[ { "path": ".dockerignore", "content": ".git\ndoc\ncodecov.yml\nCONTRIBUTING.md\nLICENSE\nREADME.md\nsamples" }, { "path": ".gitignore", "content": "noms.iml\n*.pyc\n*.swp\n.vscode\n.idea\n.noms\n.nomsconfig\n.DS_Store\nnode_modules\n" }, { "path": ".travis.yml", "content": "language: go\ngo:\n- 1.x\nos:\n- linux\nenv:\n- GO111MODULE=on\nbefore_install:\n# gox simplifies building for multiple architectures\n- go get github.com/mitchellh/gox\nscript:\n- go build ./...\n- go test ./...\n- go vet ./...\n- mkdir linux\n- mkdir darwin\n- gox -os=\"linux darwin\" -arch=\"amd64\" -output=\"{{.OS}}/{{.Dir}}\" ./cmd/noms\n- gox -os=\"linux darwin\" -arch=\"amd64\" -output=\"{{.OS}}/{{.Dir}}\" ./samples/go/csv/csv-import\n- gox -os=\"linux darwin\" -arch=\"amd64\" -output=\"{{.OS}}/{{.Dir}}\" ./samples/go/csv/csv-export\n- gox -os=\"linux darwin\" -arch=\"amd64\" -output=\"{{.OS}}/{{.Dir}}\" ./samples/go/xml-import\n- mv darwin osx\n- zip -r linux linux\n- zip -r osx osx\ndeploy:\n overwrite: true\n provider: releases\n skip_cleanup: true\n api_key:\n secure: \"N2LCdQDlPquU31TK8WZwlYPRT7SSyfsGPBpNPSp5gpJPtF5hlqLf96Fd1R7SYn/LfTcri8baFMxgPVK4FowAzIsTxwkG57vCnJR24atOFVLkaKzVPdQZ30zXDHq2WO1zYw7KzAZq49YWdzwKSShzT7+SpiNZWEE2UiB5ZSQcd7/fii1TUkphzWPeHCB+d9wf1qUyJmm6HQ3PKe9yYRQHczGin6INUV5o+nzlRws2+5Kj7eg519htLgRY0oloncY0fdwTEwbSTTkkja3eoAWQrdPMJH7mDMwpbdgPl3jW8wDdTPHO5mQHRF4GvJHrY18qMJ9Kf8iQ3bdRtIS5XM8kvo8+Le22XQbYH7Q7Ryj/bJN+71KpVLwqWQhOr3fWRrL7r8DDPAG/myw0SK1uMaXCzT3KiYckJv7Q3el9MkHNblvFNxWC4tIrwE0LtP4hbSiIlZ/MV58yJxU8WXVej9AoFnKHLA7hgJUhHy0EIlfeETalDBrqNrh40iNP0maUrhpNJxLGtpOgAUhrdQ3gd//6pWwejkfvMTQ2b+1Qq11wWsSmRI/U1WGbcO/wzjKgVv2PT2sYPgx7TPwPWje5uFTZ4/sehwGG/LDcvuZ5uBXLRcpFIz9oh31nIFzsxhdatSKaaK4zlMzkxec+xqBGe0SVKeL/rW0MeQUbeSqyAf0wtBQ=\"\n file:\n - linux.zip\n - osx.zip\n on:\n repo: attic-labs/noms\n tags: true\n" }, { "path": "CONTRIBUTING.md", "content": "Contributing to Noms\n====================\n\n## Install Go\n\nFirst setup Go on your machine per https://golang.org/doc/install. You need *at least* Go version 1.11. You can test that you're setup correctly like so:\n\n```shell\n# Must be at least 1.11\ngo version\n```\n\n## Get and build Noms\n\nNoms uses [Go modules](https://github.com/golang/go/wiki/Modules) a new feature of Go. Therefore if you're an existing Go user, you need to be careful to check out into a directory **other than $GOPATH** (or else use the environment variable GO11MODULES=on to force it on). Hopefully this gets easier to understand in future Go versions when Go modules become stabilized.\n\n```shell\ncd \ngit clone https://github.com/attic-labs/noms\ncd noms\ngo install ./cmd/noms\ngo test ./...\n```\n\n## License\n\nNoms is open source software, licensed under the [Apache License, Version 2.0](LICENSE).\n\n## Contributing code\n\nDue to legal reasons, all contributors must sign a contributor agreement, either for an [individual](https://attic-labs.github.io/ca/individual.html) or [corporation](https://attic-labs.github.io/ca/corporation.html), before a pull request can be accepted.\n\n## Languages\n\n* Use Go, JS, or Python.\n* Shell script is not allowed.\n\n## Coding style\n\n* Go uses `gofmt`, advisable to hook into your editor\n* JS follows the [Airbnb Style Guide](https://github.com/airbnb/javascript)\n* Tag PRs with either `toward: #` or `fixes: #` to help establish context for why a change is happening\n* Commit messages follow [Chris Beam's awesome commit message style guide](http://chris.beams.io/posts/git-commit/)\n\n### Go error reporting\n\nIn general, for Public API in Noms, we use the Go-style of returning errors by default.\n\nFor non-exposed code, we do provide, and use, some wrappers to do Exception-style error handling. There *must* be an overriding rationale for using this style, however. One reason to use the Exception-style is that the current code doesn't know how to proceed and needs to panic, but you want to signal that a calling function somewhere up the stack might be able to recover from the failure and continue.\n\nFor these cases, please use the following family of functions to 'raise' a 'catchable' error (see [go/d/try.go](https://godoc.org/github.com/attic-labs/noms/go/d)):\n\n\t* d.PanicIfError()\n\t* d.PanicIfTrue()\n\t* d.PanicIfFalse()\n\nYou might see some old code that uses functions that seem similar starting with `d.Chk`, however we are going to remove those and don't want to use them for new code. See #3258 for details.\n\n## Submitting PRs\n\nWe follow a code review protocol dervied from the one that the [Chromium team](https://www.chromium.org/) uses:\n\n1. Create a GitHub fork of the repo you want to modify (e.g., fork `https://github.com/attic-labs/noms` to `https://github.com//noms`).\n2. Add your own fork as a remote to your github repo: `git remote add https://github.com//noms`.\n3. Push your changes to a branch at your fork: `git push `\n4. Create a PR using the branch you just created. Usually you can do this by just navigating to https://github.com/attic-labs/noms in a browser - GitHub recognizes the new branch and offers to create a PR for you.\n5. When you're ready for review, make a comment in the issue asking for a review. Sometimes people won't review until you do this because we're not sure if you think the PR is ready for review.\n6. Iterate with your reviewer using the normal Github review flow.\n7. Once the reviewer is happy with the changes, they will submit them.\n\n## Running the tests\n\nYou can use `go test` command, e.g:\n\n* `go test $(go list ./... | grep -v /vendor/)` should run every test except from vendor packages.\n\nIf you have commit rights, Jenkins automatically runs the Go tests on every PR, then every subsequent patch. To ask Jenkins to immediately run, any committer can reply (no quotes) \"Jenkins: test this\" to your PR.\n\n### Perf tests\n\nBy default, neither `go test` nor Jenkins run the perf tests, because they take a while.\n\nTo run the tests yourself, use the `-perf` and `-v` flag to `go test`, e.g.:\n\n* `go test -v ./samples/go/csv/... -perf mem`\n\nSee https://godoc.org/github.com/attic-labs/noms/go/perf/suite for full documentation and flags.\n\nTo ask Jenkins to run the perf tests for you, reply (no quotes) \"Jenkins: perf this\" to your PR. Your results will be viewable at http://perf.noms.io/?ds=http://demo.noms.io/perf::pr_$your-pull-request-number/csv-import. Again, only a committer can do this.\n" }, { "path": "Dockerfile", "content": "FROM golang:latest AS build\n\nENV NOMS_SRC=$GOPATH/src/github.com/attic-labs/noms\nENV CGO_ENABLED=1\nENV GOOS=linux\nENV DOCKER=1\n\nRUN mkdir -pv $NOMS_SRC\nCOPY . ${NOMS_SRC}\nRUN go test github.com/attic-labs/noms/...\nRUN go install -v github.com/attic-labs/noms/cmd/noms\nRUN cp $GOPATH/bin/noms /bin/noms\n\nFROM alpine:latest\n\nCOPY --from=build /bin/noms /bin/noms\n\nVOLUME /data\nEXPOSE 8000\n\nENTRYPOINT [ \"noms\" ]\n\nCMD [\"serve\", \"/data\"]\n" }, { "path": "HACKING.md", "content": "# Prerequisites\n\n* [Go 1.13 or later](https://golang.org/dl/)\n* Mac or Linux (Noms isn't currently supported on Windows)\n\n# Get\n\n```\ngit clone https://github.com/attic-labs/noms\n```\n\n# Build\n\n```\ncd noms\ngo build ./cmd/noms\n```\n\n# Test\n\n```\ncd noms\ngo test ./go/...\ngo test ./cmd/...\n```\n\n# Release\n\nTravis automatically creates releases for tagged versions, so the following should do it:\n\n```\ngit tag latest -f\ngit push origin latest\n```\n" }, { "path": "LICENSE", "content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"{}\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright {yyyy} {name of copyright owner}\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "README.md", "content": "# Warning - This project is not active\n\nNoms is not being maintained. You shouldn't use it, except maybe for fun or research.\n\nIf you are interested in something like Noms, you probably want Dolt (https://github.com/dolthub/dolt) which is a fork of this project and actively maintained.\n\nSend me (aaron at aaronboodman.com) a message if you have questions.\n\n
\n\n\n\n\n[Use Cases](#use-cases)  |  [Setup](#setup)  |  [Status](#status)  |  [Documentation](./doc/intro.md)  |  [Contact](#contact-us)\n

\n\n[![Build Status](https://travis-ci.org/attic-labs/noms.svg?branch=master)](https://travis-ci.org/attic-labs/noms)\n[![Docker Build Status](https://img.shields.io/docker/build/noms/noms.svg)](https://hub.docker.com/r/noms/noms/)\n[![GoDoc](https://godoc.org/github.com/attic-labs/noms?status.svg)](https://godoc.org/github.com/attic-labs/noms)\n\n# Welcome\n\n*Noms* is a decentralized database philosophically descendant from the Git version control system.\n\nLike Git, Noms is:\n\n* **Versioned:** By default, all previous versions of the database are retained. You can trivially track how the database evolved to its current state, easily and efficiently compare any two versions, or even rewind and branch from any previous version.\n* **Synchronizable:** Instances of a single Noms database can be disconnected from each other for any amount of time, then later reconcile their changes efficiently and correctly.\n\nUnlike Git, Noms is a database, so it also:\n\n* Primarily **stores structured data**, not files and directories (see: [the Noms type system](https://github.com/attic-labs/noms/blob/master/doc/intro.md#types))\n* **Scales well** to large amounts of data and concurrent clients\n* Supports **atomic transactions** (a single instance of Noms is CP, but Noms is typically run in production backed by S3, in which case it is \"[effectively CA](https://cloud.google.com/spanner/docs/whitepapers/SpannerAndCap.pdf)\")\n* Supports **efficient indexes** (see: [Noms prolly-trees](https://github.com/attic-labs/noms/blob/master/doc/intro.md#prolly-trees-probabilistic-b-trees))\n* Features a **flexible query model** (see: [GraphQL](./go/ngql/README.md))\n\nA Noms database can reside within a file system or in the cloud:\n\n* The (built-in) [NBS](./go/nbs) `ChunkStore` implementation provides two back-ends which provide persistence for Noms databases: one for storage in a file system and one for storage in an S3 bucket.\n\nFinally, because Noms is content-addressed, it yields a very pleasant programming model.\n\nWorking with Noms is ***declarative***. You don't `INSERT` new data, `UPDATE` existing data, or `DELETE` old data. You simply *declare* what the data ought to be right now. If you commit the same data twice, it will be deduplicated because of content-addressing. If you commit _almost_ the same data, only the part that is different will be written.\n\n
\n\n## Use Cases\n\n#### [Decentralization](./doc/decent/about.md)\n\nBecause Noms is very good at sync, it makes a decent basis for rich, collaborative, fully-decentralized applications.\n\n#### Mobile Offline-First Database\n\nEmbed Noms into mobile applications, making it easier to build offline-first, fully synchronizing mobile applications.\n\n
\n\n## Install\n\n1. Download the latest release:\n - [**Linux**](https://github.com/attic-labs/noms/releases/download/latest/linux.zip)\n - [**Mac OS**](https://github.com/attic-labs/noms/releases/download/latest/osx.zip)\n2. Unzip the directory somewhere and add it to your `$PATH`\n3. Verify Noms is installed correctly:\n\n```\n$ noms version\nformat version: 7.18\nbuilt from \n```\n\n
\n\n## Run\n\nImport some data:\n\n```shell\ngo install github.com/attic-labs/noms/samples/go/csv/csv-import\ncurl 'https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv?accessType=DOWNLOAD' > /tmp/data.csv\ncsv-import /tmp/data.csv /tmp/noms::nycdemo\n```\n\nExplore:\n\n```shell\nnoms show /tmp/noms::nycdemo\n```\n\nShould show:\n\n```go\nstruct Commit {\n meta: struct Meta {\n date: \"2017-09-19T19:33:01Z\",\n inputFile: \"/tmp/data.csv\",\n },\n parents: set {},\n value: [ // 236 items\n struct Row {\n countAmericanIndian: \"0\",\n countAsianNonHispanic: \"3\",\n countBlackNonHispanic: \"21\",\n countCitizenStatusTotal: \"44\",\n countCitizenStatusUnknown: \"0\",\n countEthnicityTotal: \"44\",\n...\n```\n\n
\n\n## Status\n\nNobody is working on this right now. You shouldn't rely on it unless you're willing to take over development yourself.\n\n### Major Open Issues\n\nThese are the major things you'd probably want to fix before relying on this for most systems.\n\n* Sync performance with long commit chains (https://github.com/attic-labs/noms/issues/2233)\n* Migration (https://github.com/attic-labs/noms/issues/3363)\n* Garbage Collection (https://github.com/attic-labs/noms/issues/3374)\n* Query language\n * We started trying to hack in GraphQL but it's incomplete and maybe not the right thing. See: [ngql](./go/ngql/README.md)\n* [Various other smaller bugs and improvements](https://github.com/attic-labs/noms/issues?q=is%3Aissue+is%3Aopen+label%3AP0)\n\n
\n\n## Learn More About Noms\n\nFor the decentralized web: [The Decentralized Database](doc/decent/about.md)\n\nLearn the basics: [Technical Overview](doc/intro.md)\n\nTour the CLI: [Command-Line Interface Tour](doc/cli-tour.md)\n\nTour the Go API: [Go SDK Tour](doc/go-tour.md)\n\n
\n\n## Contact Us\n\nInterested in using Noms? Awesome! We would be happy to work with you to help understand whether Noms is a fit for your problem. Reach out at:\n\n- [Mailing List](https://groups.google.com/forum/#!forum/nomsdb)\n- [Twitter](https://twitter.com/nomsdb)\n\n## Licensing\n\nNoms is open source software, licensed by Attic Labs, Inc. under the Apache License, Version 2.0.\n" }, { "path": "cmd/util/kingpin_command.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage util\n\nimport \"github.com/attic-labs/kingpin\"\n\ntype KingpinHandler func(input string) (exitCode int)\ntype KingpinCommand func(*kingpin.Application) (*kingpin.CmdClause, KingpinHandler)\n" }, { "path": "codecov.yml", "content": "codecov:\n branch: master\n bot: \"mikegray\"\n ci:\n - \"jenkins3.noms.io\"\n\ncoverage:\n precision: 2 # how many decimal places to display in the UI: 0 <= value <= 4\n round: down # how coverage is rounded: down/up/nearest\n range: 70...100 # custom range of coverage colors from red -> yellow -> green\n\n notify:\n slack:\n default:\n url: \"secret:n+BYhIXTXsaCiMKB3vOf6yP68ytdKd3WpXtJFWPEUsEWXDiGnU5dTB5DO2yv8tR0COdxvs7K31hVpEfHEXdoXOaQhUw3FKf3fh8KZDLN7CGTbeDhw1uNGGyBr2d2TWnopzYtcXomdwMmuckARtiWQx0YXJiZY9YyCrIoDK9HIJQ=\"\n branches: null\n threshold: 5.0\n attachments: \"tree, diff\"\n\n status:\n project:\n default:\n enabled: yes\n target: auto\n branches: null\n threshold: null\n if_no_uploads: error\n if_not_found: success\n if_ci_failed: error\n\n patch:\n default:\n enabled: yes\n target: auto\n branches: null\n threshold: null\n if_no_uploads: error\n if_not_found: success\n if_ci_failed: error\n\n changes:\n default:\n enabled: yes\n branches: null\n if_no_uploads: error\n if_not_found: success\n if_ci_failed: error\n\ncomment:\n layout: \"tree\"\n branches: null\n behavior: default\n" }, { "path": "doc/cli-tour.md", "content": "[Home](../README.md) »\n\n[Technical Overview](intro.md)  |  [Use Cases](../README.md#use-cases)  |  **Command-Line Interface**  |  [Go bindings Tour](go-tour.md) |  [Path Syntax](spelling.md)  |  [FAQ](faq.md) \n

\n# A Short Tour of the Noms CLI\n\nThis is a quick introduction to the Noms command-line interface. It should only take a few minutes to read. There's also a screencast covering some of this if you prefer:\n\n[](https://www.youtube.com/watch?v=ncITL4xdXh4)\n\n## Install Noms\n\n... if you haven't already. Follow the instructions [here](https://github.com/attic-labs/noms#setup).\n\n## The `noms` command\n\nNow you should be able to run `noms`:\n\n```shell\n> noms\nNoms is a tool for goofing with Noms data.\n\nUsage:\n\n noms command [arguments]\n\nThe commands are:\n\n diff Shows the difference between two objects\n ds Noms dataset management\n log Displays the history of a Noms dataset\n serve Serves a Noms database over HTTP\n show Shows a serialization of a Noms object\n sync Moves datasets between or within databases\n version Display noms version\n\nUse \"noms help [command]\" for more information about a command.\n```\n\nWithout any arguments, `noms` lists out all available commands. To get information on a specific command, we can use `noms help [command]`:\n\n```shell\n> noms help sync\nusage: noms sync [options] \n\nSee Spelling Objects at https://github.com/attic-labs/noms/blob/master/doc/spelling.md for details on the object and dataset arguments.\n\n...\n```\n\n## noms ds\n\nThe `noms ds` command lists the _datasets_ within a particular database:\n\n```shell\n> noms ds http://demo.noms.io\n...\nsf-film-locations/raw\nsf-film-locations\n...\n```\n\n## noms log\n\nNoms datasets are versioned. You can see the history with `log`:\n\n```shell\n> !? noms log http://demo.noms.io::sf-film-locations\ncommit aprsmg0j2eegk8eehbgj7cd3tmmd1be8\nParent: None\nDate: \"2017-09-19T21:42:46Z\"\nInputPath: \"http://localhost:8000::#dksek6tuf8ens06bi4culq85tfp5q4cg.value\"\n\n...\n```\n\nNote that Noms is a typed system. What is being shown here for each entry is not text, but a serialization of the diff between two datasets.\n\n## noms show\n\nYou can see the entire serialization of any object in the database with `noms show`:\n\n```shell\n> noms show 'http://demo.noms.io::#aprsmg0j2eegk8eehbgj7cd3tmmd1be8'\n\nstruct Commit {\n meta: struct {},\n parents: Set>>,\n value: List,\n}({\n meta: Meta {\n date: \"2016-07-25T18:34:00+0000\",\n inputPath: \"http://localhost:8000::sf-film-locations/raw.value\",\n },\n parents: {\n c506ta03786j48a07he83ju669u78qa2,\n },\n value: [ // 1,241 items\n Row {\n Actor1: \"Siddarth\",\n...\n```\n\n## noms sync\n\nYou can work with Noms databases that are remote exactly the same as you work with local databases. But it's frequently useful to move data to a local machine, for example, to make a private fork or to work with the data disconnected from the source database.\n\nMoving data in Noms is done with the `sync` command. Note that unlike Git, we do not make a distinction between _push_ and _pull_. It's the same operation in both directions:\n\n```shell\n> noms sync http://demo.noms.io::sf-film-locations /tmp/noms::films\n> noms ds /tmp/noms\nfilms\n```\n\nWe can now make an edit locally:\n\n```shell\n> go install github.com/attic-labs/noms/samples/go/csv/...\n> csv-export /tmp/noms::films > /tmp/film-locations.csv\n```\n\nopen /tmp/film-location.csv and edit it, then:\n\n```shell\n> csv-import --column-types=String,String,String,String,String,String,String,String,Number,String,String \\\n /tmp/film-locations.csv /tmp/noms::films\n```\n\n## noms diff\n\nThe `noms diff` command can show you the differences between any two values. Let's see our change:\n\n```shell\n> noms diff http://demo.noms.io::sf-film-locations /tmp/noms::films\n\n./.meta {\n- \"date\": \"2016-07-25T18:51:23+0000\"\n+ \"date\": \"2016-07-25T22:51:14+0000\"\n+ \"inputFile\": \"/tmp/film-locations.csv\"\n- \"inputPath\": \"http://demo.noms.io::sf-film-locations/raw.value\"\n./.parents {\n- pckdvpvr9br1fie6c3pjudrlthe7na18\n+ q4jcc2i7kntkjiipvjgpr5r02ldroj0g\n }\n./.value[0] {\n- \"Locations\": \"Epic Roasthouse (399 Embarcadero)\"\n+ \"Locations\": \"Epic Roadhouse (399 Embarcadero)\"\n```\n" }, { "path": "doc/decent/about.md", "content": "[Home](../../README.md) » [Use Cases](../../README.md#use-cases) » **Decentralized** »\n\n**About**  |  [Quickstart](quickstart.md)  |  [Architectures](architectures.md)  |  [P2P Chat Demo](demo-p2p-chat.md)  |  [IPFS Chat Demo](demo-ipfs-chat.md)\n

\n# Noms — The Decentralized Database\n\n[Noms](http://noms.io) makes it ~~easy~~ tractable to create rich,\nmultiuser, collaborative, fully-decentralized applications.\n\nLike most databases, Noms features a rich data model, atomic\ntransactions, support for large-scale data, and efficient searches,\nscans, reads, and updates.\n\nUnlike any other database, Noms has built-in multiparty sync and\nconflict resolution. This feature makes Noms a very good fit for P2P\ndecentralized applications.\n\nAny number of dapp peers in a P2P network can\nconcurrently modify the same logical Noms database, and continuously\nand efficiently sync their changes with each other. All peers will\nconverge to the same state.\n\nFor many applications, peers can store an entire local copy of the\ndata they are interested in. For larger applications, it should be\npossible to back Noms by a decentralized blockstore like IPFS, Swarm,\nor Sia (or in the future, Filecoin), and store large-scale data in a\ncompletely decentralized way, without replicating it on every\nnode. Noms also has a blockstore for S3, which is ideal for\napplications that have some centralized components.\n\n**We'd love to talk to you about the possibility of using noms in your project** so please don't hestitate to contact us at [noms@attic.io](mailto:noms@attic.io). \n\n## How it Works\n\nThink of Noms like a programmable Git: changes are bundled as commits\nwhich reference previous states of the database. Apps pull changes\nfrom peers and merge them using a principled set of APIs and\nstrategies. Except that rather than users manually pulling and\nmerging, applications typically do this continuously, automatically\nconverging to a shared state.\n\nYour application uses a [Go client\nlibrary](https://github.com/attic-labs/noms/blob/master/doc/go-tour.md)\nto interact with Noms data. There is also a [command-line\ninterface](https://github.com/attic-labs/noms/blob/master/doc/cli-tour.md)\nfor working with data and initial support for a [GraphQL-based query\nlanguage](https://github.com/attic-labs/noms/blob/master/go/ngql/README.md).\n\nSome additional features include:\n* **Versioning**: It’s easy to use, compare, or revert to older database versions\n* **Efficient diffs**: diffing even huge datasets is efficient due to\n noms’ use of a novel BTree-like data structure called a [Prolly\n Tree](https://github.com/attic-labs/noms/blob/master/doc/intro.md#prolly-trees-probabilistic-b-trees)\n* **Efficient storage**: data are chunked and content-addressable, so\n there is exactly one copy of each chunk in the database, shared by\n other data that reference it. Small changes to massive data\n structures always result in small operations.\n* **Verifiable**: The entire database rolls up to a single 20-byte hash\n that uniquely represents the database at that moment - anyone can\n verify that a particular database hashes to the same value\n\nRead the [Noms design overview](https://github.com/attic-labs/noms/blob/master/doc/decent/intro.md).\n\n## Status\n\nFor overall status of the database, see [Noms Status](../../README.md#status).\n\nFor the decentralized use case in particular: we are fairly confident in this approach and are actively looking for partners to work with to build it out.\n\n- [x] Demonstrate core concept of using Noms to continuously sync across many users (Done! See noms-chat demos)\n- [ ] Demonstrate using libp2p or similar to traverse NATs\n- [ ] Investigate backing IPFS with Noms rather than the reverse - this should improve stability and dramatically improve local performance\n- [ ] Demonstrate using IPFS with a schema that permits nodes to disappear\n\n**_If you would like to use noms in your project we’d love to hear from you_**:\ndrop us an email ([noms@attic.io](mailto:noms@attic.io)) or send us a\nmessage in slack ([slack.noms.io](http://slack.noms.io)).\n" }, { "path": "doc/decent/architectures.md", "content": "[Home](../../README.md) » [Use Cases](../../README.md#use-cases) » **Decentralized** »\n\n[About](about.md)  |  [Quickstart](quickstart.md)  | **Architectures**  |  [P2P Chat Demo](demo-p2p-chat.md)  |  [IPFS Chat Demo](demo-ipfs-chat.md)\n

\n\n# Architectures\n\nThere are many possible ways to use Noms as part of a decentralized application. Noms can naturally be mixed and matched with other decentralized tools like blockchains, IPFS, etc. This page lists a few approaches we find promising.\n\n## Classic P2P Architecture\n\nNoms can be used to implement apps in a peer-to-peer configuration. Each instance of the application (i.e., each \"node\") maintains a database locally with the data that is relevant to it. When a node creates new data, it commits that data to it's database and broadcasts a message to it's peers that contains the hash of it's lastest commit.\n \n![P2P Architecture](./p2p-arch.png)\n \nPeers that are listening for these message can decide if that data is relevant to them. Those that are interested can pull the new data from the publisher. The two clients efficiently communicate so that only data that isn't present in the requesting client is transmitted (much the same way that one git client sends source changes to another).\n\nPeers can use a flow similar to the following in order to sync changes with one another:\n\n```nohighlight\nfor {\n listen for new message\n if new msg is relevant {\n if new msg is ancestor of current commit {\n // nothing to do\n continue\n }\n pull new data from sender of msg\n if current head is ancestor of new msg {\n // fast forward to the new commit\n set head of dataset to new commit\n continue\n }\n merge new with current head and commit\n publish new commit\n }\n}\n```\n\nNoms has a default [merge policy](https://github.com/attic-labs/noms/blob/2d0e9e738370d49cc09e8fa6e290ceca1c3e2005/go/merge/three_way.go#L14) that covers many classes of concurrent operations. If the application restricts itself to only operations that are mergeable by this policy, then Noms can automatically merge all concurrent changes. In this case, the entire database is effectively a CRDT.\n\nIf this is not sufficient, then applications can create their own merge policies, implementing whatever merge is appropriate for their use case.\n\n# Decentralized Chunkstore Architecture\n\nAnother potential architecture for decentralized apps uses a decentralized chunkstore (such as IPFS, Swarm, or Sia) rather than local databases. In this case, rather than each node maintaining a local datastore, Noms chunks are stored in a decentralized chunkstore. The underlying chunkstore is responsible for making chunks available when needed. \n\n![Decentralized Architecture](./dist-arch.png)\n\nThe flow used by peers to sync with one another is similar to the peer-to-peer architecture. The main difference is data is not duplicated on local machines and doesn't have to be pulled during sync. Each app keeps track of it's latest commit in the chunk store.\n\n```nohighlight\nfor {\n listen for new message\n if new msg is relevant {\n if new msg is ancestor of current commit {\n // nothing to do\n continue\n }\n // No pull necessary\n if current head is ancestor of new msg {\n // fast forward to the new commit\n set head of dataset to new commit\n continue\n }\n merge new with current head and commit\n publish new commit\n }\n}\n```\nWe have a prototype implementation of an IPFS-based chunkstore. If you are interested in pursuing this direction, let us know!\n" }, { "path": "doc/decent/demo-ipfs-chat.md", "content": "[Home](../../README.md) » [Use Cases](../../README.md#use-cases) » **Decentralized** »\n\n[About](about.md)  |  [Quickstart](quickstart.md)  |  [Architectures](architectures.md)  |  [P2P Chat Demo](demo-p2p-chat.md)  |  **IPFS Chat Demo**\n

\n# Demo App: IPFS-based Decentralized Chat\n\nThis sample app demonstrates backing a P2P noms app by a decentralized blockstore (in this case, IPFS). Data is pulled off the network dynamically as needed - each client doesn't need a complete copy.\n\n# Build and Run\n\nDemo app code is in the\n[ipfs-chat](https://github.com/attic-labs/noms/tree/master/samples/go/decent/ipfs-chat/)\ndirectory. To get it up and running take the following steps:\n\n* Use git to clone the noms repository onto your computer:\n\n```shell\ngo get github.com/attic-labs/noms/samples/go/decent/ipfs-chat\n```\n\n* From the noms/samples/go/decent/ipfs-chat directory, build the program with the following command:\n\n```shell\ngo build\n```\n\n* Run the ipfs-chat client with the following command:\n\n```shell\n./ipfs-chat client --username --node-idx=1 ipfs:/tmp/ipfs1::chat >& /tmp/err1\n```\n\n* Run a second ipfs-chat client with the following command:\n\n```shell\n./ipfs-chat client --username --node-idx=2 ipfs:/tmp/ipfs2::chat >& /tmp/err2\n```\n \nIf desired, ipfs-chat can be run as a daemon which will replicate all\nchat content in a local store which will enable clients to go offline\nwithout causing data to become unavailable to other clients:\n\n```shell\n./ipfs-chat daemon --node-idx=3 ipfs:/tmp/ipfs3::chat\n```\n\nNote: the 'node-idx' argument ensures that each IPFS-based program\nuses a distinct set of ports. This is useful when running multiple\nIPFS-based programs on the same machine.\n" }, { "path": "doc/decent/demo-p2p-chat.md", "content": "[Home](../../README.md) » [Use Cases](../../README.md#use-cases) » **Decentralized** »\n\n[About](about.md)  |  [Quickstart](quickstart.md)  |  [Architectures](architectures.md)  |  **P2P Chat Demo**  |  [IPFS Chat Demo](demo-ipfs-chat.md)\n

\n# Demo App: P2P Decentralized Chat\n\nThis sample demonstrates the simplest possible case of building a p2p app on top of Noms. Each node stores a complete copy of the data it is interested in, and peers find each other using [IPFS pubsub](https://ipfs.io/blog/25-pubsub/).\n\nCurrently, nodes have to have a publicly routable IP, but it should be possible to use [libP2P](https://github.com/libp2p) or similar to connect to most nodes.\n\n# Build and Run\n\nDemo app code is in the\n[p2p](https://github.com/attic-labs/noms/tree/master/samples/go/decent/p2p-chat)\ndirectory. To get it up and running take the following steps:\n\n* Use git to clone the noms repository onto your computer:\n\n```shell\ngo get github.com/attic-labs/noms/samples/go/decent/p2p-chat\n```\n\n* From the noms/samples/go/decent/p2p-chat directory, build the program with the following command:\n\n```shell\ngo build\n```\n\n* Run the p2p client with the following command:\n\n```shell\nmkdir /tmp/noms1\n./p2p-chat client --username= --node-idx=1 /tmp/noms1 >& /tmp/err1\n```\n\n* Run a second p2p client with the following command:\n\n```shell\nmkdir /tmp/noms2\n./p2p-chat client --username= --node-idx=2 /tmp/noms2 >& /tmp/err2\n```\n \nNote: the p2p client relies on IPFS for it's pub/sub implementation. The\n'node-idx' argument ensures that each IPFS-based node uses a distinct set\nof ports. This is useful when running multiple IPFS-based programs on\nthe same machine.\n" }, { "path": "doc/decent/quickstart.md", "content": "[Home](../../README.md) » [Use Cases](../../README.md#use-cases) » **Decentralized** »\n\n[About](about.md)  |  **Quickstart**  |  [Architectures](architectures.md)  |  [P2P Chat Demo](demo-p2p-chat.md)  |  [IPFS Chat Demo](demo-ipfs-chat.md)\n

\n# How to Use Noms in a Decentralized App\n\nIf you’d like to use noms in your project we’d love to hear from you:\ndrop us an email ([noms@attic.io](mailto:noms@attic.io)) or send us a\nmessage in slack ([slack.noms.io](http://slack.noms.io)).\n\nThe steps you’ll need to take are:\n\n1. Decide how you’ll model your problem using noms’ datatypes: boolean,\n number, string, blob, map, list, set, structs, ref, and\n union. (Note: if you are interested in using CRDTs as an alternative\n to classic datatypes please let us know.)\n2. Consider...\n * How peers will discover each other\n * How peers will notify each other of changes\n * How and when they will pull changes, and \n * What potential there is for conflicting changes. Consider modeling\n your problem so that changes commute in order to make merging\n easier. \n\n In our [p2p sample](https://github.com/attic-labs/noms/blob/master/doc/decent/demo-p2p-chat.md) application, all peers periodically broadcast their HEAD on a known channel using [IPFS pubsub](https://ipfs.io/blog/25-pubsub/), pull each others' changes immediately, and avoid conflicts by using operations that can be resolved with Noms' built in merge policies.\n \n This is basically the simplest possible approach, but lots of options are possible. For example, an alternate approach for discoverability could be to keep a registry of all participating nodes in a blockchain (e.g., by storing them in an Ethereum smart contract). One could store either the current HEAD of each node (updated whenever the node changes state), or just an IPNS name that the node is writing to.\n \n As an example of changes that commute consider modeling a stream\n of chat messages. Appending messages from both parties to a list\n is not commutative; the result depends on the order in which\n messages are added to the list. An example of a commutative\n strategy is adding the messages to a `Map` keyed by\n `Struct{sender, ordinal}`: the resulting `Map` is the same no\n matter what order messages are added.\n\n3. Vendor the code into your project. \n4. Decide which type of storage you'd like to use: memory (convenient for playing around), disk, IPFS, or S3. (If you want to implement a store on top of another type of storage that's possible too; email us or reach out on slack and we can help.)\n5. Set up and instantiate a database for your storage. Generally, you use the spec package to parse a [dataset spec](https://github.com/attic-labs/noms/blob/master/doc/spelling.md) like `mem::mydataset` which you can then ask for [`Database`](https://github.com/attic-labs/noms/blob/master/go/datas/database.go) and [`Dataset`](https://github.com/attic-labs/noms/blob/master/go/datas/dataset.go).\n * **Memory**: no setup required, just instantiate it:\n\n```go\nsp := spec.ForDataset(\"mem::test\") // Dataset name is \"test\"\n```\n \n * **Disk**: identify a directory for storage, say `/path/to/chunks`, and then instantiate:\n \n```go\nsp := spec.ForDataset(\"/path/to/chunks::test\") // Dataset name is \"test\"\n```\n \n * **IPFS**: identify an IPFS node by directory. If an IPFS node doesn't exist at that directory, one will be created:\n\n```go\nsp := spec.ForDataset(\"ipfs:/path/to/ipfs_repo::test\") // Dataset name is \"test\"\n```\n\n * **S3**: Follow the [S3 setup instructions](https://github.com/attic-labs/noms/blob/master/go/nbs/NBS-on-AWS.md) then instantiate a database and dataset:\n\n```go\nsess := session.Must(session.NewSession(aws.NewConfig().WithRegion(\"us-west-2\")))\nstore := nbs.NewAWSStore(\"dynamo-table\", \"store-name\", \"s3-bucket\", s3.New(sess), dynamodb.New(sess), 1<<28))\ndatabase := datas.NewDatabase(store)\ndataset := database.GetDataset(\"aws://dynamo-table:s3-bucket/store-name::test\") // Dataset name is \"test\"\n```\n\n7. Implement using the [Go API](https://github.com/attic-labs/noms/blob/master/doc/go-tour.md). If you're just playing around you could try something like this:\n\n```go\n package main\n \n import (\n \"fmt\"\n \"os\"\n \n \"github.com/attic-labs/noms/go/spec\"\n \"github.com/attic-labs/noms/go/types\"\n )\n \n // Usage: quickstart /path/to/store::ds\n func main() {\n sp, err := spec.ForDataset(os.Args[1])\n if err != nil {\n fmt.Fprintf(os.Stderr, \"Unable to parse spec: %s, error: %s\\n\", sp, err)\n os.Exit(1)\n }\n defer sp.Close()\n \n db := sp.GetDatabase()\n if headValue, ok := sp.GetDataset().MaybeHeadValue(); !ok {\n data := types.NewList(sp.GetDatabase(),\n newPerson(\"Rickon\", true),\n newPerson(\"Bran\", true),\n newPerson(\"Arya\", false),\n newPerson(\"Sansa\", false),\n )\n \n fmt.Fprintf(os.Stdout, \"data type: %v\\n\", types.TypeOf(data).Describe())\n _, err = db.CommitValue(sp.GetDataset(), data)\n if err != nil {\n fmt.Fprint(os.Stderr, \"Error commiting: %s\\n\", err)\n os.Exit(1)\n }\n } else {\n // type assertion to convert Head to List\n personList := headValue.(types.List)\n // type assertion to convert List Value to Struct\n personStruct := personList.Get(0).(types.Struct)\n // prints: Rickon\n fmt.Fprintf(os.Stdout, \"given: %v\\n\", personStruct.Get(\"given\"))\n }\n }\n \n func newPerson(givenName string, male bool) types.Struct {\n return types.NewStruct(\"Person\", types.StructData{\n \"given\": types.String(givenName),\n \"male\": types.Bool(male),\n })\n }\n```\n\n8. You can inspect data that you've committed via the [noms command-line interface](https://github.com/attic-labs/noms/blob/master/doc/cli-tour.md). For example:\n\n```shell\nnoms log /path/to/store::ds\nnoms show /path/to/store::ds\n```\n\n> Note that Memory tables won't be inspectable because they exist only in the memory of the process that created them. \n\n9. Implement pull and merge. The [pull API](../../go/datas/pull.go) is used pull changes from a peer and the [merge API](../../go/merge/) is used to merge changes before commit. There's an [example of merging in the IPFS-based-chat sample\n app](https://github.com/attic-labs/noms/blob/master/samples/go/ipfs-chat/pubsub.go). \n" }, { "path": "doc/faq.md", "content": "[Home](../README.md) »\n\n[Technical Overview](intro.md)  |  [Use Cases](../README.md#use-cases)  |  [Command-Line Interface](cli-tour.md)  |  [Go bindings Tour](go-tour.md) |  [Path Syntax](spelling.md)  |  **FAQ** \n

\n# Frequently Asked Questions\n\n### Decentralized like BitTorrent?\n\nNo, decentralized like Git.\n\nSpecifically, Noms isn't itself a peer-to-peer network. If you can get two instances to share data, somehow, then they can synchronize. Noms doesn't define how this should happen though.\n\nCurrently, instances mainly share data via either HTTP/DNS or a filesystem. But it should be easy to add other mechanisms. For example, it seems like Noms could run well on top of BitTorrent, or IPFS. You should [look into it](https://github.com/attic-labs/noms/issues/2123).\n\n### Isn't it wasteful to store every version?\n\nNoms deduplicates chunks of data that are identical within one database. So if multiple versions of one dataset share a lot of data, or if the same data is present in multiple datasets, Noms only stores one copy.\n\nThat said, it is definitely possible to have write patterns that defeat this. Deduplication is done at the chunk level, and chunks are currently set to an average size of 4KB. So if you change about 1 byte in every 4096 in a single commit, and those changed bytes are well-distributed throughout the dataset, then we will end up making a complete copy of the dataset.\n\n### Is there a way to not store the entire history?\n\nTheoretically, definitely. In Git, for example, the concept of \"shallow clones\" exists, and we could do something similar in Noms. This has not been implemented yet.\n\n### How does Noms handle conflicts?\n\nNoms provides several built-in policies that can automatically merge common cases of conflicts. For example concurrent edits to sets are always mergeable and concurrent edits to different keys in a map or struct are also mergeable.\n\nThe conflict resolution system is pluggable so new policies that are application-specific can be added. However, it's possible to build surprisingly complex applications with just the built-in policies.\n\n### Why don't you just use CRDTs?\n\n[Convergent (or Commutative) Replicated Data Types (CRDTs)](http://hal.upmc.fr/inria-00555588/document) are a class of distributed data structures that provably converge to some agreed-upon state with no synchronization. Stated differently: CRDTs define a merge policy that is commutative over all their operations.\n\nCRDTs are nice because they require no custom conflict/merge code from the developer.\n\nNoms defines a set of intutive built-in merge policies for its core datatypes. For example, the default policy makes all operations on Noms Sets commute (add wins in the case of concurrent remove/add). This means that with the default policy, Noms Sets are a CRDT.\n\nIf your application uses only operations on Noms datatypes that can be merged with whatever merge policy you are using, then your schema is a CRDT. It's possible to build surprisingly complex applications this way with just the default policy.\n\nNoms also allows you to provide your own custom policy. If your policy commutes, then the resulting datatype will be a CRDT.\n\nHowever, it would be nice if application developers could more easily opt-in to using only mergeable operations, thereby enforcing that their schema is a CRDT, and providing confidence that custom merge logic doesn't need to be implemented.\n\nMore generally, perhaps there could be a way to test that all possible conflict cases have been handled by the developer. This would allow developers to implement their own custom CRDTs. This is something we'd like to research in the future.\n\n### Why don't you support Windows?\n\nWe are a tiny team and we all personally use Macs as our development machines, and we use Linux in production. These two platforms are very close to identical, and so we can generally test on Mac and assume it will work on Linux. Adding Windows would add significant complexity to our code and build processes which we're not willing to take on.\n\n### But you'll accept patches for Windows, right?\n\nNo, because then we'll have to maintain those patches.\n\n### Are there any workaround for Windows?\n\nYou can use it in a virtual machine. We have also heard Noms works OK with gitbash or cygwin, but that's coincidence.\n\n### Why is it called Noms?\n\n1. It's insert-only. OMNOMNOM.\n2. It's content addressed. Every value has its own hash, or [name](http://dictionary.reverso.net/french-english/nom).\n\n### Are you sure Noms doesn't stand for something?\n\nPretty sure. But if you like, you can pretend it stands for Non-Mutable Store.\n" }, { "path": "doc/go-tour.md", "content": "[Home](../README.md) »\n\n[Technical Overview](intro.md)  |  [Use Cases](../README.md#use-cases)  |  [Command-Line Interface](cli-tour.md)  |  **Go bindings Tour** |  [Path Syntax](spelling.md)  |  [FAQ](faq.md) \n

\n# A Short Tour of Noms for Go\n\nThis is a short introduction to using Noms from Go. It should only take a few minutes if you have some familiarity with Go.\n\nDuring the tour, you can refer to the complete [Go SDK Reference](https://godoc.org/github.com/attic-labs/noms) for more information on anything you see.\n\n\n\n## Requirements\n\n* [Noms command-line tools](https://github.com/attic-labs/noms#setup)\n* [Go v1.6+](https://golang.org/dl/)\n* Ensure your [$GOPATH](https://github.com/golang/go/wiki/GOPATH) is configured\n\n## Start a Local Database\n\nLet's create a local database to play with:\n\n```sh\n> mkdir /tmp/noms-go-tour\n> noms serve /tmp/noms-go-tour\n```\n\n## [Database](https://github.com/attic-labs/noms/blob/master/go/datas/database.go)\nLeave the server running, and in a separate terminal:\n\n```sh\n> mkdir noms-tour\n> cd noms-tour\n```\n\nThen use your favorite editor so that we can start to play with code. To get started with Noms, first create a Database:\n\n```go\npackage main\n\nimport (\n \"fmt\"\n \"os\"\n\n \"github.com/attic-labs/noms/go/spec\"\n)\n\nfunc main() {\n sp, err := spec.ForDatabase(\"http://localhost:8000\")\n if err != nil {\n fmt.Fprintf(os.Stderr, \"Could not access database: %s\\n\", err)\n return\n }\n defer sp.Close()\n}\n```\n\nNow let's run it:\n\n```sh\n> go run noms-tour.go\n```\n\nIf you did not leave the server running you would see output of ```Could not access database``` here, otherwise your program should exit cleanly.\n\nSee [Spelling in Noms](https://github.com/attic-labs/noms/blob/master/doc/spelling.md) for more information on database spec strings.\n\n\n## [Dataset](https://github.com/attic-labs/noms/blob/master/go/dataset/dataset.go)\n\nDatasets are the main interface you'll use to work with Noms. Let's update our example to use a Dataset spec string:\n\n```go\npackage main\n\nimport (\n \"fmt\"\n \"os\"\n\n \"github.com/attic-labs/noms/go/spec\"\n)\n\nfunc main() {\n sp, err := spec.ForDataset(\"http://localhost:8000::people\")\n if err != nil {\n fmt.Fprintf(os.Stderr, \"Could not create dataset: %s\\n\", err)\n return\n }\n defer sp.Close()\n\n if _, ok := sp.GetDataset().MaybeHeadValue(); !ok {\n fmt.Fprintf(os.Stdout, \"head is empty\\n\")\n }\n}\n```\n\nNow let's run it:\n\n```sh\n> go run noms-tour.go\nhead is empty\n```\n\nSince the dataset does not yet have any values you see ```head is empty```. Let's add some data to make it more interesting:\n\n```go\npackage main\n\nimport (\n \"fmt\"\n \"os\"\n\n \"github.com/attic-labs/noms/go/spec\"\n \"github.com/attic-labs/noms/go/types\"\n)\n\nfunc newPerson(givenName string, male bool) types.Struct {\n return types.NewStruct(\"Person\", types.StructData{\n \"given\": types.String(givenName),\n \"male\": types.Bool(male),\n })\n}\n\nfunc main() {\n sp, err := spec.ForDataset(\"http://localhost:8000::people\")\n if err != nil {\n fmt.Fprintf(os.Stderr, \"Could not create dataset: %s\\n\", err)\n return\n }\n defer sp.Close()\n\n db := sp.GetDatabase()\n\n data := types.NewList(db,\n newPerson(\"Rickon\", true),\n newPerson(\"Bran\", true),\n newPerson(\"Arya\", false),\n newPerson(\"Sansa\", false),\n )\n\n fmt.Fprintf(os.Stdout, \"data type: %v\\n\", types.TypeOf(data).Describe())\n\n _, err = db.CommitValue(sp.GetDataset(), data)\n if err != nil {\n fmt.Fprint(os.Stderr, \"Error commiting: %s\\n\", err)\n }\n}\n```\n\nNow you will get output of the data type of our Dataset value:\n\n```shell\n> go run noms-tour.go\ndata type: List\n```\n\nNow you can access the data via your program:\n\n```go\npackage main\n\nimport (\n \"fmt\"\n \"os\"\n\n \"github.com/attic-labs/noms/go/spec\"\n \"github.com/attic-labs/noms/go/types\"\n)\n\nfunc main() {\n sp, err := spec.ForDataset(\"http://localhost:8000::people\")\n if err != nil {\n fmt.Fprintf(os.Stderr, \"Could not create dataset: %s\\n\", err)\n return\n }\n defer sp.Close()\n\n if headValue, ok := sp.GetDataset().MaybeHeadValue(); !ok {\n fmt.Fprintf(os.Stdout, \"head is empty\\n\")\n } else {\n // type assertion to convert Head to List\n personList := headValue.(types.List)\n // type assertion to convert List Value to Struct\n personStruct := personList.Get(0).(types.Struct)\n // prints: Rickon\n fmt.Fprintf(os.Stdout, \"given: %v\\n\", personStruct.Get(\"given\"))\n }\n}\n```\n\nRunning it now:\n\n```sh\n> go run noms-tour.go\ngiven: Rickon\n```\n\nYou can see this data using the command-line too:\n\n```sh\n> noms ds http://localhost:8000\npeople\n\n> noms show http://localhost:8000::people\nstruct Commit {\n meta: struct {},\n parents: set {},\n value: [ // 4 items\n struct Person {\n given: \"Rickon\",\n male: true,\n },\n struct Person {\n given: \"Bran\",\n male: true,\n },\n struct Person {\n given: \"Arya\",\n male: false,\n },\n struct Person {\n given: \"Sansa\",\n male: false,\n },\n ],\n}\n```\n\nLet's add some more data.\n\n```go\npackage main\n\nimport (\n \"fmt\"\n \"os\"\n\n \"github.com/attic-labs/noms/go/spec\"\n \"github.com/attic-labs/noms/go/types\"\n)\n\nfunc main() {\n sp, err := spec.ForDataset(\"http://localhost:8000::people\")\n if err != nil {\n fmt.Fprintf(os.Stderr, \"Could not create dataset: %s\\n\", err)\n return\n }\n defer sp.Close()\n\n if headValue, ok := sp.GetDataset().MaybeHeadValue(); !ok {\n fmt.Fprintf(os.Stdout, \"head is empty\\n\")\n } else {\n // type assertion to convert Head to List\n personList := headValue.(types.List)\n personEditor := personList.Edit()\n data := personEditor.Append(\n types.NewStruct(\"Person\", types.StructData{\n \"given\": types.String(\"Jon\"),\n \"family\": types.String(\"Snow\"),\n \"male\": types.Bool(true),\n }),\n ).List()\n\n fmt.Fprintf(os.Stdout, \"data type: %v\\n\", types.TypeOf(data).Describe())\n\n _, err = sp.GetDatabase().CommitValue(sp.GetDataset(), data)\n if err != nil {\n fmt.Fprint(os.Stderr, \"Error commiting: %s\\n\", err)\n }\n }\n}\n```\n\nRunning this:\n\n```sh\n> go run noms-tour.go\ndata type: List\n```\n\nDatasets are versioned. When you *commit* a new value, you aren't overwriting the old value, but adding to a historical log of values:\n\n```sh\n> noms log http://localhost:8000::people\ncommit ba3lvopbgcqqnofm3qk7sk4j2doroj1l\nParent: f0b1befu9jp82r1vcd4gmuhdno27uobi\n(root) {\n+ struct Person {\n+ family: \"Snow\",\n+ given: \"Jon\",\n+ male: true,\n+ }\n }\n\ncommit f0b1befu9jp82r1vcd4gmuhdno27uobi\nParent: hshltip9kss28uu910qadq04mhk9kuko\n\ncommit hshltip9kss28uu910qadq04mhk9kuko\nParent: None\n```\n\n## Values\n\nNoms supports a [variety of datatypes](https://github.com/attic-labs/noms/blob/master/doc/intro.md#types) beyond List, Struct, String, and Bool we used above.\n\n## Samples\n\nYou can continue learning more about the Noms Go SDK by looking at the documentation and by reviewing the [samples](https://github.com/attic-labs/noms/blob/master/samples/go). The [hr sample](https://github.com/attic-labs/noms/blob/master/samples/go/hr) is a more complete implementation of our example above and will help you to see further usage of the other datatypes.\n" }, { "path": "doc/intro.md", "content": "[Home](../README.md) »\n\n**Technical Overview**  |  [Use Cases](../README.md#use-cases)  |  [Command-Line Interface](cli-tour.md)  |  [Go bindings Tour](go-tour.md) |  [Path Syntax](spelling.md)  |  [FAQ](faq.md) \n

\n# Noms Technical Overview\n\nMost conventional database systems share two central properties:\n\n1. Data is modeled as a single point-in-time. Once a transaction commits, the previous state of the database is either lost, or available only as a fallback by reconstructing from transaction logs.\n\n2. Data is modeled as a single source of truth. Even large-scale distributed databases which are internally a fault-tolerant network of nodes, present the abstraction to clients of being a single logical master, with which clients must coordinate in order to change state.\n\nNoms blends the properties of decentralized systems, such as [Git](https://git-scm.com/), with properties of traditional databases in order to create a general-purpose decentralized database, in which:\n\n1. Any peer’s state is as valid as any other.\n\n2. All commits of the database are retained and available at any time.\n\n3. Any peer is free to move forward independently of communication from any other—while retaining the ability to reconcile changes at some point in the future.\n\n4. The basic properties of structured databases (efficient queries, updates, and range scans) are retained.\n\n5. Diffs between any two sets of data can be computed efficiently.\n\n6. Synchronization between disconnected copies of the database can be performed efficiently and correctly.\n\n## Basics\n\nAs in Git, [Bitcoin](https://bitcoin.org/en/), [Ethereum](https://www.ethereum.org/), [IPFS](https://ipfs.io/), [Camlistore](https://camlistore.org/), [bup](https://bup.github.io/), and other systems, Noms models data as a [directed acyclic graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph) of nodes in which every node has a _hash_. A node's hash is derived from the values encoded in the node and (transitively) from the values encoded in all nodes which are reachable from that node.\n\nIn other words, a Noms database is a single large [Merkle DAG](https://github.com/jbenet/random-ideas/issues/20).\n\nWhen two nodes have the same hash, they represent identical logical values and the respective subgraph of nodes reachable from each are topologically equivalent. Importantly, in Noms, the reverse is also true: a single logical value has one and only one hash. When two nodes have differnet hashes, they represent different logical values.\n\nNoms extends the ideas of prior systems to enable efficiently computing and reconciling differences, synchronizing state, and building indexes over large-scale, structured data.\n\n## Databases and Datasets\n\nA _database_ is the top-level abstraction in Noms.\n\nA database has two responsibilities: it provides storage of [content-addressed](https://en.wikipedia.org/wiki/Content-addressable_storage) chunks of data, and it keeps track of zero or more _datasets_.\n\nA Noms database can be implemented on top of any underlying storage system that provides key/value storage with at least optional optimistic concurrency. We only use optimistic concurrency to store the current value of each dataset. Chunks themselves are immutable.\n\nWe have implementations of Noms databases on top of our own file-backed store [Noms Block Store (NBS)](https://github.com/attic-labs/noms/tree/master/go/nbs) (usually used locally), our own [HTTP protocol](https://github.com/attic-labs/noms/blob/master/go/datas/database_server.go) (used for working with a remote database), [Amazon DynamoDB](https://aws.amazon.com/dynamodb/), and [memory](https://github.com/attic-labs/noms/blob/master/go/chunks/memory_store.go) (mainly used for testing).\n\nHere's an example of creating an http-backed database using the [Go Noms SDK](go-tour.md):\n\n```go\npackage main\n\nimport (\n \"fmt\"\n \"os\"\n\n \"github.com/attic-labs/noms/go/spec\"\n)\n\nfunc main() {\n sp, err := spec.ForDatabase(\"http://localhost:8000\")\n if err != nil {\n fmt.Fprintf(os.Stderr, \"Could not access database: %s\\n\", err)\n return\n }\n defer sp.Close()\n}\n```\n\nA dataset is nothing more than a named pointer into the DAG. Consider the following command to copy the dataset named `foo` to the dataset named `bar` within a database:\n\n```shell\nnoms sync http://localhost:8000::foo http://localhost:8000::bar\n```\n\nThis command is trivial and causes basically zero IO. Noms first resolves the dataset name `foo` in `http://localhost:8000`. This results in a hash. Noms then checks whether that hash exists in the destination database (which in this case is the same as the source database), finds that it does, and then adds a new dataset pointing at that chunk.\n\nSyncs across database can be efficient by the same logic if the destination database already has all or most of the chunks required chunks.\n\n## Time\n\nAll data in Noms is immutable. Once a piece of data is stored, it is never changed. To represent state changes, Noms uses a progression of `Commit` structures.\n\n[TODO - diagram]\n\nAs in Git, Commits typically have one _parent_, which is the previous commit in time. But in the cases of merges, a Noms commit can have multiple parents.\n\n### Chunks\n\nWhen a value is stored in Noms, it is stored as one or more chunks of data. Chunk boundaries are typically created implicitly, as a way to store large collections efficiently (see [Prolly Trees](#prolly-trees-probabilistic-b-trees)). Programmers can also create explicit chunk boundaries using the `Ref` type (see [Types](#types )).\n\n[TODO - Diagram]\n\nEvery chunk encodes a single logical value (which may be a component of another value and/or be composed of sub-values). Chunks are [addressed](https://en.wikipedia.org/wiki/Content-addressable_storage) in the Noms persistence layer by the hash of the value they encode.\n\n## Types\n\nNoms is a typed system, meaning that every Noms value is classified into one of the following _types_:\n\n* `Boolean`\n* `Number` (arbitrary precision binary)\n* `String` (utf8-encoded)\n* `Blob` (raw binary data)\n* `Set`\n* `List`\n* `Map`\n* Unions: `T|U|V|...`\n* `Ref` (explicit out-of-line references)\n* `Struct` (user-defined record types, e.g., `Struct Person { name: String, age?: Number })`\n* `Type` (A value that stores a Noms type)\n\nBlobs, sets, lists, and maps can be gigantic - Noms will _chunk_ these types into reasonable sized parts internally for efficient storage, searching, and updating (see [Prolly Trees](#prolly-trees-probabilistic-b-trees) below for more on this).\n\nStrings, numbers, unions, and structs are not chunked, and should be used for \"reasonably-sized\" values. Use `Ref` if you need to force a particular value to be in a different chunk for some reason.\n\nTypes serve several purposes in Noms:\n\n1. Most importantly, types make Noms data self-describing. You can use the `types.TypeOf` function on any Noms `Value`, no matter how large, and get a very precise description of the entire value and all values reachable from it. This allows software to interoperate without prior agreement or planning.\n\n2. Users of Noms can define their own structures and publish data that uses them. This allows for ad-hoc standardization of types within communities working on similar data.\n\n3. Types can be used _structurally_. A program can check incoming data against a required type. If the incoming root chunk matches the type, or is a superset of it, then the program can proceed with certainty of the shape of all accessible data. This enables richer interoperability between software, since schemas can be expanded over time as long as a compatible subset remains.\n\n4. Eventually, we plan to add type restrictions to datasets, which would enforce the allowed types that can be committed to a dataset. This would allow something akin to schema validation in traditional databases.\n\n### Refs vs Hashes\n\nA _hash_ in Noms is just like the hashes used elsewhere in computing: a short string of bytes that uniquely identifies a larger value. Every value in Noms has a hash. Noms currently uses the [sha2-512](https://github.com/attic-labs/noms/blob/master/go/hash/hash.go#L7) hash function, but that can change in future versions of the system.\n\nA _ref_ is different in subtle, but important ways. A `Ref` is a part of the type system - a `Ref` is a value. Anywhere you can find a Noms value, you can find a `Ref`. For example, you can commit a `Ref` to a dataset, but you can't commit a bare hash.\n\nThe difference is that `Ref` carries the type of its target, along with the hash. This allows us to efficiently validate commits that include `Ref`, among other things.\n\n### Type Accretion\n\nNoms is an immutable database, which leads to the question: How do you change the schema? If I have a dataset containing `Set`, and I later decide that it should be `Set`, what do I do?\n\nYou might say that you just commit the new type, but that would mean that users can't look at a dataset and understand what types previous versions contained, without manually exploring every one of those commits.\n\nWe call our solution to this problem _Type Accretion_.\n\nIf you construct a `Set` containing only `Number`s, its type will be `Set`. If you then insert a string into this set, the type of the resulting value is `Set`.\n\nThis is usually completely implicit, done based on the data you store (you can set types explicitly though, which is useful in some cases).\n\nWe do the same thing for datasets. If you commit a `Set`, the type of the commit we create for you is:\n\n```go\nStruct Commit {\n\tValue: Set\n\tParents: Set>>\n}\n```\n\nThis tells you that the current and all previous commits have values of type `Set`.\n\nBut if you then commit a `Set` to this same dataset, then the type of that commit will be:\n\n```go\nStruct Commit {\n\tValue: Set\n\tParents: Set> |\n\t\tRef\n\t\t\tParents: Cycle\n\t\t}>>\n\t}\n}\n```\n\nThis tells you that the dataset's current commit has a value of type `Set` and that previous commits are either the same, or else have a value of type `Set`.\n\nType accretion has a number of benefits related to schema changes:\n\n1. You can widen the type of any container (list, set, map) without rewriting any existing data. `Set` becomes `Set | Struct { name: String, age: Number }>>` and all existing data is reused.\n2. You can widen containers in ways that other databases wouldn't allow. For example, you can go from `Set` to `Set`. Existing data is still reused.\n3. You can change the type of a dataset in either direction - either widening or narrowing it, and the dataset remains self-documenting as to its current and previous types.\n\n## Prolly Trees: Probabilistic B-Trees\n\nA critical invariant of Noms is [history-independence](https://arxiv.org/pdf/1501.06508.pdf): the same Noms value will be represented by the same graph of physical chunks, and the same hashes, regardless of what past sequence of logical mutations resulted in the value. This is what makes fast diff, sync, and merge possible in Noms: we can compare two values just by looking at their hash. If their hashes are identical, we know the values are identical without additional work. By modeling collections as trees of values, the same trick can be used to quickly find the differences between larges sets of values.\n\nBut Noms is also a database, and needs to do what databases do: efficiently search, scan, and mutate large collections. The classic data structures that enable these features inside databases — B-Trees and LSM Trees — can't be used by Noms because they aren't history-independent: their internal state depends upon their mutation history.\n\nIn order to model large mutable collections in Noms, of the type where B-Trees would typically be used, while preserving efficient diff, sync, and merge, Noms introduces _Prolly Trees_.\n\n### Prolly Tree Structure\n\nA Prolly Tree is a [search tree](https://en.wikipedia.org/wiki/Search_tree) where the number of values stored in each node is determined probabilistically, based on the data which is stored in the tree.\n\nA Prolly Tree is similar in many ways to a B-Tree, except that the number of values in each node has a probabilistic average rather than an enforced upper and lower bound, and the set of values in each node is determined by the output of a rolling hash function over the values, rather than via split and join operations when upper and lower bounds are exceeded.\n\nLike B-Trees, Prolly Trees can model lists, maps, tables, and sets. Below is an example of a small set of ASCII characters stored in a Prolly Tree:\n\n![Prolly Tree Diagram](prolly-tree-structure.png)\n\n### Prolly Tree Construction\n\nProlly Trees can be constructed from scratch using a variation of [content-slicing](https://en.wikipedia.org/wiki/Rolling_hash#Content-based_slicing_using_a_rolling_hash), as used in bup, rsync, Camlistore, and many other systems.\n\n![Prolly Tree Construction](prolly-tree-construction.png)\n\nTo start, we \"chunk\" the serialization of a larged sorted sequence by sliding a fixed-size window through it, one byte at a time.\n\nAt each position, we compute a hash of the bytes in the window. Any hash can be used, but in Noms a [rolling hash](https://en.wikipedia.org/wiki/Rolling_hash) is used for performance.\n\nWithin each hash, we look for a pattern that has a known probability of occuring. If the pattern is found, that position is a _boundary_. We slide the window forward to the end of the containing item, and write a new _chunk_ containing the bytes between this boundary and the previous, if any. The resulting chunk is stored in a [content-addressed storage system](https://en.wikipedia.org/wiki/Content-addressable_storage). Again, any hash can be used for this, but in Noms [we use truncated SHA-512](https://github.com/attic-labs/noms/blob/master/go/hash/hash.go).\n\nBy adjusting the pattern we look for, we can control the average size of the chunks our tree will be decomposed into.\n\nIn Noms, the pattern we look for is the [12 high bits being 1](https://github.com/attic-labs/noms/blob/master/go/types/rolling_value_hasher.go). Since this has a probability of 1/2^12, the average chunk size in Noms is 4kb.\n\nOnce we've created an initial pass of chunks this way, we build an index describing the contents of each of those chunks, and perform the chunking operation again on the serialization of that index. This continues recursively, until we are left with a node that doesn't chunk. This is the root of the tree.\n\nNoms uses a window size of 64 bytes, so the probability of any 1 bit change moving a boundary is about 64/4kb ~= 0.016.\n\n### Prolly Tree Mutation\n\nTo mutate a Prolly Tree, conceptually we build a new Prolly Tree from scratch, except that we can reuse everything from the previous tree that we know cannot have been affected (because it is outside the window).\n\n![Prolly Tree Mutation](prolly-tree-mutation.png)\n\nIn the example above, we insert the value _I_ into the set. The chunk boundary is unchanged in this case so the subtrees before and after the modified chunk can be reused as-is.\n\nEvery so often (1.6% of the time in Noms) a write will move a chunk boundary. This results in one extra chunk getting written at that level in the tree. This can happen at each level, so the expected number of operations to make a change to a prolly tree is 1.016*treedepth.\n\nA 4-level Prolly Tree in Noms can hold 4096^4 ~= 281TB of data. We can make a single mutation to that tree with about 4 4kb writes.\n\n### Some Properties of Prolly Trees\n\nOperation | B-Trees | Patricia Trees† / HAMTs | Prolly Trees\n--------- | ------- | ----------------------- | ------------\n1 Random Read | 🎉logk(n) | 🎉logk(n) | 🎉logk(n)\n1 Random Write | 🎉logk(n) | 💩2*logk(n) | 👍(1+k/w)*logk(n)\nOrdered scan of one item with size z | 🎉z/k | ❌ | 🎉z/k\nCalculate diff of size d | 💩n | 🎉d | 🎉d\nVerification, proofs | ❌ | 🙌 | 🙌\nStructured sharing | ❌ | 🙌 | 🙌\n\n**†** assuming hashed keys, unhashed destroys perf — **n**: total leaf data in tree, **k**: average block size, **w**: window width\n\n### Indexing and Searching with Prolly Trees\n\nLike B-Trees, Prolly Trees are sorted. In Noms, we sort keys of type Boolean, Number, and String by their natural order. We sort other key types by their hash.\n\nBecause of this sorting, Noms collections can be used as efficient indexes, in the same manner as primary and secondary indexes in traditional databases.\n\nFor example, say you want to quickly be able to find `Person` structs by their age. You could build a map of type `Map>`. This would allow you to quickly (~logk(n) seeks) find all the people of an exact age. But it would _also_ allow you to find all people within a range of ages efficiently (~num_results/logk(n) seeks), even if the ages are non-integral.\n\nAlso, because Noms collections are ordered search trees, it is possible to implement set operations like union and intersect efficiently on them.\n\nSo, for example, if you wanted to find all the people of a particular age AND having a particular hair color, you could construct a second map having type `Map>`, and intersect the two sets.\n\nOver time, we plan to develop this basic capability into support for some kind of generalized query system.\n" }, { "path": "doc/spelling.md", "content": "[Home](../README.md) »\n\n[Technical Overview](intro.md)  |  [Use Cases](../README.md#use-cases)  |  [Command-Line Interface](cli-tour.md)  |  [Go bindings Tour](go-tour.md) |  **Path Syntax**  |  [FAQ](faq.md) \n

\n# Spelling in Noms\n\nMany commands and APIs in Noms accept database, dataset, or value specifications as arguments. This document describes how to construct these specifications.\n\n## Spelling Databases\n\ndatabase specifications take the form:\n\n```nohighlight\n[:]\n```\n\nThe `path` part of the name is interpreted differently depending on the protocol:\n\n- **http(s)** specs describe a remote database to be accessed over HTTP. In this case, the entire database spec is a normal http(s) URL. For example: `https://dev.noms.io/aa`.\n- **mem** specs describe an ephemeral memory-backed database. In this case, the path component is not used and must be empty.\n- **nbs** specs describe a local [Noms Block Store (NBS)](https://github.com/attic-labs/noms/tree/master/go/nbs)-backed database. In this case, the path component should be a relative or absolute path on disk to a directory in which to store the data, e.g. `nbs:/tmp/noms-data`.\n - In Go, `nbs:` can be ommitted (just `/tmp/noms-data` will work).\n- **aws** specs describe a remote Noms Block Store backed directly by Amazon Web Services, specifically DynamoDB and S3. The format is a URI containing the names of the DynamoDB table to use, the S3 bucket to use, and the database to serve. For example: `aws:dynamo-table/s3-bucket/database`.\n\n## Spelling Datasets\n\nDataset specifications take the form:\n\n```nohighlight\n::\n```\n\nSee [spelling databases](#spelling-databases) for how to build the `database` part of the name. The `dataset` part is just any string matching the regex `^[a-zA-Z0-9\\-_/]+$`.\n\nExample datasets:\n\n```nohighlight\n/tmp/test-db::my-dataset\nnbs:/tmp/test-db::my-dataset\nhttp://localhost:8000::registered-businesses\nhttps://demo.noms.io/aa::music\n```\n\n## Spelling Values\n\nValue specifications take the form:\n\n```nohighlight\n::\n```\n\nSee [spelling databases](#spelling-databases) for how to build the database part of the name.\n\nThe `root` part can be either a hash or a dataset name. If `root` begins with `#` it will be interpreted as a hash otherwise it is used as a dataset name. See [spelling datasets](#spelling-datasets) for how to build the dataset part of the name.\n\nThe `path` part is relative to the `root` provided.\n\n### Specifying Struct Fields\nElements of a Noms struct can be referenced using a period `.`.\n\nFor example, if the `root` is a dataset, then one can use `.value` to get the root of the data in the dataset. In this case `.value` selects the `value` field from the `Commit` struct at the top of the dataset. One could instead use `.meta` to select the `meta` struct from the `Commit` struct. The `root` does not need to be a dataset though, so if it is a hash that references a struct, the same notation still works: `#o38hugtf3l1e8rqtj89mijj1dq57eh4m.field`.\n\n### Specifying Collection Values\nElements of a Noms list, map, or set can be retrieved using brackets `[...]`.\n\nFor example, if the dataset is a Noms map of number to struct then one could use `.value[42]` to get the Noms struct associated with the key 42. Similarly selecting the first element from a Noms list would be `.value[0]`. If the Noms map was keyed by string, then using `.value[\"0000024-02-999\"]` would reference the Noms struct associated with key \"0000024-02-999\".\n\nNoms lists also support indexing from the back, using `.value[-1]` to mean the last element of a last, `.value[-2]` for the 2nd last, and so on.\n\nIf the key of a Noms map or set is a Noms struct or a more complex value, then indexing into the collection can be done using the hash of that more complex value. For example, if the `root` of our dataset is a Noms set of Noms structs, then if you provide the hash of the struct element then you can index into the map using the brackets as described above. e.g. http://localhost:8000::dataset.value[#o38hugtf3l1e8rqtj89mijj1dq57eh4m].field\n\nSimilarly, the key is addressable using `@key` syntax. One use for this is when you have the hash of a complex value, but want need to retrieve the key (rather than or in addition to the value) in a Noms map. The syntax is to append `@key` after the closing bracket of the index specifier. e.g. http://localhost:8000::dataset.value[#o38hugtf3l1e8rqtj89mijj1dq57eh4m]@key would retrieve the key element specified by the hash key `#o38hugtf3l1e8rqtj89mijj1dq57eh4m` from the `dataset.value` collection.\n\n### Specifying Collection Positions\nElements of a Noms list, map, or set can be retrived _by their position_ using the `@at(index)` annotation.\n\nFor lists, this is exactly equivalent to `[index]`. For sets and maps, note that Noms has a stable ordering, so `@at(0)` will always return the smallest element, `@at(1)` the 2nd smallest, and so on. `@at(-1)` will return the largest. For maps, adding the `@key` annotation will retrieve the key of the map entry instead of the value.\n\n### Examples\n\n```sh\n# “sf-registered-business” dataset at https://demo.noms.io/cli-tour\nhttps://demo.noms.io/cli-tour::sf-registered-business\n\n# value o38hugtf3l1e8rqtj89mijj1dq57eh4m at https://localhost:8000\nhttps://localhost:8000/monkey::#o38hugtf3l1e8rqtj89mijj1dq57eh4m\n\n# “bonk” dataset at /foo/bar\n/foo/bar::bonk\n\n# from https://demo.noms.io/cli-tour, select the \"sf-registered-business\" dataset,\n# the root value is a Noms map, select the value of the Noms map identified by string\n# key \"0000024-02-999\", then from that resulting struct select the Ownership_Name field\nhttps://demo.noms.io/cli-tour::sf-registered-business.value[\"0000024-02-999\"].Ownership_Name\n```\n\nBe careful with shell escaping. Your shell might require escaping of the double quotes and other characters or use single quotes around the entire command line argument. e.g.:\n\n```sh\n> noms show https://demo.noms.io/cli-tour::sf-registered-business.value[\"0000024-02-999\"].Ownership_Name\nerror: Invalid index: 0000024-02-999\n\n> noms show https://demo.noms.io/cli-tour::sf-registered-business.value[\\\"0000024-02-999\\\"].Ownership_Name\n\"EASTMAN KODAK CO\"\n\n> noms show 'https://demo.noms.io/cli-tour::sf-registered-business.value[\"0000024-02-999\"].Ownership_Name'\n\"EASTMAN KODAK CO\"\n```\n" }, { "path": "go/chunks/chunk.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package chunks provides facilities for representing, storing, and fetching content-addressed chunks of Noms data.\npackage chunks\n\nimport (\n\t\"bytes\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n// Chunk is a unit of stored data in noms\ntype Chunk struct {\n\tr hash.Hash\n\tdata []byte\n}\n\nvar EmptyChunk = NewChunk([]byte{})\n\nfunc (c Chunk) Hash() hash.Hash {\n\treturn c.r\n}\n\nfunc (c Chunk) Data() []byte {\n\treturn c.data\n}\n\nfunc (c Chunk) IsEmpty() bool {\n\treturn len(c.data) == 0\n}\n\n// NewChunk creates a new Chunk backed by data. This means that the returned Chunk has ownership of this slice of memory.\nfunc NewChunk(data []byte) Chunk {\n\tr := hash.Of(data)\n\treturn Chunk{r, data}\n}\n\n// NewChunkWithHash creates a new chunk with a known hash. The hash is not re-calculated or verified. This should obviously only be used in cases where the caller already knows the specified hash is correct.\nfunc NewChunkWithHash(r hash.Hash, data []byte) Chunk {\n\treturn Chunk{r, data}\n}\n\n// ChunkWriter wraps an io.WriteCloser, additionally providing the ability to grab the resulting Chunk for all data written through the interface. Calling Chunk() or Close() on an instance disallows further writing.\ntype ChunkWriter struct {\n\tbuffer *bytes.Buffer\n\tc Chunk\n}\n\nfunc NewChunkWriter() *ChunkWriter {\n\tb := &bytes.Buffer{}\n\treturn &ChunkWriter{\n\t\tbuffer: b,\n\t}\n}\n\nfunc (w *ChunkWriter) Write(data []byte) (int, error) {\n\tif w.buffer == nil {\n\t\td.Panic(\"Write() cannot be called after Hash() or Close().\")\n\t}\n\tsize, err := w.buffer.Write(data)\n\td.Chk.NoError(err)\n\treturn size, nil\n}\n\n// Chunk() closes the writer and returns the resulting Chunk.\nfunc (w *ChunkWriter) Chunk() Chunk {\n\td.Chk.NoError(w.Close())\n\treturn w.c\n}\n\n// Close() closes computes the hash and Puts it into the ChunkSink Note: The Write() method never returns an error. Instead, like other noms interfaces, errors are reported via panic.\nfunc (w *ChunkWriter) Close() error {\n\tif w.buffer == nil {\n\t\treturn nil\n\t}\n\n\tw.c = NewChunk(w.buffer.Bytes())\n\tw.buffer = nil\n\treturn nil\n}\n" }, { "path": "go/chunks/chunk_serializer.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"bytes\"\n\t\"encoding/binary\"\n\t\"io\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n/*\n Chunk Serialization:\n Chunk 0\n Chunk 1\n ..\n Chunk N\n\n Chunk:\n Hash // 20-byte hash\n Len // 4-byte int\n Data // len(Data) == Len\n*/\n\n// Serialize a single Chunk to writer.\nfunc Serialize(chunk Chunk, writer io.Writer) {\n\td.PanicIfFalse(chunk.data != nil)\n\n\th := chunk.Hash()\n\tn, err := io.Copy(writer, bytes.NewReader(h[:]))\n\td.Chk.NoError(err)\n\td.PanicIfFalse(int64(hash.ByteLen) == n)\n\n\t// Because of chunking at higher levels, no chunk should never be more than 4GB\n\tchunkSize := uint32(len(chunk.Data()))\n\terr = binary.Write(writer, binary.BigEndian, chunkSize)\n\td.Chk.NoError(err)\n\n\tn, err = io.Copy(writer, bytes.NewReader(chunk.Data()))\n\td.Chk.NoError(err)\n\td.PanicIfFalse(uint32(n) == chunkSize)\n}\n\n// Deserialize reads off of |reader| until EOF, sending chunks to\n// chunkChan in the order they are read. Objects sent over chunkChan are\n// *Chunk.\nfunc Deserialize(reader io.Reader, chunkChan chan<- *Chunk) (err error) {\n\tfor {\n\t\tvar c Chunk\n\t\tc, err = deserializeChunk(reader)\n\t\tif err != nil {\n\t\t\tbreak\n\t\t}\n\t\td.Chk.NotEqual(EmptyChunk.Hash(), c.Hash())\n\t\tchunkChan <- &c\n\t}\n\tif err == io.EOF {\n\t\terr = nil\n\t}\n\treturn\n}\n\nfunc deserializeChunk(reader io.Reader) (Chunk, error) {\n\th := hash.Hash{}\n\tn, err := io.ReadFull(reader, h[:])\n\tif err != nil {\n\t\treturn EmptyChunk, err\n\t}\n\td.PanicIfFalse(int(hash.ByteLen) == n)\n\n\tchunkSize := uint32(0)\n\tif err = binary.Read(reader, binary.BigEndian, &chunkSize); err != nil {\n\t\treturn EmptyChunk, err\n\t}\n\n\tdata := make([]byte, int(chunkSize))\n\tif n, err = io.ReadFull(reader, data); err != nil {\n\t\treturn EmptyChunk, err\n\t}\n\td.PanicIfFalse(int(chunkSize) == n)\n\tc := NewChunk(data)\n\tif h != c.Hash() {\n\t\td.Panic(\"%s != %s\", h, c.Hash().String())\n\t}\n\treturn c, nil\n}\n" }, { "path": "go/chunks/chunk_serializer_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"bytes\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestSerializeRoundTrip(t *testing.T) {\n\tassert := assert.New(t)\n\tinputs := [][]byte{[]byte(\"abc\"), []byte(\"def\")}\n\tchnx := make([]Chunk, len(inputs))\n\tfor i, data := range inputs {\n\t\tchnx[i] = NewChunk(data)\n\t}\n\n\tbuf := &bytes.Buffer{}\n\tSerialize(chnx[0], buf)\n\tSerialize(chnx[1], buf)\n\n\tchunkChan := make(chan *Chunk)\n\tgo func() {\n\t\tdefer close(chunkChan)\n\t\terr := Deserialize(bytes.NewReader(buf.Bytes()), chunkChan)\n\t\tassert.NoError(err)\n\t}()\n\n\tfor c := range chunkChan {\n\t\tassert.Equal(chnx[0].Hash(), c.Hash())\n\t\tchnx = chnx[1:]\n\t}\n\tassert.Len(chnx, 0)\n}\n\nfunc TestBadSerialization(t *testing.T) {\n\tbad := []byte{0, 1} // Not enough bytes to read first length\n\tch := make(chan *Chunk)\n\tdefer close(ch)\n\tassert.Error(t, Deserialize(bytes.NewReader(bad), ch))\n}\n" }, { "path": "go/chunks/chunk_store.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"io\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n// ChunkStore is the core storage abstraction in noms. We can put data\n// anyplace we have a ChunkStore implementation for.\ntype ChunkStore interface {\n\t// Get the Chunk for the value of the hash in the store. If the hash is\n\t// absent from the store EmptyChunk is returned.\n\tGet(h hash.Hash) Chunk\n\n\t// GetMany gets the Chunks with |hashes| from the store. On return,\n\t// |foundChunks| will have been fully sent all chunks which have been\n\t// found. Any non-present chunks will silently be ignored.\n\tGetMany(hashes hash.HashSet, foundChunks chan *Chunk)\n\n\t// Returns true iff the value at the address |h| is contained in the\n\t// store\n\tHas(h hash.Hash) bool\n\n\t// Returns a new HashSet containing any members of |hashes| that are\n\t// absent from the store.\n\tHasMany(hashes hash.HashSet) (absent hash.HashSet)\n\n\t// Put caches c in the ChunkSource. Upon return, c must be visible to\n\t// subsequent Get and Has calls, but must not be persistent until a call\n\t// to Flush(). Put may be called concurrently with other calls to Put(),\n\t// Get(), GetMany(), Has() and HasMany().\n\tPut(c Chunk)\n\n\t// Returns the NomsVersion with which this ChunkSource is compatible.\n\tVersion() string\n\n\t// Rebase brings this ChunkStore into sync with the persistent storage's\n\t// current root.\n\tRebase()\n\n\t// Root returns the root of the database as of the time the ChunkStore\n\t// was opened or the most recent call to Rebase.\n\tRoot() hash.Hash\n\n\t// Commit atomically attempts to persist all novel Chunks and update the\n\t// persisted root hash from last to current (or keeps it the same).\n\t// If last doesn't match the root in persistent storage, returns false.\n\tCommit(current, last hash.Hash) bool\n\n\t// Stats may return some kind of struct that reports statistics about the\n\t// ChunkStore instance. The type is implementation-dependent, and impls\n\t// may return nil\n\tStats() interface{}\n\n\t// StatsSummary may return a string containing summarized statistics for\n\t// this ChunkStore. It must return \"Unsupported\" if this operation is not\n\t// supported.\n\tStatsSummary() string\n\n\t// Close tears down any resources in use by the implementation. After\n\t// Close(), the ChunkStore may not be used again. It is NOT SAFE to call\n\t// Close() concurrently with any other ChunkStore method; behavior is\n\t// undefined and probably crashy.\n\tio.Closer\n}\n\n// Factory allows the creation of namespaced ChunkStore instances. The details\n// of how namespaces are separated is left up to the particular implementation\n// of Factory and ChunkStore.\ntype Factory interface {\n\tCreateStore(ns string) ChunkStore\n\n\t// CreateStoreFromCache allows caller to signal to the factory that it's\n\t// willing to tolerate an out-of-date ChunkStore.\n\tCreateStoreFromCache(ns string) ChunkStore\n\n\t// Shutter shuts down the factory. Subsequent calls to CreateStore() will fail.\n\tShutter()\n}\n" }, { "path": "go/chunks/chunk_store_common_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"github.com/stretchr/testify/suite\"\n\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype ChunkStoreTestSuite struct {\n\tsuite.Suite\n\tFactory Factory\n}\n\nfunc (suite *ChunkStoreTestSuite) TestChunkStorePut() {\n\tstore := suite.Factory.CreateStore(\"ns\")\n\tinput := \"abc\"\n\tc := NewChunk([]byte(input))\n\tstore.Put(c)\n\th := c.Hash()\n\n\t// Reading it via the API should work.\n\tassertInputInStore(input, h, store, suite.Assert())\n}\n\nfunc (suite *ChunkStoreTestSuite) TestChunkStoreRoot() {\n\tstore := suite.Factory.CreateStore(\"ns\")\n\toldRoot := store.Root()\n\tsuite.True(oldRoot.IsEmpty())\n\n\tbogusRoot := hash.Parse(\"8habda5skfek1265pc5d5l1orptn5dr0\")\n\tnewRoot := hash.Parse(\"8la6qjbh81v85r6q67lqbfrkmpds14lg\")\n\n\t// Try to update root with bogus oldRoot\n\tresult := store.Commit(newRoot, bogusRoot)\n\tsuite.False(result)\n\n\t// Now do a valid root update\n\tresult = store.Commit(newRoot, oldRoot)\n\tsuite.True(result)\n}\n\nfunc (suite *ChunkStoreTestSuite) TestChunkStoreCommitPut() {\n\tname := \"ns\"\n\tstore := suite.Factory.CreateStore(name)\n\tinput := \"abc\"\n\tc := NewChunk([]byte(input))\n\tstore.Put(c)\n\th := c.Hash()\n\n\t// Reading it via the API should work...\n\tassertInputInStore(input, h, store, suite.Assert())\n\t// ...but it shouldn't be persisted yet\n\tassertInputNotInStore(input, h, suite.Factory.CreateStore(name), suite.Assert())\n\n\tstore.Commit(h, store.Root()) // Commit persists Chunks\n\tassertInputInStore(input, h, store, suite.Assert())\n\tassertInputInStore(input, h, suite.Factory.CreateStore(name), suite.Assert())\n}\n\nfunc (suite *ChunkStoreTestSuite) TestChunkStoreGetNonExisting() {\n\tstore := suite.Factory.CreateStore(\"ns\")\n\th := hash.Parse(\"11111111111111111111111111111111\")\n\tc := store.Get(h)\n\tsuite.True(c.IsEmpty())\n}\n\nfunc (suite *ChunkStoreTestSuite) TestChunkStoreVersion() {\n\tstore := suite.Factory.CreateStore(\"ns\")\n\toldRoot := store.Root()\n\tsuite.True(oldRoot.IsEmpty())\n\tnewRoot := hash.Parse(\"11111222223333344444555556666677\")\n\tsuite.True(store.Commit(newRoot, oldRoot))\n\n\tsuite.Equal(constants.NomsVersion, store.Version())\n}\n\nfunc (suite *ChunkStoreTestSuite) TestChunkStoreCommitUnchangedRoot() {\n\tstore1, store2 := suite.Factory.CreateStore(\"ns\"), suite.Factory.CreateStore(\"ns\")\n\tinput := \"abc\"\n\tc := NewChunk([]byte(input))\n\tstore1.Put(c)\n\th := c.Hash()\n\n\t// Reading c from store1 via the API should work...\n\tassertInputInStore(input, h, store1, suite.Assert())\n\t// ...but not store2.\n\tassertInputNotInStore(input, h, store2, suite.Assert())\n\n\tstore1.Commit(store1.Root(), store1.Root())\n\tstore2.Rebase()\n\t// Now, reading c from store2 via the API should work...\n\tassertInputInStore(input, h, store2, suite.Assert())\n}\n" }, { "path": "go/chunks/chunk_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestChunk(t *testing.T) {\n\tc := NewChunk([]byte(\"abc\"))\n\th := c.Hash()\n\t// See http://www.di-mgt.com.au/sha_testvectors.html\n\tassert.Equal(t, \"rmnjb8cjc5tblj21ed4qs821649eduie\", h.String())\n}\n\nfunc TestChunkWriteAfterCloseFails(t *testing.T) {\n\tassert := assert.New(t)\n\tinput := \"abc\"\n\tw := NewChunkWriter()\n\t_, err := w.Write([]byte(input))\n\tassert.NoError(err)\n\n\tassert.NoError(w.Close())\n\tassert.Panics(func() { w.Write([]byte(input)) }, \"Write() after Close() should barf!\")\n}\n\nfunc TestChunkWriteAfterChunkFails(t *testing.T) {\n\tassert := assert.New(t)\n\tinput := \"abc\"\n\tw := NewChunkWriter()\n\t_, err := w.Write([]byte(input))\n\tassert.NoError(err)\n\n\t_ = w.Chunk()\n\tassert.Panics(func() { w.Write([]byte(input)) }, \"Write() after Chunk() should barf!\")\n}\n\nfunc TestChunkChunkCloses(t *testing.T) {\n\tassert := assert.New(t)\n\tinput := \"abc\"\n\tw := NewChunkWriter()\n\t_, err := w.Write([]byte(input))\n\tassert.NoError(err)\n\n\tw.Chunk()\n\tassert.Panics(func() { w.Write([]byte(input)) }, \"Write() after Close() should barf!\")\n}\n" }, { "path": "go/chunks/memory_store.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n// MemoryStorage provides a \"persistent\" storage layer to back multiple\n// MemoryStoreViews. A MemoryStorage instance holds the ground truth for the\n// root and set of chunks that are visible to all MemoryStoreViews vended by\n// NewView(), allowing them to implement the transaction-style semantics that\n// ChunkStore requires.\ntype MemoryStorage struct {\n\tdata map[hash.Hash]Chunk\n\trootHash hash.Hash\n\tmu sync.RWMutex\n}\n\n// NewView vends a MemoryStoreView backed by this MemoryStorage. It's\n// initialized with the currently \"persisted\" root.\nfunc (ms *MemoryStorage) NewView() ChunkStore {\n\treturn &MemoryStoreView{storage: ms, rootHash: ms.rootHash}\n}\n\n// Get retrieves the Chunk with the Hash h, returning EmptyChunk if it's not\n// present.\nfunc (ms *MemoryStorage) Get(h hash.Hash) Chunk {\n\tms.mu.RLock()\n\tdefer ms.mu.RUnlock()\n\tif c, ok := ms.data[h]; ok {\n\t\treturn c\n\t}\n\treturn EmptyChunk\n}\n\n// Has returns true if the Chunk with the Hash h is present in ms.data, false\n// if not.\nfunc (ms *MemoryStorage) Has(r hash.Hash) bool {\n\tms.mu.RLock()\n\tdefer ms.mu.RUnlock()\n\t_, ok := ms.data[r]\n\treturn ok\n}\n\n// Len returns the number of Chunks in ms.data.\nfunc (ms *MemoryStorage) Len() int {\n\tms.mu.RLock()\n\tdefer ms.mu.RUnlock()\n\treturn len(ms.data)\n}\n\n// Root returns the currently \"persisted\" root hash of this in-memory store.\nfunc (ms *MemoryStorage) Root() hash.Hash {\n\tms.mu.RLock()\n\tdefer ms.mu.RUnlock()\n\treturn ms.rootHash\n}\n\n// Update checks the \"persisted\" root against last and, iff it matches,\n// updates the root to current, adds all of novel to ms.data, and returns\n// true. Otherwise returns false.\nfunc (ms *MemoryStorage) Update(current, last hash.Hash, novel map[hash.Hash]Chunk) bool {\n\tms.mu.Lock()\n\tdefer ms.mu.Unlock()\n\tif last != ms.rootHash {\n\t\treturn false\n\t}\n\tif ms.data == nil {\n\t\tms.data = map[hash.Hash]Chunk{}\n\t}\n\tfor h, c := range novel {\n\t\tms.data[h] = c\n\t}\n\tms.rootHash = current\n\treturn true\n}\n\n// MemoryStoreView is an in-memory implementation of store.ChunkStore. Useful\n// mainly for tests.\n// The proper way to get one:\n// storage := &MemoryStorage{}\n// ms := storage.NewView()\ntype MemoryStoreView struct {\n\tpending map[hash.Hash]Chunk\n\trootHash hash.Hash\n\tmu sync.RWMutex\n\n\tstorage *MemoryStorage\n}\n\nfunc (ms *MemoryStoreView) Get(h hash.Hash) Chunk {\n\tms.mu.RLock()\n\tdefer ms.mu.RUnlock()\n\tif c, ok := ms.pending[h]; ok {\n\t\treturn c\n\t}\n\treturn ms.storage.Get(h)\n}\n\nfunc (ms *MemoryStoreView) GetMany(hashes hash.HashSet, foundChunks chan *Chunk) {\n\tfor h := range hashes {\n\t\tc := ms.Get(h)\n\t\tif !c.IsEmpty() {\n\t\t\tfoundChunks <- &c\n\t\t}\n\t}\n\treturn\n}\n\nfunc (ms *MemoryStoreView) Has(h hash.Hash) bool {\n\tms.mu.RLock()\n\tdefer ms.mu.RUnlock()\n\tif _, ok := ms.pending[h]; ok {\n\t\treturn true\n\t}\n\treturn ms.storage.Has(h)\n}\n\nfunc (ms *MemoryStoreView) HasMany(hashes hash.HashSet) hash.HashSet {\n\tabsent := hash.HashSet{}\n\tfor h := range hashes {\n\t\tif !ms.Has(h) {\n\t\t\tabsent.Insert(h)\n\t\t}\n\t}\n\treturn absent\n}\n\nfunc (ms *MemoryStoreView) Version() string {\n\treturn constants.NomsVersion\n}\n\nfunc (ms *MemoryStoreView) Put(c Chunk) {\n\tms.mu.Lock()\n\tdefer ms.mu.Unlock()\n\tif ms.pending == nil {\n\t\tms.pending = map[hash.Hash]Chunk{}\n\t}\n\tms.pending[c.Hash()] = c\n}\n\nfunc (ms *MemoryStoreView) Len() int {\n\tms.mu.RLock()\n\tdefer ms.mu.RUnlock()\n\treturn len(ms.pending) + ms.storage.Len()\n}\n\nfunc (ms *MemoryStoreView) Rebase() {\n\tms.mu.Lock()\n\tdefer ms.mu.Unlock()\n\tms.rootHash = ms.storage.Root()\n}\n\nfunc (ms *MemoryStoreView) Root() hash.Hash {\n\tms.mu.RLock()\n\tdefer ms.mu.RUnlock()\n\treturn ms.rootHash\n}\n\nfunc (ms *MemoryStoreView) Commit(current, last hash.Hash) bool {\n\tms.mu.Lock()\n\tdefer ms.mu.Unlock()\n\tif last != ms.rootHash {\n\t\treturn false\n\t}\n\n\tsuccess := ms.storage.Update(current, last, ms.pending)\n\tif success {\n\t\tms.pending = nil\n\t}\n\tms.rootHash = ms.storage.Root()\n\treturn success\n}\n\nfunc (ms *MemoryStoreView) Stats() interface{} {\n\treturn nil\n}\n\nfunc (ms *MemoryStoreView) StatsSummary() string {\n\treturn \"Unsupported\"\n}\n\nfunc (ms *MemoryStoreView) Close() error {\n\treturn nil\n}\n\ntype memoryStoreFactory struct {\n\tstores map[string]*MemoryStorage\n\tmu *sync.Mutex\n}\n\nfunc NewMemoryStoreFactory() Factory {\n\treturn &memoryStoreFactory{map[string]*MemoryStorage{}, &sync.Mutex{}}\n}\n\nfunc (f *memoryStoreFactory) CreateStoreFromCache(ns string) ChunkStore {\n\treturn f.CreateStore(ns)\n}\n\nfunc (f *memoryStoreFactory) CreateStore(ns string) ChunkStore {\n\tf.mu.Lock()\n\tdefer f.mu.Unlock()\n\n\tif f.stores == nil {\n\t\td.Panic(\"Cannot use memoryStoreFactory after Shutter().\")\n\t}\n\tif ms, present := f.stores[ns]; present {\n\t\treturn ms.NewView()\n\t}\n\tf.stores[ns] = &MemoryStorage{}\n\treturn f.stores[ns].NewView()\n}\n\nfunc (f *memoryStoreFactory) Shutter() {\n\tf.stores = nil\n}\n" }, { "path": "go/chunks/memory_store_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc TestMemoryStoreTestSuite(t *testing.T) {\n\tsuite.Run(t, &MemoryStoreTestSuite{})\n}\n\ntype MemoryStoreTestSuite struct {\n\tChunkStoreTestSuite\n}\n\nfunc (suite *MemoryStoreTestSuite) SetupTest() {\n\tsuite.Factory = NewMemoryStoreFactory()\n}\n\nfunc (suite *MemoryStoreTestSuite) TearDownTest() {\n\tsuite.Factory.Shutter()\n}\n" }, { "path": "go/chunks/put_cache.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\nfunc newUnwrittenPutCache() *unwrittenPutCache {\n\treturn &unwrittenPutCache{map[hash.Hash]Chunk{}, &sync.Mutex{}}\n}\n\ntype unwrittenPutCache struct {\n\tunwrittenPuts map[hash.Hash]Chunk\n\tmu *sync.Mutex\n}\n\nfunc (p *unwrittenPutCache) Add(c Chunk) bool {\n\tp.mu.Lock()\n\tdefer p.mu.Unlock()\n\tif _, ok := p.unwrittenPuts[c.Hash()]; !ok {\n\t\tp.unwrittenPuts[c.Hash()] = c\n\t\treturn true\n\t}\n\n\treturn false\n}\n\nfunc (p *unwrittenPutCache) Has(c Chunk) (has bool) {\n\tp.mu.Lock()\n\tdefer p.mu.Unlock()\n\t_, has = p.unwrittenPuts[c.Hash()]\n\treturn\n}\n\nfunc (p *unwrittenPutCache) Get(r hash.Hash) Chunk {\n\tp.mu.Lock()\n\tdefer p.mu.Unlock()\n\tif c, ok := p.unwrittenPuts[r]; ok {\n\t\treturn c\n\t}\n\treturn EmptyChunk\n}\n\nfunc (p *unwrittenPutCache) Clear(chunks []Chunk) {\n\tp.mu.Lock()\n\tdefer p.mu.Unlock()\n\tfor _, c := range chunks {\n\t\tdelete(p.unwrittenPuts, c.Hash())\n\t}\n}\n" }, { "path": "go/chunks/remote_requests.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype ReadRequest interface {\n\tHashes() hash.HashSet\n\tOutstanding() OutstandingRequest\n}\n\nfunc NewGetRequest(r hash.Hash, ch chan<- *Chunk) GetRequest {\n\treturn GetRequest{hash.HashSet{r: struct{}{}}, ch}\n}\n\ntype GetRequest struct {\n\thashes hash.HashSet\n\tch chan<- *Chunk\n}\n\nfunc NewGetManyRequest(hashes hash.HashSet, wg *sync.WaitGroup, ch chan<- *Chunk) GetManyRequest {\n\treturn GetManyRequest{hashes, wg, ch}\n}\n\ntype GetManyRequest struct {\n\thashes hash.HashSet\n\twg *sync.WaitGroup\n\tch chan<- *Chunk\n}\n\nfunc NewAbsentRequest(r hash.Hash, ch chan<- bool) AbsentRequest {\n\treturn AbsentRequest{hash.HashSet{r: struct{}{}}, ch}\n}\n\ntype AbsentRequest struct {\n\thashes hash.HashSet\n\tch chan<- bool\n}\n\nfunc NewAbsentManyRequest(hashes hash.HashSet, wg *sync.WaitGroup, ch chan<- hash.Hash) AbsentManyRequest {\n\treturn AbsentManyRequest{hashes, wg, ch}\n}\n\ntype AbsentManyRequest struct {\n\thashes hash.HashSet\n\twg *sync.WaitGroup\n\tch chan<- hash.Hash\n}\n\nfunc (g GetRequest) Hashes() hash.HashSet {\n\treturn g.hashes\n}\n\nfunc (g GetRequest) Outstanding() OutstandingRequest {\n\treturn OutstandingGet(g.ch)\n}\n\nfunc (g GetManyRequest) Hashes() hash.HashSet {\n\treturn g.hashes\n}\n\nfunc (g GetManyRequest) Outstanding() OutstandingRequest {\n\treturn OutstandingGetMany{g.wg, g.ch}\n}\n\nfunc (h AbsentRequest) Hashes() hash.HashSet {\n\treturn h.hashes\n}\n\nfunc (h AbsentRequest) Outstanding() OutstandingRequest {\n\treturn OutstandingAbsent(h.ch)\n}\n\nfunc (h AbsentManyRequest) Hashes() hash.HashSet {\n\treturn h.hashes\n}\n\nfunc (h AbsentManyRequest) Outstanding() OutstandingRequest {\n\treturn OutstandingAbsentMany{h.wg, h.ch}\n}\n\ntype OutstandingRequest interface {\n\tSatisfy(h hash.Hash, c *Chunk)\n\tFail()\n}\n\ntype OutstandingGet chan<- *Chunk\ntype OutstandingGetMany struct {\n\twg *sync.WaitGroup\n\tch chan<- *Chunk\n}\ntype OutstandingAbsent chan<- bool\ntype OutstandingAbsentMany struct {\n\twg *sync.WaitGroup\n\tch chan<- hash.Hash\n}\n\nfunc (r OutstandingGet) Satisfy(h hash.Hash, c *Chunk) {\n\tr <- c\n}\n\nfunc (r OutstandingGet) Fail() {\n\tr <- &EmptyChunk\n}\n\nfunc (ogm OutstandingGetMany) Satisfy(h hash.Hash, c *Chunk) {\n\togm.ch <- c\n\togm.wg.Done()\n}\n\nfunc (ogm OutstandingGetMany) Fail() {\n\togm.wg.Done()\n}\n\nfunc (oh OutstandingAbsent) Satisfy(h hash.Hash, c *Chunk) {\n\toh <- false\n}\n\nfunc (oh OutstandingAbsent) Fail() {\n\toh <- true\n}\n\nfunc (ohm OutstandingAbsentMany) Satisfy(h hash.Hash, c *Chunk) {\n\tohm.ch <- h\n\tohm.wg.Done()\n}\n\nfunc (ohm OutstandingAbsentMany) Fail() {\n\tohm.wg.Done()\n}\n\n// ReadBatch represents a set of queued Get/Has requests, each of which are blocking on a receive channel for a response.\ntype ReadBatch map[hash.Hash][]OutstandingRequest\n\n// Close ensures that callers to Get() and Has() are failed correctly if the corresponding chunk wasn't in the response from the server (i.e. it wasn't found).\nfunc (rb *ReadBatch) Close() error {\n\tfor _, reqs := range *rb {\n\t\tfor _, req := range reqs {\n\t\t\treq.Fail()\n\t\t}\n\t}\n\treturn nil\n}\n" }, { "path": "go/chunks/remote_requests_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"sync\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestGetRequestBatch(t *testing.T) {\n\tassert := assert.New(t)\n\th0 := hash.Parse(\"00000000000000000000000000000000\")\n\tc1 := NewChunk([]byte(\"abc\"))\n\th1 := c1.Hash()\n\tc2 := NewChunk([]byte(\"123\"))\n\th2 := c2.Hash()\n\n\ttally := func(b bool, trueCnt, falseCnt *int) {\n\t\tif b {\n\t\t\t*trueCnt++\n\t\t} else {\n\t\t\t*falseCnt++\n\t\t}\n\t}\n\n\treq0chan := make(chan bool, 1)\n\treq1chan := make(chan *Chunk, 1)\n\treq2chan := make(chan bool, 1)\n\treq3chan := make(chan bool, 1)\n\treq4chan := make(chan *Chunk, 1)\n\tdefer func() { close(req0chan); close(req1chan); close(req2chan); close(req3chan); close(req4chan) }()\n\n\tbatch := ReadBatch{\n\t\th0: []OutstandingRequest{OutstandingAbsent(req0chan), OutstandingGet(req1chan)},\n\t\th1: []OutstandingRequest{OutstandingAbsent(req2chan)},\n\t\th2: []OutstandingRequest{OutstandingAbsent(req3chan), OutstandingGet(req4chan)},\n\t}\n\tgo func() {\n\t\tfor requestedHash, reqs := range batch {\n\t\t\tfor _, req := range reqs {\n\t\t\t\tif requestedHash == h1 {\n\t\t\t\t\treq.Satisfy(h1, &c1)\n\t\t\t\t\tdelete(batch, h1)\n\t\t\t\t} else if requestedHash == h2 {\n\t\t\t\t\treq.Satisfy(h2, &c2)\n\t\t\t\t\tdelete(batch, h2)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tbatch.Close()\n\t}()\n\n\tvar r0True, r0False, r2True, r2False, r3True, r3False int\n\tb := <-req0chan\n\ttally(b, &r0True, &r0False)\n\tc := <-req1chan\n\tassert.EqualValues(EmptyChunk.Hash(), c.Hash())\n\tb = <-req2chan\n\ttally(b, &r2True, &r2False)\n\tb = <-req3chan\n\ttally(b, &r3True, &r3False)\n\tc = <-req4chan\n\tassert.EqualValues(c2.Hash(), c.Hash())\n\n\tassert.Equal(1, r0True)\n\tassert.Equal(0, r0False)\n\tassert.Equal(0, r2True)\n\tassert.Equal(1, r2False)\n\tassert.Equal(0, r3True)\n\tassert.Equal(1, r3False)\n}\n\nfunc TestGetManyRequestBatch(t *testing.T) {\n\tassert := assert.New(t)\n\th0 := hash.Parse(\"00000000000000000000000000000000\")\n\tc1 := NewChunk([]byte(\"abc\"))\n\th1 := c1.Hash()\n\tc2 := NewChunk([]byte(\"123\"))\n\th2 := c2.Hash()\n\n\tchunks := make(chan *Chunk)\n\thashes := hash.NewHashSet(h0, h1, h2)\n\twg := &sync.WaitGroup{}\n\twg.Add(len(hashes))\n\tgo func() { wg.Wait(); close(chunks) }()\n\n\treq := NewGetManyRequest(hashes, wg, chunks)\n\tbatch := ReadBatch{\n\t\th0: {req.Outstanding()},\n\t\th1: {req.Outstanding()},\n\t\th2: {req.Outstanding()},\n\t}\n\tgo func() {\n\t\tfor reqHash, reqs := range batch {\n\t\t\tfor _, req := range reqs {\n\t\t\t\tif reqHash == h1 {\n\t\t\t\t\treq.Satisfy(h1, &c1)\n\t\t\t\t\tdelete(batch, h1)\n\t\t\t\t} else if reqHash == h2 {\n\t\t\t\t\treq.Satisfy(h2, &c2)\n\t\t\t\t\tdelete(batch, h2)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tbatch.Close()\n\t}()\n\n\tfor c := range chunks {\n\t\thashes.Remove(c.Hash())\n\t}\n\tassert.Len(hashes, 1)\n\tassert.True(hashes.Has(h0))\n}\n\nfunc TestAbsentManyRequestBatch(t *testing.T) {\n\tassert := assert.New(t)\n\th0 := hash.Parse(\"00000000000000000000000000000000\")\n\tc1 := NewChunk([]byte(\"abc\"))\n\th1 := c1.Hash()\n\tc2 := NewChunk([]byte(\"123\"))\n\th2 := c2.Hash()\n\n\tfound := make(chan hash.Hash)\n\thashes := hash.NewHashSet(h0, h1, h2)\n\twg := &sync.WaitGroup{}\n\twg.Add(len(hashes))\n\tgo func() { wg.Wait(); close(found) }()\n\n\treq := NewAbsentManyRequest(hashes, wg, found)\n\tbatch := ReadBatch{}\n\tfor h := range req.Hashes() {\n\t\tbatch[h] = []OutstandingRequest{req.Outstanding()}\n\t}\n\tgo func() {\n\t\tfor reqHash, reqs := range batch {\n\t\t\tfor _, req := range reqs {\n\t\t\t\tif reqHash == h1 {\n\t\t\t\t\treq.Satisfy(h1, &EmptyChunk)\n\t\t\t\t\tdelete(batch, h1)\n\t\t\t\t} else if reqHash == h2 {\n\t\t\t\t\treq.Satisfy(h2, &EmptyChunk)\n\t\t\t\t\tdelete(batch, h2)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tbatch.Close()\n\t}()\n\n\tfor h := range found {\n\t\thashes.Remove(h)\n\t}\n\tassert.Len(hashes, 1)\n\tassert.True(hashes.Has(h0))\n}\n" }, { "path": "go/chunks/test_utils.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage chunks\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc assertInputInStore(input string, h hash.Hash, s ChunkStore, assert *assert.Assertions) {\n\tchunk := s.Get(h)\n\tassert.False(chunk.IsEmpty(), \"Shouldn't get empty chunk for %s\", h.String())\n\tassert.Equal(input, string(chunk.Data()))\n}\n\nfunc assertInputNotInStore(input string, h hash.Hash, s ChunkStore, assert *assert.Assertions) {\n\tchunk := s.Get(h)\n\tassert.True(chunk.IsEmpty(), \"Shouldn't get non-empty chunk for %s: %v\", h.String(), chunk)\n}\n\ntype TestStorage struct {\n\tMemoryStorage\n}\n\nfunc (t *TestStorage) NewView() *TestStoreView {\n\treturn &TestStoreView{ChunkStore: t.MemoryStorage.NewView()}\n}\n\ntype TestStoreView struct {\n\tChunkStore\n\tReads int\n\tHases int\n\tWrites int\n}\n\nfunc (s *TestStoreView) Get(h hash.Hash) Chunk {\n\ts.Reads++\n\treturn s.ChunkStore.Get(h)\n}\n\nfunc (s *TestStoreView) GetMany(hashes hash.HashSet, foundChunks chan *Chunk) {\n\ts.Reads += len(hashes)\n\ts.ChunkStore.GetMany(hashes, foundChunks)\n}\n\nfunc (s *TestStoreView) Has(h hash.Hash) bool {\n\ts.Hases++\n\treturn s.ChunkStore.Has(h)\n}\n\nfunc (s *TestStoreView) HasMany(hashes hash.HashSet) hash.HashSet {\n\ts.Hases += len(hashes)\n\treturn s.ChunkStore.HasMany(hashes)\n}\n\nfunc (s *TestStoreView) Put(c Chunk) {\n\ts.Writes++\n\ts.ChunkStore.Put(c)\n}\n\ntype TestStoreFactory struct {\n\tstores map[string]*TestStorage\n}\n\nfunc NewTestStoreFactory() *TestStoreFactory {\n\treturn &TestStoreFactory{map[string]*TestStorage{}}\n}\n\nfunc (f *TestStoreFactory) CreateStore(ns string) ChunkStore {\n\tif f.stores == nil {\n\t\td.Panic(\"Cannot use TestStoreFactory after Shutter().\")\n\t}\n\tif ts, present := f.stores[ns]; present {\n\t\treturn ts.NewView()\n\t}\n\tf.stores[ns] = &TestStorage{}\n\treturn f.stores[ns].NewView()\n}\n\nfunc (f *TestStoreFactory) Shutter() {\n\tf.stores = nil\n}\n" }, { "path": "go/config/config.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage config\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\n\t\"github.com/BurntSushi/toml\"\n\t\"github.com/attic-labs/noms/go/spec\"\n)\n\ntype Config struct {\n\tFile string\n\tDb map[string]DbConfig\n}\n\ntype DbConfig struct {\n\tUrl string\n}\n\nconst (\n\tNomsConfigFile = \".nomsconfig\"\n\tDefaultDbAlias = \"default\"\n)\n\nvar NoConfig = errors.New(fmt.Sprintf(\"no %s found\", NomsConfigFile))\n\n// Find the closest directory containing .nomsconfig starting\n// in cwd and then searching up ancestor tree.\n// Look first looking in cwd and then up through its ancestors\nfunc FindNomsConfig() (*Config, error) {\n\tcurDir, err := os.Getwd()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tfor {\n\t\tnomsConfig := filepath.Join(curDir, NomsConfigFile)\n\t\tinfo, err := os.Stat(nomsConfig)\n\t\tif err == nil && !info.IsDir() {\n\t\t\t// found\n\t\t\treturn ReadConfig(nomsConfig)\n\t\t} else if err != nil && !os.IsNotExist(err) {\n\t\t\t// can't read\n\t\t\treturn nil, err\n\t\t}\n\t\tnextDir := filepath.Dir(curDir)\n\t\tif nextDir == curDir {\n\t\t\t// stop at root\n\t\t\treturn nil, NoConfig\n\t\t}\n\t\tcurDir = nextDir\n\t}\n}\n\nfunc ReadConfig(name string) (*Config, error) {\n\tdata, err := ioutil.ReadFile(name)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tc, err := NewConfig(string(data))\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tc.File = name\n\treturn qualifyPaths(name, c)\n}\n\nfunc NewConfig(data string) (*Config, error) {\n\tc := new(Config)\n\tif _, err := toml.Decode(data, c); err != nil {\n\t\treturn nil, err\n\t}\n\treturn c, nil\n}\n\nfunc (c *Config) WriteTo(configHome string) (string, error) {\n\tfile := filepath.Join(configHome, NomsConfigFile)\n\tif err := os.MkdirAll(filepath.Dir(file), os.ModePerm); err != nil {\n\t\treturn \"\", err\n\t}\n\tif err := ioutil.WriteFile(file, []byte(c.writeableString()), os.ModePerm); err != nil {\n\t\treturn \"\", err\n\t}\n\treturn file, nil\n}\n\n// Replace relative directory in path part of spec with an absolute\n// directory. Assumes the path is relative to the location of the config file\nfunc absDbSpec(configHome string, url string) string {\n\tdbSpec, err := spec.ForDatabase(url)\n\tif err != nil {\n\t\treturn url\n\t}\n\tif dbSpec.Protocol != \"nbs\" {\n\t\treturn url\n\t}\n\tdbName := dbSpec.DatabaseName\n\tif !filepath.IsAbs(dbName) {\n\t\tdbName = filepath.Join(configHome, dbName)\n\t}\n\treturn \"nbs:\" + dbName\n}\n\nfunc qualifyPaths(configPath string, c *Config) (*Config, error) {\n\tfile, err := filepath.Abs(configPath)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tdir := filepath.Dir(file)\n\tqc := *c\n\tqc.File = file\n\tfor k, r := range c.Db {\n\t\tqc.Db[k] = DbConfig{absDbSpec(dir, r.Url)}\n\t}\n\treturn &qc, nil\n}\n\nfunc (c *Config) String() string {\n\tvar buffer bytes.Buffer\n\tif c.File != \"\" {\n\t\tbuffer.WriteString(fmt.Sprintf(\"file = %s\\n\", c.File))\n\t}\n\tbuffer.WriteString(c.writeableString())\n\treturn buffer.String()\n}\n\nfunc (c *Config) writeableString() string {\n\tvar buffer bytes.Buffer\n\tfor k, r := range c.Db {\n\t\tbuffer.WriteString(fmt.Sprintf(\"[db.%s]\\n\", k))\n\t\tbuffer.WriteString(fmt.Sprintf(\"\\t\"+`url = \"%s\"`+\"\\n\", r.Url))\n\t}\n\treturn buffer.String()\n}\n" }, { "path": "go/config/config_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage config\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nconst (\n\tnbsSpec = \"nbs:./local\"\n\tmemSpec = \"mem\"\n\thttpSpec = \"http://test.com:8080/foo\"\n\tnbsAbsSpec = \"nbs:/tmp/noms\"\n\tremoteAlias = \"origin\"\n)\n\nvar (\n\tctestRoot = os.TempDir()\n\n\tldbConfig = &Config{\n\t\t\"\",\n\t\tmap[string]DbConfig{\n\t\t\tDefaultDbAlias: {nbsSpec},\n\t\t\tremoteAlias: {httpSpec},\n\t\t},\n\t}\n\n\thttpConfig = &Config{\n\t\t\"\",\n\t\tmap[string]DbConfig{\n\t\t\tDefaultDbAlias: {httpSpec},\n\t\t\tremoteAlias: {nbsSpec},\n\t\t},\n\t}\n\n\tmemConfig = &Config{\n\t\t\"\",\n\t\tmap[string]DbConfig{\n\t\t\tDefaultDbAlias: {memSpec},\n\t\t\tremoteAlias: {httpSpec},\n\t\t},\n\t}\n\n\tldbAbsConfig = &Config{\n\t\t\"\",\n\t\tmap[string]DbConfig{\n\t\t\tDefaultDbAlias: {nbsAbsSpec},\n\t\t\tremoteAlias: {httpSpec},\n\t\t},\n\t}\n)\n\ntype paths struct {\n\thome string\n\tconfig string\n}\n\nfunc getPaths(assert *assert.Assertions, base string) paths {\n\tabs, err := filepath.Abs(ctestRoot)\n\tassert.NoError(err)\n\tabs, err = filepath.EvalSymlinks(ctestRoot)\n\tassert.NoError(err)\n\thome := filepath.Join(abs, base)\n\tconfig := filepath.Join(home, NomsConfigFile)\n\treturn paths{home, config}\n}\n\nfunc qualifyFilePath(assert *assert.Assertions, path string) string {\n\tp, err := filepath.Abs(path)\n\tassert.NoError(err)\n\treturn p\n}\n\nfunc assertDbSpecsEquiv(assert *assert.Assertions, expected string, actual string) {\n\te, err := spec.ForDatabase(expected)\n\tassert.NoError(err)\n\tif e.Protocol != \"nbs\" {\n\t\tassert.Equal(expected, actual)\n\t} else {\n\t\ta, err := spec.ForDatabase(actual)\n\t\tassert.NoError(err)\n\t\tassert.Equal(e.Protocol, a.Protocol, actual)\n\t\tif filepath.IsAbs(e.DatabaseName) {\n\t\t\tassert.Equal(e.DatabaseName, a.DatabaseName, actual)\n\t\t} else {\n\t\t\t// If the original path is relative, it will return as absolute.\n\t\t\t// All we do here is ensure that the path suffix is the same.\n\t\t\teName := strings.TrimPrefix(e.DatabaseName, \".\")\n\t\t\tassert.True(strings.HasSuffix(a.DatabaseName, eName),\n\t\t\t\t\"expected: %s; actual: %s\", eName, actual)\n\t\t}\n\t}\n}\n\nfunc validateConfig(assert *assert.Assertions, file string, e *Config, a *Config) {\n\tassert.Equal(qualifyFilePath(assert, file), qualifyFilePath(assert, a.File))\n\tassert.Equal(len(e.Db), len(a.Db))\n\tfor k, er := range e.Db {\n\t\tar, ok := a.Db[k]\n\t\tassert.True(ok)\n\t\tassertDbSpecsEquiv(assert, er.Url, ar.Url)\n\t}\n}\n\nfunc writeConfig(assert *assert.Assertions, c *Config, home string) string {\n\tfile, err := c.WriteTo(home)\n\tassert.NoError(err, home)\n\treturn file\n}\n\nfunc TestConfig(t *testing.T) {\n\tassert := assert.New(t)\n\tpath := getPaths(assert, \"home\")\n\twriteConfig(assert, ldbConfig, path.home)\n\n\t// Test from home\n\tassert.NoError(os.Chdir(path.home))\n\tc, err := FindNomsConfig()\n\tassert.NoError(err, path.config)\n\tvalidateConfig(assert, path.config, ldbConfig, c)\n\n\t// Test from subdir\n\tsubdir := filepath.Join(path.home, \"subdir\")\n\tassert.NoError(os.MkdirAll(subdir, os.ModePerm))\n\tassert.NoError(os.Chdir(subdir))\n\tc, err = FindNomsConfig()\n\tassert.NoError(err, path.config)\n\tvalidateConfig(assert, path.config, ldbConfig, c)\n\n\t// Test from subdir with intervening .nomsconfig directory\n\tnomsDir := filepath.Join(subdir, NomsConfigFile)\n\terr = os.MkdirAll(nomsDir, os.ModePerm)\n\tassert.NoError(err, nomsDir)\n\tassert.NoError(os.Chdir(subdir))\n\tc, err = FindNomsConfig()\n\tassert.NoError(err, path.config)\n\tvalidateConfig(assert, path.config, ldbConfig, c)\n}\n\nfunc TestUnreadableConfig(t *testing.T) {\n\t// BUG 3816\n\tif os.Getenv(\"DOCKER\") != \"\" {\n\t\tt.Skip(\"Skipping testing in Docker environment\")\n\t}\n\n\tassert := assert.New(t)\n\tpath := getPaths(assert, \"home.unreadable\")\n\twriteConfig(assert, ldbConfig, path.home)\n\tassert.NoError(os.Chmod(path.config, 0333)) // write-only\n\tassert.NoError(os.Chdir(path.home))\n\t_, err := FindNomsConfig()\n\tassert.Error(err, path.config)\n}\n\nfunc TestNoConfig(t *testing.T) {\n\tassert := assert.New(t)\n\tpath := getPaths(assert, \"home.none\")\n\tassert.NoError(os.MkdirAll(path.home, os.ModePerm))\n\tassert.NoError(os.Chdir(path.home))\n\t_, err := FindNomsConfig()\n\tassert.Equal(NoConfig, err)\n}\n\nfunc TestBadConfig(t *testing.T) {\n\tassert := assert.New(t)\n\tpath := getPaths(assert, \"home.bad\")\n\tcfile := writeConfig(assert, ldbConfig, path.home)\n\t// overwrite with something invalid\n\tassert.NoError(ioutil.WriteFile(cfile, []byte(\"invalid config\"), os.ModePerm))\n\tassert.NoError(os.Chdir(path.home))\n\t_, err := FindNomsConfig()\n\tassert.Error(err, path.config)\n}\n\nfunc TestQualifyingPaths(t *testing.T) {\n\tassert := assert.New(t)\n\tpath := getPaths(assert, \"home\")\n\tassert.NoError(os.Chdir(path.home))\n\n\tfor _, tc := range []*Config{httpConfig, memConfig, ldbAbsConfig} {\n\t\twriteConfig(assert, tc, path.home)\n\t\tac, err := FindNomsConfig()\n\t\tassert.NoError(err, path.config)\n\t\tvalidateConfig(assert, path.config, tc, ac)\n\t}\n}\n\nfunc TestCwd(t *testing.T) {\n\tassert := assert.New(t)\n\tcwd, err := os.Getwd()\n\tassert.NoError(err)\n\tcwd = filepath.Join(cwd, \"test\")\n\tabs, err := filepath.Abs(\"test\")\n\tassert.NoError(err)\n\n\tassert.Equal(cwd, abs)\n}\n" }, { "path": "go/config/resolver.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage config\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/verbose\"\n)\n\ntype Resolver struct {\n\tconfig *Config\n\tdotDatapath string // set to the first datapath that was resolved\n}\n\n// A Resolver enables using db defaults, db aliases and dataset '.' replacement in command\n// line arguments when a .nomsconfig file is present. To use it, create a config resolver\n// before command line processing and use it to resolve each dataspec argument in\n// succession.\nfunc NewResolver() *Resolver {\n\tc, err := FindNomsConfig()\n\tif err != nil {\n\t\tif err != NoConfig {\n\t\t\tpanic(fmt.Errorf(\"Failed to read .nomsconfig due to: %v\", err))\n\t\t}\n\t\treturn &Resolver{}\n\t}\n\treturn &Resolver{c, \"\"}\n}\n\n// Print replacement if one occurred\nfunc (r *Resolver) verbose(orig string, replacement string) string {\n\tif orig != replacement {\n\t\tif orig == \"\" {\n\t\t\torig = `\"\"`\n\t\t}\n\t\tverbose.Log(\"\\tresolving %s -> %s\\n\", orig, replacement)\n\t}\n\treturn replacement\n}\n\n// Resolve string to database name. If config is defined:\n// - replace the empty string with the default db url\n// - replace any db alias with it's url\nfunc (r *Resolver) ResolveDbSpec(str string) string {\n\tif r.config != nil {\n\t\tif str == \"\" {\n\t\t\treturn r.config.Db[DefaultDbAlias].Url\n\t\t}\n\t\tif val, ok := r.config.Db[str]; ok {\n\t\t\treturn val.Url\n\t\t}\n\t}\n\treturn str\n}\n\n// Resolve string to dataset or path name.\n// - replace database name as described in ResolveDatabase\n// - if this is the first call to ResolvePath, remember the\n// datapath part for subsequent calls.\n// - if this is not the first call and a \".\" is used, replace\n// it with the first datapath.\nfunc (r *Resolver) ResolvePathSpec(str string) string {\n\tif r.config != nil {\n\t\tsplit := strings.SplitN(str, spec.Separator, 2)\n\t\tdb, rest := \"\", split[0]\n\t\tif len(split) > 1 {\n\t\t\tdb, rest = split[0], split[1]\n\t\t}\n\t\tif r.dotDatapath == \"\" {\n\t\t\tr.dotDatapath = rest\n\t\t} else if rest == \".\" {\n\t\t\trest = r.dotDatapath\n\t\t}\n\t\treturn r.ResolveDbSpec(db) + spec.Separator + rest\n\t}\n\treturn str\n}\n\n// Resolve string to database spec. If a config is present,\n// - resolve a db alias to its db spec\n// - resolve \"\" to the default db spec\nfunc (r *Resolver) GetDatabase(str string) (datas.Database, error) {\n\tsp, err := spec.ForDatabase(r.verbose(str, r.ResolveDbSpec(str)))\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn sp.GetDatabase(), nil\n}\n\n// Resolve string to a chunkstore. Like ResolveDatabase, but returns the underlying ChunkStore\nfunc (r *Resolver) GetChunkStore(str string) (chunks.ChunkStore, error) {\n\tsp, err := spec.ForDatabase(r.verbose(str, r.ResolveDbSpec(str)))\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn sp.NewChunkStore(), nil\n}\n\n// Resolve string to a dataset. If a config is present,\n// - if no db prefix is present, assume the default db\n// - if the db prefix is an alias, replace it\nfunc (r *Resolver) GetDataset(str string) (datas.Database, datas.Dataset, error) {\n\tsp, err := spec.ForDataset(r.verbose(str, r.ResolvePathSpec(str)))\n\tif err != nil {\n\t\treturn nil, datas.Dataset{}, err\n\t}\n\treturn sp.GetDatabase(), sp.GetDataset(), nil\n}\n\n// Resolve string to a value path. If a config is present,\n// - if no db spec is present, assume the default db\n// - if the db spec is an alias, replace it\nfunc (r *Resolver) GetPath(str string) (datas.Database, types.Value, error) {\n\tsp, err := spec.ForPath(r.verbose(str, r.ResolvePathSpec(str)))\n\tif err != nil {\n\t\treturn nil, nil, err\n\t}\n\treturn sp.GetDatabase(), sp.GetValue(), nil\n}\n" }, { "path": "go/config/resolver_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage config\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nconst (\n\tlocalSpec = nbsSpec\n\tremoteSpec = httpSpec\n\ttestDs = \"testds\"\n\ttestObject = \"#pckdvpvr9br1fie6c3pjudrlthe7na18\"\n)\n\ntype testData struct {\n\tinput string\n\texpected string\n}\n\nvar (\n\trtestRoot = os.TempDir()\n\n\trtestConfig = &Config{\n\t\t\"\",\n\t\tmap[string]DbConfig{\n\t\t\tDefaultDbAlias: {localSpec},\n\t\t\tremoteAlias: {remoteSpec},\n\t\t},\n\t}\n\n\tdbTestsNoAliases = []testData{\n\t\t{localSpec, localSpec},\n\t\t{remoteSpec, remoteSpec},\n\t}\n\n\tdbTestsWithAliases = []testData{\n\t\t{\"\", localSpec},\n\t\t{remoteAlias, remoteSpec},\n\t}\n\n\tpathTestsNoAliases = []testData{\n\t\t{remoteSpec + \"::\" + testDs, remoteSpec + \"::\" + testDs},\n\t\t{remoteSpec + \"::\" + testObject, remoteSpec + \"::\" + testObject},\n\t}\n\n\tpathTestsWithAliases = []testData{\n\t\t{testDs, localSpec + \"::\" + testDs},\n\t\t{remoteAlias + \"::\" + testDs, remoteSpec + \"::\" + testDs},\n\t\t{testObject, localSpec + \"::\" + testObject},\n\t\t{remoteAlias + \"::\" + testObject, remoteSpec + \"::\" + testObject},\n\t}\n)\n\nfunc withConfig(t *testing.T) *Resolver {\n\tassert := assert.New(t)\n\tdir := filepath.Join(rtestRoot, \"with-config\")\n\t_, err := rtestConfig.WriteTo(dir)\n\tassert.NoError(err, dir)\n\tassert.NoError(os.Chdir(dir))\n\tr := NewResolver() // resolver must be created after changing directory\n\treturn r\n\n}\n\nfunc withoutConfig(t *testing.T) *Resolver {\n\tassert := assert.New(t)\n\tdir := filepath.Join(rtestRoot, \"without-config\")\n\tassert.NoError(os.MkdirAll(dir, os.ModePerm), dir)\n\tassert.NoError(os.Chdir(dir))\n\tr := NewResolver() // resolver must be created after changing directory\n\treturn r\n}\n\nfunc assertPathSpecsEquiv(assert *assert.Assertions, expected string, actual string) {\n\te, err := spec.ForPath(expected)\n\tassert.NoError(err)\n\ta, err := spec.ForPath(actual)\n\tassert.NoError(err)\n\n\tdatabaseSpec := func(sp spec.Spec) string {\n\t\treturn fmt.Sprintf(\"%s:%s\", sp.Protocol, sp.DatabaseName)\n\t}\n\n\tassertDbSpecsEquiv(assert, databaseSpec(e), databaseSpec(a))\n\tassert.Equal(e.Path.String(), a.Path.String())\n}\n\nfunc TestResolveDatabaseWithConfig(t *testing.T) {\n\tr := withConfig(t)\n\tassert := assert.New(t)\n\tfor _, d := range append(dbTestsNoAliases, dbTestsWithAliases...) {\n\t\tdb := r.ResolveDbSpec(d.input)\n\t\tassertDbSpecsEquiv(assert, d.expected, db)\n\t}\n}\n\nfunc TestResolvePathWithConfig(t *testing.T) {\n\tr := withConfig(t)\n\tassert := assert.New(t)\n\tfor _, d := range append(pathTestsNoAliases, pathTestsWithAliases...) {\n\t\tpath := r.ResolvePathSpec(d.input)\n\t\tassertPathSpecsEquiv(assert, d.expected, path)\n\t}\n}\n\nfunc TestResolveDatabaseWithoutConfig(t *testing.T) {\n\tr := withoutConfig(t)\n\tassert := assert.New(t)\n\tfor _, d := range dbTestsNoAliases {\n\t\tdb := r.ResolveDbSpec(d.input)\n\t\tassert.Equal(d.expected, db, d.input)\n\t}\n}\n\nfunc TestResolvePathWithoutConfig(t *testing.T) {\n\tr := withoutConfig(t)\n\tassert := assert.New(t)\n\tfor _, d := range pathTestsNoAliases {\n\t\tpath := r.ResolvePathSpec(d.input)\n\t\tassertPathSpecsEquiv(assert, d.expected, path)\n\t}\n\n}\n\nfunc TestResolveDestPathWithDot(t *testing.T) {\n\tr := withConfig(t)\n\tassert := assert.New(t)\n\n\tdata := []struct {\n\t\tsrc string\n\t\tdest string\n\t\texpSrc string\n\t\texpDest string\n\t}{\n\t\t{testDs, remoteSpec + \"::.\", localSpec + \"::\" + testDs, remoteSpec + \"::\" + testDs},\n\t\t{remoteSpec + \"::\" + testDs, \".\", remoteSpec + \"::\" + testDs, localSpec + \"::\" + testDs},\n\t}\n\tfor _, d := range data {\n\t\tsrc := r.ResolvePathSpec(d.src)\n\t\tdest := r.ResolvePathSpec(d.dest)\n\t\tassertPathSpecsEquiv(assert, d.expSrc, src)\n\t\tassertPathSpecsEquiv(assert, d.expDest, dest)\n\t}\n\n}\n" }, { "path": "go/constants/http.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage constants\n\nconst (\n\tRootPath = \"/root/\"\n\tGetRefsPath = \"/getRefs/\"\n\tGetBlobPath = \"/getBlob/\"\n\tHasRefsPath = \"/hasRefs/\"\n\tWriteValuePath = \"/writeValue/\"\n\tBasePath = \"/\"\n\n\tGraphQLPath = \"/graphql/\"\n\tStatsPath = \"/stats/\"\n)\n" }, { "path": "go/constants/version.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package constants collects common constants used in Noms, such as the Noms data format version.\npackage constants\n\nconst NomsVersion = \"7.18\"\n\nvar NomsGitSHA = \"\"\n" }, { "path": "go/d/check_error.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage d\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\n\t\"github.com/attic-labs/kingpin\"\n\t\"github.com/attic-labs/noms/go/util/exit\"\n)\n\nfunc CheckError(err error) {\n\tif err != nil {\n\t\tfmt.Fprintf(os.Stderr, \"error: %s\\n\", err)\n\t\tkingpin.Usage()\n\t\texit.Fail()\n\t}\n}\n\nfunc CheckErrorNoUsage(err error) {\n\tif err != nil {\n\t\tfmt.Fprintf(os.Stderr, \"error: %s\\n\", err)\n\t\texit.Fail()\n\t}\n}\n" }, { "path": "go/d/try.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package d implements several debug, error and assertion functions used throughout Noms.\npackage d\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"reflect\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\n// d.Chk.() -- used in test cases and as assertions\nvar (\n\tChk = assert.New(&panicker{})\n)\n\ntype panicker struct {\n}\n\nfunc (s panicker) Errorf(format string, args ...interface{}) {\n\tpanic(fmt.Sprintf(format, args...))\n}\n\n// Panic(err) creates an error using format and args and wraps it in a\n// WrappedError which can be handled using Try() and TryCatch()\nfunc Panic(format string, args ...interface{}) {\n\n\tif len(args) == 0 {\n\t\terr := errors.New(format)\n\t\tpanic(Wrap(err))\n\t}\n\terr := fmt.Errorf(format, args...)\n\tpanic(Wrap(err))\n}\n\n// PanicIfError(err) && PanicIfTrue(expr) can be used to panic in a way that's\n// easily handled by Try() and TryCatch()\nfunc PanicIfError(err error) {\n\tif err != nil {\n\t\tpanic(Wrap(err))\n\t}\n}\n\n// If b is true, creates a default error, wraps it and panics.\nfunc PanicIfTrue(b bool) {\n\tif b {\n\t\tpanic(Wrap(errors.New(\"Expected true\")))\n\t}\n}\n\n// If b is false, creates a default error, wraps it and panics.\nfunc PanicIfFalse(b bool) {\n\tif !b {\n\t\tpanic(Wrap(errors.New(\"Expected false\")))\n\t}\n}\n\n// If 'f' panics with a WrappedError then recover that error.\n// If types is empty, return the WrappedError.\n// if types is not empty and cause is not one of the listed types, re-panic.\n// if types is not empty and cause is one of the types, return 'cause'\nfunc Try(f func(), types ...interface{}) (err error) {\n\tdefer recoverWrappedTypes(&err, types)\n\tf()\n\treturn\n}\n\n// If 'f' panics with a WrappedError then recover that error and return it.\n// If types is empty, return the WrappedError.\n// if types is not empty and cause is not one of the listed types, re-panic.\n// if types is not empty and cause is one of the types, return 'cause'\nfunc TryCatch(f func(), catch func(err error) error) (err error) {\n\tdefer recoverWrapped(&err, catch)\n\tf()\n\treturn\n}\n\ntype WrappedError interface {\n\tError() string\n\tCause() error\n}\n\n// Wraps an error. The enclosing error has a default Error() that contains the error msg along\n// with a backtrace. The original error can be retrieved by calling err.Cause().\nfunc Wrap(err error) WrappedError {\n\tif err == nil {\n\t\treturn nil\n\t}\n\tif we, ok := err.(WrappedError); ok {\n\t\treturn we\n\t}\n\n\tst := stackTracer{}\n\tassert := assert.New(&st)\n\tassert.Fail(err.Error())\n\n\treturn wrappedError{st.stackTrace, err}\n}\n\n// If err is a WrappedError, then Cause() is returned, otherwise returns err.\nfunc Unwrap(err error) error {\n\tcause := err\n\twe, ok := err.(WrappedError)\n\tif ok {\n\t\tcause = we.Cause()\n\t}\n\treturn cause\n}\n\nfunc causeInTypes(err error, types ...interface{}) bool {\n\tcause := Unwrap(err)\n\ttyp := reflect.TypeOf(cause)\n\tfor _, t := range types {\n\t\tif typ == reflect.TypeOf(t) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\n// Utility method, that checks type of error and panics with wrapped error not one of the listed types.\nfunc PanicIfNotType(err error, types ...interface{}) error {\n\tif err == nil {\n\t\treturn nil\n\t}\n\tif !causeInTypes(err, types...) {\n\t\twe, ok := err.(WrappedError)\n\t\tif !ok {\n\t\t\twe = Wrap(err)\n\t\t}\n\t\tpanic(we)\n\t}\n\treturn Unwrap(err)\n}\n\ntype wrappedError struct {\n\tmsg string\n\tcause error\n}\n\nfunc (we wrappedError) Error() string { return we.msg }\nfunc (we wrappedError) Cause() error { return we.cause }\nfunc (we wrappedError) Unwrap() error { return we.cause }\n\ntype stackTracer struct {\n\tstackTrace string\n}\n\nfunc (s *stackTracer) Errorf(format string, args ...interface{}) {\n\ts.stackTrace = fmt.Sprintf(format, args...)\n}\n\nfunc recoverWrappedTypes(errp *error, types []interface{}) {\n\tif r := recover(); r != nil {\n\t\tif wrapper, ok := r.(wrappedError); !ok {\n\t\t\tpanic(r)\n\t\t} else if len(types) > 0 && !causeInTypes(wrapper, types...) {\n\t\t\tpanic(r)\n\t\t} else if len(types) > 0 {\n\t\t\t*errp = wrapper.Cause()\n\t\t} else {\n\t\t\t*errp = wrapper\n\t\t}\n\t}\n}\n\nfunc recoverWrapped(errp *error, catch func(err error) error) {\n\tif r := recover(); r != nil {\n\t\twe, ok := r.(wrappedError)\n\t\tif !ok {\n\t\t\tpanic(r)\n\t\t}\n\t\tif catch != nil {\n\t\t\t*errp = catch(we)\n\t\t} else {\n\t\t\t*errp = Unwrap(we)\n\t\t}\n\t}\n}\n" }, { "path": "go/d/try_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage d\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nvar (\n\tte = testError{\"te\"}\n\tte2 = testError2{\"te2\"}\n)\n\ntype testError struct {\n\ts string\n}\n\nfunc (e testError) Error() string { return e.s }\n\ntype testError2 struct {\n\ts string\n}\n\nfunc (e testError2) Error() string { return e.s }\n\nfunc TestTry2(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassert.Panics(func() {\n\t\tTry(func() {\n\t\t\tpanic(te)\n\t\t})\n\t})\n\n\tassert.Panics(func() {\n\t\tTry(func() {\n\t\t\tPanicIfError(te)\n\t\t}, te2)\n\t})\n\n\tassert.Error(func() error {\n\t\treturn Try(func() {\n\t\t\tPanicIfError(te)\n\t\t})\n\t}())\n\n\tassert.Error(func() error {\n\t\treturn Try(func() {\n\t\t\tPanicIfError(te)\n\t\t}, testError{})\n\t}())\n\n\tassert.Nil(func() error {\n\t\treturn Try(func() {\n\t\t\tPanicIfError(nil)\n\t\t})\n\t}())\n}\n\nfunc TestTryCatch(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassert.Panics(func() {\n\t\tTryCatch(func() {\n\t\t\tpanic(Wrap(te))\n\t\t},\n\t\t\tfunc(err error) error {\n\t\t\t\tif !causeInTypes(err, testError2{}) {\n\t\t\t\t\tpanic(err)\n\t\t\t\t}\n\t\t\t\treturn Unwrap(err)\n\t\t\t})\n\t})\n\n\tassert.Panics(func() {\n\t\tTryCatch(func() {\n\t\t\tpanic(te)\n\t\t},\n\t\t\tfunc(err error) error {\n\t\t\t\tif !causeInTypes(err, testError{}) {\n\t\t\t\t\tpanic(err)\n\t\t\t\t}\n\t\t\t\treturn Unwrap(err)\n\t\t\t})\n\t})\n\n\tassert.IsType(wrappedError{}, func() error {\n\t\treturn TryCatch(func() {\n\t\t\tpanic(Wrap(te))\n\t\t},\n\t\t\tfunc(err error) error {\n\t\t\t\treturn err\n\t\t\t})\n\t}())\n\n\tassert.Error(func() error {\n\t\treturn TryCatch(func() {\n\t\t\tpanic(Wrap(te))\n\t\t},\n\t\t\tfunc(err error) error {\n\t\t\t\tif !causeInTypes(err, testError2{}, testError{}) {\n\t\t\t\t\tpanic(err)\n\t\t\t\t}\n\t\t\t\treturn Unwrap(err)\n\t\t\t})\n\t}())\n}\n\nfunc TestUnwrap(t *testing.T) {\n\tassert := assert.New(t)\n\n\terr := errors.New(\"test\")\n\twe := wrappedError{\"test msg\", err}\n\tassert.Equal(err, Unwrap(err))\n\tassert.Equal(err, Unwrap(we))\n}\n\nfunc TestPanicIfTrue(t *testing.T) {\n\tassert := assert.New(t)\n\n\targ := \"arg value\"\n\tformat := \"could be a format: %s\"\n\tformatted := fmt.Sprintf(format, arg)\n\n\tassert.Panics(func() {\n\t\tPanicIfTrue(true)\n\t})\n\n\tassert.Panics(func() {\n\t\tPanicIfTrue(true)\n\t})\n\n\tassert.NotPanics(func() {\n\t\tPanicIfTrue(false)\n\t})\n\n\terr := Try(func() {\n\t\tPanic(format)\n\t})\n\tassert.Equal(errors.New(format), Unwrap(err))\n\n\terr = Try(func() {\n\t\tPanic(format, arg)\n\t})\n\tassert.Equal(errors.New(formatted), Unwrap(err))\n}\n\nfunc TestPanicIfFalse(t *testing.T) {\n\tassert := assert.New(t)\n\n\targ := \"arg value\"\n\tformat := \"could be a format: %s\"\n\tformatted := fmt.Sprintf(format, arg)\n\n\tassert.Panics(func() {\n\t\tPanicIfFalse(false)\n\t})\n\n\tassert.Panics(func() {\n\t\tPanicIfFalse(false)\n\t})\n\n\tassert.NotPanics(func() {\n\t\tPanicIfFalse(true)\n\t})\n\n\terr := Try(func() {\n\t\tPanic(format)\n\t})\n\tassert.Equal(errors.New(format), Unwrap(err))\n\n\terr = Try(func() {\n\t\tPanic(format, arg)\n\t})\n\tassert.Equal(errors.New(formatted), Unwrap(err))\n}\n\nfunc TestPanicIfNotType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tte := testError{\"te\"}\n\tte2 := testError2{\"te2\"}\n\n\tassert.Panics(func() {\n\t\tPanicIfNotType(te, te2)\n\t})\n\n\tassert.Equal(te, PanicIfNotType(te, te))\n\tassert.Equal(te2, PanicIfNotType(te2, te, te2))\n}\n\nfunc TestCauseInTypes(t *testing.T) {\n\tassert := assert.New(t)\n\n\tte := testError{\"te\"}\n\tte2 := testError2{\"te2\"}\n\n\tassert.True(causeInTypes(te, te))\n\tassert.True(causeInTypes(te, te2, te))\n\tassert.False(causeInTypes(te, te2))\n\tassert.False(causeInTypes(te))\n}\n\nfunc TestWrap(t *testing.T) {\n\tassert := assert.New(t)\n\n\tte := testError{\"te\"}\n\twe := Wrap(te)\n\tassert.Equal(te, we.Cause())\n\tassert.Equal(te, errors.Unwrap(we))\n\tassert.IsType(wrappedError{}, we)\n\tassert.Equal(we, Wrap(we))\n\tfmt.Printf(\"st: %s, cause: %s\\n\", we.Error(), we.Cause())\n\tassert.Nil(Wrap(nil))\n}\n" }, { "path": "go/datas/commit.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/nomdl\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\nconst (\n\tParentsField = \"parents\"\n\tValueField = \"value\"\n\tMetaField = \"meta\"\n\tcommitName = \"Commit\"\n)\n\nvar commitTemplate = types.MakeStructTemplate(commitName, []string{MetaField, ParentsField, ValueField})\n\nvar valueCommitType = nomdl.MustParseType(`Struct Commit {\n meta: Struct {},\n parents: Set>>,\n value: Value,\n}`)\n\n// NewCommit creates a new commit object.\n//\n// A commit has the following type:\n//\n// ```\n// struct Commit {\n// meta: M,\n// parents: Set>>,\n// value: T,\n// }\n// ```\n// where M is a struct type and T is any type.\nfunc NewCommit(value types.Value, parents types.Set, meta types.Struct) types.Struct {\n\treturn commitTemplate.NewStruct([]types.Value{meta, parents, value})\n}\n\n// FindCommonAncestor returns the most recent common ancestor of c1 and c2, if\n// one exists, setting ok to true. If there is no common ancestor, ok is set\n// to false.\nfunc FindCommonAncestor(c1, c2 types.Ref, vr types.ValueReader) (a types.Ref, ok bool) {\n\tif !IsRefOfCommitType(types.TypeOf(c1)) {\n\t\td.Panic(\"FindCommonAncestor() called on %s\", types.TypeOf(c1).Describe())\n\t}\n\tif !IsRefOfCommitType(types.TypeOf(c2)) {\n\t\td.Panic(\"FindCommonAncestor() called on %s\", types.TypeOf(c2).Describe())\n\t}\n\n\tc1Q, c2Q := &types.RefByHeight{c1}, &types.RefByHeight{c2}\n\tfor !c1Q.Empty() && !c2Q.Empty() {\n\t\tc1Ht, c2Ht := c1Q.MaxHeight(), c2Q.MaxHeight()\n\t\tif c1Ht == c2Ht {\n\t\t\tc1Parents, c2Parents := c1Q.PopRefsOfHeight(c1Ht), c2Q.PopRefsOfHeight(c2Ht)\n\t\t\tif common, ok := findCommonRef(c1Parents, c2Parents); ok {\n\t\t\t\treturn common, true\n\t\t\t}\n\t\t\tparentsToQueue(c1Parents, c1Q, vr)\n\t\t\tparentsToQueue(c2Parents, c2Q, vr)\n\t\t} else if c1Ht > c2Ht {\n\t\t\tparentsToQueue(c1Q.PopRefsOfHeight(c1Ht), c1Q, vr)\n\t\t} else {\n\t\t\tparentsToQueue(c2Q.PopRefsOfHeight(c2Ht), c2Q, vr)\n\t\t}\n\t}\n\treturn\n}\n\nfunc parentsToQueue(refs types.RefSlice, q *types.RefByHeight, vr types.ValueReader) {\n\tfor _, r := range refs {\n\t\tc := r.TargetValue(vr).(types.Struct)\n\t\tp := c.Get(ParentsField).(types.Set)\n\t\tp.IterAll(func(v types.Value) {\n\t\t\tq.PushBack(v.(types.Ref))\n\t\t})\n\t}\n\tsort.Sort(q)\n}\n\nfunc findCommonRef(a, b types.RefSlice) (types.Ref, bool) {\n\ttoRefSet := func(s types.RefSlice) map[hash.Hash]types.Ref {\n\t\tout := map[hash.Hash]types.Ref{}\n\t\tfor _, r := range s {\n\t\t\tout[r.TargetHash()] = r\n\t\t}\n\t\treturn out\n\t}\n\n\taSet, bSet := toRefSet(a), toRefSet(b)\n\tfor s, r := range aSet {\n\t\tif _, present := bSet[s]; present {\n\t\t\treturn r, true\n\t\t}\n\t}\n\treturn types.Ref{}, false\n}\n\nfunc makeCommitStructType(metaType, parentsType, valueType *types.Type) *types.Type {\n\treturn types.MakeStructType(\"Commit\",\n\t\ttypes.StructField{\n\t\t\tName: MetaField,\n\t\t\tType: metaType,\n\t\t},\n\t\ttypes.StructField{\n\t\t\tName: ParentsField,\n\t\t\tType: parentsType,\n\t\t},\n\t\ttypes.StructField{\n\t\t\tName: ValueField,\n\t\t\tType: valueType,\n\t\t},\n\t)\n}\n\nfunc getRefElementType(t *types.Type) *types.Type {\n\td.PanicIfFalse(t.TargetKind() == types.RefKind)\n\treturn t.Desc.(types.CompoundDesc).ElemTypes[0]\n}\n\nfunc IsCommitType(t *types.Type) bool {\n\treturn types.IsSubtype(valueCommitType, t)\n}\n\nfunc IsCommit(v types.Value) bool {\n\treturn types.IsValueSubtypeOf(v, valueCommitType)\n}\n\nfunc IsRefOfCommitType(t *types.Type) bool {\n\treturn t.TargetKind() == types.RefKind && IsCommitType(getRefElementType(t))\n}\n" }, { "path": "go/datas/commit_options.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"github.com/attic-labs/noms/go/merge\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// CommitOptions is used to pass options into Commit.\ntype CommitOptions struct {\n\t// Parents, if provided is the parent commits of the commit we are\n\t// creating.\n\tParents types.Set\n\n\t// Meta is a Struct that describes arbitrary metadata about this Commit,\n\t// e.g. a timestamp or descriptive text.\n\tMeta types.Struct\n\n\t// Policy will be called to attempt to merge this Commit with the current\n\t// Head, if this is not a fast-forward. If Policy is nil, no merging will\n\t// be attempted. Note that because Commit() retries in some cases, Policy\n\t// might also be called multiple times with different values.\n\tPolicy merge.Policy\n}\n" }, { "path": "go/datas/commit_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/nomdl\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestNewCommit(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassertTypeEquals := func(e, a *types.Type) {\n\t\tassert.True(a.Equals(e), \"Actual: %s\\nExpected %s\", a.Describe(), e.Describe())\n\t}\n\n\tstorage := &chunks.TestStorage{}\n\tdb := NewDatabase(storage.NewView())\n\tdefer db.Close()\n\n\tcommit := NewCommit(types.Number(1), types.NewSet(db), types.EmptyStruct)\n\tat := types.TypeOf(commit)\n\tet := makeCommitStructType(\n\t\ttypes.EmptyStructType,\n\t\ttypes.MakeSetType(types.MakeUnionType()),\n\t\ttypes.NumberType,\n\t)\n\tassertTypeEquals(et, at)\n\n\t// Committing another Number\n\tcommit2 := NewCommit(types.Number(2), types.NewSet(db, types.NewRef(commit)), types.EmptyStruct)\n\tat2 := types.TypeOf(commit2)\n\tet2 := nomdl.MustParseType(`Struct Commit {\n meta: Struct {},\n parents: Set>>,\n value: Number,\n }`)\n\tassertTypeEquals(et2, at2)\n\n\t// Now commit a String\n\tcommit3 := NewCommit(types.String(\"Hi\"), types.NewSet(db, types.NewRef(commit2)), types.EmptyStruct)\n\tat3 := types.TypeOf(commit3)\n\tet3 := nomdl.MustParseType(`Struct Commit {\n meta: Struct {},\n parents: Set>>,\n value: Number | String,\n }`)\n\tassertTypeEquals(et3, at3)\n\n\t// Now commit a String with MetaInfo\n\tmeta := types.NewStruct(\"Meta\", types.StructData{\"date\": types.String(\"some date\"), \"number\": types.Number(9)})\n\tmetaType := nomdl.MustParseType(`Struct Meta {\n date: String,\n number: Number,\n\t}`)\n\tassertTypeEquals(metaType, types.TypeOf(meta))\n\tcommit4 := NewCommit(types.String(\"Hi\"), types.NewSet(db, types.NewRef(commit2)), meta)\n\tat4 := types.TypeOf(commit4)\n\tet4 := nomdl.MustParseType(`Struct Commit {\n meta: Struct {} | Struct Meta {\n date: String,\n number: Number,\n \t},\n parents: Set>>,\n value: Number | String,\n }`)\n\tassertTypeEquals(et4, at4)\n\n\t// Merge-commit with different parent types\n\tcommit5 := NewCommit(types.String(\"Hi\"), types.NewSet(db, types.NewRef(commit2), types.NewRef(commit3)), types.EmptyStruct)\n\tat5 := types.TypeOf(commit5)\n\tet5 := nomdl.MustParseType(`Struct Commit {\n meta: Struct {},\n parents: Set>>,\n value: Number | String,\n }`)\n\tassertTypeEquals(et5, at5)\n}\n\nfunc TestCommitWithoutMetaField(t *testing.T) {\n\tassert := assert.New(t)\n\n\tstorage := &chunks.TestStorage{}\n\tdb := NewDatabase(storage.NewView())\n\tdefer db.Close()\n\n\tmetaCommit := types.NewStruct(\"Commit\", types.StructData{\n\t\t\"value\": types.Number(9),\n\t\t\"parents\": types.NewSet(db),\n\t\t\"meta\": types.EmptyStruct,\n\t})\n\tassert.True(IsCommit(metaCommit))\n\tassert.True(IsCommitType(types.TypeOf(metaCommit)))\n\n\tnoMetaCommit := types.NewStruct(\"Commit\", types.StructData{\n\t\t\"value\": types.Number(9),\n\t\t\"parents\": types.NewSet(db),\n\t})\n\tassert.False(IsCommit(noMetaCommit))\n\tassert.False(IsCommitType(types.TypeOf(noMetaCommit)))\n}\n\n// Convert list of Struct's to Set\nfunc toRefSet(vrw types.ValueReadWriter, commits ...types.Struct) types.Set {\n\tset := types.NewSet(vrw).Edit()\n\tfor _, p := range commits {\n\t\tset.Insert(types.NewRef(p))\n\t}\n\treturn set.Set()\n}\n\n// Convert Set> to a string of Struct.Get(\"value\")'s\nfunc toValuesString(refSet types.Set, vr types.ValueReader) string {\n\tvalues := []string{}\n\trefSet.IterAll(func(v types.Value) {\n\t\tvalues = append(values, fmt.Sprintf(\"%v\", v.(types.Ref).TargetValue(vr).(types.Struct).Get(\"value\")))\n\t})\n\treturn strings.Join(values, \",\")\n}\n\nfunc TestFindCommonAncestor(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.TestStorage{}\n\tdb := NewDatabase(storage.NewView())\n\tdefer db.Close()\n\n\t// Add a commit and return it\n\taddCommit := func(datasetID string, val string, parents ...types.Struct) types.Struct {\n\t\tds := db.GetDataset(datasetID)\n\t\tvar err error\n\t\tds, err = db.Commit(ds, types.String(val), CommitOptions{Parents: toRefSet(db, parents...)})\n\t\tassert.NoError(err)\n\t\treturn ds.Head()\n\t}\n\n\t// Assert that c is the common ancestor of a and b\n\tassertCommonAncestor := func(expected, a, b types.Struct) {\n\t\tif found, ok := FindCommonAncestor(types.NewRef(a), types.NewRef(b), db); assert.True(ok) {\n\t\t\tancestor := found.TargetValue(db).(types.Struct)\n\t\t\tassert.True(\n\t\t\t\texpected.Equals(ancestor),\n\t\t\t\t\"%s should be common ancestor of %s, %s. Got %s\",\n\t\t\t\texpected.Get(ValueField),\n\t\t\t\ta.Get(ValueField),\n\t\t\t\tb.Get(ValueField),\n\t\t\t\tancestor.Get(ValueField),\n\t\t\t)\n\t\t}\n\t}\n\n\t// Build commit DAG\n\t//\n\t// ds-a: a1<-a2<-a3<-a4<-a5<-a6\n\t// ^ ^ ^ |\n\t// | \\ \\----\\ /-/\n\t// | \\ \\V\n\t// ds-b: \\ b3<-b4<-b5\n\t// \\\n\t// \\\n\t// ds-c: c2<-c3\n\t// /\n\t// /\n\t// V\n\t// ds-d: d1<-d2\n\t//\n\ta, b, c, d := \"ds-a\", \"ds-b\", \"ds-c\", \"ds-d\"\n\ta1 := addCommit(a, \"a1\")\n\td1 := addCommit(d, \"d1\")\n\ta2 := addCommit(a, \"a2\", a1)\n\tc2 := addCommit(c, \"c2\", a1)\n\td2 := addCommit(d, \"d2\", d1)\n\ta3 := addCommit(a, \"a3\", a2)\n\tb3 := addCommit(b, \"b3\", a2)\n\tc3 := addCommit(c, \"c3\", c2, d2)\n\ta4 := addCommit(a, \"a4\", a3)\n\tb4 := addCommit(b, \"b4\", b3)\n\ta5 := addCommit(a, \"a5\", a4)\n\tb5 := addCommit(b, \"b5\", b4, a3)\n\ta6 := addCommit(a, \"a6\", a5, b5)\n\n\tassertCommonAncestor(a1, a1, a1) // All self\n\tassertCommonAncestor(a1, a1, a2) // One side self\n\tassertCommonAncestor(a2, a3, b3) // Common parent\n\tassertCommonAncestor(a2, a4, b4) // Common grandparent\n\tassertCommonAncestor(a1, a6, c3) // Traversing multiple parents on both sides\n\n\t// No common ancestor\n\tif found, ok := FindCommonAncestor(types.NewRef(d2), types.NewRef(a6), db); !assert.False(ok) {\n\t\tassert.Fail(\n\t\t\t\"Unexpected common ancestor!\",\n\t\t\t\"Should be no common ancestor of %s, %s. Got %s\",\n\t\t\td2.Get(ValueField),\n\t\t\ta6.Get(ValueField),\n\t\t\tfound.TargetValue(db).(types.Struct).Get(ValueField),\n\t\t)\n\t}\n}\n\nfunc TestNewCommitRegressionTest(t *testing.T) {\n\tstorage := &chunks.TestStorage{}\n\tdb := NewDatabase(storage.NewView())\n\tdefer db.Close()\n\n\tc1 := NewCommit(types.String(\"one\"), types.NewSet(db), types.EmptyStruct)\n\tcx := NewCommit(types.Bool(true), types.NewSet(db), types.EmptyStruct)\n\tvalue := types.String(\"two\")\n\tparents := types.NewSet(db, types.NewRef(c1))\n\tmeta := types.NewStruct(\"\", types.StructData{\n\t\t\"basis\": cx,\n\t})\n\n\t// Used to fail\n\tNewCommit(value, parents, meta)\n}\n" }, { "path": "go/datas/database.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package datas defines and implements the database layer used in Noms.\npackage datas\n\nimport (\n\t\"io\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// Database provides versioned storage for noms values. While Values can be\n// directly read and written from a Database, it is generally more appropriate\n// to read data by inspecting the Head of a Dataset and write new data by\n// updating the Head of a Dataset via Commit() or similar. Particularly, new\n// data is not guaranteed to be persistent until after a Commit (Delete,\n// SetHead, or FastForward) operation completes.\n// The Database API is stateful, meaning that calls to GetDataset() or\n// Datasets() occurring after a call to Commit() (et al) will represent the\n// result of the Commit().\ntype Database interface {\n\t// To implement types.ValueWriter, Database implementations provide\n\t// WriteValue(). WriteValue() writes v to this Database, though v is not\n\t// guaranteed to be be persistent until after a subsequent Commit(). The\n\t// return value is the Ref of v.\n\t// Written values won't be persisted until a commit-alike\n\ttypes.ValueReadWriter\n\n\t// Close must have no side-effects\n\tio.Closer\n\n\t// Datasets returns the root of the database which is a\n\t// Map> where string is a datasetID.\n\tDatasets() types.Map\n\n\t// GetDataset returns a Dataset struct containing the current mapping of\n\t// datasetID in the above Datasets Map.\n\tGetDataset(datasetID string) Dataset\n\n\t// Rebase brings this Database's view of the world inline with upstream.\n\tRebase()\n\n\t// Commit updates the Commit that ds.ID() in this database points at. All\n\t// Values that have been written to this Database are guaranteed to be\n\t// persistent after Commit() returns.\n\t// The new Commit struct is constructed using v, opts.Parents, and\n\t// opts.Meta. If opts.Parents is the zero value (types.Set{}) then\n\t// the current head is used. If opts.Meta is the zero value\n\t// (types.Struct{}) then a fully initialized empty Struct is passed to\n\t// NewCommit.\n\t// The returned Dataset is always the newest snapshot, regardless of\n\t// success or failure, and Datasets() is updated to match backing storage\n\t// upon return as well. If the update cannot be performed, e.g., because\n\t// of a conflict, Commit returns an 'ErrMergeNeeded' error.\n\tCommit(ds Dataset, v types.Value, opts CommitOptions) (Dataset, error)\n\n\t// CommitValue updates the Commit that ds.ID() in this database points at.\n\t// All Values that have been written to this Database are guaranteed to be\n\t// persistent after Commit().\n\t// The new Commit struct is constructed using `v`, and the current Head of\n\t// `ds` as the lone Parent.\n\t// The returned Dataset is always the newest snapshot, regardless of\n\t// success or failure, and Datasets() is updated to match backing storage\n\t// upon return as well. If the update cannot be performed, e.g., because\n\t// of a conflict, Commit returns an 'ErrMergeNeeded' error.\n\tCommitValue(ds Dataset, v types.Value) (Dataset, error)\n\n\t// Delete removes the Dataset named ds.ID() from the map at the root of\n\t// the Database. The Dataset data is not necessarily cleaned up at this\n\t// time, but may be garbage collected in the future.\n\t// The returned Dataset is always the newest snapshot, regardless of\n\t// success or failure, and Datasets() is updated to match backing storage\n\t// upon return as well. If the update cannot be performed, e.g., because\n\t// of a conflict, Delete returns an 'ErrMergeNeeded' error.\n\tDelete(ds Dataset) (Dataset, error)\n\n\t// SetHead ignores any lineage constraints (e.g. the current Head being in\n\t// commit’s Parent set) and force-sets a mapping from datasetID: commit in\n\t// this database.\n\t// All Values that have been written to this Database are guaranteed to be\n\t// persistent after SetHead(). If the update cannot be performed, e.g.,\n\t// because another process moved the current Head out from under you,\n\t// error will be non-nil.\n\t// The newest snapshot of the Dataset is always returned, so the caller an\n\t// easily retry using the latest.\n\t// Regardless, Datasets() is updated to match backing storage upon return.\n\tSetHead(ds Dataset, newHeadRef types.Ref) (Dataset, error)\n\n\t// FastForward takes a types.Ref to a Commit object and makes it the new\n\t// Head of ds iff it is a descendant of the current Head. Intended to be\n\t// used e.g. after a call to Pull(). If the update cannot be performed,\n\t// e.g., because another process moved the current Head out from under\n\t// you, err will be non-nil.\n\t// The newest snapshot of the Dataset is always returned, so the caller\n\t// can easily retry using the latest.\n\t// Regardless, Datasets() is updated to match backing storage upon return.\n\tFastForward(ds Dataset, newHeadRef types.Ref) (Dataset, error)\n\n\t// Stats may return some kind of struct that reports statistics about the\n\t// ChunkStore that backs this Database instance. The type is\n\t// implementation-dependent, and impls may return nil\n\tStats() interface{}\n\n\t// StatsSummary may return a string containing summarized statistics for\n\t// the ChunkStore that backs this Database. It must return \"Unsupported\"\n\t// if this operation is not supported.\n\tStatsSummary() string\n\n\tFlush()\n\n\t// chunkStore returns the ChunkStore used to read and write\n\t// groups of values to the database efficiently. This interface is a low-\n\t// level detail of the database that should infrequently be needed by\n\t// clients.\n\tchunkStore() chunks.ChunkStore\n}\n\nfunc NewDatabase(cs chunks.ChunkStore) Database {\n\treturn newDatabase(cs)\n}\n" }, { "path": "go/datas/database_common.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/merge\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/random\"\n)\n\ntype database struct {\n\t*types.ValueStore\n\trt rootTracker\n}\n\nvar (\n\tErrOptimisticLockFailed = errors.New(\"Optimistic lock failed on database Root update\")\n\tErrMergeNeeded = errors.New(\"Dataset head is not ancestor of commit\")\n)\n\n// rootTracker is a narrowing of the ChunkStore interface, to keep Database disciplined about working directly with Chunks\ntype rootTracker interface {\n\tRebase()\n\tRoot() hash.Hash\n\tCommit(current, last hash.Hash) bool\n}\n\nfunc newDatabase(cs chunks.ChunkStore) *database {\n\tvs := types.NewValueStore(cs)\n\tif _, ok := cs.(*httpChunkStore); ok {\n\t\tvs.SetEnforceCompleteness(false)\n\t}\n\n\treturn &database{\n\t\tValueStore: vs, // ValueStore is responsible for closing |cs|\n\t\trt: vs,\n\t}\n}\n\nfunc (db *database) chunkStore() chunks.ChunkStore {\n\treturn db.ChunkStore()\n}\n\nfunc (db *database) Stats() interface{} {\n\treturn db.ChunkStore().Stats()\n}\n\nfunc (db *database) StatsSummary() string {\n\treturn db.ChunkStore().StatsSummary()\n}\n\nfunc (db *database) Flush() {\n\t// TODO: This is a pretty ghetto hack - do better.\n\t// See: https://github.com/attic-labs/noms/issues/3530\n\tds := db.GetDataset(fmt.Sprintf(\"-/flush/%s\", random.Id()))\n\tr := db.WriteValue(types.Bool(true))\n\tds, err := db.CommitValue(ds, r)\n\td.PanicIfError(err)\n\t_, err = db.Delete(ds)\n\td.PanicIfError(err)\n}\n\nfunc (db *database) Datasets() types.Map {\n\trootHash := db.rt.Root()\n\tif rootHash.IsEmpty() {\n\t\treturn types.NewMap(db)\n\t}\n\n\treturn db.ReadValue(rootHash).(types.Map)\n}\n\nfunc (db *database) GetDataset(datasetID string) Dataset {\n\tif !DatasetFullRe.MatchString(datasetID) {\n\t\td.Panic(\"Invalid dataset ID: %s\", datasetID)\n\t}\n\tvar head types.Value\n\tif r, ok := db.Datasets().MaybeGet(types.String(datasetID)); ok {\n\t\thead = r.(types.Ref).TargetValue(db)\n\t}\n\n\treturn newDataset(db, datasetID, head)\n}\n\nfunc (db *database) Rebase() {\n\tdb.rt.Rebase()\n}\n\nfunc (db *database) Close() error {\n\treturn db.ValueStore.Close()\n}\n\nfunc (db *database) SetHead(ds Dataset, newHeadRef types.Ref) (Dataset, error) {\n\treturn db.doHeadUpdate(ds, func(ds Dataset) error { return db.doSetHead(ds, newHeadRef) })\n}\n\nfunc (db *database) doSetHead(ds Dataset, newHeadRef types.Ref) error {\n\tif currentHeadRef, ok := ds.MaybeHeadRef(); ok && newHeadRef.Equals(currentHeadRef) {\n\t\treturn nil\n\t}\n\tcommit := db.validateRefAsCommit(newHeadRef)\n\n\tcurrentRootHash, currentDatasets := db.rt.Root(), db.Datasets()\n\tcommitRef := db.WriteValue(commit) // will be orphaned if the tryCommitChunks() below fails\n\n\tcurrentDatasets = currentDatasets.Edit().Set(types.String(ds.ID()), types.ToRefOfValue(commitRef)).Map()\n\treturn db.tryCommitChunks(currentDatasets, currentRootHash)\n}\n\nfunc (db *database) FastForward(ds Dataset, newHeadRef types.Ref) (Dataset, error) {\n\treturn db.doHeadUpdate(ds, func(ds Dataset) error { return db.doFastForward(ds, newHeadRef) })\n}\n\nfunc (db *database) doFastForward(ds Dataset, newHeadRef types.Ref) error {\n\tcurrentHeadRef, ok := ds.MaybeHeadRef()\n\tif ok && newHeadRef.Equals(currentHeadRef) {\n\t\treturn nil\n\t}\n\n\tif ok && newHeadRef.Height() <= currentHeadRef.Height() {\n\t\treturn ErrMergeNeeded\n\t}\n\n\tcommit := db.validateRefAsCommit(newHeadRef)\n\treturn db.doCommit(ds.ID(), commit, nil)\n}\n\nfunc (db *database) Commit(ds Dataset, v types.Value, opts CommitOptions) (Dataset, error) {\n\treturn db.doHeadUpdate(\n\t\tds,\n\t\tfunc(ds Dataset) error { return db.doCommit(ds.ID(), buildNewCommit(ds, v, opts), opts.Policy) },\n\t)\n}\n\nfunc (db *database) CommitValue(ds Dataset, v types.Value) (Dataset, error) {\n\treturn db.Commit(ds, v, CommitOptions{})\n}\n\n// doCommit manages concurrent access the single logical piece of mutable state: the current Root. doCommit is optimistic in that it is attempting to update head making the assumption that currentRootHash is the hash of the current head. The call to Commit below will return an 'ErrOptimisticLockFailed' error if that assumption fails (e.g. because of a race with another writer) and the entire algorithm must be tried again. This method will also fail and return an 'ErrMergeNeeded' error if the |commit| is not a descendent of the current dataset head\nfunc (db *database) doCommit(datasetID string, commit types.Struct, mergePolicy merge.Policy) error {\n\tif !IsCommit(commit) {\n\t\td.Panic(\"Can't commit a non-Commit struct to dataset %s\", datasetID)\n\t}\n\n\t// This could loop forever, given enough simultaneous committers. BUG 2565\n\tvar err error\n\tfor err = ErrOptimisticLockFailed; err == ErrOptimisticLockFailed; {\n\t\tcurrentRootHash, currentDatasets := db.rt.Root(), db.Datasets()\n\t\tcommitRef := db.WriteValue(commit) // will be orphaned if the tryCommitChunks() below fails\n\n\t\t// If there's nothing in the DB yet, skip all this logic.\n\t\tif !currentRootHash.IsEmpty() {\n\t\t\tr, hasHead := currentDatasets.MaybeGet(types.String(datasetID))\n\n\t\t\t// First commit in dataset is always fast-forward, so go through all this iff there's already a Head for datasetID.\n\t\t\tif hasHead {\n\t\t\t\thead := r.(types.Ref).TargetValue(db)\n\t\t\t\tcurrentHeadRef := types.NewRef(head)\n\t\t\t\tancestorRef, found := FindCommonAncestor(commitRef, currentHeadRef, db)\n\t\t\t\tif !found {\n\t\t\t\t\treturn ErrMergeNeeded\n\t\t\t\t}\n\n\t\t\t\t// This covers all cases where currentHeadRef is not an ancestor of commit, including the following edge cases:\n\t\t\t\t// - commit is a duplicate of currentHead.\n\t\t\t\t// - we hit an ErrOptimisticLockFailed and looped back around because some other process changed the Head out from under us.\n\t\t\t\tif currentHeadRef.TargetHash() != ancestorRef.TargetHash() || currentHeadRef.TargetHash() == commitRef.TargetHash() {\n\t\t\t\t\tif mergePolicy == nil {\n\t\t\t\t\t\treturn ErrMergeNeeded\n\t\t\t\t\t}\n\n\t\t\t\t\tancestor, currentHead := db.validateRefAsCommit(ancestorRef), db.validateRefAsCommit(currentHeadRef)\n\t\t\t\t\tmerged, err := mergePolicy(commit.Get(ValueField), currentHead.Get(ValueField), ancestor.Get(ValueField), db, nil)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\treturn err\n\t\t\t\t\t}\n\t\t\t\t\tcommitRef = db.WriteValue(NewCommit(merged, types.NewSet(db, commitRef, currentHeadRef), types.EmptyStruct))\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tcurrentDatasets = currentDatasets.Edit().Set(types.String(datasetID), types.ToRefOfValue(commitRef)).Map()\n\t\terr = db.tryCommitChunks(currentDatasets, currentRootHash)\n\t}\n\treturn err\n}\n\nfunc (db *database) Delete(ds Dataset) (Dataset, error) {\n\treturn db.doHeadUpdate(ds, func(ds Dataset) error { return db.doDelete(ds.ID()) })\n}\n\n// doDelete manages concurrent access the single logical piece of mutable state: the current Root. doDelete is optimistic in that it is attempting to update head making the assumption that currentRootHash is the hash of the current head. The call to Commit below will return an 'ErrOptimisticLockFailed' error if that assumption fails (e.g. because of a race with another writer) and the entire algorithm must be tried again.\nfunc (db *database) doDelete(datasetIDstr string) error {\n\tdatasetID := types.String(datasetIDstr)\n\tcurrentRootHash, currentDatasets := db.rt.Root(), db.Datasets()\n\tvar initialHead types.Ref\n\tif r, hasHead := currentDatasets.MaybeGet(datasetID); !hasHead {\n\t\treturn nil\n\t} else {\n\t\tinitialHead = r.(types.Ref)\n\t}\n\n\tvar err error\n\tfor {\n\t\tcurrentDatasets = currentDatasets.Edit().Remove(datasetID).Map()\n\t\terr = db.tryCommitChunks(currentDatasets, currentRootHash)\n\t\tif err != ErrOptimisticLockFailed {\n\t\t\tbreak\n\t\t}\n\t\t// If the optimistic lock failed because someone changed the Head of datasetID, then return ErrMergeNeeded. If it failed because someone changed a different Dataset, we should try again.\n\t\tcurrentRootHash, currentDatasets = db.rt.Root(), db.Datasets()\n\t\tif r, hasHead := currentDatasets.MaybeGet(datasetID); !hasHead || (hasHead && !initialHead.Equals(r)) {\n\t\t\terr = ErrMergeNeeded\n\t\t\tbreak\n\t\t}\n\t}\n\treturn err\n}\n\nfunc (db *database) tryCommitChunks(currentDatasets types.Map, currentRootHash hash.Hash) (err error) {\n\tnewRootHash := db.WriteValue(currentDatasets).TargetHash()\n\n\tif !db.rt.Commit(newRootHash, currentRootHash) {\n\t\terr = ErrOptimisticLockFailed\n\t}\n\treturn\n}\n\nfunc (db *database) validateRefAsCommit(r types.Ref) types.Struct {\n\tv := db.ReadValue(r.TargetHash())\n\n\tif v == nil {\n\t\tpanic(r.TargetHash().String() + \" not found\")\n\t}\n\tif !IsCommit(v) {\n\t\tpanic(\"Not a commit: \" + types.EncodedValueMaxLines(v, 10) + \" ...\\n\")\n\t}\n\treturn v.(types.Struct)\n}\n\nfunc buildNewCommit(ds Dataset, v types.Value, opts CommitOptions) types.Struct {\n\tparents := opts.Parents\n\tif (parents == types.Set{}) {\n\t\tparents = types.NewSet(ds.Database())\n\t\tif headRef, ok := ds.MaybeHeadRef(); ok {\n\t\t\tparents = parents.Edit().Insert(headRef).Set()\n\t\t}\n\t}\n\n\tmeta := opts.Meta\n\tif meta.IsZeroValue() {\n\t\tmeta = types.EmptyStruct\n\t}\n\treturn NewCommit(v, parents, meta)\n}\n\nfunc (db *database) doHeadUpdate(ds Dataset, updateFunc func(ds Dataset) error) (Dataset, error) {\n\terr := updateFunc(ds)\n\treturn db.GetDataset(ds.ID()), err\n}\n" }, { "path": "go/datas/database_server.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"net\"\n\t\"net/http\"\n\t\"strconv\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/julienschmidt/httprouter\"\n)\n\ntype connectionState struct {\n\tc net.Conn\n\tcs http.ConnState\n}\n\ntype RemoteDatabaseServer struct {\n\tcs chunks.ChunkStore\n\taddress string\n\tport int\n\tl *net.Listener\n\tcsChan chan *connectionState\n\tclosing bool\n\t// Called just before the server is started.\n\tReady func()\n}\n\nfunc NewRemoteDatabaseServer(cs chunks.ChunkStore, address string, port int) *RemoteDatabaseServer {\n\tdataVersion := cs.Version()\n\tif constants.NomsVersion != dataVersion {\n\t\td.Panic(\"SDK version %s is incompatible with data of version %s\", constants.NomsVersion, dataVersion)\n\t}\n\treturn &RemoteDatabaseServer{\n\t\tcs, address, port, nil, make(chan *connectionState, 16), false, func() {},\n\t}\n}\n\n// Port is the actual port used. This may be different than the port passed in to NewRemoteDatabaseServer.\nfunc (s *RemoteDatabaseServer) Port() int {\n\treturn s.port\n}\n\nfunc Router(cs chunks.ChunkStore, prefix string) *httprouter.Router {\n\trouter := httprouter.New()\n\n\trouter.POST(prefix+constants.GetRefsPath, corsHandle(makeHandle(HandleGetRefs, cs)))\n\trouter.GET(prefix+constants.GetBlobPath, corsHandle(makeHandle(HandleGetBlob, cs)))\n\trouter.OPTIONS(prefix+constants.GetRefsPath, corsHandle(noopHandle))\n\trouter.POST(prefix+constants.HasRefsPath, corsHandle(makeHandle(HandleHasRefs, cs)))\n\trouter.OPTIONS(prefix+constants.HasRefsPath, corsHandle(noopHandle))\n\trouter.GET(prefix+constants.RootPath, corsHandle(makeHandle(HandleRootGet, cs)))\n\trouter.POST(prefix+constants.RootPath, corsHandle(makeHandle(HandleRootPost, cs)))\n\trouter.OPTIONS(prefix+constants.RootPath, corsHandle(noopHandle))\n\trouter.POST(prefix+constants.WriteValuePath, corsHandle(makeHandle(HandleWriteValue, cs)))\n\trouter.OPTIONS(prefix+constants.WriteValuePath, corsHandle(noopHandle))\n\trouter.GET(prefix+constants.BasePath, corsHandle(makeHandle(HandleBaseGet, cs)))\n\n\trouter.GET(prefix+constants.GraphQLPath, corsHandle(makeHandle(HandleGraphQL, cs)))\n\trouter.POST(prefix+constants.GraphQLPath, corsHandle(makeHandle(HandleGraphQL, cs)))\n\trouter.OPTIONS(prefix+constants.GraphQLPath, corsHandle(noopHandle))\n\n\trouter.GET(prefix+constants.StatsPath, corsHandle(makeHandle(HandleStats, cs)))\n\trouter.OPTIONS(prefix+constants.StatsPath, corsHandle(noopHandle))\n\n\treturn router\n}\n\n// Run blocks while the RemoteDatabaseServer is listening. Running on a separate go routine is supported.\nfunc (s *RemoteDatabaseServer) Run() {\n\n\tl, err := net.Listen(\"tcp\", fmt.Sprintf(\"%s:%d\", s.address, s.port))\n\td.Chk.NoError(err)\n\ts.l = &l\n\t_, port, err := net.SplitHostPort(l.Addr().String())\n\td.Chk.NoError(err)\n\ts.port, err = strconv.Atoi(port)\n\td.Chk.NoError(err)\n\tlog.Printf(\"Listening on %s:%d...\\n\", s.address, s.port)\n\n\trouter := Router(s.cs, \"\")\n\n\tsrv := &http.Server{\n\t\tHandler: http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {\n\t\t\trouter.ServeHTTP(w, req)\n\t\t}),\n\t\tConnState: s.connState,\n\t}\n\n\tgo func() {\n\t\tm := map[net.Conn]http.ConnState{}\n\t\tfor connState := range s.csChan {\n\t\t\tswitch connState.cs {\n\t\t\tcase http.StateNew, http.StateActive, http.StateIdle:\n\t\t\t\tm[connState.c] = connState.cs\n\t\t\tdefault:\n\t\t\t\tdelete(m, connState.c)\n\t\t\t}\n\t\t}\n\t\tfor c := range m {\n\t\t\tc.Close()\n\t\t}\n\t}()\n\n\tgo s.Ready()\n\tsrv.Serve(l)\n}\n\nfunc makeHandle(hndlr Handler, cs chunks.ChunkStore) httprouter.Handle {\n\treturn func(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\thndlr(w, req, ps, cs)\n\t}\n}\n\nfunc noopHandle(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {\n}\n\nfunc corsHandle(f httprouter.Handle) httprouter.Handle {\n\t// TODO: Implement full pre-flighting?\n\t// See: http://www.html5rocks.com/static/images/cors_server_flowchart.png\n\treturn func(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {\n\t\t// Can't use * when clients are using cookies.\n\t\tw.Header().Add(\"Access-Control-Allow-Origin\", r.Header.Get(\"Origin\"))\n\t\tw.Header().Add(\"Access-Control-Allow-Methods\", \"GET, POST\")\n\t\tw.Header().Add(\"Access-Control-Allow-Headers\", \"*\")\n\t\tw.Header().Add(\"Access-Control-Expose-Headers\", NomsVersionHeader)\n\t\tw.Header().Add(NomsVersionHeader, constants.NomsVersion)\n\t\tf(w, r, ps)\n\t}\n}\n\nfunc (s *RemoteDatabaseServer) connState(c net.Conn, cs http.ConnState) {\n\tif s.closing {\n\t\td.PanicIfFalse(cs == http.StateClosed)\n\t\treturn\n\t}\n\ts.csChan <- &connectionState{c, cs}\n}\n\n// Will cause the RemoteDatabaseServer to stop listening and an existing call to Run() to continue.\nfunc (s *RemoteDatabaseServer) Stop() {\n\ts.closing = true\n\t(*s.l).Close()\n\t(s.cs).Close()\n\tclose(s.csChan)\n}\n" }, { "path": "go/datas/database_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/merge\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc TestLocalDatabase(t *testing.T) {\n\tsuite.Run(t, &LocalDatabaseSuite{})\n}\n\nfunc TestRemoteDatabase(t *testing.T) {\n\tsuite.Run(t, &RemoteDatabaseSuite{})\n}\n\nfunc TestValidateRef(t *testing.T) {\n\tst := &chunks.TestStorage{}\n\tdb := NewDatabase(st.NewView()).(*database)\n\tdefer db.Close()\n\tb := types.Bool(true)\n\tr := db.WriteValue(b)\n\n\tassert.Panics(t, func() { db.validateRefAsCommit(r) })\n\tassert.Panics(t, func() { db.validateRefAsCommit(types.NewRef(b)) })\n}\n\ntype DatabaseSuite struct {\n\tsuite.Suite\n\tstorage *chunks.TestStorage\n\tdb Database\n\tmakeDb func(chunks.ChunkStore) Database\n}\n\ntype LocalDatabaseSuite struct {\n\tDatabaseSuite\n}\n\nfunc (suite *LocalDatabaseSuite) SetupTest() {\n\tsuite.storage = &chunks.TestStorage{}\n\tsuite.makeDb = NewDatabase\n\tsuite.db = suite.makeDb(suite.storage.NewView())\n}\n\ntype RemoteDatabaseSuite struct {\n\tDatabaseSuite\n}\n\nfunc (suite *RemoteDatabaseSuite) SetupTest() {\n\tsuite.storage = &chunks.TestStorage{}\n\tsuite.makeDb = func(cs chunks.ChunkStore) Database {\n\t\treturn NewDatabase(newHTTPChunkStoreForTest(cs))\n\t}\n\tsuite.db = suite.makeDb(suite.storage.NewView())\n}\n\nfunc (suite *DatabaseSuite) TearDownTest() {\n\tsuite.db.Close()\n}\n\nfunc (suite *RemoteDatabaseSuite) TestWriteRefToNonexistentValue() {\n\tds := suite.db.GetDataset(\"foo\")\n\tr := types.NewRef(types.Bool(true))\n\tsuite.Panics(func() { suite.db.CommitValue(ds, r) })\n}\n\nfunc (suite *DatabaseSuite) TestTolerateUngettableRefs() {\n\tsuite.Nil(suite.db.ReadValue(hash.Hash{}))\n}\n\nfunc (suite *DatabaseSuite) TestCompletenessCheck() {\n\tdatasetID := \"ds1\"\n\tds1 := suite.db.GetDataset(datasetID)\n\n\tse := types.NewSet(suite.db).Edit()\n\tfor i := 0; i < 100; i++ {\n\t\tse.Insert(suite.db.WriteValue(types.Number(100)))\n\t}\n\ts := se.Set()\n\n\tds1, err := suite.db.CommitValue(ds1, s)\n\tsuite.NoError(err)\n\n\ts = ds1.HeadValue().(types.Set)\n\ts = s.Edit().Insert(types.NewRef(types.Number(1000))).Set() // danging ref\n\tsuite.Panics(func() {\n\t\tds1, err = suite.db.CommitValue(ds1, s)\n\t})\n}\n\nfunc (suite *DatabaseSuite) TestRebase() {\n\tdatasetID := \"ds1\"\n\tds1 := suite.db.GetDataset(datasetID)\n\tvar err error\n\n\t// Setup:\n\t// ds1: |a| <- |b|\n\tds1, err = suite.db.CommitValue(ds1, types.String(\"a\"))\n\tb := types.String(\"b\")\n\tds1, err = suite.db.CommitValue(ds1, b)\n\tsuite.NoError(err)\n\tsuite.True(ds1.HeadValue().Equals(b))\n\n\tinterloper := suite.makeDb(suite.storage.NewView())\n\tdefer interloper.Close()\n\n\t// Concurrent change, to move root out from under my feet:\n\t// ds1: |a| <- |b| <- |e|\n\te := types.String(\"e\")\n\tiDS, concErr := interloper.CommitValue(interloper.GetDataset(datasetID), e)\n\tsuite.NoError(concErr)\n\tsuite.True(iDS.HeadValue().Equals(e))\n\n\t// suite.ds shouldn't see the above change yet\n\tsuite.True(suite.db.GetDataset(datasetID).HeadValue().Equals(b))\n\n\tsuite.db.Rebase()\n\tsuite.True(suite.db.GetDataset(datasetID).HeadValue().Equals(e))\n\n\tcs := suite.storage.NewView()\n\tnoChangeDB := suite.makeDb(cs)\n\tnoChangeDB.Datasets()\n\tcs.Reads = 0 // New baseline\n\n\tnoChangeDB.Rebase()\n\tsuite.Zero(cs.Reads)\n}\n\nfunc (suite *DatabaseSuite) TestCommitProperlyTracksRoot() {\n\tid1, id2 := \"testdataset\", \"othertestdataset\"\n\n\tdb1 := suite.makeDb(suite.storage.NewView())\n\tdefer db1.Close()\n\tds1 := db1.GetDataset(id1)\n\tds1HeadVal := types.String(\"Commit value for \" + id1)\n\tds1, err := db1.CommitValue(ds1, ds1HeadVal)\n\tsuite.NoError(err)\n\n\tdb2 := suite.makeDb(suite.storage.NewView())\n\tdefer db2.Close()\n\tds2 := db2.GetDataset(id2)\n\tds2HeadVal := types.String(\"Commit value for \" + id2)\n\tds2, err = db2.CommitValue(ds2, ds2HeadVal)\n\tsuite.NoError(err)\n\n\tsuite.EqualValues(ds1HeadVal, ds1.HeadValue())\n\tsuite.EqualValues(ds2HeadVal, ds2.HeadValue())\n\tsuite.False(ds2.HeadValue().Equals(ds1HeadVal))\n\tsuite.False(ds1.HeadValue().Equals(ds2HeadVal))\n}\n\nfunc (suite *DatabaseSuite) TestDatabaseCommit() {\n\tdatasetID := \"ds1\"\n\tdatasets := suite.db.Datasets()\n\tsuite.Zero(datasets.Len())\n\n\t// |a|\n\tds := suite.db.GetDataset(datasetID)\n\ta := types.String(\"a\")\n\tds2, err := suite.db.CommitValue(ds, a)\n\tsuite.NoError(err)\n\n\t// ds2 matches the Datasets Map in suite.db\n\tsuite.True(ds2.HeadRef().Equals(suite.db.GetDataset(datasetID).HeadRef()))\n\n\t// ds2 has |a| at its head\n\th, ok := ds2.MaybeHeadValue()\n\tsuite.True(ok)\n\tsuite.True(h.Equals(a))\n\tsuite.Equal(uint64(1), ds2.HeadRef().Height())\n\n\tds = ds2\n\taCommitRef := ds.HeadRef() // to be used to test disallowing of non-fastforward commits below\n\n\t// |a| <- |b|\n\tb := types.String(\"b\")\n\tds, err = suite.db.CommitValue(ds, b)\n\tsuite.NoError(err)\n\tsuite.True(ds.HeadValue().Equals(b))\n\tsuite.Equal(uint64(2), ds.HeadRef().Height())\n\n\t// |a| <- |b|\n\t// \\----|c|\n\t// Should be disallowed.\n\tc := types.String(\"c\")\n\tds, err = suite.db.Commit(ds, c, newOpts(suite.db, aCommitRef))\n\tsuite.Error(err)\n\tsuite.True(ds.HeadValue().Equals(b))\n\n\t// |a| <- |b| <- |d|\n\td := types.String(\"d\")\n\tds, err = suite.db.CommitValue(ds, d)\n\tsuite.NoError(err)\n\tsuite.True(ds.HeadValue().Equals(d))\n\tsuite.Equal(uint64(3), ds.HeadRef().Height())\n\n\t// Attempt to recommit |b| with |a| as parent.\n\t// Should be disallowed.\n\tds, err = suite.db.Commit(ds, b, newOpts(suite.db, aCommitRef))\n\tsuite.Error(err)\n\tsuite.True(ds.HeadValue().Equals(d))\n\n\t// Add a commit to a different datasetId\n\t_, err = suite.db.CommitValue(suite.db.GetDataset(\"otherDS\"), a)\n\tsuite.NoError(err)\n\n\t// Get a fresh database, and verify that both datasets are present\n\tnewDB := suite.makeDb(suite.storage.NewView())\n\tdefer newDB.Close()\n\tdatasets2 := newDB.Datasets()\n\tsuite.Equal(uint64(2), datasets2.Len())\n}\n\nfunc (suite *DatabaseSuite) TestDatasetsMapType() {\n\tdsID1, dsID2 := \"ds1\", \"ds2\"\n\n\tdatasets := suite.db.Datasets()\n\tds, err := suite.db.CommitValue(suite.db.GetDataset(dsID1), types.String(\"a\"))\n\tsuite.NoError(err)\n\tsuite.NotPanics(func() { assertMapOfStringToRefOfCommit(suite.db.Datasets(), datasets, suite.db) })\n\n\tdatasets = suite.db.Datasets()\n\t_, err = suite.db.CommitValue(suite.db.GetDataset(dsID2), types.Number(42))\n\tsuite.NoError(err)\n\tsuite.NotPanics(func() { assertMapOfStringToRefOfCommit(suite.db.Datasets(), datasets, suite.db) })\n\n\tdatasets = suite.db.Datasets()\n\t_, err = suite.db.Delete(ds)\n\tsuite.NoError(err)\n\tsuite.NotPanics(func() { assertMapOfStringToRefOfCommit(suite.db.Datasets(), datasets, suite.db) })\n}\n\nfunc newOpts(vrw types.ValueReadWriter, parents ...types.Value) CommitOptions {\n\treturn CommitOptions{Parents: types.NewSet(vrw, parents...)}\n}\n\nfunc (suite *DatabaseSuite) TestDatabaseDuplicateCommit() {\n\tdatasetID := \"ds1\"\n\tds := suite.db.GetDataset(datasetID)\n\tdatasets := suite.db.Datasets()\n\tsuite.Zero(datasets.Len())\n\n\tv := types.String(\"Hello\")\n\t_, err := suite.db.CommitValue(ds, v)\n\tsuite.NoError(err)\n\n\t_, err = suite.db.CommitValue(ds, v)\n\tsuite.IsType(ErrMergeNeeded, err)\n}\n\nfunc (suite *DatabaseSuite) TestDatabaseCommitMerge() {\n\tdatasetID1, datasetID2 := \"ds1\", \"ds2\"\n\tds1, ds2 := suite.db.GetDataset(datasetID1), suite.db.GetDataset(datasetID2)\n\n\tvar err error\n\tv := types.NewMap(suite.db, types.String(\"Hello\"), types.Number(42))\n\tds1, err = suite.db.CommitValue(ds1, v)\n\tds1First := ds1\n\tsuite.NoError(err)\n\tds1, err = suite.db.CommitValue(ds1, v.Edit().Set(types.String(\"Friends\"), types.Bool(true)).Map())\n\tsuite.NoError(err)\n\n\tds2, err = suite.db.CommitValue(ds2, types.String(\"Goodbye\"))\n\tsuite.NoError(err)\n\n\t// No common ancestor\n\t_, err = suite.db.Commit(ds1, types.Number(47), newOpts(suite.db, ds2.HeadRef()))\n\tsuite.IsType(ErrMergeNeeded, err, \"%s\", err)\n\n\t// Unmergeable\n\t_, err = suite.db.Commit(ds1, types.Number(47), newOptsWithMerge(suite.db, merge.None, ds1First.HeadRef()))\n\tsuite.IsType(&merge.ErrMergeConflict{}, err, \"%s\", err)\n\n\t// Merge policies\n\tnewV := v.Edit().Set(types.String(\"Friends\"), types.Bool(false)).Map()\n\t_, err = suite.db.Commit(ds1, newV, newOptsWithMerge(suite.db, merge.None, ds1First.HeadRef()))\n\tsuite.IsType(&merge.ErrMergeConflict{}, err, \"%s\", err)\n\n\ttheirs, err := suite.db.Commit(ds1, newV, newOptsWithMerge(suite.db, merge.Theirs, ds1First.HeadRef()))\n\tsuite.NoError(err)\n\tsuite.True(types.Bool(true).Equals(theirs.HeadValue().(types.Map).Get(types.String(\"Friends\"))))\n\n\tnewV = v.Edit().Set(types.String(\"Friends\"), types.Number(47)).Map()\n\tours, err := suite.db.Commit(ds1First, newV, newOptsWithMerge(suite.db, merge.Ours, ds1First.HeadRef()))\n\tsuite.NoError(err)\n\tsuite.True(types.Number(47).Equals(ours.HeadValue().(types.Map).Get(types.String(\"Friends\"))))\n}\n\nfunc newOptsWithMerge(vrw types.ValueReadWriter, policy merge.ResolveFunc, parents ...types.Value) CommitOptions {\n\treturn CommitOptions{Parents: types.NewSet(vrw, parents...), Policy: merge.NewThreeWay(policy)}\n}\n\nfunc (suite *DatabaseSuite) TestDatabaseDelete() {\n\tdatasetID1, datasetID2 := \"ds1\", \"ds2\"\n\tds1, ds2 := suite.db.GetDataset(datasetID1), suite.db.GetDataset(datasetID2)\n\tdatasets := suite.db.Datasets()\n\tsuite.Zero(datasets.Len())\n\n\t// ds1: |a|\n\tvar err error\n\ta := types.String(\"a\")\n\tds1, err = suite.db.CommitValue(ds1, a)\n\tsuite.NoError(err)\n\tsuite.True(ds1.HeadValue().Equals(a))\n\n\t// ds1: |a|, ds2: |b|\n\tb := types.String(\"b\")\n\tds2, err = suite.db.CommitValue(ds2, b)\n\tsuite.NoError(err)\n\tsuite.True(ds2.HeadValue().Equals(b))\n\n\tds1, err = suite.db.Delete(ds1)\n\tsuite.NoError(err)\n\tsuite.True(suite.db.GetDataset(datasetID2).HeadValue().Equals(b))\n\t_, present := suite.db.GetDataset(datasetID1).MaybeHead()\n\tsuite.False(present, \"Dataset %s should not be present\", datasetID1)\n\n\t// Get a fresh database, and verify that only ds2 is present\n\tnewDB := suite.makeDb(suite.storage.NewView())\n\tdefer newDB.Close()\n\tdatasets = newDB.Datasets()\n\tsuite.Equal(uint64(1), datasets.Len())\n\t_, present = newDB.GetDataset(datasetID2).MaybeHeadRef()\n\tsuite.True(present, \"Dataset %s should be present\", datasetID2)\n}\n\ntype waitDuringUpdateRootChunkStore struct {\n\tchunks.ChunkStore\n\tpreUpdateRootHook func()\n}\n\nfunc (w *waitDuringUpdateRootChunkStore) Commit(current, last hash.Hash) bool {\n\tif w.preUpdateRootHook != nil {\n\t\tw.preUpdateRootHook()\n\t}\n\treturn w.ChunkStore.Commit(current, last)\n}\n\nfunc (suite *DatabaseSuite) TestCommitWithConcurrentChunkStoreUse() {\n\tdatasetID := \"ds1\"\n\tds1 := suite.db.GetDataset(datasetID)\n\tvar err error\n\n\t// Setup:\n\t// ds1: |a| <- |b|\n\tds1, err = suite.db.CommitValue(ds1, types.String(\"a\"))\n\tb := types.String(\"b\")\n\tds1, err = suite.db.CommitValue(ds1, b)\n\tsuite.NoError(err)\n\tsuite.True(ds1.HeadValue().Equals(b))\n\n\t// Craft DB that will allow me to move the backing ChunkStore while suite.db isn't looking\n\tinterloper := suite.makeDb(suite.storage.NewView())\n\tdefer interloper.Close()\n\n\t// Change ds2 behind suite.db's back. This shouldn't block changes to ds1 via suite.db below.\n\t// ds1: |a| <- |b|\n\t// ds2: |stuff|\n\tstf := types.String(\"stuff\")\n\tds2, concErr := interloper.CommitValue(interloper.GetDataset(\"ds2\"), stf)\n\tsuite.NoError(concErr)\n\tsuite.True(ds2.HeadValue().Equals(stf))\n\n\t// Change ds1 via suite.db, which should proceed without a problem\n\tc := types.String(\"c\")\n\tds1, err = suite.db.CommitValue(ds1, c)\n\tsuite.NoError(err)\n\tsuite.True(ds1.HeadValue().Equals(c))\n\n\t// Change ds1 behind suite.db's back. Will block changes to ds1 below.\n\t// ds1: |a| <- |b| <- |c| <- |e|\n\te := types.String(\"e\")\n\tinterloper.Rebase()\n\tiDS, concErr := interloper.CommitValue(interloper.GetDataset(\"ds1\"), e)\n\tsuite.NoError(concErr)\n\tsuite.True(iDS.HeadValue().Equals(e))\n\n\t// Attempted Concurrent change, which should fail due to the above\n\tnope := types.String(\"nope\")\n\tds1, err = suite.db.CommitValue(ds1, nope)\n\tsuite.Error(err)\n\tv := ds1.HeadValue()\n\tsuite.True(v.Equals(e), \"%s\", v.(types.String))\n}\n\nfunc (suite *DatabaseSuite) TestDeleteWithConcurrentChunkStoreUse() {\n\tdatasetID := \"ds1\"\n\tds1 := suite.db.GetDataset(datasetID)\n\tvar err error\n\n\t// Setup:\n\t// ds1: |a| <- |b|\n\tds1, err = suite.db.CommitValue(ds1, types.String(\"a\"))\n\tb := types.String(\"b\")\n\tds1, err = suite.db.CommitValue(ds1, b)\n\tsuite.NoError(err)\n\tsuite.True(ds1.HeadValue().Equals(b))\n\n\t// Craft DB that will allow me to move the backing ChunkStore while suite.db isn't looking\n\tinterloper := suite.makeDb(suite.storage.NewView())\n\tdefer interloper.Close()\n\n\t// Concurrent change, to move root out from under my feet:\n\t// ds1: |a| <- |b| <- |e|\n\te := types.String(\"e\")\n\tiDS, concErr := interloper.CommitValue(interloper.GetDataset(datasetID), e)\n\tsuite.NoError(concErr)\n\tsuite.True(iDS.HeadValue().Equals(e))\n\n\t// Attempt to delete ds1 via suite.db, which should fail due to the above\n\tds1, err = suite.db.Delete(ds1)\n\tsuite.Error(err)\n\tsuite.True(ds1.HeadValue().Equals(e))\n\n\t// Concurrent change, but to some other dataset. This shouldn't stop changes to ds1.\n\t// ds1: |a| <- |b| <- |e|\n\t// ds2: |stuff|\n\tstf := types.String(\"stuff\")\n\tiDS, concErr = interloper.CommitValue(suite.db.GetDataset(\"other\"), stf)\n\tsuite.NoError(concErr)\n\tsuite.True(iDS.HeadValue().Equals(stf))\n\n\t// Attempted concurrent delete, which should proceed without a problem\n\tds1, err = suite.db.Delete(ds1)\n\tsuite.NoError(err)\n\t_, present := ds1.MaybeHeadRef()\n\tsuite.False(present, \"Dataset %s should not be present\", datasetID)\n}\n\nfunc (suite *DatabaseSuite) TestSetHead() {\n\tvar err error\n\tdatasetID := \"ds1\"\n\n\t// |a| <- |b|\n\tds := suite.db.GetDataset(datasetID)\n\ta := types.String(\"a\")\n\tds, err = suite.db.CommitValue(ds, a)\n\tsuite.NoError(err)\n\taCommitRef := ds.HeadRef() // To use in non-FF SetHead() below.\n\n\tb := types.String(\"b\")\n\tds, err = suite.db.CommitValue(ds, b)\n\tsuite.NoError(err)\n\tsuite.True(ds.HeadValue().Equals(b))\n\tbCommitRef := ds.HeadRef() // To use in FF SetHead() below.\n\n\tds, err = suite.db.SetHead(ds, aCommitRef)\n\tsuite.NoError(err)\n\tsuite.True(ds.HeadValue().Equals(a))\n\n\tds, err = suite.db.SetHead(ds, bCommitRef)\n\tsuite.NoError(err)\n\tsuite.True(ds.HeadValue().Equals(b))\n}\n\nfunc (suite *DatabaseSuite) TestFastForward() {\n\tvar err error\n\tdatasetID := \"ds1\"\n\n\t// |a| <- |b| <- |c|\n\tds := suite.db.GetDataset(datasetID)\n\ta := types.String(\"a\")\n\tds, err = suite.db.CommitValue(ds, a)\n\tsuite.NoError(err)\n\taCommitRef := ds.HeadRef() // To use in non-FF cases below.\n\n\tb := types.String(\"b\")\n\tds, err = suite.db.CommitValue(ds, b)\n\tsuite.NoError(err)\n\tsuite.True(ds.HeadValue().Equals(b))\n\n\tc := types.String(\"c\")\n\tds, err = suite.db.CommitValue(ds, c)\n\tsuite.NoError(err)\n\tsuite.True(ds.HeadValue().Equals(c))\n\tcCommitRef := ds.HeadRef() // To use in FastForward() below.\n\n\t// FastForward should disallow this, as |a| is not a descendant of |c|\n\tds, err = suite.db.FastForward(ds, aCommitRef)\n\tsuite.Error(err)\n\tsuite.True(ds.HeadValue().Equals(c))\n\n\t// Move Head back to something earlier in the lineage, so we can test FastForward\n\tds, err = suite.db.SetHead(ds, aCommitRef)\n\tsuite.NoError(err)\n\tsuite.True(ds.HeadValue().Equals(a))\n\n\t// This should succeed, because while |a| is not a direct parent of |c|, it is an ancestor.\n\tds, err = suite.db.FastForward(ds, cCommitRef)\n\tsuite.NoError(err)\n\tsuite.True(ds.HeadValue().Equals(c))\n}\n\nfunc (suite *DatabaseSuite) TestDatabaseHeightOfRefs() {\n\tr1 := suite.db.WriteValue(types.String(\"hello\"))\n\tsuite.Equal(uint64(1), r1.Height())\n\n\tr2 := suite.db.WriteValue(r1)\n\tsuite.Equal(uint64(2), r2.Height())\n\tsuite.Equal(uint64(3), suite.db.WriteValue(r2).Height())\n}\n\nfunc (suite *DatabaseSuite) TestDatabaseHeightOfCollections() {\n\tsetOfStringType := types.MakeSetType(types.StringType)\n\tsetOfRefOfStringType := types.MakeSetType(types.MakeRefType(types.StringType))\n\n\t// Set\n\tv1 := types.String(\"hello\")\n\tv2 := types.String(\"world\")\n\ts1 := types.NewSet(suite.db, v1, v2)\n\tsuite.Equal(uint64(1), suite.db.WriteValue(s1).Height())\n\n\t// Set>\n\ts2 := types.NewSet(suite.db, suite.db.WriteValue(v1), suite.db.WriteValue(v2))\n\tsuite.Equal(uint64(2), suite.db.WriteValue(s2).Height())\n\n\t// List>\n\tv3 := types.String(\"foo\")\n\tv4 := types.String(\"bar\")\n\ts3 := types.NewSet(suite.db, v3, v4)\n\tl1 := types.NewList(suite.db, s1, s3)\n\tsuite.Equal(uint64(1), suite.db.WriteValue(l1).Height())\n\n\t// List>\n\tl2 := types.NewList(suite.db, suite.db.WriteValue(s1), suite.db.WriteValue(s3))\n\tsuite.Equal(uint64(2), suite.db.WriteValue(l2).Height())\n\n\t// List>>\n\ts4 := types.NewSet(suite.db, suite.db.WriteValue(v3), suite.db.WriteValue(v4))\n\tl3 := types.NewList(suite.db, suite.db.WriteValue(s4))\n\tsuite.Equal(uint64(3), suite.db.WriteValue(l3).Height())\n\n\t// List | RefValue>>\n\tl4 := types.NewList(suite.db, s1, suite.db.WriteValue(s3))\n\tsuite.Equal(uint64(2), suite.db.WriteValue(l4).Height())\n\tl5 := types.NewList(suite.db, suite.db.WriteValue(s1), s3)\n\tsuite.Equal(uint64(2), suite.db.WriteValue(l5).Height())\n\n\t// Familiar with the \"New Jersey Turnpike\" drink? Here's the noms version of that...\n\teverything := []types.Value{v1, v2, s1, s2, v3, v4, s3, l1, l2, s4, l3, l4, l5}\n\tandMore := make([]types.Value, 0, len(everything)*3+2)\n\tfor _, v := range everything {\n\t\tandMore = append(andMore, v, types.TypeOf(v), suite.db.WriteValue(v))\n\t}\n\tandMore = append(andMore, setOfStringType, setOfRefOfStringType)\n\n\tsuite.db.WriteValue(types.NewList(suite.db, andMore...))\n}\n\nfunc (suite *DatabaseSuite) TestMetaOption() {\n\tds := suite.db.GetDataset(\"ds1\")\n\tm := types.NewStruct(\"M\", types.StructData{\n\t\t\"author\": types.String(\"arv\"),\n\t})\n\n\tds, err := suite.db.Commit(ds, types.String(\"a\"), CommitOptions{Meta: m})\n\tsuite.NoError(err)\n\tc := ds.Head()\n\tsuite.Equal(types.String(\"arv\"), c.Get(\"meta\").(types.Struct).Get(\"author\"))\n}\n" }, { "path": "go/datas/dataset.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"regexp\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// DatasetRe is a regexp that matches a legal Dataset name anywhere within the\n// target string.\nvar DatasetRe = regexp.MustCompile(`[a-zA-Z0-9\\-_/]+`)\n\n// DatasetFullRe is a regexp that matches a only a target string that is\n// entirely legal Dataset name.\nvar DatasetFullRe = regexp.MustCompile(\"^\" + DatasetRe.String() + \"$\")\n\n// Dataset is a named Commit within a Database.\ntype Dataset struct {\n\tdb Database\n\tid string\n\thead types.Value\n}\n\nfunc newDataset(db Database, id string, head types.Value) Dataset {\n\td.PanicIfFalse(head == nil || IsCommit(head))\n\treturn Dataset{db, id, head}\n}\n\n// Database returns the Database object in which this Dataset is stored.\n// WARNING: This method is under consideration for deprecation.\nfunc (ds Dataset) Database() Database {\n\treturn ds.db\n}\n\n// ID returns the name of this Dataset.\nfunc (ds Dataset) ID() string {\n\treturn ds.id\n}\n\n// MaybeHead returns the current Head Commit of this Dataset, which contains\n// the current root of the Dataset's value tree, if available. If not, it\n// returns a new Commit and 'false'.\nfunc (ds Dataset) MaybeHead() (types.Struct, bool) {\n\tif ds.head == nil {\n\t\treturn types.Struct{}, false\n\t}\n\treturn ds.head.(types.Struct), true\n}\n\n// Head returns the current head Commit, which contains the current root of\n// the Dataset's value tree.\nfunc (ds Dataset) Head() types.Struct {\n\tc, ok := ds.MaybeHead()\n\tif !ok {\n\t\td.Panic(\"Dataset \\\"%s\\\" does not exist\", ds.id)\n\t}\n\treturn c\n}\n\n// MaybeHeadRef returns the Ref of the current Head Commit of this Dataset,\n// which contains the current root of the Dataset's value tree, if available.\n// If not, it returns an empty Ref and 'false'.\nfunc (ds Dataset) MaybeHeadRef() (types.Ref, bool) {\n\tif ds.head == nil {\n\t\treturn types.Ref{}, false\n\t}\n\treturn types.NewRef(ds.head), true\n}\n\n// HasHead() returns 'true' if this dataset has a Head Commit, false otherwise.\nfunc (ds Dataset) HasHead() bool {\n\treturn ds.head != nil\n}\n\n// HeadRef returns the Ref of the current head Commit, which contains the\n// current root of the Dataset's value tree.\nfunc (ds Dataset) HeadRef() types.Ref {\n\tr, ok := ds.MaybeHeadRef()\n\tif !ok {\n\t\td.Panic(\"Dataset \\\"%s\\\" does not exist\", ds.id)\n\t}\n\treturn r\n}\n\n// MaybeHeadValue returns the Value field of the current head Commit, if\n// available. If not it returns nil and 'false'.\nfunc (ds Dataset) MaybeHeadValue() (types.Value, bool) {\n\tif c, ok := ds.MaybeHead(); ok {\n\t\treturn c.Get(ValueField), true\n\t}\n\treturn nil, false\n}\n\n// HeadValue returns the Value field of the current head Commit.\nfunc (ds Dataset) HeadValue() types.Value {\n\tc := ds.Head()\n\treturn c.Get(ValueField)\n}\n\nfunc IsValidDatasetName(name string) bool {\n\treturn DatasetFullRe.MatchString(name)\n}\n" }, { "path": "go/datas/dataset_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestExplicitBranchUsingDatasets(t *testing.T) {\n\tassert := assert.New(t)\n\tid1 := \"testdataset\"\n\tid2 := \"othertestdataset\"\n\tstg := &chunks.MemoryStorage{}\n\tstore := NewDatabase(stg.NewView())\n\tdefer store.Close()\n\n\tds1 := store.GetDataset(id1)\n\n\t// ds1: |a|\n\ta := types.String(\"a\")\n\tds1, err := store.CommitValue(ds1, a)\n\tassert.NoError(err)\n\tassert.True(ds1.Head().Get(ValueField).Equals(a))\n\n\t// ds1: |a|\n\t// \\ds2\n\tds2 := store.GetDataset(id2)\n\tds2, err = store.Commit(ds2, ds1.HeadValue(), CommitOptions{Parents: types.NewSet(store, ds1.HeadRef())})\n\tassert.NoError(err)\n\tassert.True(ds2.Head().Get(ValueField).Equals(a))\n\n\t// ds1: |a| <- |b|\n\tb := types.String(\"b\")\n\tds1, err = store.CommitValue(ds1, b)\n\tassert.NoError(err)\n\tassert.True(ds1.Head().Get(ValueField).Equals(b))\n\n\t// ds1: |a| <- |b|\n\t// \\ds2 <- |c|\n\tc := types.String(\"c\")\n\tds2, err = store.CommitValue(ds2, c)\n\tassert.NoError(err)\n\tassert.True(ds2.Head().Get(ValueField).Equals(c))\n\n\t// ds1: |a| <- |b| <--|d|\n\t// \\ds2 <- |c| <--/\n\tmergeParents := types.NewSet(store, types.NewRef(ds1.Head()), types.NewRef(ds2.Head()))\n\td := types.String(\"d\")\n\tds2, err = store.Commit(ds2, d, CommitOptions{Parents: mergeParents})\n\tassert.NoError(err)\n\tassert.True(ds2.Head().Get(ValueField).Equals(d))\n\n\tds1, err = store.Commit(ds1, d, CommitOptions{Parents: mergeParents})\n\tassert.NoError(err)\n\tassert.True(ds1.Head().Get(ValueField).Equals(d))\n}\n\nfunc TestTwoClientsWithEmptyDataset(t *testing.T) {\n\tassert := assert.New(t)\n\tid1 := \"testdataset\"\n\tstg := &chunks.MemoryStorage{}\n\tstore := NewDatabase(stg.NewView())\n\tdefer store.Close()\n\n\tdsx := store.GetDataset(id1)\n\tdsy := store.GetDataset(id1)\n\n\t// dsx: || -> |a|\n\ta := types.String(\"a\")\n\tdsx, err := store.CommitValue(dsx, a)\n\tassert.NoError(err)\n\tassert.True(dsx.Head().Get(ValueField).Equals(a))\n\n\t// dsy: || -> |b|\n\t_, ok := dsy.MaybeHead()\n\tassert.False(ok)\n\tb := types.String(\"b\")\n\tdsy, err = store.CommitValue(dsy, b)\n\tassert.Error(err)\n\t// Commit failed, but dsy now has latest head, so we should be able to just try again.\n\t// dsy: |a| -> |b|\n\tdsy, err = store.CommitValue(dsy, b)\n\tassert.NoError(err)\n\tassert.True(dsy.Head().Get(ValueField).Equals(b))\n}\n\nfunc TestTwoClientsWithNonEmptyDataset(t *testing.T) {\n\tassert := assert.New(t)\n\tid1 := \"testdataset\"\n\tstg := &chunks.MemoryStorage{}\n\tstore := NewDatabase(stg.NewView())\n\tdefer store.Close()\n\n\ta := types.String(\"a\")\n\t{\n\t\t// ds1: || -> |a|\n\t\tds1 := store.GetDataset(id1)\n\t\tds1, err := store.CommitValue(ds1, a)\n\t\tassert.NoError(err)\n\t\tassert.True(ds1.Head().Get(ValueField).Equals(a))\n\t}\n\n\tdsx := store.GetDataset(id1)\n\tdsy := store.GetDataset(id1)\n\n\t// dsx: |a| -> |b|\n\tassert.True(dsx.Head().Get(ValueField).Equals(a))\n\tb := types.String(\"b\")\n\tdsx, err := store.CommitValue(dsx, b)\n\tassert.NoError(err)\n\tassert.True(dsx.Head().Get(ValueField).Equals(b))\n\n\t// dsy: |a| -> |c|\n\tassert.True(dsy.Head().Get(ValueField).Equals(a))\n\tc := types.String(\"c\")\n\tdsy, err = store.CommitValue(dsy, c)\n\tassert.Error(err)\n\tassert.True(dsy.Head().Get(ValueField).Equals(b))\n\t// Commit failed, but dsy now has latest head, so we should be able to just try again.\n\t// dsy: |b| -> |c|\n\tdsy, err = store.CommitValue(dsy, c)\n\tassert.NoError(err)\n\tassert.True(dsy.Head().Get(ValueField).Equals(c))\n}\n\nfunc TestIdValidation(t *testing.T) {\n\tassert := assert.New(t)\n\tstg := &chunks.MemoryStorage{}\n\tstore := NewDatabase(stg.NewView())\n\n\tinvalidDatasetNames := []string{\" \", \"\", \"a \", \" a\", \"$\", \"#\", \":\", \"\\n\", \"💩\"}\n\tfor _, id := range invalidDatasetNames {\n\t\tassert.Panics(func() {\n\t\t\tstore.GetDataset(id)\n\t\t})\n\t}\n}\n\nfunc TestHeadValueFunctions(t *testing.T) {\n\tassert := assert.New(t)\n\n\tid1 := \"testdataset\"\n\tid2 := \"otherdataset\"\n\tstg := &chunks.MemoryStorage{}\n\tstore := NewDatabase(stg.NewView())\n\tdefer store.Close()\n\n\tds1 := store.GetDataset(id1)\n\tassert.False(ds1.HasHead())\n\n\t// ds1: |a|\n\ta := types.String(\"a\")\n\tds1, err := store.CommitValue(ds1, a)\n\tassert.NoError(err)\n\tassert.True(ds1.HasHead())\n\n\thv := ds1.Head().Get(ValueField)\n\tassert.Equal(a, hv)\n\tassert.Equal(a, ds1.HeadValue())\n\n\thv, ok := ds1.MaybeHeadValue()\n\tassert.True(ok)\n\tassert.Equal(a, hv)\n\n\tds2 := store.GetDataset(id2)\n\tassert.Panics(func() {\n\t\tds2.HeadValue()\n\t})\n\t_, ok = ds2.MaybeHeadValue()\n\tassert.False(ok)\n}\n\nfunc TestIsValidDatasetName(t *testing.T) {\n\tassert := assert.New(t)\n\tcases := []struct {\n\t\tname string\n\t\tvalid bool\n\t}{\n\t\t{\"foo\", true},\n\t\t{\"foo/bar\", true},\n\t\t{\"f1\", true},\n\t\t{\"1f\", true},\n\t\t{\"\", false},\n\t\t{\"f!!\", false},\n\t}\n\tfor _, c := range cases {\n\t\tassert.Equal(c.valid, IsValidDatasetName(c.name),\n\t\t\t\"Expected %s validity to be %t\", c.name, c.valid)\n\t}\n}\n" }, { "path": "go/datas/http_chunk_store.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"compress/gzip\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"strings\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/nbs\"\n\t\"github.com/attic-labs/noms/go/util/verbose\"\n\t\"github.com/golang/snappy\"\n\t\"github.com/julienschmidt/httprouter\"\n)\n\nconst (\n\thttpChunkStoreConcurrency = 6\n\treadThreshold = 1 << 12 // 4K\n)\n\nvar customHTTPTransport = http.Transport{\n\t// Since we limit ourselves to a maximum of httpChunkStoreConcurrency concurrent http requests, we think it's OK to up MaxIdleConnsPerHost so that one connection stays open for each concurrent request\n\tMaxIdleConnsPerHost: httpChunkStoreConcurrency,\n\t// This sets, essentially, an idle-timeout. The timer starts counting AFTER the client has finished sending the entire request to the server. As soon as the client receives the server's response headers, the timeout is canceled.\n\tResponseHeaderTimeout: time.Duration(4) * time.Minute,\n}\n\ntype httpChunkStore struct {\n\thost *url.URL\n\thttpClient httpDoer\n\tauth string\n\tgetQueue chan chunks.ReadRequest\n\thasQueue chan chunks.ReadRequest\n\tfinishedChan chan struct{}\n\trateLimit chan struct{}\n\tworkerWg *sync.WaitGroup\n\n\tcacheMu *sync.RWMutex\n\tunwrittenPuts *nbs.NomsBlockCache\n\n\trootMu *sync.RWMutex\n\troot hash.Hash\n\tversion string\n}\n\nfunc NewHTTPChunkStore(baseURL, auth string) chunks.ChunkStore {\n\t// Custom http.Client to give control of idle connections and timeouts\n\treturn newHTTPChunkStoreWithClient(baseURL, auth, &http.Client{Transport: &customHTTPTransport})\n}\n\nfunc newHTTPChunkStoreWithClient(baseURL, auth string, client httpDoer) *httpChunkStore {\n\tu, err := url.Parse(baseURL)\n\td.PanicIfError(err)\n\tif u.Scheme != \"http\" && u.Scheme != \"https\" {\n\t\td.Panic(\"Unrecognized scheme: %s\", u.Scheme)\n\t}\n\thcs := &httpChunkStore{\n\t\thost: u,\n\t\thttpClient: client,\n\t\tauth: auth,\n\t\tgetQueue: make(chan chunks.ReadRequest),\n\t\thasQueue: make(chan chunks.ReadRequest),\n\t\tfinishedChan: make(chan struct{}),\n\t\trateLimit: make(chan struct{}, httpChunkStoreConcurrency),\n\t\tworkerWg: &sync.WaitGroup{},\n\t\tcacheMu: &sync.RWMutex{},\n\t\tunwrittenPuts: nbs.NewCache(),\n\t\trootMu: &sync.RWMutex{},\n\t}\n\thcs.root, hcs.version = hcs.getRoot(false)\n\thcs.batchGetRequests()\n\thcs.batchHasRequests()\n\treturn hcs\n}\n\ntype httpDoer interface {\n\tDo(req *http.Request) (resp *http.Response, err error)\n}\n\nfunc (hcs *httpChunkStore) Version() string {\n\treturn hcs.version\n}\n\nfunc (hcs *httpChunkStore) Close() (e error) {\n\thcs.rootMu.Lock()\n\tdefer hcs.rootMu.Unlock()\n\n\tclose(hcs.finishedChan)\n\thcs.workerWg.Wait()\n\n\tclose(hcs.getQueue)\n\tclose(hcs.hasQueue)\n\tclose(hcs.rateLimit)\n\n\thcs.cacheMu.Lock()\n\tdefer hcs.cacheMu.Unlock()\n\thcs.unwrittenPuts.Destroy()\n\treturn\n}\n\nfunc (hcs *httpChunkStore) Stats() interface{} {\n\treturn nil\n}\n\nfunc checkStatus(status int, res *http.Response, body io.Reader) {\n\tif status == res.StatusCode {\n\t\treturn\n\t}\n\tbuf, _ := ioutil.ReadAll(body)\n\td.Panic(\"Unexpected response: %s: %s\", http.StatusText(res.StatusCode), strings.TrimSpace(string(buf)))\n}\n\nfunc (hcs *httpChunkStore) StatsSummary() string {\n\t// GET http:///stats. Response will be string containing a summary of database stats.\n\tu := *hcs.host\n\tu.Path = httprouter.CleanPath(hcs.host.Path + constants.StatsPath)\n\tres, err := hcs.httpClient.Do(newRequest(\"GET\", hcs.auth, u.String(), nil, nil))\n\td.PanicIfError(err)\n\tdefer closeResponse(res.Body)\n\n\tcheckStatus(http.StatusOK, res, res.Body)\n\tdata, err := ioutil.ReadAll(res.Body)\n\td.PanicIfError(err)\n\n\treturn string(data)\n}\n\nfunc (hcs *httpChunkStore) Get(h hash.Hash) chunks.Chunk {\n\tcheckCache := func(h hash.Hash) chunks.Chunk {\n\t\thcs.cacheMu.RLock()\n\t\tdefer hcs.cacheMu.RUnlock()\n\t\treturn hcs.unwrittenPuts.Get(h)\n\t}\n\tif pending := checkCache(h); !pending.IsEmpty() {\n\t\treturn pending\n\t}\n\n\tch := make(chan *chunks.Chunk)\n\tdefer close(ch)\n\n\tselect {\n\tcase <-hcs.finishedChan:\n\t\td.Panic(\"Tried to Get %s from closed ChunkStore\", h)\n\tcase hcs.getQueue <- chunks.NewGetRequest(h, ch):\n\t}\n\n\treturn *(<-ch)\n}\n\nfunc (hcs *httpChunkStore) GetMany(hashes hash.HashSet, foundChunks chan *chunks.Chunk) {\n\tcachedChunks := make(chan *chunks.Chunk)\n\tgo func() {\n\t\thcs.cacheMu.RLock()\n\t\tdefer hcs.cacheMu.RUnlock()\n\t\tdefer close(cachedChunks)\n\t\thcs.unwrittenPuts.GetMany(hashes, cachedChunks)\n\t}()\n\tremaining := hash.HashSet{}\n\tfor h := range hashes {\n\t\tremaining.Insert(h)\n\t}\n\tfor c := range cachedChunks {\n\t\tremaining.Remove(c.Hash())\n\t\tfoundChunks <- c\n\t}\n\n\tif len(remaining) == 0 {\n\t\treturn\n\t}\n\twg := &sync.WaitGroup{}\n\twg.Add(len(remaining))\n\tselect {\n\tcase <-hcs.finishedChan:\n\t\td.Panic(\"Tried to GetMany from closed ChunkStore\")\n\tcase hcs.getQueue <- chunks.NewGetManyRequest(remaining, wg, foundChunks):\n\t}\n\twg.Wait()\n}\n\nfunc (hcs *httpChunkStore) batchGetRequests() {\n\thcs.batchReadRequests(hcs.getQueue, hcs.getRefs)\n}\n\nfunc (hcs *httpChunkStore) Has(h hash.Hash) bool {\n\tcheckCache := func(h hash.Hash) bool {\n\t\thcs.cacheMu.RLock()\n\t\tdefer hcs.cacheMu.RUnlock()\n\t\treturn hcs.unwrittenPuts.Has(h)\n\t}\n\tif checkCache(h) {\n\t\treturn true\n\t}\n\n\tch := make(chan bool)\n\tdefer close(ch)\n\tselect {\n\tcase <-hcs.finishedChan:\n\t\td.Panic(\"Tried to Has %s on closed ChunkStore\", h)\n\tcase hcs.hasQueue <- chunks.NewAbsentRequest(h, ch):\n\t}\n\n\treturn <-ch\n}\n\nfunc (hcs *httpChunkStore) HasMany(hashes hash.HashSet) (absent hash.HashSet) {\n\tvar remaining hash.HashSet\n\tfunc() {\n\t\thcs.cacheMu.RLock()\n\t\tdefer hcs.cacheMu.RUnlock()\n\t\tremaining = hcs.unwrittenPuts.HasMany(hashes)\n\t}()\n\tif len(remaining) == 0 {\n\t\treturn remaining\n\t}\n\n\tnotFoundChunks := make(chan hash.Hash)\n\twg := &sync.WaitGroup{}\n\twg.Add(len(remaining))\n\tselect {\n\tcase <-hcs.finishedChan:\n\t\td.Panic(\"Tried to HasMany on closed ChunkStore\")\n\tcase hcs.hasQueue <- chunks.NewAbsentManyRequest(remaining, wg, notFoundChunks):\n\t}\n\tgo func() { defer close(notFoundChunks); wg.Wait() }()\n\n\tabsent = hash.HashSet{}\n\tfor notFound := range notFoundChunks {\n\t\tabsent.Insert(notFound)\n\t}\n\treturn absent\n}\n\nfunc (hcs *httpChunkStore) batchHasRequests() {\n\thcs.batchReadRequests(hcs.hasQueue, hcs.hasRefs)\n}\n\ntype batchGetter func(batch chunks.ReadBatch)\n\nfunc (hcs *httpChunkStore) batchReadRequests(queue <-chan chunks.ReadRequest, getter batchGetter) {\n\thcs.workerWg.Add(1)\n\tgo func() {\n\t\tdefer hcs.workerWg.Done()\n\t\tfor done := false; !done; {\n\t\t\tselect {\n\t\t\tcase req := <-queue:\n\t\t\t\thcs.sendReadRequests(req, queue, getter)\n\t\t\tcase <-hcs.finishedChan:\n\t\t\t\tdone = true\n\t\t\t}\n\t\t}\n\t}()\n}\n\nfunc (hcs *httpChunkStore) sendReadRequests(req chunks.ReadRequest, queue <-chan chunks.ReadRequest, getter batchGetter) {\n\tbatch := chunks.ReadBatch{}\n\n\taddReq := func(req chunks.ReadRequest) {\n\t\tfor h := range req.Hashes() {\n\t\t\tbatch[h] = append(batch[h], req.Outstanding())\n\t\t}\n\t}\n\n\taddReq(req)\n\tfor drained := false; !drained && len(batch) < readThreshold; {\n\t\tselect {\n\t\tcase req := <-queue:\n\t\t\taddReq(req)\n\t\tdefault:\n\t\t\tdrained = true\n\t\t}\n\t}\n\n\thcs.rateLimit <- struct{}{}\n\tgo func() {\n\t\tdefer batch.Close()\n\t\tdefer func() { <-hcs.rateLimit }()\n\n\t\tgetter(batch)\n\t}()\n}\n\nfunc (hcs *httpChunkStore) getRefs(batch chunks.ReadBatch) {\n\t// POST http:///getRefs/. Post body: ref=hash0&ref=hash1& Response will be chunk data if present, 404 if absent.\n\tu := *hcs.host\n\tu.Path = httprouter.CleanPath(hcs.host.Path + constants.GetRefsPath)\n\n\t// Indicate to the server that we're OK reading chunks from any store that knows about our root\n\tq := \"root=\" + hcs.root.String()\n\tif u.RawQuery != \"\" {\n\t\tq = u.RawQuery + \"&\" + q\n\t}\n\tu.RawQuery = q\n\n\treq := newRequest(\"POST\", hcs.auth, u.String(), buildHashesRequest(batch), http.Header{\n\t\t\"Accept-Encoding\": {\"x-snappy-framed\"},\n\t\t\"Content-Type\": {\"application/octet-stream\"},\n\t})\n\treq.ContentLength = int64(serializedLength(batch))\n\n\tres, err := hcs.httpClient.Do(req)\n\td.Chk.NoError(err)\n\texpectVersion(hcs.version, res)\n\treader := resBodyReader(res)\n\tdefer closeResponse(reader)\n\n\tcheckStatus(http.StatusOK, res, reader)\n\n\tchunkChan := make(chan *chunks.Chunk, 16)\n\tgo func() { defer close(chunkChan); chunks.Deserialize(reader, chunkChan) }()\n\n\tfor c := range chunkChan {\n\t\th := c.Hash()\n\t\tfor _, or := range batch[h] {\n\t\t\tgo or.Satisfy(h, c)\n\t\t}\n\t\tdelete(batch, c.Hash())\n\t}\n}\n\nfunc (hcs *httpChunkStore) hasRefs(batch chunks.ReadBatch) {\n\t// POST http:///hasRefs/. Post body: ref=sha1---&ref=sha1---& Response will be text of lines containing \"|ref| |bool|\".\n\tu := *hcs.host\n\tu.Path = httprouter.CleanPath(hcs.host.Path + constants.HasRefsPath)\n\n\treq := newRequest(\"POST\", hcs.auth, u.String(), buildHashesRequest(batch), http.Header{\n\t\t\"Accept-Encoding\": {\"x-snappy-framed\"},\n\t\t\"Content-Type\": {\"application/octet-stream\"},\n\t})\n\treq.ContentLength = int64(serializedLength(batch))\n\n\tres, err := hcs.httpClient.Do(req)\n\td.Chk.NoError(err)\n\texpectVersion(hcs.version, res)\n\treader := resBodyReader(res)\n\tdefer closeResponse(reader)\n\n\tcheckStatus(http.StatusOK, res, reader)\n\n\tscanner := bufio.NewScanner(reader)\n\tscanner.Split(bufio.ScanWords)\n\tfor scanner.Scan() {\n\t\th := hash.Parse(scanner.Text())\n\t\tfor _, outstanding := range batch[h] {\n\t\t\toutstanding.Satisfy(h, &chunks.EmptyChunk)\n\t\t}\n\t\tdelete(batch, h)\n\t}\n}\n\nfunc resBodyReader(res *http.Response) (reader io.ReadCloser) {\n\treader = res.Body\n\tif strings.Contains(res.Header.Get(\"Content-Encoding\"), \"gzip\") {\n\t\tgr, err := gzip.NewReader(reader)\n\t\td.Chk.NoError(err)\n\t\treader = gr\n\t} else if strings.Contains(res.Header.Get(\"Content-Encoding\"), \"x-snappy-framed\") {\n\t\tsr := snappy.NewReader(reader)\n\t\treader = ioutil.NopCloser(sr)\n\t}\n\treturn\n}\n\nfunc (hcs *httpChunkStore) Put(c chunks.Chunk) {\n\thcs.cacheMu.RLock()\n\tdefer hcs.cacheMu.RUnlock()\n\tselect {\n\tcase <-hcs.finishedChan:\n\t\td.Panic(\"Tried to Put %s into closed ChunkStore\", c.Hash())\n\tdefault:\n\t}\n\thcs.unwrittenPuts.Insert(c)\n}\n\nfunc sendWriteRequest(u url.URL, auth, vers string, p *nbs.NomsBlockCache, cli httpDoer) {\n\tchunkChan := make(chan *chunks.Chunk, 1024)\n\tgo func() {\n\t\tp.ExtractChunks(chunkChan)\n\t\tclose(chunkChan)\n\t}()\n\n\tbody := buildWriteValueRequest(chunkChan)\n\tn := int64(0)\n\n\t// Sad that we have to buffer this, but required for servers that need a content-length.\n\t// See: https://spectrum.chat/zeit/now/are-streaming-request-bodies-supported~8f085d13-2e35-4613-9cc0-818abcd04dfe\n\tnb := &bytes.Buffer{}\n\tvar err error\n\tn, err = io.Copy(nb, body)\n\td.PanicIfError(err)\n\tbody = ioutil.NopCloser(nb)\n\n\treq := newRequest(\"POST\", auth, u.String(), body, http.Header{\n\t\t\"Content-Encoding\": {\"x-snappy-framed\"},\n\t\t\"Content-Type\": {\"application/octet-stream\"},\n\t})\n\treq.ContentLength = n\n\n\tres, err := cli.Do(req)\n\td.PanicIfError(err)\n\texpectVersion(vers, res)\n\tdefer closeResponse(res.Body)\n\n\tcheckStatus(http.StatusCreated, res, res.Body)\n}\n\nfunc (hcs *httpChunkStore) Root() hash.Hash {\n\thcs.rootMu.RLock()\n\tdefer hcs.rootMu.RUnlock()\n\treturn hcs.root\n}\n\nfunc (hcs *httpChunkStore) Rebase() {\n\troot, _ := hcs.getRoot(true)\n\thcs.rootMu.Lock()\n\tdefer hcs.rootMu.Unlock()\n\thcs.root = root\n}\n\nfunc (hcs *httpChunkStore) getRoot(checkVers bool) (root hash.Hash, vers string) {\n\t// GET http:///root. Response will be ref of root.\n\tres := hcs.requestRoot(\"GET\", hash.Hash{}, hash.Hash{})\n\tif checkVers {\n\t\texpectVersion(hcs.version, res)\n\t}\n\tdefer closeResponse(res.Body)\n\n\tcheckStatus(http.StatusOK, res, res.Body)\n\tdata, err := ioutil.ReadAll(res.Body)\n\td.PanicIfError(err)\n\n\treturn hash.Parse(string(data)), res.Header.Get(NomsVersionHeader)\n}\n\nfunc (hcs *httpChunkStore) Commit(current, last hash.Hash) bool {\n\thcs.rootMu.Lock()\n\tdefer hcs.rootMu.Unlock()\n\thcs.cacheMu.Lock()\n\tdefer hcs.cacheMu.Unlock()\n\n\tselect {\n\tcase <-hcs.finishedChan:\n\t\td.Panic(\"Tried to Commit %s to closed ChunkStore\", current)\n\tcase hcs.rateLimit <- struct{}{}:\n\t\tdefer func() { <-hcs.rateLimit }()\n\t}\n\n\tif count := hcs.unwrittenPuts.Count(); count > 0 {\n\t\turl := *hcs.host\n\t\turl.Path = httprouter.CleanPath(hcs.host.Path + constants.WriteValuePath)\n\t\tverbose.Log(\"Sending %d chunks\", count)\n\t\tsendWriteRequest(url, hcs.auth, hcs.version, hcs.unwrittenPuts, hcs.httpClient)\n\t\tverbose.Log(\"Finished sending %d hashes\", count)\n\t\thcs.unwrittenPuts.Destroy()\n\t\thcs.unwrittenPuts = nbs.NewCache()\n\t}\n\n\t// POST http:///root?current=&last=. Response will be 200 on success, 409 if current is outdated. Regardless, the server returns its current root for this store\n\tres := hcs.requestRoot(\"POST\", current, last)\n\texpectVersion(hcs.version, res)\n\tdefer closeResponse(res.Body)\n\n\tvar success bool\n\tswitch res.StatusCode {\n\tcase http.StatusOK:\n\t\tsuccess = true\n\tcase http.StatusConflict:\n\t\tsuccess = false\n\tdefault:\n\t\tbuf := bytes.Buffer{}\n\t\tbuf.ReadFrom(res.Body)\n\t\tbody := buf.String()\n\t\td.Chk.Fail(\n\t\t\tfmt.Sprintf(\"Unexpected response: %s: %s\",\n\t\t\t\thttp.StatusText(res.StatusCode),\n\t\t\t\tbody))\n\t\treturn false\n\t}\n\tdata, err := ioutil.ReadAll(res.Body)\n\td.PanicIfError(err)\n\thcs.root = hash.Parse(string(data))\n\treturn success\n}\n\nfunc (hcs *httpChunkStore) requestRoot(method string, current, last hash.Hash) *http.Response {\n\tu := *hcs.host\n\tu.Path = httprouter.CleanPath(hcs.host.Path + constants.RootPath)\n\tif method == \"POST\" {\n\t\tparams := u.Query()\n\t\tparams.Add(\"last\", last.String())\n\t\tparams.Add(\"current\", current.String())\n\t\tu.RawQuery = params.Encode()\n\t}\n\n\treq := newRequest(method, hcs.auth, u.String(), nil, nil)\n\n\tres, err := hcs.httpClient.Do(req)\n\td.PanicIfError(err)\n\n\treturn res\n}\n\nfunc newRequest(method, auth, url string, body io.Reader, header http.Header) *http.Request {\n\treq, err := http.NewRequest(method, url, body)\n\td.Chk.NoError(err)\n\treq.Header.Set(NomsVersionHeader, constants.NomsVersion)\n\tfor k, vals := range header {\n\t\tfor _, v := range vals {\n\t\t\treq.Header.Add(k, v)\n\t\t}\n\t}\n\tif auth != \"\" {\n\t\treq.Header.Set(\"Authorization\", auth)\n\t}\n\treturn req\n}\n\nfunc expectVersion(expected string, res *http.Response) {\n\tdataVersion := res.Header.Get(NomsVersionHeader)\n\tif expected != dataVersion {\n\t\tb, _ := ioutil.ReadAll(res.Body)\n\t\tres.Body.Close()\n\t\td.Panic(\n\t\t\t\"Version skew\\n\\r\"+\n\t\t\t\t\"\\tServer data version changed from '%s' to '%s'\\n\\r\"+\n\t\t\t\t\"\\tHTTP Response: %d (%s): %s\\n\",\n\t\t\texpected, dataVersion,\n\t\t\tres.StatusCode, res.Status, string(b))\n\t}\n}\n\n// In order for keep alive to work we must read to EOF on every response. We may want to add a timeout so that a server that left its connection open can't cause all of ports to be eaten up.\nfunc closeResponse(rc io.ReadCloser) error {\n\tioutil.ReadAll(rc)\n\t// Bug #2069. It's not clear what the behavior is here. These checks are currently not enabled because they are shadowing information about a failure which occurs earlier.\n\t// d.Chk.NoError(err)\n\t// d.PanicIfFalse(0 == len(data), string(data))\n\treturn rc.Close()\n}\n" }, { "path": "go/datas/http_chunk_store_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"encoding/binary\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/julienschmidt/httprouter\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst testAuthToken = \"aToken123\"\n\nfunc TestHTTPChunkStore(t *testing.T) {\n\tsuite.Run(t, &HTTPChunkStoreSuite{})\n}\n\ntype HTTPChunkStoreSuite struct {\n\tsuite.Suite\n\tserverCS *chunks.TestStoreView\n\thttp *httpChunkStore\n}\n\ntype inlineServer struct {\n\t*httprouter.Router\n}\n\nfunc (serv inlineServer) Do(req *http.Request) (resp *http.Response, err error) {\n\tw := httptest.NewRecorder()\n\tserv.ServeHTTP(w, req)\n\treturn &http.Response{\n\t\t\tStatusCode: w.Code,\n\t\t\tStatus: http.StatusText(w.Code),\n\t\t\tHeader: w.HeaderMap,\n\t\t\tBody: ioutil.NopCloser(w.Body),\n\t\t},\n\t\tnil\n}\n\nfunc (suite *HTTPChunkStoreSuite) SetupTest() {\n\tstorage := &chunks.TestStorage{}\n\tsuite.serverCS = storage.NewView()\n\tsuite.http = newHTTPChunkStoreForTest(suite.serverCS)\n}\n\nfunc newHTTPChunkStoreForTest(cs chunks.ChunkStore) *httpChunkStore {\n\t// Ideally, this function (and its bretheren below) would take a *TestStorage and mint a fresh TestStoreView in each handler call below. That'd break a bunch of tests in pull_test.go that want to pass in a single TestStoreView and then inspect it after doing a bunch of work. The cs.Rebase() calls here are a good compromise for now, but BUG 3415 tracks Making This Right.\n\tserv := inlineServer{httprouter.New()}\n\tserv.POST(\n\t\tconstants.WriteValuePath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tHandleWriteValue(w, req, ps, cs)\n\t\t},\n\t)\n\tserv.POST(\n\t\tconstants.GetRefsPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tHandleGetRefs(w, req, ps, cs)\n\t\t},\n\t)\n\tserv.POST(\n\t\tconstants.HasRefsPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tHandleHasRefs(w, req, ps, cs)\n\t\t},\n\t)\n\tserv.POST(\n\t\tconstants.RootPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tHandleRootPost(w, req, ps, cs)\n\t\t},\n\t)\n\tserv.GET(\n\t\tconstants.RootPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tHandleRootGet(w, req, ps, cs)\n\t\t},\n\t)\n\tserv.GET(\n\t\tconstants.StatsPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tHandleStats(w, req, ps, cs)\n\t\t},\n\t)\n\treturn newHTTPChunkStoreWithClient(\"http://localhost:9000\", \"\", serv)\n}\n\nfunc newAuthenticatingHTTPChunkStoreForTest(assert *assert.Assertions, cs chunks.ChunkStore, hostUrl string) *httpChunkStore {\n\tauthenticate := func(req *http.Request) {\n\t\tassert.Equal(testAuthToken, req.URL.Query().Get(\"access_token\"))\n\t}\n\n\tserv := inlineServer{httprouter.New()}\n\tserv.POST(\n\t\tconstants.RootPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tauthenticate(req)\n\t\t\tHandleRootPost(w, req, ps, cs)\n\t\t},\n\t)\n\tserv.GET(\n\t\tconstants.RootPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tHandleRootGet(w, req, ps, cs)\n\t\t},\n\t)\n\treturn newHTTPChunkStoreWithClient(hostUrl, \"\", serv)\n}\n\nfunc newBadVersionHTTPChunkStoreForTest(cs chunks.ChunkStore) *httpChunkStore {\n\tserv := inlineServer{httprouter.New()}\n\tserv.POST(\n\t\tconstants.RootPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tHandleRootPost(w, req, ps, cs)\n\t\t\tw.Header().Set(NomsVersionHeader, \"BAD\")\n\t\t},\n\t)\n\tserv.GET(\n\t\tconstants.RootPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tcs.Rebase()\n\t\t\tHandleRootGet(w, req, ps, cs)\n\t\t},\n\t)\n\treturn newHTTPChunkStoreWithClient(\"http://localhost\", \"\", serv)\n}\n\nfunc (suite *HTTPChunkStoreSuite) TearDownTest() {\n\tsuite.http.Close()\n\tsuite.serverCS.Close()\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestPutChunk() {\n\tc := types.EncodeValue(types.String(\"abc\"))\n\tsuite.http.Put(c)\n\tsuite.True(suite.http.Has(c.Hash()))\n\n\tsuite.True(suite.http.Commit(hash.Hash{}, hash.Hash{}))\n\tsuite.Equal(1, suite.serverCS.Writes)\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestPutChunksInOrder() {\n\tvals := []types.Value{\n\t\ttypes.String(\"abc\"),\n\t\ttypes.String(\"def\"),\n\t}\n\tvs := types.NewValueStore(suite.serverCS)\n\tdefer vs.Close()\n\tle := types.NewList(vs).Edit()\n\tfor _, val := range vals {\n\t\tsuite.http.Put(types.EncodeValue(val))\n\t\tle.Append(types.NewRef(val))\n\t}\n\tsuite.http.Put(types.EncodeValue(le.List()))\n\tsuite.True(suite.http.Commit(hash.Hash{}, hash.Hash{}))\n\n\tsuite.Equal(3, suite.serverCS.Writes)\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestStats() {\n\tsuite.http.Put(types.EncodeValue(types.String(\"abc\")))\n\tsuite.http.Put(types.EncodeValue(types.String(\"def\")))\n\n\tsuite.True(suite.http.Commit(hash.Hash{}, hash.Hash{}))\n\n\tsuite.NotEmpty(suite.http.StatsSummary())\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestRebase() {\n\tsuite.Equal(hash.Hash{}, suite.http.Root())\n\tdb := NewDatabase(suite.serverCS)\n\tdefer db.Close()\n\tc := types.EncodeValue(types.NewMap(db))\n\tsuite.serverCS.Put(c)\n\tsuite.True(suite.serverCS.Commit(c.Hash(), hash.Hash{})) // change happens behind our backs\n\tsuite.Equal(hash.Hash{}, suite.http.Root()) // shouldn't be visible yet\n\n\tsuite.http.Rebase()\n\tsuite.Equal(c.Hash(), suite.serverCS.Root())\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestRoot() {\n\tdb := NewDatabase(suite.serverCS)\n\tdefer db.Close()\n\tc := types.EncodeValue(types.NewMap(db))\n\tsuite.serverCS.Put(c)\n\tsuite.True(suite.http.Commit(c.Hash(), hash.Hash{}))\n\tsuite.Equal(c.Hash(), suite.serverCS.Root())\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestVersionMismatch() {\n\tstore := newBadVersionHTTPChunkStoreForTest(suite.serverCS)\n\tvs := types.NewValueStore(store)\n\tdefer vs.Close()\n\tc := types.EncodeValue(types.NewMap(vs))\n\tsuite.serverCS.Put(c)\n\tsuite.Panics(func() { store.Commit(c.Hash(), hash.Hash{}) })\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestCommit() {\n\tdb := NewDatabase(suite.serverCS)\n\tdefer db.Close()\n\tc := types.EncodeValue(types.NewMap(db))\n\tsuite.serverCS.Put(c)\n\tsuite.True(suite.http.Commit(c.Hash(), hash.Hash{}))\n\tsuite.Equal(c.Hash(), suite.serverCS.Root())\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestEmptyHashCommit() {\n\tsuite.True(suite.http.Commit(hash.Hash{}, hash.Hash{}))\n\tsuite.Equal(hash.Hash{}, suite.serverCS.Root())\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestCommitWithParams() {\n\tu := fmt.Sprintf(\"http://localhost:9000?access_token=%s&other=19\", testAuthToken)\n\tstore := newAuthenticatingHTTPChunkStoreForTest(suite.Assert(), suite.serverCS, u)\n\tvs := types.NewValueStore(store)\n\tdefer vs.Close()\n\tc := types.EncodeValue(types.NewMap(vs))\n\tsuite.serverCS.Put(c)\n\tsuite.True(store.Commit(c.Hash(), hash.Hash{}))\n\tsuite.Equal(c.Hash(), suite.serverCS.Root())\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestGet() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tfor _, c := range chnx {\n\t\tsuite.serverCS.Put(c)\n\t}\n\tgot := suite.http.Get(chnx[0].Hash())\n\tsuite.Equal(chnx[0].Hash(), got.Hash())\n\tgot = suite.http.Get(chnx[1].Hash())\n\tsuite.Equal(chnx[1].Hash(), got.Hash())\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestGetMany() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tnotPresent := chunks.NewChunk([]byte(\"ghi\")).Hash()\n\tfor _, c := range chnx {\n\t\tsuite.serverCS.Put(c)\n\t}\n\tpersistChunks(suite.serverCS)\n\n\thashes := hash.NewHashSet(chnx[0].Hash(), chnx[1].Hash(), notPresent)\n\tfoundChunks := make(chan *chunks.Chunk)\n\tgo func() { suite.http.GetMany(hashes, foundChunks); close(foundChunks) }()\n\n\tfor c := range foundChunks {\n\t\thashes.Remove(c.Hash())\n\t}\n\tsuite.Len(hashes, 1)\n\tsuite.True(hashes.Has(notPresent))\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestOverGetThreshold_Issue3589() {\n\tif testing.Short() {\n\t\tsuite.T().Skip(\"Skipping test in short mode.\")\n\t}\n\t// BUG 3589 happened because we requested enough hashes that the body was over 10MB. The new way of encoding getRefs request bodies means that 10MB will no longer be a limitation. This test will generate a request larger than 10MB.\n\tcount := ((10 * (1 << 20)) / hash.ByteLen) + 1\n\thashes := make(hash.HashSet, count)\n\tfor i := 0; i < count-1; i++ {\n\t\th := hash.Hash{}\n\t\tbinary.BigEndian.PutUint64(h[hash.ByteLen-8:], uint64(i))\n\t\thashes.Insert(h)\n\t}\n\n\tpresent := chunks.NewChunk([]byte(\"ghi\"))\n\tsuite.serverCS.Put(present)\n\tpersistChunks(suite.serverCS)\n\thashes.Insert(present.Hash())\n\n\tfoundChunks := make(chan *chunks.Chunk)\n\tgo func() { suite.http.GetMany(hashes, foundChunks); close(foundChunks) }()\n\n\tfound := hash.HashSet{}\n\tfor c := range foundChunks {\n\t\tfound.Insert(c.Hash())\n\t}\n\tsuite.Len(found, 1)\n\tsuite.True(found.Has(present.Hash()))\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestGetManyAllCached() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tfor _, c := range chnx {\n\t\tsuite.http.Put(c)\n\t}\n\n\thashes := hash.NewHashSet(chnx[0].Hash(), chnx[1].Hash())\n\tfoundChunks := make(chan *chunks.Chunk)\n\tgo func() { suite.http.GetMany(hashes, foundChunks); close(foundChunks) }()\n\n\tfor c := range foundChunks {\n\t\thashes.Remove(c.Hash())\n\t}\n\tsuite.Len(hashes, 0)\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestGetManySomeCached() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tcached := chunks.NewChunk([]byte(\"ghi\"))\n\tfor _, c := range chnx {\n\t\tsuite.serverCS.Put(c)\n\t}\n\tpersistChunks(suite.serverCS)\n\tsuite.http.Put(cached)\n\n\thashes := hash.NewHashSet(chnx[0].Hash(), chnx[1].Hash(), cached.Hash())\n\tfoundChunks := make(chan *chunks.Chunk)\n\tgo func() { suite.http.GetMany(hashes, foundChunks); close(foundChunks) }()\n\n\tfor c := range foundChunks {\n\t\thashes.Remove(c.Hash())\n\t}\n\tsuite.Len(hashes, 0)\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestGetSame() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tfor _, c := range chnx {\n\t\tsuite.serverCS.Put(c)\n\t}\n\tgot := suite.http.Get(chnx[0].Hash())\n\tsuite.Equal(chnx[0].Hash(), got.Hash())\n\tgot = suite.http.Get(chnx[1].Hash())\n\tsuite.Equal(chnx[1].Hash(), got.Hash())\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestGetWithRoot() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tfor _, c := range chnx {\n\t\tsuite.serverCS.Put(c)\n\t}\n\tsuite.serverCS.Commit(chnx[0].Hash(), hash.Hash{})\n\n\tserv := inlineServer{httprouter.New()}\n\tserv.GET(\n\t\tconstants.RootPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tsuite.serverCS.Rebase()\n\t\t\tHandleRootGet(w, req, ps, suite.serverCS)\n\t\t},\n\t)\n\tserv.POST(\n\t\tconstants.GetRefsPath,\n\t\tfunc(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\t\tr := req.URL.Query().Get(\"root\")\n\t\t\tsuite.Equal(chnx[0].Hash().String(), r)\n\t\t\tsuite.serverCS.Rebase()\n\t\t\tHandleGetRefs(w, req, ps, suite.serverCS)\n\t\t},\n\t)\n\tstore := newHTTPChunkStoreWithClient(\"http://localhost:9000\", \"\", serv)\n\n\tgot := store.Get(chnx[1].Hash())\n\tsuite.Equal(chnx[1].Hash(), got.Hash())\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestHas() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tfor _, c := range chnx {\n\t\tsuite.serverCS.Put(c)\n\t}\n\tsuite.True(suite.http.Has(chnx[0].Hash()))\n\tsuite.True(suite.http.Has(chnx[1].Hash()))\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestHasMany() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tfor _, c := range chnx {\n\t\tsuite.serverCS.Put(c)\n\t}\n\tpersistChunks(suite.serverCS)\n\tnotPresent := chunks.NewChunk([]byte(\"ghi\")).Hash()\n\n\thashes := hash.NewHashSet(chnx[0].Hash(), chnx[1].Hash(), notPresent)\n\tabsent := suite.http.HasMany(hashes)\n\n\tsuite.Len(absent, 1)\n\tfor _, c := range chnx {\n\t\tsuite.False(absent.Has(c.Hash()), \"%s present in %v\", c.Hash(), absent)\n\t}\n\tsuite.True(absent.Has(notPresent))\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestHasManyAllCached() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tfor _, c := range chnx {\n\t\tsuite.http.Put(c)\n\t}\n\tpersistChunks(suite.serverCS)\n\n\thashes := hash.NewHashSet(chnx[0].Hash(), chnx[1].Hash())\n\tabsent := suite.http.HasMany(hashes)\n\n\tsuite.Len(absent, 0)\n\tfor _, c := range chnx {\n\t\tsuite.False(absent.Has(c.Hash()), \"%s present in %v\", c.Hash(), absent)\n\t}\n}\n\nfunc (suite *HTTPChunkStoreSuite) TestHasManySomeCached() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tcached := chunks.NewChunk([]byte(\"ghi\"))\n\tfor _, c := range chnx {\n\t\tsuite.serverCS.Put(c)\n\t}\n\tpersistChunks(suite.serverCS)\n\tsuite.http.Put(cached)\n\n\thashes := hash.NewHashSet(chnx[0].Hash(), chnx[1].Hash(), cached.Hash())\n\tabsent := suite.http.HasMany(hashes)\n\n\tsuite.Len(absent, 0)\n\tfor _, c := range chnx {\n\t\tsuite.False(absent.Has(c.Hash()), \"%s present in %v\", c.Hash(), absent)\n\t}\n\tsuite.False(absent.Has(cached.Hash()), \"%s present in %v\", cached.Hash(), absent)\n}\n" }, { "path": "go/datas/pull.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"math\"\n\t\"math/rand\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/golang/snappy\"\n)\n\ntype PullProgress struct {\n\tDoneCount, KnownCount, ApproxWrittenBytes uint64\n}\n\nconst (\n\tbytesWrittenSampleRate = .10\n\tbatchSize = 1 << 12 // 4096 chunks\n)\n\n// Pull objects that descend from sourceRef from srcDB to sinkDB.\nfunc Pull(srcDB, sinkDB Database, sourceRef types.Ref, progressCh chan PullProgress) {\n\t// Sanity Check\n\td.PanicIfFalse(srcDB.chunkStore().Has(sourceRef.TargetHash()))\n\n\tif sinkDB.chunkStore().Has(sourceRef.TargetHash()) {\n\t\treturn // already up to date\n\t}\n\n\tvar doneCount, knownCount, approxBytesWritten uint64\n\tupdateProgress := func(moreDone, moreKnown, moreApproxBytesWritten uint64) {\n\t\tif progressCh == nil {\n\t\t\treturn\n\t\t}\n\t\tdoneCount, knownCount, approxBytesWritten = doneCount+moreDone, knownCount+moreKnown, approxBytesWritten+moreApproxBytesWritten\n\t\tprogressCh <- PullProgress{doneCount, knownCount, approxBytesWritten}\n\t}\n\tvar sampleSize, sampleCount uint64\n\n\t// TODO: This batches based on limiting the _number_ of chunks processed at the same time. We really want to batch based on the _amount_ of chunk data being processed simultaneously. We also want to consider the chunks in a particular order, however, and the current GetMany() interface doesn't provide any ordering guarantees. Once BUG 3750 is fixed, we should be able to revisit this and do a better job.\n\tabsent := hash.HashSlice{sourceRef.TargetHash()}\n\tfor absentCount := len(absent); absentCount != 0; absentCount = len(absent) {\n\t\tupdateProgress(0, uint64(absentCount), 0)\n\n\t\t// For gathering up the hashes in the next level of the tree\n\t\tnextLevel := hash.HashSet{}\n\t\tuniqueOrdered := hash.HashSlice{}\n\n\t\t// Process all absent chunks in this level of the tree in quanta of at most |batchSize|\n\t\tfor start, end := 0, batchSize; start < absentCount; start, end = end, end+batchSize {\n\t\t\tif end > absentCount {\n\t\t\t\tend = absentCount\n\t\t\t}\n\t\t\tbatch := absent[start:end]\n\n\t\t\t// Concurrently pull all chunks from this batch that the sink is missing out of the source\n\t\t\tneededChunks := map[hash.Hash]*chunks.Chunk{}\n\t\t\tfound := make(chan *chunks.Chunk)\n\t\t\tgo func() { defer close(found); srcDB.chunkStore().GetMany(batch.HashSet(), found) }()\n\t\t\tfor c := range found {\n\t\t\t\tneededChunks[c.Hash()] = c\n\n\t\t\t\t// Randomly sample amount of data written\n\t\t\t\tif rand.Float64() < bytesWrittenSampleRate {\n\t\t\t\t\tsampleSize += uint64(len(snappy.Encode(nil, c.Data())))\n\t\t\t\t\tsampleCount++\n\t\t\t\t}\n\t\t\t\tupdateProgress(1, 0, sampleSize/uint64(math.Max(1, float64(sampleCount))))\n\t\t\t}\n\n\t\t\t// Now, put the absent chunks into the sink IN ORDER.\n\t\t\t// At the same time, gather up an ordered, uniquified list of all the children of the chunks in |batch| and add them to those in previous batches. This list is what we'll use to descend to the next level of the tree.\n\t\t\tfor _, h := range batch {\n\t\t\t\tc := neededChunks[h]\n\t\t\t\tsinkDB.chunkStore().Put(*c)\n\t\t\t\ttypes.WalkRefs(*c, func(r types.Ref) {\n\t\t\t\t\tif !nextLevel.Has(r.TargetHash()) {\n\t\t\t\t\t\tuniqueOrdered = append(uniqueOrdered, r.TargetHash())\n\t\t\t\t\t\tnextLevel.Insert(r.TargetHash())\n\t\t\t\t\t}\n\t\t\t\t})\n\t\t\t}\n\t\t}\n\n\t\t// Ask sinkDB which of the next level's hashes it doesn't have.\n\t\tabsentSet := sinkDB.chunkStore().HasMany(nextLevel)\n\t\tabsent = absent[:0]\n\t\tfor _, h := range uniqueOrdered {\n\t\t\tif absentSet.Has(h) {\n\t\t\t\tabsent = append(absent, h)\n\t\t\t}\n\t\t}\n\t}\n\n\tpersistChunks(sinkDB.chunkStore())\n}\n" }, { "path": "go/datas/pull_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst datasetID = \"ds1\"\n\nfunc TestLocalToLocalPulls(t *testing.T) {\n\tsuite.Run(t, &LocalToLocalSuite{})\n}\n\nfunc TestRemoteToLocalPulls(t *testing.T) {\n\tsuite.Run(t, &RemoteToLocalSuite{})\n}\n\nfunc TestLocalToRemotePulls(t *testing.T) {\n\tsuite.Run(t, &LocalToRemoteSuite{})\n}\n\nfunc TestRemoteToRemotePulls(t *testing.T) {\n\tsuite.Run(t, &RemoteToRemoteSuite{})\n}\n\ntype PullSuite struct {\n\tsuite.Suite\n\tsinkCS *chunks.TestStoreView\n\tsourceCS *chunks.TestStoreView\n\tsink Database\n\tsource Database\n\tcommitReads int // The number of reads triggered by commit differs across chunk store impls\n}\n\nfunc makeTestStoreViews() (ts1, ts2 *chunks.TestStoreView) {\n\tst1, st2 := &chunks.TestStorage{}, &chunks.TestStorage{}\n\treturn st1.NewView(), st2.NewView()\n}\n\ntype LocalToLocalSuite struct {\n\tPullSuite\n}\n\nfunc (suite *LocalToLocalSuite) SetupTest() {\n\tsuite.sinkCS, suite.sourceCS = makeTestStoreViews()\n\tsuite.sink = NewDatabase(suite.sinkCS)\n\tsuite.source = NewDatabase(suite.sourceCS)\n}\n\ntype RemoteToLocalSuite struct {\n\tPullSuite\n}\n\nfunc (suite *RemoteToLocalSuite) SetupTest() {\n\tsuite.sinkCS, suite.sourceCS = makeTestStoreViews()\n\tsuite.sink = NewDatabase(suite.sinkCS)\n\tsuite.source = makeRemoteDb(suite.sourceCS)\n}\n\ntype LocalToRemoteSuite struct {\n\tPullSuite\n}\n\nfunc (suite *LocalToRemoteSuite) SetupTest() {\n\tsuite.sinkCS, suite.sourceCS = makeTestStoreViews()\n\tsuite.sink = makeRemoteDb(suite.sinkCS)\n\tsuite.source = NewDatabase(suite.sourceCS)\n\tsuite.commitReads = 1\n}\n\ntype RemoteToRemoteSuite struct {\n\tPullSuite\n}\n\nfunc (suite *RemoteToRemoteSuite) SetupTest() {\n\tsuite.sinkCS, suite.sourceCS = makeTestStoreViews()\n\tsuite.sink = makeRemoteDb(suite.sinkCS)\n\tsuite.source = makeRemoteDb(suite.sourceCS)\n\tsuite.commitReads = 1\n}\n\nfunc makeRemoteDb(cs chunks.ChunkStore) Database {\n\treturn NewDatabase(newHTTPChunkStoreForTest(cs))\n}\n\nfunc (suite *PullSuite) TearDownTest() {\n\tsuite.sink.Close()\n\tsuite.source.Close()\n\tsuite.sinkCS.Close()\n\tsuite.sourceCS.Close()\n}\n\ntype progressTracker struct {\n\tCh chan PullProgress\n\tdoneCh chan []PullProgress\n}\n\nfunc startProgressTracker() *progressTracker {\n\tpt := &progressTracker{make(chan PullProgress), make(chan []PullProgress)}\n\tgo func() {\n\t\tprogress := []PullProgress{}\n\t\tfor info := range pt.Ch {\n\t\t\tprogress = append(progress, info)\n\t\t}\n\t\tpt.doneCh <- progress\n\t}()\n\treturn pt\n}\n\nfunc (pt *progressTracker) Validate(suite *PullSuite) {\n\tclose(pt.Ch)\n\tprogress := <-pt.doneCh\n\n\t// Expecting exact progress would be unreliable and not necessary meaningful. Instead, just validate that it's useful and consistent.\n\tsuite.NotEmpty(progress)\n\n\tfirst := progress[0]\n\tsuite.Zero(first.DoneCount)\n\tsuite.True(first.KnownCount > 0)\n\tsuite.Zero(first.ApproxWrittenBytes)\n\n\tlast := progress[len(progress)-1]\n\tsuite.True(last.DoneCount > 0)\n\tsuite.Equal(last.DoneCount, last.KnownCount)\n\n\tfor i, prog := range progress {\n\t\tsuite.True(prog.KnownCount >= prog.DoneCount)\n\t\tif i > 0 {\n\t\t\tprev := progress[i-1]\n\t\t\tsuite.True(prog.DoneCount >= prev.DoneCount)\n\t\t\tsuite.True(prog.ApproxWrittenBytes >= prev.ApproxWrittenBytes)\n\t\t}\n\t}\n}\n\n// Source: -3-> C(L2) -1-> N\n// \\ -2-> L1 -1-> N\n// \\ -1-> L0\n//\n// Sink: Nada\nfunc (suite *PullSuite) TestPullEverything() {\n\texpectedReads := suite.sinkCS.Reads\n\n\tl := buildListOfHeight(2, suite.source)\n\tsourceRef := suite.commitToSource(l, types.NewSet(suite.source))\n\tpt := startProgressTracker()\n\n\tPull(suite.source, suite.sink, sourceRef, pt.Ch)\n\tsuite.True(expectedReads-suite.sinkCS.Reads <= suite.commitReads)\n\tpt.Validate(suite)\n\n\tv := suite.sink.ReadValue(sourceRef.TargetHash()).(types.Struct)\n\tsuite.NotNil(v)\n\tsuite.True(l.Equals(v.Get(ValueField)))\n}\n\n// Source: -6-> C3(L5) -1-> N\n// . \\ -5-> L4 -1-> N\n// . \\ -4-> L3 -1-> N\n// . \\ -3-> L2 -1-> N\n// 5 \\ -2-> L1 -1-> N\n// . \\ -1-> L0\n// C2(L4) -1-> N\n// . \\ -4-> L3 -1-> N\n// . \\ -3-> L2 -1-> N\n// . \\ -2-> L1 -1-> N\n// 3 \\ -1-> L0\n// .\n// C1(L2) -1-> N\n// \\ -2-> L1 -1-> N\n// \\ -1-> L0\n//\n// Sink: -3-> C1(L2) -1-> N\n// \\ -2-> L1 -1-> N\n// \\ -1-> L0\nfunc (suite *PullSuite) TestPullMultiGeneration() {\n\tsinkL := buildListOfHeight(2, suite.sink)\n\tsuite.commitToSink(sinkL, types.NewSet(suite.sink))\n\texpectedReads := suite.sinkCS.Reads\n\n\tsrcL := buildListOfHeight(2, suite.source)\n\tsourceRef := suite.commitToSource(srcL, types.NewSet(suite.source))\n\tsrcL = buildListOfHeight(4, suite.source)\n\tsourceRef = suite.commitToSource(srcL, types.NewSet(suite.source, sourceRef))\n\tsrcL = buildListOfHeight(5, suite.source)\n\tsourceRef = suite.commitToSource(srcL, types.NewSet(suite.source, sourceRef))\n\n\tpt := startProgressTracker()\n\n\tPull(suite.source, suite.sink, sourceRef, pt.Ch)\n\n\tsuite.True(expectedReads-suite.sinkCS.Reads <= suite.commitReads)\n\tpt.Validate(suite)\n\n\tv := suite.sink.ReadValue(sourceRef.TargetHash()).(types.Struct)\n\tsuite.NotNil(v)\n\tsuite.True(srcL.Equals(v.Get(ValueField)))\n}\n\n// Source: -6-> C2(L5) -1-> N\n// . \\ -5-> L4 -1-> N\n// . \\ -4-> L3 -1-> N\n// . \\ -3-> L2 -1-> N\n// 4 \\ -2-> L1 -1-> N\n// . \\ -1-> L0\n// C1(L3) -1-> N\n// \\ -3-> L2 -1-> N\n// \\ -2-> L1 -1-> N\n// \\ -1-> L0\n//\n// Sink: -5-> C3(L3') -1-> N\n// . \\ -3-> L2 -1-> N\n// . \\ \\ -2-> L1 -1-> N\n// . \\ \\ -1-> L0\n// . \\ - \"oy!\"\n// 4\n// .\n// C1(L3) -1-> N\n// \\ -3-> L2 -1-> N\n// \\ -2-> L1 -1-> N\n// \\ -1-> L0\nfunc (suite *PullSuite) TestPullDivergentHistory() {\n\tsinkL := buildListOfHeight(3, suite.sink)\n\tsinkRef := suite.commitToSink(sinkL, types.NewSet(suite.sink))\n\tsrcL := buildListOfHeight(3, suite.source)\n\tsourceRef := suite.commitToSource(srcL, types.NewSet(suite.source))\n\n\tsinkL = sinkL.Edit().Append(types.String(\"oy!\")).List()\n\tsinkRef = suite.commitToSink(sinkL, types.NewSet(suite.sink, sinkRef))\n\tsrcL = srcL.Edit().Set(1, buildListOfHeight(5, suite.source)).List()\n\tsourceRef = suite.commitToSource(srcL, types.NewSet(suite.source, sourceRef))\n\tpreReads := suite.sinkCS.Reads\n\n\tpt := startProgressTracker()\n\n\tPull(suite.source, suite.sink, sourceRef, pt.Ch)\n\n\tsuite.True(preReads-suite.sinkCS.Reads <= suite.commitReads)\n\tpt.Validate(suite)\n\n\tv := suite.sink.ReadValue(sourceRef.TargetHash()).(types.Struct)\n\tsuite.NotNil(v)\n\tsuite.True(srcL.Equals(v.Get(ValueField)))\n}\n\n// Source: -6-> C2(L4) -1-> N\n// . \\ -4-> L3 -1-> N\n// . \\ -3-> L2 -1-> N\n// . \\ - \"oy!\"\n// 5 \\ -2-> L1 -1-> N\n// . \\ -1-> L0\n// C1(L4) -1-> N\n// \\ -4-> L3 -1-> N\n// \\ -3-> L2 -1-> N\n// \\ -2-> L1 -1-> N\n// \\ -1-> L0\n// Sink: -5-> C1(L4) -1-> N\n// \\ -4-> L3 -1-> N\n// \\ -3-> L2 -1-> N\n// \\ -2-> L1 -1-> N\n// \\ -1-> L0\nfunc (suite *PullSuite) TestPullUpdates() {\n\tsinkL := buildListOfHeight(4, suite.sink)\n\tsuite.commitToSink(sinkL, types.NewSet(suite.sink))\n\texpectedReads := suite.sinkCS.Reads\n\n\tsrcL := buildListOfHeight(4, suite.source)\n\tsourceRef := suite.commitToSource(srcL, types.NewSet(suite.source))\n\tL3 := srcL.Get(1).(types.Ref).TargetValue(suite.source).(types.List)\n\tL2 := L3.Get(1).(types.Ref).TargetValue(suite.source).(types.List)\n\tL2 = L2.Edit().Append(suite.source.WriteValue(types.String(\"oy!\"))).List()\n\tL3 = L3.Edit().Set(1, suite.source.WriteValue(L2)).List()\n\tsrcL = srcL.Edit().Set(1, suite.source.WriteValue(L3)).List()\n\tsourceRef = suite.commitToSource(srcL, types.NewSet(suite.source, sourceRef))\n\n\tpt := startProgressTracker()\n\n\tPull(suite.source, suite.sink, sourceRef, pt.Ch)\n\n\tsuite.True(expectedReads-suite.sinkCS.Reads <= suite.commitReads)\n\tpt.Validate(suite)\n\n\tv := suite.sink.ReadValue(sourceRef.TargetHash()).(types.Struct)\n\tsuite.NotNil(v)\n\tsuite.True(srcL.Equals(v.Get(ValueField)))\n}\n\nfunc (suite *PullSuite) commitToSource(v types.Value, p types.Set) types.Ref {\n\tds := suite.source.GetDataset(datasetID)\n\tds, err := suite.source.Commit(ds, v, CommitOptions{Parents: p})\n\tsuite.NoError(err)\n\treturn ds.HeadRef()\n}\n\nfunc (suite *PullSuite) commitToSink(v types.Value, p types.Set) types.Ref {\n\tds := suite.sink.GetDataset(datasetID)\n\tds, err := suite.sink.Commit(ds, v, CommitOptions{Parents: p})\n\tsuite.NoError(err)\n\treturn ds.HeadRef()\n}\n\nfunc buildListOfHeight(height int, vrw types.ValueReadWriter) types.List {\n\tunique := 0\n\tl := types.NewList(vrw, types.Number(unique), types.Number(unique+1))\n\tunique += 2\n\n\tfor i := 0; i < height; i++ {\n\t\tr1, r2 := vrw.WriteValue(types.Number(unique)), vrw.WriteValue(l)\n\t\tunique++\n\t\tl = types.NewList(vrw, r1, r2)\n\t}\n\treturn l\n}\n" }, { "path": "go/datas/pulling.md", "content": "# Dataset pulling algorithm\nThe approach is to explore the chunk graph of both sink and source in order of decreasing ref-height. As the code walks, it uses the knowledge gained about which chunks are present in the sink to both prune the source-graph-walk and build up a set of `hints` that can be sent to a remote Database to aid in chunk validation.\n\n## Basic algorithm\n\n- let `sink` be the *sink* database\n- let `source` be the *source* database\n- let `snkQ` and `srcQ` be priority queues of `Ref` prioritized by highest `Ref.height`\n- let `hints` be a map of `hash => hash`\n- let `reachableChunks` be a set of hashes\n- let `snkHdRef` be the ref (of `Commit`) of the head of the *sink* dataset\n- let `srcHdRef` be the ref of the *source* `Commit`, which must descend from the `Commit` indicated by `snkHdRef`\n\n- let `traverseSource(srcRef, srcQ, sink, source, reachableChunks)` be\n - pop `srcRef` from `srcQ`\n - if `!sink.has(srcRef)`\n - let `c` = `source.batchStore().Get(srcRef.targetHash)`\n - let `v` = `types.DecodeValue(c, source)`\n - insert all child refs, `cr`, from `v` into `srcQ` and into reachableRefs\n - `sink.batchStore().Put(c, srcRef.height, no hints)`\n - (hints will all be gathered and handed to sink.batchStore at the end)\n\n\n- let `traverseSink(sinkRef, snkQ, sink, hints)` be\n - pop `snkRef` from `snkQ`\n - if `snkRef.height` > 1\n - let `v` = `sink.readValue(snkRef.targetHash)`\n - insert all child refs, `cr`, from `v` into `snkQ` and `hints[cr] = snkRef`\n\n\n- let `traverseCommon(comRef, snkHdRef, snkQ, srcQ, sink, hints)` be\n - pop `comRef` from both `snkQ` and `srcQ`\n - if `comRef.height` > 1\n - if `comRef` is a `Ref` of `Commit`\n - let `v` = `sink.readValue(comRef.targetHash)`\n - if `comRef` == snkHdRef\n - *ignore all parent refs*\n - insert each other child ref `cr` from `v` into `snkQ` *only*, set `hints[cr] = comRef`\n - else\n - insert each child ref `cr` from `v` into both `snkQ` and `srcQ`, set `hints[cr] = comRef`\n\n\n- let `pull(source, sink, srcHdRef, sinkHdRef)\n - insert `snkHdRef` into `snkQ` and `srcHdRef` into `srcQ`\n - create empty `hints` and `reachableChunks`\n - while `srcQ` is non-empty\n - let `srcHt` and `snkHt` be the respective heights of the *top* `Ref` in each of `srcQ` and `snkQ`\n - if `srcHt` > `snkHt`, for every `srcHdRef` in `srcQ` which is of greater height than `snkHt`\n - `traverseSource(srcHdRef, srcQ, sink, source)`\n - else if `snkHt` > `srcHt`, for every `snkHdRef` in `snkQ` which is of greater height than `srcHt`\n - `traverseSink(snkHdRef, snkQ, sink)`\n - else\n - for every `comRef` in which is common to `snkQ` and `srcQ` which is of height `srcHt` (and `snkHt`)\n - `traverseCommon(comRef, snkHdRef, snkQ, srcQ, sink, hints)`\n - for every `ref` in `srcQ` which is of height `srcHt`\n - `traverseSource(ref, srcQ, sink, source, reachableChunks)`\n - for every `ref` in `snkQ` which is of height `snkHt`\n - `traverseSink(ref, snkQ, sink, hints)`\n - for all `hash` in `reachableChunks`\n - sink.batchStore().addHint(hints[hash])\n\n\n## Isomorphic, but less clear, algorithm\n\n- let all identifiers be as above\n- let `traverseSource`, `traverseSink`, and `traverseCommon` be as above\n\n- let `higherThan(refA, refB)` be\n - if refA.height == refB.height\n - return refA.targetHash < refB.targetHash\n - return refA.height > refB.height\n\n- let `pull(source, sink, srcHdRef, sinkHdRef)\n - insert `snkHdRef` into `snkQ` and `srcHdRef` into `srcQ`\n - create empty `hints` and `reachableChunks`\n - while `srcQ` is non-empty\n - if `sinkQ` is empty\n - pop `ref` from `srcQ`\n - `traverseSource(ref, srcQ, sink, source, reachableChunks))\n - else if `higherThan(head of srcQ, head of snkQ)`\n - pop `ref` from `srcQ`\n - `traverseSource(ref, srcQ, sink, source, reachableChunks))\n - else if `higherThan(head of snkQ, head of srcQ)`\n - pop `ref` from `snkQ`\n - `traverseSink(ref, snkQ, sink, hints)`\n - else, heads of both queues are the same\n - pop `comRef` from `snkQ` and `srcQ`\n - `traverseCommon(comRef, snkHdRef, snkQ, srcQ, sink, hints)`\n - for all `hash` in `reachableChunks`\n - sink.batchStore().addHint(hints[hash])\n\n\n" }, { "path": "go/datas/remote_database_handlers.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"compress/gzip\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"log\"\n\t\"net/http\"\n\t\"os\"\n\t\"runtime\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/ngql\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/verbose\"\n\t\"github.com/golang/snappy\"\n)\n\ntype URLParams interface {\n\tByName(string) string\n}\n\ntype Handler func(w http.ResponseWriter, req *http.Request, ps URLParams, cs chunks.ChunkStore)\n\nconst (\n\t// NomsVersionHeader is the name of the header that Noms clients and\n\t// servers must set in every request/response.\n\tNomsVersionHeader = \"x-noms-vers\"\n\tnomsBaseHTML = \"

Hi. This is a Noms HTTP server.

To learn more, visit our GitHub project.

\"\n\tmaxGetBatchSize = 1 << 14 // Limit GetMany() to ~16k chunks, or ~64MB of data\n)\n\nvar (\n\t// HandleWriteValue is meant to handle HTTP POST requests to the\n\t// writeValue/ server endpoint. The payload should be an appropriately-\n\t// ordered sequence of Chunks to be validated and stored on the server.\n\t// TODO: Nice comment about what headers it expects/honors, payload\n\t// format, and error responses.\n\tHandleWriteValue = createHandler(handleWriteValue, true)\n\n\t// HandleGetRefs is meant to handle HTTP POST requests to the getRefs/\n\t// server endpoint. Given a sequence of Chunk hashes, the server will\n\t// fetch and return them.\n\t// TODO: Nice comment about what headers it\n\t// expects/honors, payload format, and responses.\n\tHandleGetRefs = createHandler(handleGetRefs, true)\n\n\t// HandleGetBlob is a custom endpoint whose sole purpose is to directly\n\t// fetch the *bytes* contained in a Blob value. It expects a single query\n\t// param of `h` to be the ref of the Blob.\n\t// TODO: Support retrieving blob contents via a path.\n\tHandleGetBlob = createHandler(handleGetBlob, false)\n\n\t// HandleWriteValue is meant to handle HTTP POST requests to the hasRefs/\n\t// server endpoint. Given a sequence of Chunk hashes, the server check for\n\t// their presence and return a list of true/false responses.\n\t// TODO: Nice comment about what headers it expects/honors, payload\n\t// format, and responses.\n\tHandleHasRefs = createHandler(handleHasRefs, true)\n\n\t// HandleRootGet is meant to handle HTTP GET requests to the root/ server\n\t// endpoint. The server returns the hash of the Root as a string.\n\t// TODO: Nice comment about what headers it expects/honors, payload\n\t// format, and responses.\n\tHandleRootGet = createHandler(handleRootGet, true)\n\n\t// HandleWriteValue is meant to handle HTTP POST requests to the root/\n\t// server endpoint. This is used to update the Root to point to a new\n\t// Chunk.\n\t// TODO: Nice comment about what headers it expects/honors, payload\n\t// format, and error responses.\n\tHandleRootPost = createHandler(handleRootPost, true)\n\n\t// HandleBaseGet is meant to handle HTTP GET requests to the / server\n\t// endpoint. This is used to give a friendly message to users.\n\t// TODO: Nice comment about what headers it expects/honors, payload\n\t// format, and error responses.\n\tHandleBaseGet = handleBaseGet\n\n\tHandleGraphQL = createHandler(handleGraphQL, false)\n\n\tHandleStats = createHandler(handleStats, false)\n\n\twriteValueConcurrency = runtime.NumCPU()\n)\n\nfunc createHandler(hndlr Handler, versionCheck bool) Handler {\n\treturn func(w http.ResponseWriter, req *http.Request, ps URLParams, cs chunks.ChunkStore) {\n\t\tw.Header().Set(NomsVersionHeader, constants.NomsVersion)\n\n\t\tif versionCheck && req.Header.Get(NomsVersionHeader) != constants.NomsVersion {\n\t\t\tlog.Printf(\"returning version mismatch error\")\n\t\t\thttp.Error(\n\t\t\t\tw,\n\t\t\t\tfmt.Sprintf(\"Error: SDK version %s is incompatible with data of version %s\", req.Header.Get(NomsVersionHeader), constants.NomsVersion),\n\t\t\t\thttp.StatusBadRequest,\n\t\t\t)\n\t\t\treturn\n\t\t}\n\n\t\terr := d.Try(func() { hndlr(w, req, ps, cs) })\n\t\tif err != nil {\n\t\t\terr = d.Unwrap(err)\n\t\t\tlog.Printf(\"returning bad request error: %v\", err)\n\t\t\thttp.Error(w, fmt.Sprintf(\"Error: %v\", err), http.StatusBadRequest)\n\t\t\treturn\n\t\t}\n\t}\n}\n\nfunc handleWriteValue(w http.ResponseWriter, req *http.Request, ps URLParams, cs chunks.ChunkStore) {\n\tif req.Method != \"POST\" {\n\t\td.Panic(\"Expected post method.\")\n\t}\n\n\tt1 := time.Now()\n\ttotalDataWritten := 0\n\tchunkCount := 0\n\n\tverbose.Log(\"Handling WriteValue from \" + req.RemoteAddr)\n\tdefer func() {\n\t\tverbose.Log(\"Wrote %d Kb as %d chunks from %s in %s\", totalDataWritten/1024, chunkCount, req.RemoteAddr, time.Since(t1))\n\t}()\n\n\treader := bodyReader(req)\n\tdefer func() {\n\t\t// Ensure all data on reader is consumed\n\t\tio.Copy(ioutil.Discard, reader)\n\t\treader.Close()\n\t}()\n\tvdc := types.NewValidatingDecoder(cs)\n\n\t// Deserialize chunks from reader in background, recovering from errors\n\terrChan := make(chan error)\n\tchunkChan := make(chan *chunks.Chunk, writeValueConcurrency)\n\n\tgo func() {\n\t\tvar err error\n\t\tdefer func() { errChan <- err; close(errChan) }()\n\t\tdefer close(chunkChan)\n\t\terr = chunks.Deserialize(reader, chunkChan)\n\t}()\n\n\tdecoded := make(chan chan types.DecodedChunk, writeValueConcurrency)\n\n\tgo func() {\n\t\tdefer close(decoded)\n\t\tfor c := range chunkChan {\n\t\t\tch := make(chan types.DecodedChunk)\n\t\t\tdecoded <- ch\n\n\t\t\tgo func(ch chan types.DecodedChunk, c *chunks.Chunk) {\n\t\t\t\tch <- vdc.Decode(c)\n\t\t\t}(ch, c)\n\t\t}\n\t}()\n\n\tunresolvedRefs := hash.HashSet{}\n\tfor ch := range decoded {\n\t\tdc := <-ch\n\t\tif dc.Chunk != nil && dc.Value != nil {\n\t\t\t(*dc.Value).WalkRefs(func(r types.Ref) {\n\t\t\t\tunresolvedRefs.Insert(r.TargetHash())\n\t\t\t})\n\n\t\t\ttotalDataWritten += len(dc.Chunk.Data())\n\t\t\tcs.Put(*dc.Chunk)\n\t\t\tchunkCount++\n\t\t\tif chunkCount%100 == 0 {\n\t\t\t\tverbose.Log(\"Enqueued %d chunks\", chunkCount)\n\t\t\t}\n\t\t}\n\t}\n\n\t// If there was an error during chunk deserialization, raise so it can be logged and responded to.\n\tif err := <-errChan; err != nil {\n\t\td.Panic(\"Deserialization failure: %v\", err)\n\t}\n\n\tif chunkCount > 0 {\n\t\ttypes.PanicIfDangling(unresolvedRefs, cs)\n\t\tpersistChunks(cs)\n\t}\n\n\tw.WriteHeader(http.StatusCreated)\n}\n\n// Contents of the returned io.ReadCloser are snappy-compressed.\nfunc buildWriteValueRequest(chunkChan chan *chunks.Chunk) io.ReadCloser {\n\tbody, pw := io.Pipe()\n\n\tgo func() {\n\t\tsw := snappy.NewBufferedWriter(pw)\n\t\tdefer checkClose(pw)\n\t\tdefer checkClose(sw)\n\t\tfor c := range chunkChan {\n\t\t\tchunks.Serialize(*c, sw)\n\t\t}\n\t}()\n\n\treturn body\n}\n\nfunc checkClose(c io.Closer) {\n\td.PanicIfError(c.Close())\n}\n\nfunc bodyReader(req *http.Request) (reader io.ReadCloser) {\n\treader = req.Body\n\tif strings.Contains(req.Header.Get(\"Content-Encoding\"), \"gzip\") {\n\t\tgr, err := gzip.NewReader(reader)\n\t\td.PanicIfError(err)\n\t\treader = gr\n\t} else if strings.Contains(req.Header.Get(\"Content-Encoding\"), \"x-snappy-framed\") {\n\t\tsr := snappy.NewReader(reader)\n\t\treader = ioutil.NopCloser(sr)\n\t}\n\treturn\n}\n\nfunc respWriter(req *http.Request, w http.ResponseWriter) (writer io.WriteCloser) {\n\twriter = wc{w.(io.Writer)}\n\tif strings.Contains(req.Header.Get(\"Accept-Encoding\"), \"gzip\") {\n\t\tw.Header().Add(\"Content-Encoding\", \"gzip\")\n\t\tgw := gzip.NewWriter(w)\n\t\twriter = gw\n\t} else if strings.Contains(req.Header.Get(\"Accept-Encoding\"), \"x-snappy-framed\") {\n\t\tw.Header().Add(\"Content-Encoding\", \"x-snappy-framed\")\n\t\tsw := snappy.NewBufferedWriter(w)\n\t\twriter = sw\n\t}\n\treturn\n}\n\ntype wc struct {\n\tio.Writer\n}\n\nfunc (wc wc) Close() error {\n\treturn nil\n}\n\nfunc persistChunks(cs chunks.ChunkStore) {\n\tfor !cs.Commit(cs.Root(), cs.Root()) {\n\t}\n}\n\nfunc handleGetRefs(w http.ResponseWriter, req *http.Request, ps URLParams, cs chunks.ChunkStore) {\n\tif req.Method != \"POST\" {\n\t\td.Panic(\"Expected post method.\")\n\t}\n\n\thashes := extractHashes(req)\n\n\tverbose.Log(\"Handling getRefs request for: %v\\n\", hashes)\n\n\tw.Header().Add(\"Content-Type\", \"application/octet-stream\")\n\twriter := respWriter(req, w)\n\tdefer writer.Close()\n\n\tfor len(hashes) > 0 {\n\t\tbatch := hashes\n\n\t\t// Limit RAM consumption by streaming chunks in ~8MB batches\n\t\tif len(batch) > maxGetBatchSize {\n\t\t\tbatch = batch[:maxGetBatchSize]\n\t\t}\n\n\t\tchunkChan := make(chan *chunks.Chunk, maxGetBatchSize)\n\t\tabsent := batch.HashSet()\n\t\tgo func() {\n\t\t\tcs.GetMany(batch.HashSet(), chunkChan)\n\t\t\tclose(chunkChan)\n\t\t}()\n\n\t\tfor c := range chunkChan {\n\t\t\tchunks.Serialize(*c, writer)\n\t\t\tabsent.Remove(c.Hash())\n\t\t}\n\n\t\tif len(absent) > 0 {\n\t\t\tfmt.Fprintf(os.Stderr, \"ERROR: Could not get chunks: %v\\n\", absent)\n\t\t}\n\n\t\thashes = hashes[len(batch):]\n\t}\n}\n\nfunc handleGetBlob(w http.ResponseWriter, req *http.Request, ps URLParams, cs chunks.ChunkStore) {\n\trefStr := req.URL.Query().Get(\"h\")\n\tif refStr == \"\" {\n\t\td.Panic(\"Expected h param\")\n\t}\n\n\th := hash.Parse(refStr)\n\tif (h == hash.Hash{}) {\n\t\td.Panic(\"h failed to parse\")\n\t}\n\n\tvs := types.NewValueStore(cs)\n\tv := vs.ReadValue(h)\n\tb, ok := v.(types.Blob)\n\tif !ok {\n\t\td.Panic(\"h is not a Blob\")\n\t}\n\n\tw.Header().Add(\"Content-Type\", \"application/octet-stream\")\n\tw.Header().Add(\"Content-Length\", fmt.Sprintf(\"%d\", b.Len()))\n\tw.Header().Add(\"Cache-Control\", fmt.Sprintf(\"max-age=%d\", 60*60*24*365))\n\n\tb.Copy(w)\n}\n\nfunc extractHashes(req *http.Request) hash.HashSlice {\n\treader := bodyReader(req)\n\tdefer reader.Close()\n\tdefer io.Copy(ioutil.Discard, reader) // Ensure all data on reader is consumed\n\treturn deserializeHashes(reader)\n}\n\nfunc BuildHashesRequestForTest(hashes hash.HashSet) io.ReadCloser {\n\tbatch := chunks.ReadBatch{}\n\tfor h := range hashes {\n\t\tbatch[h] = nil\n\t}\n\treturn buildHashesRequest(batch)\n}\n\nfunc buildHashesRequest(batch chunks.ReadBatch) io.ReadCloser {\n\tbody, pw := io.Pipe()\n\tgo func() {\n\t\tdefer checkClose(pw)\n\t\tserializeHashes(pw, batch)\n\t}()\n\treturn body\n}\n\nfunc handleHasRefs(w http.ResponseWriter, req *http.Request, ps URLParams, cs chunks.ChunkStore) {\n\tif req.Method != \"POST\" {\n\t\td.Panic(\"Expected post method.\")\n\t}\n\n\thashes := extractHashes(req)\n\n\tw.Header().Add(\"Content-Type\", \"text/plain\")\n\twriter := respWriter(req, w)\n\tdefer writer.Close()\n\n\tabsent := cs.HasMany(hashes.HashSet())\n\tfor h := range absent {\n\t\tfmt.Fprintln(writer, h.String())\n\t}\n}\n\nfunc handleRootGet(w http.ResponseWriter, req *http.Request, ps URLParams, rt chunks.ChunkStore) {\n\tif req.Method != \"GET\" {\n\t\td.Panic(\"Expected get method.\")\n\t}\n\tfmt.Fprintf(w, \"%v\", rt.Root().String())\n\tw.Header().Add(\"content-type\", \"text/plain\")\n}\n\nfunc handleStats(w http.ResponseWriter, req *http.Request, ps URLParams, cs chunks.ChunkStore) {\n\tif req.Method != \"GET\" {\n\t\td.Panic(\"Expected get method.\")\n\t}\n\tfmt.Fprint(w, cs.StatsSummary())\n\tw.Header().Add(\"content-type\", \"text/plain\")\n}\n\nfunc handleRootPost(w http.ResponseWriter, req *http.Request, ps URLParams, cs chunks.ChunkStore) {\n\tif req.Method != \"POST\" {\n\t\td.Panic(\"Expected post method.\")\n\t}\n\n\tparams := req.URL.Query()\n\ttokens := params[\"last\"]\n\tif len(tokens) != 1 {\n\t\td.Panic(`Expected \"last\" query param value`)\n\t}\n\tlast := hash.Parse(tokens[0])\n\t// \"current\" should really, really be called \"proposed\" or something in the wire API\n\ttokens = params[\"current\"]\n\tif len(tokens) != 1 {\n\t\td.Panic(`Expected \"current\" query param value`)\n\t}\n\tproposed := hash.Parse(tokens[0])\n\n\tvs := types.NewValueStore(cs)\n\n\t// Even though the Root is actually a Map>, its Noms Type is Map> in order to prevent the root chunk from getting bloated with type info. That means that the Value of the proposed new Root needs to be manually type-checked. The simplest way to do that would be to iterate over the whole thing and pull the target of each Ref from |cs|. That's a lot of reads, though, and it's more efficient to just read the Value indicated by |last|, diff the proposed new root against it, and validate whatever new entries appear.\n\tlastMap := validateLast(last, vs)\n\n\tproposedMap := validateProposed(proposed, last, vs)\n\tif !proposedMap.Empty() {\n\t\tassertMapOfStringToRefOfCommit(proposedMap, lastMap, vs)\n\t}\n\n\t// If some other client has committed to |vs| since it had |from| at the\n\t// root, this call to vs.Commit() will fail. Used to be that we'd always\n\t// propagate that failure back to the client and let them try again. This\n\t// made one very common operation annoyingly expensive, though, as clients\n\t// simultaneously committing to different Datasets would cause conflicts\n\t// with this vs.Commit() right here. In this common case, the server\n\t// already knows everything it needs to try again, so now we cut out the\n\t// round trip to the client and just retry inline.\n\tfor to, from := proposed, last; !vs.Commit(to, from); {\n\t\t// If committing failed, we go read out the map of Datasets at the root of the store, which is a Map[string]Ref\n\t\trootMap := types.NewMap(vs)\n\t\troot := vs.Root()\n\t\tif v := vs.ReadValue(root); v != nil {\n\t\t\trootMap = v.(types.Map)\n\t\t}\n\n\t\t// Since we know that lastMap is an ancestor of both proposedMap and\n\t\t// rootMap, we can try to do a three-way merge here. We don't want to\n\t\t// traverse the Refs stored in the maps, though, just\n\t\t// basically merge the maps together as long the changes to rootMap\n\t\t// and proposedMap were in different Datasets.\n\t\tmerged, err := mergeDatasetMaps(proposedMap, rootMap, lastMap, vs)\n\t\tif err != nil {\n\t\t\tverbose.Log(\"Attempted root map auto-merge failed: %s\", err)\n\t\t\tw.WriteHeader(http.StatusConflict)\n\t\t\tbreak\n\t\t}\n\t\tto, from = vs.WriteValue(merged).TargetHash(), root\n\t}\n\n\t// If committing succeeded, the root of the store might be |proposed|...or\n\t// it might be some result of the merge performed above. So, we need to\n\t// tell the client what the new root is. If the commit failed, obviously\n\t// we need to inform the client of the actual current root.\n\tw.Header().Add(\"content-type\", \"text/plain\")\n\tfmt.Fprintf(w, \"%v\", vs.Root().String())\n}\n\nfunc validateLast(last hash.Hash, vrw types.ValueReadWriter) types.Map {\n\tif last.IsEmpty() {\n\t\treturn types.NewMap(vrw)\n\t}\n\tlastVal := vrw.ReadValue(last)\n\tif lastVal == nil {\n\t\td.Panic(\"Can't Commit from a non-present Chunk\")\n\t}\n\treturn lastVal.(types.Map)\n}\n\nfunc validateProposed(proposed, last hash.Hash, vrw types.ValueReadWriter) types.Map {\n\t// Only allowed to skip this check if both last and proposed are empty, because that represents the special case of someone flushing chunks into an empty store.\n\tif last.IsEmpty() && proposed.IsEmpty() {\n\t\treturn types.NewMap(vrw)\n\t}\n\t// Ensure that proposed new Root is present in vr, is a Map and, if it has anything in it, that it's >\n\tproposedVal := vrw.ReadValue(proposed)\n\tif proposedVal == nil {\n\t\td.Panic(\"Can't set Root to a non-present Chunk\")\n\t}\n\n\tproposedMap, ok := proposedVal.(types.Map)\n\tif !ok {\n\t\td.Panic(\"Root of a Database must be a Map\")\n\t}\n\treturn proposedMap\n}\n\nfunc assertMapOfStringToRefOfCommit(proposed, datasets types.Map, vr types.ValueReader) {\n\tstopChan := make(chan struct{})\n\tdefer close(stopChan)\n\tchanges := make(chan types.ValueChanged)\n\tgo func() {\n\t\tdefer close(changes)\n\t\tproposed.Diff(datasets, changes, stopChan)\n\t}()\n\tfor change := range changes {\n\t\tswitch change.ChangeType {\n\t\tcase types.DiffChangeAdded, types.DiffChangeModified:\n\t\t\t// Since this is a Map Diff, change.V is the key at which a change was detected.\n\t\t\t// Go get the Value there, which should be a Ref, deref it, and then ensure the target is a Commit.\n\t\t\tval := change.NewValue\n\t\t\tref, ok := val.(types.Ref)\n\t\t\tif !ok {\n\t\t\t\td.Panic(\"Root of a Database must be a Map>, but key %s maps to a %s\", change.Key.(types.String), types.TypeOf(val).Describe())\n\t\t\t}\n\t\t\tif targetValue := ref.TargetValue(vr); !IsCommit(targetValue) {\n\t\t\t\td.Panic(\"Root of a Database must be a Map>, but the ref at key %s points to a %s\", change.Key.(types.String), types.TypeOf(targetValue).Describe())\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc mergeDatasetMaps(a, b, parent types.Map, vrw types.ValueReadWriter) (types.Map, error) {\n\taChangeChan, bChangeChan := make(chan types.ValueChanged), make(chan types.ValueChanged)\n\tstopChan := make(chan struct{})\n\n\tgo func() {\n\t\tdefer close(aChangeChan)\n\t\ta.Diff(parent, aChangeChan, stopChan)\n\t}()\n\tgo func() {\n\t\tdefer close(bChangeChan)\n\t\tb.Diff(parent, bChangeChan, stopChan)\n\t}()\n\tdefer func() {\n\t\tclose(stopChan)\n\t\tfor range aChangeChan {\n\t\t}\n\t\tfor range bChangeChan {\n\t\t}\n\t}()\n\n\tapply := func(target *types.MapEditor, change types.ValueChanged, newVal types.Value) *types.MapEditor {\n\t\tswitch change.ChangeType {\n\t\tcase types.DiffChangeAdded, types.DiffChangeModified:\n\t\t\treturn target.Set(change.Key, newVal)\n\t\tcase types.DiffChangeRemoved:\n\t\t\treturn target.Remove(change.Key)\n\t\tdefault:\n\t\t\tpanic(\"Not Reached\")\n\t\t}\n\t}\n\n\tmerged := parent.Edit()\n\taChange, bChange := types.ValueChanged{}, types.ValueChanged{}\n\tfor {\n\t\tif aChange.Key == nil {\n\t\t\taChange = <-aChangeChan\n\t\t}\n\t\tif bChange.Key == nil {\n\t\t\tbChange = <-bChangeChan\n\t\t}\n\n\t\t// Both channels are producing zero values, so we're done.\n\t\tif aChange.Key == nil && bChange.Key == nil {\n\t\t\tbreak\n\t\t}\n\n\t\tif aChange.Key != nil && (bChange.Key == nil || aChange.Key.Less(bChange.Key)) {\n\t\t\tmerged = apply(merged, aChange, a.Get(aChange.Key))\n\t\t\taChange = types.ValueChanged{}\n\t\t\tcontinue\n\t\t} else if bChange.Key != nil && (aChange.Key == nil || bChange.Key.Less(aChange.Key)) {\n\t\t\tmerged = apply(merged, bChange, b.Get(bChange.Key))\n\t\t\tbChange = types.ValueChanged{}\n\t\t\tcontinue\n\t\t}\n\n\t\td.PanicIfFalse(aChange.Key.Equals(bChange.Key))\n\t\t// If the two diffs generate different kinds of changes at the same key, conflict.\n\t\tif aChange.ChangeType != bChange.ChangeType {\n\t\t\treturn parent, errors.New(\"Incompatible changes at \" + types.EncodedValue(aChange.Key))\n\t\t}\n\n\t\t// Otherwise, we're OK IFF the two diffs made exactly the same change\n\t\taValue := a.Get(aChange.Key)\n\t\tif aChange.ChangeType != types.DiffChangeRemoved && !aValue.Equals(b.Get(bChange.Key)) {\n\t\t\treturn parent, errors.New(\"Incompatible changes at \" + types.EncodedValue(aChange.Key))\n\t\t}\n\t\tmerged = apply(merged, aChange, aValue)\n\t\taChange, bChange = types.ValueChanged{}, types.ValueChanged{}\n\t}\n\treturn merged.Map(), nil\n}\n\nfunc handleGraphQL(w http.ResponseWriter, req *http.Request, ps URLParams, cs chunks.ChunkStore) {\n\tif req.Method != http.MethodGet && req.Method != http.MethodPost {\n\t\td.Panic(\"Unexpected method\")\n\t}\n\n\tds := req.FormValue(\"ds\")\n\th := req.FormValue(\"h\")\n\n\tif (ds == \"\") == (h == \"\") {\n\t\td.Panic(\"Must specify one (and only one) of ds (dataset) or h (hash)\")\n\t}\n\n\tvar query string\n\tif req.Header.Get(\"Content-Type\") == \"application/json\" {\n\t\tvar body struct {\n\t\t\tQuery string\n\t\t}\n\t\terr := json.NewDecoder(req.Body).Decode(&body)\n\t\tif err != nil {\n\t\t\td.Panic(\"invalid query: %s\", err)\n\t\t}\n\t\tquery = body.Query\n\t} else {\n\t\tquery = req.FormValue(\"query\")\n\t\tif query == \"\" {\n\t\t\td.Panic(\"Expected query\")\n\t\t}\n\t}\n\n\t// Note: we don't close this becaues |cs| will be closed by the generic endpoint handler\n\tdb := NewDatabase(cs)\n\n\tvar rootValue types.Value\n\tvar err error\n\tif ds != \"\" {\n\t\tdataset := db.GetDataset(ds)\n\t\tvar ok bool\n\t\trootValue, ok = dataset.MaybeHead()\n\t\tif !ok {\n\t\t\terr = fmt.Errorf(\"Dataset %s not found\", ds)\n\t\t}\n\t} else {\n\t\trootValue = db.ReadValue(hash.Parse(h))\n\t\tif rootValue == nil {\n\t\t\terr = errors.New(\"Root value not found\")\n\t\t}\n\t}\n\n\tw.Header().Add(\"Content-Type\", \"application/json\")\n\twriter := respWriter(req, w)\n\tdefer writer.Close()\n\n\tif err != nil {\n\t\tngql.Error(err, writer)\n\t} else {\n\t\tngql.Query(rootValue, query, db, writer)\n\t}\n}\n\nfunc handleBaseGet(w http.ResponseWriter, req *http.Request, ps URLParams, rt chunks.ChunkStore) {\n\tif req.Method != \"GET\" {\n\t\td.Panic(\"Expected get method.\")\n\t}\n\n\tw.Header().Add(\"Content-Type\", \"text/html\")\n\tfmt.Fprintf(w, nomsBaseHTML)\n}\n" }, { "path": "go/datas/remote_database_handlers_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"net/url\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/golang/snappy\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestHandleWriteValue(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.TestStorage{}\n\tdb := NewDatabase(storage.NewView())\n\n\tl := types.NewList(\n\t\tdb,\n\t\tdb.WriteValue(types.Bool(true)),\n\t\tdb.WriteValue(types.Bool(false)),\n\t)\n\tr := db.WriteValue(l)\n\t_, err := db.CommitValue(db.GetDataset(\"datasetID\"), r)\n\tassert.NoError(err)\n\n\tnewItem := types.NewEmptyBlob(db)\n\titemChunk := types.EncodeValue(newItem)\n\tl2 := l.Edit().Insert(1, types.NewRef(newItem)).List()\n\tlistChunk := types.EncodeValue(l2)\n\n\tbody := &bytes.Buffer{}\n\tchunks.Serialize(itemChunk, body)\n\tchunks.Serialize(listChunk, body)\n\n\tw := httptest.NewRecorder()\n\tHandleWriteValue(w, newRequest(\"POST\", \"\", \"\", body, nil), params{}, storage.NewView())\n\n\tif assert.Equal(http.StatusCreated, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes())) {\n\t\tdb2 := NewDatabase(storage.NewView())\n\t\tv := db2.ReadValue(l2.Hash())\n\t\tif assert.NotNil(v) {\n\t\t\tassert.True(v.Equals(l2), \"%+v != %+v\", v, l2)\n\t\t}\n\t}\n}\n\nfunc TestHandleWriteValuePanic(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\n\tbody := &bytes.Buffer{}\n\tbody.WriteString(\"Bogus\")\n\n\tw := httptest.NewRecorder()\n\tHandleWriteValue(w, newRequest(\"POST\", \"\", \"\", body, nil), params{}, storage.NewView())\n\n\tassert.Equal(http.StatusBadRequest, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes()))\n}\n\nfunc TestHandleWriteValueDupChunks(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := NewDatabase(storage.NewView())\n\tdefer db.Close()\n\n\tnewItem := types.NewEmptyBlob(db)\n\titemChunk := types.EncodeValue(newItem)\n\n\tbody := &bytes.Buffer{}\n\t// Write the same chunk to body enough times to be certain that at least one of the concurrent deserialize/decode passes completes before the last one can continue.\n\tfor i := 0; i <= writeValueConcurrency; i++ {\n\t\tchunks.Serialize(itemChunk, body)\n\t}\n\n\tw := httptest.NewRecorder()\n\tHandleWriteValue(w, newRequest(\"POST\", \"\", \"\", body, nil), params{}, storage.NewView())\n\n\tif assert.Equal(http.StatusCreated, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes())) {\n\t\tdb := NewDatabase(storage.NewView())\n\t\tv := db.ReadValue(newItem.Hash())\n\t\tif assert.NotNil(v) {\n\t\t\tassert.True(v.Equals(newItem), \"%+v != %+v\", v, newItem)\n\t\t}\n\t}\n}\n\nfunc TestBuildWriteValueRequest(t *testing.T) {\n\tassert := assert.New(t)\n\tinput1, input2 := \"abc\", \"def\"\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(input1)),\n\t\tchunks.NewChunk([]byte(input2)),\n\t}\n\n\tinChunkChan := make(chan *chunks.Chunk, 2)\n\tinChunkChan <- &chnx[0]\n\tinChunkChan <- &chnx[1]\n\tclose(inChunkChan)\n\n\tcompressed := buildWriteValueRequest(inChunkChan)\n\tgr := snappy.NewReader(compressed)\n\n\toutChunkChan := make(chan *chunks.Chunk, len(chnx))\n\tchunks.Deserialize(gr, outChunkChan)\n\tclose(outChunkChan)\n\n\tfor c := range outChunkChan {\n\t\tassert.Equal(chnx[0].Hash(), c.Hash())\n\t\tchnx = chnx[1:]\n\t}\n\tassert.Empty(chnx)\n}\n\nfunc serializeChunks(chnx []chunks.Chunk, assert *assert.Assertions) io.Reader {\n\tbody := &bytes.Buffer{}\n\tsw := snappy.NewBufferedWriter(body)\n\tfor _, chunk := range chnx {\n\t\tchunks.Serialize(chunk, sw)\n\t}\n\tassert.NoError(sw.Close())\n\treturn body\n}\n\nfunc TestBuildHashesRequest(t *testing.T) {\n\tassert := assert.New(t)\n\tbatch := chunks.ReadBatch{\n\t\thash.Parse(\"00000000000000000000000000000002\"): nil,\n\t\thash.Parse(\"00000000000000000000000000000003\"): nil,\n\t}\n\tr := buildHashesRequest(batch)\n\tdefer r.Close()\n\trequested := deserializeHashes(r)\n\n\tfor _, h := range requested {\n\t\t_, present := batch[h]\n\t\tassert.True(present, \"Query contains %s, which is not in initial refs\", h)\n\t}\n}\n\nfunc TestHandleGetRefs(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tcs := storage.NewView()\n\tinput1, input2 := \"abc\", \"def\"\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(input1)),\n\t\tchunks.NewChunk([]byte(input2)),\n\t}\n\tfor _, c := range chnx {\n\t\tcs.Put(c)\n\t}\n\tpersistChunks(cs)\n\n\tbody := buildHashesRequest(chunks.ReadBatch{chnx[0].Hash(): nil, chnx[1].Hash(): nil})\n\n\tw := httptest.NewRecorder()\n\tHandleGetRefs(\n\t\tw,\n\t\tnewRequest(\"POST\", \"\", \"\", body, http.Header{\n\t\t\t\"Content-Type\": {\"application/octet-stream\"},\n\t\t}),\n\t\tparams{},\n\t\tstorage.NewView(),\n\t)\n\n\tif assert.Equal(http.StatusOK, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes())) {\n\t\tchunkChan := make(chan *chunks.Chunk, len(chnx))\n\t\tchunks.Deserialize(w.Body, chunkChan)\n\t\tclose(chunkChan)\n\n\t\tfoundHashes := hash.HashSet{}\n\t\tfor c := range chunkChan {\n\t\t\tfoundHashes[c.Hash()] = struct{}{}\n\t\t}\n\n\t\tassert.True(len(foundHashes) == 2)\n\t\t_, hasC1 := foundHashes[chnx[0].Hash()]\n\t\tassert.True(hasC1)\n\t\t_, hasC2 := foundHashes[chnx[1].Hash()]\n\t\tassert.True(hasC2)\n\t}\n}\n\nfunc TestHandleGetBlob(t *testing.T) {\n\tassert := assert.New(t)\n\n\tblobContents := \"I am a blob\"\n\tstorage := &chunks.MemoryStorage{}\n\tdb := NewDatabase(storage.NewView())\n\tds := db.GetDataset(\"foo\")\n\n\t// Test missing h\n\tw := httptest.NewRecorder()\n\tHandleGetBlob(\n\t\tw,\n\t\tnewRequest(\"GET\", \"\", \"/getBlob/\", strings.NewReader(\"\"), http.Header{}),\n\t\tparams{},\n\t\tstorage.NewView(),\n\t)\n\tassert.Equal(http.StatusBadRequest, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes()))\n\n\tb := types.NewBlob(db, bytes.NewBuffer([]byte(blobContents)))\n\n\t// Test non-present hash\n\tw = httptest.NewRecorder()\n\tHandleGetBlob(\n\t\tw,\n\t\tnewRequest(\"GET\", \"\", fmt.Sprintf(\"/getBlob/?h=%s\", b.Hash().String()), strings.NewReader(\"\"), http.Header{}),\n\t\tparams{},\n\t\tstorage.NewView(),\n\t)\n\tassert.Equal(http.StatusBadRequest, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes()))\n\n\tr := db.WriteValue(b)\n\tds, err := db.CommitValue(ds, r)\n\tassert.NoError(err)\n\n\t// Valid\n\tw = httptest.NewRecorder()\n\tHandleGetBlob(\n\t\tw,\n\t\tnewRequest(\"GET\", \"\", fmt.Sprintf(\"/getBlob/?h=%s\", r.TargetHash().String()), strings.NewReader(\"\"), http.Header{}),\n\t\tparams{},\n\t\tstorage.NewView(),\n\t)\n\n\tif assert.Equal(http.StatusOK, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes())) {\n\t\tout, _ := ioutil.ReadAll(w.Body)\n\t\tassert.Equal(string(out), blobContents)\n\t}\n\n\t// Test non-blob\n\tr2 := db.WriteValue(types.Number(1))\n\t_, err = db.CommitValue(ds, r2)\n\tassert.NoError(err)\n\n\tw = httptest.NewRecorder()\n\tHandleGetBlob(\n\t\tw,\n\t\tnewRequest(\"GET\", \"\", fmt.Sprintf(\"/getBlob/?h=%s\", r2.TargetHash().String()), strings.NewReader(\"\"), http.Header{}),\n\t\tparams{},\n\t\tstorage.NewView(),\n\t)\n\tassert.Equal(http.StatusBadRequest, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes()))\n}\n\nfunc TestHandleHasRefs(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tinput1, input2, input3 := \"abc\", \"def\", \"ghi\"\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(input1)),\n\t\tchunks.NewChunk([]byte(input2)),\n\t}\n\tpresent := chunks.NewChunk([]byte(input3))\n\tcs := storage.NewView()\n\tcs.Put(present)\n\tpersistChunks(cs)\n\n\tbody := buildHashesRequest(chunks.ReadBatch{\n\t\tchnx[0].Hash(): nil,\n\t\tchnx[1].Hash(): nil,\n\t\tpresent.Hash(): nil,\n\t})\n\n\tw := httptest.NewRecorder()\n\tHandleHasRefs(\n\t\tw,\n\t\tnewRequest(\"POST\", \"\", \"\", body, http.Header{\n\t\t\t\"Content-Type\": {\"application/octet-stream\"},\n\t\t}),\n\t\tparams{},\n\t\tstorage.NewView(),\n\t)\n\n\tabsent := hash.HashSet{}\n\tif assert.Equal(http.StatusOK, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes())) {\n\t\tscanner := bufio.NewScanner(w.Body)\n\t\tscanner.Split(bufio.ScanWords)\n\t\tfor scanner.Scan() {\n\t\t\tabsent.Insert(hash.Parse(scanner.Text()))\n\t\t}\n\t}\n\tif assert.Len(absent, len(chnx)) {\n\t\tfor _, c := range chnx {\n\t\t\tassert.True(absent.Has(c.Hash()))\n\t\t}\n\t\tassert.False(absent.Has(present.Hash()))\n\t}\n}\n\nfunc TestHandleGetRoot(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tcs := storage.NewView()\n\tc := chunks.NewChunk([]byte(\"abc\"))\n\tcs.Put(c)\n\tassert.True(cs.Commit(c.Hash(), hash.Hash{}))\n\n\tw := httptest.NewRecorder()\n\tHandleRootGet(w, newRequest(\"GET\", \"\", \"\", nil, nil), params{}, storage.NewView())\n\n\tif assert.Equal(http.StatusOK, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes())) {\n\t\troot := hash.Parse(string(w.Body.Bytes()))\n\t\tassert.Equal(c.Hash(), root)\n\t}\n}\n\nfunc TestHandleGetBase(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\n\tw := httptest.NewRecorder()\n\tHandleBaseGet(w, newRequest(\"GET\", \"\", \"\", nil, nil), params{}, storage.NewView())\n\n\tif assert.Equal(http.StatusOK, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes())) {\n\t\tassert.Equal([]byte(nomsBaseHTML), w.Body.Bytes())\n\t}\n}\n\nfunc TestHandlePostRoot(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tcs := storage.NewView()\n\tvs := types.NewValueStore(cs)\n\n\tvalidate := func(code int, root hash.Hash, w *httptest.ResponseRecorder) {\n\t\tassert.Equal(code, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes()))\n\t\tassert.Equal(root, hash.Parse(string(w.Body.Bytes())))\n\t}\n\n\t// Empty -> Empty should be OK.\n\turl := buildPostRootURL(hash.Hash{}, hash.Hash{})\n\tw := httptest.NewRecorder()\n\tHandleRootPost(w, newRequest(\"POST\", \"\", url, nil, nil), params{}, storage.NewView())\n\tvalidate(http.StatusOK, hash.Hash{}, w)\n\n\tcommit := buildTestCommit(vs, types.String(\"head\"))\n\tcommitRef := vs.WriteValue(commit)\n\tfirstHead := types.NewMap(vs, types.String(\"dataset1\"), types.ToRefOfValue(commitRef))\n\tfirstHeadRef := vs.WriteValue(firstHead)\n\tvs.Commit(vs.Root(), vs.Root())\n\n\tcommit = buildTestCommit(vs, types.String(\"second\"), commitRef)\n\tnewHead := types.NewMap(vs, types.String(\"dataset1\"), types.ToRefOfValue(vs.WriteValue(commit)))\n\tnewHeadRef := vs.WriteValue(newHead)\n\tvs.Commit(vs.Root(), vs.Root())\n\n\t// First attempt should fail, as 'last' won't match.\n\turl = buildPostRootURL(newHeadRef.TargetHash(), firstHeadRef.TargetHash())\n\tw = httptest.NewRecorder()\n\tHandleRootPost(w, newRequest(\"POST\", \"\", url, nil, nil), params{}, storage.NewView())\n\tvalidate(http.StatusConflict, hash.Hash{}, w)\n\n\t// Now, update the root manually to 'last' and try again.\n\tassert.True(cs.Commit(firstHeadRef.TargetHash(), hash.Hash{}))\n\tw = httptest.NewRecorder()\n\tHandleRootPost(w, newRequest(\"POST\", \"\", url, nil, nil), params{}, storage.NewView())\n\tvalidate(http.StatusOK, newHeadRef.TargetHash(), w)\n}\n\nfunc buildPostRootURL(current, last hash.Hash) string {\n\tu := &url.URL{}\n\tqueryParams := url.Values{}\n\tqueryParams.Add(\"last\", last.String())\n\tqueryParams.Add(\"current\", current.String())\n\tu.RawQuery = queryParams.Encode()\n\treturn u.String()\n}\n\nfunc buildTestCommit(vrw types.ValueReadWriter, v types.Value, parents ...types.Value) types.Struct {\n\treturn NewCommit(v, types.NewSet(vrw, parents...), types.NewStruct(\"Meta\", types.StructData{}))\n}\n\nfunc TestRejectPostRoot(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tcs := storage.NewView()\n\tvs := types.NewValueStore(cs)\n\tdefer vs.Close()\n\n\tnewHead := types.NewMap(vs, types.String(\"dataset1\"), types.String(\"Not a Head\"))\n\tchunk := types.EncodeValue(newHead)\n\tcs.Put(chunk)\n\tpersistChunks(cs)\n\n\t// Attempt should fail, as newHead isn't the right type.\n\turl := buildPostRootURL(chunk.Hash(), hash.Hash{})\n\tw := httptest.NewRecorder()\n\tHandleRootPost(w, newRequest(\"POST\", \"\", url, nil, nil), params{}, storage.NewView())\n\tassert.Equal(http.StatusBadRequest, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes()))\n\n\t// Put in a legit commit\n\tcommit := buildTestCommit(vs, types.String(\"commit\"))\n\thead := types.NewMap(vs, types.String(\"dataset1\"), types.ToRefOfValue(vs.WriteValue(commit)))\n\theadRef := vs.WriteValue(head)\n\tassert.True(vs.Commit(headRef.TargetHash(), vs.Root()))\n\n\t// Attempt to update head to empty hash should fail\n\turl = buildPostRootURL(hash.Hash{}, headRef.TargetHash())\n\tw = httptest.NewRecorder()\n\tHandleRootPost(w, newRequest(\"POST\", \"\", url, nil, nil), params{}, storage.NewView())\n\tassert.Equal(http.StatusBadRequest, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes()))\n\n\t// Attempt to update from a non-present chunks should fail\n\turl = buildPostRootURL(headRef.TargetHash(), chunks.EmptyChunk.Hash())\n\tw = httptest.NewRecorder()\n\tHandleRootPost(w, newRequest(\"POST\", \"\", url, nil, nil), params{}, storage.NewView())\n\tassert.Equal(http.StatusBadRequest, w.Code, \"Handler error:\\n%s\", string(w.Body.Bytes()))\n}\n\ntype params map[string]string\n\nfunc (p params) ByName(k string) string {\n\treturn p[k]\n}\n" }, { "path": "go/datas/serialize_hashes.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"encoding/binary\"\n\t\"io\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\nfunc serializedLength(batch chunks.ReadBatch) uint32 {\n\treturn uint32(len(batch)*hash.ByteLen + binary.Size(uint32(0)))\n}\n\nfunc serializeHashes(w io.Writer, batch chunks.ReadBatch) {\n\terr := binary.Write(w, binary.BigEndian, uint32(len(batch))) // 4 billion hashes is probably absurd. Maybe this should be smaller?\n\td.PanicIfError(err)\n\tfor h := range batch {\n\t\tserializeHash(w, h)\n\t}\n}\n\nfunc serializeHash(w io.Writer, h hash.Hash) {\n\t_, err := w.Write(h[:])\n\td.PanicIfError(err)\n}\n\nfunc deserializeHashes(reader io.Reader) hash.HashSlice {\n\tcount := uint32(0)\n\terr := binary.Read(reader, binary.BigEndian, &count)\n\td.PanicIfError(err)\n\n\thashes := make(hash.HashSlice, count)\n\tfor i := range hashes {\n\t\thashes[i] = deserializeHash(reader)\n\t}\n\treturn hashes\n}\n\nfunc deserializeHash(reader io.Reader) hash.Hash {\n\th := hash.Hash{}\n\tn, err := io.ReadFull(reader, h[:])\n\td.PanicIfError(err)\n\td.PanicIfFalse(int(hash.ByteLen) == n)\n\treturn h\n}\n" }, { "path": "go/datas/serialize_hashes_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datas\n\nimport (\n\t\"bytes\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestHashRoundTrip(t *testing.T) {\n\tb := &bytes.Buffer{}\n\tinput := chunks.ReadBatch{\n\t\thash.Parse(\"00000000000000000000000000000000\"): nil,\n\t\thash.Parse(\"00000000000000000000000000000001\"): nil,\n\t\thash.Parse(\"00000000000000000000000000000002\"): nil,\n\t\thash.Parse(\"00000000000000000000000000000003\"): nil,\n\t}\n\tdefer input.Close()\n\n\tserializeHashes(b, input)\n\tserializedLen := b.Len()\n\toutput := deserializeHashes(b)\n\tassert.Len(t, output, len(input), \"Output has different number of elements than input: %v, %v\", output, input)\n\tfor _, h := range output {\n\t\t_, present := input[h]\n\t\tassert.True(t, present, \"%s is in output but not in input\", h)\n\t}\n\n\tassert.Equal(t, uint32(serializedLen), serializedLength(input))\n}\n" }, { "path": "go/diff/apply_patch.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage diff\n\nimport (\n\t\"fmt\"\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// Apply applies a Patch (list of diffs) to a graph. It fulfills the\n// following contract:\n// Given 2 Noms graphs: a1 and a2:\n// ApplyPatch(a1, Diff(a1, a2)) == a2\n// This is useful for IncrementalUpdate() and possibly other problems. See\n// updater.go for more information.\n//\n// This function uses a patchStack to maintain state of the graph as it cycles\n// through the diffs in a patch, applying them to 'root' one by one. Because the\n// Difference objects in the patch can be sorted according to their path, each\n// one is applied in order. When done in combination with the stack, this enables\n// all Differences that change a particular node to be applied to that node\n// before it gets assigned back to it's parent.\nfunc Apply(root types.Value, patch Patch) types.Value {\n\tif len(patch) == 0 {\n\t\treturn root\n\t}\n\n\tvar lastPath types.Path\n\tstack := patchStack{}\n\tsort.Sort(patch)\n\n\t// Push the element on the stack that corresponds to the root\n\t// node.\n\tstack.push(nil, nil, types.DiffChangeModified, root, nil, nil)\n\n\tfor _, dif := range patch {\n\t\t// get the path where this dif needs to be applied\n\t\tp := dif.Path\n\n\t\t// idx will hold the index of the last common element between p and\n\t\t// lastPath (p from the last iteration).\n\t\tvar idx int\n\n\t\t// p can be identical to lastPath in certain cases. For example, when\n\t\t// one item gets removed from a list at the same place another item\n\t\t// is added to it. In this case, we need pop the last operation of the\n\t\t// stack early and set the idx to be the len(p) - 1.\n\t\t// Otherwise, if the paths are different we can call commonPrefixCount()\n\t\tif len(p) > 0 && p.Equals(lastPath) {\n\t\t\tstack.pop()\n\t\t\tidx = len(p) - 1\n\t\t} else {\n\t\t\tidx = commonPrefixCount(lastPath, p)\n\t\t}\n\t\tlastPath = p\n\n\t\t// if the stack has elements on it leftover from the last iteration. Pop\n\t\t// those elements until the stack only has values in it that are\n\t\t// referenced by this p. Popping an element on the stack, folds that\n\t\t// value into it's parent.\n\t\tfor idx < stack.Len()-1 {\n\t\t\tstack.pop()\n\t\t}\n\n\t\t// tail is the part of the current path that has not yet been pushed\n\t\t// onto the stack. Iterate over those pathParts and push those values\n\t\t// onto the stack.\n\t\ttail := p[idx:]\n\t\tfor i, pp := range tail {\n\t\t\ttop := stack.top()\n\t\t\tparent := top.newestValue()\n\t\t\toldValue := pp.Resolve(parent, nil)\n\t\t\tvar newValue types.Value\n\t\t\tif i == len(tail)-1 { // last pathPart in this path\n\t\t\t\tnewValue = oldValue\n\t\t\t\toldValue = dif.OldValue\n\t\t\t}\n\t\t\t// Any intermediate elements on the stack will have a changeType\n\t\t\t// of modified. Leaf elements will be updated below to reflect the\n\t\t\t// actual changeType.\n\t\t\tstack.push(p, pp, types.DiffChangeModified, oldValue, newValue, dif.NewKeyValue)\n\t\t}\n\n\t\t// Update the top element in the stack with changeType from the dif and\n\t\t// the NewValue from the diff\n\t\tse := stack.top()\n\t\tse.newValue = dif.NewValue\n\t\tse.changeType = dif.ChangeType\n\t}\n\n\t// We're done applying diffs to the graph. Pop any elements left on the\n\t// stack and return the new root.\n\tvar newRoot stackElem\n\tfor stack.Len() > 0 {\n\t\tnewRoot = stack.pop()\n\t}\n\treturn newRoot.newValue\n}\n\n// updateNode handles the actual update of a node. It uses 'pp' to get the\n// information that it needs to update 'parent' with 'newVal'. 'oldVal' is also\n// passed in so that Sets can be updated correctly. This function is used by\n// the patchStack Pop() function to merge values into a new graph.\nfunc (stack *patchStack) updateNode(top *stackElem, parent types.Value) types.Value {\n\td.PanicIfTrue(parent == nil)\n\tswitch part := top.pathPart.(type) {\n\tcase types.FieldPath:\n\t\tswitch top.changeType {\n\t\tcase types.DiffChangeAdded:\n\t\t\treturn parent.(types.Struct).Set(part.Name, top.newValue)\n\t\tcase types.DiffChangeRemoved:\n\t\t\treturn parent.(types.Struct).Delete(part.Name)\n\t\tcase types.DiffChangeModified:\n\t\t\treturn parent.(types.Struct).Set(part.Name, top.newValue)\n\t\t}\n\tcase types.IndexPath:\n\t\tswitch el := parent.(type) {\n\t\tcase types.List:\n\t\t\tidx := uint64(part.Index.(types.Number))\n\t\t\toffset := stack.adjustIndexOffset(top.path, top.changeType)\n\t\t\trealIdx := idx + uint64(offset)\n\t\t\tvar nv types.Value\n\t\t\tswitch top.changeType {\n\t\t\tcase types.DiffChangeAdded:\n\t\t\t\tif realIdx > el.Len() {\n\t\t\t\t\tnv = el.Edit().Append(top.newValue).List()\n\t\t\t\t} else {\n\t\t\t\t\tnv = el.Edit().Insert(realIdx, top.newValue).List()\n\t\t\t\t}\n\t\t\tcase types.DiffChangeRemoved:\n\t\t\t\tnv = el.Edit().RemoveAt(realIdx).List()\n\t\t\tcase types.DiffChangeModified:\n\t\t\t\tnv = el.Edit().Set(realIdx, top.newValue).List()\n\t\t\t}\n\t\t\treturn nv\n\t\tcase types.Map:\n\t\t\tswitch top.changeType {\n\t\t\tcase types.DiffChangeAdded:\n\t\t\t\treturn el.Edit().Set(part.Index, top.newValue).Map()\n\t\t\tcase types.DiffChangeRemoved:\n\t\t\t\treturn el.Edit().Remove(part.Index).Map()\n\t\t\tcase types.DiffChangeModified:\n\t\t\t\tif part.IntoKey {\n\t\t\t\t\tnewPart := types.IndexPath{Index: part.Index}\n\t\t\t\t\tov := newPart.Resolve(parent, nil)\n\t\t\t\t\treturn el.Edit().Remove(part.Index).Set(top.newValue, ov).Map()\n\t\t\t\t}\n\t\t\t\treturn el.Edit().Set(part.Index, top.newValue).Map()\n\t\t\t}\n\t\tcase types.Set:\n\t\t\tif top.oldValue != nil {\n\t\t\t\tel = el.Edit().Remove(top.oldValue).Set()\n\t\t\t}\n\t\t\tif top.newValue != nil {\n\t\t\t\tel = el.Edit().Insert(top.newValue).Set()\n\t\t\t}\n\t\t\treturn el\n\t\t}\n\tcase types.HashIndexPath:\n\t\tswitch el := parent.(type) {\n\t\tcase types.Set:\n\t\t\tswitch top.changeType {\n\t\t\tcase types.DiffChangeAdded:\n\t\t\t\treturn el.Edit().Insert(top.newValue).Set()\n\t\t\tcase types.DiffChangeRemoved:\n\t\t\t\treturn el.Edit().Remove(top.oldValue).Set()\n\t\t\tcase types.DiffChangeModified:\n\t\t\t\treturn el.Edit().Remove(top.oldValue).Insert(top.newValue).Set()\n\t\t\t}\n\t\tcase types.Map:\n\t\t\tkeyPart := types.HashIndexPath{Hash: part.Hash, IntoKey: true}\n\t\t\tk := keyPart.Resolve(parent, nil)\n\t\t\tswitch top.changeType {\n\t\t\tcase types.DiffChangeAdded:\n\t\t\t\tk := top.newKeyValue\n\t\t\t\treturn el.Edit().Set(k, top.newValue).Map()\n\t\t\tcase types.DiffChangeRemoved:\n\t\t\t\treturn el.Edit().Remove(k).Map()\n\t\t\tcase types.DiffChangeModified:\n\t\t\t\tif part.IntoKey {\n\t\t\t\t\tv := el.Get(k)\n\t\t\t\t\treturn el.Edit().Remove(k).Set(top.newValue, v).Map()\n\t\t\t\t}\n\t\t\t\treturn el.Edit().Set(k, top.newValue).Map()\n\t\t\t}\n\t\t}\n\t}\n\tpanic(fmt.Sprintf(\"unreachable, pp.(type): %T\", top.pathPart))\n}\n\n// Returns the count of the number of PathParts that two paths have in a common\n// prefix. The paths '.field1' and '.field2' have a 0 length common prefix.\n// Todo: move to types.Path?\nfunc commonPrefixCount(p1, p2 types.Path) int {\n\tcnt := 0\n\n\tfor i, pp1 := range p1 {\n\t\tvar pp2 types.PathPart\n\t\tif i < len(p2) {\n\t\t\tpp2 = p2[i]\n\t\t}\n\t\tif pp1 != pp2 {\n\t\t\treturn cnt\n\t\t}\n\t\tcnt += 1\n\t}\n\treturn cnt\n}\n\ntype stackElem struct {\n\tpath types.Path\n\tpathPart types.PathPart // from parent Value to this Value\n\tchangeType types.DiffChangeType\n\toldValue types.Value // can be nil if newValue is not nil\n\tnewValue types.Value // can be nil if oldValue is not nil\n\tnewKeyValue types.Value\n}\n\n// newestValue returns newValue if not nil, otherwise oldValue. This is useful\n// when merging. Elements on the stack were 'push'ed there with the oldValue.\n// newValue may have been set when a value was 'pop'ed above it. This method\n// returns the last value that has been set.\nfunc (se stackElem) newestValue() types.Value {\n\tif se.newValue != nil {\n\t\treturn se.newValue\n\t}\n\treturn se.oldValue\n}\n\ntype patchStack struct {\n\tvals []stackElem\n\tlastPath types.Path\n\taddCnt int\n\trmCnt int\n}\n\nfunc (stack *patchStack) push(p types.Path, pp types.PathPart, changeType types.DiffChangeType, oldValue, newValue, newKeyValue types.Value) {\n\tstack.vals = append(stack.vals, stackElem{path: p, pathPart: pp, changeType: changeType, oldValue: oldValue, newValue: newValue, newKeyValue: newKeyValue})\n}\n\nfunc (stack *patchStack) top() *stackElem {\n\treturn &stack.vals[len(stack.vals)-1]\n}\n\n// pop applies the change to the graph. When an element is 'pop'ed from the stack,\n// this function uses the pathPart to merge that value into it's parent.\nfunc (stack *patchStack) pop() stackElem {\n\ttop := stack.top()\n\tstack.vals = stack.vals[:len(stack.vals)-1]\n\tif stack.Len() > 0 {\n\t\tnewTop := stack.top()\n\t\tparent := newTop.newestValue()\n\t\tnewTop.newValue = stack.updateNode(top, parent)\n\t}\n\treturn *top\n}\n\nfunc (stack *patchStack) Len() int {\n\treturn len(stack.vals)\n}\n\n// adjustIndexOffset returns an offset that needs to be added to list indexes\n// when applying diffs to lists. Diffs are applied to lists beginning at the 0th\n// element. Changes to the list mean that subsequent changes to the same list\n// have to be adjusted accordingly. The stack keeps state for each list as it's\n// processed so updateNode() can get the correct index.\n// Whenever a list is encountered, diffs consist of add & remove operations. The\n// offset is calculated by keeping a count of each add & remove. Due to the way\n// way diffs are calculated, no offset is ever needed for 'add' operations. The\n// offset for 'remove' operations are calculated as:\n// stack.addCnt - stack.rmCnt\nfunc (stack *patchStack) adjustIndexOffset(p types.Path, changeType types.DiffChangeType) (res int) {\n\tparentPath := p[:len(p)-1]\n\n\t// parentPath is different than the last parentPath so reset counters\n\tif stack.lastPath == nil || !stack.lastPath.Equals(parentPath) {\n\t\tstack.lastPath = parentPath\n\t\tstack.addCnt = 0\n\t\tstack.rmCnt = 0\n\t}\n\n\t// offset for 'Add' operations are always 0, 'Remove' ops offset are\n\t// calculated here\n\tif changeType == types.DiffChangeRemoved {\n\t\tres = stack.addCnt - stack.rmCnt\n\t}\n\n\t// Bump up the appropriate cnt for this operation.\n\tswitch changeType {\n\tcase types.DiffChangeAdded:\n\t\tstack.addCnt += 1\n\tcase types.DiffChangeRemoved:\n\t\tstack.rmCnt += 1\n\t}\n\treturn\n}\n" }, { "path": "go/diff/apply_patch_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage diff\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestCommonPrefixCount(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttestCases := [][]interface{}{\n\t\t{\".value[#94a2oa20oka0jdv5lha03vuvvumul1vb].sizes[#316j9oc39b09fbc2qf3klenm6p1o1d7h]\", 0},\n\t\t{\".value[#94a2oa20oka0jdv5lha03vuvvumul1vb].sizes[#77eavttned7llu1pkvhaei9a9qgcagir]\", 3},\n\t\t{\".value[#94a2oa20oka0jdv5lha03vuvvumul1vb].sizes[#hboaq9581drq4g9jf62d3s06al3us49s]\", 3},\n\t\t{\".value[#94a2oa20oka0jdv5lha03vuvvumul1vb].sizes[#l0hpa7sbr7qutrcfn5173kar4j2847m1]\", 3},\n\t\t{\".value[#9vj5m3049mav94bttcujhgfdfqcavsbn].sizes[#33f6tb4h8agh57s2bqlmi9vbhlkbtmct]\", 1},\n\t\t{\".value[#9vj5m3049mav94bttcujhgfdfqcavsbn].sizes[#a43ne9a8kotcqph4up5pqqdmr1e1qcsl]\", 3},\n\t\t{\".value[#9vj5m3049mav94bttcujhgfdfqcavsbn].sizes[#ppqg6pem2sb64h2i2ptnh8ckj8gogj9h]\", 3},\n\t\t{\".value[#9vj5m3049mav94bttcujhgfdfqcavsbn].sizes[#s7r2vpnqlk20sd72mg8ijerg9cmauaqo]\", 3},\n\t\t{\".value[#bpspmmlc41pk0r144a7682oah0tmge1e].sizes[#9vuc1gg3c3eude5v3j5deqopjsobe3no]\", 1},\n\t\t{\".value[#bpspmmlc41pk0r144a7682oah0tmge1e].sizes[#qo3gfdsf14v3dh0oer82vn1bg4o8nlsc]\", 3},\n\t\t{\".value[#bpspmmlc41pk0r144a7682oah0tmge1e].sizes[#rlidki5ipbjdofsm2rq3a66v908m5fpl]\", 3},\n\t\t{\".value[#bpspmmlc41pk0r144a7682oah0tmge1e].sizes[#st1n96rh89c2vgo090dt9lknd5ip4kck]\", 3},\n\t\t{\".value[#hjh5hpn55591k0gjvgckc14erli968ao].sizes[#267889uv3mtih6fij3fhio2jiqtl6nho]\", 1},\n\t\t{\".value[#hjh5hpn55591k0gjvgckc14erli968ao].sizes[#7ncb7guoip9e400bm2lcvr0dda29o9jn]\", 3},\n\t\t{\".value[#hjh5hpn55591k0gjvgckc14erli968ao].sizes[#afscb0on7rt8bq6eutup8juusmid7i96]\", 3},\n\t\t{\".value[#hjh5hpn55591k0gjvgckc14erli968ao].sizes[#drqe4lr0vdfdtmvejsjun1l3mfv6ums5]\", 3},\n\t}\n\n\tvar lastPath types.Path\n\n\tfor i, tc := range testCases {\n\t\tpath, expected := tc[0].(string), tc[1].(int)\n\t\tp, err := types.ParsePath(path)\n\t\tassert.NoError(err)\n\t\tassert.Equal(expected, commonPrefixCount(lastPath, p), \"failed for paths[%d]: %s\", i, path)\n\t\tlastPath = p\n\t}\n}\n\ntype testFunc func(parent types.Value) types.Value\ntype testKey struct {\n\tX, Y int\n}\n\nvar (\n\tvm map[string]types.Value\n)\n\nfunc vfk(keys ...string) []types.Value {\n\tvar values []types.Value\n\tfor _, k := range keys {\n\t\tvalues = append(values, vm[k])\n\t}\n\treturn values\n}\n\nfunc testValues(vrw types.ValueReadWriter) map[string]types.Value {\n\tif vm == nil {\n\t\tvm = map[string]types.Value{\n\t\t\t\"k1\": types.String(\"k1\"),\n\t\t\t\"k2\": types.String(\"k2\"),\n\t\t\t\"k3\": types.String(\"k3\"),\n\t\t\t\"s1\": types.String(\"string1\"),\n\t\t\t\"s2\": types.String(\"string2\"),\n\t\t\t\"s3\": types.String(\"string3\"),\n\t\t\t\"s4\": types.String(\"string4\"),\n\t\t\t\"n1\": types.Number(1),\n\t\t\t\"n2\": types.Number(2),\n\t\t\t\"n3\": types.Number(3.3),\n\t\t\t\"n4\": types.Number(4.4),\n\t\t\t\"b1\": mustMarshal(true),\n\t\t\t\"b2\": mustMarshal(false),\n\t\t\t\"l1\": mustMarshal([]string{}),\n\t\t\t\"l2\": mustMarshal([]string{\"one\", \"two\", \"three\", \"four\"}),\n\t\t\t\"l3\": mustMarshal([]string{\"two\", \"three\", \"four\", \"five\"}),\n\t\t\t\"l4\": mustMarshal([]string{\"two\", \"three\", \"four\"}),\n\t\t\t\"l5\": mustMarshal([]string{\"one\", \"two\", \"three\", \"four\", \"five\"}),\n\t\t\t\"l6\": mustMarshal([]string{\"one\", \"four\"}),\n\t\t\t\"struct1\": types.NewStruct(\"test1\", types.StructData{\"f1\": types.Number(1), \"f2\": types.Number(2)}),\n\t\t\t\"struct2\": types.NewStruct(\"test1\", types.StructData{\"f1\": types.Number(11111), \"f2\": types.Number(2)}),\n\t\t\t\"struct3\": types.NewStruct(\"test1\", types.StructData{\"f1\": types.Number(1), \"f2\": types.Number(2), \"f3\": types.Number(3)}),\n\t\t\t\"struct4\": types.NewStruct(\"test1\", types.StructData{\"f2\": types.Number(2)}),\n\t\t\t\"m1\": mustMarshal(map[string]int{}),\n\t\t\t\"m2\": mustMarshal(map[string]int{\"k1\": 1, \"k2\": 2, \"k3\": 3}),\n\t\t\t\"m3\": mustMarshal(map[string]int{\"k2\": 2, \"k3\": 3, \"k4\": 4}),\n\t\t\t\"m4\": mustMarshal(map[string]int{\"k1\": 1, \"k3\": 3}),\n\t\t\t\"m5\": mustMarshal(map[string]int{\"k1\": 1, \"k2\": 2222, \"k3\": 3}),\n\t\t\t\"ms1\": mustMarshal(map[testKey]int{{1, 1}: 1, {2, 2}: 2, {3, 3}: 3}),\n\t\t\t\"ms2\": mustMarshal(map[testKey]int{{1, 1}: 1, {4, 4}: 4, {5, 5}: 5}),\n\t\t}\n\n\t\tvm[\"mh1\"] = types.NewMap(vrw, vfk(\"k1\", \"struct1\", \"k2\", \"l1\")...)\n\t\tvm[\"mh2\"] = types.NewMap(vrw, vfk(\"k1\", \"n1\", \"k2\", \"l2\", \"k3\", \"l3\")...)\n\t\tvm[\"set1\"] = types.NewSet(vrw)\n\t\tvm[\"set2\"] = types.NewSet(vrw, vfk(\"s1\", \"s2\")...)\n\t\tvm[\"set3\"] = types.NewSet(vrw, vfk(\"s1\", \"s2\", \"s3\")...)\n\t\tvm[\"set1\"] = types.NewSet(vrw, vfk(\"s2\")...)\n\t\tvm[\"seth1\"] = types.NewSet(vrw, vfk(\"struct1\", \"struct2\", \"struct3\")...)\n\t\tvm[\"seth2\"] = types.NewSet(vrw, vfk(\"struct2\", \"struct3\")...)\n\t\tvm[\"setj3\"] = types.NewSet(vrw, vfk(\"struct1\")...)\n\t\tvm[\"mk1\"] = types.NewMap(vrw, vfk(\"struct1\", \"s1\", \"struct2\", \"s2\")...)\n\t\tvm[\"mk2\"] = types.NewMap(vrw, vfk(\"struct1\", \"s3\", \"struct4\", \"s4\")...)\n\t}\n\treturn vm\n}\n\nfunc newTestValueStore() *types.ValueStore {\n\tst := &chunks.TestStorage{}\n\treturn types.NewValueStore(st.NewView())\n}\n\nfunc getPatch(g1, g2 types.Value) Patch {\n\tdChan := make(chan Difference)\n\tsChan := make(chan struct{})\n\tgo func() {\n\t\tDiff(g1, g2, dChan, sChan, true)\n\t\tclose(dChan)\n\t}()\n\n\tpatch := Patch{}\n\tfor dif := range dChan {\n\t\tpatch = append(patch, dif)\n\t}\n\treturn patch\n}\n\nfunc checkApplyPatch(assert *assert.Assertions, g1, expectedG2 types.Value, k1, k2 string) {\n\tpatch := getPatch(g1, expectedG2)\n\tg2 := Apply(g1, patch)\n\tassert.True(expectedG2.Equals(g2), \"failed to apply diffs for k1: %s and k2: %s\", k1, k2)\n}\n\nfunc TestPatches(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tcnt := 0\n\tfor k1, g1 := range testValues(vs) {\n\t\tfor k2, expectedG2 := range testValues(vs) {\n\t\t\tif k1 != k2 {\n\t\t\t\tcnt++\n\t\t\t\tcheckApplyPatch(assert, g1, expectedG2, k1, k2)\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc TestNestedLists(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tol1 := mustMarshal([]string{\"one\", \"two\", \"three\", \"four\"})\n\tnl1 := mustMarshal([]string{\"two\", \"three\"})\n\tol2 := mustMarshal([]int{2, 3})\n\tnl2 := mustMarshal([]int{1, 2, 3, 4})\n\tnl3 := mustMarshal([]bool{true, false, true})\n\tg1 := types.NewList(vs, ol1, ol2)\n\tg2 := types.NewList(vs, nl1, nl2, nl3)\n\tcheckApplyPatch(assert, g1, g2, \"g1\", \"g2\")\n}\n\nfunc TestUpdateNode(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tdoTest := func(pp types.PathPart, parent, ov, nv, exp types.Value, f testFunc) {\n\t\tstack := &patchStack{}\n\t\tse := &stackElem{path: []types.PathPart{pp}, pathPart: pp, changeType: types.DiffChangeModified, oldValue: ov, newValue: nv}\n\t\tupdated := stack.updateNode(se, parent)\n\t\ttestVal := f(updated)\n\t\tassert.True(exp.Equals(testVal), \"%s != %s\", nv, testVal)\n\t}\n\n\tvar pp types.PathPart\n\toldVal := types.String(\"Yo\")\n\tnewVal := types.String(\"YooHoo\")\n\n\ts1 := types.NewStruct(\"TestStruct\", types.StructData{\"f1\": types.Number(1), \"f2\": oldVal})\n\tpp = types.FieldPath{Name: \"f2\"}\n\tdoTest(pp, s1, oldVal, newVal, newVal, func(parent types.Value) types.Value {\n\t\treturn parent.(types.Struct).Get(\"f2\")\n\t})\n\n\tl1 := types.NewList(vs, types.String(\"one\"), oldVal, types.String(\"three\"))\n\tpp = types.IndexPath{Index: types.Number(1)}\n\tdoTest(pp, l1, oldVal, newVal, newVal, func(parent types.Value) types.Value {\n\t\treturn parent.(types.List).Get(1)\n\t})\n\n\tm1 := types.NewMap(vs, types.String(\"k1\"), types.Number(1), types.String(\"k2\"), oldVal)\n\tpp = types.IndexPath{Index: types.String(\"k2\")}\n\tdoTest(pp, m1, oldVal, newVal, newVal, func(parent types.Value) types.Value {\n\t\treturn parent.(types.Map).Get(types.String(\"k2\"))\n\t})\n\n\tk1 := types.NewStruct(\"Sizes\", types.StructData{\"height\": types.Number(200), \"width\": types.Number(300)})\n\tvs.WriteValue(k1)\n\tm1 = types.NewMap(vs, k1, oldVal)\n\tpp = types.HashIndexPath{Hash: k1.Hash()}\n\tdoTest(pp, m1, oldVal, newVal, newVal, func(parent types.Value) types.Value {\n\t\treturn parent.(types.Map).Get(k1)\n\t})\n\n\tset1 := types.NewSet(vs, oldVal, k1)\n\tpp = types.IndexPath{Index: oldVal}\n\texp := types.NewSet(vs, newVal, k1)\n\tdoTest(pp, set1, oldVal, newVal, exp, func(parent types.Value) types.Value {\n\t\treturn parent\n\t})\n\n\tk2 := types.NewStruct(\"Sizes\", types.StructData{\"height\": types.Number(300), \"width\": types.Number(500)})\n\tset1 = types.NewSet(vs, oldVal, k1)\n\tpp = types.HashIndexPath{Hash: k1.Hash()}\n\texp = types.NewSet(vs, oldVal, k2)\n\tdoTest(pp, set1, k1, k2, exp, func(parent types.Value) types.Value {\n\t\treturn parent\n\t})\n}\n\nfunc checkApplyDiffs(a *assert.Assertions, n1, n2 types.Value, leftRight bool) {\n\tdChan := make(chan Difference)\n\tsChan := make(chan struct{})\n\tgo func() {\n\t\tDiff(n1, n2, dChan, sChan, leftRight)\n\t\tclose(dChan)\n\t}()\n\n\tdifs := Patch{}\n\tfor dif := range dChan {\n\t\tdifs = append(difs, dif)\n\t}\n\n\tres := Apply(n1, difs)\n\ta.True(n2.Equals(res))\n}\n\nfunc tryApplyDiff(a *assert.Assertions, a1, a2 interface{}) {\n\tn1 := mustMarshal(a1)\n\tn2 := mustMarshal(a2)\n\n\tcheckApplyDiffs(a, n1, n2, true)\n\tcheckApplyDiffs(a, n1, n2, false)\n\tcheckApplyDiffs(a, n2, n1, true)\n\tcheckApplyDiffs(a, n2, n1, false)\n}\n\nfunc TestUpdateList(t *testing.T) {\n\ta := assert.New(t)\n\n\t// insert at beginning\n\ta1 := []interface{}{\"five\", \"ten\", \"fifteen\"}\n\ta2 := []interface{}{\"one\", \"two\", \"three\", \"five\", \"ten\", \"fifteen\"}\n\ttryApplyDiff(a, a1, a2)\n\n\t// append at end\n\ta1 = []interface{}{\"five\", \"ten\", \"fifteen\"}\n\ta2 = []interface{}{\"five\", \"ten\", \"fifteen\", \"twenty\", \"twenty-five\"}\n\ttryApplyDiff(a, a1, a2)\n\n\t// insert interleaved\n\ta1 = []interface{}{\"one\", \"three\", \"five\", \"seven\"}\n\ta2 = []interface{}{\"one\", \"two\", \"three\", \"four\", \"five\", \"six\", \"seven\"}\n\ttryApplyDiff(a, a1, a2)\n\n\t// delete from beginning and append to end\n\ta1 = []interface{}{\"one\", \"two\", \"three\", \"four\", \"five\"}\n\ta2 = []interface{}{\"four\", \"five\", \"six\", \"seven\"}\n\ttryApplyDiff(a, a1, a2)\n\n\t// replace entries at beginning\n\ta1 = []interface{}{\"one\", \"two\", \"three\", \"four\", \"five\"}\n\ta2 = []interface{}{\"3.5\", \"four\", \"five\"}\n\ttryApplyDiff(a, a1, a2)\n\n\t// replace entries at end\n\ta1 = []interface{}{\"one\", \"two\", \"three\"}\n\ta2 = []interface{}{\"one\", \"four\"}\n\ttryApplyDiff(a, a1, a2)\n\n\t// insert at beginning, replace at end\n\ta1 = []interface{}{\"five\", \"ten\", \"fifteen\"}\n\ta2 = []interface{}{\"one\", \"two\", \"five\", \"eight\", \"eleven\", \"sixteen\", \"twenty\"}\n\ttryApplyDiff(a, a1, a2)\n\n\t// remove everything\n\ta1 = []interface{}{\"five\", \"ten\", \"fifteen\"}\n\ta2 = []interface{}{}\n\ttryApplyDiff(a, a1, a2)\n}\n\nfunc TestUpdateMap(t *testing.T) {\n\ta := assert.New(t)\n\n\t// insertions, deletions, and replacements\n\ta1 := map[string]int{\"five\": 5, \"ten\": 10, \"fifteen\": 15, \"twenty\": 20}\n\ta2 := map[string]int{\"one\": 1, \"two\": 2, \"three\": 3, \"five\": 5, \"ten\": 10, \"fifteen\": 15, \"twenty\": 2020}\n\ttryApplyDiff(a, a1, a2)\n\n\t// delete everything\n\ta1 = map[string]int{\"five\": 5, \"ten\": 10, \"fifteen\": 15, \"twenty\": 20}\n\ta2 = map[string]int{}\n\ttryApplyDiff(a, a1, a2)\n}\n\nfunc TestUpdateStruct(t *testing.T) {\n\ta := assert.New(t)\n\n\ta1 := types.NewStruct(\"tStruct\", types.StructData{\n\t\t\"f1\": types.Number(1),\n\t\t\"f2\": types.String(\"two\"),\n\t\t\"f3\": mustMarshal([]string{\"one\", \"two\", \"three\"}),\n\t})\n\ta2 := types.NewStruct(\"tStruct\", types.StructData{\n\t\t\"f1\": types.Number(2),\n\t\t\"f2\": types.String(\"twotwo\"),\n\t\t\"f3\": mustMarshal([]interface{}{0, \"one\", 1, \"two\", 2, \"three\", 3}),\n\t})\n\tcheckApplyDiffs(a, a1, a2, true)\n\tcheckApplyDiffs(a, a1, a2, false)\n\n\ta2 = types.NewStruct(\"tStruct\", types.StructData{\n\t\t\"f1\": types.Number(2),\n\t\t\"f2\": types.String(\"two\"),\n\t\t\"f3\": mustMarshal([]interface{}{0, \"one\", 1, \"two\", 2, \"three\", 3}),\n\t\t\"f4\": types.Bool(true),\n\t})\n\tcheckApplyDiffs(a, a1, a2, true)\n\tcheckApplyDiffs(a, a1, a2, false)\n}\n\nfunc TestUpdateSet(t *testing.T) {\n\ta := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ta1 := types.NewSet(vs, types.Number(1), types.String(\"two\"), mustMarshal([]string{\"one\", \"two\", \"three\"}))\n\ta2 := types.NewSet(vs, types.Number(3), types.String(\"three\"), mustMarshal([]string{\"one\", \"two\", \"three\", \"four\"}))\n\n\tcheckApplyDiffs(a, a1, a2, true)\n\tcheckApplyDiffs(a, a1, a2, false)\n\tcheckApplyDiffs(a, a2, a1, true)\n\tcheckApplyDiffs(a, a2, a1, false)\n}\n\nfunc mustMarshal(v interface{}) types.Value {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tv1, err := marshal.Marshal(vs, v)\n\td.Chk.NoError(err)\n\treturn v1\n}\n" }, { "path": "go/diff/diff.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage diff\n\nimport (\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\ntype (\n\tdiffFunc func(changeChan chan<- types.ValueChanged, stopChan <-chan struct{})\n\tpathPartFunc func(v types.Value) types.PathPart\n\tvalueFunc func(k types.Value) types.Value\n)\n\n// Difference represents a \"diff\" between two Noms graphs.\ntype Difference struct {\n\t// Path to the Value that has changed\n\tPath types.Path\n\t// ChangeType indicates the type of diff: modified, added, deleted\n\tChangeType types.DiffChangeType\n\t// OldValue is Value before the change, can be nil if Value was added\n\tOldValue types.Value\n\t// NewValue is Value after the change, can be nil if Value was removed\n\tNewValue types.Value\n\t// NewKeyValue is used for when elements are added to diffs with a\n\t// non-primitive key. The new key must available when the map gets updated.\n\tNewKeyValue types.Value\n}\n\nfunc (dif Difference) IsEmpty() bool {\n\treturn dif.Path == nil && dif.OldValue == nil && dif.NewValue == nil\n}\n\n// differ is used internally to hold information necessary for diffing two graphs.\ntype differ struct {\n\t// Channel used to send Difference objects back to caller\n\tdiffChan chan<- Difference\n\t// Channel that caller should close() to terminate Diff function.\n\tstopChan chan struct{}\n\t// Use LeftRight diff as opposed to TopDown\n\tleftRight bool\n}\n\n// Diff traverses two graphs simultaneously looking for differences. It returns\n// two channels: a DiffReceiveChan that the caller can use to iterate over the\n// diffs in the graph and a StopSendChanel that a caller can use to signal the\n// Diff function to stop processing.\n// Diff returns the Differences in depth-first first order. A 'diff' is defined\n// as one of the following conditions:\n// * a Value is Added or Removed from a node in the graph\n// * the type of a Value has changed in the graph\n// * a primitive (i.e. Bool, Number, String, Ref or Blob) Value has changed\n//\n// A Difference is not returned when a non-primitive value has been modified. For\n// example, a struct field has been changed from one Value of type Employee to\n// another. Those modifications are accounted for by the Differences described\n// above at a lower point in the graph.\n//\n// If leftRight is true then the left-right diff is used for ordered sequences\n// - see Diff vs DiffLeftRight in Set and Map.\n//\n// Note: the function sends messages on diffChan and checks whether stopChan has\n// been closed to know if it needs to terminate diffing early. To function\n// properly it needs to be executed concurrently with code that reads values from\n// diffChan. The following is a typical invocation of Diff():\n// dChan := make(chan Difference)\n// sChan := make(chan struct{})\n// go func() {\n// d.Diff(s3, s4, dChan, sChan, leftRight)\n// close(dChan)\n// }()\n// for dif := range dChan {\n// \n// }\nfunc Diff(v1, v2 types.Value, dChan chan<- Difference, stopChan chan struct{}, leftRight bool) {\n\td := differ{diffChan: dChan, stopChan: stopChan, leftRight: leftRight}\n\tif !v1.Equals(v2) {\n\t\tif !shouldDescend(v1, v2) {\n\t\t\td.sendDiff(Difference{Path: nil, ChangeType: types.DiffChangeModified, OldValue: v1, NewValue: v2})\n\t\t} else {\n\t\t\td.diff(nil, v1, v2)\n\t\t}\n\t}\n}\n\nfunc (d differ) diff(p types.Path, v1, v2 types.Value) bool {\n\tswitch v1.Kind() {\n\tcase types.ListKind:\n\t\treturn d.diffLists(p, v1.(types.List), v2.(types.List))\n\tcase types.MapKind:\n\t\treturn d.diffMaps(p, v1.(types.Map), v2.(types.Map))\n\tcase types.SetKind:\n\t\treturn d.diffSets(p, v1.(types.Set), v2.(types.Set))\n\tcase types.StructKind:\n\t\treturn d.diffStructs(p, v1.(types.Struct), v2.(types.Struct))\n\tdefault:\n\t\tpanic(\"Unrecognized type in diff function\")\n\t}\n}\n\nfunc (d differ) diffLists(p types.Path, v1, v2 types.List) (stop bool) {\n\tspliceChan := make(chan types.Splice)\n\tstopChan := make(chan struct{}, 1) // buffer size of 1s, so this won't block if diff already finished\n\n\tgo func() {\n\t\tv2.Diff(v1, spliceChan, stopChan)\n\t\tclose(spliceChan)\n\t}()\n\n\tfor splice := range spliceChan {\n\t\tif stop {\n\t\t\tbreak\n\t\t}\n\t\tif splice.SpRemoved == splice.SpAdded {\n\t\t\t// Heuristic: list only has modifications.\n\t\t\tfor i := uint64(0); i < splice.SpRemoved; i++ {\n\t\t\t\tlastEl := v1.Get(splice.SpAt + i)\n\t\t\t\tnewEl := v2.Get(splice.SpFrom + i)\n\t\t\t\tif shouldDescend(lastEl, newEl) {\n\t\t\t\t\tidx := types.Number(splice.SpAt + i)\n\t\t\t\t\tstop = d.diff(append(p, types.NewIndexPath(idx)), lastEl, newEl)\n\t\t\t\t} else {\n\t\t\t\t\tp1 := p.Append(types.NewIndexPath(types.Number(splice.SpAt + i)))\n\t\t\t\t\tdif := Difference{p1, types.DiffChangeModified, v1.Get(splice.SpAt + i), v2.Get(splice.SpFrom + i), nil}\n\t\t\t\t\tstop = !d.sendDiff(dif)\n\t\t\t\t}\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\t// Heuristic: list only has additions/removals.\n\t\tfor i := uint64(0); i < splice.SpRemoved && !stop; i++ {\n\t\t\tp1 := p.Append(types.NewIndexPath(types.Number(splice.SpAt + i)))\n\t\t\tdif := Difference{Path: p1, ChangeType: types.DiffChangeRemoved, OldValue: v1.Get(splice.SpAt + i), NewValue: nil}\n\t\t\tstop = !d.sendDiff(dif)\n\t\t}\n\t\tfor i := uint64(0); i < splice.SpAdded && !stop; i++ {\n\t\t\tp1 := p.Append(types.NewIndexPath(types.Number(splice.SpFrom + i)))\n\t\t\tdif := Difference{Path: p1, ChangeType: types.DiffChangeAdded, OldValue: nil, NewValue: v2.Get(splice.SpFrom + i)}\n\t\t\tstop = !d.sendDiff(dif)\n\t\t}\n\t}\n\n\tif stop {\n\t\tstopChan <- struct{}{}\n\t\t// Wait for diff to stop.\n\t\tfor range spliceChan {\n\t\t}\n\t}\n\treturn\n}\n\nfunc (d differ) diffMaps(p types.Path, v1, v2 types.Map) bool {\n\treturn d.diffOrdered(p,\n\t\tfunc(v types.Value) types.PathPart {\n\t\t\tif types.ValueCanBePathIndex(v) {\n\t\t\t\treturn types.NewIndexPath(v)\n\t\t\t} else {\n\t\t\t\treturn types.NewHashIndexPath(v.Hash())\n\t\t\t}\n\t\t},\n\t\tfunc(cc chan<- types.ValueChanged, sc <-chan struct{}) {\n\t\t\tif d.leftRight {\n\t\t\t\tv2.DiffLeftRight(v1, cc, sc)\n\t\t\t} else {\n\t\t\t\tv2.DiffHybrid(v1, cc, sc)\n\t\t\t}\n\t\t},\n\t\tfunc(k types.Value) types.Value { return k },\n\t\tfunc(k types.Value) types.Value { return v1.Get(k) },\n\t\tfunc(k types.Value) types.Value { return v2.Get(k) },\n\t)\n}\n\nfunc (d differ) diffStructs(p types.Path, v1, v2 types.Struct) bool {\n\tstr := func(v types.Value) string {\n\t\treturn string(v.(types.String))\n\t}\n\treturn d.diffOrdered(p,\n\t\tfunc(v types.Value) types.PathPart { return types.NewFieldPath(str(v)) },\n\t\tfunc(cc chan<- types.ValueChanged, sc <-chan struct{}) {\n\t\t\tv2.Diff(v1, cc, sc)\n\t\t},\n\t\tfunc(k types.Value) types.Value { return k },\n\t\tfunc(k types.Value) types.Value { return v1.Get(str(k)) },\n\t\tfunc(k types.Value) types.Value { return v2.Get(str(k)) },\n\t)\n}\n\nfunc (d differ) diffSets(p types.Path, v1, v2 types.Set) bool {\n\treturn d.diffOrdered(p,\n\t\tfunc(v types.Value) types.PathPart {\n\t\t\tif types.ValueCanBePathIndex(v) {\n\t\t\t\treturn types.NewIndexPath(v)\n\t\t\t}\n\t\t\treturn types.NewHashIndexPath(v.Hash())\n\t\t},\n\t\tfunc(cc chan<- types.ValueChanged, sc <-chan struct{}) {\n\t\t\tif d.leftRight {\n\t\t\t\tv2.DiffLeftRight(v1, cc, sc)\n\t\t\t} else {\n\t\t\t\tv2.DiffHybrid(v1, cc, sc)\n\t\t\t}\n\t\t},\n\t\tfunc(k types.Value) types.Value { return k },\n\t\tfunc(k types.Value) types.Value { return k },\n\t\tfunc(k types.Value) types.Value { return k },\n\t)\n}\n\nfunc (d differ) diffOrdered(p types.Path, ppf pathPartFunc, df diffFunc, kf, v1, v2 valueFunc) (stop bool) {\n\tchangeChan := make(chan types.ValueChanged)\n\tstopChan := make(chan struct{}, 1) // buffer size of 1, so this won't block if diff already finished\n\n\tgo func() {\n\t\tdf(changeChan, stopChan)\n\t\tclose(changeChan)\n\t}()\n\n\tfor change := range changeChan {\n\t\tif stop {\n\t\t\tbreak\n\t\t}\n\n\t\tk := kf(change.Key)\n\t\tp1 := p.Append(ppf(k))\n\n\t\tswitch change.ChangeType {\n\t\tcase types.DiffChangeAdded:\n\t\t\tdif := Difference{Path: p1, ChangeType: types.DiffChangeAdded, OldValue: nil, NewValue: v2(change.Key), NewKeyValue: k}\n\t\t\tstop = !d.sendDiff(dif)\n\t\tcase types.DiffChangeRemoved:\n\t\t\tdif := Difference{Path: p1, ChangeType: types.DiffChangeRemoved, OldValue: v1(change.Key), NewValue: nil}\n\t\t\tstop = !d.sendDiff(dif)\n\t\tcase types.DiffChangeModified:\n\t\t\tc1, c2 := v1(change.Key), v2(change.Key)\n\t\t\tif shouldDescend(c1, c2) {\n\t\t\t\tstop = d.diff(p1, c1, c2)\n\t\t\t} else {\n\t\t\t\tdif := Difference{Path: p1, ChangeType: types.DiffChangeModified, OldValue: c1, NewValue: c2}\n\t\t\t\tstop = !d.sendDiff(dif)\n\t\t\t}\n\t\tdefault:\n\t\t\tpanic(\"unknown change type\")\n\t\t}\n\t}\n\n\tif stop {\n\t\tstopChan <- struct{}{}\n\t\tfor range changeChan {\n\t\t}\n\t}\n\n\treturn\n}\n\n// shouldDescend returns true, if Value is not primitive or is a Ref.\nfunc shouldDescend(v1, v2 types.Value) bool {\n\tkind := v1.Kind()\n\treturn !types.IsPrimitiveKind(kind) && kind == v2.Kind() && kind != types.RefKind\n}\n\n// stopSent returns true if a message has been sent to this StopChannel\nfunc (d differ) sendDiff(dif Difference) bool {\n\tselect {\n\tcase <-d.stopChan:\n\t\treturn false\n\tcase d.diffChan <- dif:\n\t\treturn true\n\t}\n}\n" }, { "path": "go/diff/diff_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage diff\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/test\"\n\t\"github.com/attic-labs/noms/go/util/writers\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nvar (\n\taa1 = createMap(\"a1\", \"a-one\", \"a2\", \"a-two\", \"a3\", \"a-three\", \"a4\", \"a-four\")\n\taa1x = createMap(\"a1\", \"a-one-diff\", \"a2\", \"a-two\", \"a3\", \"a-three\", \"a4\", \"a-four\")\n\n\tmm1 = createMap(\"k1\", \"k-one\", \"k2\", \"k-two\", \"k3\", \"k-three\", \"k4\", aa1)\n\tmm2 = createMap(\"l1\", \"l-one\", \"l2\", \"l-two\", \"l3\", \"l-three\", \"l4\", aa1)\n\tmm3 = createMap(\"m1\", \"m-one\", \"v2\", \"m-two\", \"m3\", \"m-three\", \"m4\", aa1)\n\tmm3x = createMap(\"m1\", \"m-one\", \"v2\", \"m-two\", \"m3\", \"m-three-diff\", \"m4\", aa1x)\n\tmm4 = createMap(\"n1\", \"n-one\", \"n2\", \"n-two\", \"n3\", \"n-three\", \"n4\", aa1)\n)\n\nfunc valToTypesValue(v interface{}) types.Value {\n\tvar v1 types.Value\n\tswitch t := v.(type) {\n\tcase string:\n\t\tv1 = types.String(t)\n\tcase int:\n\t\tv1 = types.Number(t)\n\tcase types.Value:\n\t\tv1 = t\n\t}\n\treturn v1\n}\n\nfunc valsToTypesValues(kv ...interface{}) []types.Value {\n\tkeyValues := []types.Value{}\n\tfor _, e := range kv {\n\t\tv := valToTypesValue(e)\n\t\tkeyValues = append(keyValues, v)\n\t}\n\treturn keyValues\n}\n\nfunc createMap(kv ...interface{}) types.Map {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\tkeyValues := valsToTypesValues(kv...)\n\treturn types.NewMap(vs, keyValues...)\n}\n\nfunc createSet(kv ...interface{}) types.Set {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\tkeyValues := valsToTypesValues(kv...)\n\treturn types.NewSet(vs, keyValues...)\n}\n\nfunc createList(kv ...interface{}) types.List {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\tkeyValues := valsToTypesValues(kv...)\n\treturn types.NewList(vs, keyValues...)\n}\n\nfunc createStruct(name string, kv ...interface{}) types.Struct {\n\tfields := types.StructData{}\n\tfor i := 0; i < len(kv); i += 2 {\n\t\tfields[kv[i].(string)] = valToTypesValue(kv[i+1])\n\t}\n\treturn types.NewStruct(name, fields)\n}\n\nfunc pathsFromDiff(v1, v2 types.Value, leftRight bool) []string {\n\tdChan := make(chan Difference)\n\tsChan := make(chan struct{})\n\n\tgo func() {\n\t\tDiff(v1, v2, dChan, sChan, leftRight)\n\t\tclose(dChan)\n\t}()\n\n\tvar paths []string\n\tfor d := range dChan {\n\t\tpaths = append(paths, d.Path.String())\n\t}\n\treturn paths\n}\n\nfunc mustParsePath(assert *assert.Assertions, s string) types.Path {\n\tif s == \"\" {\n\t\treturn nil\n\t}\n\tp, err := types.ParsePath(s)\n\tassert.NoError(err)\n\treturn p\n}\n\nfunc TestNomsDiffPrintMap(t *testing.T) {\n\tassert := assert.New(t)\n\texpected := `[\"map-3\"] {\n- \"m3\": \"m-three\"\n+ \"m3\": \"m-three-diff\"\n }\n[\"map-3\"][\"m4\"] {\n- \"a1\": \"a-one\"\n+ \"a1\": \"a-one-diff\"\n }\n`\n\texpectedPaths := []string{\n\t\t`[\"map-3\"][\"m3\"]`,\n\t\t`[\"map-3\"][\"m4\"][\"a1\"]`,\n\t}\n\n\ttf := func(leftRight bool) {\n\t\tm1 := createMap(\"map-1\", mm1, \"map-2\", mm2, \"map-3\", mm3, \"map-4\", mm4)\n\t\tm2 := createMap(\"map-1\", mm1, \"map-2\", mm2, \"map-3\", mm3x, \"map-4\", mm4)\n\t\tbuf := &bytes.Buffer{}\n\t\tPrintDiff(buf, m1, m2, leftRight)\n\t\tassert.Equal(expected, buf.String())\n\n\t\tpaths := pathsFromDiff(m1, m2, leftRight)\n\t\tassert.Equal(expectedPaths, paths)\n\t}\n\n\ttf(true)\n\ttf(false)\n}\n\nfunc TestNomsDiffPrintSet(t *testing.T) {\n\tassert := assert.New(t)\n\n\texpected1 := `(root) {\n- \"five\"\n+ \"five-diff\"\n }\n`\n\texpectedPaths1 := []string{\n\t\t`[\"five\"]`,\n\t\t`[\"five-diff\"]`,\n\t}\n\n\texpected2 := `(root) {\n- map { // 4 items\n- \"m1\": \"m-one\",\n- \"m3\": \"m-three\",\n- \"m4\": map { // 4 items\n- \"a1\": \"a-one\",\n- \"a2\": \"a-two\",\n- \"a3\": \"a-three\",\n- \"a4\": \"a-four\",\n- },\n- \"v2\": \"m-two\",\n- }\n+ map { // 4 items\n+ \"m1\": \"m-one\",\n+ \"m3\": \"m-three-diff\",\n+ \"m4\": map { // 4 items\n+ \"a1\": \"a-one-diff\",\n+ \"a2\": \"a-two\",\n+ \"a3\": \"a-three\",\n+ \"a4\": \"a-four\",\n+ },\n+ \"v2\": \"m-two\",\n+ }\n }\n`\n\texpectedPaths2 := []string{\n\t\tfmt.Sprintf(\"[#%s]\", mm3.Hash()),\n\t\tfmt.Sprintf(\"[#%s]\", mm3x.Hash()),\n\t}\n\n\ts1 := createSet(\"one\", \"three\", \"five\", \"seven\", \"nine\")\n\ts2 := createSet(\"one\", \"three\", \"five-diff\", \"seven\", \"nine\")\n\ts3 := createSet(mm1, mm2, mm3, mm4)\n\ts4 := createSet(mm1, mm2, mm3x, mm4)\n\n\ttf := func(leftRight bool) {\n\t\tbuf := &bytes.Buffer{}\n\t\tPrintDiff(buf, s1, s2, leftRight)\n\t\tassert.Equal(expected1, buf.String())\n\n\t\tpaths := pathsFromDiff(s1, s2, leftRight)\n\t\tassert.Equal(expectedPaths1, paths)\n\n\t\tbuf = &bytes.Buffer{}\n\t\tPrintDiff(buf, s3, s4, leftRight)\n\t\tassert.Equal(expected2, buf.String())\n\n\t\tpaths = pathsFromDiff(s3, s4, leftRight)\n\t\tassert.Equal(expectedPaths2, paths)\n\t}\n\n\ttf(true)\n\ttf(false)\n}\n\n// This function tests stop functionality in PrintDiff and Diff.\nfunc TestNomsDiffPrintStop(t *testing.T) {\n\tassert := assert.New(t)\n\n\texpected1 := `(root) {\n- \"five\"\n`\n\n\texpected2 := `(root) {\n- map { // 4 items\n`\n\n\ts1 := createSet(\"one\", \"three\", \"five\", \"seven\", \"nine\")\n\ts2 := createSet(\"one\", \"three\", \"five-diff\", \"seven\", \"nine\")\n\ts3 := createSet(mm1, mm2, mm3, mm4)\n\ts4 := createSet(mm1, mm2, mm3x, mm4)\n\n\ttf := func(leftRight bool) {\n\t\tbuf := &bytes.Buffer{}\n\t\tmlw := &writers.MaxLineWriter{Dest: buf, MaxLines: 2}\n\t\tPrintDiff(mlw, s1, s2, leftRight)\n\t\tassert.Equal(expected1, buf.String())\n\n\t\tbuf = &bytes.Buffer{}\n\t\tmlw = &writers.MaxLineWriter{Dest: buf, MaxLines: 2}\n\t\tPrintDiff(mlw, s3, s4, leftRight)\n\t\tassert.Equal(expected2, buf.String())\n\t}\n\n\ttf(true)\n\ttf(false)\n}\n\nfunc TestNomsDiffPrintStruct(t *testing.T) {\n\tassert := assert.New(t)\n\n\texpected1 := `(root) {\n- \"four\": \"four\"\n+ \"four\": \"four-diff\"\n }\n[\"three\"] {\n- field1: \"field1-data\"\n- field3: \"field3-data\"\n+ field3: \"field3-data-diff\"\n+ field4: \"field4-data\"\n }\n`\n\texpectedPaths1 := []string{\n\t\t`[\"four\"]`,\n\t\t`[\"three\"].field1`,\n\t\t`[\"three\"].field3`,\n\t\t`[\"three\"].field4`,\n\t}\n\n\texpected2 := `(root) {\n- four: \"four\"\n+ four: \"four-diff\"\n }\n.three {\n- field1: \"field1-data\"\n- field3: \"field3-data\"\n+ field3: \"field3-data-diff\"\n+ field4: \"field4-data\"\n }\n`\n\texpectedPaths2 := []string{\n\t\t`.four`,\n\t\t`.three.field1`,\n\t\t`.three.field3`,\n\t\t`.three.field4`,\n\t}\n\n\ts1 := createStruct(\"TestData\",\n\t\t\"field1\", \"field1-data\",\n\t\t\"field2\", \"field2-data\",\n\t\t\"field3\", \"field3-data\",\n\t)\n\ts2 := createStruct(\"TestData\",\n\t\t\"field2\", \"field2-data\",\n\t\t\"field3\", \"field3-data-diff\",\n\t\t\"field4\", \"field4-data\",\n\t)\n\n\tm1 := createMap(\"one\", 1, \"two\", 2, \"three\", s1, \"four\", \"four\")\n\tm2 := createMap(\"one\", 1, \"two\", 2, \"three\", s2, \"four\", \"four-diff\")\n\n\ts3 := createStruct(\"\", \"one\", 1, \"two\", 2, \"three\", s1, \"four\", \"four\")\n\ts4 := createStruct(\"\", \"one\", 1, \"two\", 2, \"three\", s2, \"four\", \"four-diff\")\n\n\ttf := func(leftRight bool) {\n\t\tbuf := &bytes.Buffer{}\n\t\tPrintDiff(buf, m1, m2, leftRight)\n\t\tassert.Equal(expected1, buf.String())\n\n\t\tpaths := pathsFromDiff(m1, m2, leftRight)\n\t\tassert.Equal(expectedPaths1, paths)\n\n\t\tbuf = &bytes.Buffer{}\n\t\tPrintDiff(buf, s3, s4, leftRight)\n\t\tassert.Equal(expected2, buf.String())\n\n\t\tpaths = pathsFromDiff(s3, s4, leftRight)\n\t\tassert.Equal(expectedPaths2, paths)\n\t}\n\n\ttf(true)\n\ttf(false)\n}\n\nfunc TestNomsDiffPrintMapWithStructKeys(t *testing.T) {\n\ta := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tk1 := createStruct(\"TestKey\", \"name\", \"n1\", \"label\", \"l1\")\n\n\texpected1 := `(root) {\n- struct TestKey {\n- label: \"l1\",\n- name: \"n1\",\n- }: true\n+ struct TestKey {\n+ label: \"l1\",\n+ name: \"n1\",\n+ }: false\n }\n`\n\n\tm1 := types.NewMap(vs, k1, types.Bool(true))\n\tm2 := types.NewMap(vs, k1, types.Bool(false))\n\ttf := func(leftRight bool) {\n\t\tbuf := &bytes.Buffer{}\n\t\tPrintDiff(buf, m1, m2, leftRight)\n\t\ta.Equal(expected1, buf.String())\n\t}\n\ttf(true)\n\ttf(false)\n}\n\nfunc TestNomsDiffPrintList(t *testing.T) {\n\tassert := assert.New(t)\n\n\texpected1 := `(root) {\n- 2\n+ 22\n- 44\n }\n`\n\texpectedPaths1 := []string{\n\t\t`[1]`,\n\t\t`[4]`,\n\t}\n\n\tl1 := createList(1, 2, 3, 4, 44, 5, 6)\n\tl2 := createList(1, 22, 3, 4, 5, 6)\n\n\texpected2 := `(root) {\n+ \"seven\"\n }\n`\n\texpectedPaths2 := []string{\n\t\t`[6]`,\n\t}\n\n\tl3 := createList(\"one\", \"two\", \"three\", \"four\", \"five\", \"six\")\n\tl4 := createList(\"one\", \"two\", \"three\", \"four\", \"five\", \"six\", \"seven\")\n\n\texpected3 := `[2] {\n- \"m3\": \"m-three\"\n+ \"m3\": \"m-three-diff\"\n }\n[2][\"m4\"] {\n- \"a1\": \"a-one\"\n+ \"a1\": \"a-one-diff\"\n }\n`\n\texpectedPaths3 := []string{\n\t\t`[2][\"m3\"]`,\n\t\t`[2][\"m4\"][\"a1\"]`,\n\t}\n\n\tl5 := createList(mm1, mm2, mm3, mm4)\n\tl6 := createList(mm1, mm2, mm3x, mm4)\n\n\ttf := func(leftRight bool) {\n\t\tbuf := &bytes.Buffer{}\n\t\tPrintDiff(buf, l1, l2, leftRight)\n\t\tassert.Equal(expected1, buf.String())\n\n\t\tpaths := pathsFromDiff(l1, l2, leftRight)\n\t\tassert.Equal(expectedPaths1, paths)\n\n\t\tbuf = &bytes.Buffer{}\n\t\tPrintDiff(buf, l3, l4, leftRight)\n\t\tassert.Equal(expected2, buf.String())\n\n\t\tpaths = pathsFromDiff(l3, l4, leftRight)\n\t\tassert.Equal(expectedPaths2, paths)\n\n\t\tbuf = &bytes.Buffer{}\n\t\tPrintDiff(buf, l5, l6, leftRight)\n\t\tassert.Equal(expected3, buf.String())\n\n\t\tpaths = pathsFromDiff(l5, l6, leftRight)\n\t\tassert.Equal(expectedPaths3, paths)\n\t}\n\n\ttf(true)\n\ttf(false)\n}\n\nfunc TestNomsDiffPrintBlob(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\texpected := \"- Blob (2.0 kB)\\n+ Blob (11 B)\\n\"\n\texpectedPaths1 := []string{``}\n\tb1 := types.NewBlob(vs, strings.NewReader(strings.Repeat(\"x\", 2*1024)))\n\tb2 := types.NewBlob(vs, strings.NewReader(\"Hello World\"))\n\n\ttf := func(leftRight bool) {\n\t\tbuf := &bytes.Buffer{}\n\t\tPrintDiff(buf, b1, b2, leftRight)\n\t\tassert.Equal(expected, buf.String())\n\n\t\tpaths := pathsFromDiff(b1, b2, leftRight)\n\t\tassert.Equal(expectedPaths1, paths)\n\t}\n\n\ttf(true)\n\ttf(false)\n}\n\nfunc TestNomsDiffPrintType(t *testing.T) {\n\tassert := assert.New(t)\n\n\texpected1 := \"- List\\n+ List\\n\"\n\texpectedPaths1 := []string{\"\"}\n\tt1 := types.MakeListType(types.NumberType)\n\tt2 := types.MakeListType(types.StringType)\n\n\texpected2 := \"- List\\n+ Set\\n\"\n\texpectedPaths2 := []string{``}\n\tt3 := types.MakeListType(types.NumberType)\n\tt4 := types.MakeSetType(types.StringType)\n\n\ttf := func(leftRight bool) {\n\t\tbuf := &bytes.Buffer{}\n\t\tPrintDiff(buf, t1, t2, leftRight)\n\t\tassert.Equal(expected1, buf.String())\n\n\t\tpaths := pathsFromDiff(t1, t2, leftRight)\n\t\tassert.Equal(expectedPaths1, paths)\n\n\t\tbuf = &bytes.Buffer{}\n\t\tPrintDiff(buf, t3, t4, leftRight)\n\t\tassert.Equal(expected2, buf.String())\n\n\t\tpaths = pathsFromDiff(t3, t4, leftRight)\n\t\tassert.Equal(expectedPaths2, paths)\n\t}\n\n\ttf(true)\n\ttf(false)\n}\n\nfunc TestNomsDiffPrintRef(t *testing.T) {\n\tassert := assert.New(t)\n\n\texpected := \"- #fckcbt7nk5jl4arco2dk7r9nj7abb6ci\\n+ #i7d3u5gekm48ot419t2cot6cnl7ltcah\\n\"\n\texpectedPaths1 := []string{``}\n\tl1 := createList(1)\n\tl2 := createList(2)\n\tr1 := types.NewRef(l1)\n\tr2 := types.NewRef(l2)\n\n\ttf := func(leftRight bool) {\n\t\tbuf := &bytes.Buffer{}\n\t\tPrintDiff(buf, r1, r2, leftRight)\n\t\ttest.EqualsIgnoreHashes(t, expected, buf.String())\n\n\t\tpaths := pathsFromDiff(r1, r2, leftRight)\n\t\tassert.Equal(expectedPaths1, paths)\n\t}\n\n\ttf(true)\n\ttf(false)\n}\n" }, { "path": "go/diff/patch.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage diff\n\nimport (\n\t\"bytes\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// Patch is a list of difference objects that can be applied to a graph\n// using ApplyPatch(). Patch implements a sort order that is useful for\n// applying the patch in an efficient way.\ntype Patch []Difference\n\nfunc (r Patch) Swap(i, j int) {\n\tr[i], r[j] = r[j], r[i]\n}\n\nfunc (r Patch) Len() int {\n\treturn len(r)\n}\n\nvar vals = map[types.DiffChangeType]int{types.DiffChangeRemoved: 0, types.DiffChangeModified: 1, types.DiffChangeAdded: 2}\n\nfunc (r Patch) Less(i, j int) bool {\n\tif r[i].Path.Equals(r[j].Path) {\n\t\treturn vals[r[i].ChangeType] < vals[r[j].ChangeType]\n\t}\n\treturn pathIsLess(r[i].Path, r[j].Path)\n}\n\n// Utility methods on path\n// TODO: Should these be on types.Path & types.PathPart?\nfunc pathIsLess(p1, p2 types.Path) bool {\n\tfor i, pp1 := range p1 {\n\t\tif len(p2) == i {\n\t\t\treturn false // p1 > p2\n\t\t}\n\t\tswitch pathPartCompare(pp1, p2[i]) {\n\t\tcase -1:\n\t\t\treturn true // p1 < p2\n\t\tcase 1:\n\t\t\treturn false // p1 > p2\n\t\t}\n\t}\n\n\treturn len(p2) > len(p1) // if true p1 < p2, else p1 == p2\n}\n\nfunc fieldPathCompare(pp types.FieldPath, o types.PathPart) int {\n\tswitch opp := o.(type) {\n\tcase types.FieldPath:\n\t\tif pp.Name == opp.Name {\n\t\t\treturn 0\n\t\t}\n\t\tif pp.Name < opp.Name {\n\t\t\treturn -1\n\t\t}\n\t\treturn 1\n\tcase types.IndexPath:\n\t\treturn -1\n\tcase types.HashIndexPath:\n\t\treturn -1\n\t}\n\tpanic(\"unreachable\")\n}\n\nfunc indexPathCompare(pp types.IndexPath, o types.PathPart) int {\n\tswitch opp := o.(type) {\n\tcase types.FieldPath:\n\t\treturn 1\n\tcase types.IndexPath:\n\t\tif pp.Index.Equals(opp.Index) {\n\t\t\tif pp.IntoKey == opp.IntoKey {\n\t\t\t\treturn 0\n\t\t\t}\n\t\t\tif pp.IntoKey {\n\t\t\t\treturn -1\n\t\t\t}\n\t\t\treturn 1\n\t\t}\n\t\tif pp.Index.Less(opp.Index) {\n\t\t\treturn -1\n\t\t}\n\t\treturn 1\n\tcase types.HashIndexPath:\n\t\treturn -1\n\t}\n\tpanic(\"unreachable\")\n}\n\nfunc hashIndexPathCompare(pp types.HashIndexPath, o types.PathPart) int {\n\tswitch opp := o.(type) {\n\tcase types.FieldPath:\n\t\treturn 1\n\tcase types.IndexPath:\n\t\treturn 1\n\tcase types.HashIndexPath:\n\t\tswitch bytes.Compare(pp.Hash[:], opp.Hash[:]) {\n\t\tcase -1:\n\t\t\treturn -1\n\t\tcase 0:\n\t\t\tif pp.IntoKey == opp.IntoKey {\n\t\t\t\treturn 0\n\t\t\t}\n\t\t\tif pp.IntoKey {\n\t\t\t\treturn -1\n\t\t\t}\n\t\t\treturn 1\n\t\tcase 1:\n\t\t\treturn 1\n\t\t}\n\t}\n\tpanic(\"unreachable\")\n}\n\nfunc pathPartCompare(pp, pp2 types.PathPart) int {\n\tswitch pp1 := pp.(type) {\n\tcase types.FieldPath:\n\t\treturn fieldPathCompare(pp1, pp2)\n\tcase types.IndexPath:\n\t\treturn indexPathCompare(pp1, pp2)\n\tcase types.HashIndexPath:\n\t\treturn hashIndexPathCompare(pp1, pp2)\n\t}\n\tpanic(\"unreachable\")\n}\n" }, { "path": "go/diff/patch_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage diff\n\nimport (\n\t\"math/rand\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestPatchPathPartCompare(t *testing.T) {\n\tassert := assert.New(t)\n\n\tfieldPath1 := mustParsePath(assert, `.field1`)[0]\n\tfieldPath2 := mustParsePath(assert, `.field2`)[0]\n\tindexPath1 := mustParsePath(assert, `[\"field1\"]`)[0]\n\tindexPath2 := mustParsePath(assert, `[\"field2\"]`)[0]\n\tindexPathKey1 := mustParsePath(assert, `[\"field1\"]@key`)[0]\n\tindexPathKey2 := mustParsePath(assert, `[\"field2\"]@key`)[0]\n\thashIndexPath1 := mustParsePath(assert, `[#01234567890123456789012345678901]`)[0]\n\thashIndexPath2 := mustParsePath(assert, `[#0123456789abcdef0123456789abcdef]`)[0]\n\thashIndexPathKey1 := mustParsePath(assert, `[#01234567890123456789012345678901]`)[0]\n\thashIndexPathKey2 := mustParsePath(assert, `[#0123456789abcdef0123456789abcdef]`)[0]\n\n\ttestCases := [][]types.PathPart{\n\t\t{fieldPath1, fieldPath2},\n\t\t{indexPath1, indexPath2},\n\t\t{indexPathKey1, indexPathKey2},\n\t\t{hashIndexPath1, hashIndexPath2},\n\t\t{hashIndexPathKey1, hashIndexPathKey2},\n\t\t{fieldPath2, indexPath1},\n\t\t{fieldPath2, indexPathKey1},\n\t\t{fieldPath2, hashIndexPath1},\n\t\t{fieldPath2, hashIndexPathKey1},\n\t\t{indexPath2, hashIndexPath1},\n\t\t{indexPath2, hashIndexPathKey1},\n\t}\n\n\tfor i, tc := range testCases {\n\t\tassert.Equal(-1, pathPartCompare(tc[0], tc[1]), \"test case %d failed, pp0: %s, pp1: %s\", i, tc[0], tc[1])\n\t\tassert.Equal(0, pathPartCompare(tc[0], tc[0]), \"test case %d failed, pp0: %s, pp1: %s\", i, tc[0], tc[1])\n\t\tassert.Equal(1, pathPartCompare(tc[1], tc[0]), \"test case %d failed, pp0: %s, pp1: %s\", i, tc[0], tc[1])\n\t}\n}\n\nfunc TestPatchPathIsLess(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttestCases := [][]string{\n\t\t{``, `[\"field1\"]`},\n\t\t{`[\"field1\"]`, `[\"field1\"].f1`},\n\t\t{`[\"field1\"].f1`, `[\"field1\"][\"f1\"]`},\n\t\t{`[\"field1\"][\"f1\"]@key`, `[\"field1\"][\"f1\"]`},\n\t\t{`[\"field1\"][\"f1\"]`, `[\"field1\"][#01234567890123456789012345678901]`},\n\t\t{`[\"field1\"][#01234567890123456789012345678901]`, `[\"field1\"][#0123456789abcdef0123456789abcdef]`},\n\t}\n\n\tfor i, tc := range testCases {\n\t\tp0 := mustParsePath(assert, tc[0])\n\t\tp1 := mustParsePath(assert, tc[1])\n\t\tassert.True(pathIsLess(p0, p1), \"test case %d failed\", i)\n\t\tassert.False(pathIsLess(p0, p0), \"test case %d failed\", i)\n\t\tassert.False(pathIsLess(p1, p0), \"test case %d failed\", i)\n\t}\n\t//p := mustParsePath(assert, `#0123456789abcdef0123456789abcdef.value`)\n\t//fmt.Printf(\"p[0]: %s, type: %T\\n\", p[0], p[0])\n}\n\nfunc TestPatchSort(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsortedPaths := Patch{\n\t\t{Path: mustParsePath(assert, `[\"field1\"]`)},\n\t\t{Path: mustParsePath(assert, `[\"field1\"].f1`)},\n\t\t{Path: mustParsePath(assert, `[\"field1\"][\"f1\"]`), ChangeType: types.DiffChangeRemoved},\n\t\t{Path: mustParsePath(assert, `[\"field1\"][\"f1\"]`), ChangeType: types.DiffChangeModified},\n\t\t{Path: mustParsePath(assert, `[\"field1\"][\"f1\"]`), ChangeType: types.DiffChangeAdded},\n\t\t{Path: mustParsePath(assert, `[\"field1\"][#01234567890123456789012345678901]`)},\n\t\t{Path: mustParsePath(assert, `[\"field1\"][#0123456789abcdef0123456789abcdef]`)},\n\t}\n\n\trand.Perm(len(sortedPaths))\n\tshuffledPaths := Patch{}\n\tfor _, idx := range rand.Perm(len(sortedPaths)) {\n\t\tshuffledPaths = append(shuffledPaths, sortedPaths[idx])\n\t}\n\n\tsort.Sort(shuffledPaths)\n\tassert.Equal(sortedPaths, shuffledPaths)\n}\n" }, { "path": "go/diff/print_diff.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage diff\n\nimport (\n\t\"io\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/writers\"\n\t\"github.com/dustin/go-humanize\"\n)\n\ntype prefixOp string\n\nconst (\n\tADD = \"+ \"\n\tDEL = \"- \"\n)\n\ntype (\n\tprintFunc func(w io.Writer, op prefixOp, key, val types.Value) error\n)\n\n// PrintDiff writes a textual reprensentation of the diff from |v1| to |v2|\n// to |w|. If |leftRight| is true then the left-right diff is used for ordered\n// sequences - see Diff vs DiffLeftRight in Set and Map.\nfunc PrintDiff(w io.Writer, v1, v2 types.Value, leftRight bool) (err error) {\n\t// In the case where the diff involves two simple values, just print out the\n\t// diff and return. This is needed because the code below assumes that the\n\t// values being compared have a parent.\n\tif !shouldDescend(v1, v2) {\n\t\tline(w, DEL, nil, v1)\n\t\treturn line(w, ADD, nil, v2)\n\t}\n\n\tdChan := make(chan Difference, 16)\n\tstopChan := make(chan struct{})\n\tstopDiff := func() {\n\t\tclose(stopChan)\n\t\tfor range dChan {\n\t\t}\n\t}\n\n\t// From here on, we can assume that every Difference will have at least one\n\t// element in the Path\n\tgo func() {\n\t\tDiff(v1, v2, dChan, stopChan, leftRight)\n\t\tclose(dChan)\n\t}()\n\n\tvar lastParentPath types.Path\n\twroteHdr := false\n\tfirstTime := true\n\n\tfor d := range dChan {\n\t\tparentPath := d.Path[:len(d.Path)-1]\n\t\tparentPathChanged := !parentPath.Equals(lastParentPath)\n\t\tlastParentPath = parentPath\n\t\tif parentPathChanged && wroteHdr {\n\t\t\terr = writeFooter(w, &wroteHdr)\n\t\t}\n\t\tif parentPathChanged || firstTime {\n\t\t\tfirstTime = false\n\t\t\terr = writeHeader(w, parentPath, &wroteHdr)\n\t\t}\n\n\t\tlastPart := d.Path[len(d.Path)-1]\n\t\tparentEl := parentPath.Resolve(v1, nil)\n\n\t\tvar key types.Value\n\t\tvar pfunc printFunc = line\n\n\t\tswitch parent := parentEl.(type) {\n\t\tcase types.Map:\n\t\t\tif indexPath, ok := lastPart.(types.IndexPath); ok {\n\t\t\t\tkey = indexPath.Index\n\t\t\t} else if hip, ok := lastPart.(types.HashIndexPath); ok {\n\t\t\t\t// In this case, the map has a non-primitive key so the value\n\t\t\t\t// is a ref to the key. We need the actual key, not a ref to it.\n\t\t\t\thip1 := hip\n\t\t\t\thip1.IntoKey = true\n\t\t\t\tkey = hip1.Resolve(parent, nil)\n\t\t\t} else {\n\t\t\t\tpanic(\"unexpected Path type\")\n\t\t\t}\n\t\tcase types.Set:\n\t\t\t// default values are ok\n\t\tcase types.Struct:\n\t\t\tkey = types.String(lastPart.(types.FieldPath).Name)\n\t\t\tpfunc = field\n\t\tcase types.List:\n\t\t\t// default values are ok\n\t\t}\n\n\t\tif d.OldValue != nil {\n\t\t\terr = pfunc(w, DEL, key, d.OldValue)\n\t\t}\n\t\tif d.NewValue != nil {\n\t\t\terr = pfunc(w, ADD, key, d.NewValue)\n\t\t}\n\t\tif err != nil {\n\t\t\tstopDiff()\n\t\t\tbreak\n\t\t}\n\t}\n\terr = writeFooter(w, &wroteHdr)\n\treturn\n}\n\nfunc writeHeader(w io.Writer, p types.Path, wroteHdr *bool) error {\n\tif *wroteHdr {\n\t\treturn nil\n\t}\n\t*wroteHdr = true\n\thdr := \"(root)\"\n\tif len(p) > 0 {\n\t\thdr = p.String()\n\t}\n\treturn write(w, []byte(hdr+\" {\\n\"))\n}\n\nfunc writeFooter(w io.Writer, wroteHdr *bool) error {\n\tif !*wroteHdr {\n\t\treturn nil\n\t}\n\t*wroteHdr = false\n\treturn write(w, []byte(\" }\\n\"))\n}\n\nfunc line(w io.Writer, op prefixOp, key, val types.Value) error {\n\tgenPrefix := func(w *writers.PrefixWriter) []byte {\n\t\treturn []byte(op)\n\t}\n\tpw := &writers.PrefixWriter{Dest: w, PrefixFunc: genPrefix, NeedsPrefix: true}\n\tif key != nil {\n\t\twriteEncodedValue(pw, key)\n\t\twrite(w, []byte(\": \"))\n\t}\n\twriteEncodedValue(pw, val)\n\treturn write(w, []byte(\"\\n\"))\n}\n\nfunc field(w io.Writer, op prefixOp, name, val types.Value) error {\n\tgenPrefix := func(w *writers.PrefixWriter) []byte {\n\t\treturn []byte(op)\n\t}\n\tpw := &writers.PrefixWriter{Dest: w, PrefixFunc: genPrefix, NeedsPrefix: true}\n\twrite(pw, []byte(name.(types.String)))\n\twrite(w, []byte(\": \"))\n\twriteEncodedValue(pw, val)\n\treturn write(w, []byte(\"\\n\"))\n}\n\nfunc writeEncodedValue(w io.Writer, v types.Value) error {\n\tif v.Kind() != types.BlobKind {\n\t\treturn types.WriteEncodedValue(w, v)\n\t}\n\twrite(w, []byte(\"Blob (\"))\n\twrite(w, []byte(humanize.Bytes(v.(types.Blob).Len())))\n\treturn write(w, []byte(\")\"))\n}\n\nfunc write(w io.Writer, b []byte) error {\n\t_, err := w.Write(b)\n\treturn err\n}\n" }, { "path": "go/diff/summary.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage diff\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/status\"\n\thumanize \"github.com/dustin/go-humanize\"\n)\n\n// Summary prints a summary of the diff between two values to stdout.\nfunc Summary(value1, value2 types.Value) {\n\tif datas.IsCommit(value1) && datas.IsCommit(value2) {\n\t\tfmt.Println(\"Comparing commit values\")\n\t\tvalue1 = value1.(types.Struct).Get(datas.ValueField)\n\t\tvalue2 = value2.(types.Struct).Get(datas.ValueField)\n\t}\n\n\tvar singular, plural string\n\tif value1.Kind() == value2.Kind() {\n\t\tswitch value1.Kind() {\n\t\tcase types.StructKind:\n\t\t\tsingular = \"field\"\n\t\t\tplural = \"fields\"\n\t\tcase types.MapKind:\n\t\t\tsingular = \"entry\"\n\t\t\tplural = \"entries\"\n\t\tdefault:\n\t\t\tsingular = \"value\"\n\t\t\tplural = \"values\"\n\t\t}\n\t}\n\n\tch := make(chan diffSummaryProgress)\n\tgo func() {\n\t\tdiffSummary(ch, value1, value2)\n\t\tclose(ch)\n\t}()\n\n\tacc := diffSummaryProgress{}\n\tfor p := range ch {\n\t\tacc.Adds += p.Adds\n\t\tacc.Removes += p.Removes\n\t\tacc.Changes += p.Changes\n\t\tacc.NewSize += p.NewSize\n\t\tacc.OldSize += p.OldSize\n\t\tif status.WillPrint() {\n\t\t\tformatStatus(acc, singular, plural)\n\t\t}\n\t}\n\tformatStatus(acc, singular, plural)\n\tstatus.Done()\n}\n\ntype diffSummaryProgress struct {\n\tAdds, Removes, Changes, NewSize, OldSize uint64\n}\n\nfunc diffSummary(ch chan diffSummaryProgress, v1, v2 types.Value) {\n\tif !v1.Equals(v2) {\n\t\tif shouldDescend(v1, v2) {\n\t\t\tswitch v1.Kind() {\n\t\t\tcase types.ListKind:\n\t\t\t\tdiffSummaryList(ch, v1.(types.List), v2.(types.List))\n\t\t\tcase types.MapKind:\n\t\t\t\tdiffSummaryMap(ch, v1.(types.Map), v2.(types.Map))\n\t\t\tcase types.SetKind:\n\t\t\t\tdiffSummarySet(ch, v1.(types.Set), v2.(types.Set))\n\t\t\tcase types.StructKind:\n\t\t\t\tdiffSummaryStructs(ch, v1.(types.Struct), v2.(types.Struct))\n\t\t\tdefault:\n\t\t\t\tpanic(\"Unrecognized type in diff function: \" + types.TypeOf(v1).Describe() + \" and \" + types.TypeOf(v2).Describe())\n\t\t\t}\n\t\t} else {\n\t\t\tch <- diffSummaryProgress{Adds: 1, Removes: 1, NewSize: 1, OldSize: 1}\n\t\t}\n\t}\n}\n\nfunc diffSummaryList(ch chan<- diffSummaryProgress, v1, v2 types.List) {\n\tch <- diffSummaryProgress{OldSize: v1.Len(), NewSize: v2.Len()}\n\n\tspliceChan := make(chan types.Splice)\n\tstopChan := make(chan struct{}, 1) // buffer size of 1, so this won't block if diff already finished\n\n\tgo func() {\n\t\tv2.Diff(v1, spliceChan, stopChan)\n\t\tclose(spliceChan)\n\t}()\n\n\tfor splice := range spliceChan {\n\t\tif splice.SpRemoved == splice.SpAdded {\n\t\t\tch <- diffSummaryProgress{Changes: splice.SpRemoved}\n\t\t} else {\n\t\t\tch <- diffSummaryProgress{Adds: splice.SpAdded, Removes: splice.SpRemoved}\n\t\t}\n\t}\n}\n\nfunc diffSummaryMap(ch chan<- diffSummaryProgress, v1, v2 types.Map) {\n\tdiffSummaryValueChanged(ch, v1.Len(), v2.Len(), func(changeChan chan<- types.ValueChanged, stopChan <-chan struct{}) {\n\t\tv2.Diff(v1, changeChan, stopChan)\n\t})\n}\n\nfunc diffSummarySet(ch chan<- diffSummaryProgress, v1, v2 types.Set) {\n\tdiffSummaryValueChanged(ch, v1.Len(), v2.Len(), func(changeChan chan<- types.ValueChanged, stopChan <-chan struct{}) {\n\t\tv2.Diff(v1, changeChan, stopChan)\n\t})\n}\n\nfunc diffSummaryStructs(ch chan<- diffSummaryProgress, v1, v2 types.Struct) {\n\t// TODO: Operate on values directly\n\tsize1 := uint64(types.TypeOf(v1).Desc.(types.StructDesc).Len())\n\tsize2 := uint64(types.TypeOf(v2).Desc.(types.StructDesc).Len())\n\tdiffSummaryValueChanged(ch, size1, size2, func(changeChan chan<- types.ValueChanged, stopChan <-chan struct{}) {\n\t\tv2.Diff(v1, changeChan, stopChan)\n\t})\n}\n\nfunc diffSummaryValueChanged(ch chan<- diffSummaryProgress, oldSize, newSize uint64, f diffFunc) {\n\tch <- diffSummaryProgress{OldSize: oldSize, NewSize: newSize}\n\n\tchangeChan := make(chan types.ValueChanged)\n\tstopChan := make(chan struct{}, 1) // buffer size of 1, so this won't block if diff already finished\n\n\tgo func() {\n\t\tf(changeChan, stopChan)\n\t\tclose(changeChan)\n\t}()\n\treportChanges(ch, changeChan)\n}\n\nfunc reportChanges(ch chan<- diffSummaryProgress, changeChan chan types.ValueChanged) {\n\tfor change := range changeChan {\n\t\tswitch change.ChangeType {\n\t\tcase types.DiffChangeAdded:\n\t\t\tch <- diffSummaryProgress{Adds: 1}\n\t\tcase types.DiffChangeRemoved:\n\t\t\tch <- diffSummaryProgress{Removes: 1}\n\t\tcase types.DiffChangeModified:\n\t\t\tch <- diffSummaryProgress{Changes: 1}\n\t\tdefault:\n\t\t\tpanic(\"unknown change type\")\n\t\t}\n\t}\n}\n\nfunc formatStatus(acc diffSummaryProgress, singular, plural string) {\n\tpluralize := func(singular, plural string, n uint64) string {\n\t\tvar noun string\n\t\tif n != 1 {\n\t\t\tnoun = plural\n\t\t} else {\n\t\t\tnoun = singular\n\t\t}\n\t\treturn fmt.Sprintf(\"%s %s\", humanize.Comma(int64(n)), noun)\n\t}\n\n\tinsertions := pluralize(\"insertion\", \"insertions\", acc.Adds)\n\tdeletions := pluralize(\"deletion\", \"deletions\", acc.Removes)\n\tchanges := pluralize(\"change\", \"changes\", acc.Changes)\n\n\toldValues := pluralize(singular, plural, acc.OldSize)\n\tnewValues := pluralize(singular, plural, acc.NewSize)\n\n\tstatus.Printf(\"%s (%.2f%%), %s (%.2f%%), %s (%.2f%%), (%s vs %s)\", insertions, (float64(100*acc.Adds) / float64(acc.OldSize)), deletions, (float64(100*acc.Removes) / float64(acc.OldSize)), changes, (float64(100*acc.Changes) / float64(acc.OldSize)), oldValues, newValues)\n}\n" }, { "path": "go/hash/base32.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage hash\n\nimport \"encoding/base32\"\n\nvar encoding = base32.NewEncoding(\"0123456789abcdefghijklmnopqrstuv\")\n\nfunc encode(data []byte) string {\n\treturn encoding.EncodeToString(data)\n}\n\nfunc decode(s string) []byte {\n\tslice, _ := encoding.DecodeString(s)\n\treturn slice\n}\n" }, { "path": "go/hash/base32_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage hash\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestBase32Encode(t *testing.T) {\n\tassert := assert.New(t)\n\n\td := make([]byte, 20, 20)\n\tassert.Equal(\"00000000000000000000000000000000\", encode(d))\n\td[19] = 1\n\tassert.Equal(\"00000000000000000000000000000001\", encode(d))\n\td[19] = 10\n\tassert.Equal(\"0000000000000000000000000000000a\", encode(d))\n\td[19] = 20\n\tassert.Equal(\"0000000000000000000000000000000k\", encode(d))\n\td[19] = 31\n\tassert.Equal(\"0000000000000000000000000000000v\", encode(d))\n\td[19] = 32\n\tassert.Equal(\"00000000000000000000000000000010\", encode(d))\n\td[19] = 63\n\tassert.Equal(\"0000000000000000000000000000001v\", encode(d))\n\td[19] = 64\n\tassert.Equal(\"00000000000000000000000000000020\", encode(d))\n\n\t// Largest!\n\tfor i := 0; i < 20; i++ {\n\t\td[i] = 0xff\n\t}\n\tassert.Equal(\"vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv\", encode(d))\n}\n\nfunc TestBase32Decode(t *testing.T) {\n\tassert := assert.New(t)\n\n\td := make([]byte, 20, 20)\n\tassert.Equal(d, decode(\"00000000000000000000000000000000\"))\n\n\td[19] = 1\n\tassert.Equal(d, decode(\"00000000000000000000000000000001\"))\n\td[19] = 10\n\tassert.Equal(d, decode(\"0000000000000000000000000000000a\"))\n\td[19] = 20\n\tassert.Equal(d, decode(\"0000000000000000000000000000000k\"))\n\td[19] = 31\n\tassert.Equal(d, decode(\"0000000000000000000000000000000v\"))\n\td[19] = 32\n\tassert.Equal(d, decode(\"00000000000000000000000000000010\"))\n\td[19] = 63\n\tassert.Equal(d, decode(\"0000000000000000000000000000001v\"))\n\td[19] = 64\n\tassert.Equal(d, decode(\"00000000000000000000000000000020\"))\n\n\t// Largest!\n\tfor i := 0; i < 20; i++ {\n\t\td[i] = 0xff\n\t}\n\tassert.Equal(d, decode(\"vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv\"))\n}\n" }, { "path": "go/hash/hash.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package hash implements the hash function used throughout Noms.\n//\n// Noms serialization from version 4-onward uses the first 20 bytes of sha-512 for hashes.\n//\n// sha-512 was chosen because:\n//\n// - sha-1 is no longer recommended.\n// - sha-3 is brand new, not a lot of platform support.\n// - blake is not commonly used, not a lot of platform support.\n// - within sha-2, sha-512 is faster than sha-256 on 64 bit.\n//\n// Our specific truncation scheme (first 20 bytes) was chosen because:\n//\n// - The \"standard\" truncation schemes are not widely supported. For example, at time of writing, there is no fast native implementation of sha512/256 on Node.\n// - The smallest standard truncation of sha512 is 28 bytes, but we don't need this many. And because we are a database, the size of the hashes matters. Bigger hashes mean less data in each chunk, which means less tree fan-out, which means slower iteration and searching. 20 bytes is a good balance between collision resistance and wide trees.\n// - 20 bytes leads to a nice round number of base32 digits: 32.\n//\n// The textual serialization of hashes uses big-endian base32 with the alphabet {0-9,a-v}. This scheme was chosen because:\n//\n// - It's easy to convert to and from base32 without bignum arithemetic.\n// - No special chars: you can double-click to select in GUIs.\n// - Sorted hashes will be sorted textually, making it easy to scan for humans.\n//\n// In Noms, the hash function is a component of the serialization version, which is constant over the entire lifetime of a single database. So clients do not need to worry about encountering multiple hash functions in the same database.\npackage hash\n\nimport (\n\t\"bytes\"\n\t\"crypto/sha512\"\n\t\"fmt\"\n\t\"regexp\"\n\t\"strconv\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nconst (\n\t// ByteLen is the number of bytes used to represent the Hash.\n\tByteLen = 20\n\n\t// StringLen is the number of characters need to represent the Hash using Base32.\n\tStringLen = 32 // 20 * 8 / log2(32)\n)\n\nvar (\n\tpattern = regexp.MustCompile(\"^([0-9a-v]{\" + strconv.Itoa(StringLen) + \"})$\")\n\temptyHash = Hash{}\n)\n\n// Hash is used to represent the hash of a Noms Value.\ntype Hash [ByteLen]byte\n\n// IsEmpty determines if this Hash is equal to the empty hash (all zeroes).\nfunc (h Hash) IsEmpty() bool {\n\treturn h == emptyHash\n}\n\n// String returns a string representation of the hash using Base32 encoding.\nfunc (h Hash) String() string {\n\treturn encode(h[:])\n}\n\n// Of computes a new Hash from data.\nfunc Of(data []byte) Hash {\n\tr := sha512.Sum512(data)\n\th := Hash{}\n\tcopy(h[:], r[:ByteLen])\n\treturn h\n}\n\n// New creates a new Hash backed by data, ensuring that data is an acceptable length.\nfunc New(data []byte) Hash {\n\td.PanicIfFalse(len(data) == ByteLen)\n\th := Hash{}\n\tcopy(h[:], data)\n\treturn h\n}\n\n// MaybeParse parses a string representing a hash as a Base32 encoded byte array.\n// If the string is not well formed then this returns (emptyHash, false).\nfunc MaybeParse(s string) (Hash, bool) {\n\tmatch := pattern.FindStringSubmatch(s)\n\tif match == nil {\n\t\treturn emptyHash, false\n\t}\n\treturn New(decode(s)), true\n}\n\n// Parse parses a string representing a hash as a Base32 encoded byte array.\n// If the string is not well formed then this panics.\nfunc Parse(s string) Hash {\n\tr, ok := MaybeParse(s)\n\tif !ok {\n\t\td.PanicIfError(fmt.Errorf(\"Cound not parse Hash: %s\", s))\n\t}\n\treturn r\n}\n\n// Less compares two hashes returning whether this Hash is less than other.\nfunc (h Hash) Less(other Hash) bool {\n\treturn bytes.Compare(h[:], other[:]) < 0\n}\n\n// Greater compares two hashes returning whether this Hash is greater than other.\nfunc (h Hash) Greater(other Hash) bool {\n\t// TODO: Remove this\n\treturn bytes.Compare(h[:], other[:]) > 0\n}\n\n// HashSet is a set of Hashes.\ntype HashSet map[Hash]struct{}\n\nfunc NewHashSet(hashes ...Hash) HashSet {\n\tout := make(HashSet, len(hashes))\n\tfor _, h := range hashes {\n\t\tout.Insert(h)\n\t}\n\treturn out\n}\n\n// Insert adds a Hash to the set.\nfunc (hs HashSet) Insert(hash Hash) {\n\ths[hash] = struct{}{}\n}\n\n// Has returns true if the HashSet contains hash.\nfunc (hs HashSet) Has(hash Hash) (has bool) {\n\t_, has = hs[hash]\n\treturn\n}\n\n// Remove removes hash from the HashSet.\nfunc (hs HashSet) Remove(hash Hash) {\n\tdelete(hs, hash)\n}\n" }, { "path": "go/hash/hash_slice.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage hash\n\ntype HashSlice []Hash\n\nfunc (rs HashSlice) Len() int {\n\treturn len(rs)\n}\n\nfunc (rs HashSlice) Less(i, j int) bool {\n\treturn rs[i].Less(rs[j])\n}\n\nfunc (rs HashSlice) Swap(i, j int) {\n\trs[i], rs[j] = rs[j], rs[i]\n}\n\nfunc (rs HashSlice) Equals(other HashSlice) bool {\n\tif len(rs) != len(other) {\n\t\treturn false\n\t}\n\tfor i := 0; i < len(rs); i++ {\n\t\tif rs[i] != other[i] {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\nfunc (rs HashSlice) HashSet() HashSet {\n\ths := make(HashSet, len(rs))\n\tfor _, h := range rs {\n\t\ths[h] = struct{}{}\n\t}\n\n\treturn hs\n}\n" }, { "path": "go/hash/hash_slice_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage hash\n\nimport (\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestHashSliceSort(t *testing.T) {\n\tassert := assert.New(t)\n\n\trs := HashSlice{}\n\tfor i := 1; i <= 3; i++ {\n\t\tfor j := 1; j <= 3; j++ {\n\t\t\th := Hash{}\n\t\t\tfor k := 1; k <= j; k++ {\n\t\t\t\th[k-1] = byte(i)\n\t\t\t}\n\t\t\trs = append(rs, h)\n\t\t}\n\t}\n\n\trs2 := HashSlice(make([]Hash, len(rs)))\n\tcopy(rs2, rs)\n\tsort.Sort(sort.Reverse(rs2))\n\tassert.False(rs.Equals(rs2))\n\n\tsort.Sort(rs2)\n\tassert.True(rs.Equals(rs2))\n}\n" }, { "path": "go/hash/hash_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage hash\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestParseError(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassertParseError := func(s string) {\n\t\te := d.Try(func() { Parse(s) })\n\t\t_, ok := e.(d.WrappedError)\n\t\tassert.True(ok)\n\t}\n\n\tassertParseError(\"foo\")\n\n\t// too few digits\n\tassertParseError(\"0000000000000000000000000000000\")\n\n\t// too many digits\n\tassertParseError(\"000000000000000000000000000000000\")\n\n\t// 'w' not valid base32\n\tassertParseError(\"00000000000000000000000000000000w\")\n\n\t// no prefix\n\tassertParseError(\"sha1-00000000000000000000000000000000\")\n\tassertParseError(\"sha2-00000000000000000000000000000000\")\n\n\tr := Parse(\"00000000000000000000000000000000\")\n\tassert.NotNil(r)\n}\n\nfunc TestMaybeParse(t *testing.T) {\n\tassert := assert.New(t)\n\n\tparse := func(s string, success bool) {\n\t\tr, ok := MaybeParse(s)\n\t\tassert.Equal(success, ok, \"Expected success=%t for %s\", success, s)\n\t\tif ok {\n\t\t\tassert.Equal(s, r.String())\n\t\t} else {\n\t\t\tassert.Equal(emptyHash, r)\n\t\t}\n\t}\n\n\tparse(\"00000000000000000000000000000000\", true)\n\tparse(\"00000000000000000000000000000001\", true)\n\tparse(\"\", false)\n\tparse(\"adsfasdf\", false)\n\tparse(\"sha2-00000000000000000000000000000000\", false)\n\tparse(\"0000000000000000000000000000000w\", false)\n}\n\nfunc TestEquals(t *testing.T) {\n\tassert := assert.New(t)\n\n\tr0 := Parse(\"00000000000000000000000000000000\")\n\tr01 := Parse(\"00000000000000000000000000000000\")\n\tr1 := Parse(\"00000000000000000000000000000001\")\n\n\tassert.Equal(r0, r01)\n\tassert.Equal(r01, r0)\n\tassert.NotEqual(r0, r1)\n\tassert.NotEqual(r1, r0)\n}\n\nfunc TestString(t *testing.T) {\n\ts := \"0123456789abcdefghijklmnopqrstuv\"\n\tr := Parse(s)\n\tassert.Equal(t, s, r.String())\n}\n\nfunc TestOf(t *testing.T) {\n\tr := Of([]byte(\"abc\"))\n\tassert.Equal(t, \"rmnjb8cjc5tblj21ed4qs821649eduie\", r.String())\n}\n\nfunc TestIsEmpty(t *testing.T) {\n\tr1 := Hash{}\n\tassert.True(t, r1.IsEmpty())\n\n\tr2 := Parse(\"00000000000000000000000000000000\")\n\tassert.True(t, r2.IsEmpty())\n\n\tr3 := Parse(\"rmnjb8cjc5tblj21ed4qs821649eduie\")\n\tassert.False(t, r3.IsEmpty())\n}\n\nfunc TestLess(t *testing.T) {\n\tassert := assert.New(t)\n\n\tr1 := Parse(\"00000000000000000000000000000001\")\n\tr2 := Parse(\"00000000000000000000000000000002\")\n\n\tassert.False(r1.Less(r1))\n\tassert.True(r1.Less(r2))\n\tassert.False(r2.Less(r1))\n\tassert.False(r2.Less(r2))\n\n\tr0 := Hash{}\n\tassert.False(r0.Less(r0))\n\tassert.True(r0.Less(r2))\n\tassert.False(r2.Less(r0))\n}\n\nfunc TestGreater(t *testing.T) {\n\tassert := assert.New(t)\n\n\tr1 := Parse(\"00000000000000000000000000000001\")\n\tr2 := Parse(\"00000000000000000000000000000002\")\n\n\tassert.False(r1.Greater(r1))\n\tassert.False(r1.Greater(r2))\n\tassert.True(r2.Greater(r1))\n\tassert.False(r2.Greater(r2))\n\n\tr0 := Hash{}\n\tassert.False(r0.Greater(r0))\n\tassert.False(r0.Greater(r2))\n\tassert.True(r2.Greater(r0))\n}\n" }, { "path": "go/marshal/decode.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage marshal\n\nimport (\n\t\"fmt\"\n\t\"reflect\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// Unmarshal converts a Noms value into a Go value. It decodes v and stores the\n// result in the value pointed to by out.\n//\n// Unmarshal uses the inverse of the encodings that Marshal uses with the\n// following additional rules:\n//\n// To unmarshal a Noms struct into a Go struct, Unmarshal matches incoming\n// object fields to the fields used by Marshal (either the struct field name or\n// its tag). Unmarshal will only set exported fields of the struct. The name\n// of the Go struct must match (ignoring case) the name of the Noms struct. All\n// exported fields on the Go struct must be present in the Noms struct, unless\n// the field on the Go struct is marked with the \"omitempty\" tag. Go struct\n// fields also support the \"original\" tag which causes the Go field to receive\n// the entire original unmarshaled Noms struct.\n//\n// To unmarshal a Noms list or set into a slice, Unmarshal resets the slice\n// length to zero and then appends each element to the slice. If the Go slice\n// was nil a new slice is created when an element is added.\n//\n// To unmarshal a Noms list into a Go array, Unmarshal decodes Noms list\n// elements into corresponding Go array elements.\n//\n// To unmarshal a Noms map into a Go map, Unmarshal decodes Noms key and values\n// into corresponding Go array elements. If the Go map was nil a new map is\n// created if any value is set.\n//\n// To unmarshal a Noms set into a Go map, the field must be tagged with `noms:\",set\"`,\n// and it must have a type of map[]struct{}. Unmarshal decodes into\n// Go map keys corresponding to the set values and assigns each key a value of struct{}{}.\n//\n// When unmarshalling onto interface{} the following rules are used:\n// - types.Bool -> bool\n// - types.List -> []T, where T is determined recursively using the same rules.\n// - types.Set -> depends on `noms:\",set\"` annotation and field type:\n// - without the annotation, same as types.List\n// - with the annotation, same as types.Map for map[T]struct{} fields and same as types.List for slice fields\n// - types.Map -> map[T]V, where T and V is determined recursively using the\n// same rules.\n// - types.Number -> float64\n// - types.String -> string\n// - *types.Type -> *types.Type\n// - types.Union -> interface\n// - Everything else an error\n//\n// Unmarshal returns an UnmarshalTypeMismatchError if:\n// - a Noms value is not appropriate for a given target type\n// - a Noms number overflows the target type\n// - a Noms list is decoded into a Go array of a different length\nfunc Unmarshal(v types.Value, out interface{}) (err error) {\n\treturn UnmarshalOpt(v, Opt{}, out)\n}\n\n// UnmarshalOpt is like Unmarshal but provides additional options.\nfunc UnmarshalOpt(v types.Value, opt Opt, out interface{}) (err error) {\n\tdefer func() {\n\t\tif r := recover(); r != nil {\n\t\t\tswitch r := r.(type) {\n\t\t\tcase *UnmarshalTypeMismatchError, *UnsupportedTypeError, *InvalidTagError, *InvalidUnmarshalError:\n\t\t\t\terr = r.(error)\n\t\t\tcase *unmarshalNomsError:\n\t\t\t\terr = r.err\n\t\t\tdefault:\n\t\t\t\tpanic(r)\n\t\t\t}\n\t\t}\n\t}()\n\n\tMustUnmarshalOpt(v, opt, out)\n\treturn\n}\n\n// Unmarshals a Noms value into a Go value using the same rules as Unmarshal().\n// Panics on failure.\nfunc MustUnmarshal(v types.Value, out interface{}) {\n\tMustUnmarshalOpt(v, Opt{}, out)\n}\n\n// MustUnmarshalOpt is like MustUnmarshal but with additional options.\nfunc MustUnmarshalOpt(v types.Value, opt Opt, out interface{}) {\n\trv := reflect.ValueOf(out)\n\tif rv.Kind() != reflect.Ptr || rv.IsNil() {\n\t\tpanic(&InvalidUnmarshalError{reflect.TypeOf(out)})\n\t}\n\trv = rv.Elem()\n\tnt := nomsTags{\n\t\tset: opt.Set,\n\t}\n\td := typeDecoder(rv.Type(), nt)\n\td(v, rv)\n}\n\n// Unmarshaler is an interface types can implement to provide their own\n// decoding.\n//\n// You probably want to implement this on a pointer to a type, otherwise\n// calling UnmarshalNoms will effectively do nothing. For example, to unmarshal\n// a MyType you would define:\n//\n// func (t *MyType) UnmarshalNoms(v types.Value) error {}\ntype Unmarshaler interface {\n\t// UnmarshalNoms decodes v, or returns an error.\n\tUnmarshalNoms(v types.Value) error\n}\n\nvar unmarshalerInterface = reflect.TypeOf((*Unmarshaler)(nil)).Elem()\n\n// InvalidUnmarshalError describes an invalid argument passed to Unmarshal. (The\n// argument to Unmarshal must be a non-nil pointer.)\ntype InvalidUnmarshalError struct {\n\tType reflect.Type\n}\n\nfunc (e *InvalidUnmarshalError) Error() string {\n\tif e.Type == nil {\n\t\treturn \"Cannot unmarshal into Go nil value\"\n\t}\n\n\tif e.Type.Kind() != reflect.Ptr {\n\t\treturn \"Cannot unmarshal into Go non pointer of type \" + e.Type.String()\n\t}\n\treturn \"Cannot unmarshal into Go nil pointer of type \" + e.Type.String()\n}\n\n// UnmarshalTypeMismatchError describes a Noms value that was not appropriate\n// for a value of a specific Go type.\ntype UnmarshalTypeMismatchError struct {\n\tValue types.Value\n\tType reflect.Type // type of Go value it could not be assigned to\n\tdetails string\n}\n\nfunc (e *UnmarshalTypeMismatchError) Error() string {\n\tvar ts string\n\tif e.Type == nil {\n\t\tts = \"nil\"\n\t} else {\n\t\tts = e.Type.String()\n\t}\n\treturn fmt.Sprintf(\"Cannot unmarshal %s into Go value of type %s%s\", types.TypeOf(e.Value).Describe(), ts, e.details)\n}\n\nfunc overflowError(v types.Number, t reflect.Type) *UnmarshalTypeMismatchError {\n\treturn &UnmarshalTypeMismatchError{v, t, fmt.Sprintf(\" (%g does not fit in %s)\", v, t)}\n}\n\n// unmarshalNomsError wraps errors from Marshaler.UnmarshalNoms. These should\n// be unwrapped and never leak to the caller of Unmarshal.\ntype unmarshalNomsError struct {\n\terr error\n}\n\nfunc (e *unmarshalNomsError) Error() string {\n\treturn e.err.Error()\n}\n\ntype decoderFunc func(v types.Value, rv reflect.Value)\n\nfunc typeDecoder(t reflect.Type, tags nomsTags) decoderFunc {\n\tif reflect.PtrTo(t).Implements(unmarshalerInterface) {\n\t\treturn marshalerDecoder(t)\n\t}\n\n\tswitch t.Kind() {\n\tcase reflect.Bool:\n\t\treturn boolDecoder\n\tcase reflect.Float32, reflect.Float64:\n\t\treturn floatDecoder\n\tcase reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:\n\t\treturn intDecoder\n\tcase reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:\n\t\treturn uintDecoder\n\tcase reflect.String:\n\t\treturn stringDecoder\n\tcase reflect.Struct:\n\t\treturn structDecoder(t)\n\tcase reflect.Interface:\n\t\treturn interfaceDecoder(t)\n\tcase reflect.Slice:\n\t\treturn sliceDecoder(t)\n\tcase reflect.Array:\n\t\treturn arrayDecoder(t)\n\tcase reflect.Map:\n\t\tif shouldMapDecodeFromSet(t, tags) {\n\t\t\treturn mapFromSetDecoder(t)\n\t\t}\n\t\treturn mapDecoder(t, tags)\n\tcase reflect.Ptr:\n\t\t// Allow implementations of types.Value (like *types.Type)\n\t\tif t.Implements(nomsValueInterface) {\n\t\t\treturn nomsValueDecoder\n\t\t}\n\t\tfallthrough\n\tdefault:\n\t\tpanic(&UnsupportedTypeError{Type: t})\n\t}\n}\n\nfunc boolDecoder(v types.Value, rv reflect.Value) {\n\tif b, ok := v.(types.Bool); ok {\n\t\trv.SetBool(bool(b))\n\t} else {\n\t\tpanic(&UnmarshalTypeMismatchError{v, rv.Type(), \"\"})\n\t}\n}\n\nfunc stringDecoder(v types.Value, rv reflect.Value) {\n\tif s, ok := v.(types.String); ok {\n\t\trv.SetString(string(s))\n\t} else {\n\t\tpanic(&UnmarshalTypeMismatchError{v, rv.Type(), \"\"})\n\t}\n}\n\nfunc floatDecoder(v types.Value, rv reflect.Value) {\n\tif n, ok := v.(types.Number); ok {\n\t\trv.SetFloat(float64(n))\n\t} else {\n\t\tpanic(&UnmarshalTypeMismatchError{v, rv.Type(), \"\"})\n\t}\n}\n\nfunc intDecoder(v types.Value, rv reflect.Value) {\n\tif n, ok := v.(types.Number); ok {\n\t\ti := int64(n)\n\t\tif rv.OverflowInt(i) {\n\t\t\tpanic(overflowError(n, rv.Type()))\n\t\t}\n\t\trv.SetInt(i)\n\t} else {\n\t\tpanic(&UnmarshalTypeMismatchError{v, rv.Type(), \"\"})\n\t}\n}\n\nfunc uintDecoder(v types.Value, rv reflect.Value) {\n\tif n, ok := v.(types.Number); ok {\n\t\tu := uint64(n)\n\t\tif rv.OverflowUint(u) {\n\t\t\tpanic(overflowError(n, rv.Type()))\n\t\t}\n\t\trv.SetUint(u)\n\t} else {\n\t\tpanic(&UnmarshalTypeMismatchError{v, rv.Type(), \"\"})\n\t}\n}\n\ntype decoderCacheT struct {\n\tsync.RWMutex\n\tm map[reflect.Type]decoderFunc\n}\n\nvar decoderCache = &decoderCacheT{}\n\n// Separate Set decoder cache because the same type with and without the\n// `noms:\",set\"` tag decode differently (Set vs Map).\nvar setDecoderCache = &decoderCacheT{}\n\nfunc (c *decoderCacheT) get(t reflect.Type) decoderFunc {\n\tc.RLock()\n\tdefer c.RUnlock()\n\treturn c.m[t]\n}\n\nfunc (c *decoderCacheT) set(t reflect.Type, d decoderFunc) {\n\tc.Lock()\n\tdefer c.Unlock()\n\tif c.m == nil {\n\t\tc.m = map[reflect.Type]decoderFunc{}\n\t}\n\tc.m[t] = d\n}\n\ntype decField struct {\n\tname string\n\tdecoder decoderFunc\n\tindex []int\n\tomitEmpty bool\n\toriginal bool\n}\n\nfunc structDecoderFields(t reflect.Type) []decField {\n\tfields := make([]decField, 0, t.NumField())\n\tfor i := 0; i < t.NumField(); i++ {\n\t\tindex := make([]int, 1)\n\t\tindex[0] = i\n\t\tf := t.Field(i)\n\t\ttags := getTags(f)\n\t\tif tags.skip {\n\t\t\tcontinue\n\t\t}\n\n\t\tif f.Anonymous && f.PkgPath == \"\" && !tags.hasName {\n\t\t\tembeddedFields := structDecoderFields(f.Type)\n\t\t\tfor _, ef := range embeddedFields {\n\t\t\t\tef.index = append(index, ef.index...)\n\t\t\t\tfields = append(fields, ef)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tvalidateField(f, t)\n\n\t\tfields = append(fields, decField{\n\t\t\tname: tags.name,\n\t\t\tdecoder: typeDecoder(f.Type, tags),\n\t\t\tindex: index,\n\t\t\tomitEmpty: tags.omitEmpty,\n\t\t\toriginal: tags.original,\n\t\t})\n\t}\n\treturn fields\n}\n\nfunc structDecoder(t reflect.Type) decoderFunc {\n\tif t.Implements(nomsValueInterface) {\n\t\treturn nomsValueDecoder\n\t}\n\n\td := decoderCache.get(t)\n\tif d != nil {\n\t\treturn d\n\t}\n\n\tfields := structDecoderFields(t)\n\n\td = func(v types.Value, rv reflect.Value) {\n\t\ts, ok := v.(types.Struct)\n\t\tif !ok {\n\t\t\tpanic(&UnmarshalTypeMismatchError{v, rv.Type(), \", expected struct\"})\n\t\t}\n\n\t\tfor _, f := range fields {\n\t\t\tsf := rv.FieldByIndex(f.index)\n\t\t\tif f.original {\n\t\t\t\tif sf.Type() != reflect.TypeOf(s) {\n\t\t\t\t\tpanic(&UnmarshalTypeMismatchError{v, rv.Type(), \", field with tag \\\"original\\\" must have type Struct\"})\n\t\t\t\t}\n\t\t\t\tsf.Set(reflect.ValueOf(s))\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tfv, ok := s.MaybeGet(f.name)\n\t\t\tif ok {\n\t\t\t\tf.decoder(fv, sf)\n\t\t\t} else if !f.omitEmpty {\n\t\t\t\tpanic(&UnmarshalTypeMismatchError{v, rv.Type(), \", missing field \\\"\" + f.name + \"\\\"\"})\n\t\t\t}\n\t\t}\n\t}\n\n\tdecoderCache.set(t, d)\n\treturn d\n}\n\nfunc nomsValueDecoder(v types.Value, rv reflect.Value) {\n\tif !reflect.TypeOf(v).AssignableTo(rv.Type()) {\n\t\tpanic(&UnmarshalTypeMismatchError{v, rv.Type(), \"\"})\n\t}\n\trv.Set(reflect.ValueOf(v))\n}\n\nfunc marshalerDecoder(t reflect.Type) decoderFunc {\n\treturn func(v types.Value, rv reflect.Value) {\n\t\tptr := reflect.New(t)\n\t\terr := ptr.Interface().(Unmarshaler).UnmarshalNoms(v)\n\t\tif err != nil {\n\t\t\tpanic(&unmarshalNomsError{err})\n\t\t}\n\t\trv.Set(ptr.Elem())\n\t}\n}\n\nfunc iterListOrSlice(v types.Value, t reflect.Type, f func(c types.Value, i uint64)) {\n\tswitch v := v.(type) {\n\tcase types.List:\n\t\tv.IterAll(f)\n\tcase types.Set:\n\t\ti := uint64(0)\n\t\tv.IterAll(func(cv types.Value) {\n\t\t\tf(cv, i)\n\t\t\ti++\n\t\t})\n\tdefault:\n\t\tpanic(&UnmarshalTypeMismatchError{v, t, \"\"})\n\t}\n}\n\nfunc sliceDecoder(t reflect.Type) decoderFunc {\n\td := decoderCache.get(t)\n\tif d != nil {\n\t\treturn d\n\t}\n\n\tvar decoder decoderFunc\n\tvar init sync.RWMutex\n\tinit.Lock()\n\tdefer init.Unlock()\n\td = func(v types.Value, rv reflect.Value) {\n\t\tvar slice reflect.Value\n\t\tif rv.IsNil() {\n\t\t\tslice = rv\n\t\t} else {\n\t\t\tslice = rv.Slice(0, 0)\n\t\t}\n\t\tinit.RLock()\n\t\tdefer init.RUnlock()\n\t\titerListOrSlice(v, t, func(v types.Value, _ uint64) {\n\t\t\telemRv := reflect.New(t.Elem()).Elem()\n\t\t\tdecoder(v, elemRv)\n\t\t\tslice = reflect.Append(slice, elemRv)\n\t\t})\n\t\trv.Set(slice)\n\t}\n\n\tdecoderCache.set(t, d)\n\tdecoder = typeDecoder(t.Elem(), nomsTags{})\n\treturn d\n}\n\nfunc arrayDecoder(t reflect.Type) decoderFunc {\n\td := decoderCache.get(t)\n\tif d != nil {\n\t\treturn d\n\t}\n\n\tvar decoder decoderFunc\n\tvar init sync.RWMutex\n\tinit.Lock()\n\tdefer init.Unlock()\n\td = func(v types.Value, rv reflect.Value) {\n\t\tsize := t.Len()\n\t\tlist, ok := v.(types.Collection)\n\t\tif !ok {\n\t\t\tpanic(&UnmarshalTypeMismatchError{v, t, \"\"})\n\t\t}\n\n\t\tl := int(list.Len())\n\t\tif l != size {\n\t\t\tpanic(&UnmarshalTypeMismatchError{v, t, \", length does not match\"})\n\t\t}\n\t\tinit.RLock()\n\t\tdefer init.RUnlock()\n\t\titerListOrSlice(list, t, func(v types.Value, i uint64) {\n\t\t\tdecoder(v, rv.Index(int(i)))\n\t\t})\n\t}\n\n\tdecoderCache.set(t, d)\n\tdecoder = typeDecoder(t.Elem(), nomsTags{})\n\treturn d\n}\n\nfunc mapFromSetDecoder(t reflect.Type) decoderFunc {\n\td := setDecoderCache.get(t)\n\tif d != nil {\n\t\treturn d\n\t}\n\n\tvar decoder decoderFunc\n\tvar init sync.RWMutex\n\tinit.Lock()\n\tdefer init.Unlock()\n\td = func(v types.Value, rv reflect.Value) {\n\t\tm := rv\n\n\t\tnomsSet, ok := v.(types.Set)\n\t\tif !ok {\n\t\t\tpanic(&UnmarshalTypeMismatchError{v, t, `, field has \"set\" tag`})\n\t\t}\n\n\t\tinit.RLock()\n\t\tdefer init.RUnlock()\n\t\tnomsSet.IterAll(func(v types.Value) {\n\t\t\tkeyRv := reflect.New(t.Key()).Elem()\n\t\t\tdecoder(v, keyRv)\n\t\t\tif m.IsNil() {\n\t\t\t\tm = reflect.MakeMap(t)\n\t\t\t}\n\t\t\tm.SetMapIndex(keyRv, reflect.New(t.Elem()).Elem())\n\t\t})\n\t\trv.Set(m)\n\t}\n\n\tsetDecoderCache.set(t, d)\n\tdecoder = typeDecoder(t.Key(), nomsTags{})\n\treturn d\n}\n\nfunc mapDecoder(t reflect.Type, tags nomsTags) decoderFunc {\n\td := decoderCache.get(t)\n\tif d != nil {\n\t\treturn d\n\t}\n\n\tvar keyDecoder decoderFunc\n\tvar valueDecoder decoderFunc\n\tvar init sync.RWMutex\n\tinit.Lock()\n\tdefer init.Unlock()\n\td = func(v types.Value, rv reflect.Value) {\n\t\tm := rv\n\n\t\t// Special case decoding failure if it looks like the \"set\" tag is missing,\n\t\t// because it's helpful.\n\t\tif _, ok := v.(types.Set); ok && !tags.set {\n\t\t\tpanic(&UnmarshalTypeMismatchError{v, t, `, field missing \"set\" tag`})\n\t\t}\n\n\t\tnomsMap, ok := v.(types.Map)\n\t\tif !ok {\n\t\t\tpanic(&UnmarshalTypeMismatchError{v, t, \"\"})\n\t\t}\n\n\t\tinit.RLock()\n\t\tdefer init.RUnlock()\n\t\tnomsMap.IterAll(func(k, v types.Value) {\n\t\t\tkeyRv := reflect.New(t.Key()).Elem()\n\t\t\tkeyDecoder(k, keyRv)\n\t\t\tvalueRv := reflect.New(t.Elem()).Elem()\n\t\t\tvalueDecoder(v, valueRv)\n\t\t\tif m.IsNil() {\n\t\t\t\tm = reflect.MakeMap(t)\n\t\t\t}\n\t\t\tm.SetMapIndex(keyRv, valueRv)\n\t\t})\n\t\trv.Set(m)\n\t}\n\n\tdecoderCache.set(t, d)\n\tkeyDecoder = typeDecoder(t.Key(), nomsTags{})\n\tvalueDecoder = typeDecoder(t.Elem(), nomsTags{})\n\treturn d\n}\n\nfunc interfaceDecoder(t reflect.Type) decoderFunc {\n\tif t.Implements(nomsValueInterface) {\n\t\treturn nomsValueDecoder\n\t}\n\n\tif t != emptyInterface {\n\t\tpanic(&UnsupportedTypeError{Type: t})\n\t}\n\n\treturn func(v types.Value, rv reflect.Value) {\n\t\t// TODO: Go directly from value to go type\n\t\tt := getGoTypeForNomsType(types.TypeOf(v), rv.Type(), v)\n\t\ti := reflect.New(t).Elem()\n\t\ttypeDecoder(t, nomsTags{})(v, i)\n\t\trv.Set(i)\n\t}\n}\n\nfunc getGoTypeForNomsType(nt *types.Type, rt reflect.Type, v types.Value) reflect.Type {\n\tswitch nt.TargetKind() {\n\tcase types.BoolKind:\n\t\treturn reflect.TypeOf(false)\n\tcase types.NumberKind:\n\t\treturn reflect.TypeOf(float64(0))\n\tcase types.StringKind:\n\t\treturn reflect.TypeOf(\"\")\n\tcase types.ListKind, types.SetKind:\n\t\tet := getGoTypeForNomsType(nt.Desc.(types.CompoundDesc).ElemTypes[0], rt, v)\n\t\treturn reflect.SliceOf(et)\n\tcase types.MapKind:\n\t\tkt := getGoTypeForNomsType(nt.Desc.(types.CompoundDesc).ElemTypes[0], rt, v)\n\t\tvt := getGoTypeForNomsType(nt.Desc.(types.CompoundDesc).ElemTypes[1], rt, v)\n\t\treturn reflect.MapOf(kt, vt)\n\tcase types.UnionKind:\n\t\t// Visit union types to raise potential errors\n\t\tfor _, ut := range nt.Desc.(types.CompoundDesc).ElemTypes {\n\t\t\tgetGoTypeForNomsType(ut, rt, v)\n\t\t}\n\t\treturn emptyInterface\n\t// case types.StructKind:\n\t// \treflect.StructOf was not added until Go 1.7\n\tdefault:\n\t\tpanic(&UnmarshalTypeMismatchError{Value: v, Type: rt})\n\t}\n}\n\nfunc shouldMapDecodeFromSet(rt reflect.Type, tags nomsTags) bool {\n\t// map[T]struct{} `noms:,\"set\"`\n\treturn tags.set &&\n\t\trt.Elem().Kind() == reflect.Struct &&\n\t\trt.Elem().NumField() == 0\n}\n" }, { "path": "go/marshal/decode_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage marshal\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"fmt\"\n\t\"math\"\n\t\"reflect\"\n\t\"regexp\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestDecode(tt *testing.T) {\n\tassert := assert.New(tt)\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tt := func(v types.Value, ptr interface{}, expected interface{}) {\n\t\tp := reflect.ValueOf(ptr)\n\t\tassert.Equal(reflect.Ptr, p.Type().Kind())\n\t\terr := Unmarshal(v, p.Interface())\n\t\tassert.NoError(err)\n\t\tif expectedValue, ok := expected.(types.Value); ok {\n\t\t\tassert.True(expectedValue.Equals(p.Elem().Interface().(types.Value)))\n\t\t} else {\n\t\t\tassert.Equal(expected, p.Elem().Interface())\n\t\t}\n\n\t\t// Also test that types.Value is passed through\n\t\tvar v2 types.Value\n\t\terr = Unmarshal(v, &v2)\n\t\tassert.NoError(err)\n\t\tassert.True(v.Equals(v2))\n\t}\n\n\tfor _, n := range []float32{0, 42, 3.14159265359, math.MaxFloat32} {\n\t\tvar f32 float32\n\t\tt(types.Number(n), &f32, float32(n))\n\t}\n\n\tfor _, n := range []float64{0, 42, 3.14159265359, math.MaxFloat64} {\n\t\tvar f64 float64\n\t\tt(types.Number(n), &f64, float64(n))\n\t}\n\n\tfor _, n := range []int8{0, 42, math.MaxInt8} {\n\t\tvar i8 int8\n\t\tt(types.Number(n), &i8, int8(n))\n\t}\n\n\tfor _, n := range []int16{0, 42, math.MaxInt16} {\n\t\tvar i16 int16\n\t\tt(types.Number(n), &i16, int16(n))\n\t}\n\n\tfor _, n := range []int32{0, 42, math.MaxInt32} {\n\t\tvar i32 int32\n\t\tt(types.Number(n), &i32, int32(n))\n\t}\n\n\t// int is at least int32\n\tfor _, n := range []int{0, 42, math.MaxInt32} {\n\t\tvar i int\n\t\tt(types.Number(n), &i, int(n))\n\t}\n\n\t// There is precision loss for values above Math.pow(2, 53) - 1\n\tfor _, n := range []int64{0, 42, int64(math.Pow(2, 53) - 1)} {\n\t\tvar i64 int64\n\t\tt(types.Number(n), &i64, int64(n))\n\t}\n\n\tfor _, n := range []uint8{0, 42, math.MaxUint8} {\n\t\tvar ui8 uint8\n\t\tt(types.Number(n), &ui8, uint8(n))\n\t}\n\n\tfor _, n := range []uint16{0, 42, math.MaxUint16} {\n\t\tvar ui16 uint16\n\t\tt(types.Number(n), &ui16, uint16(n))\n\t}\n\n\tfor _, n := range []uint32{0, 42, math.MaxInt32} {\n\t\tvar ui32 uint32\n\t\tt(types.Number(n), &ui32, uint32(n))\n\t}\n\n\t// uint is at least uint32\n\tfor _, n := range []uint{0, 42, math.MaxInt32} {\n\t\tvar ui uint\n\t\tt(types.Number(n), &ui, uint(n))\n\t}\n\n\t// There is precision loss for values above Math.pow(2, 53) - 1\n\tfor _, n := range []uint64{0, 42, uint64(math.Pow(2, 53) - 1)} {\n\t\tvar ui64 uint64\n\t\tt(types.Number(n), &ui64, uint64(n))\n\t}\n\n\tvar b bool\n\tt(types.Bool(true), &b, true)\n\tt(types.Bool(false), &b, false)\n\n\tfor _, s := range []string{\"\", \"s\", \"hello\", \"💩\"} {\n\t\tvar s2 string\n\t\tt(types.String(s), &s2, s)\n\t}\n\n\tvar list types.List\n\tlist2 := types.NewList(vs, types.Number(42))\n\tt(list2, &list, list2)\n\n\tvar m types.Map\n\tmap2 := types.NewMap(vs, types.Number(42), types.String(\"Hi\"))\n\tt(map2, &m, map2)\n\n\tvar set types.Set\n\tset2 := types.NewSet(vs, types.String(\"Bye\"))\n\tt(set2, &set, set2)\n\n\tvar blob types.Blob\n\tblob2 := types.NewBlob(vs, bytes.NewBufferString(\"hello\"))\n\tt(blob2, &blob, blob2)\n\n\ttype TestStruct struct {\n\t\tB bool\n\t\tA float64\n\t\tC string\n\t}\n\tvar ts TestStruct\n\tt(types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\"b\": types.Bool(true),\n\t\t\"a\": types.Number(42),\n\t\t\"c\": types.String(\"hi\"),\n\t}), &ts, TestStruct{true, 42, \"hi\"})\n\t// again to test the caching\n\tt(types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\"b\": types.Bool(false),\n\t\t\"a\": types.Number(555),\n\t\t\"c\": types.String(\"hello\"),\n\t}), &ts, TestStruct{false, 555, \"hello\"})\n\n\tvar as struct {\n\t\tX int32\n\t\tY bool\n\t}\n\tt(types.NewStruct(\"\", types.StructData{\n\t\t\"y\": types.Bool(true),\n\t\t\"x\": types.Number(42),\n\t}), &as, struct {\n\t\tX int32\n\t\tY bool\n\t}{\n\t\t42,\n\t\ttrue,\n\t})\n\n\t// extra fields\n\ttype T3 struct {\n\t\tB string\n\t}\n\tvar t3 T3\n\tt(types.NewStruct(\"T3\", types.StructData{\n\t\t\"b\": types.String(\"abc\"),\n\t\t\"a\": types.Number(42),\n\t}), &t3, T3{\"abc\"})\n\n\t// Case of struct name is not relevant when unmarshalling.\n\ttype aBc struct {\n\t\tE bool\n\t}\n\tvar t4 aBc\n\tt(types.NewStruct(\"abc\", types.StructData{\n\t\t\"e\": types.Bool(true),\n\t}), &t4, aBc{true})\n\tt(types.NewStruct(\"Abc\", types.StructData{\n\t\t\"e\": types.Bool(false),\n\t}), &t4, aBc{false})\n\n\t// Name of struct is irrelevant to unmarshalling structs.\n\ttype SomeOtherName struct {\n\t\tA int\n\t}\n\tvar t5 SomeOtherName\n\tt(types.NewStruct(\"aeiou\", types.StructData{\n\t\t\"a\": types.Number(42),\n\t}), &t5, SomeOtherName{42})\n\n\tvar t6 SomeOtherName\n\tt(types.NewStruct(\"SomeOtherName\", types.StructData{\n\t\t\"a\": types.Number(42),\n\t}), &t6, SomeOtherName{42})\n\n\tvar t7 struct {\n\t\tA int\n\t}\n\tt(types.NewStruct(\"SomeOtherName\", types.StructData{\n\t\t\"a\": types.Number(42),\n\t}), &t7, struct{ A int }{42})\n}\n\nfunc TestDecodeStructWithNomsValue(t *testing.T) {\n\t// This is split out of TestDecode because we cannot use testify Equal\n\t// on a go struct with a field that is a Noms value.\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype TestStruct struct {\n\t\tB bool\n\t\tA float64\n\t\tC string\n\t}\n\ttype T2 struct {\n\t\tAbc TestStruct\n\t\tDef types.List\n\t}\n\n\tv := types.NewStruct(\"T2\", types.StructData{\n\t\t\"abc\": types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\t\"a\": types.Number(1),\n\t\t\t\"b\": types.Bool(false),\n\t\t\t\"c\": types.String(\"bye\"),\n\t\t}),\n\t\t\"def\": types.NewList(vs, types.Number(42)),\n\t})\n\tvar t2 T2\n\tMustUnmarshal(v, &t2)\n\tassert.IsType(t, T2{}, t2)\n\tassert.Equal(t, TestStruct{false, 1, \"bye\"}, t2.Abc)\n\tassert.True(t, t2.Def.Equals(types.NewList(vs, types.Number(42))))\n}\n\nfunc TestDecodeNilPointer(t *testing.T) {\n\tvar x *bool\n\tassertDecodeErrorMessage(t, types.Bool(true), x, \"Cannot unmarshal into Go nil pointer of type *bool\")\n}\n\nfunc TestDecodeNonPointer(t *testing.T) {\n\tb := true\n\tassertDecodeErrorMessage(t, types.Bool(true), b, \"Cannot unmarshal into Go non pointer of type bool\")\n}\n\nfunc TestDecodeNil(t *testing.T) {\n\terr := Unmarshal(types.Bool(true), nil)\n\tassert.Error(t, err)\n\tassert.Equal(t, \"Cannot unmarshal into Go nil value\", err.Error())\n}\n\nfunc newTestValueStore() *types.ValueStore {\n\tst := &chunks.TestStorage{}\n\treturn types.NewValueStore(st.NewView())\n}\n\nfunc TestDecodeTypeMismatch(t *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar b bool\n\tassertDecodeErrorMessage(t, types.Number(42), &b, \"Cannot unmarshal Number into Go value of type bool\")\n\n\tvar blob types.Blob\n\tassertDecodeErrorMessage(t, types.NewList(vs), &blob, \"Cannot unmarshal List<> into Go value of type types.Blob\")\n\n\ttype S struct {\n\t\tX int\n\t}\n\tvar s S\n\tassertDecodeErrorMessage(t, types.String(\"hi!\"), &s, \"Cannot unmarshal String into Go value of type marshal.S, expected struct\")\n\tassertDecodeErrorMessage(t, types.NewStruct(\"S\", types.StructData{\n\t\t\"x\": types.String(\"hi\"),\n\t}), &s, \"Cannot unmarshal String into Go value of type int\")\n}\n\nfunc assertDecodeErrorMessage(t *testing.T, v types.Value, ptr interface{}, msg string) {\n\tp := reflect.ValueOf(ptr)\n\terr := Unmarshal(v, p.Interface())\n\tassert.Error(t, err)\n\tassert.Equal(t, msg, err.Error())\n}\n\nfunc TestDecodeInvalidTypes(tt *testing.T) {\n\tt := func(p interface{}, ts string) {\n\t\tassertDecodeErrorMessage(tt, types.Number(42), p, \"Type is not supported, type: \"+ts)\n\t}\n\n\tvar ptr *bool\n\tt(&ptr, \"*bool\")\n\n\tvar c chan bool\n\tt(&c, \"chan bool\")\n\n\ttype Nested struct {\n\t\tX *bool\n\t}\n\tvar n Nested\n\tt(&n, \"*bool\")\n}\n\nfunc TestDecodeOverflows(tt *testing.T) {\n\tt := func(p interface{}, n float64, ts string) {\n\t\tassertDecodeErrorMessage(tt, types.Number(n), p, fmt.Sprintf(\"Cannot unmarshal Number into Go value of type %s (%g does not fit in %s)\", ts, n, ts))\n\t}\n\n\tvar ui8 uint8\n\tt(&ui8, 256, \"uint8\")\n\tt(&ui8, -1, \"uint8\")\n\n\tvar ui16 uint16\n\tt(&ui16, math.Pow(2, 16), \"uint16\")\n\tt(&ui16, -1, \"uint16\")\n\n\tvar ui32 uint32\n\tt(&ui32, math.Pow(2, 32), \"uint32\")\n\tt(&ui32, -1, \"uint32\")\n\n\tvar i8 int8\n\tt(&i8, 128, \"int8\")\n\tt(&i8, -128-1, \"int8\")\n\n\tvar i16 int16\n\tt(&i16, math.Pow(2, 15), \"int16\")\n\tt(&i16, -math.Pow(2, 15)-1, \"int16\")\n\n\tvar i32 int32\n\tt(&i32, math.Pow(2, 31), \"int32\")\n\tt(&i32, -math.Pow(2, 31)-1, \"int32\")\n}\n\nfunc TestDecodeMissingField(t *testing.T) {\n\ttype S struct {\n\t\tA int32\n\t\tB bool\n\t}\n\tvar s S\n\tassertDecodeErrorMessage(t, types.NewStruct(\"S\", types.StructData{\n\t\t\"a\": types.Number(42),\n\t}), &s, \"Cannot unmarshal Struct S {\\n a: Number,\\n} into Go value of type marshal.S, missing field \\\"b\\\"\")\n}\n\nfunc TestDecodeEmbeddedStruct(tt *testing.T) {\n\tassert := assert.New(tt)\n\n\ttype EmbeddedStruct struct {\n\t\tX int\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct\n\t}\n\tvar ts TestStruct\n\terr := Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"x\": types.Number(1),\n\t}), &ts)\n\tassert.NoError(err)\n\tassert.Equal(TestStruct{EmbeddedStruct{1}}, ts)\n\n\ttype OuterTest struct {\n\t\tY bool\n\t\tTestStruct\n\t}\n\tvar ts2 OuterTest\n\terr = Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"x\": types.Number(2),\n\t\t\"y\": types.Bool(true),\n\t}), &ts2)\n\tassert.NoError(err)\n\tassert.Equal(OuterTest{true, TestStruct{EmbeddedStruct{2}}}, ts2)\n}\n\nfunc TestDecodeEmbeddedStructSkip(tt *testing.T) {\n\tassert := assert.New(tt)\n\n\ttype EmbeddedStruct struct {\n\t\tX int\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct `noms:\"-\"`\n\t\tY int\n\t}\n\tts := TestStruct{EmbeddedStruct: EmbeddedStruct{42}}\n\terr := Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"y\": types.Number(2),\n\t}), &ts)\n\tassert.NoError(err)\n\tassert.Equal(TestStruct{EmbeddedStruct{42}, 2}, ts)\n}\n\nfunc TestDecodeEmbeddedStructNamed(tt *testing.T) {\n\tassert := assert.New(tt)\n\n\ttype EmbeddedStruct struct {\n\t\tX int\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct `noms:\"em\"`\n\t\tY int\n\t}\n\tts := TestStruct{EmbeddedStruct: EmbeddedStruct{42}}\n\terr := Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"em\": types.NewStruct(\"S\", types.StructData{\n\t\t\t\"x\": types.Number(1),\n\t\t}),\n\t\t\"y\": types.Number(2),\n\t}), &ts)\n\tassert.NoError(err)\n\tassert.Equal(TestStruct{EmbeddedStruct{1}, 2}, ts)\n}\n\nfunc TestDecodeEmbeddedStructOriginal(tt *testing.T) {\n\tassert := assert.New(tt)\n\n\ttype EmbeddedStruct struct {\n\t\tX int\n\t\tO types.Struct `noms:\",original\"`\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct\n\t}\n\tvar ts TestStruct\n\tnomsStruct := types.NewStruct(\"S\", types.StructData{\n\t\t\"x\": types.Number(1),\n\t})\n\terr := Unmarshal(nomsStruct, &ts)\n\tassert.NoError(err)\n\texpected := TestStruct{\n\t\tEmbeddedStruct: EmbeddedStruct{\n\t\t\tX: 1,\n\t\t\tO: nomsStruct,\n\t\t},\n\t}\n\tassert.Equal(expected, ts)\n}\n\nfunc TestDecodeNonExportedField(tt *testing.T) {\n\ttype TestStruct struct {\n\t\tx int\n\t}\n\tvar ts TestStruct\n\tassertDecodeErrorMessage(tt, types.String(\"hi\"), &ts, \"Non exported fields are not supported, type: marshal.TestStruct\")\n}\n\nfunc TestDecodeTaggingSkip(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tA int32 `noms:\"-\"`\n\t\tB bool\n\t}\n\tvar s S\n\terr := Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"b\": types.Bool(true),\n\t}), &s)\n\tassert.NoError(err)\n\tassert.Equal(S{0, true}, s)\n\n\tvar s2 S\n\tUnmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"a\": types.Number(42),\n\t\t\"b\": types.Bool(true),\n\t}), &s2)\n\tassert.Equal(S{0, true}, s2)\n\n\ts3 := S{555, true}\n\terr = Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"a\": types.Number(42),\n\t\t\"b\": types.Bool(false),\n\t}), &s3)\n\tassert.NoError(err)\n\tassert.Equal(S{555, false}, s3)\n}\n\nfunc TestDecodeNamedFields(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tAaa int `noms:\"a\"`\n\t\tBbb bool `noms:\"B\"`\n\t\tCcc string\n\t}\n\tvar s S\n\terr := Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"a\": types.Number(42),\n\t\t\"B\": types.Bool(true),\n\t\t\"ccc\": types.String(\"Hi\"),\n\t}), &s)\n\tassert.NoError(err)\n\tassert.Equal(S{42, true, \"Hi\"}, s)\n}\n\nfunc TestDecodeInvalidNamedFields(t *testing.T) {\n\ttype S struct {\n\t\tA int `noms:\"1a\"`\n\t}\n\tvar s S\n\tassertDecodeErrorMessage(t, types.NewStruct(\"S\", types.StructData{\n\t\t\"a\": types.Number(42),\n\t}), &s, \"Invalid struct field name: 1a\")\n}\n\nfunc TestDecodeInvalidNomsType(t *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tA types.List\n\t}\n\tvar s S\n\tassertDecodeErrorMessage(t, types.NewStruct(\"S\", types.StructData{\n\t\t\"a\": types.NewMap(vs, types.String(\"A\"), types.Number(1)),\n\t}), &s, \"Cannot unmarshal Map into Go value of type types.List\")\n}\n\nfunc TestDecodeNomsTypePtr(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttestUnmarshal := func(v types.Value, dest interface{}, expected interface{}) {\n\t\terr := Unmarshal(v, dest)\n\t\tassert.NoError(err)\n\t\tassert.Equal(expected, dest)\n\t}\n\n\ttype S struct{ Type *types.Type }\n\tvar s S\n\n\tprimitive := types.StringType\n\ttestUnmarshal(types.NewStruct(\"S\", types.StructData{\"type\": primitive}), &s, &S{primitive})\n\n\tcomplex := types.MakeStructType(\"Complex\",\n\t\ttypes.StructField{\n\t\t\tName: \"stuff\",\n\t\t\tType: types.StringType,\n\t\t},\n\t)\n\ttestUnmarshal(types.NewStruct(\"S\", types.StructData{\"type\": complex}), &s, &S{complex})\n}\n\nfunc ExampleUnmarshal() {\n\ttype Person struct {\n\t\tGiven string\n\t\tMale bool\n\t}\n\tvar rickon Person\n\terr := Unmarshal(types.NewStruct(\"Person\", types.StructData{\n\t\t\"given\": types.String(\"Rickon\"),\n\t\t\"male\": types.Bool(true),\n\t}), &rickon)\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n\n\tfmt.Printf(\"Given: %s, Male: %t\\n\", rickon.Given, rickon.Male)\n\t// Output: Given: Rickon, Male: true\n}\n\nfunc TestDecodeSlice(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar s []string\n\n\terr := Unmarshal(types.NewList(vs, types.String(\"a\"), types.String(\"b\"), types.String(\"c\")), &s)\n\tassert.NoError(err)\n\tassert.Equal([]string{\"a\", \"b\", \"c\"}, s)\n\n\terr = Unmarshal(types.NewSet(vs, types.String(\"a\"), types.String(\"b\"), types.String(\"c\")), &s)\n\tassert.NoError(err)\n\tassert.Equal([]string{\"a\", \"b\", \"c\"}, s)\n}\n\nfunc TestDecodeSliceEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar s []string\n\n\terr := Unmarshal(types.NewList(vs), &s)\n\tassert.NoError(err)\n\tassert.Equal([]string(nil), s)\n\n\terr = Unmarshal(types.NewSet(vs), &s)\n\tassert.NoError(err)\n\tassert.Equal([]string(nil), s)\n\n\ts2 := []string{}\n\terr = Unmarshal(types.NewList(vs), &s2)\n\tassert.NoError(err)\n\tassert.Equal([]string{}, s2)\n\n\terr = Unmarshal(types.NewSet(vs), &s2)\n\tassert.NoError(err)\n\tassert.Equal([]string{}, s2)\n}\n\nfunc TestDecodeSliceReuse(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ts := []string{\"A\", \"B\", \"C\", \"D\"}\n\ts2 := s[1:3]\n\terr := Unmarshal(types.NewList(vs, types.String(\"a\"), types.String(\"b\")), &s)\n\tassert.NoError(err)\n\tassert.Equal([]string{\"a\", \"b\"}, s)\n\tassert.Equal([]string{\"b\", \"C\"}, s2)\n\n\terr = Unmarshal(types.NewSet(vs, types.String(\"a\"), types.String(\"b\")), &s)\n\tassert.NoError(err)\n\tassert.Equal([]string{\"a\", \"b\"}, s)\n\tassert.Equal([]string{\"b\", \"C\"}, s2)\n}\n\nfunc TestDecodeArray(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ts := [3]string{\"\", \"\", \"\"}\n\n\terr := Unmarshal(types.NewList(vs, types.String(\"a\"), types.String(\"b\"), types.String(\"c\")), &s)\n\tassert.NoError(err)\n\tassert.Equal([3]string{\"a\", \"b\", \"c\"}, s)\n\n\terr = Unmarshal(types.NewSet(vs, types.String(\"a\"), types.String(\"b\"), types.String(\"c\")), &s)\n\tassert.NoError(err)\n\tassert.Equal([3]string{\"a\", \"b\", \"c\"}, s)\n}\n\nfunc TestDecodeArrayEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar s [0]string\n\n\terr := Unmarshal(types.NewList(vs), &s)\n\tassert.NoError(err)\n\tassert.Equal([0]string{}, s)\n\n\terr = Unmarshal(types.NewSet(vs), &s)\n\tassert.NoError(err)\n\tassert.Equal([0]string{}, s)\n}\n\nfunc TestDecodeStructWithSlice(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tList []int\n\t}\n\tvar s S\n\terr := Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"list\": types.NewList(vs, types.Number(1), types.Number(2), types.Number(3)),\n\t}), &s)\n\tassert.NoError(err)\n\tassert.Equal(S{[]int{1, 2, 3}}, s)\n\n\terr = Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"list\": types.NewSet(vs, types.Number(1), types.Number(2), types.Number(3)),\n\t}), &s)\n\tassert.NoError(err)\n\tassert.Equal(S{[]int{1, 2, 3}}, s)\n}\n\nfunc TestDecodeStructWithArrayOfNomsValue(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tList [1]types.Set\n\t}\n\tvar s S\n\terr := Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"list\": types.NewList(vs, types.NewSet(vs, types.Bool(true))),\n\t}), &s)\n\tassert.NoError(err)\n\tassert.Equal(S{[1]types.Set{types.NewSet(vs, types.Bool(true))}}, s)\n}\n\nfunc TestDecodeWrongArrayLength(t *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar l [2]string\n\tassertDecodeErrorMessage(t, types.NewList(vs, types.String(\"hi\")), &l, \"Cannot unmarshal List into Go value of type [2]string, length does not match\")\n}\n\nfunc TestDecodeWrongArrayType(t *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar l [1]string\n\tassertDecodeErrorMessage(t, types.NewList(vs, types.Number(1)), &l, \"Cannot unmarshal Number into Go value of type string\")\n}\n\nfunc TestDecodeWrongSliceType(t *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar l []string\n\tassertDecodeErrorMessage(t, types.NewList(vs, types.Number(1)), &l, \"Cannot unmarshal Number into Go value of type string\")\n}\n\nfunc TestDecodeSliceWrongNomsType(t *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar l []string\n\tassertDecodeErrorMessage(t, types.NewMap(vs, types.String(\"a\"), types.Number(1)), &l, \"Cannot unmarshal Map into Go value of type []string\")\n}\n\nfunc TestDecodeArrayWrongNomsType(t *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar l [1]string\n\tassertDecodeErrorMessage(t, types.NewMap(vs, types.String(\"a\"), types.Number(1)), &l, \"Cannot unmarshal Map into Go value of type [1]string\")\n}\n\nfunc TestDecodeRecursive(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype Node struct {\n\t\tValue int\n\t\tChildren []Node\n\t}\n\n\tv := types.NewStruct(\"Node\", types.StructData{\n\t\t\"children\": types.NewList(\n\t\t\tvs,\n\t\t\ttypes.NewStruct(\"Node\", types.StructData{\n\t\t\t\t\"children\": types.NewList(vs),\n\t\t\t\t\"value\": types.Number(2),\n\t\t\t}),\n\t\t\ttypes.NewStruct(\"Node\", types.StructData{\n\t\t\t\t\"children\": types.NewList(vs),\n\t\t\t\t\"value\": types.Number(3),\n\t\t\t}),\n\t\t),\n\t\t\"value\": types.Number(1),\n\t})\n\n\tvar n Node\n\terr := Unmarshal(v, &n)\n\tassert.NoError(err)\n\n\tassert.Equal(Node{\n\t\t1, []Node{\n\t\t\t{2, nil},\n\t\t\t{3, nil},\n\t\t},\n\t}, n)\n}\n\nfunc TestDecodeMap(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar m map[string]int\n\n\ttestMap := types.NewMap(\n\t\tvs,\n\t\ttypes.String(\"a\"), types.Number(1),\n\t\ttypes.String(\"b\"), types.Number(2),\n\t\ttypes.String(\"c\"), types.Number(3))\n\texpectedMap := map[string]int{\"a\": 1, \"b\": 2, \"c\": 3}\n\terr := Unmarshal(testMap, &m)\n\tassert.NoError(err)\n\tassert.Equal(expectedMap, m)\n\n\tm = map[string]int{\"b\": 2, \"c\": 333}\n\terr = Unmarshal(types.NewMap(\n\t\tvs,\n\t\ttypes.String(\"a\"), types.Number(1),\n\t\ttypes.String(\"c\"), types.Number(3)), &m)\n\tassert.NoError(err)\n\tassert.Equal(expectedMap, m)\n\n\ttype S struct {\n\t\tN string\n\t}\n\n\tvar m2 map[S]bool\n\terr = Unmarshal(types.NewMap(\n\t\tvs,\n\t\ttypes.NewStruct(\"S\", types.StructData{\"n\": types.String(\"Yes\")}), types.Bool(true),\n\t\ttypes.NewStruct(\"S\", types.StructData{\"n\": types.String(\"No\")}), types.Bool(false)), &m2)\n\tassert.NoError(err)\n\tassert.Equal(map[S]bool{S{\"Yes\"}: true, S{\"No\"}: false}, m2)\n}\n\nfunc TestDecodeMapEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar m map[string]int\n\terr := Unmarshal(types.NewMap(vs), &m)\n\tassert.NoError(err)\n\tassert.Equal(map[string]int(nil), m)\n\n\tm2 := map[string]int{}\n\terr = Unmarshal(types.NewMap(vs), &m2)\n\tassert.NoError(err)\n\tassert.Equal(map[string]int{}, m2)\n}\n\nfunc TestDecodeMapWrongNomsType(t *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar m map[string]int\n\tassertDecodeErrorMessage(t, types.NewList(vs, types.String(\"a\"), types.Number(1)), &m, \"Cannot unmarshal List into Go value of type map[string]int\")\n}\n\nfunc TestDecodeOntoInterface(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar i interface{}\n\terr := Unmarshal(types.Number(1), &i)\n\tassert.NoError(err)\n\tassert.Equal(float64(1), i)\n\n\terr = Unmarshal(types.String(\"abc\"), &i)\n\tassert.NoError(err)\n\tassert.Equal(\"abc\", i)\n\n\terr = Unmarshal(types.Bool(true), &i)\n\tassert.NoError(err)\n\tassert.Equal(true, i)\n\n\terr = Unmarshal(types.NewList(vs, types.String(\"abc\")), &i)\n\tassert.NoError(err)\n\tassert.Equal([]string{\"abc\"}, i)\n\n\terr = Unmarshal(types.NewMap(vs, types.String(\"abc\"), types.Number(1)), &i)\n\tassert.NoError(err)\n\tassert.Equal(map[string]float64{\"abc\": float64(1)}, i)\n\n\terr = Unmarshal(types.NewList(vs, types.String(\"a\"), types.Bool(true), types.Number(42)), &i)\n\tassert.NoError(err)\n\tassert.Equal([]interface{}{\"a\", true, float64(42)}, i)\n\n\terr = Unmarshal(types.NewMap(vs, types.String(\"a\"), types.Bool(true), types.Number(42), types.NewList(vs)), &i)\n\tassert.NoError(err)\n\tassert.Equal(map[interface{}]interface{}{\"a\": true, float64(42): []interface{}(nil)}, i)\n}\n\nfunc TestDecodeOntoNonSupportedInterface(t *testing.T) {\n\ttype I interface {\n\t\tM() int\n\t}\n\tvar i I\n\tassertDecodeErrorMessage(t, types.Number(1), &i, \"Type is not supported, type: marshal.I\")\n}\n\nfunc TestDecodeOntoInterfaceStruct(t *testing.T) {\n\t// Not implemented because it requires Go 1.7.\n\tvar i interface{}\n\tassertDecodeErrorMessage(t, types.NewStruct(\"\", types.StructData{}), &i, \"Cannot unmarshal Struct {} into Go value of type interface {}\")\n}\n\nfunc TestDecodeSet(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype T struct {\n\t\tA map[int]struct{} `noms:\",set\"`\n\t\tB map[int]struct{}\n\t\tC map[string]struct{} `noms:\",set\"`\n\t\tD map[string]struct{}\n\t\tE []int\n\t\tF []int `noms:\",set\"`\n\t\tG []int\n\t}\n\n\tns := types.NewStruct(\"T\", types.StructData{\n\t\t\"a\": types.NewSet(vs, types.Number(0), types.Number(1), types.Number(2)),\n\t\t\"b\": types.NewMap(vs, types.Number(3), types.EmptyStruct, types.Number(4), types.EmptyStruct, types.Number(5), types.EmptyStruct),\n\t\t\"c\": types.NewSet(vs, types.String(\"0\"), types.String(\"1\"), types.String(\"2\")),\n\t\t\"d\": types.NewMap(vs, types.String(\"3\"), types.EmptyStruct, types.String(\"4\"), types.EmptyStruct, types.String(\"5\"), types.EmptyStruct),\n\t\t\"e\": types.NewSet(vs, types.Number(6), types.Number(7), types.Number(8)),\n\t\t\"f\": types.NewSet(vs, types.Number(9), types.Number(10), types.Number(11)),\n\t\t\"g\": types.NewList(vs, types.Number(12), types.Number(13), types.Number(14)),\n\t})\n\n\tgs := T{}\n\tassert.NoError(Unmarshal(ns, &gs))\n\tassert.Equal(T{\n\t\tA: map[int]struct{}{0: {}, 1: {}, 2: {}},\n\t\tB: map[int]struct{}{3: {}, 4: {}, 5: {}},\n\t\tC: map[string]struct{}{\"0\": {}, \"1\": {}, \"2\": {}},\n\t\tD: map[string]struct{}{\"3\": {}, \"4\": {}, \"5\": {}},\n\t\tE: []int{6, 7, 8},\n\t\tF: []int{9, 10, 11},\n\t\tG: []int{12, 13, 14},\n\t}, gs)\n\n\tns2 := types.NewStruct(\"T\", types.StructData{\n\t\t\"a\": types.NewSet(vs),\n\t\t\"b\": types.NewMap(vs),\n\t\t\"c\": types.NewSet(vs),\n\t\t\"d\": types.NewMap(vs),\n\t\t\"e\": types.NewSet(vs),\n\t\t\"f\": types.NewSet(vs),\n\t\t\"g\": types.NewList(vs),\n\t})\n\n\tgs2 := T{\n\t\tA: map[int]struct{}{},\n\t}\n\tassert.NoError(Unmarshal(ns2, &gs2))\n\tassert.Equal(T{\n\t\tA: map[int]struct{}{},\n\t}, gs2)\n}\n\nfunc TestDecodeOpt(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttc := []struct {\n\t\tin types.Value\n\t\topt Opt\n\t\tonto interface{}\n\t\twantValue interface{}\n\t\twantError string\n\t}{\n\t\t{\n\t\t\ttypes.NewSet(vs, types.String(\"a\"), types.String(\"b\")),\n\t\t\tOpt{},\n\t\t\t&[]string{},\n\t\t\t&[]string{\"a\", \"b\"},\n\t\t\t\"\",\n\t\t},\n\t\t{\n\t\t\ttypes.NewSet(vs, types.String(\"a\"), types.String(\"b\")),\n\t\t\tOpt{Set: true},\n\t\t\t&[]string{},\n\t\t\t&[]string{\"a\", \"b\"},\n\t\t\t\"\",\n\t\t},\n\t\t{\n\t\t\ttypes.NewSet(vs, types.String(\"a\"), types.String(\"b\")),\n\t\t\tOpt{Set: true},\n\t\t\t&map[string]struct{}{},\n\t\t\t&map[string]struct{}{\"a\": struct{}{}, \"b\": struct{}{}},\n\t\t\t\"\",\n\t\t},\n\t\t{\n\t\t\ttypes.NewSet(vs, types.String(\"a\"), types.String(\"b\")),\n\t\t\tOpt{},\n\t\t\t&map[string]struct{}{},\n\t\t\t&map[string]struct{}{},\n\t\t\t\"Cannot unmarshal Set into Go value of type map[string]struct {}, field missing \\\"set\\\" tag\",\n\t\t},\n\t}\n\n\tfor _, t := range tc {\n\t\terr := UnmarshalOpt(t.in, t.opt, t.onto)\n\t\tassert.Equal(t.wantValue, t.onto)\n\t\tif t.wantError == \"\" {\n\t\t\tassert.Nil(err)\n\t\t} else {\n\t\t\tassert.Equal(t.wantError, err.Error())\n\t\t}\n\t}\n}\n\nfunc TestDecodeNamedSet(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype T struct {\n\t\tA map[int]struct{} `noms:\"foo,set\"`\n\t}\n\n\tns := types.NewStruct(\"T\", types.StructData{\n\t\t\"a\": types.NewSet(vs, types.Number(0)),\n\t\t\"foo\": types.NewSet(vs, types.Number(1)),\n\t})\n\n\tgs := T{}\n\tassert.NoError(Unmarshal(ns, &gs))\n\tassert.Equal(T{\n\t\tmap[int]struct{}{1: {}},\n\t}, gs)\n}\n\nfunc TestDecodeSetWrongMapType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype T1 struct {\n\t\tA map[int]int `noms:\",set\"`\n\t}\n\n\terr := Unmarshal(types.NewStruct(\"T1\", types.StructData{\n\t\t\"a\": types.NewSet(vs, types.Number(0)),\n\t}), &T1{})\n\tassert.Error(err)\n\tassert.Equal(\"Cannot unmarshal Set into Go value of type map[int]int\", err.Error())\n\n\ttype T2 struct {\n\t\tA map[int]struct{}\n\t}\n\n\terr = Unmarshal(types.NewStruct(\"T2\", types.StructData{\n\t\t\"a\": types.NewSet(vs, types.Number(0)),\n\t}), &T2{})\n\tassert.Error(err)\n\tassert.Equal(`Cannot unmarshal Set into Go value of type map[int]struct {}, field missing \"set\" tag`, err.Error())\n\n\ttype T3 struct {\n\t\tA map[int]struct{} `noms:\",set\"`\n\t}\n\n\terr = Unmarshal(types.NewStruct(\"T3\", types.StructData{\n\t\t\"a\": types.NewMap(vs, types.Number(0), types.EmptyStruct),\n\t}), &T3{})\n\tassert.Error(err)\n\tassert.Equal(`Cannot unmarshal Map into Go value of type map[int]struct {}, field has \"set\" tag`, err.Error())\n}\n\nfunc TestDecodeOmitEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tFoo int `noms:\",omitempty\"`\n\t\tBar struct {\n\t\t\tBaz int\n\t\t\tHotdog int `noms:\",omitempty\"`\n\t\t}\n\t}\n\texpected := S{\n\t\tBar: struct {\n\t\t\tBaz int\n\t\t\tHotdog int `noms:\",omitempty\"`\n\t\t}{\n\t\t\tBaz: 42,\n\t\t},\n\t}\n\tvar actual S\n\terr := Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"bar\": types.NewStruct(\"\", types.StructData{\n\t\t\t\"baz\": types.Number(42),\n\t\t}),\n\t}), &actual)\n\tassert.NoError(err)\n\tassert.Equal(expected, actual)\n}\n\nfunc TestDecodeOriginal(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tFoo int `noms:\",omitempty\"`\n\t\tBar types.Struct `noms:\",original\"`\n\t\tBaz types.Struct `noms:\",original\"`\n\t}\n\tinput := types.NewStruct(\"S\", types.StructData{\n\t\t\"foo\": types.Number(42),\n\t})\n\texpected := S{\n\t\tFoo: 42,\n\t\tBar: input,\n\t\tBaz: input,\n\t}\n\tvar actual S\n\terr := Unmarshal(input, &actual)\n\tassert.NoError(err)\n\tassert.True(expected.Bar.Equals(actual.Bar))\n}\n\nfunc TestDecodeOriginalReceiveTypeError(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tFoo types.Value `noms:\",original\"`\n\t}\n\tinput := types.NewStruct(\"S\", types.StructData{})\n\tvar actual S\n\terr := Unmarshal(input, &actual)\n\tassert.Error(err)\n\tassert.Equal(`Cannot unmarshal Struct S {} into Go value of type marshal.S, field with tag \"original\" must have type Struct`, err.Error())\n}\n\nfunc TestDecodeCanSkipUnexportedField(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tAbc int\n\t\tnotExported bool `noms:\"-\"`\n\t}\n\tvar s S\n\terr := Unmarshal(types.NewStruct(\"S\", types.StructData{\n\t\t\"abc\": types.Number(42),\n\t}), &s)\n\tassert.NoError(err)\n\tassert.Equal(S{42, false}, s)\n}\n\nfunc (u *primitiveType) UnmarshalNoms(v types.Value) error {\n\t*u = primitiveType(v.(types.Number) - 1)\n\treturn nil\n}\n\nfunc TestUnmarshalerPrimitiveType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tv := types.Number(43)\n\tu := primitiveType(0)\n\tassert.NoError(Unmarshal(v, &u))\n\tassert.Equal(primitiveType(42), u)\n}\n\nfunc (u *primitiveSliceType) UnmarshalNoms(v types.Value) error {\n\tsv := string(v.(types.String))\n\tspl := strings.Split(sv, \",\")\n\t*u = make(primitiveSliceType, len(spl))\n\tfor i, s := range spl {\n\t\t(*u)[i] = s\n\t}\n\treturn nil\n}\n\nfunc TestUnmarshalerPrimitiveSliceType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tv := types.String(\"a,b,c\")\n\tu := primitiveSliceType{}\n\tassert.NoError(Unmarshal(v, &u))\n\tassert.Equal(primitiveSliceType{\"a\", \"b\", \"c\"}, u)\n}\n\nfunc (u *primitiveMapType) UnmarshalNoms(v types.Value) error {\n\t*u = primitiveMapType{}\n\tv.(types.Set).IterAll(func(v types.Value) {\n\t\tsv := v.(types.String)\n\t\tspl := strings.Split(string(sv), \",\")\n\t\td.PanicIfFalse(len(spl) == 2)\n\t\t(*u)[spl[0]] = spl[1]\n\t})\n\treturn nil\n}\n\nfunc TestUnmarshalerPrimitiveMapType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tv := types.NewSet(vs, types.String(\"a,foo\"), types.String(\"b,bar\"))\n\tu := primitiveMapType{}\n\tassert.NoError(Unmarshal(v, &u))\n\tassert.Equal(primitiveMapType(map[string]string{\n\t\t\"a\": \"foo\",\n\t\t\"b\": \"bar\",\n\t}), u)\n}\n\nfunc (u *primitiveStructType) UnmarshalNoms(v types.Value) error {\n\tn := int(v.(types.Number))\n\tu.x = n / 3\n\tu.y = n % 3\n\treturn nil\n}\n\nfunc TestUnmarshalerPrimitiveStructType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tv := types.Number(10)\n\tu := primitiveStructType{}\n\tassert.NoError(Unmarshal(v, &u))\n\tassert.Equal(primitiveStructType{3, 1}, u)\n}\n\nfunc (u *builtinType) UnmarshalNoms(v types.Value) error {\n\tsv := v.(types.String)\n\t*u = builtinType(*regexp.MustCompile(string(sv)))\n\treturn nil\n}\n\nfunc TestUnmarshalerBuiltinType(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts := \"[a-z]+$\"\n\tv := types.String(s)\n\tu := builtinType{}\n\tassert.NoError(Unmarshal(v, &u))\n\tr := regexp.Regexp(u)\n\tassert.Equal(s, r.String())\n}\n\nfunc (u *wrappedMarshalerType) UnmarshalNoms(v types.Value) error {\n\tn := v.(types.Number)\n\t*u = wrappedMarshalerType(int(n) - 2)\n\treturn nil\n}\n\nfunc TestUnmarshalerWrappedMarshalerType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tv := types.Number(44)\n\tu := wrappedMarshalerType(0)\n\tassert.NoError(Unmarshal(v, &u))\n\tassert.Equal(wrappedMarshalerType(42), u)\n}\n\nfunc TestUnmarshalerComplexStructType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ts := \"foo|bar\"\n\tr := regexp.MustCompile(s)\n\tv := types.NewStruct(\"TestComplexStructType\", types.StructData{\n\t\t\"p\": types.Number(43),\n\t\t\"ps\": types.NewList(vs, types.Number(2), types.Number(3)),\n\t\t\"pm\": types.NewMap(vs, types.String(\"x\"), types.Number(101), types.String(\"y\"), types.Number(102)),\n\t\t\"pslice\": types.String(\"a,b,c\"),\n\t\t\"pmap\": types.NewSet(vs, types.String(\"c,123\"), types.String(\"d,456\")),\n\t\t\"pstruct\": types.Number(5),\n\t\t\"b\": types.String(s),\n\t})\n\tu := TestComplexStructType{}\n\tassert.NoError(Unmarshal(v, &u))\n\tassert.Equal(TestComplexStructType{\n\t\tP: 42,\n\t\tPs: []primitiveType{1, 2},\n\t\tPm: map[string]primitiveType{\n\t\t\t\"x\": 100,\n\t\t\t\"y\": 101,\n\t\t},\n\t\tPslice: primitiveSliceType{\"a\", \"b\", \"c\"},\n\t\tPmap: primitiveMapType{\n\t\t\t\"c\": \"123\",\n\t\t\t\"d\": \"456\",\n\t\t},\n\t\tPstruct: primitiveStructType{1, 2},\n\t\tB: builtinType(*r),\n\t}, u)\n}\n\nfunc (u *returnsMarshalerError) UnmarshalNoms(v types.Value) error {\n\t// Can't use u.err because an empty returnsMarshalerError is created for each\n\t// call to UnmarshalNoms.\n\treturn errors.New(\"foo bar baz\")\n}\n\nfunc (u panicsMarshaler) UnmarshalNoms(v types.Value) error {\n\tpanic(\"panic\")\n}\n\nfunc TestUnmarshalerError(t *testing.T) {\n\tassert := assert.New(t)\n\n\tm1 := returnsMarshalerError{}\n\terr := Unmarshal(types.EmptyStruct, &m1)\n\tassert.Equal(errors.New(\"foo bar baz\"), err)\n\n\tm2 := panicsMarshaler{}\n\tassert.Panics(func() { Unmarshal(types.EmptyStruct, &m2) })\n}\n\ntype notPointer struct {\n\tx int\n}\n\nfunc (u notPointer) UnmarshalNoms(v types.Value) error {\n\tu.x++\n\treturn nil\n}\n\nfunc TestUnmarshalNomsNotPointerDoesNotShareState(t *testing.T) {\n\tassert := assert.New(t)\n\n\tu := notPointer{0}\n\tassert.NoError(Unmarshal(types.EmptyStruct, &u))\n\tassert.NoError(Unmarshal(types.EmptyStruct, &u))\n\tassert.NoError(Unmarshal(types.EmptyStruct, &u))\n\tassert.Equal(notPointer{0}, u)\n}\n\nfunc TestUnmarshalMustUnmarshal(t *testing.T) {\n\ta := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype TestStruct struct{ F1 int }\n\n\tv := MustMarshal(vs, types.Number(1))\n\tvar out TestStruct\n\ta.Panics(func() { MustUnmarshal(v, &out) })\n\n\tv = MustMarshal(vs, TestStruct{2})\n\ta.NotPanics(func() { MustUnmarshal(v, &out) })\n}\n" }, { "path": "go/marshal/encode.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package marshal implements encoding and decoding of Noms values. The mapping\n// between Noms objects and Go values is described in the documentation for the\n// Marshal and Unmarshal functions.\npackage marshal\n\nimport (\n\t\"fmt\"\n\t\"reflect\"\n\t\"sort\"\n\t\"strings\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// Marshal converts a Go value to a Noms value.\n//\n// Marshal traverses the value v recursively. Marshal uses the following\n// type-dependent encodings:\n//\n// Boolean values are encoded as Noms types.Bool.\n//\n// Floating point and integer values are encoded as Noms types.Number. At the\n// moment this might lead to some loss in precision because types.Number\n// currently takes a float64.\n//\n// String values are encoded as Noms types.String.\n//\n// Slices and arrays are encoded as Noms types.List by default. If a\n// field is tagged with `noms:\"set\", it will be encoded as Noms types.Set\n// instead.\n//\n// Maps are encoded as Noms types.Map, or a types.Set if the value type is\n// struct{} and the field is tagged with `noms:\"set\"`.\n//\n// Struct values are encoded as Noms structs (types.Struct). Each exported Go\n// struct field becomes a member of the Noms struct unless\n// - The field's tag is \"-\"\n// - The field is empty and its tag specifies the \"omitempty\" option.\n// - The field has the \"original\" tag, in which case the field is used as an\n// initial value onto which the fields of the Go type are added. When\n// combined with the corresponding support for \"original\" in Unmarshal(),\n// this allows one to find and modify any values of a known subtype.\n//\n// Additionally, user-defined types can implement the Marshaler interface to\n// provide a custom encoding.\n//\n// The empty values are false, 0, any nil pointer or interface value, and any\n// array, slice, map, or string of length zero.\n//\n// The Noms struct default field name is the Go struct field name where the\n// first character is lower cased, but can be specified in the Go struct field's\n// tag value. The \"noms\" key in the Go struct field's tag value is the field\n// name. Examples:\n//\n// // Field is ignored.\n// Field int `noms:\"-\"`\n//\n// // Field appears in a Noms struct as field \"myName\".\n// MyName int\n//\n// // Field appears in a Noms struct as key \"myName\".\n// Field int `noms:\"myName\"`\n//\n// // Field appears in a Noms struct as key \"myName\" and the field is\n// // omitted from the object if its value is empty, as defined above.\n// Field int `noms:\"myName,omitempty\"\n//\n// // Field appears in a Noms struct as key \"field\" and the field is\n// // omitted from the object if its value is empty, as defined above.\n// Field int `noms:\",omitempty\"\n//\n// The name of the Noms struct is the name of the Go struct where the first\n// character is changed to upper case. You can also implement the\n// StructNameMarshaler interface to get more control over the actual struct\n// name.\n//\n// Anonymous struct fields are usually marshaled as if their inner exported\n// fields were fields in the outer struct, subject to the usual Go visibility.\n// An anonymous struct field with a name given in its Noms tag is treated as\n// having that name, rather than being anonymous.\n//\n// Noms values (values implementing types.Value) are copied over without any\n// change.\n//\n// When marshalling interface{} the dynamic type is used.\n//\n// Go pointers, complex, function are not supported. Attempting to encode such a\n// value causes Marshal to return an UnsupportedTypeError.\nfunc Marshal(vrw types.ValueReadWriter, v interface{}) (types.Value, error) {\n\treturn MarshalOpt(vrw, v, Opt{})\n}\n\n// MarshalOpt is like Marshal but provides additional options.\nfunc MarshalOpt(vrw types.ValueReadWriter, v interface{}, opt Opt) (nomsValue types.Value, err error) {\n\tdefer func() {\n\t\tif r := recover(); r != nil {\n\t\t\tswitch r := r.(type) {\n\t\t\tcase *UnsupportedTypeError, *InvalidTagError:\n\t\t\t\terr = r.(error)\n\t\t\tcase *marshalNomsError:\n\t\t\t\terr = r.err\n\t\t\tdefault:\n\t\t\t\tpanic(r)\n\t\t\t}\n\t\t}\n\t}()\n\tnomsValue = MustMarshalOpt(vrw, v, opt)\n\treturn\n}\n\n// MustMarshal marshals a Go value to a Noms value using the same rules as\n// Marshal(). Panics on failure.\nfunc MustMarshal(vrw types.ValueReadWriter, v interface{}) types.Value {\n\treturn MustMarshalOpt(vrw, v, Opt{})\n}\n\n// MustMarshalOpt is like MustMarshal, but with additional options.\nfunc MustMarshalOpt(vrw types.ValueReadWriter, v interface{}, opt Opt) types.Value {\n\trv := reflect.ValueOf(v)\n\tnt := nomsTags{\n\t\tset: opt.Set,\n\t}\n\tencoder := typeEncoder(rv.Type(), map[string]reflect.Type{}, nt)\n\treturn encoder(rv, vrw)\n}\n\n// Marshaler is an interface types can implement to provide their own encoding.\ntype Marshaler interface {\n\t// MarshalNoms returns the Noms Value encoding of a type, or an error.\n\t// nil is not a valid return val - if both val and err are nil, Marshal will\n\t// panic.\n\tMarshalNoms(vrw types.ValueReadWriter) (val types.Value, err error)\n}\n\n// StructNameMarshaler is an interface that can be implemented to define the\n// name of a Noms struct.\ntype StructNameMarshaler interface {\n\tMarshalNomsStructName() string\n}\n\n// UnsupportedTypeError is returned by encode when attempting to encode a type\n// that isn't supported.\ntype UnsupportedTypeError struct {\n\tType reflect.Type\n\tMessage string\n}\n\nfunc (e *UnsupportedTypeError) Error() string {\n\tmsg := e.Message\n\tif msg == \"\" {\n\t\tmsg = \"Type is not supported\"\n\t}\n\treturn msg + \", type: \" + e.Type.String()\n}\n\n// InvalidTagError is returned by encode and decode when the struct field tag is\n// invalid. For example if the field name is not a valid Noms struct field name.\ntype InvalidTagError struct {\n\tmessage string\n}\n\nfunc (e *InvalidTagError) Error() string {\n\treturn e.message\n}\n\n// marshalNomsError wraps errors from Marshaler.MarshalNoms. These should be\n// unwrapped and never leak to the caller of Marshal.\ntype marshalNomsError struct {\n\terr error\n}\n\nfunc (e *marshalNomsError) Error() string {\n\treturn e.err.Error()\n}\n\ntype Opt struct {\n\t// Marshal []T or map[T]struct{} to Set, or Unmarhsal Set to map[T]struct{}.\n\tSet bool\n}\n\ntype nomsTags struct {\n\tname string\n\tomitEmpty bool\n\toriginal bool\n\tset bool\n\tskip bool\n\thasName bool\n}\n\nvar nomsValueInterface = reflect.TypeOf((*types.Value)(nil)).Elem()\nvar emptyInterface = reflect.TypeOf((*interface{})(nil)).Elem()\nvar marshalerInterface = reflect.TypeOf((*Marshaler)(nil)).Elem()\nvar structNameMarshalerInterface = reflect.TypeOf((*StructNameMarshaler)(nil)).Elem()\n\ntype encoderFunc func(v reflect.Value, vrw types.ValueReadWriter) types.Value\n\nfunc boolEncoder(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\treturn types.Bool(v.Bool())\n}\n\nfunc float64Encoder(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\treturn types.Number(v.Float())\n}\n\nfunc intEncoder(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\treturn types.Number(float64(v.Int()))\n}\n\nfunc uintEncoder(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\treturn types.Number(float64(v.Uint()))\n}\n\nfunc stringEncoder(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\treturn types.String(v.String())\n}\n\nfunc nomsValueEncoder(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\treturn v.Interface().(types.Value)\n}\n\nfunc marshalerEncoder(t reflect.Type) encoderFunc {\n\treturn func(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\t\tval, err := v.Interface().(Marshaler).MarshalNoms(vrw)\n\t\tif err != nil {\n\t\t\tpanic(&marshalNomsError{err})\n\t\t}\n\t\tif val == nil {\n\t\t\tpanic(fmt.Errorf(\"nil result from %s.MarshalNoms\", t.String()))\n\t\t}\n\t\treturn val\n\t}\n}\n\nfunc typeEncoder(t reflect.Type, seenStructs map[string]reflect.Type, tags nomsTags) encoderFunc {\n\tif t.Implements(marshalerInterface) {\n\t\treturn marshalerEncoder(t)\n\t}\n\n\tswitch t.Kind() {\n\tcase reflect.Bool:\n\t\treturn boolEncoder\n\tcase reflect.Float64, reflect.Float32:\n\t\treturn float64Encoder\n\tcase reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:\n\t\treturn intEncoder\n\tcase reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:\n\t\treturn uintEncoder\n\tcase reflect.String:\n\t\treturn stringEncoder\n\tcase reflect.Struct:\n\t\treturn structEncoder(t, seenStructs)\n\tcase reflect.Slice, reflect.Array:\n\t\tif shouldEncodeAsSet(t, tags) {\n\t\t\treturn setFromListEncoder(t, seenStructs)\n\t\t}\n\t\treturn listEncoder(t, seenStructs)\n\tcase reflect.Map:\n\t\tif shouldEncodeAsSet(t, tags) {\n\t\t\treturn setEncoder(t, seenStructs)\n\t\t}\n\t\treturn mapEncoder(t, seenStructs)\n\tcase reflect.Interface:\n\t\treturn func(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\t\t\t// Get the dynamic type.\n\t\t\tv2 := reflect.ValueOf(v.Interface())\n\t\t\treturn typeEncoder(v2.Type(), seenStructs, tags)(v2, vrw)\n\t\t}\n\tcase reflect.Ptr:\n\t\t// Allow implementations of types.Value (like *types.Type)\n\t\tif t.Implements(nomsValueInterface) {\n\t\t\treturn nomsValueEncoder\n\t\t}\n\t\tfallthrough\n\tdefault:\n\t\tpanic(&UnsupportedTypeError{Type: t})\n\t}\n}\n\nfunc getStructName(t reflect.Type) string {\n\tif t.Implements(structNameMarshalerInterface) {\n\t\tv := reflect.Zero(t)\n\t\treturn v.Interface().(StructNameMarshaler).MarshalNomsStructName()\n\t}\n\treturn strings.Title(t.Name())\n}\n\nfunc structEncoder(t reflect.Type, seenStructs map[string]reflect.Type) encoderFunc {\n\tif t.Implements(nomsValueInterface) {\n\t\treturn nomsValueEncoder\n\t}\n\n\te := encoderCache.get(t)\n\tif e != nil {\n\t\treturn e\n\t}\n\n\tstructName := getStructName(t)\n\n\tseenStructs[t.Name()] = t\n\tfields, knownShape, originalFieldIndex := typeFields(t, seenStructs, false, false)\n\tif knownShape {\n\t\tfieldNames := make([]string, len(fields))\n\t\tfor i, f := range fields {\n\t\t\tfieldNames[i] = f.name\n\t\t}\n\n\t\tstructTemplate := types.MakeStructTemplate(structName, fieldNames)\n\t\te = func(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\t\t\tvalues := make(types.ValueSlice, len(fields))\n\t\t\tfor i, f := range fields {\n\t\t\t\tvalues[i] = f.encoder(v.FieldByIndex(f.index), vrw)\n\t\t\t}\n\t\t\treturn structTemplate.NewStruct(values)\n\t\t}\n\t} else if originalFieldIndex == nil {\n\t\t// Slower path: cannot precompute the Noms type since there are Noms collections,\n\t\t// but at least there are a set number of fields.\n\t\te = func(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\t\t\tdata := make(types.StructData, len(fields))\n\t\t\tfor _, f := range fields {\n\t\t\t\tfv := v.FieldByIndex(f.index)\n\t\t\t\tif !fv.IsValid() || f.omitEmpty && isEmptyValue(fv) {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tdata[f.name] = f.encoder(fv, vrw)\n\t\t\t}\n\t\t\treturn types.NewStruct(structName, data)\n\t\t}\n\t} else {\n\t\t// Slowest path - we are extending some other struct. We need to start with the\n\t\t// type of that struct and extend.\n\t\te = func(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\t\t\tfv := v.FieldByIndex(originalFieldIndex)\n\t\t\tret := fv.Interface().(types.Struct)\n\t\t\tif ret.IsZeroValue() {\n\t\t\t\tret = types.NewStruct(structName, nil)\n\t\t\t}\n\t\t\tfor _, f := range fields {\n\t\t\t\tfv := v.FieldByIndex(f.index)\n\t\t\t\tif !fv.IsValid() || f.omitEmpty && isEmptyValue(fv) {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tret = ret.Set(f.name, f.encoder(fv, vrw))\n\t\t\t}\n\t\t\treturn ret\n\t\t}\n\t}\n\n\tencoderCache.set(t, e)\n\treturn e\n}\n\nfunc isEmptyValue(v reflect.Value) bool {\n\tswitch v.Kind() {\n\tcase reflect.Array, reflect.Map, reflect.Slice, reflect.String:\n\t\treturn v.Len() == 0\n\tcase reflect.Bool:\n\t\treturn !v.Bool()\n\tcase reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:\n\t\treturn v.Int() == 0\n\tcase reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:\n\t\treturn v.Uint() == 0\n\tcase reflect.Float32, reflect.Float64:\n\t\treturn v.Float() == 0\n\tcase reflect.Struct:\n\t\tz := reflect.Zero(v.Type())\n\t\treturn reflect.DeepEqual(z.Interface(), v.Interface())\n\tcase reflect.Interface:\n\t\treturn v.IsNil()\n\t}\n\treturn false\n}\n\ntype field struct {\n\tname string\n\tencoder encoderFunc\n\tindex []int\n\tnomsType *types.Type\n\tomitEmpty bool\n}\n\ntype fieldSlice []field\n\nfunc (fs fieldSlice) Len() int { return len(fs) }\nfunc (fs fieldSlice) Swap(i, j int) { fs[i], fs[j] = fs[j], fs[i] }\nfunc (fs fieldSlice) Less(i, j int) bool { return fs[i].name < fs[j].name }\n\ntype encoderCacheT struct {\n\tsync.RWMutex\n\tm map[reflect.Type]encoderFunc\n}\n\nvar encoderCache = &encoderCacheT{}\n\n// Separate Set encoder cache because the same type with and without the\n// `noms:\",set\"` tag encode differently (Set vs Map).\nvar setEncoderCache = &encoderCacheT{}\n\nfunc (c *encoderCacheT) get(t reflect.Type) encoderFunc {\n\tc.RLock()\n\tdefer c.RUnlock()\n\treturn c.m[t]\n}\n\nfunc (c *encoderCacheT) set(t reflect.Type, e encoderFunc) {\n\tc.Lock()\n\tdefer c.Unlock()\n\tif c.m == nil {\n\t\tc.m = map[reflect.Type]encoderFunc{}\n\t}\n\tc.m[t] = e\n}\n\nfunc getTags(f reflect.StructField) (tags nomsTags) {\n\treflectTags := f.Tag.Get(\"noms\")\n\tif reflectTags == \"-\" {\n\t\ttags.skip = true\n\t\treturn\n\t}\n\n\ttagsSlice := strings.Split(reflectTags, \",\")\n\n\t// The first tag is always the name, or empty to use the field as the name.\n\tif len(tagsSlice) == 0 || tagsSlice[0] == \"\" {\n\t\ttags.name = strings.ToLower(f.Name[:1]) + f.Name[1:]\n\t} else {\n\t\ttags.name = tagsSlice[0]\n\t\ttags.hasName = true\n\t}\n\n\tif !types.IsValidStructFieldName(tags.name) {\n\t\tpanic(&InvalidTagError{\"Invalid struct field name: \" + tags.name})\n\t}\n\n\tfor i := 1; i < len(tagsSlice); i++ {\n\t\tswitch tag := tagsSlice[i]; tag {\n\t\tcase \"omitempty\":\n\t\t\ttags.omitEmpty = true\n\t\tcase \"original\":\n\t\t\ttags.original = true\n\t\tcase \"set\":\n\t\t\ttags.set = true\n\t\tdefault:\n\t\t\tpanic(&InvalidTagError{\"Unrecognized tag: \" + tag})\n\t\t}\n\t}\n\treturn\n}\n\nfunc validateField(f reflect.StructField, t reflect.Type) {\n\t// PkgPath is the package path that qualifies a lower case (unexported)\n\t// field name. It is empty for upper case (exported) field names.\n\t// See https://golang.org/ref/spec#Uniqueness_of_identifiers\n\tif f.PkgPath != \"\" && !f.Anonymous { // unexported\n\t\tpanic(&UnsupportedTypeError{t, \"Non exported fields are not supported\"})\n\t}\n}\n\nfunc typeFields(t reflect.Type, seenStructs map[string]reflect.Type, computeType, embedded bool) (fields fieldSlice, knownShape bool, originalFieldIndex []int) {\n\tknownShape = true\n\tfor i := 0; i < t.NumField(); i++ {\n\t\tindex := make([]int, 1)\n\t\tindex[0] = i\n\t\tf := t.Field(i)\n\t\ttags := getTags(f)\n\t\tif tags.skip {\n\t\t\tcontinue\n\t\t}\n\n\t\tif tags.original {\n\t\t\toriginalFieldIndex = f.Index\n\t\t\tcontinue\n\t\t}\n\n\t\tif f.Anonymous && f.PkgPath == \"\" && !tags.hasName {\n\t\t\tembeddedFields, embeddedKnownShape, embeddedOriginalFieldIndex := typeFields(f.Type, seenStructs, computeType, true)\n\t\t\tif embeddedOriginalFieldIndex != nil {\n\t\t\t\toriginalFieldIndex = append(index, embeddedOriginalFieldIndex...)\n\t\t\t}\n\t\t\tknownShape = knownShape && embeddedKnownShape\n\n\t\t\tfor _, ef := range embeddedFields {\n\t\t\t\tef.index = append(index, ef.index...)\n\t\t\t\tfields = append(fields, ef)\n\t\t\t}\n\n\t\t\tcontinue\n\t\t}\n\n\t\tvar nt *types.Type\n\t\tvalidateField(f, t)\n\t\tif computeType {\n\t\t\tnt = encodeType(f.Type, seenStructs, tags)\n\t\t\tif nt == nil {\n\t\t\t\tknownShape = false\n\t\t\t}\n\t\t}\n\n\t\tif tags.omitEmpty && !computeType {\n\t\t\tknownShape = false\n\t\t}\n\n\t\tfields = append(fields, field{\n\t\t\tname: tags.name,\n\t\t\tencoder: typeEncoder(f.Type, seenStructs, tags),\n\t\t\tindex: index,\n\t\t\tnomsType: nt,\n\t\t\tomitEmpty: tags.omitEmpty,\n\t\t})\n\t}\n\n\tif !embedded {\n\t\tsort.Sort(fields)\n\t}\n\t// If embedded then the fields gets sorted once we return to the caller.\n\n\treturn\n}\n\nfunc listEncoder(t reflect.Type, seenStructs map[string]reflect.Type) encoderFunc {\n\te := encoderCache.get(t)\n\tif e != nil {\n\t\treturn e\n\t}\n\n\tvar elemEncoder encoderFunc\n\t// lock e until encoder(s) are initialized\n\tvar init sync.RWMutex\n\tinit.Lock()\n\tdefer init.Unlock()\n\te = func(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\t\tinit.RLock()\n\t\tdefer init.RUnlock()\n\t\tvalues := make([]types.Value, v.Len())\n\t\tfor i := 0; i < v.Len(); i++ {\n\t\t\tvalues[i] = elemEncoder(v.Index(i), vrw)\n\t\t}\n\t\treturn types.NewList(vrw, values...)\n\t}\n\n\tencoderCache.set(t, e)\n\telemEncoder = typeEncoder(t.Elem(), seenStructs, nomsTags{})\n\treturn e\n}\n\n// Encode set from array or slice\nfunc setFromListEncoder(t reflect.Type, seenStructs map[string]reflect.Type) encoderFunc {\n\te := setEncoderCache.get(t)\n\tif e != nil {\n\t\treturn e\n\t}\n\n\tvar elemEncoder encoderFunc\n\t// lock e until encoder(s) are initialized\n\tvar init sync.RWMutex\n\tinit.Lock()\n\tdefer init.Unlock()\n\te = func(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\t\tinit.RLock()\n\t\tdefer init.RUnlock()\n\t\tvalues := make([]types.Value, v.Len())\n\t\tfor i := 0; i < v.Len(); i++ {\n\t\t\tvalues[i] = elemEncoder(v.Index(i), vrw)\n\t\t}\n\t\treturn types.NewSet(vrw, values...)\n\t}\n\n\tsetEncoderCache.set(t, e)\n\telemEncoder = typeEncoder(t.Elem(), seenStructs, nomsTags{})\n\treturn e\n}\n\nfunc setEncoder(t reflect.Type, seenStructs map[string]reflect.Type) encoderFunc {\n\te := setEncoderCache.get(t)\n\tif e != nil {\n\t\treturn e\n\t}\n\n\tvar encoder encoderFunc\n\t// lock e until encoder(s) are initialized\n\tvar init sync.RWMutex\n\tinit.Lock()\n\tdefer init.Unlock()\n\te = func(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\t\tinit.RLock()\n\t\tdefer init.RUnlock()\n\t\tvalues := make([]types.Value, v.Len(), v.Len())\n\t\tfor i, k := range v.MapKeys() {\n\t\t\tvalues[i] = encoder(k, vrw)\n\t\t}\n\t\treturn types.NewSet(vrw, values...)\n\t}\n\n\tsetEncoderCache.set(t, e)\n\tencoder = typeEncoder(t.Key(), seenStructs, nomsTags{})\n\treturn e\n}\n\nfunc mapEncoder(t reflect.Type, seenStructs map[string]reflect.Type) encoderFunc {\n\te := encoderCache.get(t)\n\tif e != nil {\n\t\treturn e\n\t}\n\n\tvar keyEncoder encoderFunc\n\tvar valueEncoder encoderFunc\n\t// lock e until encoder(s) are initialized\n\tvar init sync.RWMutex\n\tinit.Lock()\n\tdefer init.Unlock()\n\te = func(v reflect.Value, vrw types.ValueReadWriter) types.Value {\n\t\tinit.RLock()\n\t\tdefer init.RUnlock()\n\t\tkeys := v.MapKeys()\n\t\tkvs := make([]types.Value, 2*len(keys))\n\t\tfor i, k := range keys {\n\t\t\tkvs[2*i] = keyEncoder(k, vrw)\n\t\t\tkvs[2*i+1] = valueEncoder(v.MapIndex(k), vrw)\n\t\t}\n\t\treturn types.NewMap(vrw, kvs...)\n\t}\n\n\tencoderCache.set(t, e)\n\tkeyEncoder = typeEncoder(t.Key(), seenStructs, nomsTags{})\n\tvalueEncoder = typeEncoder(t.Elem(), seenStructs, nomsTags{})\n\treturn e\n}\n\nfunc shouldEncodeAsSet(t reflect.Type, tags nomsTags) bool {\n\tswitch t.Kind() {\n\tcase reflect.Slice, reflect.Array:\n\t\treturn tags.set\n\tcase reflect.Map:\n\t\t// map[T]struct{} `noms:,\"set\"`\n\t\treturn tags.set &&\n\t\t\tt.Elem().Kind() == reflect.Struct &&\n\t\t\tt.Elem().NumField() == 0\n\tdefault:\n\t\tpanic(fmt.Errorf(\"called with unexpected kind %v\", t.Kind()))\n\t}\n}\n" }, { "path": "go/marshal/encode_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage marshal\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"fmt\"\n\t\"math\"\n\t\"regexp\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestEncode(tt *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tt := func(exp types.Value, v interface{}) {\n\t\tactual, err := Marshal(vs, v)\n\t\tassert.NoError(tt, err)\n\t\tassert.True(tt, exp.Equals(actual))\n\n\t\t// Encode again for fallthrough\n\t\tactual2, err := Marshal(vs, actual)\n\t\tassert.NoError(tt, err)\n\t\tassert.True(tt, exp.Equals(actual2))\n\t}\n\n\tfor _, n := range []float32{0, 42, 3.14159265359, math.MaxFloat32} {\n\t\tt(types.Number(n), n)\n\t\tt(types.Number(-n), -n)\n\t}\n\n\tfor _, n := range []float64{0, 42, 3.14159265359, 9007199254740991, math.MaxFloat64} {\n\t\tt(types.Number(n), n)\n\t\tt(types.Number(-n), -n)\n\t}\n\n\tfor _, n := range []int8{0, 42, math.MaxInt8} {\n\t\tt(types.Number(n), n)\n\t\tt(types.Number(-n), -n)\n\t}\n\n\tfor _, n := range []int16{0, 42, math.MaxInt16} {\n\t\tt(types.Number(n), n)\n\t\tt(types.Number(-n), -n)\n\t}\n\n\tfor _, n := range []int32{0, 42, math.MaxInt32} {\n\t\tt(types.Number(n), n)\n\t\tt(types.Number(-n), -n)\n\t}\n\n\t// int is at least int32\n\tfor _, n := range []int{0, 42, math.MaxInt32} {\n\t\tt(types.Number(n), n)\n\t\tt(types.Number(-n), -n)\n\t}\n\n\tfor _, n := range []int64{0, 42, math.MaxInt64} {\n\t\tt(types.Number(n), n)\n\t\tt(types.Number(-n), -n)\n\t}\n\n\tfor _, n := range []uint8{0, 42, math.MaxUint8} {\n\t\tt(types.Number(n), n)\n\t}\n\n\tfor _, n := range []uint16{0, 42, math.MaxUint16} {\n\t\tt(types.Number(n), n)\n\t}\n\n\tfor _, n := range []uint32{0, 42, math.MaxUint32} {\n\t\tt(types.Number(n), n)\n\t}\n\n\t// uint is at least uint32\n\tfor _, n := range []uint{0, 42, math.MaxUint32} {\n\t\tt(types.Number(n), n)\n\t}\n\n\tfor _, n := range []uint64{0, 42, math.MaxUint64} {\n\t\tt(types.Number(n), n)\n\t}\n\n\tt(types.Bool(true), true)\n\tt(types.Bool(false), false)\n\n\tfor _, s := range []string{\"\", \"s\", \"hello\", \"💩\"} {\n\t\tt(types.String(s), s)\n\t}\n\n\tt(types.NewList(vs, types.Number(42)), types.NewList(vs, types.Number(42)))\n\tt(types.NewMap(vs, types.Number(42), types.String(\"hi\")), types.NewMap(vs, types.Number(42), types.String(\"hi\")))\n\tt(types.NewSet(vs, types.String(\"bye\")), types.NewSet(vs, types.String(\"bye\")))\n\tt(types.NewBlob(vs, bytes.NewBufferString(\"hello\")), types.NewBlob(vs, bytes.NewBufferString(\"hello\")))\n\n\ttype TestStruct struct {\n\t\tStr string\n\t\tNum float64\n\t}\n\tt(types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\"num\": types.Number(42),\n\t\t\"str\": types.String(\"Hello\"),\n\t}), TestStruct{Str: \"Hello\", Num: 42})\n\t// Same again to test caching\n\tt(types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\"num\": types.Number(1),\n\t\t\"str\": types.String(\"Bye\"),\n\t}), TestStruct{Str: \"Bye\", Num: 1})\n\n\tanonStruct := struct {\n\t\tB bool\n\t}{\n\t\ttrue,\n\t}\n\tt(types.NewStruct(\"\", types.StructData{\n\t\t\"b\": types.Bool(true),\n\t}), anonStruct)\n\n\ttype TestNestedStruct struct {\n\t\tA types.List\n\t\tB TestStruct\n\t\tC float64\n\t}\n\tt(types.NewStruct(\"TestNestedStruct\", types.StructData{\n\t\t\"a\": types.NewList(vs, types.String(\"hi\")),\n\t\t\"b\": types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\t\"str\": types.String(\"bye\"),\n\t\t\t\"num\": types.Number(5678),\n\t\t}),\n\t\t\"c\": types.Number(1234),\n\t}), TestNestedStruct{\n\t\tA: types.NewList(vs, types.String(\"hi\")),\n\t\tB: TestStruct{\n\t\t\tStr: \"bye\",\n\t\t\tNum: 5678,\n\t\t},\n\t\tC: 1234,\n\t})\n\n\ttype testStruct struct {\n\t\tStr string\n\t\tNum float64\n\t}\n\tt(types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\"num\": types.Number(42),\n\t\t\"str\": types.String(\"Hello\"),\n\t}), testStruct{Str: \"Hello\", Num: 42})\n}\n\nfunc assertEncodeErrorMessage(t *testing.T, v interface{}, expectedMessage string) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\t_, err := Marshal(vs, v)\n\tassert.Error(t, err)\n\tassert.Equal(t, expectedMessage, err.Error())\n}\n\nfunc TestInvalidTypes(t *testing.T) {\n\tassertEncodeErrorMessage(t, make(chan int), \"Type is not supported, type: chan int\")\n\tx := 42\n\tassertEncodeErrorMessage(t, &x, \"Type is not supported, type: *int\")\n}\n\nfunc TestEncodeEmbeddedStructSkip(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype EmbeddedStruct struct {\n\t\tX int\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct `noms:\"-\"`\n\t\tY int\n\t}\n\ts := TestStruct{EmbeddedStruct{1}, 2}\n\tv, err := Marshal(vs, s)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\"y\": types.Number(2),\n\t}).Equals(v))\n}\n\nfunc TestEncodeEmbeddedStructWithName(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype EmbeddedStruct struct {\n\t\tX int\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct `noms:\"em\"`\n\t\tY int\n\t}\n\ts := TestStruct{EmbeddedStruct{1}, 2}\n\tv, err := Marshal(vs, s)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\"em\": types.NewStruct(\"EmbeddedStruct\", types.StructData{\n\t\t\t\"x\": types.Number(1),\n\t\t}),\n\t\t\"y\": types.Number(2),\n\t}).Equals(v))\n}\n\nfunc TestEncodeEmbeddedStruct(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype EmbeddedStruct struct {\n\t\tX int\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct\n\t}\n\ts := TestStruct{EmbeddedStruct{1}}\n\tv, err := Marshal(vs, s)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\"x\": types.Number(1),\n\t}).Equals(v))\n\n\ttype TestOuter struct {\n\t\tA int\n\t\tTestStruct\n\t\tB int\n\t}\n\ts2 := TestOuter{0, TestStruct{EmbeddedStruct{1}}, 2}\n\tv2, err := Marshal(vs, s2)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"TestOuter\", types.StructData{\n\t\t\"a\": types.Number(0),\n\t\t\"b\": types.Number(2),\n\t\t\"x\": types.Number(1),\n\t}).Equals(v2))\n}\n\nfunc TestEncodeEmbeddedStructOriginal(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype EmbeddedStruct struct {\n\t\tX int\n\t\tO types.Struct `noms:\",original\"`\n\t\tB bool\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct\n\t}\n\ts := TestStruct{\n\t\tEmbeddedStruct: EmbeddedStruct{\n\t\t\tX: 1,\n\t\t\tB: true,\n\t\t},\n\t}\n\tv, err := Marshal(vs, s)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"TestStruct\", types.StructData{\n\t\t\"b\": types.Bool(true),\n\t\t\"x\": types.Number(1),\n\t}).Equals(v))\n}\n\nfunc TestEncodeNonExportedField(t *testing.T) {\n\ttype TestStruct struct {\n\t\tx int\n\t}\n\tassertEncodeErrorMessage(t, TestStruct{1}, \"Non exported fields are not supported, type: marshal.TestStruct\")\n}\n\nfunc TestEncodeTaggingSkip(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tAbc int `noms:\"-\"`\n\t\tDef bool\n\t}\n\ts := S{42, true}\n\tv, err := Marshal(vs, s)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S\", types.StructData{\n\t\t\"def\": types.Bool(true),\n\t}).Equals(v))\n}\n\nfunc TestEncodeNamedFields(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tAaa int `noms:\"a\"`\n\t\tBbb bool `noms:\"B\"`\n\t\tCcc string\n\t}\n\ts := S{42, true, \"Hi\"}\n\tv, err := Marshal(vs, s)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S\", types.StructData{\n\t\t\"a\": types.Number(42),\n\t\t\"B\": types.Bool(true),\n\t\t\"ccc\": types.String(\"Hi\"),\n\t}).Equals(v))\n}\n\nfunc TestEncodeInvalidNamedFields(t *testing.T) {\n\ttype S struct {\n\t\tA int `noms:\"1a\"`\n\t}\n\tassertEncodeErrorMessage(t, S{42}, \"Invalid struct field name: 1a\")\n}\n\nfunc TestEncodeOmitEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tString string `noms:\",omitempty\"`\n\t\tBool bool `noms:\",omitempty\"`\n\t\tInt int `noms:\",omitempty\"`\n\t\tInt8 int8 `noms:\",omitempty\"`\n\t\tInt16 int16 `noms:\",omitempty\"`\n\t\tInt32 int32 `noms:\",omitempty\"`\n\t\tInt64 int64 `noms:\",omitempty\"`\n\t\tUint uint `noms:\",omitempty\"`\n\t\tUint8 uint8 `noms:\",omitempty\"`\n\t\tUint16 uint16 `noms:\",omitempty\"`\n\t\tUint32 uint32 `noms:\",omitempty\"`\n\t\tUint64 uint64 `noms:\",omitempty\"`\n\t\tFloat32 float32 `noms:\",omitempty\"`\n\t\tFloat64 float64 `noms:\",omitempty\"`\n\t}\n\ts := S{\n\t\tString: \"s\",\n\t\tBool: true,\n\t\tInt: 1,\n\t\tInt8: 1,\n\t\tInt16: 1,\n\t\tInt32: 1,\n\t\tInt64: 1,\n\t\tUint: 1,\n\t\tUint8: 1,\n\t\tUint16: 1,\n\t\tUint32: 1,\n\t\tUint64: 1,\n\t\tFloat32: 1,\n\t\tFloat64: 1,\n\t}\n\tv, err := Marshal(vs, s)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S\", types.StructData{\n\t\t\"string\": types.String(\"s\"),\n\t\t\"bool\": types.Bool(true),\n\t\t\"int\": types.Number(1),\n\t\t\"int8\": types.Number(1),\n\t\t\"int16\": types.Number(1),\n\t\t\"int32\": types.Number(1),\n\t\t\"int64\": types.Number(1),\n\t\t\"uint\": types.Number(1),\n\t\t\"uint8\": types.Number(1),\n\t\t\"uint16\": types.Number(1),\n\t\t\"uint32\": types.Number(1),\n\t\t\"uint64\": types.Number(1),\n\t\t\"float32\": types.Number(1),\n\t\t\"float64\": types.Number(1),\n\t}).Equals(v))\n\n\ts2 := S{\n\t\tString: \"\",\n\t\tBool: false,\n\t\tInt: 0,\n\t\tInt8: 0,\n\t\tInt16: 0,\n\t\tInt32: 0,\n\t\tInt64: 0,\n\t\tUint: 0,\n\t\tUint8: 0,\n\t\tUint16: 0,\n\t\tUint32: 0,\n\t\tUint64: 0,\n\t\tFloat32: 0,\n\t\tFloat64: 0,\n\t}\n\tv2, err := Marshal(vs, s2)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S\", types.StructData{}).Equals(v2))\n\n\ttype S2 struct {\n\t\tSlice []int `noms:\",omitempty\"`\n\t\tMap map[int]int `noms:\",omitempty\"`\n\t}\n\n\ts3 := S2{\n\t\tSlice: []int{0},\n\t\tMap: map[int]int{0: 0},\n\t}\n\tv3, err := Marshal(vs, s3)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S2\", types.StructData{\n\t\t\"slice\": types.NewList(vs, types.Number(0)),\n\t\t\"map\": types.NewMap(vs, types.Number(0), types.Number(0)),\n\t}).Equals(v3))\n\n\ts4 := S2{\n\t\tSlice: []int{},\n\t\tMap: map[int]int{},\n\t}\n\tv4, err := Marshal(vs, s4)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S2\", types.StructData{}).Equals(v4))\n\n\ts5 := S2{\n\t\tSlice: nil,\n\t\tMap: nil,\n\t}\n\tv5, err := Marshal(vs, s5)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S2\", types.StructData{}).Equals(v5))\n\n\ttype S3 struct {\n\t\tList types.List `noms:\",omitempty\"`\n\t\tValue types.Value `noms:\",omitempty\"`\n\t}\n\ts6 := S3{\n\t\tList: types.NewList(vs),\n\t\tValue: types.Number(0),\n\t}\n\tv6, err := Marshal(vs, s6)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S3\", types.StructData{\n\t\t\"list\": types.NewList(vs),\n\t\t\"value\": types.Number(0),\n\t}).Equals(v6))\n\n\ts7 := S3{\n\t\tList: types.List{},\n\t\tValue: nil,\n\t}\n\tv7, err := Marshal(vs, s7)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S3\", types.StructData{}).Equals(v7))\n\n\t// Both name and omitempty\n\ttype S4 struct {\n\t\tX int `noms:\"y,omitempty\"`\n\t}\n\ts8 := S4{\n\t\tX: 1,\n\t}\n\tv8, err := Marshal(vs, s8)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S4\", types.StructData{\n\t\t\"y\": types.Number(1),\n\t}).Equals(v8))\n\n\ts9 := S4{\n\t\tX: 0,\n\t}\n\tv9, err := Marshal(vs, s9)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S4\", types.StructData{}).Equals(v9))\n}\n\nfunc ExampleMarshal() {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype Person struct {\n\t\tGiven string\n\t\tMale bool\n\t}\n\tarya, err := Marshal(vs, Person{\"Arya\", false})\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n\n\tfmt.Printf(\"Given: %s, Male: %t\\n\", arya.(types.Struct).Get(\"given\").(types.String), arya.(types.Struct).Get(\"male\").(types.Bool))\n\t// Output: Given: Arya, Male: false\n}\n\nfunc TestEncodeSlice(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tv, err := Marshal(vs, []string{\"a\", \"b\", \"c\"})\n\tassert.NoError(err)\n\tassert.True(types.NewList(vs, types.String(\"a\"), types.String(\"b\"), types.String(\"c\")).Equals(v))\n}\n\nfunc TestEncodeArray(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tv, err := Marshal(vs, [3]int{1, 2, 3})\n\tassert.NoError(err)\n\tassert.True(types.NewList(vs, types.Number(1), types.Number(2), types.Number(3)).Equals(v))\n}\n\nfunc TestEncodeStructWithSlice(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tList []int\n\t}\n\tv, err := Marshal(vs, S{[]int{1, 2, 3}})\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S\", types.StructData{\n\t\t\"list\": types.NewList(vs, types.Number(1), types.Number(2), types.Number(3)),\n\t}).Equals(v))\n}\n\nfunc TestEncodeStructWithArrayOfNomsValue(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tList [1]types.Set\n\t}\n\tv, err := Marshal(vs, S{[1]types.Set{types.NewSet(vs, types.Bool(true))}})\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S\", types.StructData{\n\t\t\"list\": types.NewList(vs, types.NewSet(vs, types.Bool(true))),\n\t}).Equals(v))\n}\n\nfunc TestEncodeNomsTypePtr(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttestMarshal := func(g interface{}, expected types.Value) {\n\t\tv, err := Marshal(vs, g)\n\t\tassert.NoError(err)\n\t\tassert.Equal(expected, v)\n\t}\n\n\ttype S struct {\n\t\tType *types.Type\n\t}\n\n\tprimitive := types.StringType\n\ttestMarshal(S{primitive}, types.NewStruct(\"S\", types.StructData{\"type\": primitive}))\n\n\tcomplex := types.MakeStructType(\"Complex\",\n\t\ttypes.StructField{\n\t\t\tName: \"stuff\",\n\t\t\tType: types.StringType,\n\t\t},\n\t)\n\ttestMarshal(S{complex}, types.NewStruct(\"S\", types.StructData{\"type\": complex}))\n}\n\nfunc TestEncodeRecursive(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype Node struct {\n\t\tValue int\n\t\tChildren []Node\n\t}\n\tv, err := Marshal(vs, Node{\n\t\t1, []Node{\n\t\t\t{2, []Node{}},\n\t\t\t{3, []Node(nil)},\n\t\t},\n\t})\n\tassert.NoError(err)\n\n\ttyp := types.MakeStructType(\"Node\",\n\t\ttypes.StructField{\n\t\t\tName: \"children\",\n\t\t\tType: types.MakeListType(types.MakeCycleType(\"Node\")),\n\t\t},\n\t\ttypes.StructField{\n\t\t\tName: \"value\",\n\t\t\tType: types.NumberType,\n\t\t},\n\t)\n\tassert.True(typ.Equals(types.TypeOf(v)))\n\n\tassert.True(types.NewStruct(\"Node\", types.StructData{\n\t\t\"children\": types.NewList(\n\t\t\tvs,\n\t\t\ttypes.NewStruct(\"Node\", types.StructData{\n\t\t\t\t\"children\": types.NewList(vs),\n\t\t\t\t\"value\": types.Number(2),\n\t\t\t}),\n\t\t\ttypes.NewStruct(\"Node\", types.StructData{\n\t\t\t\t\"children\": types.NewList(vs),\n\t\t\t\t\"value\": types.Number(3),\n\t\t\t}),\n\t\t),\n\t\t\"value\": types.Number(1),\n\t}).Equals(v))\n}\n\nfunc TestEncodeMap(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tv, err := Marshal(vs, map[string]int{\"a\": 1, \"b\": 2, \"c\": 3})\n\tassert.NoError(err)\n\tassert.True(types.NewMap(\n\t\tvs,\n\t\ttypes.String(\"a\"), types.Number(1),\n\t\ttypes.String(\"b\"), types.Number(2),\n\t\ttypes.String(\"c\"), types.Number(3)).Equals(v))\n\n\ttype S struct {\n\t\tN string\n\t}\n\tv, err = Marshal(vs, map[S]bool{S{\"Yes\"}: true, S{\"No\"}: false})\n\tassert.NoError(err)\n\tassert.True(types.NewMap(\n\t\tvs,\n\t\ttypes.NewStruct(\"S\", types.StructData{\"n\": types.String(\"Yes\")}), types.Bool(true),\n\t\ttypes.NewStruct(\"S\", types.StructData{\"n\": types.String(\"No\")}), types.Bool(false)).Equals(v))\n\n\tv, err = Marshal(vs, map[string]int(nil))\n\tassert.NoError(err)\n\tassert.True(types.NewMap(vs).Equals(v))\n\n\tv, err = Marshal(vs, map[string]int{})\n\tassert.NoError(err)\n\tassert.True(types.NewMap(vs).Equals(v))\n}\n\nfunc TestEncodeInterface(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tvar i interface{}\n\ti = []string{\"a\", \"b\"}\n\tv, err := Marshal(vs, i)\n\tassert.NoError(err)\n\tassert.True(types.NewList(vs, types.String(\"a\"), types.String(\"b\")).Equals(v))\n\n\ti = map[interface{}]interface{}{\"a\": true, struct{ Name string }{\"b\"}: 42}\n\tv, err = Marshal(vs, i)\n\tassert.NoError(err)\n\tassert.True(types.NewMap(\n\t\tvs,\n\t\ttypes.String(\"a\"), types.Bool(true),\n\t\ttypes.NewStruct(\"\", types.StructData{\"name\": types.String(\"b\")}), types.Number(42),\n\t).Equals(v))\n}\n\nfunc TestEncodeSet(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tv, err := Marshal(vs, struct {\n\t\tA map[int]struct{} `noms:\",set\"`\n\t\tB map[int]struct{}\n\t\tC map[int]string `noms:\",set\"`\n\t\tD map[string]struct{} `noms:\",set\"`\n\t\tE map[string]struct{}\n\t\tF map[string]int `noms:\",set\"`\n\t\tG []int `noms:\",set\"`\n\t\tH string `noms:\",set\"`\n\t}{\n\t\tmap[int]struct{}{0: {}, 1: {}, 2: {}},\n\t\tmap[int]struct{}{3: {}, 4: {}, 5: {}},\n\t\tmap[int]string{},\n\t\tmap[string]struct{}{\"A\": {}, \"B\": {}, \"C\": {}},\n\t\tmap[string]struct{}{\"D\": {}, \"E\": {}, \"F\": {}},\n\t\tmap[string]int{},\n\t\t[]int{1, 2, 3},\n\t\t\"\",\n\t})\n\tassert.NoError(err)\n\ts, ok := v.(types.Struct)\n\tassert.True(ok)\n\n\texpect := map[string]types.NomsKind{\n\t\t\"a\": types.SetKind,\n\t\t\"b\": types.MapKind,\n\t\t\"c\": types.MapKind,\n\t\t\"d\": types.SetKind,\n\t\t\"e\": types.MapKind,\n\t\t\"f\": types.MapKind,\n\t\t\"g\": types.SetKind,\n\t\t\"h\": types.StringKind,\n\t}\n\tfor fieldName, kind := range expect {\n\t\tassert.Equal(kind, s.Get(fieldName).Kind())\n\t}\n\n\t// Test both the Set values are correct, and that the equivalent typed Map\n\t// are correct in case the Set marshaling interferes with it.\n\n\ta := s.Get(\"a\").(types.Set)\n\tassert.True(a.Has(types.Number(0)))\n\tassert.True(a.Has(types.Number(1)))\n\tassert.True(a.Has(types.Number(2)))\n\n\tb := s.Get(\"b\").(types.Map)\n\tassert.True(b.Has(types.Number(3)))\n\tassert.True(b.Has(types.Number(4)))\n\tassert.True(b.Has(types.Number(5)))\n\n\td := s.Get(\"d\").(types.Set)\n\tassert.True(d.Has(types.String(\"A\")))\n\tassert.True(d.Has(types.String(\"B\")))\n\tassert.True(d.Has(types.String(\"C\")))\n\n\te := s.Get(\"e\").(types.Map)\n\tassert.True(e.Has(types.String(\"D\")))\n\tassert.True(e.Has(types.String(\"E\")))\n\tassert.True(e.Has(types.String(\"F\")))\n\n\tg := s.Get(\"g\").(types.Set)\n\tassert.True(g.Has(types.Number(1)))\n\tassert.True(g.Has(types.Number(2)))\n\tassert.True(g.Has(types.Number(3)))\n}\n\nfunc TestEncodeOpt(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttc := []struct {\n\t\tin interface{}\n\t\topt Opt\n\t\twantValue types.Value\n\t}{\n\t\t{\n\t\t\t[]string{\"a\", \"b\"},\n\t\t\tOpt{},\n\t\t\ttypes.NewList(vs, types.String(\"a\"), types.String(\"b\")),\n\t\t},\n\t\t{\n\t\t\t[]string{\"a\", \"b\"},\n\t\t\tOpt{Set: true},\n\t\t\ttypes.NewSet(vs, types.String(\"a\"), types.String(\"b\")),\n\t\t},\n\t\t{\n\t\t\tmap[string]struct{}{\"a\": struct{}{}, \"b\": struct{}{}},\n\t\t\tOpt{},\n\t\t\ttypes.NewMap(vs, types.String(\"a\"), types.NewStruct(\"\", nil), types.String(\"b\"), types.NewStruct(\"\", nil)),\n\t\t},\n\t\t{\n\t\t\tmap[string]struct{}{\"a\": struct{}{}, \"b\": struct{}{}},\n\t\t\tOpt{Set: true},\n\t\t\ttypes.NewSet(vs, types.String(\"a\"), types.String(\"b\")),\n\t\t},\n\t}\n\n\tfor _, t := range tc {\n\t\tr, err := MarshalOpt(vs, t.in, t.opt)\n\t\tassert.True(t.wantValue.Equals(r))\n\t\tassert.Nil(err)\n\t}\n}\n\nfunc TestEncodeSetWithTags(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tv, err := Marshal(vs, struct {\n\t\tA map[int]struct{} `noms:\"foo,set\"`\n\t\tB map[int]struct{} `noms:\",omitempty,set\"`\n\t\tC map[int]struct{} `noms:\"bar,omitempty,set\"`\n\t}{\n\t\tA: map[int]struct{}{0: {}, 1: {}},\n\t\tC: map[int]struct{}{2: {}, 3: {}},\n\t})\n\tassert.NoError(err)\n\ts, ok := v.(types.Struct)\n\tassert.True(ok)\n\n\t_, ok = s.MaybeGet(\"a\")\n\tassert.False(ok)\n\t_, ok = s.MaybeGet(\"b\")\n\tassert.False(ok)\n\t_, ok = s.MaybeGet(\"c\")\n\tassert.False(ok)\n\n\tfoo, ok := s.Get(\"foo\").(types.Set)\n\tassert.True(ok)\n\tassert.True(types.NewSet(vs, types.Number(0), types.Number(1)).Equals(foo))\n\n\tbar, ok := s.Get(\"bar\").(types.Set)\n\tassert.True(ok)\n\tassert.True(types.NewSet(vs, types.Number(2), types.Number(3)).Equals(bar))\n}\n\nfunc TestInvalidTag(t *testing.T) {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\t_, err := Marshal(vs, struct {\n\t\tF string `noms:\",omitEmpty\"`\n\t}{\"F\"})\n\tassert.Error(t, err)\n\tassert.Equal(t, `Unrecognized tag: omitEmpty`, err.Error())\n}\n\nfunc TestEncodeCanSkipUnexportedField(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tAbc int\n\t\tnotExported bool `noms:\"-\"`\n\t}\n\ts := S{42, true}\n\tv, err := Marshal(vs, s)\n\tassert.NoError(err)\n\tassert.True(types.NewStruct(\"S\", types.StructData{\n\t\t\"abc\": types.Number(42),\n\t}).Equals(v))\n}\n\nfunc TestEncodeOriginal(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tFoo int `noms:\",omitempty\"`\n\t\tBar types.Struct `noms:\",original\"`\n\t}\n\n\tvar s S\n\tvar err error\n\tvar orig types.Struct\n\n\t// New field value clobbers old field value\n\torig = types.NewStruct(\"S\", types.StructData{\n\t\t\"foo\": types.Number(42),\n\t})\n\terr = Unmarshal(orig, &s)\n\tassert.NoError(err)\n\ts.Foo = 43\n\tassert.True(MustMarshal(vs, s).Equals(orig.Set(\"foo\", types.Number(43))))\n\n\t// New field extends old struct\n\torig = types.NewStruct(\"S\", types.StructData{})\n\terr = Unmarshal(orig, &s)\n\tassert.NoError(err)\n\ts.Foo = 43\n\tassert.True(MustMarshal(vs, s).Equals(orig.Set(\"foo\", types.Number(43))))\n\n\t// Old struct name always used\n\torig = types.NewStruct(\"Q\", types.StructData{})\n\terr = Unmarshal(orig, &s)\n\tassert.NoError(err)\n\ts.Foo = 43\n\tassert.True(MustMarshal(vs, s).Equals(orig.Set(\"foo\", types.Number(43))))\n\n\t// Field type of base are preserved\n\torig = types.NewStruct(\"S\", types.StructData{\n\t\t\"foo\": types.Number(42),\n\t})\n\terr = Unmarshal(orig, &s)\n\tassert.NoError(err)\n\ts.Foo = 43\n\tout := MustMarshal(vs, s)\n\tassert.True(out.Equals(orig.Set(\"foo\", types.Number(43))))\n\n\tst2 := types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"foo\": types.NumberType,\n\t})\n\tassert.True(types.TypeOf(out).Equals(st2))\n\n\t// It's OK to have an empty original field\n\ts = S{\n\t\tFoo: 42,\n\t}\n\tassert.True(MustMarshal(vs, s).Equals(\n\t\ttypes.NewStruct(\"S\", types.StructData{\"foo\": types.Number(float64(42))})))\n}\n\nfunc TestNomsTypes(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttype S struct {\n\t\tBlob types.Blob\n\t\tBool types.Bool\n\t\tNumber types.Number\n\t\tString types.String\n\t\tType *types.Type\n\t}\n\ts := S{\n\t\tBlob: types.NewBlob(vs),\n\t\tBool: types.Bool(true),\n\t\tNumber: types.Number(42),\n\t\tString: types.String(\"hi\"),\n\t\tType: types.NumberType,\n\t}\n\tassert.True(MustMarshal(vs, s).Equals(\n\t\ttypes.NewStruct(\"S\", types.StructData{\n\t\t\t\"blob\": types.NewBlob(vs),\n\t\t\t\"bool\": types.Bool(true),\n\t\t\t\"number\": types.Number(42),\n\t\t\t\"string\": types.String(\"hi\"),\n\t\t\t\"type\": types.NumberType,\n\t\t}),\n\t))\n}\n\ntype primitiveType int\n\nfunc (t primitiveType) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\treturn types.Number(int(t) + 1), nil\n}\n\nfunc TestMarshalerPrimitiveType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tu := primitiveType(42)\n\tv := MustMarshal(vs, u)\n\tassert.Equal(types.Number(43), v)\n}\n\ntype primitiveSliceType []string\n\nfunc (u primitiveSliceType) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\treturn types.String(strings.Join(u, \",\")), nil\n}\n\nfunc TestMarshalerPrimitiveSliceType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tu := primitiveSliceType([]string{\"a\", \"b\", \"c\"})\n\tv := MustMarshal(vs, u)\n\tassert.Equal(types.String(\"a,b,c\"), v)\n}\n\ntype primitiveMapType map[string]string\n\nfunc (u primitiveMapType) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\tvar vals types.ValueSlice\n\tfor k, v := range u {\n\t\tvals = append(vals, types.String(k+\",\"+v))\n\t}\n\treturn types.NewSet(vrw, vals...), nil\n}\n\nfunc TestMarshalerPrimitiveMapType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tu := primitiveMapType(map[string]string{\n\t\t\"a\": \"foo\",\n\t\t\"b\": \"bar\",\n\t})\n\tv := MustMarshal(vs, u)\n\tassert.True(types.NewSet(vs, types.String(\"a,foo\"), types.String(\"b,bar\")).Equals(v))\n}\n\ntype primitiveStructType struct {\n\tx, y int\n}\n\nfunc (u primitiveStructType) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\treturn types.Number(u.x + u.y), nil\n}\n\nfunc TestMarshalerPrimitiveStructType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tu := primitiveStructType{1, 2}\n\tv := MustMarshal(vs, u)\n\tassert.Equal(types.Number(3), v)\n}\n\ntype builtinType regexp.Regexp\n\nfunc (u builtinType) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\tr := regexp.Regexp(u)\n\treturn types.String(r.String()), nil\n}\n\nfunc TestMarshalerBuiltinType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ts := \"[a-z]+$\"\n\tr := regexp.MustCompile(s)\n\tu := builtinType(*r)\n\tv := MustMarshal(vs, u)\n\tassert.Equal(types.String(s), v)\n}\n\ntype wrappedMarshalerType primitiveType\n\nfunc (u wrappedMarshalerType) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\treturn types.Number(int(u) + 2), nil\n}\n\nfunc TestMarshalerWrapperMarshalerType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tu := wrappedMarshalerType(primitiveType(42))\n\tv := MustMarshal(vs, u)\n\tassert.Equal(types.Number(44), v)\n}\n\ntype TestComplexStructType struct {\n\tP primitiveType\n\tPs []primitiveType\n\tPm map[string]primitiveType\n\tPslice primitiveSliceType\n\tPmap primitiveMapType\n\tPstruct primitiveStructType\n\tB builtinType\n}\n\nfunc TestMarshalerComplexStructType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ts := \"foo|bar\"\n\tr := regexp.MustCompile(s)\n\tu := TestComplexStructType{\n\t\tP: 42,\n\t\tPs: []primitiveType{1, 2},\n\t\tPm: map[string]primitiveType{\n\t\t\t\"x\": 100,\n\t\t\t\"y\": 101,\n\t\t},\n\t\tPslice: primitiveSliceType{\"a\", \"b\", \"c\"},\n\t\tPmap: primitiveMapType{\n\t\t\t\"c\": \"123\",\n\t\t\t\"d\": \"456\",\n\t\t},\n\t\tPstruct: primitiveStructType{10, 20},\n\t\tB: builtinType(*r),\n\t}\n\n\tv := MustMarshal(vs, u)\n\n\tassert.True(types.NewStruct(\"TestComplexStructType\", types.StructData{\n\t\t\"p\": types.Number(43),\n\t\t\"ps\": types.NewList(vs, types.Number(2), types.Number(3)),\n\t\t\"pm\": types.NewMap(vs, types.String(\"x\"), types.Number(101), types.String(\"y\"), types.Number(102)),\n\t\t\"pslice\": types.String(\"a,b,c\"),\n\t\t\"pmap\": types.NewSet(vs, types.String(\"c,123\"), types.String(\"d,456\")),\n\t\t\"pstruct\": types.Number(30),\n\t\t\"b\": types.String(s),\n\t}).Equals(v))\n}\n\ntype returnsMarshalerError struct {\n\terr error\n}\n\nfunc (u returnsMarshalerError) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\treturn nil, u.err\n}\n\ntype returnsMarshalerNil struct{}\n\nfunc (u returnsMarshalerNil) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\treturn nil, nil\n}\n\ntype panicsMarshaler struct{}\n\nfunc (u panicsMarshaler) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\tpanic(\"panic\")\n}\n\nfunc TestMarshalerErrors(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\texpErr := errors.New(\"expected error\")\n\tm1 := returnsMarshalerError{expErr}\n\t_, actErr := Marshal(vs, m1)\n\tassert.Equal(expErr, actErr)\n\n\tm2 := returnsMarshalerNil{}\n\tassert.Panics(func() { Marshal(vs, m2) })\n\n\tm3 := panicsMarshaler{}\n\tassert.Panics(func() { Marshal(vs, m3) })\n}\n\ntype TestStructWithNameImpl struct {\n\tX int\n}\n\nfunc (ts TestStructWithNameImpl) MarshalNomsStructName() string {\n\treturn \"A\"\n}\nfunc TestMarshalStructName(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tts := TestStructWithNameImpl{\n\t\tX: 1,\n\t}\n\tv := MustMarshal(vs, ts)\n\tassert.True(types.NewStruct(\"A\", types.StructData{\n\t\t\"x\": types.Number(1),\n\t}).Equals(v), types.EncodedValue(v))\n}\n\ntype TestStructWithNameImpl2 struct {\n\tX int\n}\n\nfunc (ts TestStructWithNameImpl2) MarshalNomsStructName() string {\n\treturn \"\"\n}\nfunc TestMarshalStructName2(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tts := TestStructWithNameImpl2{\n\t\tX: 1,\n\t}\n\tv := MustMarshal(vs, ts)\n\tassert.True(types.NewStruct(\"\", types.StructData{\n\t\t\"x\": types.Number(1),\n\t}).Equals(v), types.EncodedValue(v))\n}\n" }, { "path": "go/marshal/encode_type.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package marshal implements encoding and decoding of Noms values. The mapping\n// between Noms objects and Go values is described in the documentation for the\n// Marshal and Unmarshal functions.\npackage marshal\n\nimport (\n\t\"fmt\"\n\t\"reflect\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// MarshalType computes a Noms type from a Go type\n//\n// The rules for MarshalType is the same as for Marshal, except for omitempty\n// which leads to an optional field since it depends on the runtime value and\n// can lead to the property not being present.\n//\n// If a Go struct contains a noms tag with original the field is skipped since\n// the Noms type depends on the original Noms value which is not available.\nfunc MarshalType(v interface{}) (nt *types.Type, err error) {\n\treturn MarshalTypeOpt(v, Opt{})\n}\n\n// MarshalTypeOpt is like MarshalType but with additional options.\nfunc MarshalTypeOpt(v interface{}, opt Opt) (nt *types.Type, err error) {\n\tdefer func() {\n\t\tif r := recover(); r != nil {\n\t\t\tswitch r := r.(type) {\n\t\t\tcase *UnsupportedTypeError, *InvalidTagError:\n\t\t\t\terr = r.(error)\n\t\t\tcase *marshalNomsError:\n\t\t\t\terr = r.err\n\t\t\tdefault:\n\t\t\t\tpanic(r)\n\t\t\t}\n\t\t}\n\t}()\n\tnt = MustMarshalTypeOpt(v, opt)\n\treturn\n}\n\n// MustMarshalType computes a Noms type from a Go type or panics if there is an\n// error.\nfunc MustMarshalType(v interface{}) (nt *types.Type) {\n\treturn MustMarshalTypeOpt(v, Opt{})\n}\n\n// MustMarshalTypeOpt is like MustMarshalType but provides additional options.\nfunc MustMarshalTypeOpt(v interface{}, opt Opt) (nt *types.Type) {\n\trv := reflect.ValueOf(v)\n\ttags := nomsTags{\n\t\tset: opt.Set,\n\t}\n\tnt = encodeType(rv.Type(), map[string]reflect.Type{}, tags)\n\n\tif nt == nil {\n\t\tpanic(&UnsupportedTypeError{Type: rv.Type()})\n\t}\n\n\treturn\n}\n\n// TypeMarshaler is an interface types can implement to provide their own\n// encoding of type.\ntype TypeMarshaler interface {\n\t// MarshalNomsType returns the Noms Type encoding of a type, or an error.\n\t// nil is not a valid return val - if both val and err are nil, MarshalType\n\t// will panic.\n\tMarshalNomsType() (t *types.Type, err error)\n}\n\nvar typeOfTypesType = reflect.TypeOf((*types.Type)(nil))\nvar typeMarshalerInterface = reflect.TypeOf((*TypeMarshaler)(nil)).Elem()\n\nfunc encodeType(t reflect.Type, seenStructs map[string]reflect.Type, tags nomsTags) *types.Type {\n\tif t.Implements(typeMarshalerInterface) {\n\t\tv := reflect.Zero(t)\n\t\ttyp, err := v.Interface().(TypeMarshaler).MarshalNomsType()\n\t\tif err != nil {\n\t\t\tpanic(&marshalNomsError{err})\n\t\t}\n\t\tif typ == nil {\n\t\t\tpanic(fmt.Errorf(\"nil result from %s.MarshalNomsType\", t))\n\t\t}\n\t\treturn typ\n\t}\n\n\tif t.Implements(marshalerInterface) {\n\t\t// There is no way to determine the noms type now. For Marshal it can be\n\t\t// different each time MarshalNoms is called and is handled further up the\n\t\t// stack.\n\t\terr := fmt.Errorf(\"Cannot marshal type which implements %s, perhaps implement %s for %s\", marshalerInterface, typeMarshalerInterface, t)\n\t\tpanic(&marshalNomsError{err})\n\t}\n\n\tif t.Implements(nomsValueInterface) {\n\t\tif t == typeOfTypesType {\n\t\t\treturn types.TypeType\n\t\t}\n\n\t\t// Use Name because List and Blob are convertible to each other on Go.\n\t\tswitch t.Name() {\n\t\tcase \"Blob\":\n\t\t\treturn types.BlobType\n\t\tcase \"Bool\":\n\t\t\treturn types.BoolType\n\t\tcase \"List\":\n\t\t\treturn types.MakeListType(types.ValueType)\n\t\tcase \"Map\":\n\t\t\treturn types.MakeMapType(types.ValueType, types.ValueType)\n\t\tcase \"Number\":\n\t\t\treturn types.NumberType\n\t\tcase \"Ref\":\n\t\t\treturn types.MakeRefType(types.ValueType)\n\t\tcase \"Set\":\n\t\t\treturn types.MakeSetType(types.ValueType)\n\t\tcase \"String\":\n\t\t\treturn types.StringType\n\t\tcase \"Value\":\n\t\t\treturn types.ValueType\n\t\t}\n\n\t\terr := fmt.Errorf(\"Cannot marshal type %s, it requires type parameters\", t)\n\t\tpanic(&marshalNomsError{err})\n\t}\n\n\tswitch t.Kind() {\n\tcase reflect.Bool:\n\t\treturn types.BoolType\n\tcase reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64:\n\t\treturn types.NumberType\n\tcase reflect.String:\n\t\treturn types.StringType\n\tcase reflect.Struct:\n\t\treturn structEncodeType(t, seenStructs)\n\tcase reflect.Array, reflect.Slice:\n\t\telemType := encodeType(t.Elem(), seenStructs, nomsTags{})\n\t\tif elemType == nil {\n\t\t\tbreak\n\t\t}\n\t\tif shouldEncodeAsSet(t, tags) {\n\t\t\treturn types.MakeSetType(elemType)\n\t\t}\n\t\treturn types.MakeListType(elemType)\n\tcase reflect.Map:\n\t\tkeyType := encodeType(t.Key(), seenStructs, nomsTags{})\n\t\tif keyType == nil {\n\t\t\tbreak\n\t\t}\n\n\t\tif shouldEncodeAsSet(t, tags) {\n\t\t\treturn types.MakeSetType(keyType)\n\t\t}\n\n\t\tvalueType := encodeType(t.Elem(), seenStructs, nomsTags{})\n\t\tif valueType != nil {\n\t\t\treturn types.MakeMapType(keyType, valueType)\n\t\t}\n\t}\n\n\t// This will be reported as an error at a different layer.\n\treturn nil\n}\n\n// structEncodeType returns the Noms types.Type if it can be determined from the\n// reflect.Type. In some cases we cannot determine the type by only looking at\n// the type but we also need to look at the value. In these cases this returns\n// nil and we have to wait until we have a value to be able to determine the\n// type.\nfunc structEncodeType(t reflect.Type, seenStructs map[string]reflect.Type) *types.Type {\n\tname := getStructName(t)\n\tif name != \"\" {\n\t\tif _, ok := seenStructs[name]; ok {\n\t\t\treturn types.MakeCycleType(name)\n\t\t}\n\t\tseenStructs[name] = t\n\t}\n\n\tfields, knownShape, _ := typeFields(t, seenStructs, true, false)\n\n\tvar structType *types.Type\n\tif knownShape {\n\t\tstructTypeFields := make([]types.StructField, len(fields))\n\t\tfor i, fs := range fields {\n\t\t\tstructTypeFields[i] = types.StructField{\n\t\t\t\tName: fs.name,\n\t\t\t\tType: fs.nomsType,\n\t\t\t\tOptional: fs.omitEmpty,\n\t\t\t}\n\t\t}\n\t\tstructType = types.MakeStructType(getStructName(t), structTypeFields...)\n\t}\n\n\treturn structType\n}\n" }, { "path": "go/marshal/encode_type_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage marshal\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"reflect\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/nomdl\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestMarshalTypeType(tt *testing.T) {\n\tt := func(exp *types.Type, ptr interface{}) {\n\t\tp := reflect.ValueOf(ptr)\n\t\tassert.NotEqual(tt, reflect.Ptr, p.Type().Kind())\n\t\tactual, err := MarshalType(p.Interface())\n\t\tassert.NoError(tt, err)\n\t\tassert.NotNil(tt, actual, \"%#v\", p.Interface())\n\t\tassert.True(tt, exp.Equals(actual))\n\t}\n\n\tt(types.NumberType, float32(0))\n\tt(types.NumberType, float64(0))\n\tt(types.NumberType, int(0))\n\tt(types.NumberType, int16(0))\n\tt(types.NumberType, int32(0))\n\tt(types.NumberType, int64(0))\n\tt(types.NumberType, int8(0))\n\tt(types.NumberType, uint(0))\n\tt(types.NumberType, uint16(0))\n\tt(types.NumberType, uint32(0))\n\tt(types.NumberType, uint64(0))\n\tt(types.NumberType, uint8(0))\n\n\tt(types.BoolType, true)\n\tt(types.StringType, \"hi\")\n\n\tvar l []int\n\tt(types.MakeListType(types.NumberType), l)\n\n\tvar m map[uint32]string\n\tt(types.MakeMapType(types.NumberType, types.StringType), m)\n\n\tt(types.MakeListType(types.ValueType), types.List{})\n\tt(types.MakeSetType(types.ValueType), types.Set{})\n\tt(types.MakeMapType(types.ValueType, types.ValueType), types.Map{})\n\tt(types.MakeRefType(types.ValueType), types.Ref{})\n\n\ttype TestStruct struct {\n\t\tStr string\n\t\tNum float64\n\t}\n\tvar str TestStruct\n\tt(types.MakeStructTypeFromFields(\"TestStruct\", types.FieldMap{\n\t\t\"str\": types.StringType,\n\t\t\"num\": types.NumberType,\n\t}), str)\n\n\t// Same again to test caching\n\tt(types.MakeStructTypeFromFields(\"TestStruct\", types.FieldMap{\n\t\t\"str\": types.StringType,\n\t\t\"num\": types.NumberType,\n\t}), str)\n\n\tanonStruct := struct {\n\t\tB bool\n\t}{\n\t\ttrue,\n\t}\n\tt(types.MakeStructTypeFromFields(\"\", types.FieldMap{\n\t\t\"b\": types.BoolType,\n\t}), anonStruct)\n\n\ttype TestNestedStruct struct {\n\t\tA []int16\n\t\tB TestStruct\n\t\tC float64\n\t}\n\tvar nestedStruct TestNestedStruct\n\tt(types.MakeStructTypeFromFields(\"TestNestedStruct\", types.FieldMap{\n\t\t\"a\": types.MakeListType(types.NumberType),\n\t\t\"b\": types.MakeStructTypeFromFields(\"TestStruct\", types.FieldMap{\n\t\t\t\"str\": types.StringType,\n\t\t\t\"num\": types.NumberType,\n\t\t}),\n\t\t\"c\": types.NumberType,\n\t}), nestedStruct)\n\n\ttype testStruct struct {\n\t\tStr string\n\t\tNum float64\n\t}\n\tvar ts testStruct\n\tt(types.MakeStructTypeFromFields(\"TestStruct\", types.FieldMap{\n\t\t\"str\": types.StringType,\n\t\t\"num\": types.NumberType,\n\t}), ts)\n}\n\n//\nfunc assertMarshalTypeErrorMessage(t *testing.T, v interface{}, expectedMessage string) {\n\t_, err := MarshalType(v)\n\tassert.Error(t, err)\n\tassert.Equal(t, expectedMessage, err.Error())\n}\n\nfunc TestMarshalTypeInvalidTypes(t *testing.T) {\n\tassertMarshalTypeErrorMessage(t, make(chan int), \"Type is not supported, type: chan int\")\n}\n\nfunc TestMarshalTypeEmbeddedStruct(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype EmbeddedStruct struct {\n\t\tB bool\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct\n\t\tA int\n\t}\n\n\tvar s TestStruct\n\ttyp := MustMarshalType(s)\n\n\tassert.True(types.MakeStructTypeFromFields(\"TestStruct\", types.FieldMap{\n\t\t\"a\": types.NumberType,\n\t\t\"b\": types.BoolType,\n\t}).Equals(typ))\n}\n\nfunc TestMarshalTypeEmbeddedStructSkip(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype EmbeddedStruct struct {\n\t\tB bool\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct `noms:\"-\"`\n\t\tA int\n\t}\n\n\tvar s TestStruct\n\ttyp := MustMarshalType(s)\n\n\tassert.True(types.MakeStructTypeFromFields(\"TestStruct\", types.FieldMap{\n\t\t\"a\": types.NumberType,\n\t}).Equals(typ))\n}\n\nfunc TestMarshalTypeEmbeddedStructNamed(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype EmbeddedStruct struct {\n\t\tB bool\n\t}\n\ttype TestStruct struct {\n\t\tEmbeddedStruct `noms:\"em\"`\n\t\tA int\n\t}\n\n\tvar s TestStruct\n\ttyp := MustMarshalType(s)\n\n\tassert.True(types.MakeStructTypeFromFields(\"TestStruct\", types.FieldMap{\n\t\t\"a\": types.NumberType,\n\t\t\"em\": types.MakeStructTypeFromFields(\"EmbeddedStruct\", types.FieldMap{\n\t\t\t\"b\": types.BoolType,\n\t\t}),\n\t}).Equals(typ))\n}\n\nfunc TestMarshalTypeEncodeNonExportedField(t *testing.T) {\n\ttype TestStruct struct {\n\t\tx int\n\t}\n\tassertMarshalTypeErrorMessage(t, TestStruct{1}, \"Non exported fields are not supported, type: marshal.TestStruct\")\n}\n\nfunc TestMarshalTypeEncodeTaggingSkip(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tAbc int `noms:\"-\"`\n\t\tDef bool\n\t}\n\tvar s S\n\ttyp, err := MarshalType(s)\n\tassert.NoError(err)\n\tassert.True(types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"def\": types.BoolType,\n\t}).Equals(typ))\n}\n\nfunc TestMarshalTypeNamedFields(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tAaa int `noms:\"a\"`\n\t\tBbb bool `noms:\"B\"`\n\t\tCcc string\n\t}\n\tvar s S\n\ttyp, err := MarshalType(s)\n\tassert.NoError(err)\n\tassert.True(types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"a\": types.NumberType,\n\t\t\"B\": types.BoolType,\n\t\t\"ccc\": types.StringType,\n\t}).Equals(typ))\n}\n\nfunc TestMarshalTypeInvalidNamedFields(t *testing.T) {\n\ttype S struct {\n\t\tA int `noms:\"1a\"`\n\t}\n\tvar s S\n\tassertMarshalTypeErrorMessage(t, s, \"Invalid struct field name: 1a\")\n}\n\nfunc TestMarshalTypeOmitEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tString string `noms:\",omitempty\"`\n\t}\n\tvar s S\n\ttyp, err := MarshalType(s)\n\tassert.NoError(err)\n\tassert.True(types.MakeStructType(\"S\", types.StructField{Name: \"string\", Type: types.StringType, Optional: true}).Equals(typ))\n}\n\nfunc ExampleMarshalType() {\n\n\ttype Person struct {\n\t\tGiven string\n\t\tFemale bool\n\t}\n\tvar person Person\n\tpersonNomsType, err := MarshalType(person)\n\tif err != nil {\n\t\tfmt.Println(err)\n\t\treturn\n\t}\n\n\tfmt.Println(personNomsType.Describe())\n\t// Output: Struct Person {\n\t// female: Bool,\n\t// given: String,\n\t// }\n}\n\nfunc TestMarshalTypeSlice(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts := []string{\"a\", \"b\", \"c\"}\n\ttyp, err := MarshalType(s)\n\tassert.NoError(err)\n\tassert.True(types.MakeListType(types.StringType).Equals(typ))\n}\n\nfunc TestMarshalTypeArray(t *testing.T) {\n\tassert := assert.New(t)\n\n\ta := [3]int{1, 2, 3}\n\ttyp, err := MarshalType(a)\n\tassert.NoError(err)\n\tassert.True(types.MakeListType(types.NumberType).Equals(typ))\n}\n\nfunc TestMarshalTypeStructWithSlice(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tList []int\n\t}\n\tvar s S\n\ttyp, err := MarshalType(s)\n\tassert.NoError(err)\n\tassert.True(types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"list\": types.MakeListType(types.NumberType),\n\t}).Equals(typ))\n}\n\nfunc TestMarshalTypeRecursive(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype Node struct {\n\t\tValue int\n\t\tChildren []Node\n\t}\n\tvar n Node\n\ttyp, err := MarshalType(n)\n\tassert.NoError(err)\n\n\ttyp2 := types.MakeStructType(\"Node\",\n\t\ttypes.StructField{\n\t\t\tName: \"children\",\n\t\t\tType: types.MakeListType(types.MakeCycleType(\"Node\")),\n\t\t},\n\t\ttypes.StructField{\n\t\t\tName: \"value\",\n\t\t\tType: types.NumberType,\n\t\t},\n\t)\n\tassert.True(typ2.Equals(typ))\n}\n\nfunc TestMarshalTypeMap(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar m map[string]int\n\ttyp, err := MarshalType(m)\n\tassert.NoError(err)\n\tassert.True(types.MakeMapType(types.StringType, types.NumberType).Equals(typ))\n\n\ttype S struct {\n\t\tN string\n\t}\n\n\tvar m2 map[S]bool\n\ttyp, err = MarshalType(m2)\n\tassert.NoError(err)\n\tassert.True(types.MakeMapType(\n\t\ttypes.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\t\"n\": types.StringType,\n\t\t}),\n\t\ttypes.BoolType).Equals(typ))\n}\n\nfunc TestMarshalTypeSet(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tA map[int]struct{} `noms:\",set\"`\n\t\tB map[int]struct{}\n\t\tC map[int]string `noms:\",set\"`\n\t\tD map[string]struct{} `noms:\",set\"`\n\t\tE map[string]struct{}\n\t\tF map[string]int `noms:\",set\"`\n\t\tG []int `noms:\",set\"`\n\t\tH string `noms:\",set\"`\n\t}\n\tvar s S\n\ttyp, err := MarshalType(s)\n\tassert.NoError(err)\n\n\temptyStructType := types.MakeStructTypeFromFields(\"\", types.FieldMap{})\n\n\tassert.True(types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"a\": types.MakeSetType(types.NumberType),\n\t\t\"b\": types.MakeMapType(types.NumberType, emptyStructType),\n\t\t\"c\": types.MakeMapType(types.NumberType, types.StringType),\n\t\t\"d\": types.MakeSetType(types.StringType),\n\t\t\"e\": types.MakeMapType(types.StringType, emptyStructType),\n\t\t\"f\": types.MakeMapType(types.StringType, types.NumberType),\n\t\t\"g\": types.MakeSetType(types.NumberType),\n\t\t\"h\": types.StringType,\n\t}).Equals(typ))\n}\n\nfunc TestEncodeTypeOpt(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttc := []struct {\n\t\tin interface{}\n\t\topt Opt\n\t\twantType *types.Type\n\t}{\n\t\t{\n\t\t\t[]string{},\n\t\t\tOpt{},\n\t\t\ttypes.MakeListType(types.StringType),\n\t\t},\n\t\t{\n\t\t\t[]string{},\n\t\t\tOpt{Set: true},\n\t\t\ttypes.MakeSetType(types.StringType),\n\t\t},\n\t\t{\n\t\t\tmap[string]struct{}{},\n\t\t\tOpt{},\n\t\t\ttypes.MakeMapType(types.StringType, types.MakeStructType(\"\")),\n\t\t},\n\t\t{\n\t\t\tmap[string]struct{}{},\n\t\t\tOpt{Set: true},\n\t\t\ttypes.MakeSetType(types.StringType),\n\t\t},\n\t}\n\n\tfor _, t := range tc {\n\t\tr, err := MarshalTypeOpt(t.in, t.opt)\n\t\tassert.True(t.wantType.Equals(r))\n\t\tassert.Nil(err)\n\t}\n}\n\nfunc TestMarshalTypeSetWithTags(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tA map[int]struct{} `noms:\"foo,set\"`\n\t\tB map[int]struct{} `noms:\",omitempty,set\"`\n\t\tC map[int]struct{} `noms:\"bar,omitempty,set\"`\n\t}\n\n\tvar s S\n\ttyp, err := MarshalType(s)\n\tassert.NoError(err)\n\tassert.True(types.MakeStructType(\"S\",\n\t\ttypes.StructField{Name: \"foo\", Type: types.MakeSetType(types.NumberType), Optional: false},\n\t\ttypes.StructField{Name: \"b\", Type: types.MakeSetType(types.NumberType), Optional: true},\n\t\ttypes.StructField{Name: \"bar\", Type: types.MakeSetType(types.NumberType), Optional: true},\n\t).Equals(typ))\n}\n\nfunc TestMarshalTypeInvalidTag(t *testing.T) {\n\n\ttype S struct {\n\t\tF string `noms:\",omitEmpty\"`\n\t}\n\tvar s S\n\t_, err := MarshalType(s)\n\tassert.Error(t, err)\n\tassert.Equal(t, `Unrecognized tag: omitEmpty`, err.Error())\n}\n\nfunc TestMarshalTypeCanSkipUnexportedField(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tAbc int\n\t\tnotExported bool `noms:\"-\"`\n\t}\n\tvar s S\n\ttyp, err := MarshalType(s)\n\tassert.NoError(err)\n\tassert.True(types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"abc\": types.NumberType,\n\t}).Equals(typ))\n}\n\nfunc TestMarshalTypeOriginal(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tFoo int `noms:\",omitempty\"`\n\t\tBar types.Struct `noms:\",original\"`\n\t}\n\n\tvar s S\n\ttyp, err := MarshalType(s)\n\tassert.NoError(err)\n\tassert.True(types.MakeStructType(\"S\",\n\t\ttypes.StructField{Name: \"foo\", Type: types.NumberType, Optional: true},\n\t).Equals(typ))\n}\n\nfunc TestMarshalTypeNomsTypes(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype S struct {\n\t\tBlob types.Blob\n\t\tBool types.Bool\n\t\tNumber types.Number\n\t\tString types.String\n\t\tType *types.Type\n\t}\n\tvar s S\n\tassert.True(MustMarshalType(s).Equals(\n\t\ttypes.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\t\"blob\": types.BlobType,\n\t\t\t\"bool\": types.BoolType,\n\t\t\t\"number\": types.NumberType,\n\t\t\t\"string\": types.StringType,\n\t\t\t\"type\": types.TypeType,\n\t\t}),\n\t))\n}\n\nfunc (t primitiveType) MarshalNomsType() (*types.Type, error) {\n\treturn types.NumberType, nil\n}\n\nfunc TestTypeMarshalerPrimitiveType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar u primitiveType\n\ttyp := MustMarshalType(u)\n\tassert.Equal(types.NumberType, typ)\n}\n\nfunc (u primitiveSliceType) MarshalNomsType() (*types.Type, error) {\n\treturn types.StringType, nil\n}\n\nfunc TestTypeMarshalerPrimitiveSliceType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar u primitiveSliceType\n\ttyp := MustMarshalType(u)\n\tassert.Equal(types.StringType, typ)\n}\n\nfunc (u primitiveMapType) MarshalNomsType() (*types.Type, error) {\n\treturn types.MakeSetType(types.StringType), nil\n}\n\nfunc TestTypeMarshalerPrimitiveMapType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar u primitiveMapType\n\ttyp := MustMarshalType(u)\n\tassert.Equal(types.MakeSetType(types.StringType), typ)\n}\n\nfunc TestTypeMarshalerPrimitiveStructTypeNoMarshalNomsType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar u primitiveStructType\n\t_, err := MarshalType(u)\n\tassert.Error(err)\n\tassert.Equal(\"Cannot marshal type which implements marshal.Marshaler, perhaps implement marshal.TypeMarshaler for marshal.primitiveStructType\", err.Error())\n}\n\nfunc (u builtinType) MarshalNomsType() (*types.Type, error) {\n\treturn types.StringType, nil\n}\n\nfunc TestTypeMarshalerBuiltinType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar u builtinType\n\ttyp := MustMarshalType(u)\n\tassert.Equal(types.StringType, typ)\n}\n\nfunc (u wrappedMarshalerType) MarshalNomsType() (*types.Type, error) {\n\treturn types.NumberType, nil\n}\n\nfunc TestTypeMarshalerWrapperMarshalerType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar u wrappedMarshalerType\n\ttyp := MustMarshalType(u)\n\tassert.Equal(types.NumberType, typ)\n}\n\nfunc (u returnsMarshalerError) MarshalNomsType() (*types.Type, error) {\n\treturn nil, errors.New(\"expected error\")\n}\n\nfunc (u returnsMarshalerNil) MarshalNomsType() (*types.Type, error) {\n\treturn nil, nil\n}\n\nfunc (u panicsMarshaler) MarshalNomsType() (*types.Type, error) {\n\tpanic(\"panic\")\n}\n\nfunc TestTypeMarshalerErrors(t *testing.T) {\n\tassert := assert.New(t)\n\n\texpErr := errors.New(\"expected error\")\n\tvar m1 returnsMarshalerError\n\t_, actErr := MarshalType(m1)\n\tassert.Equal(expErr, actErr)\n\n\tvar m2 returnsMarshalerNil\n\tassert.Panics(func() { MarshalType(m2) })\n\n\tvar m3 panicsMarshaler\n\tassert.Panics(func() { MarshalType(m3) })\n}\n\nfunc TestMarshalTypeStructName(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar ts TestStructWithNameImpl\n\ttyp := MustMarshalType(ts)\n\tassert.True(types.MakeStructType(\"A\", types.StructField{Name: \"x\", Type: types.NumberType, Optional: false}).Equals(typ), typ.Describe())\n}\n\nfunc TestMarshalTypeStructName2(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar ts TestStructWithNameImpl2\n\ttyp := MustMarshalType(ts)\n\tassert.True(types.MakeStructType(\"\", types.StructField{Name: \"x\", Type: types.NumberType, Optional: false}).Equals(typ), typ.Describe())\n}\n\ntype OutPhoto struct {\n\tFaces []OutFace `noms:\",set\"`\n\tSomeOtherFacesSet []OutFace `noms:\",set\"`\n}\n\ntype OutFace struct {\n\tBlob types.Ref\n}\n\nfunc (f OutFace) MarshalNomsStructName() string {\n\treturn \"Face\"\n}\n\nfunc TestMarshalTypeOutface(t *testing.T) {\n\n\ttyp := MustMarshalType(OutPhoto{})\n\texpectedType := nomdl.MustParseType(`Struct OutPhoto {\n faces: Set,\n }>,\n someOtherFacesSet: Set>,\n }`)\n\tassert.True(t, typ.Equals(expectedType))\n}\n" }, { "path": "go/merge/candidate.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage merge\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// candidate represents a collection that is a candidate to be merged. This\n// interface exists to wrap Maps, Sets and Structs with a common API so that\n// threeWayOrderedSequenceMerge() can remain agnostic to which kind of\n// collections it's actually working with.\ntype candidate interface {\n\tdiff(parent candidate, change chan<- types.ValueChanged, stop <-chan struct{})\n\tget(k types.Value) types.Value\n\tpathConcat(change types.ValueChanged, path types.Path) (out types.Path)\n\tgetValue() types.Value\n}\n\ntype mapCandidate struct {\n\tm types.Map\n}\n\nfunc (mc mapCandidate) diff(p candidate, change chan<- types.ValueChanged, stop <-chan struct{}) {\n\tmc.m.Diff(p.(mapCandidate).m, change, stop)\n}\n\nfunc (mc mapCandidate) get(k types.Value) types.Value {\n\treturn mc.m.Get(k)\n}\n\nfunc (mc mapCandidate) pathConcat(change types.ValueChanged, path types.Path) (out types.Path) {\n\tout = append(out, path...)\n\tif kind := change.Key.Kind(); kind == types.BoolKind || kind == types.StringKind || kind == types.NumberKind {\n\t\tout = append(out, types.NewIndexPath(change.Key))\n\t} else {\n\t\tout = append(out, types.NewHashIndexPath(change.Key.Hash()))\n\t}\n\treturn\n}\n\nfunc (mc mapCandidate) getValue() types.Value {\n\treturn mc.m\n}\n\ntype setCandidate struct {\n\ts types.Set\n}\n\nfunc (sc setCandidate) diff(p candidate, change chan<- types.ValueChanged, stop <-chan struct{}) {\n\tsc.s.Diff(p.(setCandidate).s, change, stop)\n}\n\nfunc (sc setCandidate) get(k types.Value) types.Value {\n\treturn k\n}\n\nfunc (sc setCandidate) pathConcat(change types.ValueChanged, path types.Path) (out types.Path) {\n\tout = append(out, path...)\n\tif kind := change.Key.Kind(); kind == types.BoolKind || kind == types.StringKind || kind == types.NumberKind {\n\t\tout = append(out, types.NewIndexPath(change.Key))\n\t} else {\n\t\tout = append(out, types.NewHashIndexPath(change.Key.Hash()))\n\t}\n\treturn\n}\n\nfunc (sc setCandidate) getValue() types.Value {\n\treturn sc.s\n}\n\ntype structCandidate struct {\n\ts types.Struct\n}\n\nfunc (sc structCandidate) diff(p candidate, change chan<- types.ValueChanged, stop <-chan struct{}) {\n\tsc.s.Diff(p.(structCandidate).s, change, stop)\n}\n\nfunc (sc structCandidate) get(key types.Value) types.Value {\n\tif field, ok := key.(types.String); ok {\n\t\tval, _ := sc.s.MaybeGet(string(field))\n\t\treturn val\n\t}\n\tpanic(fmt.Errorf(\"Bad key type in diff: %s\", types.TypeOf(key).Describe()))\n}\n\nfunc (sc structCandidate) pathConcat(change types.ValueChanged, path types.Path) (out types.Path) {\n\tout = append(out, path...)\n\tstr, ok := change.Key.(types.String)\n\tif !ok {\n\t\td.Panic(\"Field names must be strings, not %s\", types.TypeOf(change.Key).Describe())\n\t}\n\treturn append(out, types.NewFieldPath(string(str)))\n}\n\nfunc (sc structCandidate) getValue() types.Value {\n\treturn sc.s\n}\n" }, { "path": "go/merge/three_way.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage merge\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// Policy functors are used to merge two values (a and b) against a common\n// ancestor. All three Values and their must by wholly readable from vrw.\n// Whenever a change is merged, implementations should send a struct{} over\n// progress.\ntype Policy func(a, b, ancestor types.Value, vrw types.ValueReadWriter, progress chan struct{}) (merged types.Value, err error)\n\n// ResolveFunc is the type for custom merge-conflict resolution callbacks.\n// When the merge algorithm encounters two non-mergeable changes (aChange and\n// bChange) at the same path, it calls the ResolveFunc passed into ThreeWay().\n// The callback gets the types of the two incompatible changes (added, changed\n// or removed) and the two Values that could not be merged (if any). If the\n// ResolveFunc cannot devise a resolution, ok should be false upon return and\n// the other return values are undefined. If the conflict can be resolved, the\n// function should return the appropriate type of change to apply, the new value\n// to be used (if any), and true.\ntype ResolveFunc func(aChange, bChange types.DiffChangeType, a, b types.Value, path types.Path) (change types.DiffChangeType, merged types.Value, ok bool)\n\n// None is the no-op ResolveFunc. Any conflict results in a merge failure.\nfunc None(aChange, bChange types.DiffChangeType, a, b types.Value, path types.Path) (change types.DiffChangeType, merged types.Value, ok bool) {\n\treturn change, merged, false\n}\n\n// Ours resolves conflicts by preferring changes from the Value currently being committed.\nfunc Ours(aChange, bChange types.DiffChangeType, a, b types.Value, path types.Path) (change types.DiffChangeType, merged types.Value, ok bool) {\n\treturn aChange, a, true\n}\n\n// Theirs resolves conflicts by preferring changes in the current HEAD.\nfunc Theirs(aChange, bChange types.DiffChangeType, a, b types.Value, path types.Path) (change types.DiffChangeType, merged types.Value, ok bool) {\n\treturn bChange, b, true\n}\n\n// ErrMergeConflict indicates that a merge attempt failed and must be resolved\n// manually for the provided reason.\ntype ErrMergeConflict struct {\n\tmsg string\n}\n\nfunc (e *ErrMergeConflict) Error() string {\n\treturn e.msg\n}\n\nfunc newMergeConflict(format string, args ...interface{}) *ErrMergeConflict {\n\treturn &ErrMergeConflict{fmt.Sprintf(format, args...)}\n}\n\n// NewThreeWay creates a new Policy based on ThreeWay using the provided\n// ResolveFunc.\nfunc NewThreeWay(resolve ResolveFunc) Policy {\n\treturn func(a, b, parent types.Value, vrw types.ValueReadWriter, progress chan struct{}) (merged types.Value, err error) {\n\t\treturn ThreeWay(a, b, parent, vrw, resolve, progress)\n\t}\n}\n\n// ThreeWay attempts a three-way merge between two _candidate_ values that\n// have both changed with respect to a common _parent_ value. The result of\n// the algorithm is a _merged_ value or an error if merging could not be done.\n//\n// The algorithm works recursively, applying the following rules for each value:\n//\n// - If any of the three values have a different [kind](link): conflict\n// - If the two candidates are identical: the result is that value\n// - If the values are primitives or Blob: conflict\n// - If the values are maps:\n// - if the same key was inserted or updated in both candidates:\n// - first run this same algorithm on those two values to attempt to merge them\n// - if the two merged values are still different: conflict\n// - if a key was inserted in one candidate and removed in the other: conflict\n// - If the values are structs:\n// - Same as map, except using field names instead of map keys\n// - If the values are sets:\n// - Apply the changes from both candidates to the parent to get the result. No conflicts are possible.\n// - If the values are list:\n// - Apply list-merge (see below)\n//\n// Merge rules for List are a bit more complex than Map, Struct, and Set due\n// to a wider away of potential use patterns. A List might be a de-facto Map\n// with sequential numeric keys, or it might be a sequence of objects where\n// order matters but the caller is unlikely to go back and update the value at\n// a given index. List modifications are expressed in terms of 'splices' (see\n// types/edit_distance.go). Roughly, a splice indicates that some number of\n// elements were added and/or removed at some index in |parent|. In the\n// following example:\n//\n// parent: [a, b, c, d]\n// a: [b, c, d]\n// b: [a, b, c, d, e]\n// merged: [b, c, d, e]\n//\n// The difference from parent -> is described by the splice {0, 1}, indicating\n// that 1 element was removed from parent at index 0. The difference from\n// parent -> b is described as {4, 0, e}, indicating that 0 elements were\n// removed at parent's index 4, and the element 'e' was added. Our merge\n// algorithm will successfully merge a and b, because these splices do not\n// overlap; that is, neither one removes the index at which the other\n// operates. As a general rule, the merge algorithm will refuse to merge\n// splices that overlap, as in the following examples:\n//\n// parent: [a, b, c]\n// a: [a, d, b, c]\n// b: [a, c]\n// merged: conflict\n//\n// parent: [a, b, c]\n// a: [a, e, b, c]\n// b: [a, d, b, c]\n// merged: conflict\n//\n// The splices in the first example are {1, 0, d} (remove 0 elements at index\n// 1 and add 'd') and {1, 1} (remove 1 element at index 1). Since the latter\n// removes the element at which the former adds an element, these splices\n// overlap. Similarly, in the second example, both splices operate at index 1\n// but add different elements. Thus, they also overlap.\n//\n// There is one special case for overlapping splices. If they perform the\n// exact same operation, the algorithm considers them not to be in conflict.\n// E.g.\n//\n// parent: [a, b, c]\n// a: [a, d, e]\n// b: [a, d, e]\n// merged: [a, d, e]\nfunc ThreeWay(a, b, parent types.Value, vrw types.ValueReadWriter, resolve ResolveFunc, progress chan struct{}) (merged types.Value, err error) {\n\tdescribe := func(v types.Value) string {\n\t\tif v != nil {\n\t\t\treturn types.TypeOf(v).Describe()\n\t\t}\n\t\treturn \"nil Value\"\n\t}\n\n\tif a == nil && b == nil {\n\t\treturn parent, nil\n\t} else if unmergeable(a, b) {\n\t\treturn parent, newMergeConflict(\"Cannot merge %s with %s.\", describe(a), describe(b))\n\t}\n\n\tif resolve == nil {\n\t\tresolve = None\n\t}\n\tm := &merger{vrw, resolve, progress}\n\treturn m.threeWay(a, b, parent, types.Path{})\n}\n\n// a and b cannot be merged if they are of different NomsKind, or if at least one of the two is nil, or if either is a Noms primitive.\nfunc unmergeable(a, b types.Value) bool {\n\tif a != nil && b != nil {\n\t\taKind, bKind := a.Kind(), b.Kind()\n\t\treturn aKind != bKind || types.IsPrimitiveKind(aKind) || types.IsPrimitiveKind(bKind)\n\t}\n\treturn true\n}\n\ntype merger struct {\n\tvrw types.ValueReadWriter\n\tresolve ResolveFunc\n\tprogress chan<- struct{}\n}\n\nfunc updateProgress(progress chan<- struct{}) {\n\t// TODO: Eventually we'll want more information than a single bit :).\n\tif progress != nil {\n\t\tprogress <- struct{}{}\n\t}\n}\n\nfunc (m *merger) threeWay(a, b, parent types.Value, path types.Path) (merged types.Value, err error) {\n\tdefer updateProgress(m.progress)\n\tif a == nil || b == nil {\n\t\td.Panic(\"Merge candidates cannont be nil: a = %v, b = %v\", a, b)\n\t}\n\n\tswitch a.Kind() {\n\tcase types.ListKind:\n\t\tif aList, bList, pList, ok := listAssert(m.vrw, a, b, parent); ok {\n\t\t\treturn threeWayListMerge(aList, bList, pList)\n\t\t}\n\n\tcase types.MapKind:\n\t\tif aMap, bMap, pMap, ok := mapAssert(m.vrw, a, b, parent); ok {\n\t\t\treturn m.threeWayMapMerge(aMap, bMap, pMap, path)\n\t\t}\n\n\tcase types.RefKind:\n\t\tif aValue, bValue, pValue, ok := refAssert(a, b, parent, m.vrw); ok {\n\t\t\tmerged, err := m.threeWay(aValue, bValue, pValue, path)\n\t\t\tif err != nil {\n\t\t\t\treturn parent, err\n\t\t\t}\n\t\t\treturn m.vrw.WriteValue(merged), nil\n\t\t}\n\n\tcase types.SetKind:\n\t\tif aSet, bSet, pSet, ok := setAssert(m.vrw, a, b, parent); ok {\n\t\t\treturn m.threeWaySetMerge(aSet, bSet, pSet, path)\n\t\t}\n\n\tcase types.StructKind:\n\t\tif aStruct, bStruct, pStruct, ok := structAssert(a, b, parent); ok {\n\t\t\treturn m.threeWayStructMerge(aStruct, bStruct, pStruct, path)\n\t\t}\n\t}\n\n\tpDescription := \"\"\n\tif parent != nil {\n\t\tpDescription = types.TypeOf(parent).Describe()\n\t}\n\treturn parent, newMergeConflict(\"Cannot merge %s and %s on top of %s.\", types.TypeOf(a).Describe(), types.TypeOf(b).Describe(), pDescription)\n}\n\nfunc (m *merger) threeWayMapMerge(a, b, parent types.Map, path types.Path) (merged types.Value, err error) {\n\tapply := func(target candidate, change types.ValueChanged, newVal types.Value) candidate {\n\t\tdefer updateProgress(m.progress)\n\t\tswitch change.ChangeType {\n\t\tcase types.DiffChangeAdded, types.DiffChangeModified:\n\t\t\treturn mapCandidate{target.getValue().(types.Map).Edit().Set(change.Key, newVal).Map()}\n\t\tcase types.DiffChangeRemoved:\n\t\t\treturn mapCandidate{target.getValue().(types.Map).Edit().Remove(change.Key).Map()}\n\t\tdefault:\n\t\t\tpanic(\"Not Reached\")\n\t\t}\n\t}\n\treturn m.threeWayOrderedSequenceMerge(mapCandidate{a}, mapCandidate{b}, mapCandidate{parent}, apply, path)\n}\n\nfunc (m *merger) threeWaySetMerge(a, b, parent types.Set, path types.Path) (merged types.Value, err error) {\n\tapply := func(target candidate, change types.ValueChanged, newVal types.Value) candidate {\n\t\tdefer updateProgress(m.progress)\n\t\tswitch change.ChangeType {\n\t\tcase types.DiffChangeAdded, types.DiffChangeModified:\n\t\t\treturn setCandidate{target.getValue().(types.Set).Edit().Insert(newVal).Set()}\n\t\tcase types.DiffChangeRemoved:\n\t\t\treturn setCandidate{target.getValue().(types.Set).Edit().Remove(newVal).Set()}\n\t\tdefault:\n\t\t\tpanic(\"Not Reached\")\n\t\t}\n\t}\n\treturn m.threeWayOrderedSequenceMerge(setCandidate{a}, setCandidate{b}, setCandidate{parent}, apply, path)\n}\n\nfunc (m *merger) threeWayStructMerge(a, b, parent types.Struct, path types.Path) (merged types.Value, err error) {\n\tapply := func(target candidate, change types.ValueChanged, newVal types.Value) candidate {\n\t\tdefer updateProgress(m.progress)\n\t\t// Right now, this always iterates over all fields to create a new Struct, because there's no API for adding/removing a field from an existing struct type.\n\t\ttargetVal := target.getValue().(types.Struct)\n\t\tif f, ok := change.Key.(types.String); ok {\n\t\t\tfield := string(f)\n\t\t\tdata := types.StructData{}\n\t\t\ttargetVal.IterFields(func(name string, v types.Value) bool {\n\t\t\t\tif name != field {\n\t\t\t\t\tdata[name] = v\n\t\t\t\t}\n\n\t\t\t\treturn false\n\t\t\t})\n\t\t\tif change.ChangeType == types.DiffChangeAdded || change.ChangeType == types.DiffChangeModified {\n\t\t\t\tdata[field] = newVal\n\t\t\t}\n\t\t\treturn structCandidate{types.NewStruct(targetVal.Name(), data)}\n\t\t}\n\t\tpanic(fmt.Errorf(\"Bad key type in diff: %s\", types.TypeOf(change.Key).Describe()))\n\t}\n\treturn m.threeWayOrderedSequenceMerge(structCandidate{a}, structCandidate{b}, structCandidate{parent}, apply, path)\n}\n\nfunc listAssert(vrw types.ValueReadWriter, a, b, parent types.Value) (aList, bList, pList types.List, ok bool) {\n\tvar aOk, bOk, pOk bool\n\taList, aOk = a.(types.List)\n\tbList, bOk = b.(types.List)\n\tif parent != nil {\n\t\tpList, pOk = parent.(types.List)\n\t} else {\n\t\tpList, pOk = types.NewList(vrw), true\n\t}\n\treturn aList, bList, pList, aOk && bOk && pOk\n}\n\nfunc mapAssert(vrw types.ValueReadWriter, a, b, parent types.Value) (aMap, bMap, pMap types.Map, ok bool) {\n\tvar aOk, bOk, pOk bool\n\taMap, aOk = a.(types.Map)\n\tbMap, bOk = b.(types.Map)\n\tif parent != nil {\n\t\tpMap, pOk = parent.(types.Map)\n\t} else {\n\t\tpMap, pOk = types.NewMap(vrw), true\n\t}\n\treturn aMap, bMap, pMap, aOk && bOk && pOk\n}\n\nfunc refAssert(a, b, parent types.Value, vrw types.ValueReadWriter) (aValue, bValue, pValue types.Value, ok bool) {\n\tvar aOk, bOk, pOk bool\n\tvar aRef, bRef, pRef types.Ref\n\taRef, aOk = a.(types.Ref)\n\tbRef, bOk = b.(types.Ref)\n\tif !aOk || !bOk {\n\t\treturn\n\t}\n\n\taValue = aRef.TargetValue(vrw)\n\tbValue = bRef.TargetValue(vrw)\n\tif parent != nil {\n\t\tif pRef, pOk = parent.(types.Ref); pOk {\n\t\t\tpValue = pRef.TargetValue(vrw)\n\t\t}\n\t} else {\n\t\tpOk = true // parent == nil is still OK. It just leaves pValue as nil.\n\t}\n\treturn aValue, bValue, pValue, aOk && bOk && pOk\n}\n\nfunc setAssert(vrw types.ValueReadWriter, a, b, parent types.Value) (aSet, bSet, pSet types.Set, ok bool) {\n\tvar aOk, bOk, pOk bool\n\taSet, aOk = a.(types.Set)\n\tbSet, bOk = b.(types.Set)\n\tif parent != nil {\n\t\tpSet, pOk = parent.(types.Set)\n\t} else {\n\t\tpSet, pOk = types.NewSet(vrw), true\n\t}\n\treturn aSet, bSet, pSet, aOk && bOk && pOk\n}\n\nfunc structAssert(a, b, parent types.Value) (aStruct, bStruct, pStruct types.Struct, ok bool) {\n\tvar aOk, bOk, pOk bool\n\taStruct, aOk = a.(types.Struct)\n\tbStruct, bOk = b.(types.Struct)\n\tif aOk && bOk {\n\t\tif aStruct.Name() == bStruct.Name() {\n\t\t\tif parent != nil {\n\t\t\t\tpStruct, pOk = parent.(types.Struct)\n\t\t\t} else {\n\t\t\t\tpStruct, pOk = types.NewStruct(aStruct.Name(), nil), true\n\t\t\t}\n\t\t\treturn aStruct, bStruct, pStruct, pOk\n\t\t}\n\t}\n\treturn\n}\n" }, { "path": "go/merge/three_way_keyval_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage merge\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc TestThreeWayMapMerge(t *testing.T) {\n\tsuite.Run(t, &ThreeWayMapMergeSuite{})\n}\n\nfunc TestThreeWayStructMerge(t *testing.T) {\n\tsuite.Run(t, &ThreeWayStructMergeSuite{})\n}\n\ntype kvs []interface{}\n\nfunc (kv kvs) items() []interface{} {\n\treturn kv\n}\n\nfunc (kv kvs) remove(k interface{}) kvs {\n\tout := make(kvs, 0, len(kv))\n\tfor i := 0; i < len(kv); i += 2 {\n\t\tif kv[i] != k {\n\t\t\tout = append(out, kv[i], kv[i+1])\n\t\t}\n\t}\n\treturn out\n}\n\nfunc (kv kvs) set(k, v interface{}) kvs {\n\tout := make(kvs, len(kv))\n\tfor i := 0; i < len(kv); i += 2 {\n\t\tout[i], out[i+1] = kv[i], kv[i+1]\n\t\tif kv[i] == k {\n\t\t\tout[i+1] = v\n\t\t}\n\t}\n\treturn out\n}\n\nvar (\n\taa1 = kvs{\"a1\", \"a-one\", \"a2\", \"a-two\", \"a3\", \"a-three\", \"a4\", \"a-four\"}\n\taa1a = kvs{\"a1\", \"a-one\", \"a2\", \"a-two\", \"a3\", \"a-three-diff\", \"a4\", \"a-four\", \"a6\", \"a-six\"}\n\taa1b = kvs{\"a1\", \"a-one\", \"a3\", \"a-three-diff\", \"a4\", \"a-four\", \"a5\", \"a-five\"}\n\taaMerged = kvs{\"a1\", \"a-one\", \"a3\", \"a-three-diff\", \"a4\", \"a-four\", \"a5\", \"a-five\", \"a6\", \"a-six\"}\n\n\tmm1 = kvs{}\n\tmm1a = kvs{\"k1\", kvs{\"a\", 0}}\n\tmm1b = kvs{\"k1\", kvs{\"b\", 1}}\n\tmm1Merged = kvs{\"k1\", kvs{\"a\", 0, \"b\", 1}}\n\n\tmm2 = kvs{\"k2\", aa1, \"k3\", \"k-three\"}\n\tmm2a = kvs{\"k1\", kvs{\"a\", 0}, \"k2\", aa1a, \"k3\", \"k-three\", \"k4\", \"k-four\"}\n\tmm2b = kvs{\"k1\", kvs{\"b\", 1}, \"k2\", aa1b}\n\tmm2Merged = kvs{\"k1\", kvs{\"a\", 0, \"b\", 1}, \"k2\", aaMerged, \"k4\", \"k-four\"}\n)\n\ntype ThreeWayKeyValMergeSuite struct {\n\tThreeWayMergeSuite\n}\n\ntype ThreeWayMapMergeSuite struct {\n\tThreeWayKeyValMergeSuite\n}\n\nfunc (s *ThreeWayMapMergeSuite) SetupSuite() {\n\ts.create = func(seq seq) (val types.Value) {\n\t\tif seq != nil {\n\t\t\tkeyValues := valsToTypesValues(s.create, seq.items()...)\n\t\t\tval = types.NewMap(s.vs, keyValues...)\n\t\t}\n\t\treturn\n\t}\n\ts.typeStr = \"Map\"\n}\n\ntype ThreeWayStructMergeSuite struct {\n\tThreeWayKeyValMergeSuite\n}\n\nfunc (s *ThreeWayStructMergeSuite) SetupSuite() {\n\ts.create = func(seq seq) (val types.Value) {\n\t\tif seq != nil {\n\t\t\tkv := seq.items()\n\t\t\tfields := types.StructData{}\n\t\t\tfor i := 0; i < len(kv); i += 2 {\n\t\t\t\tfields[kv[i].(string)] = valToTypesValue(s.create, kv[i+1])\n\t\t\t}\n\t\t\tval = types.NewStruct(\"TestStruct\", fields)\n\t\t}\n\t\treturn\n\t}\n\ts.typeStr = \"Struct\"\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_DoNothing() {\n\ts.tryThreeWayMerge(nil, nil, aa1, aa1)\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_NoRecursion() {\n\ts.tryThreeWayMerge(aa1a, aa1b, aa1, aaMerged)\n\ts.tryThreeWayMerge(aa1b, aa1a, aa1, aaMerged)\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_RecursiveCreate() {\n\ts.tryThreeWayMerge(mm1a, mm1b, mm1, mm1Merged)\n\ts.tryThreeWayMerge(mm1b, mm1a, mm1, mm1Merged)\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_RecursiveCreateNil() {\n\ts.tryThreeWayMerge(mm1a, mm1b, nil, mm1Merged)\n\ts.tryThreeWayMerge(mm1b, mm1a, nil, mm1Merged)\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_RecursiveMerge() {\n\ts.tryThreeWayMerge(mm2a, mm2b, mm2, mm2Merged)\n\ts.tryThreeWayMerge(mm2b, mm2a, mm2, mm2Merged)\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_RefMerge() {\n\tstrRef := s.vs.WriteValue(types.NewStruct(\"Foo\", types.StructData{\"life\": types.Number(42)}))\n\n\tm := kvs{\"r2\", s.vs.WriteValue(s.create(aa1))}\n\tma := kvs{\"r1\", strRef, \"r2\", s.vs.WriteValue(s.create(aa1a))}\n\tmb := kvs{\"r1\", strRef, \"r2\", s.vs.WriteValue(s.create(aa1b))}\n\tmMerged := kvs{\"r1\", strRef, \"r2\", s.vs.WriteValue(s.create(aaMerged))}\n\n\ts.tryThreeWayMerge(ma, mb, m, mMerged)\n\ts.tryThreeWayMerge(mb, ma, m, mMerged)\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_RecursiveMultiLevelMerge() {\n\tm := kvs{\"mm1\", mm1, \"mm2\", s.vs.WriteValue(s.create(mm2))}\n\tma := kvs{\"mm1\", mm1a, \"mm2\", s.vs.WriteValue(s.create(mm2a))}\n\tmb := kvs{\"mm1\", mm1b, \"mm2\", s.vs.WriteValue(s.create(mm2b))}\n\tmMerged := kvs{\"mm1\", mm1Merged, \"mm2\", s.vs.WriteValue(s.create(mm2Merged))}\n\n\ts.tryThreeWayMerge(ma, mb, m, mMerged)\n\ts.tryThreeWayMerge(mb, ma, m, mMerged)\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_CustomMerge() {\n\tp := kvs{\"k1\", \"k-one\", \"k2\", \"k-two\", \"mm1\", mm1, \"s1\", \"s-one\"}\n\ta := kvs{\"k1\", \"k-won\", \"k2\", \"k-too\", \"mm1\", mm1, \"s1\", \"s-one\", \"n1\", kvs{\"a\", \"1\"}}\n\tb := kvs{\"k2\", \"k-two\", \"mm1\", \"mm-one\", \"s1\", \"s-one\", \"n1\", kvs{\"a\", \"2\"}}\n\texp := kvs{\"k2\", \"k-too\", \"mm1\", \"mm-one\", \"s1\", \"s-one\", \"n1\", kvs{\"a\", \"1\"}}\n\n\texpectedConflictPaths := [][]string{{\"k1\"}, {\"n1\", \"a\"}}\n\tconflictPaths := []types.Path{}\n\tresolve := func(aChange, bChange types.DiffChangeType, aVal, bVal types.Value, p types.Path) (change types.DiffChangeType, merged types.Value, ok bool) {\n\t\tconflictPaths = append(conflictPaths, p)\n\t\tif _, ok := aVal.(types.Map); ok || bChange == types.DiffChangeRemoved {\n\t\t\treturn bChange, bVal, true\n\t\t}\n\t\treturn aChange, aVal, true\n\t}\n\n\tmerged, err := ThreeWay(s.create(a), s.create(b), s.create(p), s.vs, resolve, nil)\n\tif s.NoError(err) {\n\t\texpected := s.create(exp)\n\t\ts.True(expected.Equals(merged), \"%s != %s\", types.EncodedValue(expected), types.EncodedValue(merged))\n\t}\n\tif s.Len(conflictPaths, len(expectedConflictPaths), \"Wrong number of conflicts!\") {\n\t\tfor i := 0; i < len(conflictPaths); i++ {\n\t\t\tfor j, c := range conflictPaths[i] {\n\t\t\t\ts.Contains(c.String(), expectedConflictPaths[i][j])\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_MergeOurs() {\n\tp := kvs{\"k1\", \"k-one\"}\n\ta := kvs{\"k1\", \"k-won\"}\n\tb := kvs{\"k1\", \"k-too\", \"k2\", \"k-two\"}\n\texp := kvs{\"k1\", \"k-won\", \"k2\", \"k-two\"}\n\n\tmerged, err := ThreeWay(s.create(a), s.create(b), s.create(p), s.vs, Ours, nil)\n\tif s.NoError(err) {\n\t\texpected := s.create(exp)\n\t\ts.True(expected.Equals(merged), \"%s != %s\", types.EncodedValue(expected), types.EncodedValue(merged))\n\t}\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_MergeTheirs() {\n\tp := kvs{\"k1\", \"k-one\"}\n\ta := kvs{\"k1\", \"k-won\"}\n\tb := kvs{\"k1\", \"k-too\", \"k2\", \"k-two\"}\n\texp := kvs{\"k1\", \"k-too\", \"k2\", \"k-two\"}\n\n\tmerged, err := ThreeWay(s.create(a), s.create(b), s.create(p), s.vs, Theirs, nil)\n\tif s.NoError(err) {\n\t\texpected := s.create(exp)\n\t\ts.True(expected.Equals(merged), \"%s != %s\", types.EncodedValue(expected), types.EncodedValue(merged))\n\t}\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_NilConflict() {\n\ts.tryThreeWayConflict(nil, s.create(mm2b), s.create(mm2), \"Cannot merge nil Value with\")\n\ts.tryThreeWayConflict(s.create(mm2a), nil, s.create(mm2), \"with nil Value.\")\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_ImmediateConflict() {\n\ts.tryThreeWayConflict(types.NewSet(s.vs), s.create(mm2b), s.create(mm2), \"Cannot merge Set<> with \"+s.typeStr)\n\ts.tryThreeWayConflict(s.create(mm2b), types.NewSet(s.vs), s.create(mm2), \"Cannot merge \"+s.typeStr)\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_RefConflict() {\n\tstrRef := s.vs.WriteValue(types.NewStruct(\"Foo\", types.StructData{\"life\": types.Number(42)}))\n\tnumRef := s.vs.WriteValue(types.Number(7))\n\n\tm := kvs{\"r2\", strRef}\n\tma := kvs{\"r1\", strRef, \"r2\", strRef}\n\tmb := kvs{\"r1\", numRef, \"r2\", strRef}\n\n\ts.tryThreeWayConflict(s.create(ma), s.create(mb), s.create(m), \"Cannot merge Struct Foo\")\n\ts.tryThreeWayConflict(s.create(mb), s.create(ma), s.create(m), \"Cannot merge Number and Struct Foo\")\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_NestedConflict() {\n\ta := mm2a.set(\"k2\", types.NewSet(s.vs))\n\ts.tryThreeWayConflict(s.create(a), s.create(mm2b), s.create(mm2), types.EncodedValue(types.NewSet(s.vs)))\n\ts.tryThreeWayConflict(s.create(a), s.create(mm2b), s.create(mm2), types.EncodedValue(s.create(aa1b)))\n}\n\nfunc (s *ThreeWayKeyValMergeSuite) TestThreeWayMerge_NestedConflictingOperation() {\n\ta := mm2a.remove(\"k2\")\n\ts.tryThreeWayConflict(s.create(a), s.create(mm2b), s.create(mm2), `removed \"k2\"`)\n\ts.tryThreeWayConflict(s.create(a), s.create(mm2b), s.create(mm2), `modded \"k2\"`)\n}\n" }, { "path": "go/merge/three_way_list.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage merge\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\nfunc threeWayListMerge(a, b, parent types.List) (merged types.List, err error) {\n\taSpliceChan, bSpliceChan := make(chan types.Splice), make(chan types.Splice)\n\taStopChan, bStopChan := make(chan struct{}, 1), make(chan struct{}, 1)\n\n\tgo func() {\n\t\ta.Diff(parent, aSpliceChan, aStopChan)\n\t\tclose(aSpliceChan)\n\t}()\n\tgo func() {\n\t\tb.Diff(parent, bSpliceChan, bStopChan)\n\t\tclose(bSpliceChan)\n\t}()\n\n\tstopAndDrain := func(stop chan<- struct{}, drain <-chan types.Splice) {\n\t\tclose(stop)\n\t\tfor range drain {\n\t\t}\n\t}\n\n\tdefer stopAndDrain(aStopChan, aSpliceChan)\n\tdefer stopAndDrain(bStopChan, bSpliceChan)\n\n\t// The algorithm below relies on determining whether one splice \"comes before\" another, and whether the splices coming from the two diffs remove/add precisely the same elements. Unfortunately, the Golang zero-value for types.Splice (which is what gets read out of a/bSpliceChan when they're closed) is actaually a valid splice, albeit a meaningless one that indicates a no-op. It \"comes before\" any other splice, so having it in play really gums up the logic below. Rather than specifically checking for it all over the place, swap the zero-splice out for one full of SPLICE_UNASSIGNED, which is really the proper invalid splice value. That splice doesn't come before ANY valid splice, so the logic below can flow more clearly.\n\tzeroSplice := types.Splice{}\n\tzeroToInvalid := func(sp types.Splice) types.Splice {\n\t\tif sp == zeroSplice {\n\t\t\treturn types.Splice{\n\t\t\t\tSpAt: types.SPLICE_UNASSIGNED,\n\t\t\t\tSpRemoved: types.SPLICE_UNASSIGNED,\n\t\t\t\tSpAdded: types.SPLICE_UNASSIGNED,\n\t\t\t\tSpFrom: types.SPLICE_UNASSIGNED,\n\t\t\t}\n\t\t}\n\t\treturn sp\n\t}\n\tinvalidSplice := zeroToInvalid(types.Splice{})\n\n\tmerged = parent\n\toffset := uint64(0)\n\taSplice, bSplice := invalidSplice, invalidSplice\n\tfor {\n\t\t// Get the next splice from both a and b. If either diff(a, parent) or diff(b, parent) is complete, aSplice or bSplice will get an invalid types.Splice. Generally, though, this allows us to proceed through both diffs in (index) order, considering the \"current\" splice from both diffs at the same time.\n\t\tif aSplice == invalidSplice {\n\t\t\taSplice = zeroToInvalid(<-aSpliceChan)\n\t\t}\n\t\tif bSplice == invalidSplice {\n\t\t\tbSplice = zeroToInvalid(<-bSpliceChan)\n\t\t}\n\t\t// Both channels are producing zero values, so we're done.\n\t\tif aSplice == invalidSplice && bSplice == invalidSplice {\n\t\t\tbreak\n\t\t}\n\t\tif overlap(aSplice, bSplice) {\n\t\t\tif canMerge(a, b, aSplice, bSplice) {\n\t\t\t\tsplice := merge(aSplice, bSplice)\n\t\t\t\tmerged = apply(a, merged, offset, splice)\n\t\t\t\toffset += splice.SpAdded - splice.SpRemoved\n\t\t\t\taSplice, bSplice = invalidSplice, invalidSplice\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\treturn parent, newMergeConflict(\"Overlapping splices: %s vs %s\", describeSplice(aSplice), describeSplice(bSplice))\n\t\t}\n\t\tif aSplice.SpAt < bSplice.SpAt {\n\t\t\tmerged = apply(a, merged, offset, aSplice)\n\t\t\toffset += aSplice.SpAdded - aSplice.SpRemoved\n\t\t\taSplice = invalidSplice\n\t\t\tcontinue\n\t\t}\n\t\tmerged = apply(b, merged, offset, bSplice)\n\t\toffset += bSplice.SpAdded - bSplice.SpRemoved\n\t\tbSplice = invalidSplice\n\t}\n\n\treturn merged, nil\n}\n\nfunc overlap(s1, s2 types.Splice) bool {\n\tearlier, later := s1, s2\n\tif s2.SpAt < s1.SpAt {\n\t\tearlier, later = s2, s1\n\t}\n\treturn s1.SpAt == s2.SpAt || earlier.SpAt+earlier.SpRemoved > later.SpAt\n}\n\n// canMerge returns whether aSplice and bSplice can be merged into a single splice that can be applied to parent. Currently, we're only willing to do this if the two splices do _precisely_ the same thing -- that is, remove the same number of elements from the same starting index and insert the exact same list of new elements.\nfunc canMerge(a, b types.List, aSplice, bSplice types.Splice) bool {\n\tif aSplice != bSplice {\n\t\treturn false\n\t}\n\taIter, bIter := a.IteratorAt(aSplice.SpFrom), b.IteratorAt(bSplice.SpFrom)\n\tfor count := uint64(0); count < aSplice.SpAdded; count++ {\n\t\taVal, bVal := aIter.Next(), bIter.Next()\n\t\tif aVal == nil || bVal == nil || !aVal.Equals(bVal) {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\n// Since merge() is only called when canMerge() is true, we know s1 and s2 are exactly equal.\nfunc merge(s1, s2 types.Splice) types.Splice {\n\treturn s1\n}\n\nfunc apply(source, target types.List, offset uint64, s types.Splice) types.List {\n\ttoAdd := make([]types.Valuable, s.SpAdded)\n\titer := source.IteratorAt(s.SpFrom)\n\tfor i := 0; uint64(i) < s.SpAdded; i++ {\n\t\tv := iter.Next()\n\t\tif v == nil {\n\t\t\td.Panic(\"List diff returned a splice that inserts a nonexistent element.\")\n\t\t}\n\t\ttoAdd[i] = v\n\t}\n\treturn target.Edit().Splice(s.SpAt+offset, s.SpRemoved, toAdd...).List()\n}\n\nfunc describeSplice(s types.Splice) string {\n\treturn fmt.Sprintf(\"%d elements removed at %d; adding %d elements\", s.SpRemoved, s.SpAt, s.SpAdded)\n}\n" }, { "path": "go/merge/three_way_list_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage merge\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc TestThreeWayListMerge(t *testing.T) {\n\tsuite.Run(t, &ThreeWayListMergeSuite{})\n}\n\ntype ThreeWayListMergeSuite struct {\n\tThreeWayMergeSuite\n}\n\nfunc (s *ThreeWayListMergeSuite) SetupSuite() {\n\ts.create = func(i seq) (val types.Value) {\n\t\tif i != nil {\n\t\t\titems := valsToTypesValues(s.create, i.items()...)\n\t\t\tval = types.NewList(s.vs, items...)\n\t\t}\n\t\treturn\n\t}\n\ts.typeStr = \"List\"\n}\n\nvar p = items{\"a\", \"b\", \"c\", \"d\", \"e\"}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_DoNothing() {\n\ts.tryThreeWayMerge(nil, nil, p, p)\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_NoLengthChange() {\n\ta := items{\"a\", 1, \"c\", \"d\", \"e\"}\n\tb := items{\"a\", \"b\", \"c\", 2, \"e\"}\n\tm := items{\"a\", 1, \"c\", 2, \"e\"}\n\ts.tryThreeWayMerge(a, b, p, m)\n\ts.tryThreeWayMerge(b, a, p, m)\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_HandleEmpty() {\n\ts.tryThreeWayMerge(p, items{}, items{}, p)\n\ts.tryThreeWayMerge(items{}, p, items{}, p)\n\ts.tryThreeWayMerge(p, p, items{}, p)\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_HandleNil() {\n\ts.tryThreeWayMerge(p, items{}, nil, p)\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_MakeLonger() {\n\ta := items{\"a\", 1, 2, \"c\", \"d\", \"e\"}\n\tb := items{\"a\", \"b\", \"c\", 3, \"e\"}\n\tm := items{\"a\", 1, 2, \"c\", 3, \"e\"}\n\ts.tryThreeWayMerge(a, b, p, m)\n\ts.tryThreeWayMerge(b, a, p, m)\n\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_MakeShorter() {\n\ta := items{\"a\", \"c\", \"d\", \"e\"}\n\tb := items{\"a\", \"b\", \"c\", 3, \"e\"}\n\tm := items{\"a\", \"c\", 3, \"e\"}\n\ts.tryThreeWayMerge(a, b, p, m)\n\ts.tryThreeWayMerge(b, a, p, m)\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_BothSidesRemove() {\n\ta := items{\"a\", \"c\", \"d\", \"e\"}\n\tb := items{\"a\", \"b\", \"c\", \"e\"}\n\tm := items{\"a\", \"c\", \"e\"}\n\ts.tryThreeWayMerge(a, b, p, m)\n\ts.tryThreeWayMerge(b, a, p, m)\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_OverlapSameRemoveNoInsert() {\n\ta := items{\"a\", \"d\", \"e\"}\n\tb := items{\"a\", \"d\", \"e\"}\n\tm := items{\"a\", \"d\", \"e\"}\n\ts.tryThreeWayMerge(a, b, p, m)\n\ts.tryThreeWayMerge(b, a, p, m)\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_OverlapSameRemoveSameInsert() {\n\ta := items{\"a\", 1, 2, 3, \"d\", \"e\"}\n\tb := items{\"a\", 1, 2, 3, \"d\", \"e\"}\n\tm := items{\"a\", 1, 2, 3, \"d\", \"e\"}\n\ts.tryThreeWayMerge(a, b, p, m)\n\ts.tryThreeWayMerge(b, a, p, m)\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_RemoveUpToOtherSideInsertionPoint() {\n\ta := items{\"a\", 1, 2, \"c\", \"d\", \"e\"}\n\tb := items{\"a\", \"b\", 3, \"c\", \"d\", \"e\"}\n\tm := items{\"a\", 1, 2, 3, \"c\", \"d\", \"e\"}\n\ts.tryThreeWayMerge(a, b, p, m)\n\ts.tryThreeWayMerge(b, a, p, m)\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_ConflictingAppends() {\n\ta := append(p, 1)\n\tb := append(p, 2)\n\ts.tryThreeWayConflict(s.create(a), s.create(b), s.create(p), \"Overlapping splices: 0 elements removed at 5; adding 1 elements\")\n\ts.tryThreeWayConflict(s.create(b), s.create(a), s.create(p), \"Overlapping splices: 0 elements removed at 5; adding 1 elements\")\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_OverlappingRemoves() {\n\ta := p[:4]\n\tb := p[:3]\n\ts.tryThreeWayConflict(s.create(a), s.create(b), s.create(p), \"Overlapping splices: 1 elements removed at 4\")\n\ts.tryThreeWayConflict(s.create(b), s.create(a), s.create(p), \"Overlapping splices: 2 elements removed at 3\")\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_SameRemoveAddPrefix() {\n\ta := items{\"a\", \"b\", \"c\", 1}\n\tb := items{\"a\", \"b\", \"c\", 1, 2}\n\ts.tryThreeWayConflict(s.create(a), s.create(b), s.create(p), \"Overlapping splices: 2 elements removed at 3; adding 1 elements\")\n\ts.tryThreeWayConflict(s.create(b), s.create(a), s.create(p), \"Overlapping splices: 2 elements removed at 3; adding 2 elements\")\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_RemoveSupersetAddPrefix() {\n\ta := items{\"a\", \"b\", \"c\", 1, 2}\n\tb := items{\"a\", \"b\", \"c\", \"d\", 1}\n\ts.tryThreeWayConflict(s.create(a), s.create(b), s.create(p), \"Overlapping splices: 2 elements removed at 3; adding 2 elements\")\n\ts.tryThreeWayConflict(s.create(b), s.create(a), s.create(p), \"Overlapping splices: 1 elements removed at 4; adding 1 elements\")\n}\n\nfunc (s *ThreeWayListMergeSuite) TestThreeWayMerge_RemoveOtherSideInsertionPoint() {\n\ta := items{\"a\", \"c\", \"d\", \"e\"}\n\tb := items{\"a\", 1, \"b\", \"c\", \"d\", \"e\"}\n\ts.tryThreeWayConflict(s.create(a), s.create(b), s.create(p), \"Overlapping splices: 1 elements removed at 1; adding 0 elements\")\n\ts.tryThreeWayConflict(s.create(b), s.create(a), s.create(p), \"Overlapping splices: 0 elements removed at 1; adding 1 elements\")\n}\n" }, { "path": "go/merge/three_way_ordered_sequence.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage merge\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\ntype applyFunc func(candidate, types.ValueChanged, types.Value) candidate\n\nfunc (m *merger) threeWayOrderedSequenceMerge(a, b, parent candidate, apply applyFunc, path types.Path) (types.Value, error) {\n\taChangeChan, bChangeChan := make(chan types.ValueChanged), make(chan types.ValueChanged)\n\taStopChan, bStopChan := make(chan struct{}, 1), make(chan struct{}, 1)\n\n\tgo func() {\n\t\ta.diff(parent, aChangeChan, aStopChan)\n\t\tclose(aChangeChan)\n\t}()\n\tgo func() {\n\t\tb.diff(parent, bChangeChan, bStopChan)\n\t\tclose(bChangeChan)\n\t}()\n\n\tdefer stopAndDrain(aStopChan, aChangeChan)\n\tdefer stopAndDrain(bStopChan, bChangeChan)\n\n\tmerged := parent\n\taChange, bChange := types.ValueChanged{}, types.ValueChanged{}\n\tfor {\n\t\t// Get the next change from both a and b. If either diff(a, parent) or diff(b, parent) is complete, aChange or bChange will get an empty types.ValueChanged containing a nil Value. Generally, though, this allows us to proceed through both diffs in (key) order, considering the \"current\" change from both diffs at the same time.\n\t\tif aChange.Key == nil {\n\t\t\taChange = <-aChangeChan\n\t\t}\n\t\tif bChange.Key == nil {\n\t\t\tbChange = <-bChangeChan\n\t\t}\n\n\t\t// Both channels are producing zero values, so we're done.\n\t\tif aChange.Key == nil && bChange.Key == nil {\n\t\t\tbreak\n\t\t}\n\n\t\t// Since diff generates changes in key-order, and we never skip over a change without processing it, we can simply compare the keys at which aChange and bChange occurred to determine if either is safe to apply to the merge result without further processing. This is because if, e.g. aChange.V.Less(bChange.V), we know that the diff of b will never generate a change at that key. If it was going to, it would have done so on an earlier iteration of this loop and been processed at that time.\n\t\t// It's also obviously OK to apply a change if only one diff is generating any changes, e.g. aChange.V is non-nil and bChange.V is nil.\n\t\tif aChange.Key != nil && (bChange.Key == nil || aChange.Key.Less(bChange.Key)) {\n\t\t\tmerged = apply(merged, aChange, a.get(aChange.Key))\n\t\t\taChange = types.ValueChanged{}\n\t\t\tcontinue\n\t\t} else if bChange.Key != nil && (aChange.Key == nil || bChange.Key.Less(aChange.Key)) {\n\t\t\tmerged = apply(merged, bChange, b.get(bChange.Key))\n\t\t\tbChange = types.ValueChanged{}\n\t\t\tcontinue\n\t\t}\n\t\tif !aChange.Key.Equals(bChange.Key) {\n\t\t\td.Panic(\"Diffs have skewed!\") // Sanity check.\n\t\t}\n\n\t\tchange, mergedVal, err := m.mergeChanges(aChange, bChange, a, b, parent, apply, path)\n\t\tif err != nil {\n\t\t\treturn parent.getValue(), err\n\t\t}\n\t\tmerged = apply(merged, change, mergedVal)\n\t\taChange, bChange = types.ValueChanged{}, types.ValueChanged{}\n\t}\n\treturn merged.getValue(), nil\n}\n\nfunc (m *merger) mergeChanges(aChange, bChange types.ValueChanged, a, b, p candidate, apply applyFunc, path types.Path) (change types.ValueChanged, mergedVal types.Value, err error) {\n\tpath = a.pathConcat(aChange, path)\n\taValue, bValue := a.get(aChange.Key), b.get(bChange.Key)\n\t// If the two diffs generate different kinds of changes at the same key, conflict.\n\tif aChange.ChangeType != bChange.ChangeType {\n\t\tif change, mergedVal, ok := m.resolve(aChange.ChangeType, bChange.ChangeType, aValue, bValue, path); ok {\n\t\t\t// TODO: Correctly encode Old/NewValue with this change report. https://github.com/attic-labs/noms/issues/3467\n\t\t\treturn types.ValueChanged{ChangeType: change, Key: aChange.Key, OldValue: nil, NewValue: nil}, mergedVal, nil\n\t\t}\n\t\treturn change, nil, newMergeConflict(\"Conflict:\\n%s\\nvs\\n%s\\n\", describeChange(aChange), describeChange(bChange))\n\t}\n\n\tif aChange.ChangeType == types.DiffChangeRemoved || aValue.Equals(bValue) {\n\t\t// If both diffs generated a remove, or if the new value is the same in both, merge is fine.\n\t\treturn aChange, aValue, nil\n\t}\n\n\t// There's one case that might still be OK even if aValue and bValue differ: different, but mergeable, compound values of the same type being added/modified at the same key, e.g. a Map being added to both a and b. If either is a primitive, or Values of different Kinds were added, though, we're in conflict.\n\tif !unmergeable(aValue, bValue) {\n\t\t// TODO: Add concurrency.\n\t\tvar err error\n\t\tif mergedVal, err = m.threeWay(aValue, bValue, p.get(aChange.Key), path); err == nil {\n\t\t\treturn aChange, mergedVal, nil\n\t\t}\n\t\treturn change, nil, err\n\t}\n\n\tif change, mergedVal, ok := m.resolve(aChange.ChangeType, bChange.ChangeType, aValue, bValue, path); ok {\n\t\t// TODO: Correctly encode Old/NewValue with this change report. https://github.com/attic-labs/noms/issues/3467\n\t\treturn types.ValueChanged{ChangeType: change, Key: aChange.Key, OldValue: nil, NewValue: nil}, mergedVal, nil\n\t}\n\treturn change, nil, newMergeConflict(\"Conflict:\\n%s = %s\\nvs\\n%s = %s\", describeChange(aChange), types.EncodedValue(aValue), describeChange(bChange), types.EncodedValue(bValue))\n}\n\nfunc stopAndDrain(stop chan<- struct{}, drain <-chan types.ValueChanged) {\n\tclose(stop)\n\tfor range drain {\n\t}\n}\n\nfunc describeChange(change types.ValueChanged) string {\n\top := \"\"\n\tswitch change.ChangeType {\n\tcase types.DiffChangeAdded:\n\t\top = \"added\"\n\tcase types.DiffChangeModified:\n\t\top = \"modded\"\n\tcase types.DiffChangeRemoved:\n\t\top = \"removed\"\n\t}\n\treturn fmt.Sprintf(\"%s %s\", op, types.EncodedValue(change.Key))\n}\n" }, { "path": "go/merge/three_way_set_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage merge\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc TestThreeWaySetMerge(t *testing.T) {\n\tsuite.Run(t, &ThreeWaySetMergeSuite{})\n}\n\ntype items []interface{}\n\nfunc (kv items) items() []interface{} {\n\treturn kv\n}\n\ntype ThreeWaySetMergeSuite struct {\n\tThreeWayMergeSuite\n}\n\nfunc (s *ThreeWaySetMergeSuite) SetupSuite() {\n\ts.create = func(i seq) (val types.Value) {\n\t\tif i != nil {\n\t\t\tkeyValues := valsToTypesValues(s.create, i.items()...)\n\t\t\tval = types.NewSet(s.vs, keyValues...)\n\t\t}\n\t\treturn\n\t}\n\ts.typeStr = \"Set\"\n}\n\nvar (\n\tflat = items{\"a1\", \"a2\", \"a3\", \"a4\"}\n\tflatA = items{\"a1\", \"a2\", \"a5\", \"a6\"}\n\tflatB = items{\"a1\", \"a4\", \"a7\", \"a5\"}\n\tflatM = items{\"a1\", \"a5\", \"a6\", \"a7\"}\n\n\tss1 = items{}\n\tss1a = items{\"k1\", flatA, items{\"a\", 0}}\n\tss1b = items{\"k1\", items{\"a\", 0}, flatB}\n\tss1Merged = items{\"k1\", items{\"a\", 0}, flatA, flatB}\n)\n\nfunc (s *ThreeWaySetMergeSuite) TestThreeWayMerge_DoNothing() {\n\ts.tryThreeWayMerge(nil, nil, flat, flat)\n}\n\nfunc (s *ThreeWaySetMergeSuite) TestThreeWayMerge_Primitives() {\n\ts.tryThreeWayMerge(flatA, flatB, flat, flatM)\n\ts.tryThreeWayMerge(flatB, flatA, flat, flatM)\n}\n\nfunc (s *ThreeWaySetMergeSuite) TestThreeWayMerge_HandleEmpty() {\n\ts.tryThreeWayMerge(ss1a, ss1b, ss1, ss1Merged)\n\ts.tryThreeWayMerge(ss1b, ss1a, ss1, ss1Merged)\n}\n\nfunc (s *ThreeWaySetMergeSuite) TestThreeWayMerge_HandleNil() {\n\ts.tryThreeWayMerge(ss1a, ss1b, nil, ss1Merged)\n\ts.tryThreeWayMerge(ss1b, ss1a, nil, ss1Merged)\n}\n\nfunc (s *ThreeWaySetMergeSuite) TestThreeWayMerge_Refs() {\n\tstrRef := s.vs.WriteValue(types.NewStruct(\"Foo\", types.StructData{\"life\": types.Number(42)}))\n\n\tm := items{s.vs.WriteValue(s.create(flatA)), s.vs.WriteValue(s.create(flatB))}\n\tma := items{\"r1\", s.vs.WriteValue(s.create(flatA))}\n\tmb := items{\"r1\", strRef, s.vs.WriteValue(s.create(flatA))}\n\tmMerged := items{\"r1\", strRef, s.vs.WriteValue(s.create(flatA))}\n\n\ts.tryThreeWayMerge(ma, mb, m, mMerged)\n\ts.tryThreeWayMerge(mb, ma, m, mMerged)\n}\n\nfunc (s *ThreeWaySetMergeSuite) TestThreeWayMerge_ImmediateConflict() {\n\ts.tryThreeWayConflict(types.NewMap(s.vs), s.create(ss1b), s.create(ss1), \"Cannot merge Map<> with \"+s.typeStr)\n\ts.tryThreeWayConflict(s.create(ss1b), types.NewMap(s.vs), s.create(ss1), \"Cannot merge \"+s.typeStr)\n}\n" }, { "path": "go/merge/three_way_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage merge\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\ntype seq interface {\n\titems() []interface{}\n}\n\ntype ThreeWayMergeSuite struct {\n\tsuite.Suite\n\tvs *types.ValueStore\n\tcreate func(seq) types.Value\n\ttypeStr string\n}\n\nfunc (s *ThreeWayMergeSuite) SetupTest() {\n\tstorage := &chunks.MemoryStorage{}\n\ts.vs = types.NewValueStore(storage.NewView())\n}\n\nfunc (s *ThreeWayMergeSuite) TearDownTest() {\n\ts.vs.Close()\n}\n\nfunc (s *ThreeWayMergeSuite) tryThreeWayMerge(a, b, p, exp seq) {\n\tmerged, err := ThreeWay(s.create(a), s.create(b), s.create(p), s.vs, nil, nil)\n\tif s.NoError(err) {\n\t\texpected := s.create(exp)\n\t\ts.True(expected.Equals(merged), \"%s != %s\", types.EncodedValue(expected), types.EncodedValue(merged))\n\t}\n}\n\nfunc (s *ThreeWayMergeSuite) tryThreeWayConflict(a, b, p types.Value, contained string) {\n\tm, err := ThreeWay(a, b, p, s.vs, nil, nil)\n\tif s.Error(err) {\n\t\ts.Contains(err.Error(), contained)\n\t\treturn\n\t}\n\ts.Fail(\"Expected error!\", \"Got successful merge: %s\", types.EncodedValue(m))\n}\n\nfunc valsToTypesValues(f func(seq) types.Value, items ...interface{}) []types.Value {\n\tkeyValues := []types.Value{}\n\tfor _, e := range items {\n\t\tv := valToTypesValue(f, e)\n\t\tkeyValues = append(keyValues, v)\n\t}\n\treturn keyValues\n}\n\nfunc valToTypesValue(f func(seq) types.Value, v interface{}) types.Value {\n\tvar v1 types.Value\n\tswitch t := v.(type) {\n\tcase string:\n\t\tv1 = types.String(t)\n\tcase int:\n\t\tv1 = types.Number(t)\n\tcase seq:\n\t\tv1 = f(t)\n\tcase types.Value:\n\t\tv1 = t\n\t}\n\treturn v1\n}\n\nfunc TestThreeWayMerge_PrimitiveConflict(t *testing.T) {\n\tthreeWayConflict := func(a, b, p types.Value, contained string) {\n\t\tmrgr := &merger{}\n\t\tm, err := mrgr.threeWay(a, b, p, nil)\n\t\tif assert.Error(t, err) {\n\t\t\tassert.Contains(t, err.Error(), contained)\n\t\t\treturn\n\t\t}\n\t\tassert.Fail(t, \"Expected error!\", \"Got successful merge: %s\", types.EncodedValue(m))\n\t}\n\n\ta, b, p := types.Number(7), types.String(\"nope\"), types.String(\"parent\")\n\n\tthreeWayConflict(a, b, p, \"Number and String on top of\")\n\tthreeWayConflict(b, a, p, \"String and Number on top of\")\n}\n" }, { "path": "go/metrics/histogram.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage metrics\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\thumanize \"github.com/dustin/go-humanize\"\n)\n\n// Histogram is a shameless and low-rent knock of the chromium project's\n// histogram:\n// https://chromium.googlesource.com/chromium/src/base/+/master/metrics/histogram.h\n//\n// It logically stores a running histogram of uint64 values and shares some\n// important features of its inspiration:\n// * It acccepts a correctness deficit in return for not needing to lock.\n// IOW, concurrent calls to Sample may clobber each other.\n// * It trades compactness and ease of arithmatic across histograms for\n// precision. Samples lose precision up to the range of the values which\n// are stored in a bucket\n//\n// Only implemented: Log2-based histogram\ntype Histogram struct {\n\tsum uint64\n\tbuckets [bucketCount]uint64\n\tToString ToStringFunc\n}\n\ntype ToStringFunc func(v uint64) string\n\nfunc identToString(v uint64) string {\n\treturn fmt.Sprintf(\"%d\", v)\n}\n\nconst bucketCount = 64\n\n// Sample adds a uint64 data point to the histogram\nfunc (h *Histogram) Sample(v uint64) {\n\td.PanicIfTrue(v == 0)\n\n\th.sum += v\n\n\tpot := 0\n\tfor v > 0 {\n\t\tv = v >> 1\n\t\tpot++\n\t}\n\n\th.buckets[pot-1]++\n}\n\n// SampleTimeSince is a convenience wrapper around Sample which takes the\n// duration since |t|, if 0, rounds to 1 and passes to Sample() as an uint64\n// number of nanoseconds.\nfunc (h *Histogram) SampleTimeSince(t time.Time) {\n\td := time.Since(t)\n\tif d == 0 {\n\t\td = 1\n\t}\n\th.Sample(uint64(d))\n}\n\n// SampleLen is a convenience wrapper around Sample which internally type\n// asserts the int to a uint64\nfunc (h *Histogram) SampleLen(l int) {\n\th.Sample(uint64(l))\n}\n\nfunc (h Histogram) bucketVal(bucket int) uint64 {\n\treturn 1 << (uint64(bucket))\n}\n\n// Sum return the sum of sampled values, note that Sum can be overflowed without\n// overflowing the histogram buckets.\nfunc (h Histogram) Sum() uint64 {\n\treturn h.sum\n}\n\n// Add returns a new Histogram which is the result of adding this and other\n// bucket-wise.\nfunc (h *Histogram) Add(other Histogram) {\n\th.sum += other.sum\n\n\tfor i := 0; i < bucketCount; i++ {\n\t\th.buckets[i] += other.buckets[i]\n\t}\n}\n\n// Delta returns a new Histogram which is the result of subtracting other from\n// this bucket-wise. The intent is to capture changes in the state of histogram\n// which is collecting samples over some time period. It will panic if any\n// bucket from other is larger than the corresponding bucket in this.\nfunc (h Histogram) Delta(other Histogram) Histogram {\n\tnh := Histogram{}\n\tnh.sum = h.sum - other.sum\n\n\tfor i := 0; i < bucketCount; i++ {\n\t\tc := h.buckets[i]\n\t\tl := other.buckets[i]\n\t\td.PanicIfTrue(l > c)\n\t\tnh.buckets[i] = c - l\n\t}\n\treturn nh\n}\n\n// Mean returns 0 if there are no samples, and h.Sum()/h.Samples otherwise.\nfunc (h Histogram) Mean() uint64 {\n\tsamples := h.Samples()\n\tif samples == 0 {\n\t\treturn 0\n\t}\n\n\treturn h.Sum() / samples\n}\n\n// Samples returns the number of samples contained in the histogram\nfunc (h Histogram) Samples() uint64 {\n\ts := uint64(0)\n\tfor i := 0; i < bucketCount; i++ {\n\t\ts += h.buckets[i]\n\t}\n\treturn s\n}\n\nfunc (h Histogram) String() string {\n\tf := h.ToString\n\tif f == nil {\n\t\tf = identToString\n\t}\n\treturn fmt.Sprintf(\"Mean: %s, Sum: %s, Samples: %d\", f(h.Mean()), f(h.Sum()), h.Samples())\n}\n\nfunc NewTimeHistogram() Histogram {\n\treturn Histogram{ToString: timeToString}\n}\n\nfunc timeToString(v uint64) string {\n\treturn time.Duration(v).String()\n}\n\n// NewByteHistogram stringifies values using humanize over byte values\nfunc NewByteHistogram() Histogram {\n\treturn Histogram{ToString: humanize.Bytes}\n}\n\nconst colWidth = 100\n\n// Report returns an ASCII graph of the non-zero range of normalized buckets.\n// IOW, it returns a basic graph of the histogram\nfunc (h Histogram) Report() string {\n\tts := h.ToString\n\tif ts == nil {\n\t\tts = identToString\n\t}\n\n\tmaxSamples := uint64(0)\n\tfoundFirstNonEmpty := false\n\tfirstNonEmpty := 0\n\tlastNonEmpty := 0\n\tfor i := 0; i < bucketCount; i++ {\n\t\tsamples := h.buckets[i]\n\n\t\tif samples > 0 {\n\t\t\tlastNonEmpty = i\n\t\t\tif !foundFirstNonEmpty {\n\t\t\t\tfoundFirstNonEmpty = true\n\t\t\t\tfirstNonEmpty = i\n\t\t\t}\n\t\t}\n\n\t\tif samples > maxSamples {\n\t\t\tmaxSamples = samples\n\t\t}\n\t}\n\n\tif maxSamples == 0 {\n\t\treturn \"\"\n\t}\n\n\tval := uint64(1)\n\n\tp := func(bucket int) string {\n\t\tsamples := h.buckets[bucket]\n\t\tval := h.bucketVal(bucket)\n\t\tadj := samples * colWidth / maxSamples\n\t\treturn fmt.Sprintf(\"%s> %s: (%d)\", strings.Repeat(\"-\", int(adj)), ts(val), samples)\n\t}\n\n\tlines := make([]string, 0)\n\tfor i := 0; i < bucketCount; i++ {\n\t\tif i >= firstNonEmpty && i <= lastNonEmpty {\n\t\t\tlines = append(lines, p(i))\n\t\t}\n\n\t\tval = val << 1\n\t}\n\n\treturn strings.Join(lines, \"\\n\")\n}\n" }, { "path": "go/metrics/histogram_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage metrics\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestHistogramBucketValue(t *testing.T) {\n\tassert := assert.New(t)\n\n\th := Histogram{}\n\tassert.Equal(uint64(1<<0), h.bucketVal(0))\n\tassert.Equal(uint64(1<<1), h.bucketVal(1))\n\tassert.Equal(uint64(1<<2), h.bucketVal(2))\n\tassert.Equal(uint64(1<<32), h.bucketVal(32))\n\tassert.Equal(uint64(1<<40), h.bucketVal(40))\n}\n\nfunc TestHistogramBasic(t *testing.T) {\n\tassert := assert.New(t)\n\n\th := Histogram{}\n\n\th.Sample(1)\n\th.Sample(1)\n\tassert.Equal(uint64(2), h.buckets[0])\n\n\th.Sample(2)\n\th.Sample(3)\n\tassert.Equal(uint64(2), h.buckets[1])\n\n\th.Sample(4)\n\th.Sample(5)\n\th.Sample(6)\n\tassert.Equal(uint64(3), h.buckets[2])\n\n\th.Sample(256)\n\th.Sample(300)\n\th.Sample(500)\n\th.Sample(511)\n\tassert.Equal(uint64(4), h.buckets[8])\n\tassert.Equal(uint64(11), h.Samples())\n\tassert.Equal(uint64(1589), h.Sum())\n\tassert.Equal(uint64(144), h.Mean())\n}\n\nfunc TestHistogramLarge(t *testing.T) {\n\tassert := assert.New(t)\n\th := Histogram{}\n\th.Sample(0xfffffffffffffe30)\n\tassert.Equal(uint64(1), h.Samples())\n\tassert.Equal(uint64(0xfffffffffffffe30), h.Sum())\n}\n\nfunc TestHistogramAdd(t *testing.T) {\n\tassert := assert.New(t)\n\n\th := Histogram{}\n\th.Sample(1)\n\th.Sample(2)\n\th.Sample(10)\n\n\th2 := Histogram{}\n\th2.Sample(3)\n\th2.Sample(1073741854)\n\n\th.Add(h2)\n\tassert.Equal(uint64(5), h.Samples())\n\tassert.Equal(uint64(1073741870), h.Sum())\n\tassert.Equal(uint64(1073741870)/uint64(5), h.Mean())\n}\n\nfunc TestHistogramString(t *testing.T) {\n\tassert := assert.New(t)\n\n\th := Histogram{}\n\th.Sample(1)\n\th.Sample(2)\n\th.Sample(10)\n\th.Sample(3034030343)\n\n\tassert.Equal(\"Mean: 758507589, Sum: 3034030356, Samples: 4\", h.String())\n\n\tth := NewTimeHistogram()\n\tth.Add(h)\n\tassert.Equal(\"Mean: 758.507589ms, Sum: 3.034030356s, Samples: 4\", th.String())\n\n\tbh := NewByteHistogram()\n\tbh.Add(h)\n\tassert.Equal(\"Mean: 758 MB, Sum: 3.0 GB, Samples: 4\", bh.String())\n}\n\nfunc TestHistogramReport(t *testing.T) {\n\tassert := assert.New(t)\n\n\th := Histogram{}\n\th.Sample(1)\n\tassert.Equal(\"----------------------------------------------------------------------------------------------------> 1: (1)\", h.Report())\n\n\th.Sample(1 << 62)\n\tassert.Equal(`----------------------------------------------------------------------------------------------------> 1: (1)\n> 2: (0)\n> 4: (0)\n> 8: (0)\n> 16: (0)\n> 32: (0)\n> 64: (0)\n> 128: (0)\n> 256: (0)\n> 512: (0)\n> 1024: (0)\n> 2048: (0)\n> 4096: (0)\n> 8192: (0)\n> 16384: (0)\n> 32768: (0)\n> 65536: (0)\n> 131072: (0)\n> 262144: (0)\n> 524288: (0)\n> 1048576: (0)\n> 2097152: (0)\n> 4194304: (0)\n> 8388608: (0)\n> 16777216: (0)\n> 33554432: (0)\n> 67108864: (0)\n> 134217728: (0)\n> 268435456: (0)\n> 536870912: (0)\n> 1073741824: (0)\n> 2147483648: (0)\n> 4294967296: (0)\n> 8589934592: (0)\n> 17179869184: (0)\n> 34359738368: (0)\n> 68719476736: (0)\n> 137438953472: (0)\n> 274877906944: (0)\n> 549755813888: (0)\n> 1099511627776: (0)\n> 2199023255552: (0)\n> 4398046511104: (0)\n> 8796093022208: (0)\n> 17592186044416: (0)\n> 35184372088832: (0)\n> 70368744177664: (0)\n> 140737488355328: (0)\n> 281474976710656: (0)\n> 562949953421312: (0)\n> 1125899906842624: (0)\n> 2251799813685248: (0)\n> 4503599627370496: (0)\n> 9007199254740992: (0)\n> 18014398509481984: (0)\n> 36028797018963968: (0)\n> 72057594037927936: (0)\n> 144115188075855872: (0)\n> 288230376151711744: (0)\n> 576460752303423488: (0)\n> 1152921504606846976: (0)\n> 2305843009213693952: (0)\n----------------------------------------------------------------------------------------------------> 4611686018427387904: (1)`, h.Report())\n\n\th = Histogram{}\n\th.Sample(4)\n\th.Sample(8)\n\n\tassert.Equal(`----------------------------------------------------------------------------------------------------> 4: (1)\n----------------------------------------------------------------------------------------------------> 8: (1)`, h.Report())\n}\n" }, { "path": "go/nbs/NBS-on-AWS.md", "content": "# Backing a Noms Block Store with AWS\n\nHow to use S3 and DynamoDB as the persistent storage layer for a Noms Block Store (NBS).\n\n## Overview\n\nWhen running atop AWS, NBS stores immutable chunk data in S3 objects and mutable state -- a 'manifest' indicating which S3 objects are live, essentially -- in DynamoDB. It is possible to have many separate Noms Block Stores backed by a single bucket/table as long as you give each a distinct name. You could also choose to spin up a separate bucket/table pair for each NBS, though this is not required -- and, indeed, probably overkill.\n\n## AWS Setup\n\nThis assumes a setup in a single AWS region.\n\n### Create an S3 bucket and DynamoDB table\n\nThere are no special requirements on the S3 bucket you create. Just choose a name and, once it's created, remember the ARN for use later.\n\nThe DynamoDB table you create, on the other hand, does need to have a particular structure. It must have a *primary partition key* that is a *string* with the name *db*. Again, remember its ARN for later use.\n\n### Access control\n\nThe NBS code honors AWS credentials files, so when running on your development machine the easiest thing to do is drop the creds of the user that created the bucket and table above into `~/.aws/credentials` and run that way. This isn't a great approach for running in on an EC2 instance in production, however. The right way to do that is to create an IAM Role, and run your instance as that role.\n\nCreate such a role using the IAM Management Console (or command line tool of your choice) and make sure it has a policy with at least the following permissions:\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Sid\": \"Stmt1453230562000\",\n \"Effect\": \"Allow\",\n \"Action\": [\n \"dynamodb:BatchGetItem\",\n \"dynamodb:BatchWriteItem\",\n \"dynamodb:DeleteItem\",\n \"dynamodb:GetItem\",\n \"dynamodb:PutItem\",\n ],\n \"Resource\": [\n \"[ARN for your DynamoDB table]\",\n ]\n },\n {\n \"Sid\": \"Stmt1454457944000\",\n \"Effect\": \"Allow\",\n \"Action\": [\n \"s3:AbortMultipartUpload\"\n \"s3:CompleteMultipartUpload\",\n \"s3:CreateMultipartUpload\",\n \"s3:GetObject\",\n \"s3:PutObject\",\n \"s3:UploadPart\",\n \"s3:UploadPartCopy\",\n ],\n \"Resource\": [\n \"[ARN for your S3 bucket]\",\n ]\n }\n ]\n}\n```\n\nThis is where the ARN for your bucket and table come in.\n\n## Instantiating an NBS-on-AWS ChunkStore\n\n### On the command line\n\n```shell\nnoms ds aws:dynamo-table/s3-bucket/store-name\n```\n\n### NewAWSStore\n\nIf your code only needs to create a store pointing to a single named stores, you can write code similar to the following:\n\n```go\nsess := session.Must(session.NewSession(aws.NewConfig().WithRegion(\"us-west-2\")))\nstore := nbs.NewAWSStore(\"dynamo-table\", \"store-name\", \"s3-bucket\", s3.New(sess), dynamodb.New(sess), 1<<28))\n```\n\n### NewAWSStoreFactory\n\nIf you find yourself wanting to create NBS instances pointing to multiple, different named stores, you can use `nbs.NewAWSStoreFactory()`, which also supports caching Noms data on disk in some cases:\n\n```go\nsess := session.Must(session.NewSession(aws.NewConfig().WithRegion(\"us-west-2\")))\nfact := nbs.NewAWSStoreFactory(\n sess, \"dynamo-table\", \"s3-bucket\",\n 128 /* Maximum number of open files in cache */,\n 1 << 28 /* Amount of index data to cache in memory */,\n 1 << 30 /* Amount of Noms data to cache on disk */,\n \"/path/to/cache\" /* Directory in which to cache Noms data */,\n)\nstore := fact.CreateStore(\"store-name\")\n```\n\n" }, { "path": "go/nbs/README.md", "content": "# Noms Block Store\n\nA horizontally-scalable storage backend for Noms.\n\n## Overview\n\nNBS is a storage layer optimized for the needs of the [Noms](https://github.com/attic-labs/noms) database.\n\nNBS can run in two configurations: either backed by local disk, or [backed by Amazon AWS](https://github.com/attic-labs/noms/blob/master/go/nbs/NBS-on-AWS.md).\n\nWhen backed by local disk, NBS is significantly faster than LevelDB for our workloads and supports full multiprocess concurrency.\n\nWhen backed by AWS, NBS stores its data mainly in S3, along with a single DynamoDB item. This configuration makes Noms \"[effectively CA](https://research.google.com/pubs/pub45855.html)\", in the sense that Noms is always consistent, and Noms+NBS is as available as DynamoDB and S3 are. This configuration also gives Noms the cost profile of S3 with power closer to that of a traditional database.\n\n## Details\n\n* NBS provides storage for a content-addressed DAG of nodes (with exactly one root), where each node is encoded as a sequence of bytes and addressed by a 20-byte hash of the byte-sequence.\n* There is no `update` or `delete` -- only `insert`, `update root` and `garbage collect`.\n* Insertion of any novel byte-sequence is durable only upon updating the root.\n* File-level multiprocess concurrency is supported, with optimistic locking for multiple writers.\n* Writers need not worry about re-writing duplicate chunks. NBS will efficiently detect and drop (most) duplicates.\n\n## Perf\n\nFor the file back-end, perf is substantially better than LevelDB mainly because LDB spends substantial IO with the goal of keeping KV pairs in key-order which doesn't benenfit Noms at all. NBS locates related chunks together and thus reading data from a NBS store can be done quite alot faster. As an example, storing & retrieving a 1.1GB MP4 video file on a MBP i5 2.9Ghz:\n\n * LDB\n * Initial import: 44 MB/s, size on disk: 1.1 GB. \n * Import exact same bytes: 35 MB/s, size on disk: 1.4 GB.\n * Export: 60 MB/s\n * NBS\n * Initial import: 72 MB/s, size on disk: 1.1 GB.\n * Import exact same bytes: 92 MB/s, size on disk: 1.1GB.\n * Export: 300 MB/s\n\n## Status\n\nNBS is more-or-less \"beta\". There's still [work we want to do](https://github.com/attic-labs/noms/issues?q=is%3Aopen+is%3Aissue+label%3ANBS), but it now works better than LevelDB for our purposes and so we have made it the default local backend for Noms:\n\n```shell\n# This uses nbs locally:\n./csv-import foo.csv /Users/bob/csv-store::data\n```\n\nThe AWS backend is available via the `aws:` scheme:\n\n```shell\n./csv-import foo.csv aws:table/bucket/database::data\n```\n" }, { "path": "go/nbs/aws_chunk_source.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nfunc newAWSChunkSource(ddb *ddbTableStore, s3 *s3ObjectReader, al awsLimits, name addr, chunkCount uint32, indexCache *indexCache, stats *Stats) chunkSource {\n\tif indexCache != nil {\n\t\tindexCache.lockEntry(name)\n\t\tdefer indexCache.unlockEntry(name)\n\t\tif index, found := indexCache.get(name); found {\n\t\t\ttra := &awsTableReaderAt{al: al, ddb: ddb, s3: s3, name: name, chunkCount: chunkCount}\n\t\t\treturn &awsChunkSource{newTableReader(index, tra, s3BlockSize), name}\n\t\t}\n\t}\n\n\tt1 := time.Now()\n\tindexBytes, tra := func() ([]byte, tableReaderAt) {\n\t\tif al.tableMayBeInDynamo(chunkCount) {\n\t\t\tdata, err := ddb.ReadTable(name, stats)\n\t\t\tif data != nil {\n\t\t\t\treturn data, &dynamoTableReaderAt{ddb: ddb, h: name}\n\t\t\t}\n\t\t\td.PanicIfTrue(err == nil) // There MUST be either data or an error\n\t\t\td.PanicIfNotType(err, tableNotInDynamoErr{})\n\t\t}\n\n\t\tsize := indexSize(chunkCount) + footerSize\n\t\tbuff := make([]byte, size)\n\n\t\tn, err := s3.ReadFromEnd(name, buff, stats)\n\t\td.PanicIfError(err)\n\t\td.PanicIfFalse(size == uint64(n))\n\t\treturn buff, &s3TableReaderAt{s3: s3, h: name}\n\t}()\n\tstats.IndexBytesPerRead.Sample(uint64(len(indexBytes)))\n\tstats.IndexReadLatency.SampleTimeSince(t1)\n\n\tindex := parseTableIndex(indexBytes)\n\tif indexCache != nil {\n\t\tindexCache.put(name, index)\n\t}\n\treturn &awsChunkSource{newTableReader(index, tra, s3BlockSize), name}\n}\n\ntype awsChunkSource struct {\n\ttableReader\n\tname addr\n}\n\nfunc (acs *awsChunkSource) hash() addr {\n\treturn acs.name\n}\n\ntype awsTableReaderAt struct {\n\tonce sync.Once\n\ttra tableReaderAt\n\n\tal awsLimits\n\tddb *ddbTableStore\n\ts3 *s3ObjectReader\n\n\tname addr\n\tchunkCount uint32\n}\n\nfunc (atra *awsTableReaderAt) hash() addr {\n\treturn atra.name\n}\n\nfunc (atra *awsTableReaderAt) ReadAtWithStats(p []byte, off int64, stats *Stats) (n int, err error) {\n\tatra.once.Do(func() { atra.tra = atra.getTableReaderAt(stats) })\n\treturn atra.tra.ReadAtWithStats(p, off, stats)\n}\n\nfunc (atra *awsTableReaderAt) getTableReaderAt(stats *Stats) tableReaderAt {\n\tif atra.al.tableMayBeInDynamo(atra.chunkCount) {\n\t\tdata, err := atra.ddb.ReadTable(atra.name, stats)\n\t\tif data != nil {\n\t\t\treturn &dynamoTableReaderAt{ddb: atra.ddb, h: atra.name}\n\t\t}\n\t\td.PanicIfTrue(err == nil) // There MUST be either data or an error\n\t\td.PanicIfNotType(err, tableNotInDynamoErr{})\n\t}\n\n\treturn &s3TableReaderAt{s3: atra.s3, h: atra.name}\n}\n" }, { "path": "go/nbs/aws_chunk_source_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestAWSChunkSource(t *testing.T) {\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\ttableData, h := buildTable(chunks)\n\n\ts3 := makeFakeS3(t)\n\tddb := makeFakeDDB(t)\n\n\ts3or := &s3ObjectReader{s3, \"bucket\", nil, nil}\n\tdts := &ddbTableStore{ddb, \"table\", nil, nil}\n\n\tmakeSrc := func(chunkMax int, ic *indexCache) chunkSource {\n\t\treturn newAWSChunkSource(\n\t\t\tdts,\n\t\t\ts3or,\n\t\t\tawsLimits{itemMax: maxDynamoItemSize, chunkMax: uint32(chunkMax)},\n\t\t\th,\n\t\t\tuint32(len(chunks)),\n\t\t\tic,\n\t\t\t&Stats{},\n\t\t)\n\t}\n\n\tt.Run(\"Dynamo\", func(t *testing.T) {\n\t\tddb.putData(fmtTableName(h), tableData)\n\n\t\tt.Run(\"NoIndexCache\", func(t *testing.T) {\n\t\t\tsrc := makeSrc(len(chunks)+1, nil)\n\t\t\tassertChunksInReader(chunks, src, assert.New(t))\n\t\t})\n\n\t\tt.Run(\"WithIndexCache\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\t\t\tindex := parseTableIndex(tableData)\n\t\t\tcache := newIndexCache(1024)\n\t\t\tcache.put(h, index)\n\n\t\t\tbaseline := ddb.numGets\n\t\t\tsrc := makeSrc(len(chunks)+1, cache)\n\n\t\t\t// constructing the table reader shouldn't have resulted in any reads\n\t\t\tassert.Zero(ddb.numGets - baseline)\n\t\t\tassertChunksInReader(chunks, src, assert)\n\t\t})\n\t})\n\n\tt.Run(\"S3\", func(t *testing.T) {\n\t\ts3.data[h.String()] = tableData\n\n\t\tt.Run(\"NoIndexCache\", func(t *testing.T) {\n\t\t\tsrc := makeSrc(len(chunks)-1, nil)\n\t\t\tassertChunksInReader(chunks, src, assert.New(t))\n\t\t})\n\n\t\tt.Run(\"WithIndexCache\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\t\t\tindex := parseTableIndex(tableData)\n\t\t\tcache := newIndexCache(1024)\n\t\t\tcache.put(h, index)\n\n\t\t\tbaseline := s3.getCount\n\t\t\tsrc := makeSrc(len(chunks)-1, cache)\n\n\t\t\t// constructing the table reader shouldn't have resulted in any reads\n\t\t\tassert.Zero(s3.getCount - baseline)\n\t\t\tassertChunksInReader(chunks, src, assert)\n\t\t})\n\t})\n}\n" }, { "path": "go/nbs/aws_table_persister.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"io\"\n\t\"net/url\"\n\t\"sort\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/util/verbose\"\n\t\"github.com/aws/aws-sdk-go/aws\"\n\t\"github.com/aws/aws-sdk-go/service/s3\"\n)\n\nconst (\n\tminS3PartSize = 5 * 1 << 20 // 5MiB\n\tmaxS3PartSize = 64 * 1 << 20 // 64MiB\n\tmaxS3Parts = 10000\n\n\tmaxDynamoChunks = 64\n\tmaxDynamoItemSize = 400 * (1 << 10) // 400k\n\n\tdefaultS3PartSize = minS3PartSize // smallest allowed by S3 allows for most throughput\n)\n\ntype awsTablePersister struct {\n\ts3 s3svc\n\tbucket string\n\trl chan struct{}\n\ttc tableCache\n\tddb *ddbTableStore\n\tlimits awsLimits\n\tindexCache *indexCache\n}\n\ntype awsLimits struct {\n\tpartTarget, partMin, partMax uint64\n\titemMax int\n\tchunkMax uint32\n}\n\nfunc (al awsLimits) tableFitsInDynamo(name addr, dataLen int, chunkCount uint32) bool {\n\tcalcItemSize := func(n addr, dataLen int) int {\n\t\treturn len(dbAttr) + len(tablePrefix) + len(n.String()) + len(dataAttr) + dataLen\n\t}\n\treturn chunkCount <= al.chunkMax && calcItemSize(name, dataLen) < al.itemMax\n}\n\nfunc (al awsLimits) tableMayBeInDynamo(chunkCount uint32) bool {\n\treturn chunkCount <= al.chunkMax\n}\n\nfunc (s3p awsTablePersister) Open(name addr, chunkCount uint32, stats *Stats) chunkSource {\n\treturn newAWSChunkSource(\n\t\ts3p.ddb,\n\t\t&s3ObjectReader{s3: s3p.s3, bucket: s3p.bucket, readRl: s3p.rl, tc: s3p.tc},\n\t\ts3p.limits,\n\t\tname,\n\t\tchunkCount,\n\t\ts3p.indexCache,\n\t\tstats,\n\t)\n}\n\ntype s3UploadedPart struct {\n\tidx int64\n\tetag string\n}\n\nfunc (s3p awsTablePersister) Persist(mt *memTable, haver chunkReader, stats *Stats) chunkSource {\n\tname, data, chunkCount := mt.write(haver, stats)\n\tif chunkCount == 0 {\n\t\treturn emptyChunkSource{}\n\t}\n\tif s3p.limits.tableFitsInDynamo(name, len(data), chunkCount) {\n\t\ts3p.ddb.Write(name, data)\n\t\treturn s3p.newReaderFromIndexData(data, name, &dynamoTableReaderAt{ddb: s3p.ddb, h: name})\n\t}\n\n\tif s3p.tc != nil {\n\t\tgo s3p.tc.store(name, bytes.NewReader(data), uint64(len(data)))\n\t}\n\ts3p.multipartUpload(data, name.String())\n\ttra := &s3TableReaderAt{&s3ObjectReader{s3: s3p.s3, bucket: s3p.bucket, readRl: s3p.rl, tc: s3p.tc}, name}\n\treturn s3p.newReaderFromIndexData(data, name, tra)\n}\n\nfunc (s3p awsTablePersister) newReaderFromIndexData(idxData []byte, name addr, tra tableReaderAt) chunkSource {\n\tindex := parseTableIndex(idxData)\n\tif s3p.indexCache != nil {\n\t\ts3p.indexCache.lockEntry(name)\n\t\tdefer s3p.indexCache.unlockEntry(name)\n\t\ts3p.indexCache.put(name, index)\n\t}\n\treturn &awsChunkSource{newTableReader(index, tra, s3BlockSize), name}\n}\n\nfunc (s3p awsTablePersister) multipartUpload(data []byte, key string) {\n\tuploadID := s3p.startMultipartUpload(key)\n\tmultipartUpload, err := s3p.uploadParts(data, key, uploadID)\n\tif err != nil {\n\t\ts3p.abortMultipartUpload(key, uploadID)\n\t\td.PanicIfError(err) // TODO: Better error handling here\n\t}\n\ts3p.completeMultipartUpload(key, uploadID, multipartUpload)\n}\n\nfunc (s3p awsTablePersister) startMultipartUpload(key string) string {\n\tresult, err := s3p.s3.CreateMultipartUpload(&s3.CreateMultipartUploadInput{\n\t\tBucket: aws.String(s3p.bucket),\n\t\tKey: aws.String(key),\n\t})\n\td.PanicIfError(err)\n\treturn *result.UploadId\n}\n\nfunc (s3p awsTablePersister) abortMultipartUpload(key, uploadID string) {\n\t_, abrtErr := s3p.s3.AbortMultipartUpload(&s3.AbortMultipartUploadInput{\n\t\tBucket: aws.String(s3p.bucket),\n\t\tKey: aws.String(key),\n\t\tUploadId: aws.String(uploadID),\n\t})\n\td.PanicIfError(abrtErr)\n}\n\nfunc (s3p awsTablePersister) completeMultipartUpload(key, uploadID string, mpu *s3.CompletedMultipartUpload) {\n\t_, err := s3p.s3.CompleteMultipartUpload(&s3.CompleteMultipartUploadInput{\n\t\tBucket: aws.String(s3p.bucket),\n\t\tKey: aws.String(key),\n\t\tMultipartUpload: mpu,\n\t\tUploadId: aws.String(uploadID),\n\t})\n\td.PanicIfError(err)\n}\n\nfunc (s3p awsTablePersister) uploadParts(data []byte, key, uploadID string) (*s3.CompletedMultipartUpload, error) {\n\tsent, failed, done := make(chan s3UploadedPart), make(chan error), make(chan struct{})\n\n\tnumParts := getNumParts(uint64(len(data)), s3p.limits.partTarget)\n\td.PanicIfTrue(numParts > maxS3Parts) // TODO: BUG 3433: handle > 10k parts\n\tvar wg sync.WaitGroup\n\tsendPart := func(partNum, start, end uint64) {\n\t\tif s3p.rl != nil {\n\t\t\ts3p.rl <- struct{}{}\n\t\t\tdefer func() { <-s3p.rl }()\n\t\t}\n\t\tdefer wg.Done()\n\n\t\t// Check if upload has been terminated\n\t\tselect {\n\t\tcase <-done:\n\t\t\treturn\n\t\tdefault:\n\t\t}\n\t\t// Upload the desired part\n\t\tif partNum == numParts { // If this is the last part, make sure it includes any overflow\n\t\t\tend = uint64(len(data))\n\t\t}\n\t\tetag, err := s3p.uploadPart(data[start:end], key, uploadID, int64(partNum))\n\t\tif err != nil {\n\t\t\tfailed <- err\n\t\t\treturn\n\t\t}\n\t\t// Try to send along part info. In the case that the upload was aborted, reading from done allows this worker to exit correctly.\n\t\tselect {\n\t\tcase sent <- s3UploadedPart{int64(partNum), etag}:\n\t\tcase <-done:\n\t\t\treturn\n\t\t}\n\t}\n\tfor i := uint64(0); i < numParts; i++ {\n\t\twg.Add(1)\n\t\tpartNum := i + 1 // Parts are 1-indexed\n\t\tstart, end := i*s3p.limits.partTarget, (i+1)*s3p.limits.partTarget\n\t\tgo sendPart(partNum, start, end)\n\t}\n\tgo func() {\n\t\twg.Wait()\n\t\tclose(sent)\n\t\tclose(failed)\n\t}()\n\n\tmultipartUpload := &s3.CompletedMultipartUpload{}\n\tvar firstFailure error\n\tfor cont := true; cont; {\n\t\tselect {\n\t\tcase sentPart, open := <-sent:\n\t\t\tif open {\n\t\t\t\tmultipartUpload.Parts = append(multipartUpload.Parts, &s3.CompletedPart{\n\t\t\t\t\tETag: aws.String(sentPart.etag),\n\t\t\t\t\tPartNumber: aws.Int64(sentPart.idx),\n\t\t\t\t})\n\t\t\t}\n\t\t\tcont = open\n\n\t\tcase err := <-failed:\n\t\t\tif err != nil && firstFailure == nil { // nil err may happen when failed gets closed\n\t\t\t\tfirstFailure = err\n\t\t\t\tclose(done)\n\t\t\t}\n\t\t}\n\t}\n\n\tif firstFailure == nil {\n\t\tclose(done)\n\t}\n\tsort.Sort(partsByPartNum(multipartUpload.Parts))\n\treturn multipartUpload, firstFailure\n}\n\nfunc getNumParts(dataLen, minPartSize uint64) uint64 {\n\tnumParts := dataLen / minPartSize\n\tif numParts == 0 {\n\t\tnumParts = 1\n\t}\n\treturn numParts\n}\n\ntype partsByPartNum []*s3.CompletedPart\n\nfunc (s partsByPartNum) Len() int {\n\treturn len(s)\n}\n\nfunc (s partsByPartNum) Less(i, j int) bool {\n\treturn *s[i].PartNumber < *s[j].PartNumber\n}\n\nfunc (s partsByPartNum) Swap(i, j int) {\n\ts[i], s[j] = s[j], s[i]\n}\n\nfunc (s3p awsTablePersister) ConjoinAll(sources chunkSources, stats *Stats) chunkSource {\n\tplan := planConjoin(sources, stats)\n\tif plan.chunkCount == 0 {\n\t\treturn emptyChunkSource{}\n\t}\n\tt1 := time.Now()\n\tname := nameFromSuffixes(plan.suffixes())\n\ts3p.executeCompactionPlan(plan, name.String())\n\tverbose.Log(\"Compacted table of %d Kb in %s\", plan.totalCompressedData/1024, time.Since(t1))\n\n\tif s3p.tc != nil {\n\t\tgo s3p.loadIntoCache(name) // load conjoined table to the cache\n\t}\n\ttra := &s3TableReaderAt{&s3ObjectReader{s3: s3p.s3, bucket: s3p.bucket, readRl: s3p.rl, tc: s3p.tc}, name}\n\treturn s3p.newReaderFromIndexData(plan.mergedIndex, name, tra)\n}\n\nfunc (s3p awsTablePersister) loadIntoCache(name addr) {\n\tinput := &s3.GetObjectInput{\n\t\tBucket: aws.String(s3p.bucket),\n\t\tKey: aws.String(name.String()),\n\t}\n\tresult, err := s3p.s3.GetObject(input)\n\td.PanicIfError(err)\n\n\ts3p.tc.store(name, result.Body, uint64(*result.ContentLength))\n}\n\nfunc (s3p awsTablePersister) executeCompactionPlan(plan compactionPlan, key string) {\n\tuploadID := s3p.startMultipartUpload(key)\n\tmultipartUpload, err := s3p.assembleTable(plan, key, uploadID)\n\tif err != nil {\n\t\ts3p.abortMultipartUpload(key, uploadID)\n\t\td.PanicIfError(err) // TODO: Better error handling here\n\t}\n\ts3p.completeMultipartUpload(key, uploadID, multipartUpload)\n}\n\nfunc (s3p awsTablePersister) assembleTable(plan compactionPlan, key, uploadID string) (*s3.CompletedMultipartUpload, error) {\n\td.PanicIfTrue(len(plan.sources) > maxS3Parts) // TODO: BUG 3433: handle > 10k parts\n\n\t// Separate plan.sources by amount of chunkData. Tables with >5MB of chunk data (copies) can be added to the new table using S3's multipart upload copy feature. Smaller tables with <5MB of chunk data (manuals) must be read, assembled into |buff|, and then re-uploaded in parts that are larger than 5MB.\n\tcopies, manuals, buff := dividePlan(plan, uint64(s3p.limits.partMin), uint64(s3p.limits.partMax))\n\n\t// Concurrently read data from small tables into |buff|\n\tvar readWg sync.WaitGroup\n\tfor _, man := range manuals {\n\t\treadWg.Add(1)\n\t\tgo func(m manualPart) {\n\t\t\tdefer readWg.Done()\n\t\t\tn, _ := m.srcR.Read(buff[m.dstStart:m.dstEnd])\n\t\t\td.PanicIfTrue(int64(n) < m.dstEnd-m.dstStart)\n\t\t}(man)\n\t}\n\treadWg.Wait()\n\n\t// sendPart calls |doUpload| to send part |partNum|, forwarding errors over |failed| or success over |sent|. Closing (or sending) on |done| will cancel all in-progress calls to sendPart.\n\tsent, failed, done := make(chan s3UploadedPart), make(chan error), make(chan struct{})\n\tvar uploadWg sync.WaitGroup\n\ttype uploadFn func() (etag string, err error)\n\tsendPart := func(partNum int64, doUpload uploadFn) {\n\t\tif s3p.rl != nil {\n\t\t\ts3p.rl <- struct{}{}\n\t\t\tdefer func() { <-s3p.rl }()\n\t\t}\n\t\tdefer uploadWg.Done()\n\n\t\t// Check if upload has been terminated\n\t\tselect {\n\t\tcase <-done:\n\t\t\treturn\n\t\tdefault:\n\t\t}\n\n\t\tetag, err := doUpload()\n\t\tif err != nil {\n\t\t\tfailed <- err\n\t\t\treturn\n\t\t}\n\t\t// Try to send along part info. In the case that the upload was aborted, reading from done allows this worker to exit correctly.\n\t\tselect {\n\t\tcase sent <- s3UploadedPart{int64(partNum), etag}:\n\t\tcase <-done:\n\t\t\treturn\n\t\t}\n\t}\n\n\t// Concurrently begin sending all parts using sendPart().\n\t// First, kick off sending all the copyable parts.\n\tpartNum := int64(1) // Part numbers are 1-indexed\n\tfor _, cp := range copies {\n\t\tuploadWg.Add(1)\n\t\tgo func(cp copyPart, partNum int64) {\n\t\t\tsendPart(partNum, func() (etag string, err error) {\n\t\t\t\treturn s3p.uploadPartCopy(cp.name, cp.srcOffset, cp.srcLen, key, uploadID, partNum)\n\t\t\t})\n\t\t}(cp, partNum)\n\t\tpartNum++\n\t}\n\n\t// Then, split buff (data from |manuals| and index) into parts and upload those concurrently.\n\tnumManualParts := getNumParts(uint64(len(buff)), s3p.limits.partTarget) // TODO: What if this is too big?\n\tfor i := uint64(0); i < numManualParts; i++ {\n\t\tstart, end := i*s3p.limits.partTarget, (i+1)*s3p.limits.partTarget\n\t\tif i+1 == numManualParts { // If this is the last part, make sure it includes any overflow\n\t\t\tend = uint64(len(buff))\n\t\t}\n\t\tuploadWg.Add(1)\n\t\tgo func(data []byte, partNum int64) {\n\t\t\tsendPart(partNum, func() (etag string, err error) {\n\t\t\t\treturn s3p.uploadPart(data, key, uploadID, partNum)\n\t\t\t})\n\t\t}(buff[start:end], partNum)\n\t\tpartNum++\n\t}\n\n\t// When all the uploads started above are done, close |sent| and |failed| so that the code below will correctly detect that we're done sending parts and move forward.\n\tgo func() {\n\t\tuploadWg.Wait()\n\t\tclose(sent)\n\t\tclose(failed)\n\t}()\n\n\t// Watch |sent| and |failed| for the results of part uploads. If ever one fails, close |done| to stop all the in-progress or pending sendPart() calls and then bail.\n\tmultipartUpload := &s3.CompletedMultipartUpload{}\n\tvar firstFailure error\n\tfor cont := true; cont; {\n\t\tselect {\n\t\tcase sentPart, open := <-sent:\n\t\t\tif open {\n\t\t\t\tmultipartUpload.Parts = append(multipartUpload.Parts, &s3.CompletedPart{\n\t\t\t\t\tETag: aws.String(sentPart.etag),\n\t\t\t\t\tPartNumber: aws.Int64(sentPart.idx),\n\t\t\t\t})\n\t\t\t}\n\t\t\tcont = open\n\n\t\tcase err := <-failed:\n\t\t\tif err != nil && firstFailure == nil { // nil err may happen when failed gets closed\n\t\t\t\tfirstFailure = err\n\t\t\t\tclose(done)\n\t\t\t}\n\t\t}\n\t}\n\n\t// If there was any failure detected above, |done| is already closed\n\tif firstFailure == nil {\n\t\tclose(done)\n\t}\n\tsort.Sort(partsByPartNum(multipartUpload.Parts)) // S3 requires that these be in part-order\n\treturn multipartUpload, firstFailure\n}\n\ntype copyPart struct {\n\tname string\n\tsrcOffset, srcLen int64\n}\n\ntype manualPart struct {\n\tsrcR io.Reader\n\tdstStart, dstEnd int64\n}\n\n// dividePlan assumes that plan.sources (which is of type chunkSourcesByDescendingDataSize) is correctly sorted by descending data size.\nfunc dividePlan(plan compactionPlan, minPartSize, maxPartSize uint64) (copies []copyPart, manuals []manualPart, buff []byte) {\n\t// NB: if maxPartSize < 2*minPartSize, splitting large copies apart isn't solvable. S3's limits are plenty far enough apart that this isn't a problem in production, but we could violate this in tests.\n\td.PanicIfTrue(maxPartSize < 2*minPartSize)\n\n\tbuffSize := uint64(len(plan.mergedIndex))\n\ti := 0\n\tfor ; i < len(plan.sources); i++ {\n\t\tsws := plan.sources[i]\n\t\tif sws.dataLen < minPartSize {\n\t\t\t// since plan.sources is sorted in descending chunk-data-length order, we know that sws and all members after it are too small to copy.\n\t\t\tbreak\n\t\t}\n\t\tif sws.dataLen <= maxPartSize {\n\t\t\tcopies = append(copies, copyPart{sws.source.hash().String(), 0, int64(sws.dataLen)})\n\t\t\tcontinue\n\t\t}\n\n\t\t// Now, we need to break the data into some number of parts such that for all parts minPartSize <= size(part) <= maxPartSize. This code tries to split the part evenly, such that all new parts satisfy the previous inequality. This gets tricky around edge cases. Consider min = 5b and max = 10b and a data length of 101b. You need to send 11 parts, but you can't just send 10 parts of 10 bytes and 1 part of 1 byte -- the last is too small. You also can't send 10 parts of 9 bytes each and 1 part of 11 bytes, because the last is too big. You have to distribute the extra bytes across all the parts so that all of them fall into the proper size range.\n\t\tlens := splitOnMaxSize(sws.dataLen, maxPartSize)\n\n\t\tvar srcStart int64\n\t\tfor _, length := range lens {\n\t\t\tcopies = append(copies, copyPart{sws.source.hash().String(), srcStart, length})\n\t\t\tsrcStart += length\n\t\t}\n\t}\n\tvar offset int64\n\tfor ; i < len(plan.sources); i++ {\n\t\tsws := plan.sources[i]\n\t\tmanuals = append(manuals, manualPart{sws.source.reader(), offset, offset + int64(sws.dataLen)})\n\t\toffset += int64(sws.dataLen)\n\t\tbuffSize += sws.dataLen\n\t}\n\tbuff = make([]byte, buffSize)\n\tcopy(buff[buffSize-uint64(len(plan.mergedIndex)):], plan.mergedIndex)\n\treturn\n}\n\n// Splits |dataLen| into the maximum number of roughly-equal part sizes such that each is <= maxPartSize.\nfunc splitOnMaxSize(dataLen, maxPartSize uint64) []int64 {\n\tnumParts := dataLen / maxPartSize\n\tif dataLen%maxPartSize > 0 {\n\t\tnumParts++\n\t}\n\tbaseSize := int64(dataLen / numParts)\n\textraBytes := dataLen % numParts\n\tsizes := make([]int64, numParts)\n\tfor i := range sizes {\n\t\tsizes[i] = baseSize\n\t\tif extraBytes > 0 {\n\t\t\tsizes[i]++\n\t\t\textraBytes--\n\t\t}\n\t}\n\treturn sizes\n}\n\nfunc (s3p awsTablePersister) uploadPartCopy(src string, srcStart, srcEnd int64, key, uploadID string, partNum int64) (etag string, err error) {\n\tres, err := s3p.s3.UploadPartCopy(&s3.UploadPartCopyInput{\n\t\t// TODO: Use url.PathEscape() once we're on go 1.8\n\t\tCopySource: aws.String(url.QueryEscape(s3p.bucket + \"/\" + src)),\n\t\tCopySourceRange: aws.String(s3RangeHeader(srcStart, srcEnd)),\n\t\tBucket: aws.String(s3p.bucket),\n\t\tKey: aws.String(key),\n\t\tPartNumber: aws.Int64(int64(partNum)),\n\t\tUploadId: aws.String(uploadID),\n\t})\n\tif err == nil {\n\t\tetag = *res.CopyPartResult.ETag\n\t}\n\treturn\n}\n\nfunc (s3p awsTablePersister) uploadPart(data []byte, key, uploadID string, partNum int64) (etag string, err error) {\n\tres, err := s3p.s3.UploadPart(&s3.UploadPartInput{\n\t\tBucket: aws.String(s3p.bucket),\n\t\tKey: aws.String(key),\n\t\tPartNumber: aws.Int64(int64(partNum)),\n\t\tUploadId: aws.String(uploadID),\n\t\tBody: bytes.NewReader(data),\n\t})\n\tif err == nil {\n\t\tetag = *res.ETag\n\t}\n\treturn\n}\n" }, { "path": "go/nbs/aws_table_persister_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"io\"\n\t\"math/rand\"\n\t\"sync\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/util/sizecache\"\n\t\"github.com/aws/aws-sdk-go/service/s3\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestAWSTablePersisterPersist(t *testing.T) {\n\tcalcPartSize := func(rdr chunkReader, maxPartNum uint64) uint64 {\n\t\treturn maxTableSize(uint64(rdr.count()), rdr.uncompressedLen()) / maxPartNum\n\t}\n\n\tmt := newMemTable(testMemTableSize)\n\tfor _, c := range testChunks {\n\t\tassert.True(t, mt.addChunk(computeAddr(c), c))\n\t}\n\n\tt.Run(\"PersistToS3\", func(t *testing.T) {\n\t\tt.Run(\"InMultipleParts\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\t\t\ts3svc, ddb := makeFakeS3(t), makeFakeDTS(makeFakeDDB(t), nil)\n\t\t\tic := newIndexCache(1024)\n\t\t\tlimits := awsLimits{partTarget: calcPartSize(mt, 3)}\n\t\t\ts3p := awsTablePersister{s3: s3svc, bucket: \"bucket\", ddb: ddb, limits: limits, indexCache: ic}\n\n\t\t\tsrc := s3p.Persist(mt, nil, &Stats{})\n\t\t\tassert.NotNil(ic.get(src.hash()))\n\n\t\t\tif assert.True(src.count() > 0) {\n\t\t\t\tif r := s3svc.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\t\tassertChunksInReader(testChunks, r, assert)\n\t\t\t\t}\n\t\t\t}\n\t\t})\n\n\t\tt.Run(\"CacheTable\", func(t *testing.T) {\n\t\t\ts3svc, ddb := makeFakeS3(t), makeFakeDTS(makeFakeDDB(t), nil)\n\t\t\tlimits := awsLimits{partTarget: calcPartSize(mt, 3)}\n\t\t\ttc := &waitOnStoreTableCache{readers: map[addr]io.ReaderAt{}}\n\t\t\ts3p := awsTablePersister{s3: s3svc, bucket: \"bucket\", ddb: ddb, limits: limits, tc: tc}\n\n\t\t\t// Persist and wait until tc.store() has completed\n\t\t\ttc.storeWG.Add(1)\n\t\t\tsrc := s3p.Persist(mt, nil, &Stats{})\n\t\t\ttc.storeWG.Wait()\n\n\t\t\t// Now, open the table that should have been cached by the above Persist() and read out all the chunks. All the reads should be serviced from tc.\n\t\t\trdr := s3p.Open(src.hash(), src.count(), &Stats{})\n\t\t\tbaseline := s3svc.getCount\n\t\t\tch := make(chan extractRecord)\n\t\t\tgo func() { defer close(ch); rdr.extract(ch) }()\n\t\t\tfor range ch {\n\t\t\t}\n\t\t\tassert.Zero(t, s3svc.getCount-baseline)\n\t\t})\n\n\t\tt.Run(\"InSinglePart\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\n\t\t\ts3svc, ddb := makeFakeS3(t), makeFakeDTS(makeFakeDDB(t), nil)\n\t\t\tlimits := awsLimits{partTarget: calcPartSize(mt, 1)}\n\t\t\ts3p := awsTablePersister{s3: s3svc, bucket: \"bucket\", ddb: ddb, limits: limits}\n\n\t\t\tsrc := s3p.Persist(mt, nil, &Stats{})\n\t\t\tif assert.True(src.count() > 0) {\n\t\t\t\tif r := s3svc.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\t\tassertChunksInReader(testChunks, r, assert)\n\t\t\t\t}\n\t\t\t}\n\t\t})\n\n\t\tt.Run(\"NoNewChunks\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\n\t\t\tmt := newMemTable(testMemTableSize)\n\t\t\texistingTable := newMemTable(testMemTableSize)\n\n\t\t\tfor _, c := range testChunks {\n\t\t\t\tassert.True(mt.addChunk(computeAddr(c), c))\n\t\t\t\tassert.True(existingTable.addChunk(computeAddr(c), c))\n\t\t\t}\n\n\t\t\ts3svc, ddb := makeFakeS3(t), makeFakeDTS(makeFakeDDB(t), nil)\n\t\t\tlimits := awsLimits{partTarget: 1 << 10}\n\t\t\ts3p := awsTablePersister{s3: s3svc, bucket: \"bucket\", ddb: ddb, limits: limits}\n\n\t\t\tsrc := s3p.Persist(mt, existingTable, &Stats{})\n\t\t\tassert.True(src.count() == 0)\n\n\t\t\t_, present := s3svc.data[src.hash().String()]\n\t\t\tassert.False(present)\n\t\t})\n\n\t\tt.Run(\"Abort\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\n\t\t\ts3svc := &failingFakeS3{makeFakeS3(t), sync.Mutex{}, 1}\n\t\t\tddb := makeFakeDTS(makeFakeDDB(t), nil)\n\t\t\tlimits := awsLimits{partTarget: calcPartSize(mt, 4)}\n\t\t\ts3p := awsTablePersister{s3: s3svc, bucket: \"bucket\", ddb: ddb, limits: limits}\n\n\t\t\tassert.Panics(func() { s3p.Persist(mt, nil, &Stats{}) })\n\t\t})\n\t})\n\n\tt.Run(\"PersistToDynamo\", func(t *testing.T) {\n\t\tt.Run(\"Success\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\n\t\t\tddb := makeFakeDDB(t)\n\t\t\ts3svc, dts := makeFakeS3(t), makeFakeDTS(ddb, nil)\n\t\t\tlimits := awsLimits{itemMax: maxDynamoItemSize, chunkMax: 2 * mt.count()}\n\t\t\ts3p := awsTablePersister{s3: s3svc, bucket: \"bucket\", ddb: dts, limits: limits}\n\n\t\t\tsrc := s3p.Persist(mt, nil, &Stats{})\n\t\t\tif assert.True(src.count() > 0) {\n\t\t\t\tif r := ddb.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\t\tassertChunksInReader(testChunks, r, assert)\n\t\t\t\t}\n\t\t\t}\n\t\t})\n\n\t\tt.Run(\"CacheOnOpen\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\n\t\t\ttc := sizecache.New(maxDynamoItemSize)\n\t\t\tddb := makeFakeDDB(t)\n\t\t\ts3svc, dts := makeFakeS3(t), makeFakeDTS(ddb, tc)\n\t\t\tlimits := awsLimits{itemMax: maxDynamoItemSize, chunkMax: 2 * mt.count()}\n\n\t\t\ts3p := awsTablePersister{s3: s3svc, bucket: \"bucket\", ddb: dts, limits: limits}\n\n\t\t\ttableData, name := buildTable(testChunks)\n\t\t\tddb.putData(fmtTableName(name), tableData)\n\n\t\t\tsrc := s3p.Open(name, uint32(len(testChunks)), &Stats{})\n\t\t\tif assert.True(src.count() > 0) {\n\t\t\t\tif r := ddb.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\t\tassertChunksInReader(testChunks, r, assert)\n\t\t\t\t}\n\t\t\t\tif data, present := tc.Get(name); assert.True(present) {\n\t\t\t\t\tassert.Equal(tableData, data.([]byte))\n\t\t\t\t}\n\t\t\t}\n\t\t})\n\n\t\tt.Run(\"FailTooManyChunks\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\n\t\t\tddb := makeFakeDDB(t)\n\t\t\ts3svc, dts := makeFakeS3(t), makeFakeDTS(ddb, nil)\n\t\t\tlimits := awsLimits{itemMax: maxDynamoItemSize, chunkMax: 1, partTarget: calcPartSize(mt, 1)}\n\t\t\ts3p := awsTablePersister{s3: s3svc, bucket: \"bucket\", ddb: dts, limits: limits}\n\n\t\t\tsrc := s3p.Persist(mt, nil, &Stats{})\n\t\t\tif assert.True(src.count() > 0) {\n\t\t\t\tif r := ddb.readerForTable(src.hash()); assert.Nil(r) {\n\t\t\t\t\tif r = s3svc.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\t\t\tassertChunksInReader(testChunks, r, assert)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t})\n\n\t\tt.Run(\"FailItemTooBig\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\n\t\t\tddb := makeFakeDDB(t)\n\t\t\ts3svc, dts := makeFakeS3(t), makeFakeDTS(ddb, nil)\n\t\t\tlimits := awsLimits{itemMax: 0, chunkMax: 2 * mt.count(), partTarget: calcPartSize(mt, 1)}\n\t\t\ts3p := awsTablePersister{s3: s3svc, bucket: \"bucket\", ddb: dts, limits: limits}\n\n\t\t\tsrc := s3p.Persist(mt, nil, &Stats{})\n\t\t\tif assert.True(src.count() > 0) {\n\t\t\t\tif r := ddb.readerForTable(src.hash()); assert.Nil(r) {\n\t\t\t\t\tif r = s3svc.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\t\t\tassertChunksInReader(testChunks, r, assert)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t})\n\t})\n}\nfunc makeFakeDTS(ddb ddbsvc, tc *sizecache.SizeCache) *ddbTableStore {\n\treturn &ddbTableStore{ddb, \"table\", nil, tc}\n}\n\ntype waitOnStoreTableCache struct {\n\treaders map[addr]io.ReaderAt\n\tmu sync.RWMutex\n\tstoreWG sync.WaitGroup\n}\n\nfunc (mtc *waitOnStoreTableCache) checkout(h addr) io.ReaderAt {\n\tmtc.mu.RLock()\n\tdefer mtc.mu.RUnlock()\n\treturn mtc.readers[h]\n}\n\nfunc (mtc *waitOnStoreTableCache) checkin(h addr) {}\n\nfunc (mtc *waitOnStoreTableCache) store(h addr, data io.Reader, size uint64) {\n\tdefer mtc.storeWG.Done()\n\tmtc.mu.Lock()\n\tdefer mtc.mu.Unlock()\n\tmtc.readers[h] = data.(io.ReaderAt)\n}\n\ntype failingFakeS3 struct {\n\t*fakeS3\n\tmu sync.Mutex\n\tnumSuccesses int\n}\n\nfunc (m *failingFakeS3) UploadPart(input *s3.UploadPartInput) (*s3.UploadPartOutput, error) {\n\tm.mu.Lock()\n\tdefer m.mu.Unlock()\n\tif m.numSuccesses > 0 {\n\t\tm.numSuccesses--\n\t\treturn m.fakeS3.UploadPart(input)\n\t}\n\treturn nil, mockAWSError(\"MalformedXML\")\n}\n\nfunc TestAWSTablePersisterDividePlan(t *testing.T) {\n\tassert := assert.New(t)\n\tminPartSize, maxPartSize := uint64(16), uint64(32)\n\ttooSmall := bytesToChunkSource([]byte(\"a\"))\n\tjustRight := bytesToChunkSource([]byte(\"123456789\"), []byte(\"abcdefghi\"))\n\tbigUns := [][]byte{make([]byte, maxPartSize-1), make([]byte, maxPartSize-1)}\n\tfor _, b := range bigUns {\n\t\trand.Read(b)\n\t}\n\ttooBig := bytesToChunkSource(bigUns...)\n\n\tsources := chunkSources{justRight, tooBig, tooSmall}\n\tplan := planConjoin(sources, &Stats{})\n\tcopies, manuals, _ := dividePlan(plan, minPartSize, maxPartSize)\n\n\tperTableDataSize := map[string]int64{}\n\tfor _, c := range copies {\n\t\tassert.True(minPartSize <= uint64(c.srcLen))\n\t\tassert.True(uint64(c.srcLen) <= maxPartSize)\n\t\ttotalSize := perTableDataSize[c.name]\n\t\ttotalSize += c.srcLen\n\t\tperTableDataSize[c.name] = totalSize\n\t}\n\tassert.Len(perTableDataSize, 2)\n\tassert.Contains(perTableDataSize, justRight.hash().String())\n\tassert.Contains(perTableDataSize, tooBig.hash().String())\n\tassert.EqualValues(calcChunkDataLen(justRight.index()), perTableDataSize[justRight.hash().String()])\n\tassert.EqualValues(calcChunkDataLen(tooBig.index()), perTableDataSize[tooBig.hash().String()])\n\n\tassert.Len(manuals, 1)\n\tassert.EqualValues(calcChunkDataLen(tooSmall.index()), manuals[0].dstEnd-manuals[0].dstStart)\n}\n\nfunc TestAWSTablePersisterCalcPartSizes(t *testing.T) {\n\tassert := assert.New(t)\n\tmin, max := uint64(8*1<<10), uint64(1+(16*1<<10))\n\n\ttestPartSizes := func(dataLen uint64) {\n\t\tlengths := splitOnMaxSize(dataLen, max)\n\t\tvar sum int64\n\t\tfor _, l := range lengths {\n\t\t\tassert.True(uint64(l) >= min)\n\t\t\tassert.True(uint64(l) <= max)\n\t\t\tsum += l\n\t\t}\n\t\tassert.EqualValues(dataLen, sum)\n\t}\n\n\ttestPartSizes(1 << 20)\n\ttestPartSizes(max + 1)\n\ttestPartSizes(10*max - 1)\n\ttestPartSizes(max + max/2)\n}\n\nfunc TestAWSTablePersisterConjoinAll(t *testing.T) {\n\ttargetPartSize := uint64(1024)\n\tminPartSize, maxPartSize := targetPartSize, 5*targetPartSize\n\tmaxItemSize, maxChunkCount := int(targetPartSize/2), uint32(4)\n\n\tic := newIndexCache(1024)\n\trl := make(chan struct{}, 8)\n\tdefer close(rl)\n\n\tnewPersister := func(s3svc s3svc, ddb *ddbTableStore) awsTablePersister {\n\t\treturn awsTablePersister{s3svc, \"bucket\", rl, nil, ddb, awsLimits{targetPartSize, minPartSize, maxPartSize, maxItemSize, maxChunkCount}, ic}\n\t}\n\n\tsmallChunks := [][]byte{}\n\trnd := rand.New(rand.NewSource(0))\n\tfor smallChunkTotal := uint64(0); smallChunkTotal <= uint64(minPartSize); {\n\t\tsmall := make([]byte, minPartSize/5)\n\t\trnd.Read(small)\n\t\tsrc := bytesToChunkSource(small)\n\t\tsmallChunks = append(smallChunks, small)\n\t\tsmallChunkTotal += calcChunkDataLen(src.index())\n\t}\n\n\tt.Run(\"Small\", func(t *testing.T) {\n\t\tmakeSources := func(s3p awsTablePersister, chunks [][]byte) (sources chunkSources) {\n\t\t\tfor i := 0; i < len(chunks); i++ {\n\t\t\t\tmt := newMemTable(uint64(2 * targetPartSize))\n\t\t\t\tmt.addChunk(computeAddr(chunks[i]), chunks[i])\n\t\t\t\tsources = append(sources, s3p.Persist(mt, nil, &Stats{}))\n\t\t\t}\n\t\t\treturn\n\t\t}\n\n\t\tt.Run(\"TotalUnderMinSize\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\t\t\ts3svc, ddb := makeFakeS3(t), makeFakeDTS(makeFakeDDB(t), nil)\n\t\t\ts3p := newPersister(s3svc, ddb)\n\n\t\t\tchunks := smallChunks[:len(smallChunks)-1]\n\t\t\tsources := makeSources(s3p, chunks)\n\t\t\tsrc := s3p.ConjoinAll(sources, &Stats{})\n\t\t\tassert.NotNil(ic.get(src.hash()))\n\n\t\t\tif assert.True(src.count() > 0) {\n\t\t\t\tif r := s3svc.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\t\tassertChunksInReader(chunks, r, assert)\n\t\t\t\t}\n\t\t\t}\n\t\t})\n\n\t\tt.Run(\"TotalOverMinSize\", func(t *testing.T) {\n\t\t\tassert := assert.New(t)\n\t\t\ts3svc, ddb := makeFakeS3(t), makeFakeDTS(makeFakeDDB(t), nil)\n\t\t\ts3p := newPersister(s3svc, ddb)\n\n\t\t\tsources := makeSources(s3p, smallChunks)\n\t\t\tsrc := s3p.ConjoinAll(sources, &Stats{})\n\t\t\tassert.NotNil(ic.get(src.hash()))\n\n\t\t\tif assert.True(src.count() > 0) {\n\t\t\t\tif r := s3svc.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\t\tassertChunksInReader(smallChunks, r, assert)\n\t\t\t\t}\n\t\t\t}\n\t\t})\n\t})\n\n\tbigUns1 := [][]byte{make([]byte, maxPartSize-1), make([]byte, maxPartSize-1)}\n\tbigUns2 := [][]byte{make([]byte, maxPartSize-1), make([]byte, maxPartSize-1)}\n\tfor _, bu := range [][][]byte{bigUns1, bigUns2} {\n\t\tfor _, b := range bu {\n\t\t\trand.Read(b)\n\t\t}\n\t}\n\n\tt.Run(\"AllOverMax\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\t\ts3svc, ddb := makeFakeS3(t), makeFakeDTS(makeFakeDDB(t), nil)\n\t\ts3p := newPersister(s3svc, ddb)\n\n\t\t// Make 2 chunk sources that each have >maxPartSize chunk data\n\t\tsources := make(chunkSources, 2)\n\t\tfor i, bu := range [][][]byte{bigUns1, bigUns2} {\n\t\t\tmt := newMemTable(uint64(2 * maxPartSize))\n\t\t\tfor _, b := range bu {\n\t\t\t\tmt.addChunk(computeAddr(b), b)\n\t\t\t}\n\t\t\tsources[i] = s3p.Persist(mt, nil, &Stats{})\n\t\t}\n\t\tsrc := s3p.ConjoinAll(sources, &Stats{})\n\t\tassert.NotNil(ic.get(src.hash()))\n\n\t\tif assert.True(src.count() > 0) {\n\t\t\tif r := s3svc.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\tassertChunksInReader(bigUns1, r, assert)\n\t\t\t\tassertChunksInReader(bigUns2, r, assert)\n\t\t\t}\n\t\t}\n\t})\n\n\tt.Run(\"SomeOverMax\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\t\ts3svc, ddb := makeFakeS3(t), makeFakeDTS(makeFakeDDB(t), nil)\n\t\ts3p := newPersister(s3svc, ddb)\n\n\t\t// Add one chunk source that has >maxPartSize data\n\t\tmtb := newMemTable(uint64(2 * maxPartSize))\n\t\tfor _, b := range bigUns1 {\n\t\t\tmtb.addChunk(computeAddr(b), b)\n\t\t}\n\n\t\t// Follow up with a chunk source where minPartSize < data size < maxPartSize\n\t\tmedChunks := make([][]byte, 2)\n\t\tmt := newMemTable(uint64(2 * maxPartSize))\n\t\tfor i := range medChunks {\n\t\t\tmedChunks[i] = make([]byte, minPartSize+1)\n\t\t\trand.Read(medChunks[i])\n\t\t\tmt.addChunk(computeAddr(medChunks[i]), medChunks[i])\n\t\t}\n\t\tsources := chunkSources{s3p.Persist(mt, nil, &Stats{}), s3p.Persist(mtb, nil, &Stats{})}\n\n\t\tsrc := s3p.ConjoinAll(sources, &Stats{})\n\t\tassert.NotNil(ic.get(src.hash()))\n\n\t\tif assert.True(src.count() > 0) {\n\t\t\tif r := s3svc.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\tassertChunksInReader(bigUns1, r, assert)\n\t\t\t\tassertChunksInReader(medChunks, r, assert)\n\t\t\t}\n\t\t}\n\t})\n\n\tt.Run(\"Mix\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\t\ts3svc, ddb := makeFakeS3(t), makeFakeDTS(makeFakeDDB(t), nil)\n\t\ts3p := newPersister(s3svc, ddb)\n\n\t\t// Start with small tables. Since total > minPartSize, will require more than one part to upload.\n\t\tsources := make(chunkSources, len(smallChunks))\n\t\tfor i := 0; i < len(smallChunks); i++ {\n\t\t\tmt := newMemTable(uint64(2 * targetPartSize))\n\t\t\tmt.addChunk(computeAddr(smallChunks[i]), smallChunks[i])\n\t\t\tsources[i] = s3p.Persist(mt, nil, &Stats{})\n\t\t}\n\n\t\t// Now, add a table with big chunks that will require more than one upload copy part.\n\t\tmt := newMemTable(uint64(2 * maxPartSize))\n\t\tfor _, b := range bigUns1 {\n\t\t\tmt.addChunk(computeAddr(b), b)\n\t\t}\n\t\tsources = append(sources, s3p.Persist(mt, nil, &Stats{}))\n\n\t\t// Last, some tables that should be directly upload-copyable\n\t\tmedChunks := make([][]byte, 2)\n\t\tmt = newMemTable(uint64(2 * maxPartSize))\n\t\tfor i := range medChunks {\n\t\t\tmedChunks[i] = make([]byte, minPartSize+1)\n\t\t\trand.Read(medChunks[i])\n\t\t\tmt.addChunk(computeAddr(medChunks[i]), medChunks[i])\n\t\t}\n\t\tsources = append(sources, s3p.Persist(mt, nil, &Stats{}))\n\n\t\tsrc := s3p.ConjoinAll(sources, &Stats{})\n\t\tassert.NotNil(ic.get(src.hash()))\n\n\t\tif assert.True(src.count() > 0) {\n\t\t\tif r := s3svc.readerForTable(src.hash()); assert.NotNil(r) {\n\t\t\t\tassertChunksInReader(smallChunks, r, assert)\n\t\t\t\tassertChunksInReader(bigUns1, r, assert)\n\t\t\t\tassertChunksInReader(medChunks, r, assert)\n\t\t\t}\n\t\t}\n\t})\n}\n\nfunc bytesToChunkSource(bs ...[]byte) chunkSource {\n\tsum := 0\n\tfor _, b := range bs {\n\t\tsum += len(b)\n\t}\n\tmaxSize := maxTableSize(uint64(len(bs)), uint64(sum))\n\tbuff := make([]byte, maxSize)\n\ttw := newTableWriter(buff, nil)\n\tfor _, b := range bs {\n\t\ttw.addChunk(computeAddr(b), b)\n\t}\n\ttableSize, name := tw.finish()\n\tdata := buff[:tableSize]\n\trdr := newTableReader(parseTableIndex(data), tableReaderAtFromBytes(data), fileBlockSize)\n\treturn chunkSourceAdapter{rdr, name}\n}\n" }, { "path": "go/nbs/block_store_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"crypto/rand\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst testMemTableSize = 1 << 8\n\nfunc TestBlockStoreSuite(t *testing.T) {\n\tsuite.Run(t, &BlockStoreSuite{})\n}\n\ntype BlockStoreSuite struct {\n\tsuite.Suite\n\tdir string\n\tstore *NomsBlockStore\n\tputCountFn func() int\n}\n\nfunc (suite *BlockStoreSuite) SetupTest() {\n\tvar err error\n\tsuite.dir, err = ioutil.TempDir(\"\", \"\")\n\tsuite.NoError(err)\n\tsuite.store = NewLocalStore(suite.dir, testMemTableSize)\n\tsuite.putCountFn = func() int {\n\t\treturn int(suite.store.putCount)\n\t}\n}\n\nfunc (suite *BlockStoreSuite) TearDownTest() {\n\tsuite.store.Close()\n\tos.RemoveAll(suite.dir)\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStoreMissingDir() {\n\tnewDir := filepath.Join(suite.dir, \"does-not-exist\")\n\tsuite.Panics(func() { NewLocalStore(newDir, testMemTableSize) })\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStoreNotDir() {\n\texistingFile := filepath.Join(suite.dir, \"path-exists-but-is-a-file\")\n\tos.Create(existingFile)\n\tsuite.Panics(func() { NewLocalStore(existingFile, testMemTableSize) })\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStorePut() {\n\tinput := []byte(\"abc\")\n\tc := chunks.NewChunk(input)\n\tsuite.store.Put(c)\n\th := c.Hash()\n\n\t// See http://www.di-mgt.com.au/sha_testvectors.html\n\tsuite.Equal(\"rmnjb8cjc5tblj21ed4qs821649eduie\", h.String())\n\n\tsuite.store.Commit(h, suite.store.Root()) // Commit writes\n\n\t// And reading it via the API should work...\n\tassertInputInStore(input, h, suite.store, suite.Assert())\n\tif suite.putCountFn != nil {\n\t\tsuite.Equal(1, suite.putCountFn())\n\t}\n\n\t// Re-writing the same data should cause a second put\n\tc = chunks.NewChunk(input)\n\tsuite.store.Put(c)\n\tsuite.Equal(h, c.Hash())\n\tassertInputInStore(input, h, suite.store, suite.Assert())\n\tsuite.store.Commit(h, suite.store.Root()) // Commit writes\n\n\tif suite.putCountFn != nil {\n\t\tsuite.Equal(2, suite.putCountFn())\n\t}\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStorePutMany() {\n\tinput1, input2 := []byte(\"abc\"), []byte(\"def\")\n\tc1, c2 := chunks.NewChunk(input1), chunks.NewChunk(input2)\n\tsuite.store.Put(c1)\n\tsuite.store.Put(c2)\n\n\tsuite.store.Commit(c1.Hash(), suite.store.Root()) // Commit writes\n\n\t// And reading it via the API should work...\n\tassertInputInStore(input1, c1.Hash(), suite.store, suite.Assert())\n\tassertInputInStore(input2, c2.Hash(), suite.store, suite.Assert())\n\tif suite.putCountFn != nil {\n\t\tsuite.Equal(2, suite.putCountFn())\n\t}\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStoreStatsSummary() {\n\tinput1, input2 := []byte(\"abc\"), []byte(\"def\")\n\tc1, c2 := chunks.NewChunk(input1), chunks.NewChunk(input2)\n\tsuite.store.Put(c1)\n\tsuite.store.Put(c2)\n\n\tsuite.store.Commit(c1.Hash(), suite.store.Root()) // Commit writes\n\n\tsummary := suite.store.StatsSummary()\n\tsuite.Contains(summary, c1.Hash().String())\n\tsuite.NotEqual(\"Unsupported\", summary)\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStorePutMoreThanMemTable() {\n\tinput1, input2 := make([]byte, testMemTableSize/2+1), make([]byte, testMemTableSize/2+1)\n\trand.Read(input1)\n\trand.Read(input2)\n\tc1, c2 := chunks.NewChunk(input1), chunks.NewChunk(input2)\n\tsuite.store.Put(c1)\n\tsuite.store.Put(c2)\n\n\tsuite.store.Commit(c1.Hash(), suite.store.Root()) // Commit writes\n\n\t// And reading it via the API should work...\n\tassertInputInStore(input1, c1.Hash(), suite.store, suite.Assert())\n\tassertInputInStore(input2, c2.Hash(), suite.store, suite.Assert())\n\tif suite.putCountFn != nil {\n\t\tsuite.Equal(2, suite.putCountFn())\n\t}\n\tsuite.Len(suite.store.tables.ToSpecs(), 2)\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStoreGetMany() {\n\tinputs := [][]byte{make([]byte, testMemTableSize/2+1), make([]byte, testMemTableSize/2+1), []byte(\"abc\")}\n\trand.Read(inputs[0])\n\trand.Read(inputs[1])\n\tchnx := make([]chunks.Chunk, len(inputs))\n\tfor i, data := range inputs {\n\t\tchnx[i] = chunks.NewChunk(data)\n\t\tsuite.store.Put(chnx[i])\n\t}\n\tsuite.store.Commit(chnx[0].Hash(), suite.store.Root()) // Commit writes\n\n\thashes := make(hash.HashSlice, len(chnx))\n\tfor i, c := range chnx {\n\t\thashes[i] = c.Hash()\n\t}\n\n\tchunkChan := make(chan *chunks.Chunk, len(hashes))\n\tsuite.store.GetMany(hashes.HashSet(), chunkChan)\n\tclose(chunkChan)\n\n\tfound := make(hash.HashSlice, 0)\n\tfor c := range chunkChan {\n\t\tfound = append(found, c.Hash())\n\t}\n\n\tsort.Sort(found)\n\tsort.Sort(hashes)\n\tsuite.True(found.Equals(hashes))\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStoreHasMany() {\n\tchnx := []chunks.Chunk{\n\t\tchunks.NewChunk([]byte(\"abc\")),\n\t\tchunks.NewChunk([]byte(\"def\")),\n\t}\n\tfor _, c := range chnx {\n\t\tsuite.store.Put(c)\n\t}\n\tsuite.store.Commit(chnx[0].Hash(), suite.store.Root()) // Commit writes\n\tnotPresent := chunks.NewChunk([]byte(\"ghi\")).Hash()\n\n\thashes := hash.NewHashSet(chnx[0].Hash(), chnx[1].Hash(), notPresent)\n\tabsent := suite.store.HasMany(hashes)\n\n\tsuite.Len(absent, 1)\n\tfor _, c := range chnx {\n\t\tsuite.False(absent.Has(c.Hash()), \"%s present in %v\", c.Hash(), absent)\n\t}\n\tsuite.True(absent.Has(notPresent))\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStoreExtractChunks() {\n\tinput1, input2 := make([]byte, testMemTableSize/2+1), make([]byte, testMemTableSize/2+1)\n\trand.Read(input1)\n\trand.Read(input2)\n\tchnx := []chunks.Chunk{chunks.NewChunk(input1), chunks.NewChunk(input2)}\n\tfor _, c := range chnx {\n\t\tsuite.store.Put(c)\n\t}\n\n\tchunkChan := make(chan *chunks.Chunk)\n\tgo func() { suite.store.extractChunks(chunkChan); close(chunkChan) }()\n\ti := 0\n\tfor c := range chunkChan {\n\t\tsuite.Equal(chnx[i].Data(), c.Data())\n\t\tsuite.Equal(chnx[i].Hash(), c.Hash())\n\t\ti++\n\t}\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStoreFlushOptimisticLockFail() {\n\tinput1, input2 := []byte(\"abc\"), []byte(\"def\")\n\tc1, c2 := chunks.NewChunk(input1), chunks.NewChunk(input2)\n\troot := suite.store.Root()\n\n\tinterloper := NewLocalStore(suite.dir, testMemTableSize)\n\tinterloper.Put(c1)\n\tsuite.True(interloper.Commit(interloper.Root(), interloper.Root()))\n\n\tsuite.store.Put(c2)\n\tsuite.True(suite.store.Commit(suite.store.Root(), suite.store.Root()))\n\n\t// Reading c2 via the API should work...\n\tassertInputInStore(input2, c2.Hash(), suite.store, suite.Assert())\n\t// And so should reading c1 via the API\n\tassertInputInStore(input1, c1.Hash(), suite.store, suite.Assert())\n\n\tsuite.True(interloper.Commit(c1.Hash(), interloper.Root())) // Commit root\n\n\t// Updating from stale root should fail...\n\tsuite.False(suite.store.Commit(c2.Hash(), root))\n\t// ...but new root should succeed\n\tsuite.True(suite.store.Commit(c2.Hash(), suite.store.Root()))\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStoreRebaseOnNoOpFlush() {\n\tinput1 := []byte(\"abc\")\n\tc1 := chunks.NewChunk(input1)\n\n\tinterloper := NewLocalStore(suite.dir, testMemTableSize)\n\tinterloper.Put(c1)\n\tsuite.True(interloper.Commit(c1.Hash(), interloper.Root()))\n\n\tsuite.False(suite.store.Has(c1.Hash()))\n\tsuite.Equal(hash.Hash{}, suite.store.Root())\n\t// Should Rebase, even though there's no work to do.\n\tsuite.True(suite.store.Commit(suite.store.Root(), suite.store.Root()))\n\n\t// Reading c1 via the API should work\n\tassertInputInStore(input1, c1.Hash(), suite.store, suite.Assert())\n\tsuite.True(suite.store.Has(c1.Hash()))\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStorePutWithRebase() {\n\tinput1, input2 := []byte(\"abc\"), []byte(\"def\")\n\tc1, c2 := chunks.NewChunk(input1), chunks.NewChunk(input2)\n\troot := suite.store.Root()\n\n\tinterloper := NewLocalStore(suite.dir, testMemTableSize)\n\tinterloper.Put(c1)\n\tsuite.True(interloper.Commit(interloper.Root(), interloper.Root()))\n\n\tsuite.store.Put(c2)\n\n\t// Reading c2 via the API should work pre-rebase\n\tassertInputInStore(input2, c2.Hash(), suite.store, suite.Assert())\n\t// Shouldn't have c1 yet.\n\tsuite.False(suite.store.Has(c1.Hash()))\n\n\tsuite.store.Rebase()\n\n\t// Reading c2 via the API should work post-rebase\n\tassertInputInStore(input2, c2.Hash(), suite.store, suite.Assert())\n\t// And so should reading c1 via the API\n\tassertInputInStore(input1, c1.Hash(), suite.store, suite.Assert())\n\n\t// Commit interloper root\n\tsuite.True(interloper.Commit(c1.Hash(), interloper.Root()))\n\n\t// suite.store should still have its initial root\n\tsuite.EqualValues(root, suite.store.Root())\n\tsuite.store.Rebase()\n\n\t// Rebase grabbed the new root, so updating should now succeed!\n\tsuite.True(suite.store.Commit(c2.Hash(), suite.store.Root()))\n\n\t// Interloper shouldn't see c2 yet....\n\tsuite.False(interloper.Has(c2.Hash()))\n\tinterloper.Rebase()\n\t// ...but post-rebase it must\n\tassertInputInStore(input2, c2.Hash(), interloper, suite.Assert())\n}\n\nfunc TestBlockStoreConjoinOnCommit(t *testing.T) {\n\tstats := &Stats{}\n\tassertContainAll := func(t *testing.T, store chunks.ChunkStore, srcs ...chunkSource) {\n\t\trdrs := make(chunkReaderGroup, len(srcs))\n\t\tfor i, src := range srcs {\n\t\t\trdrs[i] = src\n\t\t}\n\t\tchunkChan := make(chan extractRecord, rdrs.count())\n\t\trdrs.extract(chunkChan)\n\t\tclose(chunkChan)\n\n\t\tfor rec := range chunkChan {\n\t\t\tassert.True(t, store.Has(hash.Hash(rec.a)))\n\t\t}\n\t}\n\n\tmakeManifestManager := func(m manifest) manifestManager {\n\t\treturn manifestManager{m, newManifestCache(0), newManifestLocks()}\n\t}\n\n\tnewChunk := chunks.NewChunk([]byte(\"gnu\"))\n\n\tt.Run(\"NoConjoin\", func(t *testing.T) {\n\t\tmm := makeManifestManager(&fakeManifest{})\n\t\tp := newFakeTablePersister()\n\t\tc := &fakeConjoiner{}\n\n\t\tsmallTableStore := newNomsBlockStore(mm, p, c, testMemTableSize)\n\n\t\troot := smallTableStore.Root()\n\t\tsmallTableStore.Put(newChunk)\n\t\tassert.True(t, smallTableStore.Commit(newChunk.Hash(), root))\n\t\tassert.True(t, smallTableStore.Has(newChunk.Hash()))\n\t})\n\n\tmakeCanned := func(conjoinees, keepers []tableSpec, p tablePersister) cannedConjoin {\n\t\tsrcs := chunkSources{}\n\t\tfor _, sp := range conjoinees {\n\t\t\tsrcs = append(srcs, p.Open(sp.name, sp.chunkCount, nil))\n\t\t}\n\t\tconjoined := p.ConjoinAll(srcs, stats)\n\t\tcannedSpecs := []tableSpec{{conjoined.hash(), conjoined.count()}}\n\t\treturn cannedConjoin{true, append(cannedSpecs, keepers...)}\n\t}\n\n\tt.Run(\"ConjoinSuccess\", func(t *testing.T) {\n\t\tfm := &fakeManifest{}\n\t\tp := newFakeTablePersister()\n\n\t\tsrcs := makeTestSrcs([]uint32{1, 1, 3, 7}, p)\n\t\tupstream := toSpecs(srcs)\n\t\tfm.set(constants.NomsVersion, computeAddr([]byte{0xbe}), hash.Of([]byte{0xef}), upstream)\n\t\tc := &fakeConjoiner{\n\t\t\t[]cannedConjoin{makeCanned(upstream[:2], upstream[2:], p)},\n\t\t}\n\n\t\tsmallTableStore := newNomsBlockStore(makeManifestManager(fm), p, c, testMemTableSize)\n\n\t\troot := smallTableStore.Root()\n\t\tsmallTableStore.Put(newChunk)\n\t\tassert.True(t, smallTableStore.Commit(newChunk.Hash(), root))\n\t\tassert.True(t, smallTableStore.Has(newChunk.Hash()))\n\t\tassertContainAll(t, smallTableStore, srcs...)\n\t})\n\n\tt.Run(\"ConjoinRetry\", func(t *testing.T) {\n\t\tfm := &fakeManifest{}\n\t\tp := newFakeTablePersister()\n\n\t\tsrcs := makeTestSrcs([]uint32{1, 1, 3, 7, 13}, p)\n\t\tupstream := toSpecs(srcs)\n\t\tfm.set(constants.NomsVersion, computeAddr([]byte{0xbe}), hash.Of([]byte{0xef}), upstream)\n\t\tc := &fakeConjoiner{\n\t\t\t[]cannedConjoin{\n\t\t\t\tmakeCanned(upstream[:2], upstream[2:], p),\n\t\t\t\tmakeCanned(upstream[:4], upstream[4:], p),\n\t\t\t},\n\t\t}\n\n\t\tsmallTableStore := newNomsBlockStore(makeManifestManager(fm), p, c, testMemTableSize)\n\n\t\troot := smallTableStore.Root()\n\t\tsmallTableStore.Put(newChunk)\n\t\tassert.True(t, smallTableStore.Commit(newChunk.Hash(), root))\n\t\tassert.True(t, smallTableStore.Has(newChunk.Hash()))\n\t\tassertContainAll(t, smallTableStore, srcs...)\n\t})\n}\n\ntype cannedConjoin struct {\n\tshould bool\n\tspecs []tableSpec // Must name tables that are already persisted\n}\n\ntype fakeConjoiner struct {\n\tcanned []cannedConjoin\n}\n\nfunc (fc *fakeConjoiner) ConjoinRequired(ts tableSet) bool {\n\tif len(fc.canned) == 0 {\n\t\treturn false\n\t}\n\treturn fc.canned[0].should\n}\n\nfunc (fc *fakeConjoiner) Conjoin(upstream manifestContents, mm manifestUpdater, p tablePersister, stats *Stats) manifestContents {\n\td.PanicIfTrue(len(fc.canned) == 0)\n\tcanned := fc.canned[0]\n\tfc.canned = fc.canned[1:]\n\n\tnewContents := manifestContents{\n\t\tvers: constants.NomsVersion,\n\t\troot: upstream.root,\n\t\tspecs: canned.specs,\n\t\tlock: generateLockHash(upstream.root, canned.specs),\n\t}\n\tupstream = mm.Update(upstream.lock, newContents, stats, nil)\n\td.PanicIfFalse(upstream.lock == newContents.lock)\n\treturn upstream\n}\n\nfunc assertInputInStore(input []byte, h hash.Hash, s chunks.ChunkStore, assert *assert.Assertions) {\n\tc := s.Get(h)\n\tassert.False(c.IsEmpty(), \"Shouldn't get empty chunk for %s\", h.String())\n\tassert.Zero(bytes.Compare(input, c.Data()), \"%s != %s\", string(input), string(c.Data()))\n}\n\nfunc (suite *BlockStoreSuite) TestChunkStoreGetNonExisting() {\n\th := hash.Parse(\"11111111111111111111111111111111\")\n\tc := suite.store.Get(h)\n\tsuite.True(c.IsEmpty())\n}\n" }, { "path": "go/nbs/cache.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\nconst (\n\tdefaultCacheMemTableSize uint64 = 1 << 27 // 128MiB\n)\n\nfunc NewCache() *NomsBlockCache {\n\tdir, err := ioutil.TempDir(\"\", \"\")\n\td.PanicIfError(err)\n\tstore := NewLocalStore(dir, defaultCacheMemTableSize)\n\td.Chk.NoError(err, \"opening put cache in %s\", dir)\n\treturn &NomsBlockCache{store, dir}\n}\n\n// NomsBlockCache holds Chunks, allowing them to be retrieved by hash or enumerated in hash order.\ntype NomsBlockCache struct {\n\tchunks *NomsBlockStore\n\tdbDir string\n}\n\n// Insert stores c in the cache.\nfunc (nbc *NomsBlockCache) Insert(c chunks.Chunk) {\n\td.PanicIfFalse(nbc.chunks.addChunk(addr(c.Hash()), c.Data()))\n}\n\n// Has checks if the chunk referenced by hash is in the cache.\nfunc (nbc *NomsBlockCache) Has(hash hash.Hash) bool {\n\treturn nbc.chunks.Has(hash)\n}\n\n// HasMany returns a set containing the members of hashes present in the\n// cache.\nfunc (nbc *NomsBlockCache) HasMany(hashes hash.HashSet) hash.HashSet {\n\treturn nbc.chunks.HasMany(hashes)\n}\n\n// Get retrieves the chunk referenced by hash. If the chunk is not present,\n// Get returns the empty Chunk.\nfunc (nbc *NomsBlockCache) Get(hash hash.Hash) chunks.Chunk {\n\treturn nbc.chunks.Get(hash)\n}\n\n// GetMany gets the Chunks with |hashes| from the store. On return,\n// |foundChunks| will have been fully sent all chunks which have been\n// found. Any non-present chunks will silently be ignored.\nfunc (nbc *NomsBlockCache) GetMany(hashes hash.HashSet, foundChunks chan *chunks.Chunk) {\n\tnbc.chunks.GetMany(hashes, foundChunks)\n}\n\n// ExtractChunks writes the entire contents of the cache to chunkChan. The\n// chunks are extracted in insertion order.\nfunc (nbc *NomsBlockCache) ExtractChunks(chunkChan chan *chunks.Chunk) {\n\tnbc.chunks.extractChunks(chunkChan)\n}\n\n// Count returns the number of items in the cache.\nfunc (nbc *NomsBlockCache) Count() uint32 {\n\treturn nbc.chunks.Count()\n}\n\n// Destroy drops the cache and deletes any backing storage.\nfunc (nbc *NomsBlockCache) Destroy() error {\n\td.Chk.NoError(nbc.chunks.Close())\n\treturn os.RemoveAll(nbc.dbDir)\n}\n" }, { "path": "go/nbs/conjoiner.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"sort\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype conjoiner interface {\n\t// ConjoinRequired tells the caller whether or not it's time to request a\n\t// Conjoin, based upon the contents of |ts| and the conjoiner\n\t// implementation's policy.\n\tConjoinRequired(ts tableSet) bool\n\n\t// Conjoin attempts to use |p| to conjoin some number of tables referenced\n\t// by |upstream|, allowing it to update |mm| with a new, smaller, set of tables\n\t// that references precisely the same set of chunks. Conjoin() may not\n\t// actually conjoin any upstream tables, usually because some out-of-\n\t// process actor has already landed a conjoin of its own. Callers must\n\t// handle this, likely by rebasing against upstream and re-evaluating the\n\t// situation.\n\tConjoin(upstream manifestContents, mm manifestUpdater, p tablePersister, stats *Stats) manifestContents\n}\n\ntype inlineConjoiner struct {\n\tmaxTables int\n}\n\nfunc (c inlineConjoiner) ConjoinRequired(ts tableSet) bool {\n\treturn ts.Size() > c.maxTables\n}\n\nfunc (c inlineConjoiner) Conjoin(upstream manifestContents, mm manifestUpdater, p tablePersister, stats *Stats) manifestContents {\n\treturn conjoin(upstream, mm, p, stats)\n}\n\nfunc conjoin(upstream manifestContents, mm manifestUpdater, p tablePersister, stats *Stats) manifestContents {\n\tvar conjoined tableSpec\n\tvar conjoinees, keepers []tableSpec\n\n\tfor {\n\t\tif conjoinees == nil {\n\t\t\tconjoined, conjoinees, keepers = conjoinTables(p, upstream.specs, stats)\n\t\t}\n\n\t\tspecs := append(make([]tableSpec, 0, len(keepers)+1), conjoined)\n\t\tspecs = append(specs, keepers...)\n\n\t\tnewContents := manifestContents{\n\t\t\tvers: constants.NomsVersion,\n\t\t\troot: upstream.root,\n\t\t\tlock: generateLockHash(upstream.root, specs),\n\t\t\tspecs: specs,\n\t\t}\n\t\tupstream = mm.Update(upstream.lock, newContents, stats, nil)\n\n\t\tif newContents.lock == upstream.lock {\n\t\t\treturn upstream // Success!\n\t\t}\n\t\t// Optimistic lock failure. Someone else moved to the root, the set of tables, or both out from under us.\n\t\t// If we can re-use the conjoin we already performed, we want to try again. Currently, we will only do so if ALL conjoinees are still present upstream. If we can't re-use...then someone else almost certainly landed a conjoin upstream. In this case, bail and let clients ask again if they think they still can't proceed.\n\t\tconjoineeSet := map[addr]struct{}{}\n\t\tupstreamNames := map[addr]struct{}{}\n\t\tfor _, spec := range upstream.specs {\n\t\t\tupstreamNames[spec.name] = struct{}{}\n\t\t}\n\t\tfor _, c := range conjoinees {\n\t\t\tif _, present := upstreamNames[c.name]; !present {\n\t\t\t\treturn upstream // Bail!\n\t\t\t}\n\t\t\tconjoineeSet[c.name] = struct{}{}\n\t\t}\n\n\t\t// Filter conjoinees out of upstream.specs to generate new set of keepers\n\t\tkeepers = make([]tableSpec, 0, len(upstream.specs)-len(conjoinees))\n\t\tfor _, spec := range upstream.specs {\n\t\t\tif _, present := conjoineeSet[spec.name]; !present {\n\t\t\t\tkeepers = append(keepers, spec)\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc conjoinTables(p tablePersister, upstream []tableSpec, stats *Stats) (conjoined tableSpec, conjoinees, keepers []tableSpec) {\n\t// Open all the upstream tables concurrently\n\tsources := make(chunkSources, len(upstream))\n\twg := sync.WaitGroup{}\n\tfor i, spec := range upstream {\n\t\twg.Add(1)\n\t\tgo func(idx int, spec tableSpec) {\n\t\t\tsources[idx] = p.Open(spec.name, spec.chunkCount, stats)\n\t\t\twg.Done()\n\t\t}(i, spec)\n\t\ti++\n\t}\n\twg.Wait()\n\n\tt1 := time.Now()\n\n\ttoConjoin, toKeep := chooseConjoinees(sources)\n\tconjoinedSrc := p.ConjoinAll(toConjoin, stats)\n\n\tstats.ConjoinLatency.SampleTimeSince(t1)\n\tstats.TablesPerConjoin.SampleLen(len(toConjoin))\n\tstats.ChunksPerConjoin.Sample(uint64(conjoinedSrc.count()))\n\n\treturn tableSpec{conjoinedSrc.hash(), conjoinedSrc.count()}, toSpecs(toConjoin), toSpecs(toKeep)\n}\n\n// Current approach is to choose the smallest N tables which, when removed and replaced with the conjoinment, will leave the conjoinment as the smallest table.\nfunc chooseConjoinees(upstream chunkSources) (toConjoin, toKeep chunkSources) {\n\tsortedUpstream := make(chunkSources, len(upstream))\n\tcopy(sortedUpstream, upstream)\n\tsort.Sort(chunkSourcesByAscendingCount(sortedUpstream))\n\n\tpartition := 2\n\tsum := sortedUpstream[0].count() + sortedUpstream[1].count()\n\tfor partition < len(sortedUpstream) && sum > sortedUpstream[partition].count() {\n\t\tsum += sortedUpstream[partition].count()\n\t\tpartition++\n\t}\n\n\treturn sortedUpstream[:partition], sortedUpstream[partition:]\n}\n\nfunc toSpecs(srcs chunkSources) []tableSpec {\n\tspecs := make([]tableSpec, len(srcs))\n\tfor i, src := range srcs {\n\t\td.PanicIfFalse(src.count() > 0)\n\t\tspecs[i] = tableSpec{src.hash(), src.count()}\n\t}\n\treturn specs\n}\n" }, { "path": "go/nbs/conjoiner_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"encoding/binary\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype tableSpecsByAscendingCount []tableSpec\n\nfunc (ts tableSpecsByAscendingCount) Len() int { return len(ts) }\nfunc (ts tableSpecsByAscendingCount) Less(i, j int) bool {\n\ttsI, tsJ := ts[i], ts[j]\n\tif tsI.chunkCount == tsJ.chunkCount {\n\t\treturn bytes.Compare(tsI.name[:], tsJ.name[:]) < 0\n\t}\n\treturn tsI.chunkCount < tsJ.chunkCount\n}\nfunc (ts tableSpecsByAscendingCount) Swap(i, j int) { ts[i], ts[j] = ts[j], ts[i] }\n\nfunc makeTestSrcs(tableSizes []uint32, p tablePersister) (srcs chunkSources) {\n\tcount := uint32(0)\n\tnextChunk := func() (chunk []byte) {\n\t\tchunk = make([]byte, 4)\n\t\tbinary.BigEndian.PutUint32(chunk, count)\n\t\tcount++\n\t\treturn chunk\n\t}\n\n\tfor _, s := range tableSizes {\n\t\tmt := newMemTable(testMemTableSize)\n\t\tfor i := uint32(0); i < s; i++ {\n\t\t\tc := nextChunk()\n\t\t\tmt.addChunk(computeAddr(c), c)\n\t\t}\n\t\tsrcs = append(srcs, p.Persist(mt, nil, &Stats{}))\n\t}\n\treturn\n}\n\nfunc TestConjoin(t *testing.T) {\n\t// Makes a tableSet with len(tableSizes) upstream tables containing tableSizes[N] unique chunks\n\tmakeTestTableSpecs := func(tableSizes []uint32, p tablePersister) (specs []tableSpec) {\n\t\tfor _, src := range makeTestSrcs(tableSizes, p) {\n\t\t\tspecs = append(specs, tableSpec{src.hash(), src.count()})\n\t\t}\n\t\treturn\n\t}\n\n\t// Returns the chunk counts of the tables in ts.compacted & ts.upstream in ascending order\n\tgetSortedSizes := func(specs []tableSpec) (sorted []uint32) {\n\t\tall := append([]tableSpec{}, specs...)\n\t\tsort.Sort(tableSpecsByAscendingCount(all))\n\t\tfor _, ts := range all {\n\t\t\tsorted = append(sorted, ts.chunkCount)\n\t\t}\n\t\treturn\n\t}\n\n\tassertContainAll := func(t *testing.T, p tablePersister, expect, actual []tableSpec) {\n\t\topen := func(specs []tableSpec) (srcs chunkReaderGroup) {\n\t\t\tfor _, sp := range specs {\n\t\t\t\tsrcs = append(srcs, p.Open(sp.name, sp.chunkCount, nil))\n\t\t\t}\n\t\t\treturn\n\t\t}\n\t\texpectSrcs, actualSrcs := open(expect), open(actual)\n\t\tchunkChan := make(chan extractRecord, expectSrcs.count())\n\t\texpectSrcs.extract(chunkChan)\n\t\tclose(chunkChan)\n\n\t\tfor rec := range chunkChan {\n\t\t\tassert.True(t, actualSrcs.has(rec.a))\n\t\t}\n\t}\n\n\tsetup := func(lock addr, root hash.Hash, sizes []uint32) (fm *fakeManifest, p tablePersister, upstream manifestContents) {\n\t\tp = newFakeTablePersister()\n\t\tfm = &fakeManifest{}\n\t\tfm.set(constants.NomsVersion, lock, root, makeTestTableSpecs(sizes, p))\n\t\t_, upstream = fm.ParseIfExists(nil, nil)\n\t\treturn\n\t}\n\n\ttc := []struct {\n\t\tname string\n\t\tprecompact []uint32\n\t\tpostcompact []uint32\n\t}{\n\t\t{\"uniform\", []uint32{1, 1, 1, 1, 1}, []uint32{5}},\n\t\t{\"all but last\", []uint32{1, 1, 1, 1, 5}, []uint32{4, 5}},\n\t\t{\"all\", []uint32{5, 5, 5}, []uint32{15}},\n\t\t{\"first four\", []uint32{5, 6, 10, 11, 35, 64}, []uint32{32, 35, 64}},\n\t\t{\"log, first two\", []uint32{1, 2, 4, 8, 16, 32, 64}, []uint32{3, 4, 8, 16, 32, 64}},\n\t\t{\"log, all\", []uint32{2, 3, 4, 8, 16, 32, 64}, []uint32{129}},\n\t}\n\n\tstats := &Stats{}\n\tstartLock, startRoot := computeAddr([]byte(\"lock\")), hash.Of([]byte(\"root\"))\n\tt.Run(\"Success\", func(t *testing.T) {\n\t\t// Compact some tables, no one interrupts\n\t\tfor _, c := range tc {\n\t\t\tt.Run(c.name, func(t *testing.T) {\n\t\t\t\tfm, p, upstream := setup(startLock, startRoot, c.precompact)\n\n\t\t\t\tconjoin(upstream, fm, p, stats)\n\t\t\t\texists, newUpstream := fm.ParseIfExists(stats, nil)\n\t\t\t\tassert.True(t, exists)\n\t\t\t\tassert.Equal(t, c.postcompact, getSortedSizes(newUpstream.specs))\n\t\t\t\tassertContainAll(t, p, upstream.specs, newUpstream.specs)\n\t\t\t})\n\t\t}\n\t})\n\n\tt.Run(\"Retry\", func(t *testing.T) {\n\t\t// Compact some tables, interloper slips in a new table\n\t\tmakeExtra := func(p tablePersister) tableSpec {\n\t\t\tmt := newMemTable(testMemTableSize)\n\t\t\tdata := []byte{0xde, 0xad}\n\t\t\tmt.addChunk(computeAddr(data), data)\n\t\t\tsrc := p.Persist(mt, nil, &Stats{})\n\t\t\treturn tableSpec{src.hash(), src.count()}\n\t\t}\n\t\tfor _, c := range tc {\n\t\t\tt.Run(c.name, func(t *testing.T) {\n\t\t\t\tfm, p, upstream := setup(startLock, startRoot, c.precompact)\n\n\t\t\t\tnewTable := makeExtra(p)\n\t\t\t\tu := updatePreemptManifest{fm, func() {\n\t\t\t\t\tspecs := append([]tableSpec{}, upstream.specs...)\n\t\t\t\t\tfm.set(constants.NomsVersion, computeAddr([]byte(\"lock2\")), startRoot, append(specs, newTable))\n\t\t\t\t}}\n\t\t\t\tconjoin(upstream, u, p, stats)\n\t\t\t\texists, newUpstream := fm.ParseIfExists(stats, nil)\n\t\t\t\tassert.True(t, exists)\n\t\t\t\tassert.Equal(t, append([]uint32{1}, c.postcompact...), getSortedSizes(newUpstream.specs))\n\t\t\t\tassertContainAll(t, p, append(upstream.specs, newTable), newUpstream.specs)\n\t\t\t})\n\t\t}\n\t})\n\n\tt.Run(\"TablesDroppedUpstream\", func(t *testing.T) {\n\t\t// Interloper drops some compactees\n\t\tfor _, c := range tc {\n\t\t\tt.Run(c.name, func(t *testing.T) {\n\t\t\t\tfm, p, upstream := setup(startLock, startRoot, c.precompact)\n\n\t\t\t\tu := updatePreemptManifest{fm, func() {\n\t\t\t\t\tfm.set(constants.NomsVersion, computeAddr([]byte(\"lock2\")), startRoot, upstream.specs[1:])\n\t\t\t\t}}\n\t\t\t\tconjoin(upstream, u, p, stats)\n\t\t\t\texists, newUpstream := fm.ParseIfExists(stats, nil)\n\t\t\t\tassert.True(t, exists)\n\t\t\t\tassert.Equal(t, c.precompact[1:], getSortedSizes(newUpstream.specs))\n\t\t\t})\n\t\t}\n\t})\n}\n\ntype updatePreemptManifest struct {\n\tmanifest\n\tpreUpdate func()\n}\n\nfunc (u updatePreemptManifest) Update(lastLock addr, newContents manifestContents, stats *Stats, writeHook func()) manifestContents {\n\tif u.preUpdate != nil {\n\t\tu.preUpdate()\n\t}\n\treturn u.manifest.Update(lastLock, newContents, stats, writeHook)\n}\n" }, { "path": "go/nbs/dynamo_fake_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/aws/aws-sdk-go/aws\"\n\t\"github.com/aws/aws-sdk-go/service/dynamodb\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype fakeDDB struct {\n\tdata map[string]interface{}\n\tt *testing.T\n\tnumPuts, numGets int\n}\n\ntype record struct {\n\tlock, root []byte\n\tvers, specs string\n}\n\nfunc makeFakeDDB(t *testing.T) *fakeDDB {\n\treturn &fakeDDB{\n\t\tdata: map[string]interface{}{},\n\t\tt: t,\n\t}\n}\n\nfunc (m *fakeDDB) readerForTable(name addr) chunkReader {\n\tif i, present := m.data[fmtTableName(name)]; present {\n\t\tbuff, ok := i.([]byte)\n\t\tassert.True(m.t, ok)\n\t\treturn newTableReader(parseTableIndex(buff), tableReaderAtFromBytes(buff), fileBlockSize)\n\t}\n\treturn nil\n}\n\nfunc (m *fakeDDB) GetItem(input *dynamodb.GetItemInput) (*dynamodb.GetItemOutput, error) {\n\tkey := input.Key[dbAttr].S\n\tassert.NotNil(m.t, key, \"key should have been a String: %+v\", input.Key[dbAttr])\n\n\titem := map[string]*dynamodb.AttributeValue{}\n\tif e, present := m.data[*key]; present {\n\t\titem[dbAttr] = &dynamodb.AttributeValue{S: key}\n\t\tswitch e := e.(type) {\n\t\tcase record:\n\t\t\titem[nbsVersAttr] = &dynamodb.AttributeValue{S: aws.String(StorageVersion)}\n\t\t\titem[versAttr] = &dynamodb.AttributeValue{S: aws.String(e.vers)}\n\t\t\titem[rootAttr] = &dynamodb.AttributeValue{B: e.root}\n\t\t\titem[lockAttr] = &dynamodb.AttributeValue{B: e.lock}\n\t\t\tif e.specs != \"\" {\n\t\t\t\titem[tableSpecsAttr] = &dynamodb.AttributeValue{S: aws.String(e.specs)}\n\t\t\t}\n\t\tcase []byte:\n\t\t\titem[dataAttr] = &dynamodb.AttributeValue{B: e}\n\t\t}\n\t}\n\tm.numGets++\n\treturn &dynamodb.GetItemOutput{Item: item}, nil\n}\n\nfunc (m *fakeDDB) putRecord(k string, l, r []byte, v string, s string) {\n\tm.data[k] = record{l, r, v, s}\n}\n\nfunc (m *fakeDDB) putData(k string, d []byte) {\n\tm.data[k] = d\n}\n\nfunc (m *fakeDDB) PutItem(input *dynamodb.PutItemInput) (*dynamodb.PutItemOutput, error) {\n\tassert.NotNil(m.t, input.Item[dbAttr], \"%s should have been present\", dbAttr)\n\tassert.NotNil(m.t, input.Item[dbAttr].S, \"key should have been a String: %+v\", input.Item[dbAttr])\n\tkey := *input.Item[dbAttr].S\n\n\tif input.Item[dataAttr] != nil {\n\t\tassert.NotNil(m.t, input.Item[dataAttr].B, \"data should have been a blob: %+v\", input.Item[dataAttr])\n\t\tm.putData(key, input.Item[dataAttr].B)\n\t\treturn &dynamodb.PutItemOutput{}, nil\n\t}\n\n\tassert.NotNil(m.t, input.Item[nbsVersAttr], \"%s should have been present\", nbsVersAttr)\n\tassert.NotNil(m.t, input.Item[nbsVersAttr].S, \"nbsVers should have been a String: %+v\", input.Item[nbsVersAttr])\n\tassert.Equal(m.t, StorageVersion, *input.Item[nbsVersAttr].S)\n\n\tassert.NotNil(m.t, input.Item[versAttr], \"%s should have been present\", versAttr)\n\tassert.NotNil(m.t, input.Item[versAttr].S, \"nbsVers should have been a String: %+v\", input.Item[versAttr])\n\tassert.Equal(m.t, constants.NomsVersion, *input.Item[versAttr].S)\n\n\tassert.NotNil(m.t, input.Item[lockAttr], \"%s should have been present\", lockAttr)\n\tassert.NotNil(m.t, input.Item[lockAttr].B, \"lock should have been a blob: %+v\", input.Item[lockAttr])\n\tlock := input.Item[lockAttr].B\n\n\tassert.NotNil(m.t, input.Item[rootAttr], \"%s should have been present\", rootAttr)\n\tassert.NotNil(m.t, input.Item[rootAttr].B, \"root should have been a blob: %+v\", input.Item[rootAttr])\n\troot := input.Item[rootAttr].B\n\n\tspecs := \"\"\n\tif attr, present := input.Item[tableSpecsAttr]; present {\n\t\tassert.NotNil(m.t, attr.S, \"specs should have been a String: %+v\", input.Item[tableSpecsAttr])\n\t\tspecs = *attr.S\n\t}\n\n\tmustNotExist := *(input.ConditionExpression) == valueNotExistsOrEqualsExpression\n\tcurrent, present := m.data[key]\n\n\tif mustNotExist && present {\n\t\treturn nil, mockAWSError(\"ConditionalCheckFailedException\")\n\t} else if !mustNotExist && !checkCondition(current.(record), input.ExpressionAttributeValues) {\n\t\treturn nil, mockAWSError(\"ConditionalCheckFailedException\")\n\t}\n\n\tm.putRecord(key, lock, root, constants.NomsVersion, specs)\n\tm.numPuts++\n\n\treturn &dynamodb.PutItemOutput{}, nil\n}\n\nfunc checkCondition(current record, expressionAttrVals map[string]*dynamodb.AttributeValue) bool {\n\treturn current.vers == *expressionAttrVals[\":vers\"].S && bytes.Equal(current.lock, expressionAttrVals[\":prev\"].B)\n}\n" }, { "path": "go/nbs/dynamo_manifest.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/aws/aws-sdk-go/aws\"\n\t\"github.com/aws/aws-sdk-go/aws/awserr\"\n\t\"github.com/aws/aws-sdk-go/service/dynamodb\"\n)\n\nconst (\n\tdbAttr = \"db\"\n\tlockAttr = \"lck\" // 'lock' is a reserved word in dynamo\n\trootAttr = \"root\"\n\tversAttr = \"vers\"\n\tnbsVersAttr = \"nbsVers\"\n\ttableSpecsAttr = \"specs\"\n)\n\nvar (\n\tvalueEqualsExpression = fmt.Sprintf(\"(%s = :prev) and (%s = :vers)\", lockAttr, versAttr)\n\tvalueNotExistsOrEqualsExpression = fmt.Sprintf(\"attribute_not_exists(\"+lockAttr+\") or %s\", valueEqualsExpression)\n)\n\ntype ddbsvc interface {\n\tGetItem(input *dynamodb.GetItemInput) (*dynamodb.GetItemOutput, error)\n\tPutItem(input *dynamodb.PutItemInput) (*dynamodb.PutItemOutput, error)\n}\n\n// dynamoManifest assumes the existence of a DynamoDB table whose primary partition key is in String format and named `db`.\ntype dynamoManifest struct {\n\ttable, db string\n\tddbsvc ddbsvc\n}\n\nfunc newDynamoManifest(table, namespace string, ddb ddbsvc) manifest {\n\td.PanicIfTrue(table == \"\")\n\td.PanicIfTrue(namespace == \"\")\n\treturn dynamoManifest{table, namespace, ddb}\n}\n\nfunc (dm dynamoManifest) Name() string {\n\treturn dm.table + dm.db\n}\n\nfunc (dm dynamoManifest) ParseIfExists(stats *Stats, readHook func()) (exists bool, contents manifestContents) {\n\tt1 := time.Now()\n\tdefer func() { stats.ReadManifestLatency.SampleTimeSince(t1) }()\n\n\tresult, err := dm.ddbsvc.GetItem(&dynamodb.GetItemInput{\n\t\tConsistentRead: aws.Bool(true), // This doubles the cost :-(\n\t\tTableName: aws.String(dm.table),\n\t\tKey: map[string]*dynamodb.AttributeValue{\n\t\t\tdbAttr: {S: aws.String(dm.db)},\n\t\t},\n\t})\n\td.PanicIfError(err)\n\n\t// !exists(dbAttr) => unitialized store\n\tif len(result.Item) > 0 {\n\t\tvalid, hasSpecs := validateManifest(result.Item)\n\t\tif !valid {\n\t\t\td.Panic(\"Malformed manifest for %s: %+v\", dm.db, result.Item)\n\t\t}\n\t\texists = true\n\t\tcontents.vers = *result.Item[versAttr].S\n\t\tcontents.root = hash.New(result.Item[rootAttr].B)\n\t\tcopy(contents.lock[:], result.Item[lockAttr].B)\n\t\tif hasSpecs {\n\t\t\tcontents.specs = parseSpecs(strings.Split(*result.Item[tableSpecsAttr].S, \":\"))\n\t\t}\n\t}\n\treturn\n}\n\nfunc validateManifest(item map[string]*dynamodb.AttributeValue) (valid, hasSpecs bool) {\n\tif item[nbsVersAttr] != nil && item[nbsVersAttr].S != nil &&\n\t\tStorageVersion == *item[nbsVersAttr].S &&\n\t\titem[versAttr] != nil && item[versAttr].S != nil &&\n\t\titem[lockAttr] != nil && item[lockAttr].B != nil &&\n\t\titem[rootAttr] != nil && item[rootAttr].B != nil {\n\t\tif len(item) == 6 && item[tableSpecsAttr] != nil && item[tableSpecsAttr].S != nil {\n\t\t\treturn true, true\n\t\t}\n\t\treturn len(item) == 5, false\n\t}\n\treturn false, false\n}\n\nfunc (dm dynamoManifest) Update(lastLock addr, newContents manifestContents, stats *Stats, writeHook func()) manifestContents {\n\tt1 := time.Now()\n\tdefer func() { stats.WriteManifestLatency.SampleTimeSince(t1) }()\n\n\tputArgs := dynamodb.PutItemInput{\n\t\tTableName: aws.String(dm.table),\n\t\tItem: map[string]*dynamodb.AttributeValue{\n\t\t\tdbAttr: {S: aws.String(dm.db)},\n\t\t\tnbsVersAttr: {S: aws.String(StorageVersion)},\n\t\t\tversAttr: {S: aws.String(newContents.vers)},\n\t\t\trootAttr: {B: newContents.root[:]},\n\t\t\tlockAttr: {B: newContents.lock[:]},\n\t\t},\n\t}\n\tif len(newContents.specs) > 0 {\n\t\ttableInfo := make([]string, 2*len(newContents.specs))\n\t\tformatSpecs(newContents.specs, tableInfo)\n\t\tputArgs.Item[tableSpecsAttr] = &dynamodb.AttributeValue{S: aws.String(strings.Join(tableInfo, \":\"))}\n\t}\n\n\texpr := valueEqualsExpression\n\tif lastLock == (addr{}) {\n\t\texpr = valueNotExistsOrEqualsExpression\n\t}\n\n\tputArgs.ConditionExpression = aws.String(expr)\n\tputArgs.ExpressionAttributeValues = map[string]*dynamodb.AttributeValue{\n\t\t\":prev\": {B: lastLock[:]},\n\t\t\":vers\": {S: aws.String(newContents.vers)},\n\t}\n\n\t_, ddberr := dm.ddbsvc.PutItem(&putArgs)\n\tif ddberr != nil {\n\t\tif errIsConditionalCheckFailed(ddberr) {\n\t\t\texists, upstream := dm.ParseIfExists(stats, nil)\n\t\t\td.Chk.True(exists)\n\t\t\td.Chk.True(upstream.vers == constants.NomsVersion)\n\t\t\treturn upstream\n\t\t} // TODO handle other aws errors?\n\t\td.PanicIfError(ddberr)\n\t}\n\n\treturn newContents\n}\n\nfunc errIsConditionalCheckFailed(err error) bool {\n\tawsErr, ok := err.(awserr.Error)\n\treturn ok && awsErr.Code() == \"ConditionalCheckFailedException\"\n}\n" }, { "path": "go/nbs/dynamo_manifest_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nconst (\n\ttable = \"testTable\"\n\tdb = \"testDB\"\n)\n\nfunc makeDynamoManifestFake(t *testing.T) (mm manifest, ddb *fakeDDB) {\n\tddb = makeFakeDDB(t)\n\tmm = newDynamoManifest(table, db, ddb)\n\treturn\n}\n\nfunc TestDynamoManifestParseIfExists(t *testing.T) {\n\tassert := assert.New(t)\n\tmm, ddb := makeDynamoManifestFake(t)\n\tstats := &Stats{}\n\n\texists, _ := mm.ParseIfExists(stats, nil)\n\tassert.False(exists)\n\n\t// Simulate another process writing a manifest (with an old Noms version).\n\tnewLock := computeAddr([]byte(\"locker\"))\n\tnewRoot := hash.Of([]byte(\"new root\"))\n\ttableName := hash.Of([]byte(\"table1\"))\n\tddb.putRecord(db, newLock[:], newRoot[:], \"0\", tableName.String()+\":\"+\"0\")\n\n\t// ParseIfExists should now reflect the manifest written above.\n\texists, contents := mm.ParseIfExists(stats, nil)\n\tassert.True(exists)\n\tassert.Equal(\"0\", contents.vers)\n\tassert.Equal(newLock, contents.lock)\n\tassert.Equal(newRoot, contents.root)\n\tif assert.Len(contents.specs, 1) {\n\t\tassert.Equal(tableName.String(), contents.specs[0].name.String())\n\t\tassert.Equal(uint32(0), contents.specs[0].chunkCount)\n\t}\n}\n\nfunc makeContents(lock, root string, specs []tableSpec) manifestContents {\n\treturn manifestContents{constants.NomsVersion, computeAddr([]byte(lock)), hash.Of([]byte(root)), specs}\n}\n\nfunc TestDynamoManifestUpdateWontClobberOldVersion(t *testing.T) {\n\tassert := assert.New(t)\n\tmm, ddb := makeDynamoManifestFake(t)\n\tstats := &Stats{}\n\n\t// Simulate another process having already put old Noms data in dir/.\n\tlock := computeAddr([]byte(\"locker\"))\n\tbadRoot := hash.Of([]byte(\"bad root\"))\n\tddb.putRecord(db, lock[:], badRoot[:], \"0\", \"\")\n\n\tassert.Panics(func() { mm.Update(lock, manifestContents{vers: constants.NomsVersion}, stats, nil) })\n}\n\nfunc TestDynamoManifestUpdate(t *testing.T) {\n\tassert := assert.New(t)\n\tmm, ddb := makeDynamoManifestFake(t)\n\tstats := &Stats{}\n\n\t// First, test winning the race against another process.\n\tcontents := makeContents(\"locker\", \"nuroot\", []tableSpec{{computeAddr([]byte(\"a\")), 3}})\n\tupstream := mm.Update(addr{}, contents, stats, func() {\n\t\t// This should fail to get the lock, and therefore _not_ clobber the manifest. So the Update should succeed.\n\t\tlock := computeAddr([]byte(\"nolock\"))\n\t\tnewRoot2 := hash.Of([]byte(\"noroot\"))\n\t\tddb.putRecord(db, lock[:], newRoot2[:], constants.NomsVersion, \"\")\n\t})\n\tassert.Equal(contents.lock, upstream.lock)\n\tassert.Equal(contents.root, upstream.root)\n\tassert.Equal(contents.specs, upstream.specs)\n\n\t// Now, test the case where the optimistic lock fails, and someone else updated the root since last we checked.\n\trejected := makeContents(\"locker 2\", \"new root 2\", nil)\n\tupstream = mm.Update(addr{}, rejected, stats, nil)\n\tassert.Equal(contents.lock, upstream.lock)\n\tassert.Equal(contents.root, upstream.root)\n\tassert.Equal(contents.specs, upstream.specs)\n\tupstream = mm.Update(upstream.lock, rejected, stats, nil)\n\tassert.Equal(rejected.lock, upstream.lock)\n\tassert.Equal(rejected.root, upstream.root)\n\tassert.Empty(upstream.specs)\n\n\t// Now, test the case where the optimistic lock fails because someone else updated only the tables since last we checked\n\tjerkLock := computeAddr([]byte(\"jerk\"))\n\ttableName := computeAddr([]byte(\"table1\"))\n\tddb.putRecord(db, jerkLock[:], upstream.root[:], constants.NomsVersion, tableName.String()+\":1\")\n\n\tnewContents3 := makeContents(\"locker 3\", \"new root 3\", nil)\n\tupstream = mm.Update(upstream.lock, newContents3, stats, nil)\n\tassert.Equal(jerkLock, upstream.lock)\n\tassert.Equal(rejected.root, upstream.root)\n\tassert.Equal([]tableSpec{{tableName, 1}}, upstream.specs)\n}\n\nfunc TestDynamoManifestCaching(t *testing.T) {\n\tassert := assert.New(t)\n\tmm, ddb := makeDynamoManifestFake(t)\n\tstats := &Stats{}\n\n\t// ParseIfExists should hit persistent storage no matter what\n\treads := ddb.numGets\n\texists, _ := mm.ParseIfExists(stats, nil)\n\tassert.False(exists)\n\tassert.Equal(reads+1, ddb.numGets)\n\n\tlock, root := computeAddr([]byte(\"lock\")), hash.Of([]byte(\"root\"))\n\tddb.putRecord(db, lock[:], root[:], constants.NomsVersion, \"\")\n\n\treads = ddb.numGets\n\texists, _ = mm.ParseIfExists(stats, nil)\n\tassert.True(exists)\n\tassert.Equal(reads+1, ddb.numGets)\n\n\t// When failing the optimistic lock, we should hit persistent storage.\n\treads = ddb.numGets\n\tcontents := makeContents(\"lock2\", \"nuroot\", []tableSpec{{computeAddr([]byte(\"a\")), 3}})\n\tupstream := mm.Update(addr{}, contents, stats, nil)\n\tassert.NotEqual(contents.lock, upstream.lock)\n\tassert.Equal(reads+1, ddb.numGets)\n\n\t// Successful update should NOT hit persistent storage.\n\treads = ddb.numGets\n\tupstream = mm.Update(upstream.lock, contents, stats, nil)\n\tassert.Equal(contents.lock, upstream.lock)\n\tassert.Equal(reads, ddb.numGets)\n}\n\nfunc TestDynamoManifestUpdateEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\tmm, _ := makeDynamoManifestFake(t)\n\tstats := &Stats{}\n\n\tcontents := manifestContents{vers: constants.NomsVersion, lock: computeAddr([]byte{0x01})}\n\tupstream := mm.Update(addr{}, contents, stats, nil)\n\tassert.Equal(contents.lock, upstream.lock)\n\tassert.True(upstream.root.IsEmpty())\n\tassert.Empty(upstream.specs)\n}\n" }, { "path": "go/nbs/dynamo_table_reader.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/util/sizecache\"\n\t\"github.com/aws/aws-sdk-go/aws\"\n\t\"github.com/aws/aws-sdk-go/service/dynamodb\"\n)\n\nconst (\n\tdataAttr = \"data\"\n\ttablePrefix = \"*\" // I want to use NBS table names as keys when they are written to DynamoDB, but a bare table name is a legal Noms Database name as well. To avoid collisions, dynamoTableReader prepends this prefix (which is not a legal character in a Noms Database name).\n)\n\n// dynamoTableReaderAt assumes the existence of a DynamoDB table whose primary partition key is in String format and named `db`.\ntype dynamoTableReaderAt struct {\n\tddb *ddbTableStore\n\th addr\n}\n\ntype tableNotInDynamoErr struct {\n\tnbs, dynamo string\n}\n\nfunc (t tableNotInDynamoErr) Error() string {\n\treturn fmt.Sprintf(\"NBS table %s not present in DynamoDB table %s\", t.nbs, t.dynamo)\n}\n\nfunc (dtra *dynamoTableReaderAt) ReadAtWithStats(p []byte, off int64, stats *Stats) (n int, err error) {\n\tdata, err := dtra.ddb.ReadTable(dtra.h, stats)\n\td.PanicIfError(err)\n\tn = copy(p, data[off:])\n\tif n < len(p) {\n\t\terr = io.ErrUnexpectedEOF\n\t}\n\treturn\n}\n\ntype ddbTableStore struct {\n\tddb ddbsvc\n\ttable string\n\treadRl chan struct{}\n\tcache *sizecache.SizeCache // TODO: merge this with tableCache as part of BUG 3601\n}\n\nfunc (dts *ddbTableStore) ReadTable(name addr, stats *Stats) (data []byte, err error) {\n\tt1 := time.Now()\n\tif dts.cache != nil {\n\t\tif i, present := dts.cache.Get(name); present {\n\t\t\tdata = i.([]byte)\n\t\t\tdefer func() {\n\t\t\t\tstats.MemBytesPerRead.Sample(uint64(len(data)))\n\t\t\t\tstats.MemReadLatency.SampleTimeSince(t1)\n\t\t\t}()\n\t\t\treturn data, nil\n\t\t}\n\t}\n\n\tdata, err = dts.readTable(name)\n\tif data != nil {\n\t\tdefer func() {\n\t\t\tstats.DynamoBytesPerRead.Sample(uint64(len(data)))\n\t\t\tstats.DynamoReadLatency.SampleTimeSince(t1)\n\t\t}()\n\t}\n\n\tif dts.cache != nil && err == nil {\n\t\tdts.cache.Add(name, uint64(len(data)), data)\n\t}\n\treturn data, err\n}\n\nfunc (dts *ddbTableStore) readTable(name addr) (data []byte, err error) {\n\ttry := func(input *dynamodb.GetItemInput) (data []byte, err error) {\n\t\tif dts.readRl != nil {\n\t\t\tdts.readRl <- struct{}{}\n\t\t\tdefer func() {\n\t\t\t\t<-dts.readRl\n\t\t\t}()\n\t\t}\n\t\tresult, rerr := dts.ddb.GetItem(input)\n\t\tif rerr != nil {\n\t\t\treturn nil, rerr\n\t\t} else if len(result.Item) == 0 {\n\t\t\treturn nil, tableNotInDynamoErr{name.String(), dts.table}\n\t\t} else if result.Item[dataAttr] == nil || result.Item[dataAttr].B == nil {\n\t\t\treturn nil, fmt.Errorf(\"NBS table %s in DynamoDB table %s is malformed\", name, dts.table)\n\t\t}\n\t\treturn result.Item[dataAttr].B, nil\n\t}\n\n\tinput := dynamodb.GetItemInput{\n\t\tTableName: aws.String(dts.table),\n\t\tKey: map[string]*dynamodb.AttributeValue{\n\t\t\tdbAttr: {S: aws.String(fmtTableName(name))},\n\t\t},\n\t}\n\tdata, err = try(&input)\n\tif _, isNotFound := err.(tableNotInDynamoErr); isNotFound {\n\t\tlog.Printf(\"Eventually consistent read for %s failed; trying fully-consistent\", name)\n\t\tinput.ConsistentRead = aws.Bool(true)\n\t\treturn try(&input)\n\t}\n\treturn data, err\n}\n\nfunc fmtTableName(name addr) string {\n\treturn tablePrefix + name.String()\n}\n\nfunc (dts *ddbTableStore) Write(name addr, data []byte) error {\n\t_, err := dts.ddb.PutItem(&dynamodb.PutItemInput{\n\t\tTableName: aws.String(dts.table),\n\t\tItem: map[string]*dynamodb.AttributeValue{\n\t\t\tdbAttr: {S: aws.String(fmtTableName(name))},\n\t\t\tdataAttr: {B: data},\n\t\t},\n\t})\n\n\tif dts.cache != nil && err == nil {\n\t\tdts.cache.Add(name, uint64(len(data)), data)\n\t}\n\treturn err\n}\n" }, { "path": "go/nbs/dynamo_table_reader_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/util/sizecache\"\n\t\"github.com/aws/aws-sdk-go/service/dynamodb\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestDynamoTableReaderAt(t *testing.T) {\n\tddb := makeFakeDDB(t)\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\ttableData, h := buildTable(chunks)\n\tddb.putData(fmtTableName(h), tableData)\n\n\tt.Run(\"ddbTableStore\", func(t *testing.T) {\n\t\tt.Run(\"ReadTable\", func(t *testing.T) {\n\t\t\ttest := func(dts *ddbTableStore) {\n\t\t\t\tassert := assert.New(t)\n\t\t\t\tdata, err := dts.ReadTable(h, &Stats{})\n\t\t\t\tassert.NoError(err)\n\t\t\t\tassert.Equal(tableData, data)\n\n\t\t\t\tdata, err = dts.ReadTable(computeAddr([]byte{}), &Stats{})\n\t\t\t\tassert.Error(err)\n\t\t\t\tassert.IsType(tableNotInDynamoErr{}, err)\n\t\t\t\tassert.Nil(data)\n\t\t\t}\n\n\t\t\tt.Run(\"EventuallyConsistentSuccess\", func(t *testing.T) {\n\t\t\t\ttest(&ddbTableStore{ddb, \"table\", nil, nil})\n\t\t\t})\n\n\t\t\tt.Run(\"EventuallyConsistentFailure\", func(t *testing.T) {\n\t\t\t\ttest(&ddbTableStore{&eventuallyConsistentDDB{ddb}, \"table\", nil, nil})\n\t\t\t})\n\n\t\t\tt.Run(\"WithCache\", func(t *testing.T) {\n\t\t\t\ttc := sizecache.New(uint64(2 * len(tableData)))\n\t\t\t\tdts := &ddbTableStore{ddb, \"table\", nil, tc}\n\t\t\t\ttest(dts)\n\n\t\t\t\t// Table should have been cached on read\n\t\t\t\tbaseline := ddb.numGets\n\t\t\t\t_, err := dts.ReadTable(h, &Stats{})\n\t\t\t\tassert.NoError(t, err)\n\t\t\t\tassert.Zero(t, ddb.numGets-baseline)\n\t\t\t})\n\t\t})\n\n\t\tt.Run(\"WriteTable\", func(t *testing.T) {\n\t\t\tt.Run(\"WithoutCache\", func(t *testing.T) {\n\t\t\t\tassert := assert.New(t)\n\n\t\t\t\tdts := &ddbTableStore{makeFakeDDB(t), \"table\", nil, nil}\n\t\t\t\tassert.NoError(dts.Write(h, tableData))\n\n\t\t\t\tdata, err := dts.ReadTable(h, &Stats{})\n\t\t\t\tassert.NoError(err)\n\t\t\t\tassert.Equal(tableData, data)\n\t\t\t})\n\n\t\t\tt.Run(\"WithCache\", func(t *testing.T) {\n\t\t\t\tassert := assert.New(t)\n\n\t\t\t\ttc := sizecache.New(uint64(2 * len(tableData)))\n\t\t\t\tdts := &ddbTableStore{makeFakeDDB(t), \"table\", nil, tc}\n\t\t\t\tassert.NoError(dts.Write(h, tableData))\n\n\t\t\t\t// Table should have been cached on write\n\t\t\t\tbaseline := ddb.numGets\n\t\t\t\tdata, err := dts.ReadTable(h, &Stats{})\n\t\t\t\tassert.NoError(err)\n\t\t\t\tassert.Equal(tableData, data)\n\t\t\t\tassert.Zero(ddb.numGets - baseline)\n\t\t\t})\n\t\t})\n\t})\n\n\tt.Run(\"ReadAtWithCache\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\t\tstats := &Stats{}\n\n\t\ttc := sizecache.New(uint64(2 * len(tableData)))\n\t\ttra := &dynamoTableReaderAt{&ddbTableStore{ddb, \"table\", nil, tc}, h}\n\n\t\t// First, read when table is not yet cached\n\t\tscratch := make([]byte, len(tableData)/4)\n\t\tbaseline := ddb.numGets\n\t\t_, err := tra.ReadAtWithStats(scratch, 0, stats)\n\t\tassert.NoError(err)\n\t\tassert.True(ddb.numGets > baseline)\n\n\t\t// Table should have been cached on read so read again, a different slice this time\n\t\tbaseline = ddb.numGets\n\t\t_, err = tra.ReadAtWithStats(scratch, int64(len(scratch)), stats)\n\t\tassert.NoError(err)\n\t\tassert.Zero(ddb.numGets - baseline)\n\t})\n}\n\ntype eventuallyConsistentDDB struct {\n\tddbsvc\n}\n\nfunc (ec *eventuallyConsistentDDB) GetItem(input *dynamodb.GetItemInput) (*dynamodb.GetItemOutput, error) {\n\tif input.ConsistentRead != nil && *(input.ConsistentRead) {\n\t\treturn ec.ddbsvc.GetItem(input)\n\t}\n\treturn &dynamodb.GetItemOutput{}, nil\n}\n" }, { "path": "go/nbs/factory.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"path\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/util/sizecache\"\n\t\"github.com/aws/aws-sdk-go/aws/session\"\n\t\"github.com/aws/aws-sdk-go/service/dynamodb\"\n\t\"github.com/aws/aws-sdk-go/service/s3\"\n)\n\nconst (\n\tdefaultAWSReadLimit = 1024\n\tawsMaxTables = 128\n\n\tdefaultSmallTableCacheSize = 1 << 28 // 256MB\n)\n\n// AWSStoreFactory vends NomsBlockStores built on top of DynamoDB and S3.\ntype AWSStoreFactory struct {\n\tddb ddbsvc\n\tpersister tablePersister\n\ttable string\n\tconjoiner conjoiner\n\n\tmanifestLocks *manifestLocks\n\tmanifestCache *manifestCache\n}\n\n// NewAWSStoreFactory returns a ChunkStore factory that vends NomsBlockStore\n// instances that store manifests in the named DynamoDB table, and chunk data\n// in the named S3 bucket. All connections to AWS services share |sess|.\nfunc NewAWSStoreFactory(sess *session.Session, table, bucket string, maxOpenFiles int, indexCacheSize, tableCacheSize uint64, tableCacheDir string) chunks.Factory {\n\tvar indexCache *indexCache\n\tif indexCacheSize > 0 {\n\t\tindexCache = newIndexCache(indexCacheSize)\n\t}\n\tvar tc *fsTableCache\n\tif tableCacheSize > 0 {\n\t\ttc = newFSTableCache(tableCacheDir, tableCacheSize, maxOpenFiles)\n\t}\n\n\tddb := dynamodb.New(sess)\n\treadRateLimiter := make(chan struct{}, defaultAWSReadLimit)\n\treturn &AWSStoreFactory{\n\t\tddb: ddb,\n\t\tpersister: &awsTablePersister{\n\t\t\ts3.New(sess),\n\t\t\tbucket,\n\t\t\treadRateLimiter,\n\t\t\ttc,\n\t\t\t&ddbTableStore{ddb, table, readRateLimiter, sizecache.New(defaultSmallTableCacheSize)},\n\t\t\tawsLimits{defaultS3PartSize, minS3PartSize, maxS3PartSize, maxDynamoItemSize, maxDynamoChunks},\n\t\t\tindexCache,\n\t\t},\n\t\ttable: table,\n\t\tconjoiner: inlineConjoiner{awsMaxTables},\n\t\tmanifestLocks: newManifestLocks(),\n\t\tmanifestCache: newManifestCache(defaultManifestCacheSize),\n\t}\n}\n\nfunc (asf *AWSStoreFactory) CreateStore(ns string) chunks.ChunkStore {\n\tmm := manifestManager{newDynamoManifest(asf.table, ns, asf.ddb), asf.manifestCache, asf.manifestLocks}\n\treturn newNomsBlockStore(mm, asf.persister, asf.conjoiner, defaultMemTableSize)\n}\n\nfunc (asf *AWSStoreFactory) CreateStoreFromCache(ns string) chunks.ChunkStore {\n\tmm := manifestManager{newDynamoManifest(asf.table, ns, asf.ddb), asf.manifestCache, asf.manifestLocks}\n\n\tcontents, _, present := asf.manifestCache.Get(mm.Name())\n\tif present {\n\t\treturn newNomsBlockStoreWithContents(mm, contents, asf.persister, asf.conjoiner, defaultMemTableSize)\n\t}\n\treturn nil\n}\n\nfunc (asf *AWSStoreFactory) Shutter() {\n}\n\ntype LocalStoreFactory struct {\n\tdir string\n\tfc *fdCache\n\tindexCache *indexCache\n\tconjoiner conjoiner\n\n\tmanifestLocks *manifestLocks\n\tmanifestCache *manifestCache\n}\n\nfunc checkDir(dir string) error {\n\tstat, err := os.Stat(dir)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif !stat.IsDir() {\n\t\treturn fmt.Errorf(\"Path is not a directory: %s\", dir)\n\t}\n\treturn nil\n}\n\nfunc NewLocalStoreFactory(dir string, indexCacheSize uint64, maxOpenFiles int) chunks.Factory {\n\terr := checkDir(dir)\n\td.PanicIfError(err)\n\n\tvar indexCache *indexCache\n\tif indexCacheSize > 0 {\n\t\tindexCache = newIndexCache(indexCacheSize)\n\t}\n\treturn &LocalStoreFactory{\n\t\tdir: dir,\n\t\tfc: newFDCache(maxOpenFiles),\n\t\tindexCache: indexCache,\n\t\tconjoiner: inlineConjoiner{defaultMaxTables},\n\t\tmanifestLocks: newManifestLocks(),\n\t\tmanifestCache: newManifestCache(defaultManifestCacheSize),\n\t}\n}\n\nfunc (lsf *LocalStoreFactory) CreateStore(ns string) chunks.ChunkStore {\n\tpath := path.Join(lsf.dir, ns)\n\td.PanicIfError(os.MkdirAll(path, 0777))\n\n\tmm := manifestManager{fileManifest{path}, lsf.manifestCache, lsf.manifestLocks}\n\tp := newFSTablePersister(path, lsf.fc, lsf.indexCache)\n\treturn newNomsBlockStore(mm, p, lsf.conjoiner, defaultMemTableSize)\n}\n\nfunc (lsf *LocalStoreFactory) CreateStoreFromCache(ns string) chunks.ChunkStore {\n\tpath := path.Join(lsf.dir, ns)\n\tmm := manifestManager{fileManifest{path}, lsf.manifestCache, lsf.manifestLocks}\n\n\tcontents, _, present := lsf.manifestCache.Get(mm.Name())\n\tif present {\n\t\t_, err := os.Stat(path)\n\t\td.PanicIfTrue(os.IsNotExist(err))\n\t\tp := newFSTablePersister(path, lsf.fc, lsf.indexCache)\n\t\treturn newNomsBlockStoreWithContents(mm, contents, p, lsf.conjoiner, defaultMemTableSize)\n\t}\n\treturn nil\n}\n\nfunc (lsf *LocalStoreFactory) Shutter() {\n\tlsf.fc.Drop()\n}\n" }, { "path": "go/nbs/factory_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestLocalStoreFactory(t *testing.T) {\n\tassert := assert.New(t)\n\tdir := makeTempDir(t)\n\tdefer os.RemoveAll(dir)\n\n\tf := NewLocalStoreFactory(dir, 0, 8)\n\tstats := &Stats{}\n\n\tdbName := \"db\"\n\tstore := f.CreateStore(dbName)\n\n\tc := chunks.NewChunk([]byte{0xff})\n\tstore.Put(c)\n\tassert.True(store.Commit(c.Hash(), hash.Hash{}))\n\n\tdbDir := filepath.Join(dir, dbName)\n\texists, contents := fileManifest{dbDir}.ParseIfExists(stats, nil)\n\tassert.True(exists)\n\tassert.Len(contents.specs, 1)\n\n\t_, err := os.Stat(filepath.Join(dbDir, contents.specs[0].name.String()))\n\tassert.NoError(err)\n\n\t// Simulate another process writing a manifest.\n\tlock := computeAddr([]byte(\"locker\"))\n\tnewRoot := hash.Of([]byte(\"new root\"))\n\terr = clobberManifest(dbDir, strings.Join([]string{StorageVersion, constants.NomsVersion, lock.String(), newRoot.String(), contents.specs[0].name.String(), \"1\"}, \":\"))\n\tassert.NoError(err)\n\n\tcached := f.CreateStoreFromCache(dbName)\n\tassert.Equal(c.Hash(), cached.Root())\n}\n" }, { "path": "go/nbs/fd_cache.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"os\"\n\t\"sort\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nfunc newFDCache(targetSize int) *fdCache {\n\treturn &fdCache{targetSize: targetSize, cache: map[string]fdCacheEntry{}}\n}\n\n// fdCache ref-counts open file descriptors, but doesn't keep a hard cap on\n// the number of open files. Once the cache's target size is exceeded, opening\n// a new file causes the cache to try to get the cache back to the target size\n// by closing fds with zero refs. If there aren't enough such fds, fdCache\n// gives up and tries again next time a caller refs a file.\ntype fdCache struct {\n\ttargetSize int\n\tmu sync.Mutex\n\tcache map[string]fdCacheEntry\n}\n\ntype fdCacheEntry struct {\n\trefCount uint32\n\tf *os.File\n}\n\n// RefFile returns an opened *os.File for the file at |path|, or an error\n// indicating why the file could not be opened. If the cache already had an\n// entry for |path|, RefFile increments its refcount and returns the cached\n// pointer. If not, it opens the file and caches the pointer for others to\n// use. If RefFile returns an error, it's guaranteed that no refCounts were\n// changed, so it's an error to make a subsequent call to UnrefFile().\n// This is intended for clients that hold fds for extremely short periods.\nfunc (fc *fdCache) RefFile(path string) (f *os.File, err error) {\n\trefFile := func() *os.File {\n\t\tif ce, present := fc.cache[path]; present {\n\t\t\tce.refCount++\n\t\t\tfc.cache[path] = ce\n\t\t\treturn ce.f\n\t\t}\n\t\treturn nil\n\t}\n\n\tf = func() *os.File {\n\t\tfc.mu.Lock()\n\t\tdefer fc.mu.Unlock()\n\t\treturn refFile()\n\t}()\n\tif f != nil {\n\t\treturn f, nil\n\t}\n\n\t// Very much want this to be outside the lock, but the downside is that multiple callers may get here concurrently. That means we need to deal with the raciness below.\n\tf, err = os.Open(path)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tfc.mu.Lock()\n\tdefer fc.mu.Unlock()\n\tif cached := refFile(); cached != nil {\n\t\t// Someone beat us to it, so close f and return cached fd\n\t\tf.Close()\n\t\treturn cached, nil\n\t}\n\t// I won the race!\n\tfc.cache[path] = fdCacheEntry{f: f, refCount: 1}\n\treturn f, nil\n}\n\n// UnrefFile reduces the refcount of the entry at |path|. If the cache is over\n// |fc.targetSize|, UnrefFile makes a best effort to shrink the cache by dumping\n// entries with a zero refcount. If there aren't enough zero refcount entries\n// to drop to get the cache back to |fc.targetSize|, the cache will remain\n// over |fc.targetSize| until the next call to UnrefFile().\nfunc (fc *fdCache) UnrefFile(path string) {\n\tfc.mu.Lock()\n\tdefer fc.mu.Unlock()\n\tif ce, present := fc.cache[path]; present {\n\t\tce.refCount--\n\t\tfc.cache[path] = ce\n\t}\n\tif len(fc.cache) > fc.targetSize {\n\t\t// Sadly, we can't remove items from a map while iterating, so we'll record the stuff we want to drop and then do it after\n\t\tneeded := len(fc.cache) - fc.targetSize\n\t\ttoDrop := make([]string, 0, needed)\n\t\tfor p, ce := range fc.cache {\n\t\t\tif ce.refCount != 0 {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\ttoDrop = append(toDrop, p)\n\t\t\terr := ce.f.Close()\n\t\t\td.PanicIfError(err)\n\t\t\tneeded--\n\t\t\tif needed == 0 {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t\tfor _, p := range toDrop {\n\t\t\tdelete(fc.cache, p)\n\t\t}\n\t}\n}\n\n// Drop dumps the entire cache and closes all currently open files.\nfunc (fc *fdCache) Drop() {\n\tfc.mu.Lock()\n\tdefer fc.mu.Unlock()\n\tfor _, ce := range fc.cache {\n\t\tce.f.Close()\n\t}\n\tfc.cache = map[string]fdCacheEntry{}\n}\n\n// reportEntries is meant for testing.\nfunc (fc *fdCache) reportEntries() sort.StringSlice {\n\tfc.mu.Lock()\n\tdefer fc.mu.Unlock()\n\tret := make(sort.StringSlice, 0, len(fc.cache))\n\tfor p := range fc.cache {\n\t\tret = append(ret, p)\n\t}\n\tsort.Sort(ret)\n\treturn ret\n}\n" }, { "path": "go/nbs/fd_cache_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sort\"\n\t\"sync\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestFDCache(t *testing.T) {\n\tdir := makeTempDir(t)\n\tdefer os.RemoveAll(dir)\n\n\tpaths := [3]string{}\n\tfor i := range paths {\n\t\tname := fmt.Sprintf(\"file%d\", i)\n\t\tpaths[i] = filepath.Join(dir, name)\n\t\terr := ioutil.WriteFile(paths[i], []byte(name), 0644)\n\t\tassert.NoError(t, err)\n\t}\n\n\trefNoError := func(fc *fdCache, p string, assert *assert.Assertions) *os.File {\n\t\tf, err := fc.RefFile(p)\n\t\tassert.NoError(err)\n\t\tassert.NotNil(f)\n\t\treturn f\n\t}\n\n\tt.Run(\"ConcurrentOpen\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\t\tconcurrency := 3\n\t\tfc := newFDCache(3)\n\t\tdefer fc.Drop()\n\n\t\ttrigger := make(chan struct{})\n\t\twg := sync.WaitGroup{}\n\t\tfor i := 0; i < concurrency; i++ {\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tdefer wg.Done()\n\t\t\t\t<-trigger\n\t\t\t\tfc.RefFile(paths[0])\n\t\t\t}()\n\t\t}\n\t\tclose(trigger)\n\t\twg.Wait()\n\n\t\tpresent := fc.reportEntries()\n\t\tif assert.Len(present, 1) {\n\t\t\tce := fc.cache[present[0]]\n\t\t\tassert.EqualValues(concurrency, ce.refCount)\n\t\t}\n\t})\n\n\tt.Run(\"NoEvictions\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\t\tfc := newFDCache(2)\n\t\tdefer fc.Drop()\n\t\tf := refNoError(fc, paths[0], assert)\n\n\t\tf2 := refNoError(fc, paths[1], assert)\n\t\tassert.NotEqual(f, f2)\n\n\t\tdup := refNoError(fc, paths[0], assert)\n\t\tassert.Equal(f, dup)\n\t})\n\n\tt.Run(\"Evictions\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\t\tfc := newFDCache(1)\n\t\tdefer fc.Drop()\n\n\t\tf0 := refNoError(fc, paths[0], assert)\n\t\tf1 := refNoError(fc, paths[1], assert)\n\t\tassert.NotEqual(f0, f1)\n\n\t\t// f0 wasn't evicted, because that doesn't happen until UnrefFile()\n\t\tdup := refNoError(fc, paths[0], assert)\n\t\tassert.Equal(f0, dup)\n\n\t\texpected := sort.StringSlice(paths[:2])\n\t\tsort.Sort(expected)\n\t\tassert.EqualValues(expected, fc.reportEntries())\n\n\t\t// Unreffing f1 now should evict it\n\t\tfc.UnrefFile(paths[1])\n\t\tassert.EqualValues(paths[:1], fc.reportEntries())\n\n\t\t// Bring f1 back so we can test multiple evictions in a row\n\t\tf1 = refNoError(fc, paths[1], assert)\n\t\tassert.NotEqual(f0, f1)\n\n\t\t// After adding f3, we should be able to evict both f0 and f1\n\t\tf2 := refNoError(fc, paths[2], assert)\n\t\tassert.NotEqual(f0, f2)\n\t\tassert.NotEqual(f1, f2)\n\n\t\tfc.UnrefFile(paths[0])\n\t\tfc.UnrefFile(paths[0])\n\t\tfc.UnrefFile(paths[1])\n\n\t\tassert.EqualValues(paths[2:], fc.reportEntries())\n\t})\n}\n" }, { "path": "go/nbs/file_manifest.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"io\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"time\"\n\n\t\"golang.org/x/sys/unix\"\n\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\nconst (\n\tmanifestFileName = \"manifest\"\n\tlockFileName = \"LOCK\"\n)\n\n// fileManifest provides access to a NomsBlockStore manifest stored on disk in |dir|. The format\n// is currently human readable:\n//\n// |-- String --|-- String --|-------- String --------|-------- String --------|-- String --|- String --|...|-- String --|- String --|\n// | nbs version:Noms version:Base32-encoded lock hash:Base32-encoded root hash:table 1 hash:table 1 cnt:...:table N hash:table N cnt|\ntype fileManifest struct {\n\tdir string\n}\n\nfunc (fm fileManifest) Name() string {\n\treturn fm.dir\n}\n\n// ParseIfExists looks for a LOCK and manifest file in fm.dir. If it finds\n// them, it takes the lock, parses the manifest and returns its contents,\n// setting |exists| to true. If not, it sets |exists| to false and returns. In\n// that case, the other return values are undefined. If |readHook| is non-nil,\n// it will be executed while ParseIfExists() holds the manifest file lock.\n// This is to allow for race condition testing.\nfunc (fm fileManifest) ParseIfExists(stats *Stats, readHook func()) (exists bool, contents manifestContents) {\n\tt1 := time.Now()\n\tdefer func() { stats.ReadManifestLatency.SampleTimeSince(t1) }()\n\n\t// !exists(lockFileName) => unitialized store\n\tif l := openIfExists(filepath.Join(fm.dir, lockFileName)); l != nil {\n\t\tvar f io.ReadCloser\n\t\tfunc() {\n\t\t\td.PanicIfError(unix.Flock(int(l.Fd()), unix.LOCK_EX))\n\t\t\tdefer checkClose(l) // releases the flock()\n\n\t\t\tif readHook != nil {\n\t\t\t\treadHook()\n\t\t\t}\n\t\t\tf = openIfExists(filepath.Join(fm.dir, manifestFileName))\n\t\t}()\n\n\t\tif f != nil {\n\t\t\tdefer checkClose(f)\n\t\t\texists = true\n\t\t\tcontents = parseManifest(f)\n\t\t}\n\t}\n\treturn\n}\n\n// Returns nil if path does not exist\nfunc openIfExists(path string) *os.File {\n\tf, err := os.Open(path)\n\tif os.IsNotExist(err) {\n\t\treturn nil\n\t}\n\td.PanicIfError(err)\n\treturn f\n}\n\nfunc parseManifest(r io.Reader) manifestContents {\n\tmanifest, err := ioutil.ReadAll(r)\n\td.PanicIfError(err)\n\n\tslices := strings.Split(string(manifest), \":\")\n\tif len(slices) < 4 || len(slices)%2 == 1 {\n\t\td.Chk.Fail(\"Malformed manifest: \" + string(manifest))\n\t}\n\td.PanicIfFalse(StorageVersion == string(slices[0]))\n\n\treturn manifestContents{\n\t\tvers: slices[1],\n\t\tlock: ParseAddr([]byte(slices[2])),\n\t\troot: hash.Parse(slices[3]),\n\t\tspecs: parseSpecs(slices[4:]),\n\t}\n}\n\nfunc (fm fileManifest) Update(lastLock addr, newContents manifestContents, stats *Stats, writeHook func()) manifestContents {\n\tt1 := time.Now()\n\tdefer func() { stats.WriteManifestLatency.SampleTimeSince(t1) }()\n\n\t// Write a temporary manifest file, to be renamed over manifestFileName upon success.\n\t// The closure here ensures this file is closed before moving on.\n\ttempManifestPath := func() string {\n\t\ttemp, err := ioutil.TempFile(fm.dir, \"nbs_manifest_\")\n\t\td.PanicIfError(err)\n\t\tdefer checkClose(temp)\n\t\twriteManifest(temp, newContents)\n\t\treturn temp.Name()\n\t}()\n\tdefer os.Remove(tempManifestPath) // If we rename below, this will be a no-op\n\n\t// Take manifest file lock\n\tdefer checkClose(flock(filepath.Join(fm.dir, lockFileName))) // closing releases the lock\n\n\t// writeHook is for testing, allowing other code to slip in and try to do stuff while we hold the lock.\n\tif writeHook != nil {\n\t\twriteHook()\n\t}\n\n\t// Read current manifest (if it exists). The closure ensures that the file is closed before moving on, so we can rename over it later if need be.\n\tmanifestPath := filepath.Join(fm.dir, manifestFileName)\n\tupstream := func() manifestContents {\n\t\tif f := openIfExists(manifestPath); f != nil {\n\t\t\tdefer checkClose(f)\n\n\t\t\tupstream := parseManifest(f)\n\t\t\td.PanicIfFalse(constants.NomsVersion == upstream.vers)\n\t\t\treturn upstream\n\t\t}\n\t\td.Chk.True(lastLock == addr{})\n\t\treturn manifestContents{}\n\t}()\n\n\tif lastLock != upstream.lock {\n\t\treturn upstream\n\t}\n\trerr := os.Rename(tempManifestPath, manifestPath)\n\td.PanicIfError(rerr)\n\treturn newContents\n}\n\nfunc writeManifest(temp io.Writer, contents manifestContents) {\n\tstrs := make([]string, 2*len(contents.specs)+4)\n\tstrs[0], strs[1], strs[2], strs[3] = StorageVersion, contents.vers, contents.lock.String(), contents.root.String()\n\ttableInfo := strs[4:]\n\tformatSpecs(contents.specs, tableInfo)\n\t_, err := io.WriteString(temp, strings.Join(strs, \":\"))\n\td.PanicIfError(err)\n}\n\nfunc checkClose(c io.Closer) {\n\td.PanicIfError(c.Close())\n}\n\nfunc flock(lockFilePath string) io.Closer {\n\tl, err := os.Create(lockFilePath)\n\td.PanicIfError(err)\n\td.PanicIfError(unix.Flock(int(l.Fd()), unix.LOCK_EX))\n\treturn l\n}\n" }, { "path": "go/nbs/file_manifest_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"os/exec\"\n\t\"path/filepath\"\n\t\"runtime\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc makeFileManifestTempDir(t *testing.T) fileManifest {\n\tdir, err := ioutil.TempDir(\"\", \"\")\n\tassert.NoError(t, err)\n\treturn fileManifest{dir: dir} //, cache: newManifestCache(defaultManifestCacheSize)}\n}\n\nfunc TestFileManifestLoadIfExists(t *testing.T) {\n\tassert := assert.New(t)\n\tfm := makeFileManifestTempDir(t)\n\tdefer os.RemoveAll(fm.dir)\n\tstats := &Stats{}\n\n\texists, upstream := fm.ParseIfExists(stats, nil)\n\tassert.False(exists)\n\n\t// Simulate another process writing a manifest (with an old Noms version).\n\tjerk := computeAddr([]byte(\"jerk\"))\n\tnewRoot := hash.Of([]byte(\"new root\"))\n\ttableName := hash.Of([]byte(\"table1\"))\n\terr := clobberManifest(fm.dir, strings.Join([]string{StorageVersion, \"0\", jerk.String(), newRoot.String(), tableName.String(), \"0\"}, \":\"))\n\tassert.NoError(err)\n\n\t// ParseIfExists should now reflect the manifest written above.\n\texists, upstream = fm.ParseIfExists(stats, nil)\n\tassert.True(exists)\n\tassert.Equal(\"0\", upstream.vers)\n\tassert.Equal(jerk, upstream.lock)\n\tassert.Equal(newRoot, upstream.root)\n\tif assert.Len(upstream.specs, 1) {\n\t\tassert.Equal(tableName.String(), upstream.specs[0].name.String())\n\t\tassert.Equal(uint32(0), upstream.specs[0].chunkCount)\n\t}\n}\n\nfunc TestFileManifestLoadIfExistsHoldsLock(t *testing.T) {\n\tassert := assert.New(t)\n\tfm := makeFileManifestTempDir(t)\n\tdefer os.RemoveAll(fm.dir)\n\tstats := &Stats{}\n\n\t// Simulate another process writing a manifest.\n\tlock := computeAddr([]byte(\"locker\"))\n\tnewRoot := hash.Of([]byte(\"new root\"))\n\ttableName := hash.Of([]byte(\"table1\"))\n\terr := clobberManifest(fm.dir, strings.Join([]string{StorageVersion, constants.NomsVersion, lock.String(), newRoot.String(), tableName.String(), \"0\"}, \":\"))\n\tassert.NoError(err)\n\n\t// ParseIfExists should now reflect the manifest written above.\n\texists, upstream := fm.ParseIfExists(stats, func() {\n\t\t// This should fail to get the lock, and therefore _not_ clobber the manifest.\n\t\tlock := computeAddr([]byte(\"newlock\"))\n\t\tbadRoot := hash.Of([]byte(\"bad root\"))\n\t\tb, err := tryClobberManifest(fm.dir, strings.Join([]string{StorageVersion, \"0\", lock.String(), badRoot.String(), tableName.String(), \"0\"}, \":\"))\n\t\tassert.NoError(err, string(b))\n\t})\n\n\tassert.True(exists)\n\tassert.Equal(constants.NomsVersion, upstream.vers)\n\tassert.Equal(newRoot, upstream.root)\n\tif assert.Len(upstream.specs, 1) {\n\t\tassert.Equal(tableName.String(), upstream.specs[0].name.String())\n\t\tassert.Equal(uint32(0), upstream.specs[0].chunkCount)\n\t}\n}\n\nfunc TestFileManifestUpdateWontClobberOldVersion(t *testing.T) {\n\tassert := assert.New(t)\n\tfm := makeFileManifestTempDir(t)\n\tdefer os.RemoveAll(fm.dir)\n\tstats := &Stats{}\n\n\t// Simulate another process having already put old Noms data in dir/.\n\terr := clobberManifest(fm.dir, strings.Join([]string{StorageVersion, \"0\", addr{}.String(), hash.Hash{}.String()}, \":\"))\n\tassert.NoError(err)\n\n\tassert.Panics(func() { fm.Update(addr{}, manifestContents{}, stats, nil) })\n}\n\nfunc TestFileManifestUpdateEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\tfm := makeFileManifestTempDir(t)\n\tdefer os.RemoveAll(fm.dir)\n\tstats := &Stats{}\n\n\tl := computeAddr([]byte{0x01})\n\tupstream := fm.Update(addr{}, manifestContents{vers: constants.NomsVersion, lock: l}, stats, nil)\n\tassert.Equal(l, upstream.lock)\n\tassert.True(upstream.root.IsEmpty())\n\tassert.Empty(upstream.specs)\n\n\tfm2 := fileManifest{fm.dir} // Open existent, but empty manifest\n\texists, upstream := fm2.ParseIfExists(stats, nil)\n\tassert.True(exists)\n\tassert.Equal(l, upstream.lock)\n\tassert.True(upstream.root.IsEmpty())\n\tassert.Empty(upstream.specs)\n\n\tl2 := computeAddr([]byte{0x02})\n\tupstream = fm2.Update(l, manifestContents{vers: constants.NomsVersion, lock: l2}, stats, nil)\n\tassert.Equal(l2, upstream.lock)\n\tassert.True(upstream.root.IsEmpty())\n\tassert.Empty(upstream.specs)\n}\n\nfunc TestFileManifestUpdate(t *testing.T) {\n\tassert := assert.New(t)\n\tfm := makeFileManifestTempDir(t)\n\tdefer os.RemoveAll(fm.dir)\n\tstats := &Stats{}\n\n\t// First, test winning the race against another process.\n\tcontents := manifestContents{\n\t\tvers: constants.NomsVersion,\n\t\tlock: computeAddr([]byte(\"locker\")),\n\t\troot: hash.Of([]byte(\"new root\")),\n\t\tspecs: []tableSpec{{computeAddr([]byte(\"a\")), 3}},\n\t}\n\tupstream := fm.Update(addr{}, contents, stats, func() {\n\t\t// This should fail to get the lock, and therefore _not_ clobber the manifest. So the Update should succeed.\n\t\tlock := computeAddr([]byte(\"nolock\"))\n\t\tnewRoot2 := hash.Of([]byte(\"noroot\"))\n\t\tb, err := tryClobberManifest(fm.dir, strings.Join([]string{StorageVersion, constants.NomsVersion, lock.String(), newRoot2.String()}, \":\"))\n\t\tassert.NoError(err, string(b))\n\t})\n\tassert.Equal(contents.lock, upstream.lock)\n\tassert.Equal(contents.root, upstream.root)\n\tassert.Equal(contents.specs, upstream.specs)\n\n\t// Now, test the case where the optimistic lock fails, and someone else updated the root since last we checked.\n\tcontents2 := manifestContents{lock: computeAddr([]byte(\"locker 2\")), root: hash.Of([]byte(\"new root 2\"))}\n\tupstream = fm.Update(addr{}, contents2, stats, nil)\n\tassert.Equal(contents.lock, upstream.lock)\n\tassert.Equal(contents.root, upstream.root)\n\tassert.Equal(contents.specs, upstream.specs)\n\tupstream = fm.Update(upstream.lock, contents2, stats, nil)\n\tassert.Equal(contents2.lock, upstream.lock)\n\tassert.Equal(contents2.root, upstream.root)\n\tassert.Empty(upstream.specs)\n\n\t// Now, test the case where the optimistic lock fails because someone else updated only the tables since last we checked\n\tjerkLock := computeAddr([]byte(\"jerk\"))\n\ttableName := computeAddr([]byte(\"table1\"))\n\terr := clobberManifest(fm.dir, strings.Join([]string{StorageVersion, constants.NomsVersion, jerkLock.String(), contents2.root.String(), tableName.String(), \"1\"}, \":\"))\n\tassert.NoError(err)\n\n\tcontents3 := manifestContents{lock: computeAddr([]byte(\"locker 3\")), root: hash.Of([]byte(\"new root 3\"))}\n\tupstream = fm.Update(upstream.lock, contents3, stats, nil)\n\tassert.Equal(jerkLock, upstream.lock)\n\tassert.Equal(contents2.root, upstream.root)\n\tassert.Equal([]tableSpec{{tableName, 1}}, upstream.specs)\n}\n\n// tryClobberManifest simulates another process trying to access dir/manifestFileName concurrently. To avoid deadlock, it does a non-blocking lock of dir/lockFileName. If it can get the lock, it clobbers the manifest.\nfunc tryClobberManifest(dir, contents string) ([]byte, error) {\n\treturn runClobber(dir, contents)\n}\n\n// clobberManifest simulates another process writing dir/manifestFileName concurrently. It ignores the lock file, so it's up to the caller to ensure correctness.\nfunc clobberManifest(dir, contents string) error {\n\tif err := ioutil.WriteFile(filepath.Join(dir, lockFileName), nil, 0666); err != nil {\n\t\treturn err\n\t}\n\treturn ioutil.WriteFile(filepath.Join(dir, manifestFileName), []byte(contents), 0666)\n}\n\nfunc runClobber(dir, contents string) ([]byte, error) {\n\t_, filename, _, _ := runtime.Caller(1)\n\tclobber := filepath.Join(filepath.Dir(filename), \"test/manifest_clobber.go\")\n\tmkPath := func(f string) string {\n\t\treturn filepath.Join(dir, f)\n\t}\n\n\tc := exec.Command(\"go\", \"run\", clobber, mkPath(lockFileName), mkPath(manifestFileName), contents)\n\treturn c.CombinedOutput()\n}\n" }, { "path": "go/nbs/file_table_persister.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nconst tempTablePrefix = \"nbs_table_\"\n\nfunc newFSTablePersister(dir string, fc *fdCache, indexCache *indexCache) tablePersister {\n\td.PanicIfTrue(fc == nil)\n\treturn &fsTablePersister{dir, fc, indexCache}\n}\n\ntype fsTablePersister struct {\n\tdir string\n\tfc *fdCache\n\tindexCache *indexCache\n}\n\nfunc (ftp *fsTablePersister) Open(name addr, chunkCount uint32, stats *Stats) chunkSource {\n\treturn newMmapTableReader(ftp.dir, name, chunkCount, ftp.indexCache, ftp.fc)\n}\n\nfunc (ftp *fsTablePersister) Persist(mt *memTable, haver chunkReader, stats *Stats) chunkSource {\n\tname, data, chunkCount := mt.write(haver, stats)\n\treturn ftp.persistTable(name, data, chunkCount, stats)\n}\n\nfunc (ftp *fsTablePersister) persistTable(name addr, data []byte, chunkCount uint32, stats *Stats) chunkSource {\n\tif chunkCount == 0 {\n\t\treturn emptyChunkSource{}\n\t}\n\ttempName := func() string {\n\t\ttemp, err := ioutil.TempFile(ftp.dir, tempTablePrefix)\n\t\td.PanicIfError(err)\n\t\tdefer checkClose(temp)\n\t\tio.Copy(temp, bytes.NewReader(data))\n\t\tindex := parseTableIndex(data)\n\t\tif ftp.indexCache != nil {\n\t\t\tftp.indexCache.lockEntry(name)\n\t\t\tdefer ftp.indexCache.unlockEntry(name)\n\t\t\tftp.indexCache.put(name, index)\n\t\t}\n\t\treturn temp.Name()\n\t}()\n\terr := os.Rename(tempName, filepath.Join(ftp.dir, name.String()))\n\td.PanicIfError(err)\n\treturn ftp.Open(name, chunkCount, stats)\n}\n\nfunc (ftp *fsTablePersister) ConjoinAll(sources chunkSources, stats *Stats) chunkSource {\n\tplan := planConjoin(sources, stats)\n\n\tif plan.chunkCount == 0 {\n\t\treturn emptyChunkSource{}\n\t}\n\n\tname := nameFromSuffixes(plan.suffixes())\n\ttempName := func() string {\n\t\ttemp, err := ioutil.TempFile(ftp.dir, tempTablePrefix)\n\t\td.PanicIfError(err)\n\t\tdefer checkClose(temp)\n\n\t\tfor _, sws := range plan.sources {\n\t\t\tr := sws.source.reader()\n\t\t\tn, err := io.CopyN(temp, r, int64(sws.dataLen))\n\t\t\td.PanicIfError(err)\n\t\t\td.PanicIfFalse(uint64(n) == sws.dataLen)\n\t\t}\n\t\t_, err = temp.Write(plan.mergedIndex)\n\t\td.PanicIfError(err)\n\n\t\tindex := parseTableIndex(plan.mergedIndex)\n\t\tif ftp.indexCache != nil {\n\t\t\tftp.indexCache.put(name, index)\n\t\t}\n\t\treturn temp.Name()\n\t}()\n\n\terr := os.Rename(tempName, filepath.Join(ftp.dir, name.String()))\n\td.PanicIfError(err)\n\n\treturn ftp.Open(name, plan.chunkCount, stats)\n}\n" }, { "path": "go/nbs/file_table_persister_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"crypto/rand\"\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestFSTableCacheOnOpen(t *testing.T) {\n\tassert := assert.New(t)\n\tdir := makeTempDir(t)\n\tdefer os.RemoveAll(dir)\n\n\tnames := []addr{}\n\tcacheSize := 2\n\tfc := newFDCache(cacheSize)\n\tdefer fc.Drop()\n\tfts := newFSTablePersister(dir, fc, nil)\n\n\t// Create some tables manually, load them into the cache, and then blow them away\n\tfunc() {\n\t\tfor i := 0; i < cacheSize; i++ {\n\t\t\tname, err := writeTableData(dir, []byte{byte(i)})\n\t\t\tassert.NoError(err)\n\t\t\tnames = append(names, name)\n\t\t}\n\t\tfor _, name := range names {\n\t\t\tfts.Open(name, 1, nil)\n\t\t}\n\t\tremoveTables(dir, names...)\n\t}()\n\n\t// Tables should still be cached, even though they're gone from disk\n\tfor i, name := range names {\n\t\tsrc := fts.Open(name, 1, nil)\n\t\th := computeAddr([]byte{byte(i)})\n\t\tassert.True(src.has(h))\n\t}\n\n\t// Kick a table out of the cache\n\tname, err := writeTableData(dir, []byte{0xff})\n\tassert.NoError(err)\n\tfts.Open(name, 1, nil)\n\n\tpresent := fc.reportEntries()\n\t// Since 0 refcount entries are evicted randomly, the only thing we can validate is that fc remains at its target size\n\tassert.Len(present, cacheSize)\n}\n\nfunc makeTempDir(t *testing.T) string {\n\tdir, err := ioutil.TempDir(\"\", \"\")\n\tassert.NoError(t, err)\n\treturn dir\n}\n\nfunc writeTableData(dir string, chunx ...[]byte) (name addr, err error) {\n\tvar tableData []byte\n\ttableData, name = buildTable(chunx)\n\terr = ioutil.WriteFile(filepath.Join(dir, name.String()), tableData, 0666)\n\treturn\n}\n\nfunc removeTables(dir string, names ...addr) error {\n\tfor _, name := range names {\n\t\tif err := os.Remove(filepath.Join(dir, name.String())); err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc contains(s sort.StringSlice, e string) bool {\n\tfor _, c := range s {\n\t\tif c == e {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc TestFSTablePersisterPersist(t *testing.T) {\n\tassert := assert.New(t)\n\tdir := makeTempDir(t)\n\tdefer os.RemoveAll(dir)\n\tfc := newFDCache(defaultMaxTables)\n\tdefer fc.Drop()\n\tfts := newFSTablePersister(dir, fc, nil)\n\n\tsrc, err := persistTableData(fts, testChunks...)\n\tassert.NoError(err)\n\tif assert.True(src.count() > 0) {\n\t\tbuff, err := ioutil.ReadFile(filepath.Join(dir, src.hash().String()))\n\t\tassert.NoError(err)\n\t\ttr := newTableReader(parseTableIndex(buff), tableReaderAtFromBytes(buff), fileBlockSize)\n\t\tassertChunksInReader(testChunks, tr, assert)\n\t}\n}\n\nfunc persistTableData(p tablePersister, chunx ...[]byte) (src chunkSource, err error) {\n\tmt := newMemTable(testMemTableSize)\n\tfor _, c := range chunx {\n\t\tif !mt.addChunk(computeAddr(c), c) {\n\t\t\treturn nil, fmt.Errorf(\"memTable too full to add %s\", computeAddr(c))\n\t\t}\n\t}\n\treturn p.Persist(mt, nil, &Stats{}), nil\n}\n\nfunc TestFSTablePersisterPersistNoData(t *testing.T) {\n\tassert := assert.New(t)\n\tmt := newMemTable(testMemTableSize)\n\texistingTable := newMemTable(testMemTableSize)\n\n\tfor _, c := range testChunks {\n\t\tassert.True(mt.addChunk(computeAddr(c), c))\n\t\tassert.True(existingTable.addChunk(computeAddr(c), c))\n\t}\n\n\tdir := makeTempDir(t)\n\tdefer os.RemoveAll(dir)\n\tfc := newFDCache(defaultMaxTables)\n\tdefer fc.Drop()\n\tfts := newFSTablePersister(dir, fc, nil)\n\n\tsrc := fts.Persist(mt, existingTable, &Stats{})\n\tassert.True(src.count() == 0)\n\n\t_, err := os.Stat(filepath.Join(dir, src.hash().String()))\n\tassert.True(os.IsNotExist(err), \"%v\", err)\n}\n\nfunc TestFSTablePersisterCacheOnPersist(t *testing.T) {\n\tassert := assert.New(t)\n\tdir := makeTempDir(t)\n\tfc := newFDCache(1)\n\tdefer fc.Drop()\n\tfts := newFSTablePersister(dir, fc, nil)\n\tdefer os.RemoveAll(dir)\n\n\tvar name addr\n\tfunc() {\n\t\tsrc, err := persistTableData(fts, testChunks...)\n\t\tassert.NoError(err)\n\t\tname = src.hash()\n\t\tremoveTables(dir, name)\n\t}()\n\n\t// Table should still be cached, even though it's gone from disk\n\tsrc := fts.Open(name, uint32(len(testChunks)), nil)\n\tassertChunksInReader(testChunks, src, assert)\n\n\t// Evict |name| from cache\n\tsrc, err := persistTableData(fts, []byte{0xff})\n\tassert.NoError(err)\n\n\tpresent := fc.reportEntries()\n\t// Since 0 refcount entries are evicted randomly, the only thing we can validate is that fc remains at its target size\n\tassert.Len(present, 1)\n}\n\nfunc TestFSTablePersisterConjoinAll(t *testing.T) {\n\tassert := assert.New(t)\n\tassert.True(len(testChunks) > 1, \"Whoops, this test isn't meaningful\")\n\tsources := make(chunkSources, len(testChunks))\n\n\tdir := makeTempDir(t)\n\tdefer os.RemoveAll(dir)\n\tfc := newFDCache(len(sources))\n\tdefer fc.Drop()\n\tfts := newFSTablePersister(dir, fc, nil)\n\n\tfor i, c := range testChunks {\n\t\trandChunk := make([]byte, (i+1)*13)\n\t\t_, err := rand.Read(randChunk)\n\t\tassert.NoError(err)\n\t\tname, err := writeTableData(dir, c, randChunk)\n\t\tassert.NoError(err)\n\t\tsources[i] = fts.Open(name, 2, nil)\n\t}\n\n\tsrc := fts.ConjoinAll(sources, &Stats{})\n\n\tif assert.True(src.count() > 0) {\n\t\tbuff, err := ioutil.ReadFile(filepath.Join(dir, src.hash().String()))\n\t\tassert.NoError(err)\n\t\ttr := newTableReader(parseTableIndex(buff), tableReaderAtFromBytes(buff), fileBlockSize)\n\t\tassertChunksInReader(testChunks, tr, assert)\n\t}\n\n\tpresent := fc.reportEntries()\n\t// Since 0 refcount entries are evicted randomly, the only thing we can validate is that fc remains at its target size\n\tassert.Len(present, len(sources))\n}\n\nfunc TestFSTablePersisterConjoinAllDups(t *testing.T) {\n\tassert := assert.New(t)\n\tdir := makeTempDir(t)\n\tdefer os.RemoveAll(dir)\n\tfc := newFDCache(defaultMaxTables)\n\tdefer fc.Drop()\n\tfts := newFSTablePersister(dir, fc, nil)\n\n\treps := 3\n\tsources := make(chunkSources, reps)\n\tfor i := 0; i < reps; i++ {\n\t\tmt := newMemTable(1 << 10)\n\t\tfor _, c := range testChunks {\n\t\t\tmt.addChunk(computeAddr(c), c)\n\t\t}\n\t\tsources[i] = fts.Persist(mt, nil, &Stats{})\n\t}\n\tsrc := fts.ConjoinAll(sources, &Stats{})\n\n\tif assert.True(src.count() > 0) {\n\t\tbuff, err := ioutil.ReadFile(filepath.Join(dir, src.hash().String()))\n\t\tassert.NoError(err)\n\t\ttr := newTableReader(parseTableIndex(buff), tableReaderAtFromBytes(buff), fileBlockSize)\n\t\tassertChunksInReader(testChunks, tr, assert)\n\t\tassert.EqualValues(reps*len(testChunks), tr.count())\n\t}\n}\n" }, { "path": "go/nbs/fs_table_cache.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"errors\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/util/sizecache\"\n)\n\ntype tableCache interface {\n\tcheckout(h addr) io.ReaderAt\n\tcheckin(h addr)\n\tstore(h addr, data io.Reader, size uint64)\n}\n\ntype fsTableCache struct {\n\tdir string\n\tcache *sizecache.SizeCache\n\tfd *fdCache\n}\n\nfunc newFSTableCache(dir string, cacheSize uint64, maxOpenFds int) *fsTableCache {\n\tftc := &fsTableCache{dir: dir, fd: newFDCache(maxOpenFds)}\n\tftc.cache = sizecache.NewWithExpireCallback(cacheSize, func(elm interface{}) {\n\t\tftc.expire(elm.(addr))\n\t})\n\n\tftc.init(maxOpenFds)\n\treturn ftc\n}\n\nfunc (ftc *fsTableCache) init(concurrency int) {\n\ttype finfo struct {\n\t\tpath string\n\t\th addr\n\t\tsize uint64\n\t}\n\tinfos := make(chan finfo)\n\terrc := make(chan error, 1)\n\tgo func() {\n\t\tisTableFile := func(info os.FileInfo) bool {\n\t\t\treturn info.Mode().IsRegular() && ValidateAddr(info.Name())\n\t\t}\n\t\tisTempTableFile := func(info os.FileInfo) bool {\n\t\t\treturn info.Mode().IsRegular() && strings.HasPrefix(info.Name(), tempTablePrefix)\n\t\t}\n\t\tdefer close(errc)\n\t\tdefer close(infos)\n\t\t// No select needed for this send, since errc is buffered.\n\t\terrc <- filepath.Walk(ftc.dir, func(path string, info os.FileInfo, err error) error {\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tif path == ftc.dir {\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\tif isTempTableFile(info) {\n\t\t\t\tos.Remove(path)\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\tif !isTableFile(info) {\n\t\t\t\treturn errors.New(path + \" is not a table file; cache dir must contain only table files\")\n\t\t\t}\n\t\t\tinfos <- finfo{path, ParseAddr([]byte(info.Name())), uint64(info.Size())}\n\t\t\treturn nil\n\t\t})\n\t}()\n\n\twg := sync.WaitGroup{}\n\twg.Add(concurrency)\n\tfor i := 0; i < concurrency; i++ {\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\t\t\tfor info := range infos {\n\t\t\t\tftc.cache.Add(info.h, info.size, true)\n\t\t\t\tftc.fd.RefFile(info.path)\n\t\t\t\tftc.fd.UnrefFile(info.path)\n\t\t\t}\n\t\t}()\n\t}\n\twg.Wait()\n\td.PanicIfError(<-errc)\n}\n\nfunc (ftc *fsTableCache) checkout(h addr) io.ReaderAt {\n\tif _, ok := ftc.cache.Get(h); !ok {\n\t\treturn nil\n\t}\n\n\tif fd, err := ftc.fd.RefFile(filepath.Join(ftc.dir, h.String())); err == nil {\n\t\treturn fd\n\t}\n\treturn nil\n}\n\nfunc (ftc *fsTableCache) checkin(h addr) {\n\tftc.fd.UnrefFile(filepath.Join(ftc.dir, h.String()))\n}\n\nfunc (ftc *fsTableCache) store(h addr, data io.Reader, size uint64) {\n\tpath := filepath.Join(ftc.dir, h.String())\n\ttempName := func() string {\n\t\ttemp, err := ioutil.TempFile(ftc.dir, tempTablePrefix)\n\t\td.PanicIfError(err)\n\t\tdefer checkClose(temp)\n\t\tio.Copy(temp, data)\n\t\treturn temp.Name()\n\t}()\n\n\terr := os.Rename(tempName, path)\n\td.PanicIfError(err)\n\n\tftc.cache.Add(h, size, true)\n\n\tftc.fd.RefFile(path) // Prime the file in the fd cache\n\tftc.fd.UnrefFile(path)\n}\n\nfunc (ftc *fsTableCache) expire(h addr) {\n\terr := os.Remove(filepath.Join(ftc.dir, h.String()))\n\td.PanicIfError(err)\n}\n" }, { "path": "go/nbs/fs_table_cache_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestFSTableCache(t *testing.T) {\n\tdatas := [][]byte{[]byte(\"hello\"), []byte(\"world\"), []byte(\"goodbye\")}\n\tsort.SliceStable(datas, func(i, j int) bool { return len(datas[i]) < len(datas[j]) })\n\n\tt.Run(\"ExpireLRU\", func(t *testing.T) {\n\t\tt.Parallel()\n\t\tdir := makeTempDir(t)\n\t\tdefer os.RemoveAll(dir)\n\n\t\tsum := 0\n\t\tfor _, s := range datas[1:] {\n\t\t\tsum += len(s)\n\t\t}\n\n\t\ttc := newFSTableCache(dir, uint64(sum), len(datas))\n\t\tfor _, d := range datas {\n\t\t\ttc.store(computeAddr(d), bytes.NewReader(d), uint64(len(d)))\n\t\t}\n\n\t\texpiredName := computeAddr(datas[0])\n\t\tassert.Nil(t, tc.checkout(expiredName))\n\t\t_, fserr := os.Stat(filepath.Join(dir, expiredName.String()))\n\t\tassert.True(t, os.IsNotExist(fserr))\n\n\t\tfor _, d := range datas[1:] {\n\t\t\tname := computeAddr(d)\n\t\t\tr := tc.checkout(name)\n\t\t\tassert.NotNil(t, r)\n\t\t\tassertDataInReaderAt(t, d, r)\n\t\t\t_, fserr := os.Stat(filepath.Join(dir, name.String()))\n\t\t\tassert.False(t, os.IsNotExist(fserr))\n\t\t}\n\t})\n\n\tt.Run(\"Init\", func(t *testing.T) {\n\t\tt.Run(\"Success\", func(t *testing.T) {\n\t\t\tt.Parallel()\n\t\t\tdir := makeTempDir(t)\n\t\t\tdefer os.RemoveAll(dir)\n\t\t\tassert := assert.New(t)\n\n\t\t\tnames := []addr{}\n\t\t\tfor i := byte(0); i < 4; i++ {\n\t\t\t\tname := computeAddr([]byte{i})\n\t\t\t\tassert.NoError(ioutil.WriteFile(filepath.Join(dir, name.String()), nil, 0666))\n\t\t\t\tnames = append(names, name)\n\t\t\t}\n\t\t\tvar ftc *fsTableCache\n\t\t\tassert.NotPanics(func() { ftc = newFSTableCache(dir, 1024, 4) })\n\t\t\tassert.NotNil(ftc)\n\n\t\t\tfor _, name := range names {\n\t\t\t\tassert.NotNil(ftc.checkout(name))\n\t\t\t}\n\t\t})\n\n\t\tt.Run(\"BadFile\", func(t *testing.T) {\n\t\t\tt.Parallel()\n\t\t\tdir := makeTempDir(t)\n\t\t\tdefer os.RemoveAll(dir)\n\n\t\t\tassert.NoError(t, ioutil.WriteFile(filepath.Join(dir, \"boo\"), nil, 0666))\n\t\t\tassert.Panics(t, func() { newFSTableCache(dir, 1024, 4) })\n\t\t})\n\n\t\tt.Run(\"ClearTempFile\", func(t *testing.T) {\n\t\t\tt.Parallel()\n\t\t\tdir := makeTempDir(t)\n\t\t\tdefer os.RemoveAll(dir)\n\n\t\t\ttempFile := filepath.Join(dir, tempTablePrefix+\"boo\")\n\t\t\tassert.NoError(t, ioutil.WriteFile(tempFile, nil, 0666))\n\t\t\tassert.NotPanics(t, func() { newFSTableCache(dir, 1024, 4) })\n\t\t\t_, fserr := os.Stat(tempFile)\n\t\t\tassert.True(t, os.IsNotExist(fserr))\n\t\t})\n\n\t\tt.Run(\"Dir\", func(t *testing.T) {\n\t\t\tt.Parallel()\n\t\t\tdir := makeTempDir(t)\n\t\t\tdefer os.RemoveAll(dir)\n\t\t\tassert.NoError(t, os.Mkdir(filepath.Join(dir, \"sub\"), 0777))\n\t\t\tassert.Panics(t, func() { newFSTableCache(dir, 1024, 4) })\n\t\t})\n\t})\n}\n\nfunc assertDataInReaderAt(t *testing.T, data []byte, r io.ReaderAt) {\n\tp := make([]byte, len(data))\n\tn, err := r.ReadAt(p, 0)\n\tassert.NoError(t, err)\n\tassert.Equal(t, len(data), n)\n\tassert.Equal(t, data, p)\n}\n" }, { "path": "go/nbs/manifest.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"crypto/sha512\"\n\t\"strconv\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype manifest interface {\n\t// Name returns a stable, unique identifier for the store this manifest describes.\n\tName() string\n\n\t// ParseIfExists extracts and returns values from a NomsBlockStore\n\t// manifest, if one exists. Concrete implementations are responsible for\n\t// defining how to find and parse the desired manifest, e.g. a\n\t// particularly-named file in a given directory. Implementations are also\n\t// responsible for managing whatever concurrency guarantees they require\n\t// for correctness. If the manifest exists, |exists| is set to true and\n\t// manifest data is returned, including the version of the Noms data in\n\t// the store, the root root hash.Hash of the store, and a tableSpec\n\t// describing every table that comprises the store.\n\t// If the manifest doesn't exist, |exists| is set to false and the other\n\t// return values are undefined. The |readHook| parameter allows race\n\t// condition testing. If it is non-nil, it will be invoked while the\n\t// implementation is guaranteeing exclusive access to the manifest.\n\tParseIfExists(stats *Stats, readHook func()) (exists bool, contents manifestContents)\n\n\tmanifestUpdater\n}\n\ntype manifestUpdater interface {\n\t// Update optimistically tries to write a new manifest containing\n\t// |newContents|. If |lastLock| matches the lock hash in the currently\n\t// persisted manifest (logically, the lock that would be returned by\n\t// ParseIfExists), then Update succeeds and subsequent calls to both\n\t// Update and ParseIfExists will reflect a manifest containing\n\t// |newContents|. If not, Update fails. Regardless, the returned\n\t// manifestContents will reflect the current state of the world. Callers\n\t// should check that the returned root == the proposed root and, if not,\n\t// merge any desired new table information with the contents of the\n\t// returned []tableSpec before trying again.\n\t// Concrete implementations are responsible for ensuring that concurrent\n\t// Update calls (and ParseIfExists calls) are correct.\n\t// If writeHook is non-nil, it will be invoked while the implementation is\n\t// guaranteeing exclusive access to the manifest. This allows for testing\n\t// of race conditions.\n\tUpdate(lastLock addr, newContents manifestContents, stats *Stats, writeHook func()) manifestContents\n}\n\ntype manifestContents struct {\n\tvers string\n\tlock addr\n\troot hash.Hash\n\tspecs []tableSpec\n}\n\nfunc (mc manifestContents) size() (size uint64) {\n\tsize += uint64(len(mc.vers)) + addrSize + hash.ByteLen\n\tfor _, sp := range mc.specs {\n\t\tsize += uint64(len(sp.name)) + uint32Size // for sp.chunkCount\n\t}\n\treturn\n}\n\nfunc newManifestLocks() *manifestLocks {\n\treturn &manifestLocks{map[string]struct{}{}, map[string]struct{}{}, sync.NewCond(&sync.Mutex{})}\n}\n\ntype manifestLocks struct {\n\tupdating map[string]struct{}\n\tfetching map[string]struct{}\n\tcond *sync.Cond\n}\n\nfunc (ml *manifestLocks) lockForFetch(db string) {\n\tlockByName(db, ml.cond, ml.fetching)\n}\n\nfunc (ml *manifestLocks) unlockForFetch(db string) {\n\tunlockByName(db, ml.cond, ml.fetching)\n}\n\nfunc (ml *manifestLocks) lockForUpdate(db string) {\n\tlockByName(db, ml.cond, ml.updating)\n}\n\nfunc (ml *manifestLocks) unlockForUpdate(db string) {\n\tunlockByName(db, ml.cond, ml.updating)\n}\n\nfunc lockByName(db string, c *sync.Cond, locks map[string]struct{}) {\n\tc.L.Lock()\n\tdefer c.L.Unlock()\n\n\tfor {\n\t\tif _, inProgress := locks[db]; !inProgress {\n\t\t\tlocks[db] = struct{}{}\n\t\t\tbreak\n\t\t}\n\t\tc.Wait()\n\t}\n}\n\nfunc unlockByName(db string, c *sync.Cond, locks map[string]struct{}) {\n\tc.L.Lock()\n\tdefer c.L.Unlock()\n\n\t_, ok := locks[db]\n\td.PanicIfFalse(ok)\n\tdelete(locks, db)\n\n\tc.Broadcast()\n}\n\ntype manifestManager struct {\n\tm manifest\n\tcache *manifestCache\n\tlocks *manifestLocks\n}\n\nfunc (mm manifestManager) lockOutFetch() {\n\tmm.locks.lockForFetch(mm.Name())\n}\n\nfunc (mm manifestManager) allowFetch() {\n\tmm.locks.unlockForFetch(mm.Name())\n}\n\nfunc (mm manifestManager) LockForUpdate() {\n\tmm.locks.lockForUpdate(mm.Name())\n}\n\nfunc (mm manifestManager) UnlockForUpdate() {\n\tmm.locks.unlockForUpdate(mm.Name())\n}\n\nfunc (mm manifestManager) updateWillFail(lastLock addr) (cached manifestContents, doomed bool) {\n\tif upstream, _, hit := mm.cache.Get(mm.Name()); hit {\n\t\tif lastLock != upstream.lock {\n\t\t\tdoomed, cached = true, upstream\n\t\t}\n\t}\n\treturn\n}\n\nfunc (mm manifestManager) Fetch(stats *Stats) (exists bool, contents manifestContents) {\n\tentryTime := time.Now()\n\n\tmm.lockOutFetch()\n\tdefer mm.allowFetch()\n\n\tcached, t, hit := mm.cache.Get(mm.Name())\n\n\tif hit && t.After(entryTime) {\n\t\t// Cache contains a manifest which is newer than entry time.\n\t\treturn true, cached\n\t}\n\n\tt = time.Now()\n\texists, contents = mm.m.ParseIfExists(stats, nil)\n\tmm.cache.Put(mm.Name(), contents, t)\n\treturn\n}\n\n// Callers MUST protect uses of Update with Lock/UnlockForUpdate.\n// Update does not call Lock/UnlockForUpdate() on its own because it is\n// intended to be used in a larger critical section along with updateWillFail.\nfunc (mm manifestManager) Update(lastLock addr, newContents manifestContents, stats *Stats, writeHook func()) manifestContents {\n\tif upstream, _, hit := mm.cache.Get(mm.Name()); hit {\n\t\tif lastLock != upstream.lock {\n\t\t\treturn upstream\n\t\t}\n\t}\n\tt := time.Now()\n\n\tmm.lockOutFetch()\n\tdefer mm.allowFetch()\n\n\tcontents := mm.m.Update(lastLock, newContents, stats, writeHook)\n\tmm.cache.Put(mm.Name(), contents, t)\n\treturn contents\n}\n\nfunc (mm manifestManager) Name() string {\n\treturn mm.m.Name()\n}\n\ntype tableSpec struct {\n\tname addr\n\tchunkCount uint32\n}\n\nfunc parseSpecs(tableInfo []string) []tableSpec {\n\tspecs := make([]tableSpec, len(tableInfo)/2)\n\tfor i := range specs {\n\t\tspecs[i].name = ParseAddr([]byte(tableInfo[2*i]))\n\t\tc, err := strconv.ParseUint(tableInfo[2*i+1], 10, 32)\n\t\td.PanicIfError(err)\n\t\tspecs[i].chunkCount = uint32(c)\n\t}\n\treturn specs\n}\n\nfunc formatSpecs(specs []tableSpec, tableInfo []string) {\n\td.Chk.True(len(tableInfo) == 2*len(specs))\n\tfor i, t := range specs {\n\t\ttableInfo[2*i] = t.name.String()\n\t\ttableInfo[2*i+1] = strconv.FormatUint(uint64(t.chunkCount), 10)\n\t}\n}\n\n// generateLockHash returns a hash of root and the names of all the tables in\n// specs, which should be included in all persisted manifests. When a client\n// attempts to update a manifest, it must check the lock hash in the currently\n// persisted manifest against the lock hash it saw last time it loaded the\n// contents of a manifest. If they do not match, the client must not update\n// the persisted manifest.\nfunc generateLockHash(root hash.Hash, specs []tableSpec) (lock addr) {\n\tblockHash := sha512.New()\n\tblockHash.Write(root[:])\n\tfor _, spec := range specs {\n\t\tblockHash.Write(spec.name[:])\n\t}\n\tvar h []byte\n\th = blockHash.Sum(h) // Appends hash to h\n\tcopy(lock[:], h)\n\treturn\n}\n" }, { "path": "go/nbs/manifest_cache.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"container/list\"\n\t\"time\"\n\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nfunc newManifestCache(maxSize uint64) *manifestCache {\n\treturn &manifestCache{\n\t\tmaxSize: maxSize,\n\t\tcache: map[string]manifestCacheEntry{},\n\t\tmu: &sync.Mutex{},\n\t}\n}\n\ntype manifestCacheEntry struct {\n\tlruEntry *list.Element\n\tcontents manifestContents\n\tt time.Time\n}\n\ntype manifestCache struct {\n\ttotalSize uint64\n\tmaxSize uint64\n\tmu *sync.Mutex\n\tlru list.List\n\tcache map[string]manifestCacheEntry\n}\n\n// Get() checks the searches the cache for an entry. If it exists, it moves it's\n// lru entry to the back of the queue and returns (value, true). Otherwise, it\n// returns (nil, false).\nfunc (mc *manifestCache) Get(db string) (contents manifestContents, t time.Time, present bool) {\n\tmc.mu.Lock()\n\tdefer mc.mu.Unlock()\n\n\tif entry, ok := mc.entry(db); ok {\n\t\tcontents, t, present = entry.contents, entry.t, true\n\t}\n\treturn\n}\n\n// entry() checks if the value is in the cache. If not in the cache, it returns an\n// empty manifestCacheEntry and false. It it is in the cache, it moves it to\n// to the back of lru and returns the entry and true.\nfunc (mc *manifestCache) entry(key string) (manifestCacheEntry, bool) {\n\tentry, ok := mc.cache[key]\n\tif !ok {\n\t\treturn manifestCacheEntry{}, false\n\t}\n\tmc.lru.MoveToBack(entry.lruEntry)\n\treturn entry, true\n}\n\n// Put inserts |contents| into the cache with the key |db|, replacing any\n// currently cached value. Put() will add this element to the cache at the\n// back of the queue as long it's size does not exceed maxSize. If the\n// addition of this entry causes the size of the cache to exceed maxSize, the\n// necessary entries at the front of the queue will be deleted in order to\n// keep the total cache size below maxSize. |t| must be *prior* to initiating\n// the call which read/wrote |contents|.\nfunc (mc *manifestCache) Put(db string, contents manifestContents, t time.Time) {\n\tmc.mu.Lock()\n\tdefer mc.mu.Unlock()\n\n\tif entry, ok := mc.entry(db); ok {\n\t\tmc.totalSize -= entry.contents.size()\n\t\tmc.lru.Remove(entry.lruEntry)\n\t\tdelete(mc.cache, db)\n\t}\n\n\tif contents.size() <= mc.maxSize {\n\t\tnewEl := mc.lru.PushBack(db)\n\t\tce := manifestCacheEntry{lruEntry: newEl, contents: contents, t: t}\n\t\tmc.cache[db] = ce\n\t\tmc.totalSize += ce.contents.size()\n\t\tfor el := mc.lru.Front(); el != nil && mc.totalSize > mc.maxSize; {\n\t\t\tkey1 := el.Value.(string)\n\t\t\tce, ok := mc.cache[key1]\n\t\t\tif !ok {\n\t\t\t\td.Panic(\"manifestCache is missing expected value\")\n\t\t\t}\n\t\t\tnext := el.Next()\n\t\t\tdelete(mc.cache, key1)\n\t\t\tmc.totalSize -= ce.contents.size()\n\t\t\tmc.lru.Remove(el)\n\t\t\tel = next\n\t\t}\n\t}\n}\n" }, { "path": "go/nbs/manifest_cache_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestSizeCache(t *testing.T) {\n\tdefSize := manifestContents{}.size()\n\n\tt.Run(\"GetAndPut\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\n\t\tc := newManifestCache(2 * defSize)\n\t\tt1 := time.Now()\n\t\tdbA, contentsA := \"dbA\", manifestContents{lock: computeAddr([]byte(\"lockA\"))}\n\t\tdbB, contentsB := \"dbB\", manifestContents{lock: computeAddr([]byte(\"lockB\"))}\n\n\t\tc.Put(dbA, contentsA, t1)\n\t\tc.Put(dbB, contentsB, t1)\n\n\t\tcont, _, present := c.Get(dbA)\n\t\tassert.True(present)\n\t\tassert.Equal(contentsA, cont)\n\n\t\tcont, _, present = c.Get(dbB)\n\t\tassert.True(present)\n\t\tassert.Equal(contentsB, cont)\n\t})\n\n\tt.Run(\"PutDropsLRU\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\n\t\tcapacity := uint64(5)\n\t\tc := newManifestCache(capacity * defSize)\n\t\tkeys := []string{\"db1\", \"db2\", \"db3\", \"db4\", \"db5\", \"db6\", \"db7\", \"db8\", \"db9\"}\n\t\tfor i, v := range keys {\n\t\t\tc.Put(v, manifestContents{}, time.Now())\n\t\t\texpected := uint64(i + 1)\n\t\t\tif expected >= capacity {\n\t\t\t\texpected = capacity\n\t\t\t}\n\t\t\tassert.Equal(expected*defSize, c.totalSize)\n\t\t}\n\n\t\tlru := len(keys) - int(capacity)\n\t\tfor _, db := range keys[:lru] {\n\t\t\t_, _, present := c.Get(db)\n\t\t\tassert.False(present)\n\t\t}\n\t\tfor _, db := range keys[lru:] {\n\t\t\t_, _, present := c.Get(db)\n\t\t\tassert.True(present)\n\t\t}\n\n\t\t// Bump |keys[lru]| to the back of the queue, making |keys[lru+1]| the next one to be dropped\n\t\t_, _, ok := c.Get(keys[lru])\n\t\tassert.True(ok)\n\t\tlru++\n\t\tc.Put(\"novel\", manifestContents{}, time.Now())\n\t\t_, _, ok = c.Get(keys[lru])\n\t\tassert.False(ok)\n\t\t// |keys[lru]| is gone, so |keys[lru+1]| is next\n\t\tlru++\n\n\t\t// Putting a bigger value will dump multiple existing entries\n\t\tc.Put(\"big\", manifestContents{vers: \"big version\"}, time.Now())\n\t\t_, _, ok = c.Get(keys[lru])\n\t\tassert.False(ok)\n\t\tlru++\n\t\t_, _, ok = c.Get(keys[lru])\n\t\tassert.False(ok)\n\t\tlru++\n\n\t\t// Make sure expected stuff is still in the cache\n\t\tfor i := lru; i < len(keys); i++ {\n\t\t\t_, _, ok := c.Get(keys[i])\n\t\t\tassert.True(ok)\n\t\t}\n\t\tfor _, key := range []string{\"novel\", \"big\"} {\n\t\t\t_, _, ok := c.Get(key)\n\t\t\tassert.True(ok)\n\t\t}\n\t})\n\n\tt.Run(\"TooLargeValue\", func(t *testing.T) {\n\t\tc := newManifestCache(16)\n\t\tc.Put(\"db\", manifestContents{}, time.Now())\n\t\t_, _, ok := c.Get(\"db\")\n\t\tassert.False(t, ok)\n\t})\n\n\tt.Run(\"ZeroSizeCache\", func(t *testing.T) {\n\t\tc := newManifestCache(0)\n\t\tc.Put(\"db\", manifestContents{}, time.Now())\n\t\t_, _, ok := c.Get(\"db\")\n\t\tassert.False(t, ok)\n\t})\n\n}\n" }, { "path": "go/nbs/mem_table.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"sort\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype memTable struct {\n\tchunks map[addr][]byte\n\torder []hasRecord // Must maintain the invariant that these are sorted by rec.order\n\tmaxData, totalData uint64\n\n\tsnapper snappyEncoder\n}\n\nfunc newMemTable(memTableSize uint64) *memTable {\n\treturn &memTable{chunks: map[addr][]byte{}, maxData: memTableSize}\n}\n\nfunc (mt *memTable) addChunk(h addr, data []byte) bool {\n\tif len(data) == 0 {\n\t\tpanic(\"NBS blocks cannont be zero length\")\n\t}\n\tif _, ok := mt.chunks[h]; ok {\n\t\treturn true\n\t}\n\tdataLen := uint64(len(data))\n\tif mt.totalData+dataLen > mt.maxData {\n\t\treturn false\n\t}\n\tmt.totalData += dataLen\n\tmt.chunks[h] = data\n\tmt.order = append(mt.order, hasRecord{\n\t\t&h,\n\t\th.Prefix(),\n\t\tlen(mt.order),\n\t\tfalse,\n\t})\n\treturn true\n}\n\nfunc (mt *memTable) count() uint32 {\n\treturn uint32(len(mt.order))\n}\n\nfunc (mt *memTable) uncompressedLen() uint64 {\n\treturn mt.totalData\n}\n\nfunc (mt *memTable) has(h addr) (has bool) {\n\t_, has = mt.chunks[h]\n\treturn\n}\n\nfunc (mt *memTable) hasMany(addrs []hasRecord) (remaining bool) {\n\tfor i, addr := range addrs {\n\t\tif addr.has {\n\t\t\tcontinue\n\t\t}\n\n\t\tif mt.has(*addr.a) {\n\t\t\taddrs[i].has = true\n\t\t} else {\n\t\t\tremaining = true\n\t\t}\n\t}\n\treturn\n}\n\nfunc (mt *memTable) get(h addr, stats *Stats) []byte {\n\treturn mt.chunks[h]\n}\n\nfunc (mt *memTable) getMany(reqs []getRecord, foundChunks chan *chunks.Chunk, wg *sync.WaitGroup, stats *Stats) (remaining bool) {\n\tfor _, r := range reqs {\n\t\tdata := mt.chunks[*r.a]\n\t\tif data != nil {\n\t\t\tc := chunks.NewChunkWithHash(hash.Hash(*r.a), data)\n\t\t\tfoundChunks <- &c\n\t\t} else {\n\t\t\tremaining = true\n\t\t}\n\t}\n\treturn\n}\n\nfunc (mt *memTable) extract(chunks chan<- extractRecord) {\n\tfor _, hrec := range mt.order {\n\t\tchunks <- extractRecord{a: *hrec.a, data: mt.chunks[*hrec.a]}\n\t}\n\treturn\n}\n\nfunc (mt *memTable) write(haver chunkReader, stats *Stats) (name addr, data []byte, count uint32) {\n\tmaxSize := maxTableSize(uint64(len(mt.order)), mt.totalData)\n\tbuff := make([]byte, maxSize)\n\ttw := newTableWriter(buff, mt.snapper)\n\n\tif haver != nil {\n\t\tsort.Sort(hasRecordByPrefix(mt.order)) // hasMany() requires addresses to be sorted.\n\t\thaver.hasMany(mt.order)\n\t\tsort.Sort(hasRecordByOrder(mt.order)) // restore \"insertion\" order for write\n\t}\n\n\tfor _, addr := range mt.order {\n\t\tif !addr.has {\n\t\t\th := addr.a\n\t\t\ttw.addChunk(*h, mt.chunks[*h])\n\t\t\tcount++\n\t\t}\n\t}\n\ttableSize, name := tw.finish()\n\n\tif count > 0 {\n\t\tstats.BytesPerPersist.Sample(uint64(tableSize))\n\t\tstats.CompressedChunkBytesPerPersist.Sample(uint64(tw.totalCompressedData))\n\t\tstats.UncompressedChunkBytesPerPersist.Sample(uint64(tw.totalUncompressedData))\n\t\tstats.ChunksPerPersist.Sample(uint64(count))\n\t}\n\n\treturn name, buff[:tableSize], count\n}\n" }, { "path": "go/nbs/mem_table_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"sync\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/golang/snappy\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestMemTableAddHasGetChunk(t *testing.T) {\n\tassert := assert.New(t)\n\tmt := newMemTable(1024)\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\tfor _, c := range chunks {\n\t\tassert.True(mt.addChunk(computeAddr(c), c))\n\t}\n\n\tassertChunksInReader(chunks, mt, assert)\n\n\tfor _, c := range chunks {\n\t\tassert.Equal(bytes.Compare(c, mt.get(computeAddr(c), &Stats{})), 0)\n\t}\n\n\tnotPresent := []byte(\"nope\")\n\tassert.False(mt.has(computeAddr(notPresent)))\n\tassert.Nil(mt.get(computeAddr(notPresent), &Stats{}))\n}\n\nfunc TestMemTableAddOverflowChunk(t *testing.T) {\n\tmemTableSize := uint64(1024)\n\n\tassert := assert.New(t)\n\tbig := make([]byte, memTableSize)\n\tlittle := []byte{0x01}\n\t{\n\t\tbigAddr := computeAddr(big)\n\t\tmt := newMemTable(memTableSize)\n\t\tassert.True(mt.addChunk(bigAddr, big))\n\t\tassert.True(mt.has(bigAddr))\n\t\tassert.False(mt.addChunk(computeAddr(little), little))\n\t\tassert.False(mt.has(computeAddr(little)))\n\t}\n\n\t{\n\t\tbig := big[:memTableSize-1]\n\t\tbigAddr := computeAddr(big)\n\t\tmt := newMemTable(memTableSize)\n\t\tassert.True(mt.addChunk(bigAddr, big))\n\t\tassert.True(mt.has(bigAddr))\n\t\tassert.True(mt.addChunk(computeAddr(little), little))\n\t\tassert.True(mt.has(computeAddr(little)))\n\t\tother := []byte(\"o\")\n\t\tassert.False(mt.addChunk(computeAddr(other), other))\n\t\tassert.False(mt.has(computeAddr(other)))\n\t}\n}\n\nfunc TestMemTableWrite(t *testing.T) {\n\tassert := assert.New(t)\n\tmt := newMemTable(1024)\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\tfor _, c := range chunks {\n\t\tassert.True(mt.addChunk(computeAddr(c), c))\n\t}\n\n\ttd1, _ := buildTable(chunks[1:2])\n\ttd2, _ := buildTable(chunks[2:])\n\ttr1 := newTableReader(parseTableIndex(td1), tableReaderAtFromBytes(td1), fileBlockSize)\n\ttr2 := newTableReader(parseTableIndex(td2), tableReaderAtFromBytes(td2), fileBlockSize)\n\tassert.True(tr1.has(computeAddr(chunks[1])))\n\tassert.True(tr2.has(computeAddr(chunks[2])))\n\n\t_, data, count := mt.write(chunkReaderGroup{tr1, tr2}, &Stats{})\n\tassert.Equal(uint32(1), count)\n\n\toutReader := newTableReader(parseTableIndex(data), tableReaderAtFromBytes(data), fileBlockSize)\n\tassert.True(outReader.has(computeAddr(chunks[0])))\n\tassert.False(outReader.has(computeAddr(chunks[1])))\n\tassert.False(outReader.has(computeAddr(chunks[2])))\n}\n\ntype tableReaderAtAdapter struct {\n\t*bytes.Reader\n}\n\nfunc tableReaderAtFromBytes(b []byte) tableReaderAt {\n\treturn tableReaderAtAdapter{bytes.NewReader(b)}\n}\n\nfunc (adapter tableReaderAtAdapter) ReadAtWithStats(p []byte, off int64, stats *Stats) (n int, err error) {\n\treturn adapter.ReadAt(p, off)\n}\n\nfunc TestMemTableSnappyWriteOutOfLine(t *testing.T) {\n\tassert := assert.New(t)\n\tmt := newMemTable(1024)\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\tfor _, c := range chunks {\n\t\tassert.True(mt.addChunk(computeAddr(c), c))\n\t}\n\tmt.snapper = &outOfLineSnappy{[]bool{false, true, false}} // chunks[1] should trigger a panic\n\n\tassert.Panics(func() { mt.write(nil, &Stats{}) })\n}\n\ntype outOfLineSnappy struct {\n\tpolicy []bool\n}\n\nfunc (o *outOfLineSnappy) Encode(dst, src []byte) []byte {\n\toutOfLine := false\n\tif len(o.policy) > 0 {\n\t\toutOfLine = o.policy[0]\n\t\to.policy = o.policy[1:]\n\t}\n\tif outOfLine {\n\t\treturn snappy.Encode(nil, src)\n\t}\n\treturn snappy.Encode(dst, src)\n}\n\ntype chunkReaderGroup []chunkReader\n\nfunc (crg chunkReaderGroup) has(h addr) bool {\n\tfor _, haver := range crg {\n\t\tif haver.has(h) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc (crg chunkReaderGroup) get(h addr, stats *Stats) []byte {\n\tfor _, haver := range crg {\n\t\tif data := haver.get(h, stats); data != nil {\n\t\t\treturn data\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc (crg chunkReaderGroup) hasMany(addrs []hasRecord) (remaining bool) {\n\tfor _, haver := range crg {\n\t\tif !haver.hasMany(addrs) {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\nfunc (crg chunkReaderGroup) getMany(reqs []getRecord, foundChunks chan *chunks.Chunk, wg *sync.WaitGroup, stats *Stats) (remaining bool) {\n\tfor _, haver := range crg {\n\t\tif !haver.getMany(reqs, foundChunks, wg, stats) {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\nfunc (crg chunkReaderGroup) count() (count uint32) {\n\tfor _, haver := range crg {\n\t\tcount += haver.count()\n\t}\n\treturn\n}\n\nfunc (crg chunkReaderGroup) uncompressedLen() (data uint64) {\n\tfor _, haver := range crg {\n\t\tdata += haver.uncompressedLen()\n\t}\n\treturn\n}\n\nfunc (crg chunkReaderGroup) extract(chunks chan<- extractRecord) {\n\tfor _, haver := range crg {\n\t\thaver.extract(chunks)\n\t}\n}\n" }, { "path": "go/nbs/mmap_table_reader.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"io\"\n\t\"math\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strconv\"\n\t\"time\"\n\n\t\"golang.org/x/sys/unix\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype mmapTableReader struct {\n\ttableReader\n\tfc *fdCache\n\th addr\n}\n\nconst (\n\tfileBlockSize = 1 << 12\n)\n\nvar (\n\tpageSize = int64(os.Getpagesize())\n\tmaxInt = int64(math.MaxInt64)\n)\n\nfunc init() {\n\tif strconv.IntSize == 32 {\n\t\tmaxInt = math.MaxInt32\n\t}\n}\n\nfunc newMmapTableReader(dir string, h addr, chunkCount uint32, indexCache *indexCache, fc *fdCache) chunkSource {\n\tpath := filepath.Join(dir, h.String())\n\n\tvar index tableIndex\n\tfound := false\n\tif indexCache != nil {\n\t\tindexCache.lockEntry(h)\n\t\tdefer indexCache.unlockEntry(h)\n\t\tindex, found = indexCache.get(h)\n\t}\n\n\tif !found {\n\t\tf, err := fc.RefFile(path)\n\t\td.PanicIfError(err)\n\t\tdefer fc.UnrefFile(path)\n\n\t\tfi, err := f.Stat()\n\t\td.PanicIfError(err)\n\t\td.PanicIfTrue(fi.Size() < 0)\n\t\t// index. Mmap won't take an offset that's not page-aligned, so find the nearest page boundary preceding the index.\n\t\tindexOffset := fi.Size() - int64(footerSize) - int64(indexSize(chunkCount))\n\t\taligned := indexOffset / pageSize * pageSize // Thanks, integer arithmetic!\n\t\td.PanicIfTrue(fi.Size()-aligned > maxInt)\n\t\tbuff, err := unix.Mmap(int(f.Fd()), aligned, int(fi.Size()-aligned), unix.PROT_READ, unix.MAP_SHARED)\n\t\td.PanicIfError(err)\n\t\tindex = parseTableIndex(buff[indexOffset-aligned:])\n\n\t\tif indexCache != nil {\n\t\t\tindexCache.put(h, index)\n\t\t}\n\t\terr = unix.Munmap(buff)\n\t\td.PanicIfError(err)\n\t}\n\n\td.PanicIfFalse(chunkCount == index.chunkCount)\n\treturn &mmapTableReader{\n\t\tnewTableReader(index, &cacheReaderAt{path, fc}, fileBlockSize),\n\t\tfc,\n\t\th,\n\t}\n}\n\nfunc (mmtr *mmapTableReader) hash() addr {\n\treturn mmtr.h\n}\n\ntype cacheReaderAt struct {\n\tpath string\n\tfc *fdCache\n}\n\nfunc (cra *cacheReaderAt) ReadAtWithStats(p []byte, off int64, stats *Stats) (n int, err error) {\n\tvar r io.ReaderAt\n\tt1 := time.Now()\n\n\tif r, err = cra.fc.RefFile(cra.path); err != nil {\n\t\treturn\n\t}\n\tdefer func() {\n\t\tstats.FileBytesPerRead.Sample(uint64(len(p)))\n\t\tstats.FileReadLatency.SampleTimeSince(t1)\n\t}()\n\n\tdefer cra.fc.UnrefFile(cra.path)\n\n\treturn r.ReadAt(p, off)\n}\n" }, { "path": "go/nbs/mmap_table_reader_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestMmapTableReader(t *testing.T) {\n\tassert := assert.New(t)\n\tdir, err := ioutil.TempDir(\"\", \"\")\n\tassert.NoError(err)\n\tdefer os.RemoveAll(dir)\n\n\tfc := newFDCache(1)\n\tdefer fc.Drop()\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\ttableData, h := buildTable(chunks)\n\terr = ioutil.WriteFile(filepath.Join(dir, h.String()), tableData, 0666)\n\tassert.NoError(err)\n\n\ttrc := newMmapTableReader(dir, h, uint32(len(chunks)), nil, fc)\n\tassertChunksInReader(chunks, trc, assert)\n}\n" }, { "path": "go/nbs/persisting_chunk_source.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"io\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nfunc newPersistingChunkSource(mt *memTable, haver chunkReader, p tablePersister, rl chan struct{}, stats *Stats) *persistingChunkSource {\n\tt1 := time.Now()\n\n\tccs := &persistingChunkSource{mt: mt}\n\tccs.wg.Add(1)\n\trl <- struct{}{}\n\tgo func() {\n\t\tdefer ccs.wg.Done()\n\t\tcs := p.Persist(mt, haver, stats)\n\n\t\tccs.mu.Lock()\n\t\tdefer ccs.mu.Unlock()\n\t\tccs.cs = cs\n\t\tccs.mt = nil\n\t\t<-rl\n\n\t\tif cs.count() > 0 {\n\t\t\tstats.PersistLatency.SampleTimeSince(t1)\n\t\t}\n\t}()\n\treturn ccs\n}\n\ntype persistingChunkSource struct {\n\tmu sync.RWMutex\n\tmt *memTable\n\n\twg sync.WaitGroup\n\tcs chunkSource\n}\n\nfunc (ccs *persistingChunkSource) getReader() chunkReader {\n\tccs.mu.RLock()\n\tdefer ccs.mu.RUnlock()\n\tif ccs.mt != nil {\n\t\treturn ccs.mt\n\t}\n\treturn ccs.cs\n}\n\nfunc (ccs *persistingChunkSource) has(h addr) bool {\n\tcr := ccs.getReader()\n\td.Chk.True(cr != nil)\n\treturn cr.has(h)\n}\n\nfunc (ccs *persistingChunkSource) hasMany(addrs []hasRecord) bool {\n\tcr := ccs.getReader()\n\td.Chk.True(cr != nil)\n\treturn cr.hasMany(addrs)\n}\n\nfunc (ccs *persistingChunkSource) get(h addr, stats *Stats) []byte {\n\tcr := ccs.getReader()\n\td.Chk.True(cr != nil)\n\treturn cr.get(h, stats)\n}\n\nfunc (ccs *persistingChunkSource) getMany(reqs []getRecord, foundChunks chan *chunks.Chunk, wg *sync.WaitGroup, stats *Stats) bool {\n\tcr := ccs.getReader()\n\td.Chk.True(cr != nil)\n\treturn cr.getMany(reqs, foundChunks, wg, stats)\n}\n\nfunc (ccs *persistingChunkSource) count() uint32 {\n\tccs.wg.Wait()\n\td.Chk.True(ccs.cs != nil)\n\treturn ccs.cs.count()\n}\n\nfunc (ccs *persistingChunkSource) uncompressedLen() uint64 {\n\tccs.wg.Wait()\n\td.Chk.True(ccs.cs != nil)\n\treturn ccs.cs.uncompressedLen()\n}\n\nfunc (ccs *persistingChunkSource) hash() addr {\n\tccs.wg.Wait()\n\td.Chk.True(ccs.cs != nil)\n\treturn ccs.cs.hash()\n}\n\nfunc (ccs *persistingChunkSource) index() tableIndex {\n\tccs.wg.Wait()\n\td.Chk.True(ccs.cs != nil)\n\treturn ccs.cs.index()\n}\n\nfunc (ccs *persistingChunkSource) reader() io.Reader {\n\tccs.wg.Wait()\n\td.Chk.True(ccs.cs != nil)\n\treturn ccs.cs.reader()\n}\n\nfunc (ccs *persistingChunkSource) calcReads(reqs []getRecord, blockSize uint64) (reads int, remaining bool) {\n\tccs.wg.Wait()\n\td.Chk.True(ccs.cs != nil)\n\treturn ccs.cs.calcReads(reqs, blockSize)\n}\n\nfunc (ccs *persistingChunkSource) extract(chunks chan<- extractRecord) {\n\tccs.wg.Wait()\n\td.Chk.True(ccs.cs != nil)\n\tccs.cs.extract(chunks)\n}\n\ntype emptyChunkSource struct{}\n\nfunc (ecs emptyChunkSource) has(h addr) bool {\n\treturn false\n}\n\nfunc (ecs emptyChunkSource) hasMany(addrs []hasRecord) bool {\n\treturn true\n}\n\nfunc (ecs emptyChunkSource) get(h addr, stats *Stats) []byte {\n\treturn nil\n}\n\nfunc (ecs emptyChunkSource) getMany(reqs []getRecord, foundChunks chan *chunks.Chunk, wg *sync.WaitGroup, stats *Stats) bool {\n\treturn true\n}\n\nfunc (ecs emptyChunkSource) count() uint32 {\n\treturn 0\n}\n\nfunc (ecs emptyChunkSource) uncompressedLen() uint64 {\n\treturn 0\n}\n\nfunc (ecs emptyChunkSource) hash() addr {\n\treturn addr{}\n}\n\nfunc (ecs emptyChunkSource) index() tableIndex {\n\treturn tableIndex{}\n}\n\nfunc (ecs emptyChunkSource) reader() io.Reader {\n\treturn &bytes.Buffer{}\n}\n\nfunc (ecs emptyChunkSource) calcReads(reqs []getRecord, blockSize uint64) (reads int, remaining bool) {\n\treturn 0, true\n}\n\nfunc (ecs emptyChunkSource) extract(chunks chan<- extractRecord) {}\n" }, { "path": "go/nbs/persisting_chunk_source_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestPersistingChunkStoreEmpty(t *testing.T) {\n\tmt := newMemTable(testMemTableSize)\n\tccs := newPersistingChunkSource(mt, nil, newFakeTablePersister(), make(chan struct{}, 1), &Stats{})\n\tassert.Equal(t, addr{}, ccs.hash())\n\tassert.Zero(t, ccs.count())\n}\n\ntype pausingFakeTablePersister struct {\n\ttablePersister\n\ttrigger <-chan struct{}\n}\n\nfunc (ftp pausingFakeTablePersister) Persist(mt *memTable, haver chunkReader, stats *Stats) chunkSource {\n\t<-ftp.trigger\n\treturn ftp.tablePersister.Persist(mt, haver, stats)\n}\n\nfunc TestPersistingChunkStore(t *testing.T) {\n\tassert := assert.New(t)\n\tmt := newMemTable(testMemTableSize)\n\n\tfor _, c := range testChunks {\n\t\tassert.True(mt.addChunk(computeAddr(c), c))\n\t}\n\n\ttrigger := make(chan struct{})\n\tccs := newPersistingChunkSource(mt, nil, pausingFakeTablePersister{newFakeTablePersister(), trigger}, make(chan struct{}, 1), &Stats{})\n\n\tassertChunksInReader(testChunks, ccs, assert)\n\tassert.EqualValues(mt.count(), ccs.getReader().count())\n\tclose(trigger)\n\n\tassert.NotEqual(addr{}, ccs.hash())\n\tassert.EqualValues(len(testChunks), ccs.count())\n\tassertChunksInReader(testChunks, ccs, assert)\n\n\tassert.Nil(ccs.mt)\n\n\tnewChunk := []byte(\"additional\")\n\tmt.addChunk(computeAddr(newChunk), newChunk)\n\tassert.NotEqual(mt.count(), ccs.count())\n\tassert.False(ccs.has(computeAddr(newChunk)))\n}\n" }, { "path": "go/nbs/root_tracker_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"fmt\"\n\t\"sync\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestChunkStoreZeroValue(t *testing.T) {\n\tassert := assert.New(t)\n\t_, _, store := makeStoreWithFakes(t)\n\tdefer store.Close()\n\n\t// No manifest file gets written until the first call to Commit(). Prior to that, Root() will simply return hash.Hash{}.\n\tassert.Equal(hash.Hash{}, store.Root())\n\tassert.Equal(constants.NomsVersion, store.Version())\n}\n\nfunc TestChunkStoreVersion(t *testing.T) {\n\tassert := assert.New(t)\n\t_, _, store := makeStoreWithFakes(t)\n\tdefer store.Close()\n\n\tassert.Equal(constants.NomsVersion, store.Version())\n\tnewRoot := hash.Of([]byte(\"new root\"))\n\tif assert.True(store.Commit(newRoot, hash.Hash{})) {\n\t\tassert.Equal(constants.NomsVersion, store.Version())\n\t}\n}\n\nfunc TestChunkStoreRebase(t *testing.T) {\n\tassert := assert.New(t)\n\tfm, p, store := makeStoreWithFakes(t)\n\tdefer store.Close()\n\n\tassert.Equal(hash.Hash{}, store.Root())\n\tassert.Equal(constants.NomsVersion, store.Version())\n\n\t// Simulate another process writing a manifest behind store's back.\n\tnewRoot, chunks := interloperWrite(fm, p, []byte(\"new root\"), []byte(\"hello2\"), []byte(\"goodbye2\"), []byte(\"badbye2\"))\n\n\t// state in store shouldn't change\n\tassert.Equal(hash.Hash{}, store.Root())\n\tassert.Equal(constants.NomsVersion, store.Version())\n\n\tstore.Rebase()\n\n\t// NOW it should\n\tassert.Equal(newRoot, store.Root())\n\tassert.Equal(constants.NomsVersion, store.Version())\n\tassertDataInStore(chunks, store, assert)\n}\n\nfunc TestChunkStoreCommit(t *testing.T) {\n\tassert := assert.New(t)\n\t_, _, store := makeStoreWithFakes(t)\n\tdefer store.Close()\n\n\tassert.Equal(hash.Hash{}, store.Root())\n\n\tnewRootChunk := chunks.NewChunk([]byte(\"new root\"))\n\tnewRoot := newRootChunk.Hash()\n\tstore.Put(newRootChunk)\n\tif assert.True(store.Commit(newRoot, hash.Hash{})) {\n\t\tassert.True(store.Has(newRoot))\n\t\tassert.Equal(newRoot, store.Root())\n\t}\n\n\tsecondRootChunk := chunks.NewChunk([]byte(\"newer root\"))\n\tsecondRoot := secondRootChunk.Hash()\n\tstore.Put(secondRootChunk)\n\tif assert.True(store.Commit(secondRoot, newRoot)) {\n\t\tassert.Equal(secondRoot, store.Root())\n\t\tassert.True(store.Has(newRoot))\n\t\tassert.True(store.Has(secondRoot))\n\t}\n}\n\nfunc TestChunkStoreManifestAppearsAfterConstruction(t *testing.T) {\n\tassert := assert.New(t)\n\tfm, p, store := makeStoreWithFakes(t)\n\tdefer store.Close()\n\n\tassert.Equal(hash.Hash{}, store.Root())\n\tassert.Equal(constants.NomsVersion, store.Version())\n\n\t// Simulate another process writing a manifest behind store's back.\n\tinterloperWrite(fm, p, []byte(\"new root\"), []byte(\"hello2\"), []byte(\"goodbye2\"), []byte(\"badbye2\"))\n\n\t// state in store shouldn't change\n\tassert.Equal(hash.Hash{}, store.Root())\n\tassert.Equal(constants.NomsVersion, store.Version())\n}\n\nfunc TestChunkStoreManifestFirstWriteByOtherProcess(t *testing.T) {\n\tassert := assert.New(t)\n\tfm := &fakeManifest{}\n\tmm := manifestManager{fm, newManifestCache(0), newManifestLocks()}\n\tp := newFakeTablePersister()\n\n\t// Simulate another process writing a manifest behind store's back.\n\tnewRoot, chunks := interloperWrite(fm, p, []byte(\"new root\"), []byte(\"hello2\"), []byte(\"goodbye2\"), []byte(\"badbye2\"))\n\n\tstore := newNomsBlockStore(mm, p, inlineConjoiner{defaultMaxTables}, defaultMemTableSize)\n\tdefer store.Close()\n\n\tassert.Equal(newRoot, store.Root())\n\tassert.Equal(constants.NomsVersion, store.Version())\n\tassertDataInStore(chunks, store, assert)\n}\n\nfunc TestChunkStoreCommitOptimisticLockFail(t *testing.T) {\n\tassert := assert.New(t)\n\tfm, p, store := makeStoreWithFakes(t)\n\tdefer store.Close()\n\n\t// Simulate another process writing a manifest behind store's back.\n\tnewRoot, chunks := interloperWrite(fm, p, []byte(\"new root\"), []byte(\"hello2\"), []byte(\"goodbye2\"), []byte(\"badbye2\"))\n\n\tnewRoot2 := hash.Of([]byte(\"new root 2\"))\n\tassert.False(store.Commit(newRoot2, hash.Hash{}))\n\tassertDataInStore(chunks, store, assert)\n\tassert.True(store.Commit(newRoot2, newRoot))\n}\n\nfunc TestChunkStoreManifestPreemptiveOptimisticLockFail(t *testing.T) {\n\tassert := assert.New(t)\n\tfm := &fakeManifest{}\n\tmm := manifestManager{fm, newManifestCache(defaultManifestCacheSize), newManifestLocks()}\n\tp := newFakeTablePersister()\n\tc := inlineConjoiner{defaultMaxTables}\n\n\tstore := newNomsBlockStore(mm, p, c, defaultMemTableSize)\n\tdefer store.Close()\n\n\t// Simulate another goroutine writing a manifest behind store's back.\n\tinterloper := newNomsBlockStore(mm, p, c, defaultMemTableSize)\n\tdefer interloper.Close()\n\n\tchunk := chunks.NewChunk([]byte(\"hello\"))\n\tinterloper.Put(chunk)\n\tassert.True(interloper.Commit(chunk.Hash(), hash.Hash{}))\n\n\t// Try to land a new chunk in store, which should fail AND not persist the contents of store.mt\n\tchunk = chunks.NewChunk([]byte(\"goodbye\"))\n\tstore.Put(chunk)\n\tassert.NotNil(store.mt)\n\tassert.False(store.Commit(chunk.Hash(), hash.Hash{}))\n\tassert.NotNil(store.mt)\n\n\tassert.True(store.Commit(chunk.Hash(), store.Root()))\n\tassert.Nil(store.mt)\n\tassert.Equal(chunk.Hash(), store.Root())\n\tassert.Equal(constants.NomsVersion, store.Version())\n}\n\nfunc TestChunkStoreCommitLocksOutFetch(t *testing.T) {\n\tassert := assert.New(t)\n\tfm := &fakeManifest{name: \"foo\"}\n\tupm := &updatePreemptManifest{manifest: fm}\n\tmm := manifestManager{upm, newManifestCache(defaultManifestCacheSize), newManifestLocks()}\n\tp := newFakeTablePersister()\n\tc := inlineConjoiner{defaultMaxTables}\n\n\tstore := newNomsBlockStore(mm, p, c, defaultMemTableSize)\n\tdefer store.Close()\n\n\t// store.Commit() should lock out calls to mm.Fetch()\n\twg := sync.WaitGroup{}\n\tfetched := manifestContents{}\n\tupm.preUpdate = func() {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\t\t\t_, fetched = mm.Fetch(nil)\n\t\t}()\n\t}\n\n\trootChunk := chunks.NewChunk([]byte(\"new root\"))\n\tstore.Put(rootChunk)\n\tassert.True(store.Commit(rootChunk.Hash(), store.Root()))\n\n\twg.Wait()\n\tassert.Equal(store.Root(), fetched.root)\n}\n\nfunc TestChunkStoreSerializeCommits(t *testing.T) {\n\tassert := assert.New(t)\n\tfm := &fakeManifest{name: \"foo\"}\n\tupm := &updatePreemptManifest{manifest: fm}\n\tmc := newManifestCache(defaultManifestCacheSize)\n\tl := newManifestLocks()\n\tp := newFakeTablePersister()\n\tc := inlineConjoiner{defaultMaxTables}\n\n\tstore := newNomsBlockStore(manifestManager{upm, mc, l}, p, c, defaultMemTableSize)\n\tdefer store.Close()\n\n\tstoreChunk := chunks.NewChunk([]byte(\"store\"))\n\tinterloperChunk := chunks.NewChunk([]byte(\"interloper\"))\n\tupdateCount := 0\n\n\tinterloper := newNomsBlockStore(\n\t\tmanifestManager{\n\t\t\tupdatePreemptManifest{fm, func() { updateCount++ }}, mc, l,\n\t\t},\n\t\tp,\n\t\tc,\n\t\tdefaultMemTableSize)\n\tdefer interloper.Close()\n\n\twg := sync.WaitGroup{}\n\tupm.preUpdate = func() {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tdefer wg.Done()\n\t\t\tinterloper.Put(interloperChunk)\n\t\t\tassert.True(interloper.Commit(interloper.Root(), interloper.Root()))\n\t\t}()\n\n\t\tupdateCount++\n\t}\n\n\tstore.Put(storeChunk)\n\tassert.True(store.Commit(store.Root(), store.Root()))\n\n\twg.Wait()\n\tassert.Equal(2, updateCount)\n\tassert.True(interloper.Has(storeChunk.Hash()))\n\tassert.True(interloper.Has(interloperChunk.Hash()))\n}\n\nfunc makeStoreWithFakes(t *testing.T) (fm *fakeManifest, p tablePersister, store *NomsBlockStore) {\n\tfm = &fakeManifest{}\n\tmm := manifestManager{fm, newManifestCache(0), newManifestLocks()}\n\tp = newFakeTablePersister()\n\tstore = newNomsBlockStore(mm, p, inlineConjoiner{defaultMaxTables}, 0)\n\treturn\n}\n\n// Simulate another process writing a manifest behind store's back.\nfunc interloperWrite(fm *fakeManifest, p tablePersister, rootChunk []byte, chunks ...[]byte) (newRoot hash.Hash, persisted [][]byte) {\n\tnewLock, newRoot := computeAddr([]byte(\"locker\")), hash.Of(rootChunk)\n\tpersisted = append(chunks, rootChunk)\n\tsrc := p.Persist(createMemTable(persisted), nil, &Stats{})\n\tfm.set(constants.NomsVersion, newLock, newRoot, []tableSpec{{src.hash(), uint32(len(chunks))}})\n\treturn\n}\n\nfunc createMemTable(chunks [][]byte) *memTable {\n\tmt := newMemTable(1 << 10)\n\tfor _, c := range chunks {\n\t\tmt.addChunk(computeAddr(c), c)\n\t}\n\treturn mt\n}\n\nfunc assertDataInStore(slices [][]byte, store chunks.ChunkStore, assert *assert.Assertions) {\n\tfor _, data := range slices {\n\t\tassert.True(store.Has(chunks.NewChunk(data).Hash()))\n\t}\n}\n\n// fakeManifest simulates a fileManifest without touching disk.\ntype fakeManifest struct {\n\tname string\n\tcontents manifestContents\n\tmu sync.RWMutex\n}\n\nfunc (fm *fakeManifest) Name() string { return fm.name }\n\n// ParseIfExists returns any fake manifest data the caller has injected using\n// Update() or set(). It treats an empty |fm.lock| as a non-existent manifest.\nfunc (fm *fakeManifest) ParseIfExists(stats *Stats, readHook func()) (exists bool, contents manifestContents) {\n\tfm.mu.RLock()\n\tdefer fm.mu.RUnlock()\n\tif fm.contents.lock != (addr{}) {\n\t\treturn true, fm.contents\n\t}\n\treturn false, manifestContents{}\n}\n\n// Update checks whether |lastLock| == |fm.lock| and, if so, updates internal\n// fake manifest state as per the manifest.Update() contract: |fm.lock| is set\n// to |newLock|, |fm.root| is set to |newRoot|, and the contents of |specs|\n// replace |fm.tableSpecs|. If |lastLock| != |fm.lock|, then the update\n// fails. Regardless of success or failure, the current state is returned.\nfunc (fm *fakeManifest) Update(lastLock addr, newContents manifestContents, stats *Stats, writeHook func()) manifestContents {\n\tfm.mu.Lock()\n\tdefer fm.mu.Unlock()\n\tif fm.contents.lock == lastLock {\n\t\tfm.contents = manifestContents{newContents.vers, newContents.lock, newContents.root, nil}\n\t\tfm.contents.specs = make([]tableSpec, len(newContents.specs))\n\t\tcopy(fm.contents.specs, newContents.specs)\n\t}\n\treturn fm.contents\n}\n\nfunc (fm *fakeManifest) set(version string, lock addr, root hash.Hash, specs []tableSpec) {\n\tfm.contents = manifestContents{version, lock, root, specs}\n}\n\nfunc newFakeTableSet() tableSet {\n\treturn tableSet{p: newFakeTablePersister(), rl: make(chan struct{}, 1)}\n}\n\nfunc newFakeTablePersister() tablePersister {\n\treturn fakeTablePersister{map[addr]tableReader{}, &sync.RWMutex{}}\n}\n\ntype fakeTablePersister struct {\n\tsources map[addr]tableReader\n\tmu *sync.RWMutex\n}\n\nfunc (ftp fakeTablePersister) Persist(mt *memTable, haver chunkReader, stats *Stats) chunkSource {\n\tif mt.count() > 0 {\n\t\tname, data, chunkCount := mt.write(haver, stats)\n\t\tif chunkCount > 0 {\n\t\t\tftp.mu.Lock()\n\t\t\tdefer ftp.mu.Unlock()\n\t\t\tftp.sources[name] = newTableReader(parseTableIndex(data), tableReaderAtFromBytes(data), fileBlockSize)\n\t\t\treturn chunkSourceAdapter{ftp.sources[name], name}\n\t\t}\n\t}\n\treturn emptyChunkSource{}\n}\n\nfunc (ftp fakeTablePersister) ConjoinAll(sources chunkSources, stats *Stats) chunkSource {\n\tname, data, chunkCount := compactSourcesToBuffer(sources)\n\tif chunkCount > 0 {\n\t\tftp.mu.Lock()\n\t\tdefer ftp.mu.Unlock()\n\t\tftp.sources[name] = newTableReader(parseTableIndex(data), tableReaderAtFromBytes(data), fileBlockSize)\n\t\treturn chunkSourceAdapter{ftp.sources[name], name}\n\t}\n\treturn emptyChunkSource{}\n}\n\nfunc compactSourcesToBuffer(sources chunkSources) (name addr, data []byte, chunkCount uint32) {\n\ttotalData := uint64(0)\n\tfor _, src := range sources {\n\t\tchunkCount += src.count()\n\t\ttotalData += src.uncompressedLen()\n\t}\n\tif chunkCount == 0 {\n\t\treturn\n\t}\n\n\tmaxSize := maxTableSize(uint64(chunkCount), totalData)\n\tbuff := make([]byte, maxSize) // This can blow up RAM\n\ttw := newTableWriter(buff, nil)\n\terrString := \"\"\n\n\tfor _, src := range sources {\n\t\tchunks := make(chan extractRecord)\n\t\tgo func() {\n\t\t\tdefer close(chunks)\n\t\t\tdefer func() {\n\t\t\t\tif r := recover(); r != nil {\n\t\t\t\t\tchunks <- extractRecord{a: src.hash(), err: r}\n\t\t\t\t}\n\t\t\t}()\n\t\t\tsrc.extract(chunks)\n\t\t}()\n\t\tfor rec := range chunks {\n\t\t\tif rec.err != nil {\n\t\t\t\terrString += fmt.Sprintf(\"Failed to extract %s:\\n %v\\n******\\n\\n\", rec.a, rec.err)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\ttw.addChunk(rec.a, rec.data)\n\t\t}\n\t}\n\n\tif errString != \"\" {\n\t\tpanic(fmt.Errorf(errString))\n\t}\n\ttableSize, name := tw.finish()\n\treturn name, buff[:tableSize], chunkCount\n}\n\nfunc (ftp fakeTablePersister) Open(name addr, chunkCount uint32, stats *Stats) chunkSource {\n\tftp.mu.RLock()\n\tdefer ftp.mu.RUnlock()\n\treturn chunkSourceAdapter{ftp.sources[name], name}\n}\n\ntype chunkSourceAdapter struct {\n\ttableReader\n\th addr\n}\n\nfunc (csa chunkSourceAdapter) hash() addr {\n\treturn csa.h\n}\n\nfunc (csa chunkSourceAdapter) index() tableIndex {\n\treturn csa.tableIndex\n}\n" }, { "path": "go/nbs/s3_fake_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"net/url\"\n\t\"strconv\"\n\t\"strings\"\n\t\"sync\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/aws/aws-sdk-go/aws\"\n\t\"github.com/aws/aws-sdk-go/service/s3\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype mockAWSError string\n\nfunc (m mockAWSError) Error() string { return string(m) }\nfunc (m mockAWSError) Code() string { return string(m) }\nfunc (m mockAWSError) Message() string { return string(m) }\nfunc (m mockAWSError) OrigErr() error { return nil }\n\nfunc makeFakeS3(t *testing.T) *fakeS3 {\n\treturn &fakeS3{\n\t\tassert: assert.New(t),\n\t\tdata: map[string][]byte{},\n\t\tinProgress: map[string]fakeS3Multipart{},\n\t\tparts: map[string][]byte{},\n\t}\n}\n\ntype fakeS3 struct {\n\tassert *assert.Assertions\n\n\tmu sync.Mutex\n\tdata map[string][]byte\n\tinProgressCounter int\n\tinProgress map[string]fakeS3Multipart // Key -> {UploadId, Etags...}\n\tparts map[string][]byte // ETag -> data\n\tgetCount int\n}\n\ntype fakeS3Multipart struct {\n\tuploadID string\n\tetags []string\n}\n\nfunc (m *fakeS3) readerForTable(name addr) chunkReader {\n\tm.mu.Lock()\n\tdefer m.mu.Unlock()\n\tif buff, present := m.data[name.String()]; present {\n\t\treturn newTableReader(parseTableIndex(buff), tableReaderAtFromBytes(buff), s3BlockSize)\n\t}\n\treturn nil\n}\n\nfunc (m *fakeS3) AbortMultipartUpload(input *s3.AbortMultipartUploadInput) (*s3.AbortMultipartUploadOutput, error) {\n\tm.assert.NotNil(input.Bucket, \"Bucket is a required field\")\n\tm.assert.NotNil(input.Key, \"Key is a required field\")\n\tm.assert.NotNil(input.UploadId, \"UploadId is a required field\")\n\n\tm.mu.Lock()\n\tdefer m.mu.Unlock()\n\tm.assert.Equal(m.inProgress[*input.Key].uploadID, *input.UploadId)\n\tfor _, etag := range m.inProgress[*input.Key].etags {\n\t\tdelete(m.parts, etag)\n\t}\n\tdelete(m.inProgress, *input.Key)\n\treturn &s3.AbortMultipartUploadOutput{}, nil\n}\n\nfunc (m *fakeS3) CreateMultipartUpload(input *s3.CreateMultipartUploadInput) (*s3.CreateMultipartUploadOutput, error) {\n\tm.assert.NotNil(input.Bucket, \"Bucket is a required field\")\n\tm.assert.NotNil(input.Key, \"Key is a required field\")\n\n\tout := &s3.CreateMultipartUploadOutput{\n\t\tBucket: input.Bucket,\n\t\tKey: input.Key,\n\t}\n\n\tm.mu.Lock()\n\tdefer m.mu.Unlock()\n\tuploadID := strconv.Itoa(m.inProgressCounter)\n\tout.UploadId = aws.String(uploadID)\n\tm.inProgress[*input.Key] = fakeS3Multipart{uploadID, nil}\n\tm.inProgressCounter++\n\treturn out, nil\n}\n\nfunc (m *fakeS3) UploadPart(input *s3.UploadPartInput) (*s3.UploadPartOutput, error) {\n\tm.assert.NotNil(input.Bucket, \"Bucket is a required field\")\n\tm.assert.NotNil(input.Key, \"Key is a required field\")\n\tm.assert.NotNil(input.PartNumber, \"PartNumber is a required field\")\n\tm.assert.NotNil(input.UploadId, \"UploadId is a required field\")\n\tm.assert.NotNil(input.Body, \"Body is a required field\")\n\n\tdata, err := ioutil.ReadAll(input.Body)\n\tm.assert.NoError(err)\n\n\tm.mu.Lock()\n\tdefer m.mu.Unlock()\n\tetag := hash.Of(data).String() + time.Now().String()\n\tm.parts[etag] = data\n\n\tinProgress, present := m.inProgress[*input.Key]\n\tm.assert.True(present)\n\tm.assert.Equal(inProgress.uploadID, *input.UploadId)\n\tinProgress.etags = append(inProgress.etags, etag)\n\tm.inProgress[*input.Key] = inProgress\n\treturn &s3.UploadPartOutput{ETag: aws.String(etag)}, nil\n}\n\nfunc (m *fakeS3) UploadPartCopy(input *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error) {\n\tm.assert.NotNil(input.Bucket, \"Bucket is a required field\")\n\tm.assert.NotNil(input.Key, \"Key is a required field\")\n\tm.assert.NotNil(input.PartNumber, \"PartNumber is a required field\")\n\tm.assert.NotNil(input.UploadId, \"UploadId is a required field\")\n\tm.assert.NotNil(input.CopySource, \"CopySource is a required field\")\n\n\tunescaped, err := url.QueryUnescape(*input.CopySource)\n\tm.assert.NoError(err)\n\tslash := strings.LastIndex(unescaped, \"/\")\n\tm.assert.NotEqual(-1, slash, \"Malformed CopySource %s\", unescaped)\n\tsrc := unescaped[slash+1:]\n\n\tm.mu.Lock()\n\tdefer m.mu.Unlock()\n\tobj, present := m.data[src]\n\tif !present {\n\t\treturn nil, mockAWSError(\"NoSuchKey\")\n\t}\n\tif input.CopySourceRange != nil {\n\t\tstart, end := parseRange(*input.CopySourceRange, len(obj))\n\t\tobj = obj[start:end]\n\t}\n\tetag := hash.Of(obj).String() + time.Now().String()\n\tm.parts[etag] = obj\n\n\tinProgress, present := m.inProgress[*input.Key]\n\tm.assert.True(present)\n\tm.assert.Equal(inProgress.uploadID, *input.UploadId)\n\tinProgress.etags = append(inProgress.etags, etag)\n\tm.inProgress[*input.Key] = inProgress\n\treturn &s3.UploadPartCopyOutput{CopyPartResult: &s3.CopyPartResult{ETag: aws.String(etag)}}, nil\n}\n\nfunc (m *fakeS3) CompleteMultipartUpload(input *s3.CompleteMultipartUploadInput) (*s3.CompleteMultipartUploadOutput, error) {\n\tm.assert.NotNil(input.Bucket, \"Bucket is a required field\")\n\tm.assert.NotNil(input.Key, \"Key is a required field\")\n\tm.assert.NotNil(input.UploadId, \"UploadId is a required field\")\n\tm.assert.NotNil(input.MultipartUpload, \"MultipartUpload is a required field\")\n\tm.assert.True(len(input.MultipartUpload.Parts) > 0)\n\n\tm.mu.Lock()\n\tdefer m.mu.Unlock()\n\tm.assert.Equal(m.inProgress[*input.Key].uploadID, *input.UploadId)\n\tfor idx, part := range input.MultipartUpload.Parts {\n\t\tm.assert.EqualValues(idx+1, *part.PartNumber) // Part numbers are 1-indexed\n\t\tm.data[*input.Key] = append(m.data[*input.Key], m.parts[*part.ETag]...)\n\t\tdelete(m.parts, *part.ETag)\n\t}\n\tdelete(m.inProgress, *input.Key)\n\n\treturn &s3.CompleteMultipartUploadOutput{Bucket: input.Bucket, Key: input.Key}, nil\n}\n\nfunc (m *fakeS3) GetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error) {\n\tm.getCount++\n\tm.assert.NotNil(input.Bucket, \"Bucket is a required field\")\n\tm.assert.NotNil(input.Key, \"Key is a required field\")\n\n\tm.mu.Lock()\n\tdefer m.mu.Unlock()\n\tobj, present := m.data[*input.Key]\n\tif !present {\n\t\treturn nil, mockAWSError(\"NoSuchKey\")\n\t}\n\tif input.Range != nil {\n\t\tstart, end := parseRange(*input.Range, len(obj))\n\t\tobj = obj[start:end]\n\t}\n\n\treturn &s3.GetObjectOutput{\n\t\tBody: ioutil.NopCloser(bytes.NewReader(obj)),\n\t\tContentLength: aws.Int64(int64(len(obj))),\n\t}, nil\n}\n\nfunc parseRange(hdr string, total int) (start, end int) {\n\td.PanicIfFalse(len(hdr) > len(s3RangePrefix))\n\thdr = hdr[len(s3RangePrefix):]\n\td.PanicIfFalse(hdr[0] == '=')\n\thdr = hdr[1:]\n\tif hdr[0] == '-' {\n\t\t// negative range\n\t\tfromEnd, err := strconv.Atoi(hdr[1:])\n\t\td.PanicIfError(err)\n\t\treturn total - fromEnd, total\n\t}\n\tends := strings.Split(hdr, \"-\")\n\td.PanicIfFalse(len(ends) == 2)\n\tstart, err := strconv.Atoi(ends[0])\n\td.PanicIfError(err)\n\tend, err = strconv.Atoi(ends[1])\n\td.PanicIfError(err)\n\treturn start, end + 1 // insanely, the HTTP range header specifies ranges inclusively.\n}\n\nfunc (m *fakeS3) PutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error) {\n\tm.assert.NotNil(input.Bucket, \"Bucket is a required field\")\n\tm.assert.NotNil(input.Key, \"Key is a required field\")\n\n\tbuff := &bytes.Buffer{}\n\tio.Copy(buff, input.Body)\n\tm.mu.Lock()\n\tdefer m.mu.Unlock()\n\tm.data[*input.Key] = buff.Bytes()\n\n\treturn &s3.PutObjectOutput{}, nil\n}\n" }, { "path": "go/nbs/s3_table_reader.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"net\"\n\t\"os\"\n\t\"time\"\n\n\t\"golang.org/x/sys/unix\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/aws/aws-sdk-go/aws\"\n\t\"github.com/aws/aws-sdk-go/service/s3\"\n\t\"github.com/jpillora/backoff\"\n)\n\nconst (\n\ts3RangePrefix = \"bytes\"\n\ts3BlockSize = (1 << 10) * 512 // 512K\n)\n\ntype s3TableReaderAt struct {\n\ts3 *s3ObjectReader\n\th addr\n}\n\ntype s3svc interface {\n\tAbortMultipartUpload(input *s3.AbortMultipartUploadInput) (*s3.AbortMultipartUploadOutput, error)\n\tCreateMultipartUpload(input *s3.CreateMultipartUploadInput) (*s3.CreateMultipartUploadOutput, error)\n\tUploadPart(input *s3.UploadPartInput) (*s3.UploadPartOutput, error)\n\tUploadPartCopy(input *s3.UploadPartCopyInput) (*s3.UploadPartCopyOutput, error)\n\tCompleteMultipartUpload(input *s3.CompleteMultipartUploadInput) (*s3.CompleteMultipartUploadOutput, error)\n\tGetObject(input *s3.GetObjectInput) (*s3.GetObjectOutput, error)\n\tPutObject(input *s3.PutObjectInput) (*s3.PutObjectOutput, error)\n}\n\nfunc (s3tra *s3TableReaderAt) ReadAtWithStats(p []byte, off int64, stats *Stats) (n int, err error) {\n\treturn s3tra.s3.ReadAt(s3tra.h, p, off, stats)\n}\n\n// TODO: Bring all the multipart upload and remote-conjoin stuff over here and make this a better analogue to ddbTableStore\ntype s3ObjectReader struct {\n\ts3 s3svc\n\tbucket string\n\treadRl chan struct{}\n\ttc tableCache\n}\n\nfunc (s3or *s3ObjectReader) ReadAt(name addr, p []byte, off int64, stats *Stats) (n int, err error) {\n\tt1 := time.Now()\n\n\tif s3or.tc != nil {\n\t\tr := s3or.tc.checkout(name)\n\t\tif r != nil {\n\t\t\tdefer func() {\n\t\t\t\tstats.FileBytesPerRead.Sample(uint64(len(p)))\n\t\t\t\tstats.FileReadLatency.SampleTimeSince(t1)\n\t\t\t}()\n\t\t\tdefer s3or.tc.checkin(name)\n\t\t\treturn r.ReadAt(p, off)\n\t\t}\n\t}\n\n\tdefer func() {\n\t\tstats.S3BytesPerRead.Sample(uint64(len(p)))\n\t\tstats.S3ReadLatency.SampleTimeSince(t1)\n\t}()\n\treturn s3or.readRange(name, p, s3RangeHeader(off, int64(len(p))))\n}\n\nfunc s3RangeHeader(off, length int64) string {\n\tlastByte := off + length - 1 // insanely, the HTTP range header specifies ranges inclusively.\n\treturn fmt.Sprintf(\"%s=%d-%d\", s3RangePrefix, off, lastByte)\n}\n\nfunc (s3or *s3ObjectReader) ReadFromEnd(name addr, p []byte, stats *Stats) (n int, err error) {\n\t// TODO: enable this to use the tableCache. The wrinkle is the tableCache currently just returns a ReaderAt, which doesn't give you the length of the object that backs it, so you can't calculate an offset if all you know is that you want the last N bytes.\n\tdefer func(t1 time.Time) {\n\t\tstats.S3BytesPerRead.Sample(uint64(len(p)))\n\t\tstats.S3ReadLatency.SampleTimeSince(t1)\n\t}(time.Now())\n\treturn s3or.readRange(name, p, fmt.Sprintf(\"%s=-%d\", s3RangePrefix, len(p)))\n}\n\nfunc (s3or *s3ObjectReader) readRange(name addr, p []byte, rangeHeader string) (n int, err error) {\n\tread := func() (int, error) {\n\t\tif s3or.readRl != nil {\n\t\t\ts3or.readRl <- struct{}{}\n\t\t\tdefer func() {\n\t\t\t\t<-s3or.readRl\n\t\t\t}()\n\t\t}\n\n\t\tinput := &s3.GetObjectInput{\n\t\t\tBucket: aws.String(s3or.bucket),\n\t\t\tKey: aws.String(name.String()),\n\t\t\tRange: aws.String(rangeHeader),\n\t\t}\n\t\tresult, err := s3or.s3.GetObject(input)\n\t\td.PanicIfError(err)\n\t\td.PanicIfFalse(*result.ContentLength == int64(len(p)))\n\n\t\tn, err := io.ReadFull(result.Body, p)\n\t\tif err != nil {\n\t\t\tfmt.Fprintf(os.Stderr, \"Failed ranged read from S3\\n%s\\nerr type: %T\\nerror: %v\\n\", input.GoString(), err, err)\n\t\t}\n\t\treturn n, err\n\t}\n\n\tn, err = read()\n\t// We hit the point of diminishing returns investigating #3255, so add retries. In conversations with AWS people, it's not surprising to get transient failures when talking to S3, though SDKs are intended to have their own retrying. The issue may be that, in Go, making the S3 request and reading the data are separate operations, and the SDK kind of can't do its own retrying to handle failures in the latter.\n\tif isConnReset(err) {\n\t\t// We are backing off here because its possible and likely that the rate of requests to S3 is the underlying issue.\n\t\tb := &backoff.Backoff{\n\t\t\tMin: 128 * time.Microsecond,\n\t\t\tMax: 1024 * time.Millisecond,\n\t\t\tFactor: 2,\n\t\t\tJitter: true,\n\t\t}\n\t\tfor ; isConnReset(err); n, err = read() {\n\t\t\tdur := b.Duration()\n\t\t\tfmt.Fprintf(os.Stderr, \"Retrying S3 read in %s\\n\", dur.String())\n\t\t\ttime.Sleep(dur)\n\t\t}\n\t}\n\treturn\n}\n\nfunc isConnReset(err error) bool {\n\tnErr, ok := err.(*net.OpError)\n\tif !ok {\n\t\treturn false\n\t}\n\tscErr, ok := nErr.Err.(*os.SyscallError)\n\treturn ok && scErr.Err == unix.ECONNRESET\n}\n" }, { "path": "go/nbs/s3_table_reader_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"io/ioutil\"\n\t\"net\"\n\t\"os\"\n\t\"testing\"\n\n\t\"golang.org/x/sys/unix\"\n\n\t\"github.com/aws/aws-sdk-go/service/s3\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestS3TableReaderAt(t *testing.T) {\n\ts3 := makeFakeS3(t)\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\ttableData, h := buildTable(chunks)\n\ts3.data[h.String()] = tableData\n\n\tt.Run(\"TolerateFailingReads\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\n\t\tbaseline := s3.getCount\n\t\ttra := &s3TableReaderAt{&s3ObjectReader{makeFlakyS3(s3), \"bucket\", nil, nil}, h}\n\t\tscratch := make([]byte, len(tableData))\n\t\t_, err := tra.ReadAtWithStats(scratch, 0, &Stats{})\n\t\tassert.NoError(err)\n\t\t// constructing the table reader should have resulted in 2 reads\n\t\tassert.Equal(2, s3.getCount-baseline)\n\t\tassert.Equal(tableData, scratch)\n\t})\n\n\tt.Run(\"WithTableCache\", func(t *testing.T) {\n\t\tassert := assert.New(t)\n\t\tdir := makeTempDir(t)\n\t\tdefer os.RemoveAll(dir)\n\t\tstats := &Stats{}\n\n\t\ttc := newFSTableCache(dir, uint64(2*len(tableData)), 4)\n\t\ttra := &s3TableReaderAt{&s3ObjectReader{s3, \"bucket\", nil, tc}, h}\n\n\t\t// First, read when table is not yet cached\n\t\tscratch := make([]byte, len(tableData))\n\t\tbaseline := s3.getCount\n\t\t_, err := tra.ReadAtWithStats(scratch, 0, stats)\n\t\tassert.NoError(err)\n\t\tassert.True(s3.getCount > baseline)\n\n\t\t// Cache the table and read again\n\t\ttc.store(h, bytes.NewReader(tableData), uint64(len(tableData)))\n\t\tbaseline = s3.getCount\n\t\t_, err = tra.ReadAtWithStats(scratch, 0, stats)\n\t\tassert.NoError(err)\n\t\tassert.Zero(s3.getCount - baseline)\n\t})\n}\n\ntype flakyS3 struct {\n\ts3svc\n\talreadyFailed map[string]struct{}\n}\n\nfunc makeFlakyS3(svc s3svc) *flakyS3 {\n\treturn &flakyS3{svc, map[string]struct{}{}}\n}\n\nfunc (fs3 *flakyS3) GetObject(input *s3.GetObjectInput) (output *s3.GetObjectOutput, err error) {\n\toutput, err = fs3.s3svc.GetObject(input)\n\tif _, ok := fs3.alreadyFailed[*input.Key]; !ok {\n\t\tfs3.alreadyFailed[*input.Key] = struct{}{}\n\t\toutput.Body = ioutil.NopCloser(resettingReader{})\n\t}\n\treturn\n}\n\ntype resettingReader struct{}\n\nfunc (rr resettingReader) Read(p []byte) (n int, err error) {\n\treturn 0, &net.OpError{Op: \"read\", Net: \"tcp\", Err: &os.SyscallError{Syscall: \"read\", Err: unix.ECONNRESET}}\n}\n" }, { "path": "go/nbs/stats.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/attic-labs/noms/go/metrics\"\n)\n\ntype Stats struct {\n\tOpenLatency metrics.Histogram\n\tCommitLatency metrics.Histogram\n\n\tIndexReadLatency metrics.Histogram\n\tIndexBytesPerRead metrics.Histogram\n\n\tGetLatency metrics.Histogram\n\tChunksPerGet metrics.Histogram\n\n\tFileReadLatency metrics.Histogram\n\tFileBytesPerRead metrics.Histogram\n\n\tS3ReadLatency metrics.Histogram\n\tS3BytesPerRead metrics.Histogram\n\n\tMemReadLatency metrics.Histogram\n\tMemBytesPerRead metrics.Histogram\n\n\tDynamoReadLatency metrics.Histogram\n\tDynamoBytesPerRead metrics.Histogram\n\n\tHasLatency metrics.Histogram\n\tAddressesPerHas metrics.Histogram\n\n\tPutLatency metrics.Histogram\n\n\tPersistLatency metrics.Histogram\n\tBytesPerPersist metrics.Histogram\n\n\tChunksPerPersist metrics.Histogram\n\tCompressedChunkBytesPerPersist metrics.Histogram\n\tUncompressedChunkBytesPerPersist metrics.Histogram\n\n\tConjoinLatency metrics.Histogram\n\tBytesPerConjoin metrics.Histogram\n\tChunksPerConjoin metrics.Histogram\n\tTablesPerConjoin metrics.Histogram\n\n\tReadManifestLatency metrics.Histogram\n\tWriteManifestLatency metrics.Histogram\n}\n\nfunc NewStats() *Stats {\n\treturn &Stats{\n\t\tOpenLatency: metrics.NewTimeHistogram(),\n\t\tCommitLatency: metrics.NewTimeHistogram(),\n\t\tIndexReadLatency: metrics.NewTimeHistogram(),\n\t\tIndexBytesPerRead: metrics.NewByteHistogram(),\n\t\tGetLatency: metrics.NewTimeHistogram(),\n\t\tFileReadLatency: metrics.NewTimeHistogram(),\n\t\tFileBytesPerRead: metrics.NewByteHistogram(),\n\t\tS3ReadLatency: metrics.NewTimeHistogram(),\n\t\tS3BytesPerRead: metrics.NewByteHistogram(),\n\t\tMemReadLatency: metrics.NewTimeHistogram(),\n\t\tMemBytesPerRead: metrics.NewByteHistogram(),\n\t\tDynamoReadLatency: metrics.NewTimeHistogram(),\n\t\tDynamoBytesPerRead: metrics.NewByteHistogram(),\n\t\tHasLatency: metrics.NewTimeHistogram(),\n\t\tPutLatency: metrics.NewTimeHistogram(),\n\t\tPersistLatency: metrics.NewTimeHistogram(),\n\t\tBytesPerPersist: metrics.NewByteHistogram(),\n\t\tCompressedChunkBytesPerPersist: metrics.NewByteHistogram(),\n\t\tUncompressedChunkBytesPerPersist: metrics.NewByteHistogram(),\n\t\tConjoinLatency: metrics.NewTimeHistogram(),\n\t\tBytesPerConjoin: metrics.NewByteHistogram(),\n\t\tReadManifestLatency: metrics.NewTimeHistogram(),\n\t\tWriteManifestLatency: metrics.NewTimeHistogram(),\n\t}\n}\n\nfunc (s *Stats) Add(other Stats) {\n\ts.OpenLatency.Add(other.OpenLatency)\n\ts.CommitLatency.Add(other.CommitLatency)\n\n\ts.IndexReadLatency.Add(other.IndexReadLatency)\n\ts.IndexBytesPerRead.Add(other.IndexBytesPerRead)\n\n\ts.GetLatency.Add(other.GetLatency)\n\ts.ChunksPerGet.Add(other.ChunksPerGet)\n\n\ts.FileReadLatency.Add(other.FileReadLatency)\n\ts.FileBytesPerRead.Add(other.FileBytesPerRead)\n\n\ts.S3ReadLatency.Add(other.S3ReadLatency)\n\ts.S3BytesPerRead.Add(other.S3BytesPerRead)\n\n\ts.MemReadLatency.Add(other.MemReadLatency)\n\ts.MemBytesPerRead.Add(other.MemBytesPerRead)\n\n\ts.DynamoReadLatency.Add(other.DynamoReadLatency)\n\ts.DynamoBytesPerRead.Add(other.DynamoBytesPerRead)\n\n\ts.HasLatency.Add(other.HasLatency)\n\ts.AddressesPerHas.Add(other.AddressesPerHas)\n\n\ts.PutLatency.Add(other.PutLatency)\n\n\ts.PersistLatency.Add(other.PersistLatency)\n\ts.BytesPerPersist.Add(other.BytesPerPersist)\n\n\ts.ChunksPerPersist.Add(other.ChunksPerPersist)\n\ts.CompressedChunkBytesPerPersist.Add(other.CompressedChunkBytesPerPersist)\n\ts.UncompressedChunkBytesPerPersist.Add(other.UncompressedChunkBytesPerPersist)\n\n\ts.ConjoinLatency.Add(other.ConjoinLatency)\n\ts.BytesPerConjoin.Add(other.BytesPerConjoin)\n\ts.ChunksPerConjoin.Add(other.ChunksPerConjoin)\n\ts.TablesPerConjoin.Add(other.TablesPerConjoin)\n\n\ts.ReadManifestLatency.Add(other.ReadManifestLatency)\n\ts.WriteManifestLatency.Add(other.WriteManifestLatency)\n}\n\nfunc (s Stats) Delta(other Stats) Stats {\n\treturn Stats{\n\t\ts.OpenLatency.Delta(other.OpenLatency),\n\t\ts.CommitLatency.Delta(other.CommitLatency),\n\n\t\ts.IndexReadLatency.Delta(other.IndexReadLatency),\n\t\ts.IndexBytesPerRead.Delta(other.IndexBytesPerRead),\n\n\t\ts.GetLatency.Delta(other.GetLatency),\n\t\ts.ChunksPerGet.Delta(other.ChunksPerGet),\n\n\t\ts.FileReadLatency.Delta(other.FileReadLatency),\n\t\ts.FileBytesPerRead.Delta(other.FileBytesPerRead),\n\n\t\ts.S3ReadLatency.Delta(other.S3ReadLatency),\n\t\ts.S3BytesPerRead.Delta(other.S3BytesPerRead),\n\n\t\ts.MemReadLatency.Delta(other.MemReadLatency),\n\t\ts.MemBytesPerRead.Delta(other.MemBytesPerRead),\n\n\t\ts.DynamoReadLatency.Delta(other.DynamoReadLatency),\n\t\ts.DynamoBytesPerRead.Delta(other.DynamoBytesPerRead),\n\n\t\ts.HasLatency.Delta(other.HasLatency),\n\t\ts.AddressesPerHas.Delta(other.AddressesPerHas),\n\n\t\ts.PutLatency.Delta(other.PutLatency),\n\n\t\ts.PersistLatency.Delta(other.PersistLatency),\n\t\ts.BytesPerPersist.Delta(other.BytesPerPersist),\n\n\t\ts.ChunksPerPersist.Delta(other.ChunksPerPersist),\n\t\ts.CompressedChunkBytesPerPersist.Delta(other.CompressedChunkBytesPerPersist),\n\t\ts.UncompressedChunkBytesPerPersist.Delta(other.UncompressedChunkBytesPerPersist),\n\n\t\ts.ConjoinLatency.Delta(other.ConjoinLatency),\n\t\ts.BytesPerConjoin.Delta(other.BytesPerConjoin),\n\t\ts.ChunksPerConjoin.Delta(other.ChunksPerConjoin),\n\t\ts.TablesPerConjoin.Delta(other.TablesPerConjoin),\n\n\t\ts.ReadManifestLatency.Delta(other.ReadManifestLatency),\n\t\ts.WriteManifestLatency.Delta(other.WriteManifestLatency),\n\t}\n}\n\nfunc (s Stats) String() string {\n\treturn fmt.Sprintf(`---NBS Stats---\nOpenLatecy: %s\nCommitLatency: %s\nIndexReadLatency: %s\nIndexBytesPerRead: %s\nGetLatency: %s\nChunksPerGet: %s\nFileReadLatency: %s\nFileBytesPerRead: %s\nS3ReadLatency: %s\nS3BytesPerRead: %s\nMemReadLatency: %s\nMemBytesPerRead: %s\nDynamoReadLatency: %s\nDynamoBytesPerRead: %s\nHasLatency: %s\nAddressesHasGet: %s\nPutLatency: %s\nPersistLatency: %s\nBytesPerPersist: %s\nChunksPerPersist: %s\nCompressedChunkBytesPerPersist: %s\nUncompressedChunkBytesPerPersist: %s\nConjoinLatency: %s\nBytesPerConjoin: %s\nChunksPerConjoin: %s\nTablesPerConjoin: %s\nReadManifestLatency: %s\nWriteManifestLatency: %s\n`,\n\t\ts.OpenLatency,\n\t\ts.CommitLatency,\n\n\t\ts.IndexReadLatency,\n\t\ts.IndexBytesPerRead,\n\n\t\ts.GetLatency,\n\t\ts.ChunksPerGet,\n\n\t\ts.FileReadLatency,\n\t\ts.FileBytesPerRead,\n\n\t\ts.S3ReadLatency,\n\t\ts.S3BytesPerRead,\n\n\t\ts.MemReadLatency,\n\t\ts.MemBytesPerRead,\n\n\t\ts.DynamoReadLatency,\n\t\ts.DynamoBytesPerRead,\n\n\t\ts.HasLatency,\n\t\ts.AddressesPerHas,\n\n\t\ts.PutLatency,\n\n\t\ts.PersistLatency,\n\t\ts.BytesPerPersist,\n\n\t\ts.ChunksPerPersist,\n\t\ts.CompressedChunkBytesPerPersist,\n\t\ts.UncompressedChunkBytesPerPersist,\n\n\t\ts.ConjoinLatency,\n\t\ts.BytesPerConjoin,\n\t\ts.ChunksPerConjoin,\n\t\ts.TablesPerConjoin,\n\t\ts.ReadManifestLatency,\n\t\ts.WriteManifestLatency)\n}\n" }, { "path": "go/nbs/stats_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestStats(t *testing.T) {\n\tassert := assert.New(t)\n\n\tstats := func(store *NomsBlockStore) Stats {\n\t\treturn store.Stats().(Stats)\n\t}\n\n\tdir, err := ioutil.TempDir(\"\", \"\")\n\tassert.NoError(err)\n\tstore := NewLocalStore(dir, testMemTableSize)\n\n\tassert.EqualValues(1, stats(store).OpenLatency.Samples())\n\n\t// Opening a new store will still incur some read IO, to discover that the manifest doesn't exist\n\tassert.EqualValues(1, stats(store).ReadManifestLatency.Samples())\n\n\ti1, i2, i3, i4, i5 := []byte(\"abc\"), []byte(\"def\"), []byte(\"ghi\"), []byte(\"jkl\"), []byte(\"mno\")\n\n\tc1, c2, c3, c4, c5 := chunks.NewChunk(i1), chunks.NewChunk(i2), chunks.NewChunk(i3), chunks.NewChunk(i4), chunks.NewChunk(i5)\n\n\t// These just go to mem table, only operation stats\n\tstore.Put(c1)\n\tstore.Put(c2)\n\tstore.Put(c3)\n\tassert.Equal(uint64(3), stats(store).PutLatency.Samples())\n\tassert.Equal(uint64(0), stats(store).PersistLatency.Samples())\n\n\tassert.True(store.Has(c1.Hash()))\n\tassert.True(store.Has(c2.Hash()))\n\tassert.True(store.Has(c3.Hash()))\n\tassert.Equal(uint64(3), stats(store).HasLatency.Samples())\n\tassert.Equal(uint64(3), stats(store).AddressesPerHas.Sum())\n\n\tassert.False(store.Get(c1.Hash()).IsEmpty())\n\tassert.False(store.Get(c2.Hash()).IsEmpty())\n\tassert.False(store.Get(c3.Hash()).IsEmpty())\n\tassert.Equal(uint64(3), stats(store).GetLatency.Samples())\n\tassert.Equal(uint64(0), stats(store).FileReadLatency.Samples())\n\tassert.Equal(uint64(3), stats(store).ChunksPerGet.Sum())\n\n\tstore.Commit(store.Root(), store.Root())\n\n\t// Commit will update the manifest\n\tassert.EqualValues(1, stats(store).WriteManifestLatency.Samples())\n\tassert.EqualValues(1, stats(store).CommitLatency.Samples())\n\n\t// Now we have write IO\n\tassert.Equal(uint64(1), stats(store).PersistLatency.Samples())\n\tassert.Equal(uint64(3), stats(store).ChunksPerPersist.Sum())\n\tassert.Equal(uint64(131), stats(store).BytesPerPersist.Sum())\n\n\t// Now some gets that will incur read IO\n\tstore.Get(c1.Hash())\n\tstore.Get(c2.Hash())\n\tstore.Get(c3.Hash())\n\tassert.Equal(uint64(3), stats(store).FileReadLatency.Samples())\n\tassert.Equal(uint64(27), stats(store).FileBytesPerRead.Sum())\n\n\t// Try A GetMany\n\tchnx := make([]chunks.Chunk, 3)\n\tchnx[0] = c1\n\tchnx[1] = c2\n\tchnx[2] = c3\n\thashes := make(hash.HashSlice, len(chnx))\n\tfor i, c := range chnx {\n\t\thashes[i] = c.Hash()\n\t}\n\tchunkChan := make(chan *chunks.Chunk, 3)\n\tstore.GetMany(hashes.HashSet(), chunkChan)\n\tassert.Equal(uint64(4), stats(store).FileReadLatency.Samples())\n\tassert.Equal(uint64(54), stats(store).FileBytesPerRead.Sum())\n\n\t// Force a conjoin\n\tstore.c = inlineConjoiner{2}\n\tstore.Put(c4)\n\tstore.Commit(store.Root(), store.Root())\n\tstore.Put(c5)\n\tstore.Commit(store.Root(), store.Root())\n\n\tassert.Equal(uint64(1), stats(store).ConjoinLatency.Samples())\n\t// TODO: Once random conjoin hack is out, test other conjoin stats\n\n\tdefer store.Close()\n\tdefer os.RemoveAll(dir)\n}\n" }, { "path": "go/nbs/store.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"fmt\"\n\t\"sort\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\thumanize \"github.com/dustin/go-humanize\"\n)\n\n// The root of a Noms Chunk Store is stored in a 'manifest', along with the\n// names of the tables that hold all the chunks in the store. The number of\n// chunks in each table is also stored in the manifest.\n\nconst (\n\t// StorageVersion is the version of the on-disk Noms Chunks Store data format.\n\tStorageVersion = \"4\"\n\n\tdefaultMemTableSize uint64 = (1 << 20) * 128 // 128MB\n\tdefaultMaxTables = 192\n\n\tdefaultIndexCacheSize = (1 << 20) * 8 // 8MB\n\tdefaultManifestCacheSize = 1 << 23 // 8MB\n\tpreflushChunkCount = 8\n)\n\nvar (\n\tcacheOnce = sync.Once{}\n\tglobalIndexCache *indexCache\n\tmakeManifestManager func(manifest) manifestManager\n\tglobalFDCache *fdCache\n)\n\nfunc makeGlobalCaches() {\n\tglobalIndexCache = newIndexCache(defaultIndexCacheSize)\n\tglobalFDCache = newFDCache(defaultMaxTables)\n\n\tmanifestCache := newManifestCache(defaultManifestCacheSize)\n\tmanifestLocks := newManifestLocks()\n\tmakeManifestManager = func(m manifest) manifestManager { return manifestManager{m, manifestCache, manifestLocks} }\n}\n\ntype NomsBlockStore struct {\n\tmm manifestManager\n\tp tablePersister\n\tc conjoiner\n\n\tmu sync.RWMutex // protects the following state\n\tmt *memTable\n\ttables tableSet\n\tupstream manifestContents\n\n\tmtSize uint64\n\tputCount uint64\n\n\tstats *Stats\n}\n\nfunc NewAWSStore(table, ns, bucket string, s3 s3svc, ddb ddbsvc, memTableSize uint64) *NomsBlockStore {\n\tcacheOnce.Do(makeGlobalCaches)\n\treadRateLimiter := make(chan struct{}, 32)\n\tp := &awsTablePersister{\n\t\ts3,\n\t\tbucket,\n\t\treadRateLimiter,\n\t\tnil,\n\t\t&ddbTableStore{ddb, table, readRateLimiter, nil},\n\t\tawsLimits{defaultS3PartSize, minS3PartSize, maxS3PartSize, maxDynamoItemSize, maxDynamoChunks},\n\t\tglobalIndexCache,\n\t}\n\tmm := makeManifestManager(newDynamoManifest(table, ns, ddb))\n\treturn newNomsBlockStore(mm, p, inlineConjoiner{defaultMaxTables}, memTableSize)\n}\n\nfunc NewLocalStore(dir string, memTableSize uint64) *NomsBlockStore {\n\tcacheOnce.Do(makeGlobalCaches)\n\td.PanicIfError(checkDir(dir))\n\n\tmm := makeManifestManager(fileManifest{dir})\n\tp := newFSTablePersister(dir, globalFDCache, globalIndexCache)\n\treturn newNomsBlockStore(mm, p, inlineConjoiner{defaultMaxTables}, memTableSize)\n}\n\nfunc newNomsBlockStore(mm manifestManager, p tablePersister, c conjoiner, memTableSize uint64) *NomsBlockStore {\n\tif memTableSize == 0 {\n\t\tmemTableSize = defaultMemTableSize\n\t}\n\tnbs := &NomsBlockStore{\n\t\tmm: mm,\n\t\tp: p,\n\t\tc: c,\n\t\ttables: newTableSet(p),\n\t\tupstream: manifestContents{vers: constants.NomsVersion},\n\t\tmtSize: memTableSize,\n\t\tstats: NewStats(),\n\t}\n\n\tt1 := time.Now()\n\tdefer nbs.stats.OpenLatency.SampleTimeSince(t1)\n\n\tif exists, contents := nbs.mm.Fetch(nbs.stats); exists {\n\t\tnbs.upstream = contents\n\t\tnbs.tables = nbs.tables.Rebase(contents.specs, nbs.stats)\n\t}\n\n\treturn nbs\n}\n\nfunc newNomsBlockStoreWithContents(mm manifestManager, mc manifestContents, p tablePersister, c conjoiner, memTableSize uint64) *NomsBlockStore {\n\tif memTableSize == 0 {\n\t\tmemTableSize = defaultMemTableSize\n\t}\n\tstats := NewStats()\n\treturn &NomsBlockStore{\n\t\tmm: mm,\n\t\tp: p,\n\t\tc: c,\n\t\tmtSize: memTableSize,\n\t\tstats: stats,\n\n\t\tupstream: mc,\n\t\ttables: newTableSet(p).Rebase(mc.specs, stats),\n\t}\n}\n\nfunc (nbs *NomsBlockStore) Put(c chunks.Chunk) {\n\tt1 := time.Now()\n\ta := addr(c.Hash())\n\td.PanicIfFalse(nbs.addChunk(a, c.Data()))\n\tnbs.putCount++\n\n\tnbs.stats.PutLatency.SampleTimeSince(t1)\n}\n\n// TODO: figure out if there's a non-error reason for this to return false. If not, get rid of return value.\nfunc (nbs *NomsBlockStore) addChunk(h addr, data []byte) bool {\n\tnbs.mu.Lock()\n\tdefer nbs.mu.Unlock()\n\tif nbs.mt == nil {\n\t\tnbs.mt = newMemTable(nbs.mtSize)\n\t}\n\tif !nbs.mt.addChunk(h, data) {\n\t\tnbs.tables = nbs.tables.Prepend(nbs.mt, nbs.stats)\n\t\tnbs.mt = newMemTable(nbs.mtSize)\n\t\treturn nbs.mt.addChunk(h, data)\n\t}\n\treturn true\n}\n\nfunc (nbs *NomsBlockStore) Get(h hash.Hash) chunks.Chunk {\n\tt1 := time.Now()\n\tdefer func() {\n\t\tnbs.stats.GetLatency.SampleTimeSince(t1)\n\t\tnbs.stats.ChunksPerGet.Sample(1)\n\t}()\n\n\ta := addr(h)\n\tdata, tables := func() (data []byte, tables chunkReader) {\n\t\tnbs.mu.RLock()\n\t\tdefer nbs.mu.RUnlock()\n\t\tif nbs.mt != nil {\n\t\t\tdata = nbs.mt.get(a, nbs.stats)\n\t\t}\n\t\treturn data, nbs.tables\n\t}()\n\tif data != nil {\n\t\treturn chunks.NewChunkWithHash(h, data)\n\t}\n\tif data := tables.get(a, nbs.stats); data != nil {\n\t\treturn chunks.NewChunkWithHash(h, data)\n\t}\n\n\treturn chunks.EmptyChunk\n}\n\nfunc (nbs *NomsBlockStore) GetMany(hashes hash.HashSet, foundChunks chan *chunks.Chunk) {\n\tt1 := time.Now()\n\treqs := toGetRecords(hashes)\n\n\tdefer func() {\n\t\tif len(hashes) > 0 {\n\t\t\tnbs.stats.GetLatency.SampleTimeSince(t1)\n\t\t\tnbs.stats.ChunksPerGet.Sample(uint64(len(reqs)))\n\t\t}\n\t}()\n\n\twg := &sync.WaitGroup{}\n\n\ttables, remaining := func() (tables chunkReader, remaining bool) {\n\t\tnbs.mu.RLock()\n\t\tdefer nbs.mu.RUnlock()\n\t\ttables = nbs.tables\n\t\tremaining = true\n\t\tif nbs.mt != nil {\n\t\t\tremaining = nbs.mt.getMany(reqs, foundChunks, nil, nbs.stats)\n\t\t}\n\n\t\treturn\n\t}()\n\n\tif remaining {\n\t\ttables.getMany(reqs, foundChunks, wg, nbs.stats)\n\t\twg.Wait()\n\t}\n\n}\n\nfunc toGetRecords(hashes hash.HashSet) []getRecord {\n\treqs := make([]getRecord, len(hashes))\n\tidx := 0\n\tfor h := range hashes {\n\t\ta := addr(h)\n\t\treqs[idx] = getRecord{\n\t\t\ta: &a,\n\t\t\tprefix: a.Prefix(),\n\t\t}\n\t\tidx++\n\t}\n\n\tsort.Sort(getRecordByPrefix(reqs))\n\treturn reqs\n}\n\nfunc (nbs *NomsBlockStore) CalcReads(hashes hash.HashSet, blockSize uint64) (reads int, split bool) {\n\treqs := toGetRecords(hashes)\n\ttables := func() (tables tableSet) {\n\t\tnbs.mu.RLock()\n\t\tdefer nbs.mu.RUnlock()\n\t\ttables = nbs.tables\n\n\t\treturn\n\t}()\n\n\treads, split, remaining := tables.calcReads(reqs, blockSize)\n\td.Chk.False(remaining)\n\treturn\n}\n\nfunc (nbs *NomsBlockStore) extractChunks(chunkChan chan<- *chunks.Chunk) {\n\tch := make(chan extractRecord, 1)\n\tgo func() {\n\t\tdefer close(ch)\n\t\tnbs.mu.RLock()\n\t\tdefer nbs.mu.RUnlock()\n\t\t// Chunks in nbs.tables were inserted before those in nbs.mt, so extract chunks there _first_\n\t\tnbs.tables.extract(ch)\n\t\tif nbs.mt != nil {\n\t\t\tnbs.mt.extract(ch)\n\t\t}\n\t}()\n\tfor rec := range ch {\n\t\tc := chunks.NewChunkWithHash(hash.Hash(rec.a), rec.data)\n\t\tchunkChan <- &c\n\t}\n}\n\nfunc (nbs *NomsBlockStore) Count() uint32 {\n\tcount, tables := func() (count uint32, tables chunkReader) {\n\t\tnbs.mu.RLock()\n\t\tdefer nbs.mu.RUnlock()\n\t\tif nbs.mt != nil {\n\t\t\tcount = nbs.mt.count()\n\t\t}\n\t\treturn count, nbs.tables\n\t}()\n\treturn count + tables.count()\n}\n\nfunc (nbs *NomsBlockStore) Has(h hash.Hash) bool {\n\tt1 := time.Now()\n\tdefer func() {\n\t\tnbs.stats.HasLatency.SampleTimeSince(t1)\n\t\tnbs.stats.AddressesPerHas.Sample(1)\n\t}()\n\n\ta := addr(h)\n\thas, tables := func() (bool, chunkReader) {\n\t\tnbs.mu.RLock()\n\t\tdefer nbs.mu.RUnlock()\n\t\treturn nbs.mt != nil && nbs.mt.has(a), nbs.tables\n\t}()\n\thas = has || tables.has(a)\n\n\treturn has\n}\n\nfunc (nbs *NomsBlockStore) HasMany(hashes hash.HashSet) hash.HashSet {\n\tt1 := time.Now()\n\n\treqs := toHasRecords(hashes)\n\n\ttables, remaining := func() (tables chunkReader, remaining bool) {\n\t\tnbs.mu.RLock()\n\t\tdefer nbs.mu.RUnlock()\n\t\ttables = nbs.tables\n\n\t\tremaining = true\n\t\tif nbs.mt != nil {\n\t\t\tremaining = nbs.mt.hasMany(reqs)\n\t\t}\n\n\t\treturn\n\t}()\n\n\tif remaining {\n\t\ttables.hasMany(reqs)\n\t}\n\n\tif len(hashes) > 0 {\n\t\tnbs.stats.HasLatency.SampleTimeSince(t1)\n\t\tnbs.stats.AddressesPerHas.SampleLen(len(reqs))\n\t}\n\n\tabsent := hash.HashSet{}\n\tfor _, r := range reqs {\n\t\tif !r.has {\n\t\t\tabsent.Insert(hash.New(r.a[:]))\n\t\t}\n\t}\n\treturn absent\n}\n\nfunc toHasRecords(hashes hash.HashSet) []hasRecord {\n\treqs := make([]hasRecord, len(hashes))\n\tidx := 0\n\tfor h := range hashes {\n\t\ta := addr(h)\n\t\treqs[idx] = hasRecord{\n\t\t\ta: &a,\n\t\t\tprefix: a.Prefix(),\n\t\t\torder: idx,\n\t\t}\n\t\tidx++\n\t}\n\n\tsort.Sort(hasRecordByPrefix(reqs))\n\treturn reqs\n}\n\nfunc (nbs *NomsBlockStore) Rebase() {\n\tnbs.mu.Lock()\n\tdefer nbs.mu.Unlock()\n\tif exists, contents := nbs.mm.Fetch(nbs.stats); exists {\n\t\tnbs.upstream = contents\n\t\tnbs.tables = nbs.tables.Rebase(contents.specs, nbs.stats)\n\t}\n}\n\nfunc (nbs *NomsBlockStore) Root() hash.Hash {\n\tnbs.mu.RLock()\n\tdefer nbs.mu.RUnlock()\n\treturn nbs.upstream.root\n}\n\nfunc (nbs *NomsBlockStore) Commit(current, last hash.Hash) bool {\n\tt1 := time.Now()\n\tdefer nbs.stats.CommitLatency.SampleTimeSince(t1)\n\n\tanyPossiblyNovelChunks := func() bool {\n\t\tnbs.mu.Lock()\n\t\tdefer nbs.mu.Unlock()\n\t\treturn nbs.mt != nil || nbs.tables.Novel() > 0\n\t}\n\n\tif !anyPossiblyNovelChunks() && current == last {\n\t\tnbs.Rebase()\n\t\treturn true\n\t}\n\n\tfunc() {\n\t\t// This is unfortunate. We want to serialize commits to the same store\n\t\t// so that we avoid writing a bunch of unreachable small tables which result\n\t\t// from optismistic lock failures. However, this means that the time to\n\t\t// write tables is included in \"commit\" time and if all commits are\n\t\t// serialized, it means alot more waiting. Allow \"non-trivial\" tables to be\n\t\t// persisted outside of the commit-lock.\n\t\tnbs.mu.Lock()\n\t\tdefer nbs.mu.Unlock()\n\n\t\tif nbs.mt != nil && nbs.mt.count() > preflushChunkCount {\n\t\t\tnbs.tables = nbs.tables.Prepend(nbs.mt, nbs.stats)\n\t\t\tnbs.mt = nil\n\t\t}\n\t}()\n\n\tnbs.mm.LockForUpdate()\n\tdefer nbs.mm.UnlockForUpdate()\n\tfor {\n\t\tif err := nbs.updateManifest(current, last); err == nil {\n\t\t\treturn true\n\t\t} else if err == errOptimisticLockFailedRoot || err == errLastRootMismatch {\n\t\t\treturn false\n\t\t}\n\t}\n}\n\nvar (\n\terrLastRootMismatch = fmt.Errorf(\"last does not match nbs.Root()\")\n\terrOptimisticLockFailedRoot = fmt.Errorf(\"Root moved\")\n\terrOptimisticLockFailedTables = fmt.Errorf(\"Tables changed\")\n)\n\nfunc (nbs *NomsBlockStore) updateManifest(current, last hash.Hash) error {\n\tnbs.mu.Lock()\n\tdefer nbs.mu.Unlock()\n\tif nbs.upstream.root != last {\n\t\treturn errLastRootMismatch\n\t}\n\n\thandleOptimisticLockFailure := func(upstream manifestContents) error {\n\t\tnbs.upstream = upstream\n\t\tnbs.tables = nbs.tables.Rebase(upstream.specs, nbs.stats)\n\n\t\tif last != upstream.root {\n\t\t\treturn errOptimisticLockFailedRoot\n\t\t}\n\t\treturn errOptimisticLockFailedTables\n\t}\n\n\tif cached, doomed := nbs.mm.updateWillFail(nbs.upstream.lock); doomed {\n\t\t// Pre-emptive optimistic lock failure. Someone else in-process moved to the root, the set of tables, or both out from under us.\n\t\treturn handleOptimisticLockFailure(cached)\n\t}\n\n\tif nbs.mt != nil && nbs.mt.count() > 0 {\n\t\tnbs.tables = nbs.tables.Prepend(nbs.mt, nbs.stats)\n\t\tnbs.mt = nil\n\t}\n\n\tif nbs.c.ConjoinRequired(nbs.tables) {\n\t\tnbs.upstream = nbs.c.Conjoin(nbs.upstream, nbs.mm, nbs.p, nbs.stats)\n\t\tnbs.tables = nbs.tables.Rebase(nbs.upstream.specs, nbs.stats)\n\t\treturn errOptimisticLockFailedTables\n\t}\n\n\tspecs := nbs.tables.ToSpecs()\n\tnewContents := manifestContents{\n\t\tvers: constants.NomsVersion,\n\t\troot: current,\n\t\tlock: generateLockHash(current, specs),\n\t\tspecs: specs,\n\t}\n\tupstream := nbs.mm.Update(nbs.upstream.lock, newContents, nbs.stats, nil)\n\tif newContents.lock != upstream.lock {\n\t\t// Optimistic lock failure. Someone else moved to the root, the set of tables, or both out from under us.\n\t\treturn handleOptimisticLockFailure(upstream)\n\t}\n\n\tnbs.upstream = newContents\n\tnbs.tables = nbs.tables.Flatten()\n\treturn nil\n}\n\nfunc (nbs *NomsBlockStore) Version() string {\n\treturn nbs.upstream.vers\n}\n\nfunc (nbs *NomsBlockStore) Close() (err error) {\n\treturn\n}\n\nfunc (nbs *NomsBlockStore) Stats() interface{} {\n\treturn *nbs.stats\n}\n\nfunc (nbs *NomsBlockStore) StatsSummary() string {\n\tnbs.mu.Lock()\n\tdefer nbs.mu.Unlock()\n\n\treturn fmt.Sprintf(\"Root: %s; Chunk Count %d; Physical Bytes %s\", nbs.upstream.root, nbs.tables.count(), humanize.Bytes(nbs.tables.physicalLen()))\n}\n" }, { "path": "go/nbs/table.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"crypto/sha512\"\n\t\"encoding/base32\"\n\t\"encoding/binary\"\n\t\"hash/crc32\"\n\t\"io\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n)\n\n/*\n An NBS Table stores N byte slices (\"chunks\") which are addressed by a 20-byte hash of their\n contents. The footer encodes N as well as the total bytes consumed by all contained chunks.\n An Index maps each address to the position of its corresponding chunk. Addresses are logically sorted within the Index, but the corresponding chunks need not be.\n\n Table:\n +----------------+----------------+-----+----------------+-------+--------+\n | Chunk Record 0 | Chunk Record 1 | ... | Chunk Record N | Index | Footer |\n +----------------+----------------+-----+----------------+-------+--------+\n\n Chunk Record:\n +---------------------------+----------------+\n | (Chunk Length) Chunk Data | (Uint32) CRC32 |\n +---------------------------+----------------+\n\n Index:\n +------------+---------+----------+\n | Prefix Map | Lengths | Suffixes |\n +------------+---------+----------+\n\n Prefix Map:\n +--------------+--------------+-----+----------------+\n | Prefix Tuple | Prefix Tuple | ... | Prefix Tuple N |\n +--------------+--------------+-----+----------------+\n\n -The Prefix Map contains N Prefix Tuples.\n -Each Prefix Tuple corresponds to a unique Chunk Record in the Table.\n -The Prefix Tuples are sorted in increasing lexicographic order within the Prefix Map.\n -NB: THE SAME PREFIX MAY APPEAR MULTIPLE TIMES, as distinct Hashes (referring to distinct Chunks) may share the same Prefix.\n\n Prefix Tuple:\n +-----------------+------------------+\n | (8) Hash Prefix | (Uint32) Ordinal |\n +-----------------+------------------+\n\n -First 8 bytes of a Chunk's Hash\n -Ordinal is the 0-based ordinal position of the associated record within the sequence of chunk records, the associated Length within Lengths, and the associated Hash Suffix within Suffixes.\n\n Lengths:\n +-----------------+-----------------+-----+-------------------+\n | (Uint32) Length | (Uint32) Length | ... | (Uint32) Length N |\n +-----------------+-----------------+-----+-------------------+\n\n - Each Length is the length of a Chunk Record in this Table.\n - Length M must correspond to Chunk Record M for 0 <= M <= N\n\n Suffixes:\n +------------------+------------------+-----+--------------------+\n | (12) Hash Suffix | (12) Hash Suffix | ... | (12) Hash Suffix N |\n +------------------+------------------+-----+--------------------+\n\n - Each Hash Suffix is the last 12 bytes of a Chunk in this Table.\n - Hash Suffix M must correspond to Chunk Record M for 0 <= M <= N\n\n Footer:\n +----------------------+----------------------------------------+------------------+\n | (Uint32) Chunk Count | (Uint64) Total Uncompressed Chunk Data | (8) Magic Number |\n +----------------------+----------------------------------------+------------------+\n\n -Total Uncompressed Chunk Data is the sum of the uncompressed byte lengths of all contained chunk byte slices.\n -Magic Number is the first 8 bytes of the SHA256 hash of \"https://github.com/attic-labs/nbs\".\n\n NOTE: Unsigned integer quanities, hashes and hash suffix are all encoded big-endian\n\n\n Looking up Chunks in an NBS Table\n There are two phases to loading chunk data for a given Hash from an NBS Table: Checking for the chunk's presence, and fetching the chunk's bytes. When performing a has-check, only the first phase is necessary.\n\n Phase one: Chunk presence\n - Slice off the first 8 bytes of your Hash to create a Prefix\n - Since the Prefix Tuples in the Prefix Map are in lexicographic order, binary search the Prefix Map for the desired Prefix.\n - For all Prefix Tuples with a matching Prefix:\n - Load the Ordinal\n - Use the Ordinal to index into Suffixes\n - Check the Suffix of your Hash against the loaded Suffix\n - If they match, your chunk is in this Table in the Chunk Record indicated by Ordinal\n - If they don't match, continue to the next matching Prefix Tuple\n - If not found, your chunk is not in this Table.\n\n Phase two: Loading Chunk data\n - Take the Ordinal discovered in Phase one\n - Calculate the Offset of your desired Chunk Record: Sum(Lengths[0]...Lengths[Ordinal-1])\n - Load Lengths[Ordinal] bytes from Table[Offset]\n - Check the first 4 bytes of the loaded data against the last 4 bytes of your desired Hash. They should match, and the rest of the data is your Chunk data.\n*/\n\nconst (\n\taddrSize uint64 = 20\n\taddrPrefixSize uint64 = 8\n\taddrSuffixSize = addrSize - addrPrefixSize\n\tuint64Size uint64 = 8\n\tuint32Size uint64 = 4\n\tordinalSize uint64 = uint32Size\n\tlengthSize uint64 = uint32Size\n\tmagicNumber = \"\\xff\\xb5\\xd8\\xc2\\x24\\x63\\xee\\x50\"\n\tmagicNumberSize uint64 = uint64(len(magicNumber))\n\tfooterSize = uint32Size + uint64Size + magicNumberSize\n\tprefixTupleSize = addrPrefixSize + ordinalSize\n\tchecksumSize uint64 = uint32Size\n\tmaxChunkLengthSize uint64 = binary.MaxVarintLen64\n\tmaxChunkSize uint64 = 0xffffffff // Snappy won't compress slices bigger than this\n)\n\nvar crcTable = crc32.MakeTable(crc32.Castagnoli)\n\nfunc crc(b []byte) uint32 {\n\treturn crc32.Update(0, crcTable, b)\n}\n\nfunc computeAddrDefault(data []byte) addr {\n\tr := sha512.Sum512(data)\n\th := addr{}\n\tcopy(h[:], r[:addrSize])\n\treturn h\n}\n\nvar computeAddr = computeAddrDefault\n\ntype addr [addrSize]byte\n\nvar encoding = base32.NewEncoding(\"0123456789abcdefghijklmnopqrstuv\")\n\nfunc (a addr) String() string {\n\treturn encoding.EncodeToString(a[:])\n}\n\nfunc (a addr) Prefix() uint64 {\n\treturn binary.BigEndian.Uint64(a[:])\n}\n\nfunc (a addr) Checksum() uint32 {\n\treturn binary.BigEndian.Uint32(a[addrSize-checksumSize:])\n}\n\nfunc ParseAddr(b []byte) (h addr) {\n\tencoding.Decode(h[:], b)\n\treturn\n}\n\nfunc ValidateAddr(s string) bool {\n\t_, err := encoding.DecodeString(s)\n\treturn err == nil\n}\n\ntype addrSlice []addr\n\nfunc (hs addrSlice) Len() int { return len(hs) }\nfunc (hs addrSlice) Less(i, j int) bool { return bytes.Compare(hs[i][:], hs[j][:]) < 0 }\nfunc (hs addrSlice) Swap(i, j int) { hs[i], hs[j] = hs[j], hs[i] }\n\ntype hasRecord struct {\n\ta *addr\n\tprefix uint64\n\torder int\n\thas bool\n}\n\ntype hasRecordByPrefix []hasRecord\n\nfunc (hs hasRecordByPrefix) Len() int { return len(hs) }\nfunc (hs hasRecordByPrefix) Less(i, j int) bool { return hs[i].prefix < hs[j].prefix }\nfunc (hs hasRecordByPrefix) Swap(i, j int) { hs[i], hs[j] = hs[j], hs[i] }\n\ntype hasRecordByOrder []hasRecord\n\nfunc (hs hasRecordByOrder) Len() int { return len(hs) }\nfunc (hs hasRecordByOrder) Less(i, j int) bool { return hs[i].order < hs[j].order }\nfunc (hs hasRecordByOrder) Swap(i, j int) { hs[i], hs[j] = hs[j], hs[i] }\n\ntype getRecord struct {\n\ta *addr\n\tprefix uint64\n\tfound bool\n}\n\ntype getRecordByPrefix []getRecord\n\nfunc (hs getRecordByPrefix) Len() int { return len(hs) }\nfunc (hs getRecordByPrefix) Less(i, j int) bool { return hs[i].prefix < hs[j].prefix }\nfunc (hs getRecordByPrefix) Swap(i, j int) { hs[i], hs[j] = hs[j], hs[i] }\n\ntype extractRecord struct {\n\ta addr\n\tdata []byte\n\terr interface{} // only set when there was a panic during extraction.\n}\n\ntype chunkReader interface {\n\thas(h addr) bool\n\thasMany(addrs []hasRecord) bool\n\tget(h addr, stats *Stats) []byte\n\tgetMany(reqs []getRecord, foundChunks chan *chunks.Chunk, wg *sync.WaitGroup, stats *Stats) bool\n\tcount() uint32\n\tuncompressedLen() uint64\n\textract(chunks chan<- extractRecord)\n}\n\ntype chunkReadPlanner interface {\n\tfindOffsets(reqs []getRecord) (ors offsetRecSlice, remaining bool)\n\tgetManyAtOffsets(\n\t\treqs []getRecord,\n\t\toffsetRecords offsetRecSlice,\n\t\tfoundChunks chan *chunks.Chunk,\n\t\twg *sync.WaitGroup,\n\t\tstats *Stats,\n\t) (remaining bool)\n}\n\ntype chunkSource interface {\n\tchunkReader\n\thash() addr\n\tcalcReads(reqs []getRecord, blockSize uint64) (reads int, remaining bool)\n\n\t// opens a Reader to the first byte of the chunkData segment of this table.\n\treader() io.Reader\n\tindex() tableIndex\n}\n\ntype chunkSources []chunkSource\n" }, { "path": "go/nbs/table_persister.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"crypto/sha512\"\n\t\"encoding/binary\"\n\t\"sort\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/util/sizecache\"\n)\n\n// tablePersister allows interaction with persistent storage. It provides\n// primitives for pushing the contents of a memTable to persistent storage,\n// opening persistent tables for reading, and conjoining a number of existing\n// chunkSources into one. A tablePersister implementation must be goroutine-\n// safe.\ntype tablePersister interface {\n\t// Persist makes the contents of mt durable. Chunks already present in\n\t// |haver| may be dropped in the process.\n\tPersist(mt *memTable, haver chunkReader, stats *Stats) chunkSource\n\n\t// ConjoinAll conjoins all chunks in |sources| into a single, new\n\t// chunkSource.\n\tConjoinAll(sources chunkSources, stats *Stats) chunkSource\n\n\t// Open a table named |name|, containing |chunkCount| chunks.\n\tOpen(name addr, chunkCount uint32, stats *Stats) chunkSource\n}\n\n// indexCache provides sized storage for table indices. While getting and/or\n// setting the cache entry for a given table name, the caller MUST hold the\n// lock that for that entry.\ntype indexCache struct {\n\tcache *sizecache.SizeCache\n\tcond *sync.Cond\n\tlocked map[addr]struct{}\n}\n\n// Returns an indexCache which will burn roughly |size| bytes of memory.\nfunc newIndexCache(size uint64) *indexCache {\n\treturn &indexCache{sizecache.New(size), sync.NewCond(&sync.Mutex{}), map[addr]struct{}{}}\n}\n\n// Take an exclusive lock on the cache entry for |name|. Callers must do this\n// before calling get(addr) or put(addr, index)\nfunc (sic *indexCache) lockEntry(name addr) {\n\tsic.cond.L.Lock()\n\tdefer sic.cond.L.Unlock()\n\n\tfor {\n\t\tif _, present := sic.locked[name]; !present {\n\t\t\tsic.locked[name] = struct{}{}\n\t\t\tbreak\n\t\t}\n\t\tsic.cond.Wait()\n\t}\n}\n\nfunc (sic *indexCache) unlockEntry(name addr) {\n\tsic.cond.L.Lock()\n\tdefer sic.cond.L.Unlock()\n\n\t_, ok := sic.locked[name]\n\td.PanicIfFalse(ok)\n\tdelete(sic.locked, name)\n\n\tsic.cond.Broadcast()\n}\n\nfunc (sic *indexCache) get(name addr) (tableIndex, bool) {\n\tif idx, found := sic.cache.Get(name); found {\n\t\treturn idx.(tableIndex), true\n\t}\n\treturn tableIndex{}, false\n}\n\nfunc (sic *indexCache) put(name addr, idx tableIndex) {\n\tindexSize := uint64(idx.chunkCount) * (addrSize + ordinalSize + lengthSize + uint64Size)\n\tsic.cache.Add(name, indexSize, idx)\n}\n\ntype chunkSourcesByAscendingCount chunkSources\n\nfunc (csbc chunkSourcesByAscendingCount) Len() int { return len(csbc) }\nfunc (csbc chunkSourcesByAscendingCount) Less(i, j int) bool {\n\tsrcI, srcJ := csbc[i], csbc[j]\n\tif srcI.count() == srcJ.count() {\n\t\thi, hj := srcI.hash(), srcJ.hash()\n\t\treturn bytes.Compare(hi[:], hj[:]) < 0\n\t}\n\treturn srcI.count() < srcJ.count()\n}\nfunc (csbc chunkSourcesByAscendingCount) Swap(i, j int) { csbc[i], csbc[j] = csbc[j], csbc[i] }\n\ntype chunkSourcesByDescendingDataSize []sourceWithSize\n\nfunc (csbds chunkSourcesByDescendingDataSize) Len() int { return len(csbds) }\nfunc (csbds chunkSourcesByDescendingDataSize) Less(i, j int) bool {\n\tswsI, swsJ := csbds[i], csbds[j]\n\tif swsI.dataLen == swsJ.dataLen {\n\t\thi, hj := swsI.source.hash(), swsJ.source.hash()\n\t\treturn bytes.Compare(hi[:], hj[:]) < 0\n\t}\n\treturn swsI.dataLen > swsJ.dataLen\n}\nfunc (csbds chunkSourcesByDescendingDataSize) Swap(i, j int) { csbds[i], csbds[j] = csbds[j], csbds[i] }\n\ntype sourceWithSize struct {\n\tsource chunkSource\n\tdataLen uint64\n}\n\ntype compactionPlan struct {\n\tsources chunkSourcesByDescendingDataSize\n\tmergedIndex []byte\n\tchunkCount uint32\n\ttotalCompressedData uint64\n}\n\nfunc (cp compactionPlan) lengths() []byte {\n\tlengthsStart := uint64(cp.chunkCount) * prefixTupleSize\n\treturn cp.mergedIndex[lengthsStart : lengthsStart+uint64(cp.chunkCount)*lengthSize]\n}\n\nfunc (cp compactionPlan) suffixes() []byte {\n\tsuffixesStart := uint64(cp.chunkCount) * (prefixTupleSize + lengthSize)\n\treturn cp.mergedIndex[suffixesStart : suffixesStart+uint64(cp.chunkCount)*addrSuffixSize]\n}\n\nfunc planConjoin(sources chunkSources, stats *Stats) (plan compactionPlan) {\n\tvar totalUncompressedData uint64\n\tfor _, src := range sources {\n\t\ttotalUncompressedData += src.uncompressedLen()\n\t\tindex := src.index()\n\t\tplan.chunkCount += index.chunkCount\n\n\t\t// Calculate the amount of chunk data in |src|\n\t\tchunkDataLen := calcChunkDataLen(index)\n\t\tplan.sources = append(plan.sources, sourceWithSize{src, chunkDataLen})\n\t\tplan.totalCompressedData += chunkDataLen\n\t}\n\tsort.Sort(plan.sources)\n\n\tlengthsPos := lengthsOffset(plan.chunkCount)\n\tsuffixesPos := suffixesOffset(plan.chunkCount)\n\tplan.mergedIndex = make([]byte, indexSize(plan.chunkCount)+footerSize)\n\n\tprefixIndexRecs := make(prefixIndexSlice, 0, plan.chunkCount)\n\tvar ordinalOffset uint32\n\tfor _, sws := range plan.sources {\n\t\tindex := sws.source.index()\n\n\t\t// Add all the prefix tuples from this index to the list of all prefixIndexRecs, modifying the ordinals such that all entries from the 1st item in sources come after those in the 0th and so on.\n\t\tfor j, prefix := range index.prefixes {\n\t\t\trec := prefixIndexRec{prefix: prefix, order: ordinalOffset + index.ordinals[j]}\n\t\t\tprefixIndexRecs = append(prefixIndexRecs, rec)\n\t\t}\n\t\tordinalOffset += sws.source.count()\n\n\t\t// TODO: copy the lengths and suffixes as a byte-copy from src BUG #3438\n\t\t// Bring over the lengths block, in order\n\t\tfor _, length := range index.lengths {\n\t\t\tbinary.BigEndian.PutUint32(plan.mergedIndex[lengthsPos:], length)\n\t\t\tlengthsPos += lengthSize\n\t\t}\n\n\t\t// Bring over the suffixes block, in order\n\t\tn := copy(plan.mergedIndex[suffixesPos:], index.suffixes)\n\t\td.Chk.True(n == len(index.suffixes))\n\t\tsuffixesPos += uint64(n)\n\t}\n\n\t// Sort all prefixTuples by hash and then insert them starting at the beginning of plan.mergedIndex\n\tsort.Sort(prefixIndexRecs)\n\tvar pfxPos uint64\n\tfor _, pi := range prefixIndexRecs {\n\t\tbinary.BigEndian.PutUint64(plan.mergedIndex[pfxPos:], pi.prefix)\n\t\tpfxPos += addrPrefixSize\n\t\tbinary.BigEndian.PutUint32(plan.mergedIndex[pfxPos:], pi.order)\n\t\tpfxPos += ordinalSize\n\t}\n\n\twriteFooter(plan.mergedIndex[uint64(len(plan.mergedIndex))-footerSize:], plan.chunkCount, totalUncompressedData)\n\n\tstats.BytesPerConjoin.Sample(uint64(plan.totalCompressedData) + uint64(len(plan.mergedIndex)))\n\treturn plan\n}\n\nfunc nameFromSuffixes(suffixes []byte) (name addr) {\n\tsha := sha512.New()\n\tsha.Write(suffixes)\n\n\tvar h []byte\n\th = sha.Sum(h) // Appends hash to h\n\tcopy(name[:], h)\n\treturn\n}\n\nfunc calcChunkDataLen(index tableIndex) uint64 {\n\treturn index.offsets[index.chunkCount-1] + uint64(index.lengths[index.chunkCount-1])\n}\n" }, { "path": "go/nbs/table_persister_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestPlanCompaction(t *testing.T) {\n\tassert := assert.New(t)\n\ttableContents := [][][]byte{\n\t\t{[]byte(\"hello2\"), []byte(\"goodbye2\"), []byte(\"badbye2\")},\n\t\t{[]byte(\"red\"), []byte(\"blue\")},\n\t\t{[]byte(\"solo\")},\n\t}\n\n\tvar sources chunkSources\n\tvar dataLens []uint64\n\tvar totalUnc uint64\n\tfor _, content := range tableContents {\n\t\tfor _, chnk := range content {\n\t\t\ttotalUnc += uint64(len(chnk))\n\t\t}\n\t\tdata, name := buildTable(content)\n\t\tsrc := chunkSourceAdapter{newTableReader(parseTableIndex(data), tableReaderAtFromBytes(data), fileBlockSize), name}\n\t\tdataLens = append(dataLens, uint64(len(data))-indexSize(src.count())-footerSize)\n\t\tsources = append(sources, src)\n\t}\n\n\tplan := planConjoin(sources, &Stats{})\n\n\tvar totalChunks uint32\n\tfor i, src := range sources {\n\t\tassert.Equal(dataLens[i], plan.sources[i].dataLen)\n\t\ttotalChunks += src.count()\n\t}\n\n\tidx := parseTableIndex(plan.mergedIndex)\n\n\tassert.Equal(totalChunks, idx.chunkCount)\n\tassert.Equal(totalUnc, idx.totalUncompressedData)\n\n\ttr := newTableReader(idx, tableReaderAtFromBytes(nil), fileBlockSize)\n\tfor _, content := range tableContents {\n\t\tassertChunksInReader(content, tr, assert)\n\t}\n}\n" }, { "path": "go/nbs/table_reader.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"bytes\"\n\t\"encoding/binary\"\n\t\"io\"\n\t\"sort\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/golang/snappy\"\n)\n\ntype tableIndex struct {\n\tchunkCount uint32\n\ttotalUncompressedData uint64\n\tprefixes, offsets []uint64\n\tlengths, ordinals []uint32\n\tsuffixes []byte\n}\n\ntype tableReaderAt interface {\n\tReadAtWithStats(p []byte, off int64, stats *Stats) (n int, err error)\n}\n\n// tableReader implements get & has queries against a single nbs table. goroutine safe.\n// |blockSize| refers to the block-size of the underlying storage. We assume that, each time we read data, we actually have to read in blocks of this size. So, we're willing to tolerate up to |blockSize| overhead each time we read a chunk, if it helps us group more chunks together into a single read request to backing storage.\ntype tableReader struct {\n\ttableIndex\n\tr tableReaderAt\n\tblockSize uint64\n}\n\n// parses a valid nbs tableIndex from a byte stream. |buff| must end with an NBS index and footer, though it may contain an unspecified number of bytes before that data. |tableIndex| doesn't keep alive any references to |buff|.\nfunc parseTableIndex(buff []byte) tableIndex {\n\tpos := uint64(len(buff))\n\n\t// footer\n\tpos -= magicNumberSize\n\td.Chk.True(string(buff[pos:]) == magicNumber)\n\n\t// total uncompressed chunk data\n\tpos -= uint64Size\n\ttotalUncompressedData := binary.BigEndian.Uint64(buff[pos:])\n\n\tpos -= uint32Size\n\tchunkCount := binary.BigEndian.Uint32(buff[pos:])\n\n\t// index\n\tsuffixesSize := uint64(chunkCount) * addrSuffixSize\n\tpos -= suffixesSize\n\tsuffixes := make([]byte, suffixesSize)\n\tcopy(suffixes, buff[pos:])\n\n\tlengthsSize := uint64(chunkCount) * lengthSize\n\tpos -= lengthsSize\n\tlengths, offsets := computeOffsets(chunkCount, buff[pos:pos+lengthsSize])\n\n\ttuplesSize := uint64(chunkCount) * prefixTupleSize\n\tpos -= tuplesSize\n\tprefixes, ordinals := computePrefixes(chunkCount, buff[pos:pos+tuplesSize])\n\n\treturn tableIndex{\n\t\tchunkCount, totalUncompressedData,\n\t\tprefixes, offsets,\n\t\tlengths, ordinals,\n\t\tsuffixes,\n\t}\n}\n\nfunc computeOffsets(count uint32, buff []byte) (lengths []uint32, offsets []uint64) {\n\tlengths = make([]uint32, count)\n\toffsets = make([]uint64, count)\n\n\tlengths[0] = binary.BigEndian.Uint32(buff)\n\n\tfor i := uint64(1); i < uint64(count); i++ {\n\t\tlengths[i] = binary.BigEndian.Uint32(buff[i*lengthSize:])\n\t\toffsets[i] = offsets[i-1] + uint64(lengths[i-1])\n\t}\n\treturn\n}\n\nfunc computePrefixes(count uint32, buff []byte) (prefixes []uint64, ordinals []uint32) {\n\tprefixes = make([]uint64, count)\n\tordinals = make([]uint32, count)\n\n\tfor i := uint64(0); i < uint64(count); i++ {\n\t\tidx := i * prefixTupleSize\n\t\tprefixes[i] = binary.BigEndian.Uint64(buff[idx:])\n\t\tordinals[i] = binary.BigEndian.Uint32(buff[idx+addrPrefixSize:])\n\t}\n\treturn\n}\n\nfunc (ti tableIndex) prefixIdxToOrdinal(idx uint32) uint32 {\n\treturn ti.ordinals[idx]\n}\n\n// returns the first position in |tr.prefixes| whose value == |prefix|. Returns |tr.chunkCount|\n// if absent\nfunc (ti tableIndex) prefixIdx(prefix uint64) (idx uint32) {\n\t// NOTE: The golang impl of sort.Search is basically inlined here. This method can be called in\n\t// an extremely tight loop and inlining the code was a significant perf improvement.\n\tidx, j := 0, ti.chunkCount\n\tfor idx < j {\n\t\th := idx + (j-idx)/2 // avoid overflow when computing h\n\t\t// i ≤ h < j\n\t\tif ti.prefixes[h] < prefix {\n\t\t\tidx = h + 1 // preserves f(i-1) == false\n\t\t} else {\n\t\t\tj = h // preserves f(j) == true\n\t\t}\n\t}\n\n\treturn\n}\n\n// Return true IFF the suffix at insertion order |ordinal| matches the address |a|.\nfunc (ti tableIndex) ordinalSuffixMatches(ordinal uint32, h addr) bool {\n\tli := uint64(ordinal) * addrSuffixSize\n\treturn bytes.Compare(h[addrPrefixSize:], ti.suffixes[li:li+addrSuffixSize]) == 0\n}\n\n// returns the ordinal of |h| if present. returns |ti.chunkCount| if absent\nfunc (ti tableIndex) lookupOrdinal(h addr) uint32 {\n\tprefix := h.Prefix()\n\n\tfor idx := ti.prefixIdx(prefix); idx < ti.chunkCount && ti.prefixes[idx] == prefix; idx++ {\n\t\tordinal := ti.prefixIdxToOrdinal(idx)\n\t\tif ti.ordinalSuffixMatches(ordinal, h) {\n\t\t\treturn ordinal\n\t\t}\n\t}\n\n\treturn ti.chunkCount\n}\n\n// newTableReader parses a valid nbs table byte stream and returns a reader. buff must end with an NBS index and footer, though it may contain an unspecified number of bytes before that data. r should allow retrieving any desired range of bytes from the table.\nfunc newTableReader(index tableIndex, r tableReaderAt, blockSize uint64) tableReader {\n\treturn tableReader{index, r, blockSize}\n}\n\n// Scan across (logically) two ordered slices of address prefixes.\nfunc (tr tableReader) hasMany(addrs []hasRecord) (remaining bool) {\n\t// TODO: Use findInIndex if (tr.chunkCount - len(addrs)*Log2(tr.chunkCount)) > (tr.chunkCount - len(addrs))\n\n\tfilterIdx := uint32(0)\n\tfilterLen := uint32(len(tr.prefixes))\n\n\tfor i, addr := range addrs {\n\t\tif addr.has {\n\t\t\tcontinue\n\t\t}\n\n\t\tfor filterIdx < filterLen && addr.prefix > tr.prefixes[filterIdx] {\n\t\t\tfilterIdx++\n\t\t}\n\n\t\tif filterIdx >= filterLen {\n\t\t\tremaining = true\n\t\t\treturn\n\t\t}\n\n\t\tif addr.prefix != tr.prefixes[filterIdx] {\n\t\t\tremaining = true\n\t\t\tcontinue\n\t\t}\n\n\t\t// prefixes are equal, so locate and compare against the corresponding suffix\n\t\tfor j := filterIdx; j < filterLen && addr.prefix == tr.prefixes[j]; j++ {\n\t\t\tif tr.ordinalSuffixMatches(tr.prefixIdxToOrdinal(j), *addr.a) {\n\t\t\t\taddrs[i].has = true\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\n\t\tif !addrs[i].has {\n\t\t\tremaining = true\n\t\t}\n\t}\n\n\treturn\n}\n\nfunc (tr tableReader) count() uint32 {\n\treturn tr.chunkCount\n}\n\nfunc (tr tableReader) uncompressedLen() uint64 {\n\treturn tr.totalUncompressedData\n}\n\nfunc (tr tableReader) index() tableIndex {\n\treturn tr.tableIndex\n}\n\n// returns true iff |h| can be found in this table.\nfunc (tr tableReader) has(h addr) bool {\n\tordinal := tr.lookupOrdinal(h)\n\treturn ordinal < tr.count()\n}\n\n// returns the storage associated with |h|, iff present. Returns nil if absent. On success,\n// the returned byte slice directly references the underlying storage.\nfunc (tr tableReader) get(h addr, stats *Stats) (data []byte) {\n\tordinal := tr.lookupOrdinal(h)\n\tif ordinal == tr.count() {\n\t\treturn\n\t}\n\n\toffset := tr.offsets[ordinal]\n\tlength := uint64(tr.lengths[ordinal])\n\tbuff := make([]byte, length) // TODO: Avoid this allocation for every get\n\n\tn, err := tr.r.ReadAtWithStats(buff, int64(offset), stats)\n\td.Chk.NoError(err)\n\td.Chk.True(n == int(length))\n\tdata = tr.parseChunk(buff)\n\td.Chk.True(data != nil)\n\n\treturn\n}\n\ntype offsetRec struct {\n\ta *addr\n\tordinal uint32\n\toffset uint64\n}\n\ntype offsetRecSlice []offsetRec\n\nfunc (hs offsetRecSlice) Len() int { return len(hs) }\nfunc (hs offsetRecSlice) Less(i, j int) bool { return hs[i].offset < hs[j].offset }\nfunc (hs offsetRecSlice) Swap(i, j int) { hs[i], hs[j] = hs[j], hs[i] }\n\nfunc (tr tableReader) readAtOffsets(\n\treadStart, readEnd uint64,\n\treqs []getRecord,\n\toffsets offsetRecSlice,\n\tfoundChunks chan *chunks.Chunk,\n\twg *sync.WaitGroup,\n\tstats *Stats,\n) {\n\n\treadLength := readEnd - readStart\n\tbuff := make([]byte, readLength)\n\n\tn, err := tr.r.ReadAtWithStats(buff, int64(readStart), stats)\n\n\td.Chk.NoError(err)\n\td.Chk.True(uint64(n) == readLength)\n\n\tfor _, rec := range offsets {\n\t\td.Chk.True(rec.offset >= readStart)\n\t\tlocalStart := rec.offset - readStart\n\t\tlocalEnd := localStart + uint64(tr.lengths[rec.ordinal])\n\t\td.Chk.True(localEnd <= readLength)\n\t\tdata := tr.parseChunk(buff[localStart:localEnd])\n\t\tc := chunks.NewChunkWithHash(hash.Hash(*rec.a), data)\n\t\tfoundChunks <- &c\n\t}\n\n\twg.Done()\n\n}\n\n// getMany retrieves multiple stored blocks and optimizes by attempting to read in larger physical\n// blocks which contain multiple stored blocks. |reqs| must be sorted by address prefix.\nfunc (tr tableReader) getMany(\n\treqs []getRecord,\n\tfoundChunks chan *chunks.Chunk,\n\twg *sync.WaitGroup,\n\tstats *Stats,\n) (remaining bool) {\n\t// Pass #1: Iterate over |reqs| and |tr.prefixes| (both sorted by address) and build the set\n\t// of table locations which must be read in order to satisfy the getMany operation.\n\toffsetRecords, remaining := tr.findOffsets(reqs)\n\ttr.getManyAtOffsets(reqs, offsetRecords, foundChunks, wg, stats)\n\treturn remaining\n}\n\nfunc (tr tableReader) getManyAtOffsets(\n\treqs []getRecord,\n\toffsetRecords offsetRecSlice,\n\tfoundChunks chan *chunks.Chunk,\n\twg *sync.WaitGroup,\n\tstats *Stats,\n) {\n\t// Now |offsetRecords| contains all locations within the table which must be search (note\n\t// that there may be duplicates of a particular location). Sort by offset and scan forward,\n\t// grouping sequences of reads into large physical reads.\n\n\tvar batch offsetRecSlice\n\tvar readStart, readEnd uint64\n\n\tfor i := 0; i < len(offsetRecords); {\n\t\trec := offsetRecords[i]\n\t\tlength := tr.lengths[rec.ordinal]\n\n\t\tif batch == nil {\n\t\t\tbatch = make(offsetRecSlice, 1)\n\t\t\tbatch[0] = offsetRecords[i]\n\t\t\treadStart = rec.offset\n\t\t\treadEnd = readStart + uint64(length)\n\t\t\ti++\n\t\t\tcontinue\n\t\t}\n\n\t\tif newReadEnd, canRead := canReadAhead(rec, tr.lengths[rec.ordinal], readStart, readEnd, tr.blockSize); canRead {\n\t\t\tbatch = append(batch, rec)\n\t\t\treadEnd = newReadEnd\n\t\t\ti++\n\t\t\tcontinue\n\t\t}\n\n\t\twg.Add(1)\n\t\tgo tr.readAtOffsets(readStart, readEnd, reqs, batch, foundChunks, wg, stats)\n\t\tbatch = nil\n\t}\n\n\tif batch != nil {\n\t\twg.Add(1)\n\t\tgo tr.readAtOffsets(readStart, readEnd, reqs, batch, foundChunks, wg, stats)\n\t\tbatch = nil\n\t}\n\n\treturn\n}\n\n// findOffsets iterates over |reqs| and |tr.prefixes| (both sorted by\n// address) to build the set of table locations which must be read in order to\n// find each chunk specified by |reqs|. If this table contains all requested\n// chunks remaining will be set to false upon return. If some are not here,\n// then remaining will be true. The result offsetRecSlice is sorted in offset\n// order.\nfunc (tr tableReader) findOffsets(reqs []getRecord) (ors offsetRecSlice, remaining bool) {\n\tfilterIdx := uint32(0)\n\tfilterLen := uint32(len(tr.prefixes))\n\tors = make(offsetRecSlice, 0, len(reqs))\n\n\t// Iterate over |reqs| and |tr.prefixes| (both sorted by address) and build the set\n\t// of table locations which must be read in order to satisfy |reqs|.\n\tfor i, req := range reqs {\n\t\tif req.found {\n\t\t\tcontinue\n\t\t}\n\n\t\t// advance within the prefixes until we reach one which is >= req.prefix\n\t\tfor filterIdx < filterLen && tr.prefixes[filterIdx] < req.prefix {\n\t\t\tfilterIdx++\n\t\t}\n\n\t\tif filterIdx >= filterLen {\n\t\t\tremaining = true // last prefix visited.\n\t\t\tbreak\n\t\t}\n\n\t\tif req.prefix != tr.prefixes[filterIdx] {\n\t\t\tremaining = true\n\t\t\tcontinue\n\t\t}\n\n\t\t// record all offsets within the table which contain the data required.\n\t\tfor j := filterIdx; j < filterLen && req.prefix == tr.prefixes[j]; j++ {\n\t\t\tif tr.ordinalSuffixMatches(tr.prefixIdxToOrdinal(j), *req.a) {\n\t\t\t\treqs[i].found = true\n\t\t\t\tors = append(ors, offsetRec{req.a, tr.ordinals[j], tr.offsets[tr.ordinals[j]]})\n\t\t\t}\n\t\t}\n\t}\n\n\tsort.Sort(ors)\n\treturn ors, remaining\n}\n\nfunc canReadAhead(fRec offsetRec, fLength uint32, readStart, readEnd, blockSize uint64) (newEnd uint64, canRead bool) {\n\tif fRec.offset < readEnd {\n\t\t// |offsetRecords| will contain an offsetRecord for *every* chunkRecord whose address\n\t\t// prefix matches the prefix of a requested address. If the set of requests contains\n\t\t// addresses which share a common prefix, then it's possible for multiple offsetRecords\n\t\t// to reference the same table offset position. In that case, we'll see sequential\n\t\t// offsetRecords with the same fRec.offset.\n\t\treturn readEnd, true\n\t}\n\n\tif fRec.offset-readEnd > blockSize {\n\t\treturn readEnd, false\n\t}\n\n\treturn fRec.offset + uint64(fLength), true\n}\n\n// Fetches the byte stream of data logically encoded within the table starting at |pos|.\nfunc (tr tableReader) parseChunk(buff []byte) []byte {\n\tdataLen := uint64(len(buff)) - checksumSize\n\n\tchksum := binary.BigEndian.Uint32(buff[dataLen:])\n\td.Chk.True(chksum == crc(buff[:dataLen]))\n\n\tdata, err := snappy.Decode(nil, buff[:dataLen])\n\td.Chk.NoError(err)\n\n\treturn data\n}\n\nfunc (tr tableReader) calcReads(reqs []getRecord, blockSize uint64) (reads int, remaining bool) {\n\tvar offsetRecords offsetRecSlice\n\t// Pass #1: Build the set of table locations which must be read in order to find all the elements of |reqs| which are present in this table.\n\toffsetRecords, remaining = tr.findOffsets(reqs)\n\n\t// Now |offsetRecords| contains all locations within the table which must\n\t// be searched (note that there may be duplicates of a particular\n\t// location). Scan forward, grouping sequences of reads into large physical\n\t// reads.\n\n\tvar readStart, readEnd uint64\n\treadStarted := false\n\n\tfor i := 0; i < len(offsetRecords); {\n\t\trec := offsetRecords[i]\n\t\tlength := tr.lengths[rec.ordinal]\n\n\t\tif !readStarted {\n\t\t\treadStarted = true\n\t\t\treads++\n\t\t\treadStart = rec.offset\n\t\t\treadEnd = readStart + uint64(length)\n\t\t\ti++\n\t\t\tcontinue\n\t\t}\n\n\t\tif newReadEnd, canRead := canReadAhead(rec, tr.lengths[rec.ordinal], readStart, readEnd, tr.blockSize); canRead {\n\t\t\treadEnd = newReadEnd\n\t\t\ti++\n\t\t\tcontinue\n\t\t}\n\n\t\treadStarted = false\n\t}\n\n\treturn\n}\n\nfunc (tr tableReader) extract(chunks chan<- extractRecord) {\n\t// Build reverse lookup table from ordinal -> chunk hash\n\thashes := make(addrSlice, len(tr.prefixes))\n\tfor idx, prefix := range tr.prefixes {\n\t\tordinal := tr.prefixIdxToOrdinal(uint32(idx))\n\t\tbinary.BigEndian.PutUint64(hashes[ordinal][:], prefix)\n\t\tli := uint64(ordinal) * addrSuffixSize\n\t\tcopy(hashes[ordinal][addrPrefixSize:], tr.suffixes[li:li+addrSuffixSize])\n\t}\n\tchunkLen := tr.offsets[tr.chunkCount-1] + uint64(tr.lengths[tr.chunkCount-1])\n\tbuff := make([]byte, chunkLen)\n\tn, err := tr.r.ReadAtWithStats(buff, int64(tr.offsets[0]), &Stats{})\n\td.Chk.NoError(err)\n\td.Chk.True(uint64(n) == chunkLen)\n\n\tsendChunk := func(i uint32) {\n\t\tlocalOffset := tr.offsets[i] - tr.offsets[0]\n\t\tchunks <- extractRecord{a: hashes[i], data: tr.parseChunk(buff[localOffset : localOffset+uint64(tr.lengths[i])])}\n\t}\n\n\tfor i := uint32(0); i < tr.chunkCount; i++ {\n\t\tsendChunk(i)\n\t}\n}\n\nfunc (tr tableReader) reader() io.Reader {\n\treturn &readerAdapter{tr.r, 0}\n}\n\ntype readerAdapter struct {\n\trat tableReaderAt\n\toff int64\n}\n\nfunc (ra *readerAdapter) Read(p []byte) (n int, err error) {\n\tn, err = ra.rat.ReadAtWithStats(p, ra.off, &Stats{})\n\tra.off += int64(n)\n\treturn\n}\n" }, { "path": "go/nbs/table_set.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nconst concurrentCompactions = 5\n\nfunc newTableSet(persister tablePersister) tableSet {\n\treturn tableSet{p: persister, rl: make(chan struct{}, concurrentCompactions)}\n}\n\n// tableSet is an immutable set of persistable chunkSources.\ntype tableSet struct {\n\tnovel, upstream chunkSources\n\tp tablePersister\n\trl chan struct{}\n}\n\nfunc (ts tableSet) has(h addr) bool {\n\tf := func(css chunkSources) bool {\n\t\tfor _, haver := range css {\n\t\t\tif haver.has(h) {\n\t\t\t\treturn true\n\t\t\t}\n\t\t}\n\t\treturn false\n\t}\n\treturn f(ts.novel) || f(ts.upstream)\n}\n\nfunc (ts tableSet) hasMany(addrs []hasRecord) (remaining bool) {\n\tf := func(css chunkSources) (remaining bool) {\n\t\tfor _, haver := range css {\n\t\t\tif !haver.hasMany(addrs) {\n\t\t\t\treturn false\n\t\t\t}\n\t\t}\n\t\treturn true\n\t}\n\treturn f(ts.novel) && f(ts.upstream)\n}\n\nfunc (ts tableSet) get(h addr, stats *Stats) []byte {\n\tf := func(css chunkSources) []byte {\n\t\tfor _, haver := range css {\n\t\t\tif data := haver.get(h, stats); data != nil {\n\t\t\t\treturn data\n\t\t\t}\n\t\t}\n\t\treturn nil\n\t}\n\tif data := f(ts.novel); data != nil {\n\t\treturn data\n\t}\n\treturn f(ts.upstream)\n}\n\nfunc (ts tableSet) getMany(reqs []getRecord, foundChunks chan *chunks.Chunk, wg *sync.WaitGroup, stats *Stats) (remaining bool) {\n\tf := func(css chunkSources) (remaining bool) {\n\t\tfor _, haver := range css {\n\t\t\tif rp, ok := haver.(chunkReadPlanner); ok {\n\t\t\t\toffsets, remaining := rp.findOffsets(reqs)\n\t\t\t\tgo rp.getManyAtOffsets(reqs, offsets, foundChunks, wg, stats)\n\t\t\t\tif !remaining {\n\t\t\t\t\treturn false\n\t\t\t\t}\n\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif !haver.getMany(reqs, foundChunks, wg, stats) {\n\t\t\t\treturn false\n\t\t\t}\n\t\t}\n\t\treturn true\n\t}\n\treturn f(ts.novel) && f(ts.upstream)\n}\n\nfunc (ts tableSet) calcReads(reqs []getRecord, blockSize uint64) (reads int, split, remaining bool) {\n\tf := func(css chunkSources) (int, bool, bool) {\n\t\treads, split := 0, false\n\t\tfor _, haver := range css {\n\t\t\trds, rmn := haver.calcReads(reqs, blockSize)\n\t\t\treads += rds\n\t\t\tif !rmn {\n\t\t\t\treturn reads, split, false\n\t\t\t}\n\t\t\tsplit = true\n\t\t}\n\t\treturn reads, split, true\n\t}\n\treads, split, remaining = f(ts.novel)\n\tif remaining {\n\t\tvar rds int\n\t\trds, split, remaining = f(ts.upstream)\n\t\treads += rds\n\t}\n\treturn reads, split, remaining\n}\n\nfunc (ts tableSet) count() uint32 {\n\tf := func(css chunkSources) (count uint32) {\n\t\tfor _, haver := range css {\n\t\t\tcount += haver.count()\n\t\t}\n\t\treturn\n\t}\n\treturn f(ts.novel) + f(ts.upstream)\n}\n\nfunc (ts tableSet) uncompressedLen() uint64 {\n\tf := func(css chunkSources) (data uint64) {\n\t\tfor _, haver := range css {\n\t\t\tdata += haver.uncompressedLen()\n\t\t}\n\t\treturn\n\t}\n\treturn f(ts.novel) + f(ts.upstream)\n}\n\nfunc (ts tableSet) physicalLen() uint64 {\n\tf := func(css chunkSources) (data uint64) {\n\t\tfor _, haver := range css {\n\t\t\tindex := haver.index()\n\t\t\tdata += indexSize(index.chunkCount)\n\t\t\tdata += index.offsets[index.chunkCount-1] + (uint64(index.lengths[index.chunkCount-1]))\n\t\t}\n\t\treturn\n\t}\n\treturn f(ts.novel) + f(ts.upstream)\n}\n\n// Size returns the number of tables in this tableSet.\nfunc (ts tableSet) Size() int {\n\treturn len(ts.novel) + len(ts.upstream)\n}\n\n// Novel returns the number of tables containing novel chunks in this\n// tableSet.\nfunc (ts tableSet) Novel() int {\n\treturn len(ts.novel)\n}\n\n// Upstream returns the number of known-persisted tables in this tableSet.\nfunc (ts tableSet) Upstream() int {\n\treturn len(ts.upstream)\n}\n\n// Prepend adds a memTable to an existing tableSet, compacting |mt| and\n// returning a new tableSet with newly compacted table added.\nfunc (ts tableSet) Prepend(mt *memTable, stats *Stats) tableSet {\n\tnewTs := tableSet{\n\t\tnovel: make(chunkSources, len(ts.novel)+1),\n\t\tupstream: make(chunkSources, len(ts.upstream)),\n\t\tp: ts.p,\n\t\trl: ts.rl,\n\t}\n\tnewTs.novel[0] = newPersistingChunkSource(mt, ts, ts.p, ts.rl, stats)\n\tcopy(newTs.novel[1:], ts.novel)\n\tcopy(newTs.upstream, ts.upstream)\n\treturn newTs\n}\n\nfunc (ts tableSet) extract(chunks chan<- extractRecord) {\n\t// Since new tables are _prepended_ to a tableSet, extracting chunks in insertOrder requires iterating ts.upstream back to front, followed by ts.novel.\n\tfor i := len(ts.upstream) - 1; i >= 0; i-- {\n\t\tts.upstream[i].extract(chunks)\n\t}\n\tfor i := len(ts.novel) - 1; i >= 0; i-- {\n\t\tts.novel[i].extract(chunks)\n\t}\n}\n\n// Flatten returns a new tableSet with |upstream| set to the union of ts.novel\n// and ts.upstream.\nfunc (ts tableSet) Flatten() (flattened tableSet) {\n\tflattened = tableSet{\n\t\tupstream: make(chunkSources, 0, ts.Size()),\n\t\tp: ts.p,\n\t\trl: ts.rl,\n\t}\n\tfor _, src := range ts.novel {\n\t\tif src.count() > 0 {\n\t\t\tflattened.upstream = append(flattened.upstream, src)\n\t\t}\n\t}\n\tflattened.upstream = append(flattened.upstream, ts.upstream...)\n\treturn\n}\n\n// Rebase returns a new tableSet holding the novel tables managed by |ts| and\n// those specified by |specs|.\nfunc (ts tableSet) Rebase(specs []tableSpec, stats *Stats) tableSet {\n\tmerged := tableSet{\n\t\tnovel: make(chunkSources, 0, len(ts.novel)),\n\t\tupstream: make(chunkSources, 0, len(specs)),\n\t\tp: ts.p,\n\t\trl: ts.rl,\n\t}\n\n\t// Rebase the novel tables, skipping those that are actually empty (usually due to de-duping during table compaction)\n\tfor _, t := range ts.novel {\n\t\tif t.count() > 0 {\n\t\t\tmerged.novel = append(merged.novel, t)\n\t\t}\n\t}\n\n\t// Create a list of tables to open so we can open them in parallel.\n\ttablesToOpen := map[addr]tableSpec{}\n\tfor _, spec := range specs {\n\t\tif _, present := tablesToOpen[spec.name]; !present { // Filter out dups\n\t\t\ttablesToOpen[spec.name] = spec\n\t\t}\n\t}\n\n\t// Open all the new upstream tables concurrently\n\tmerged.upstream = make(chunkSources, len(tablesToOpen))\n\twg := &sync.WaitGroup{}\n\ti := 0\n\tfor _, spec := range tablesToOpen {\n\t\twg.Add(1)\n\t\tgo func(idx int, spec tableSpec) {\n\t\t\tmerged.upstream[idx] = ts.p.Open(spec.name, spec.chunkCount, stats)\n\t\t\twg.Done()\n\t\t}(i, spec)\n\t\ti++\n\t}\n\twg.Wait()\n\n\treturn merged\n}\n\nfunc (ts tableSet) ToSpecs() []tableSpec {\n\ttableSpecs := make([]tableSpec, 0, ts.Size())\n\tfor _, src := range ts.novel {\n\t\tif src.count() > 0 {\n\t\t\ttableSpecs = append(tableSpecs, tableSpec{src.hash(), src.count()})\n\t\t}\n\t}\n\tfor _, src := range ts.upstream {\n\t\td.Chk.True(src.count() > 0)\n\t\ttableSpecs = append(tableSpecs, tableSpec{src.hash(), src.count()})\n\t}\n\treturn tableSpecs\n}\n" }, { "path": "go/nbs/table_set_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nvar testChunks = [][]byte{[]byte(\"hello2\"), []byte(\"goodbye2\"), []byte(\"badbye2\")}\n\nfunc TestTableSetPrependEmpty(t *testing.T) {\n\tts := newFakeTableSet().Prepend(newMemTable(testMemTableSize), &Stats{})\n\tassert.Empty(t, ts.ToSpecs())\n}\n\nfunc TestTableSetPrepend(t *testing.T) {\n\tassert := assert.New(t)\n\tts := newFakeTableSet()\n\tassert.Empty(ts.ToSpecs())\n\tmt := newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[0]), testChunks[0])\n\tts = ts.Prepend(mt, &Stats{})\n\n\tfirstSpecs := ts.ToSpecs()\n\tassert.Len(firstSpecs, 1)\n\n\tmt = newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[1]), testChunks[1])\n\tmt.addChunk(computeAddr(testChunks[2]), testChunks[2])\n\tts = ts.Prepend(mt, &Stats{})\n\n\tsecondSpecs := ts.ToSpecs()\n\tassert.Len(secondSpecs, 2)\n\tassert.Equal(firstSpecs, secondSpecs[1:])\n}\n\nfunc TestTableSetToSpecsExcludesEmptyTable(t *testing.T) {\n\tassert := assert.New(t)\n\tts := newFakeTableSet()\n\tassert.Empty(ts.ToSpecs())\n\tmt := newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[0]), testChunks[0])\n\tts = ts.Prepend(mt, &Stats{})\n\n\tmt = newMemTable(testMemTableSize)\n\tts = ts.Prepend(mt, &Stats{})\n\n\tmt = newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[1]), testChunks[1])\n\tmt.addChunk(computeAddr(testChunks[2]), testChunks[2])\n\tts = ts.Prepend(mt, &Stats{})\n\n\tspecs := ts.ToSpecs()\n\tassert.Len(specs, 2)\n}\n\nfunc TestTableSetFlattenExcludesEmptyTable(t *testing.T) {\n\tassert := assert.New(t)\n\tts := newFakeTableSet()\n\tassert.Empty(ts.ToSpecs())\n\tmt := newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[0]), testChunks[0])\n\tts = ts.Prepend(mt, &Stats{})\n\n\tmt = newMemTable(testMemTableSize)\n\tts = ts.Prepend(mt, &Stats{})\n\n\tmt = newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[1]), testChunks[1])\n\tmt.addChunk(computeAddr(testChunks[2]), testChunks[2])\n\tts = ts.Prepend(mt, &Stats{})\n\n\tts = ts.Flatten()\n\tassert.EqualValues(ts.Size(), 2)\n}\n\nfunc TestTableSetExtract(t *testing.T) {\n\tassert := assert.New(t)\n\tts := newFakeTableSet()\n\tassert.Empty(ts.ToSpecs())\n\n\t// Put in one table\n\tmt := newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[0]), testChunks[0])\n\tts = ts.Prepend(mt, &Stats{})\n\n\t// Put in a second\n\tmt = newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[1]), testChunks[1])\n\tmt.addChunk(computeAddr(testChunks[2]), testChunks[2])\n\tts = ts.Prepend(mt, &Stats{})\n\n\tchunkChan := make(chan extractRecord)\n\tgo func() { defer close(chunkChan); ts.extract(chunkChan) }()\n\ti := 0\n\tfor rec := range chunkChan {\n\t\ta := computeAddr(testChunks[i])\n\t\tassert.NotNil(rec.data, \"Nothing for\", a)\n\t\tassert.Equal(testChunks[i], rec.data, \"Item %d: %s != %s\", i, string(testChunks[i]), string(rec.data))\n\t\tassert.Equal(a, rec.a)\n\t\ti++\n\t}\n}\n\nfunc TestTableSetRebase(t *testing.T) {\n\tassert := assert.New(t)\n\tpersister := newFakeTablePersister()\n\n\tinsert := func(ts tableSet, chunks ...[]byte) tableSet {\n\t\tfor _, c := range chunks {\n\t\t\tmt := newMemTable(testMemTableSize)\n\t\t\tmt.addChunk(computeAddr(c), c)\n\t\t\tts = ts.Prepend(mt, &Stats{})\n\t\t}\n\t\treturn ts\n\t}\n\tfullTS := newTableSet(persister)\n\tassert.Empty(fullTS.ToSpecs())\n\tfullTS = insert(fullTS, testChunks...)\n\tfullTS = fullTS.Flatten()\n\n\tts := newTableSet(persister)\n\tts = insert(ts, testChunks[0])\n\tassert.Equal(1, ts.Size())\n\tts = ts.Flatten()\n\tts = insert(ts, []byte(\"novel\"))\n\n\tts = ts.Rebase(fullTS.ToSpecs(), nil)\n\tassert.Equal(4, ts.Size())\n}\n\nfunc TestTableSetPhysicalLen(t *testing.T) {\n\tassert := assert.New(t)\n\tts := newFakeTableSet()\n\tassert.Empty(ts.ToSpecs())\n\tmt := newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[0]), testChunks[0])\n\tts = ts.Prepend(mt, &Stats{})\n\n\tmt = newMemTable(testMemTableSize)\n\tmt.addChunk(computeAddr(testChunks[1]), testChunks[1])\n\tmt.addChunk(computeAddr(testChunks[2]), testChunks[2])\n\tts = ts.Prepend(mt, &Stats{})\n\n\tassert.True(ts.physicalLen() > indexSize(ts.count()))\n}\n" }, { "path": "go/nbs/table_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"encoding/binary\"\n\t\"fmt\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc buildTable(chunks [][]byte) ([]byte, addr) {\n\ttotalData := uint64(0)\n\tfor _, chunk := range chunks {\n\t\ttotalData += uint64(len(chunk))\n\t}\n\tcapacity := maxTableSize(uint64(len(chunks)), totalData)\n\n\tbuff := make([]byte, capacity)\n\n\ttw := newTableWriter(buff, nil)\n\n\tfor _, chunk := range chunks {\n\t\ttw.addChunk(computeAddr(chunk), chunk)\n\t}\n\n\tlength, blockHash := tw.finish()\n\treturn buff[:length], blockHash\n}\n\nfunc TestSimple(t *testing.T) {\n\tassert := assert.New(t)\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\ttableData, _ := buildTable(chunks)\n\ttr := newTableReader(parseTableIndex(tableData), tableReaderAtFromBytes(tableData), fileBlockSize)\n\n\tassertChunksInReader(chunks, tr, assert)\n\n\tassert.Equal(string(chunks[0]), string(tr.get(computeAddr(chunks[0]), &Stats{})))\n\tassert.Equal(string(chunks[1]), string(tr.get(computeAddr(chunks[1]), &Stats{})))\n\tassert.Equal(string(chunks[2]), string(tr.get(computeAddr(chunks[2]), &Stats{})))\n\n\tnotPresent := [][]byte{\n\t\t[]byte(\"yo\"),\n\t\t[]byte(\"do\"),\n\t\t[]byte(\"so much to do\"),\n\t}\n\n\tassertChunksNotInReader(notPresent, tr, assert)\n\n\tassert.NotEqual(string(notPresent[0]), string(tr.get(computeAddr(notPresent[0]), &Stats{})))\n\tassert.NotEqual(string(notPresent[1]), string(tr.get(computeAddr(notPresent[1]), &Stats{})))\n\tassert.NotEqual(string(notPresent[2]), string(tr.get(computeAddr(notPresent[2]), &Stats{})))\n}\n\nfunc assertChunksInReader(chunks [][]byte, r chunkReader, assert *assert.Assertions) {\n\tfor _, c := range chunks {\n\t\tassert.True(r.has(computeAddr(c)))\n\t}\n}\n\nfunc assertChunksNotInReader(chunks [][]byte, r chunkReader, assert *assert.Assertions) {\n\tfor _, c := range chunks {\n\t\tassert.False(r.has(computeAddr(c)))\n\t}\n}\n\nfunc TestHasMany(t *testing.T) {\n\tassert := assert.New(t)\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\ttableData, _ := buildTable(chunks)\n\ttr := newTableReader(parseTableIndex(tableData), tableReaderAtFromBytes(tableData), fileBlockSize)\n\n\taddrs := addrSlice{computeAddr(chunks[0]), computeAddr(chunks[1]), computeAddr(chunks[2])}\n\thasAddrs := []hasRecord{\n\t\t{&addrs[0], binary.BigEndian.Uint64(addrs[0][:addrPrefixSize]), 0, false},\n\t\t{&addrs[1], binary.BigEndian.Uint64(addrs[1][:addrPrefixSize]), 1, false},\n\t\t{&addrs[2], binary.BigEndian.Uint64(addrs[2][:addrPrefixSize]), 2, false},\n\t}\n\tsort.Sort(hasRecordByPrefix(hasAddrs))\n\n\ttr.hasMany(hasAddrs)\n\tfor _, ha := range hasAddrs {\n\t\tassert.True(ha.has, \"Nothing for prefix %d\", ha.prefix)\n\t}\n}\n\nfunc TestHasManySequentialPrefix(t *testing.T) {\n\tassert := assert.New(t)\n\n\t// Use bogus addrs so we can generate the case of sequentially non-unique prefixes in the index\n\t// Note that these are already sorted\n\taddrStrings := []string{\n\t\t\"0rfgadopg6h3fk7d253ivbjsij4qo3nv\",\n\t\t\"0rfgadopg6h3fk7d253ivbjsij4qo4nv\",\n\t\t\"0rfgadopg6h3fk7d253ivbjsij4qo9nv\",\n\t}\n\n\taddrs := make([]addr, len(addrStrings))\n\tfor i, s := range addrStrings {\n\t\taddrs[i] = addr(hash.Parse(s))\n\t}\n\n\tbogusData := []byte(\"bogus\") // doesn't matter what this is. hasMany() won't check chunkRecords\n\ttotalData := uint64(len(bogusData) * len(addrs))\n\n\tcapacity := maxTableSize(uint64(len(addrs)), totalData)\n\tbuff := make([]byte, capacity)\n\ttw := newTableWriter(buff, nil)\n\n\tfor _, a := range addrs {\n\t\ttw.addChunk(a, bogusData)\n\t}\n\n\tlength, _ := tw.finish()\n\tbuff = buff[:length]\n\n\ttr := newTableReader(parseTableIndex(buff), tableReaderAtFromBytes(buff), fileBlockSize)\n\n\thasAddrs := make([]hasRecord, 2)\n\t// Leave out the first address\n\thasAddrs[0] = hasRecord{&addrs[1], addrs[1].Prefix(), 1, false}\n\thasAddrs[1] = hasRecord{&addrs[2], addrs[2].Prefix(), 2, false}\n\n\ttr.hasMany(hasAddrs)\n\n\tfor _, ha := range hasAddrs {\n\t\tassert.True(ha.has, fmt.Sprintf(\"Nothing for prefix %x\\n\", ha.prefix))\n\t}\n}\n\nfunc TestGetMany(t *testing.T) {\n\tassert := assert.New(t)\n\n\tdata := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\ttableData, _ := buildTable(data)\n\ttr := newTableReader(parseTableIndex(tableData), tableReaderAtFromBytes(tableData), fileBlockSize)\n\n\taddrs := addrSlice{computeAddr(data[0]), computeAddr(data[1]), computeAddr(data[2])}\n\tgetBatch := []getRecord{\n\t\t{&addrs[0], binary.BigEndian.Uint64(addrs[0][:addrPrefixSize]), false},\n\t\t{&addrs[1], binary.BigEndian.Uint64(addrs[1][:addrPrefixSize]), false},\n\t\t{&addrs[2], binary.BigEndian.Uint64(addrs[2][:addrPrefixSize]), false},\n\t}\n\tsort.Sort(getRecordByPrefix(getBatch))\n\n\twg := &sync.WaitGroup{}\n\n\tchunkChan := make(chan *chunks.Chunk, len(getBatch))\n\ttr.getMany(getBatch, chunkChan, wg, &Stats{})\n\twg.Wait()\n\tclose(chunkChan)\n\n\tgotCount := 0\n\tfor range chunkChan {\n\t\tgotCount++\n\t}\n\n\tassert.True(gotCount == len(getBatch))\n}\n\nfunc TestCalcReads(t *testing.T) {\n\tassert := assert.New(t)\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\ttableData, _ := buildTable(chunks)\n\ttr := newTableReader(parseTableIndex(tableData), tableReaderAtFromBytes(tableData), 0)\n\taddrs := addrSlice{computeAddr(chunks[0]), computeAddr(chunks[1]), computeAddr(chunks[2])}\n\tgetBatch := []getRecord{\n\t\t{&addrs[0], binary.BigEndian.Uint64(addrs[0][:addrPrefixSize]), false},\n\t\t{&addrs[1], binary.BigEndian.Uint64(addrs[1][:addrPrefixSize]), false},\n\t\t{&addrs[2], binary.BigEndian.Uint64(addrs[2][:addrPrefixSize]), false},\n\t}\n\n\tgb2 := []getRecord{getBatch[0], getBatch[2]}\n\tsort.Sort(getRecordByPrefix(getBatch))\n\n\treads, remaining := tr.calcReads(getBatch, 0)\n\tassert.False(remaining)\n\tassert.Equal(1, reads)\n\n\tsort.Sort(getRecordByPrefix(gb2))\n\treads, remaining = tr.calcReads(gb2, 0)\n\tassert.False(remaining)\n\tassert.Equal(2, reads)\n}\n\nfunc TestExtract(t *testing.T) {\n\tassert := assert.New(t)\n\n\tchunks := [][]byte{\n\t\t[]byte(\"hello2\"),\n\t\t[]byte(\"goodbye2\"),\n\t\t[]byte(\"badbye2\"),\n\t}\n\n\ttableData, _ := buildTable(chunks)\n\ttr := newTableReader(parseTableIndex(tableData), tableReaderAtFromBytes(tableData), fileBlockSize)\n\n\taddrs := addrSlice{computeAddr(chunks[0]), computeAddr(chunks[1]), computeAddr(chunks[2])}\n\n\tchunkChan := make(chan extractRecord)\n\tgo func() { tr.extract(chunkChan); close(chunkChan) }()\n\ti := 0\n\tfor rec := range chunkChan {\n\t\tassert.NotNil(rec.data, \"Nothing for\", addrs[i])\n\t\tassert.Equal(addrs[i], rec.a)\n\t\tassert.Equal(chunks[i], rec.data)\n\t\ti++\n\t}\n}\n\nfunc Test65k(t *testing.T) {\n\tassert := assert.New(t)\n\n\tcount := 1 << 16\n\tchunks := make([][]byte, count)\n\n\tdataFn := func(i int) []byte {\n\t\treturn []byte(fmt.Sprintf(\"data%d\", i*2))\n\t}\n\n\tfor i := 0; i < count; i++ {\n\t\tchunks[i] = dataFn(i)\n\t}\n\n\ttableData, _ := buildTable(chunks)\n\ttr := newTableReader(parseTableIndex(tableData), tableReaderAtFromBytes(tableData), fileBlockSize)\n\n\tfor i := 0; i < count; i++ {\n\t\tdata := dataFn(i)\n\t\th := computeAddr(data)\n\t\tassert.True(tr.has(computeAddr(data)))\n\t\tassert.Equal(string(data), string(tr.get(h, &Stats{})))\n\t}\n\n\tfor i := count; i < count*2; i++ {\n\t\tdata := dataFn(i)\n\t\th := computeAddr(data)\n\t\tassert.False(tr.has(computeAddr(data)))\n\t\tassert.NotEqual(string(data), string(tr.get(h, &Stats{})))\n\t}\n}\n\n// Ensure all addresses share the first 7 bytes. Useful for easily generating tests which have\n// \"prefix\" collisions.\nfunc computeAddrCommonPrefix(data []byte) addr {\n\ta := computeAddrDefault(data)\n\ta[0] = 0x01\n\ta[1] = 0x23\n\ta[2] = 0x45\n\ta[3] = 0x67\n\ta[4] = 0x89\n\ta[5] = 0xab\n\ta[6] = 0xcd\n\treturn a\n}\n\nfunc doTestNGetMany(t *testing.T, count int) {\n\tassert := assert.New(t)\n\n\tdata := make([][]byte, count)\n\n\tdataFn := func(i int) []byte {\n\t\treturn []byte(fmt.Sprintf(\"data%d\", i*2))\n\t}\n\n\tfor i := 0; i < count; i++ {\n\t\tdata[i] = dataFn(i)\n\t}\n\n\ttableData, _ := buildTable(data)\n\ttr := newTableReader(parseTableIndex(tableData), tableReaderAtFromBytes(tableData), fileBlockSize)\n\n\tgetBatch := make([]getRecord, len(data))\n\tfor i := 0; i < count; i++ {\n\t\ta := computeAddr(dataFn(i))\n\t\tgetBatch[i] = getRecord{&a, a.Prefix(), false}\n\t}\n\n\tsort.Sort(getRecordByPrefix(getBatch))\n\n\twg := &sync.WaitGroup{}\n\tchunkChan := make(chan *chunks.Chunk, len(getBatch))\n\ttr.getMany(getBatch, chunkChan, wg, &Stats{})\n\twg.Wait()\n\tclose(chunkChan)\n\n\tgotCount := 0\n\tfor range chunkChan {\n\t\tgotCount++\n\t}\n\n\tassert.True(gotCount == len(getBatch))\n}\n\nfunc Test65kGetMany(t *testing.T) {\n\tdoTestNGetMany(t, 1<<16)\n}\n\nfunc Test2kGetManyCommonPrefix(t *testing.T) {\n\tcomputeAddr = computeAddrCommonPrefix\n\tdefer func() {\n\t\tcomputeAddr = computeAddrDefault\n\t}()\n\n\tdoTestNGetMany(t, 1<<11)\n}\n\nfunc TestEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\n\tbuff := make([]byte, footerSize)\n\ttw := newTableWriter(buff, nil)\n\tlength, _ := tw.finish()\n\tassert.Equal(length, footerSize)\n\n\td.PanicIfError(nil)\n}\n" }, { "path": "go/nbs/table_writer.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nbs\n\nimport (\n\t\"crypto/sha512\"\n\t\"encoding/binary\"\n\t\"fmt\"\n\t\"hash\"\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/golang/snappy\"\n)\n\n// tableWriter encodes a collection of byte stream chunks into a nbs table. NOT goroutine safe.\ntype tableWriter struct {\n\tbuff []byte\n\tpos uint64\n\ttotalCompressedData uint64\n\ttotalUncompressedData uint64\n\tprefixes prefixIndexSlice // TODO: This is in danger of exploding memory\n\tblockHash hash.Hash\n\n\tsnapper snappyEncoder\n}\n\ntype snappyEncoder interface {\n\tEncode(dst, src []byte) []byte\n}\n\ntype realSnappyEncoder struct{}\n\nfunc (r realSnappyEncoder) Encode(dst, src []byte) []byte {\n\treturn snappy.Encode(dst, src)\n}\n\nfunc maxTableSize(numChunks, totalData uint64) uint64 {\n\tavgChunkSize := totalData / numChunks\n\td.Chk.True(avgChunkSize < maxChunkSize)\n\tmaxSnappySize := snappy.MaxEncodedLen(int(avgChunkSize))\n\td.Chk.True(maxSnappySize > 0)\n\treturn numChunks*(prefixTupleSize+lengthSize+addrSuffixSize+checksumSize+uint64(maxSnappySize)) + footerSize\n}\n\nfunc indexSize(numChunks uint32) uint64 {\n\treturn uint64(numChunks) * (addrSuffixSize + lengthSize + prefixTupleSize)\n}\n\nfunc lengthsOffset(numChunks uint32) uint64 {\n\treturn uint64(numChunks) * prefixTupleSize\n}\n\nfunc suffixesOffset(numChunks uint32) uint64 {\n\treturn uint64(numChunks) * (prefixTupleSize + lengthSize)\n}\n\n// len(buff) must be >= maxTableSize(numChunks, totalData)\nfunc newTableWriter(buff []byte, snapper snappyEncoder) *tableWriter {\n\tif snapper == nil {\n\t\tsnapper = realSnappyEncoder{}\n\t}\n\treturn &tableWriter{\n\t\tbuff: buff,\n\t\tblockHash: sha512.New(),\n\t\tsnapper: snapper,\n\t}\n}\n\nfunc (tw *tableWriter) addChunk(h addr, data []byte) bool {\n\tif len(data) == 0 {\n\t\tpanic(\"NBS blocks cannont be zero length\")\n\t}\n\n\t// Compress data straight into tw.buff\n\tcompressed := tw.snapper.Encode(tw.buff[tw.pos:], data)\n\tdataLength := uint64(len(compressed))\n\ttw.totalCompressedData += dataLength\n\n\t// BUG 3156 indicated that, sometimes, snappy decided that there's not enough space in tw.buff[tw.pos:] to encode into.\n\t// This _should never happen anymore be_, because we iterate over all chunks to be added and sum the max amount of space that snappy says it might need.\n\t// Since we know that |data| can't be 0-length, we also know that the compressed version of |data| has length greater than zero. The first element in a snappy-encoded blob is a Uvarint indicating how much data is present. Therefore, if there's a Uvarint-encoded 0 at tw.buff[tw.pos:], we know that snappy did not write anything there and we have a problem.\n\tif v, n := binary.Uvarint(tw.buff[tw.pos:]); v == 0 {\n\t\td.Chk.True(n != 0)\n\t\tpanic(fmt.Errorf(\"BUG 3156: unbuffered chunk %s: uncompressed %d, compressed %d, snappy max %d, tw.buff %d\\n\", h.String(), len(data), dataLength, snappy.MaxEncodedLen(len(data)), len(tw.buff[tw.pos:])))\n\t}\n\n\ttw.pos += dataLength\n\ttw.totalUncompressedData += uint64(len(data))\n\n\t// checksum (4 LSBytes, big-endian)\n\tbinary.BigEndian.PutUint32(tw.buff[tw.pos:], crc(compressed))\n\ttw.pos += checksumSize\n\n\t// Stored in insertion order\n\ttw.prefixes = append(tw.prefixes, prefixIndexRec{\n\t\th.Prefix(),\n\t\th[addrPrefixSize:],\n\t\tuint32(len(tw.prefixes)),\n\t\tuint32(checksumSize + dataLength),\n\t})\n\n\treturn true\n}\n\nfunc (tw *tableWriter) finish() (uncompressedLength uint64, blockAddr addr) {\n\ttw.writeIndex()\n\ttw.writeFooter()\n\tuncompressedLength = tw.pos\n\n\tvar h []byte\n\th = tw.blockHash.Sum(h) // Appends hash to h\n\tcopy(blockAddr[:], h)\n\treturn\n}\n\ntype prefixIndexRec struct {\n\tprefix uint64\n\tsuffix []byte\n\torder, size uint32\n}\n\ntype prefixIndexSlice []prefixIndexRec\n\nfunc (hs prefixIndexSlice) Len() int { return len(hs) }\nfunc (hs prefixIndexSlice) Less(i, j int) bool { return hs[i].prefix < hs[j].prefix }\nfunc (hs prefixIndexSlice) Swap(i, j int) { hs[i], hs[j] = hs[j], hs[i] }\n\nfunc (tw *tableWriter) writeIndex() {\n\tsort.Sort(tw.prefixes)\n\n\tpfxScratch := [addrPrefixSize]byte{}\n\n\tnumRecords := uint32(len(tw.prefixes))\n\tlengthsOffset := tw.pos + lengthsOffset(numRecords) // skip prefix and ordinal for each record\n\tsuffixesOffset := tw.pos + suffixesOffset(numRecords) // skip size for each record\n\tfor _, pi := range tw.prefixes {\n\t\tbinary.BigEndian.PutUint64(pfxScratch[:], pi.prefix)\n\n\t\t// hash prefix\n\t\tn := uint64(copy(tw.buff[tw.pos:], pfxScratch[:]))\n\t\td.Chk.True(n == addrPrefixSize)\n\t\ttw.pos += n\n\n\t\t// order\n\t\tbinary.BigEndian.PutUint32(tw.buff[tw.pos:], pi.order)\n\t\ttw.pos += ordinalSize\n\n\t\t// length\n\t\toffset := lengthsOffset + uint64(pi.order)*lengthSize\n\t\tbinary.BigEndian.PutUint32(tw.buff[offset:], pi.size)\n\n\t\t// hash suffix\n\t\toffset = suffixesOffset + uint64(pi.order)*addrSuffixSize\n\t\tn = uint64(copy(tw.buff[offset:], pi.suffix))\n\t\td.Chk.True(n == addrSuffixSize)\n\t}\n\tsuffixesLen := uint64(numRecords) * addrSuffixSize\n\ttw.blockHash.Write(tw.buff[suffixesOffset : suffixesOffset+suffixesLen])\n\ttw.pos = suffixesOffset + suffixesLen\n}\n\nfunc (tw *tableWriter) writeFooter() {\n\ttw.pos += writeFooter(tw.buff[tw.pos:], uint32(len(tw.prefixes)), tw.totalUncompressedData)\n}\n\nfunc writeFooter(dst []byte, chunkCount uint32, uncData uint64) (consumed uint64) {\n\t// chunk count\n\tbinary.BigEndian.PutUint32(dst[consumed:], chunkCount)\n\tconsumed += uint32Size\n\n\t// total uncompressed chunk data\n\tbinary.BigEndian.PutUint64(dst[consumed:], uncData)\n\tconsumed += uint64Size\n\n\t// magic number\n\tcopy(dst[consumed:], magicNumber)\n\tconsumed += magicNumberSize\n\treturn\n}\n" }, { "path": "go/nbs/test/manifest_clobber.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"flag\"\n\t\"log\"\n\t\"os\"\n\n\t\"golang.org/x/sys/unix\"\n)\n\nfunc main() {\n\tflag.Parse()\n\n\tif flag.NArg() < 3 {\n\t\tlog.Fatalln(\"Not enough arguments\")\n\t}\n\n\tl, err := os.Create(flag.Arg(0))\n\tif err != nil {\n\t\tlog.Fatalln(err)\n\t}\n\tdefer l.Close()\n\t// lock released by closing l.\n\terr = unix.Flock(int(l.Fd()), unix.LOCK_EX|unix.LOCK_NB)\n\tif err == unix.EWOULDBLOCK {\n\t\treturn\n\t}\n\tif err != nil {\n\t\tlog.Fatalln(err)\n\t}\n\n\t// Clobber manifest file at flag.Arg(1) with contents at flag.Arg(2)\n\tm, err := os.Create(flag.Arg(1))\n\tif err != nil {\n\t\tlog.Fatalln(err)\n\t}\n\tdefer m.Close()\n\tif _, err = m.WriteString(flag.Arg(2)); err != nil {\n\t\tlog.Fatalln(err)\n\t}\n}\n" }, { "path": "go/ngql/README.md", "content": "# Noms GraphQL\n\nAn experimental bridge between Noms and [GraphQL](http://graphql.org/)\n\nThis is to be used with https://github.com/attic-labs/graphql which is a fork of https://github.com/graphql-go/graphql to handle Noms values. It disables some validations that do not work for Noms due to Noms being schemaless (or more precisely the schema is a function of the value in the dataset).\n\n*ngql* provides an API to convert Noms types/values to and from GraphQL types/values, as well as some functions that can be used to implement a GraphQL endpoint using https://github.com/attic-labs/graphql.\n\n# Status\n\n * All Noms types are supported except\n * Blob\n * Type\n * Unions with non-`Struct` component types\n * GraphQL does not support unions in input types which limits the input types that can be used.\n\n# Type conversion rules\n\n## Value\n\nAllmost all Noms values can be represented by GraphQL. All Noms values except the primitives (`Bool`, `Number` & `String`) are represented by a GraphQL struct.\n\n## Bool\n\nIs represented by a non nullable `Bool`\n\n## Number\n\nIs represented by a non nullable `Float`\n\n## String\n\nIs represented by a non nullable `String`\n\n## List\n\nNoms lists are expressed as a GraphQL struct with the fields\n\n* `values` - The values in the list.\n* `size` - The number of values in the list.\n\nLists takes a few optional arguments:\n\n* `at` - The index to start at, defaults to `0`.\n* `count` - The number of values to return, defaults to all of the values.\n\n```graphql\ntype FooList {\n size: Float!\n values: [Foo!]!\n}\n```\n\n## Set\n\nNoms sets are expressed as a GraphQL struct with the fields\n\n* `values` - The values in the set.\n* `size` - The number of values in the set.\n\nSets takes a few optional arguments:\n\n* `at` - The index to start at, defaults to `0`.\n* `count` - The number of values to return, defaults to all of the values.\n* `key` - The value to start at.\n* `through` - The value to end at (inclusive).\n* `keys` - When provided only values that matches the keys are included in the result.\n\n```graphql\ntype FooSet {\n size: Float!\n values: [Foo!]!\n}\n```\n\n## Map\n\nNoms maps are expressed as a GraphQL struct with the fields\n\n* `values` - The values in the map.\n* `keys` - The keys in the map.\n* `entries` - The entries in the map. An entry is a struct with `key` and `value` fields.\n* `size` - The number of values in the map.\n\nSets takes a few optional arguments:\n\n* `at` - The index to start at, defaults to `0`\n* `count` - The number of elements to return, defaults to all of the elements.\n* `key` - The value to start at\n* `through` - The value to end at (inclusive)\n* `keys` - When provided only values/keys/entries that matches the keys are included in the result.\n\n```graphql\ntype StringFooMap {\n size: Float!\n elements: [StringFooEntry!]!\n}\n\ntype StringFloatEntry {\n key: String!\n value: Float!\n}\n```\n\n## Struct\n\nNoms structs are expressed as GraphQL structs, with an extra `hash` field.\n\nIf the field in the Noms struct is optional then the GraphQL type for that field is nullable.\n\n## Ref\n\nNoms refs are expressed as a GraphQL struct with a `targetHash` and `targetValue` field.\n\n```graphql\ntype FooRef {\n targetHash: String!\n targetValue: Foo!\n}\n```\n" }, { "path": "go/ngql/query.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage ngql\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"io\"\n\n\t\"github.com/attic-labs/graphql\"\n\t\"github.com/attic-labs/graphql/gqlerrors\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\nconst (\n\tatKey = \"at\"\n\tcountKey = \"count\"\n\telementsKey = \"elements\"\n\tentriesKey = \"entries\"\n\tkeyKey = \"key\"\n\tkeysKey = \"keys\"\n\trootKey = \"root\"\n\trootQueryKey = \"Root\"\n\tscalarValue = \"scalarValue\"\n\tsizeKey = \"size\"\n\ttargetHashKey = \"targetHash\"\n\ttargetValueKey = \"targetValue\"\n\tthroughKey = \"through\"\n\tvalueKey = \"value\"\n\tvaluesKey = \"values\"\n\tvrwKey = \"vrw\"\n)\n\n// NewRootQueryObject creates a \"root\" query object that can be used to\n// traverse the value tree of rootValue.\nfunc NewRootQueryObject(rootValue types.Value, tm *TypeMap) *graphql.Object {\n\ttc := TypeConverter{*tm, DefaultNameFunc}\n\treturn tc.NewRootQueryObject(rootValue)\n}\n\n// NewRootQueryObject creates a \"root\" query object that can be used to\n// traverse the value tree of rootValue.\nfunc (tc *TypeConverter) NewRootQueryObject(rootValue types.Value) *graphql.Object {\n\trootNomsType := types.TypeOf(rootValue)\n\trootType := tc.NomsTypeToGraphQLType(rootNomsType)\n\n\treturn graphql.NewObject(graphql.ObjectConfig{\n\t\tName: rootQueryKey,\n\t\tFields: graphql.Fields{\n\t\t\trootKey: &graphql.Field{\n\t\t\t\tType: rootType,\n\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\treturn MaybeGetScalar(rootValue), nil\n\t\t\t\t},\n\t\t\t},\n\t\t}})\n}\n\n// NewContext creates a new context.Context with the extra data added to it\n// that is required by ngql.\nfunc NewContext(vrw types.ValueReader) context.Context {\n\treturn context.WithValue(context.Background(), vrwKey, vrw)\n}\n\n// Query takes |rootValue|, builds a GraphQL scheme from rootValue.Type() and\n// executes |query| against it, encoding the result to |w|.\nfunc Query(rootValue types.Value, query string, vrw types.ValueReadWriter, w io.Writer) {\n\tschemaConfig := graphql.SchemaConfig{}\n\ttc := NewTypeConverter()\n\tqueryWithSchemaConfig(rootValue, query, schemaConfig, vrw, tc, w)\n}\n\nfunc queryWithSchemaConfig(rootValue types.Value, query string, schemaConfig graphql.SchemaConfig, vrw types.ValueReadWriter, tc *TypeConverter, w io.Writer) {\n\tschemaConfig.Query = tc.NewRootQueryObject(rootValue)\n\tschema, _ := graphql.NewSchema(schemaConfig)\n\tctx := NewContext(vrw)\n\n\tr := graphql.Do(graphql.Params{\n\t\tSchema: schema,\n\t\tRequestString: query,\n\t\tContext: ctx,\n\t})\n\n\terr := json.NewEncoder(w).Encode(r)\n\td.PanicIfError(err)\n}\n\n// Error writes an error as a GraphQL error to a writer.\nfunc Error(err error, w io.Writer) {\n\tr := graphql.Result{\n\t\tErrors: []gqlerrors.FormattedError{\n\t\t\t{Message: err.Error()},\n\t\t},\n\t}\n\n\tjsonErr := json.NewEncoder(w).Encode(r)\n\td.PanicIfError(jsonErr)\n}\n" }, { "path": "go/ngql/query_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage ngql\n\nimport (\n\t\"bytes\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/graphql\"\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/test\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\ntype QueryGraphQLSuite struct {\n\tsuite.Suite\n\tvs *types.ValueStore\n}\n\nfunc TestQueryGraphQL(t *testing.T) {\n\tsuite.Run(t, &QueryGraphQLSuite{})\n}\n\nfunc newTestValueStore() *types.ValueStore {\n\tstorage := &chunks.MemoryStorage{}\n\treturn types.NewValueStore(storage.NewView())\n}\n\nfunc (suite *QueryGraphQLSuite) SetupTest() {\n\tsuite.vs = newTestValueStore()\n}\n\nfunc (suite *QueryGraphQLSuite) assertQueryResult(v types.Value, q, expect string) {\n\tbuf := &bytes.Buffer{}\n\tQuery(v, q, suite.vs, buf)\n\tsuite.JSONEq(test.RemoveHashes(expect), test.RemoveHashes(buf.String()))\n}\n\nfunc (suite *QueryGraphQLSuite) TestScalars() {\n\tsuite.assertQueryResult(types.String(\"aaa\"), \"{root}\", `{\"data\":{\"root\":\"aaa\"}}`)\n\tsuite.assertQueryResult(types.String(\"\"), \"{root}\", `{\"data\":{\"root\":\"\"}}`)\n\n\tsuite.assertQueryResult(types.Number(0), \"{root}\", `{\"data\":{\"root\":0}}`)\n\tsuite.assertQueryResult(types.Number(1), \"{root}\", `{\"data\":{\"root\":1}}`)\n\tsuite.assertQueryResult(types.Number(-1), \"{root}\", `{\"data\":{\"root\":-1}}`)\n\tsuite.assertQueryResult(types.Number(1<<31), \"{root}\", `{\"data\":{\"root\":2.147483648e+09}}`)\n\tsuite.assertQueryResult(types.Number(-(1 << 31)), \"{root}\", `{\"data\":{\"root\":-2.147483648e+09}}`)\n\tsuite.assertQueryResult(types.Number(0.001), \"{root}\", `{\"data\":{\"root\":0.001}}`)\n\tsuite.assertQueryResult(types.Number(0.00000001), \"{root}\", `{\"data\":{\"root\":1e-08}}`)\n\n\tsuite.assertQueryResult(types.Bool(false), \"{root}\", `{\"data\":{\"root\":false}}`)\n\tsuite.assertQueryResult(types.Bool(true), \"{root}\", `{\"data\":{\"root\":true}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestStructBasic() {\n\ts1 := types.NewStruct(\"Foo\", types.StructData{\n\t\t\"a\": types.String(\"aaa\"),\n\t\t\"b\": types.Bool(true),\n\t\t\"c\": types.Number(0.1),\n\t})\n\n\tsuite.assertQueryResult(s1, \"{root{a}}\", `{\"data\":{\"root\":{\"a\":\"aaa\"}}}`)\n\tsuite.assertQueryResult(s1, \"{root{a b}}\", `{\"data\":{\"root\":{\"a\":\"aaa\",\"b\":true}}}`)\n\tsuite.assertQueryResult(s1, \"{root{a b c}}\", `{\"data\":{\"root\":{\"a\":\"aaa\",\"b\":true,\"c\":0.1}}}`)\n\tsuite.assertQueryResult(s1, \"{root{a c}}\", `{\"data\":{\"root\":{\"a\":\"aaa\",\"c\":0.1}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestEmptyStruct() {\n\ts1 := types.NewStruct(\"\", types.StructData{})\n\n\tsuite.assertQueryResult(s1, \"{root{hash}}\", `{\"data\":{\"root\":{\"hash\":\"0123456789abcdefghijklmnopqrstuv\"}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestEmbeddedStruct() {\n\ts1 := types.NewStruct(\"Foo\", types.StructData{\n\t\t\"a\": types.String(\"aaa\"),\n\t\t\"b\": types.NewStruct(\"Bar\", types.StructData{\n\t\t\t\"c\": types.Bool(true),\n\t\t\t\"d\": types.Number(0.1),\n\t\t}),\n\t})\n\n\tsuite.assertQueryResult(s1, \"{root{a}}\", `{\"data\":{\"root\":{\"a\":\"aaa\"}}}`)\n\tsuite.assertQueryResult(s1, \"{root{a b {c}}}\", `{\"data\":{\"root\":{\"a\":\"aaa\",\"b\":{\"c\":true}}}}`)\n\tsuite.assertQueryResult(s1, \"{root{a b {c d}}}\", `{\"data\":{\"root\":{\"a\":\"aaa\",\"b\":{\"c\":true,\"d\":0.1}}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestListBasic() {\n\tfor _, valuesKey := range []string{\"elements\", \"values\"} {\n\t\tlist := types.NewList(suite.vs)\n\t\tsuite.assertQueryResult(list, \"{root{size}}\", `{\"data\":{\"root\":{\"size\":0}}}`)\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"}}\", `{\"data\":{\"root\":{}}}`)\n\n\t\tlist = types.NewList(suite.vs, types.String(\"foo\"), types.String(\"bar\"), types.String(\"baz\"))\n\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"foo\",\"bar\",\"baz\"]}}}`)\n\t\tsuite.assertQueryResult(list, \"{root{size}}\", `{\"data\":{\"root\":{\"size\":3}}}`)\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"(at:1,count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"bar\",\"baz\"]}}}`)\n\n\t\tlist = types.NewList(suite.vs, types.Bool(true), types.Bool(false), types.Bool(false))\n\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[true,false,false]}}}`)\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"(at:1,count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[false,false]}}}`)\n\n\t\tlist = types.NewList(suite.vs, types.Number(1), types.Number(1.1), types.Number(-100))\n\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[1,1.1,-100]}}}`)\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"(at:1,count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[1.1,-100]}}}`)\n\n\t\tlist = types.NewList(suite.vs, types.String(\"a\"), types.String(\"b\"), types.String(\"c\"))\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"(at:4)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"(count:0)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"(count:10)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"b\",\"c\"]}}}`)\n\t\tsuite.assertQueryResult(list, \"{root{\"+valuesKey+\"(at:-1)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"b\",\"c\"]}}}`)\n\t}\n}\n\nfunc (suite *QueryGraphQLSuite) TestListOfStruct() {\n\tlist := types.NewList(suite.vs,\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(28),\n\t\t\t\"b\": types.String(\"foo\"),\n\t\t}),\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(-20.102),\n\t\t\t\"b\": types.String(\"bar\"),\n\t\t}),\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(5),\n\t\t\t\"b\": types.String(\"baz\"),\n\t\t}),\n\t)\n\n\tsuite.assertQueryResult(list, \"{root{elements{a b}}}\", `{\"data\":{\"root\":{\"elements\":[{\"a\":28,\"b\":\"foo\"},{\"a\":-20.102,\"b\":\"bar\"},{\"a\":5,\"b\":\"baz\"}]}}}`)\n\n\tsuite.assertQueryResult(list, \"{root{elements{a}}}\", `{\"data\":{\"root\":{\"elements\":[{\"a\":28},{\"a\":-20.102},{\"a\":5}]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestListOfStructWithOptionalFields() {\n\tlist := types.NewList(suite.vs,\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(1),\n\t\t}),\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(2),\n\t\t\t\"b\": types.String(\"bar\"),\n\t\t}),\n\t)\n\n\tsuite.assertQueryResult(list, \"{root{elements{a b}}}\", `{\n \"data\": {\n \"root\": {\n \"elements\": [\n {\"a\": 1, \"b\": null},\n {\"a\": 2, \"b\": \"bar\"}\n ]\n }\n }\n }`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestSetBasic() {\n\tfor _, valuesKey := range []string{\"elements\", \"values\"} {\n\t\tset := types.NewSet(suite.vs)\n\t\tsuite.assertQueryResult(set, \"{root{size}}\", `{\"data\":{\"root\":{\"size\":0}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"}}\", `{\"data\":{\"root\":{}}}`)\n\n\t\tset = types.NewSet(suite.vs, types.String(\"foo\"), types.String(\"bar\"), types.String(\"baz\"))\n\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"bar\",\"baz\",\"foo\"]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{size}}\", `{\"data\":{\"root\":{\"size\":3}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"bar\",\"baz\"]}}}`)\n\n\t\tset = types.NewSet(suite.vs, types.Bool(true), types.Bool(false))\n\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[false,true]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(count:1)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[false]}}}`)\n\n\t\tset = types.NewSet(suite.vs, types.Number(1), types.Number(1.1), types.Number(-100))\n\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[-100,1,1.1]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[-100,1]}}}`)\n\n\t\tset = types.NewSet(suite.vs, types.String(\"a\"), types.String(\"b\"), types.String(\"c\"), types.String(\"d\"))\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(count:0)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"b\"]}}}`)\n\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(at:0,count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"b\"]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(at:-1,count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"b\"]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(at:1,count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"b\",\"c\"]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(at:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\",\"d\"]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(at:2,count:1)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\"]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(at:2,count:0)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(set, \"{root{\"+valuesKey+\"(at:2,count:10)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\",\"d\"]}}}`)\n\t}\n}\n\nfunc (suite *QueryGraphQLSuite) TestSetOfStruct() {\n\tset := types.NewSet(suite.vs,\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(28),\n\t\t\t\"b\": types.String(\"foo\"),\n\t\t}),\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(-20.102),\n\t\t\t\"b\": types.String(\"bar\"),\n\t\t}),\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(5),\n\t\t\t\"b\": types.String(\"baz\"),\n\t\t}),\n\t)\n\n\tsuite.assertQueryResult(set, \"{root{values{a b}}}\",\n\t\t`{\"data\":{\"root\":{\"values\":[{\"a\":28,\"b\":\"foo\"},{\"a\":5,\"b\":\"baz\"},{\"a\":-20.102,\"b\":\"bar\"}]}}}`)\n\tsuite.assertQueryResult(set, \"{root{values{a}}}\", `{\"data\":{\"root\":{\"values\":[{\"a\":28},{\"a\":5},{\"a\":-20.102}]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestMapBasic() {\n\tfor _, entriesKey := range []string{\"elements\", \"entries\"} {\n\n\t\tm := types.NewMap(suite.vs)\n\t\tsuite.assertQueryResult(m, \"{root{size}}\", `{\"data\":{\"root\":{\"size\":0}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"}}\", `{\"data\":{\"root\":{}}}`)\n\n\t\tm = types.NewMap(suite.vs,\n\t\t\ttypes.String(\"a\"), types.Number(1),\n\t\t\ttypes.String(\"b\"), types.Number(2),\n\t\t\ttypes.String(\"c\"), types.Number(3),\n\t\t\ttypes.String(\"d\"), types.Number(4),\n\t\t)\n\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"{key value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\",\"value\":1},{\"key\":\"b\",\"value\":2},{\"key\":\"c\",\"value\":3},{\"key\":\"d\",\"value\":4}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{size}}\", `{\"data\":{\"root\":{\"size\":4}}}`)\n\t}\n}\n\nfunc (suite *QueryGraphQLSuite) TestMapOfStruct() {\n\tm := types.NewMap(suite.vs,\n\t\ttypes.String(\"foo\"), types.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(28),\n\t\t\t\"b\": types.String(\"foo\"),\n\t\t}),\n\t\ttypes.String(\"bar\"), types.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(-20.102),\n\t\t\t\"b\": types.String(\"bar\"),\n\t\t}),\n\t\ttypes.String(\"baz\"), types.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(5),\n\t\t\t\"b\": types.String(\"baz\"),\n\t\t}),\n\t)\n\n\tsuite.assertQueryResult(m, \"{root{entries{key value{a}}}}\", `{\"data\":{\"root\":{\"entries\":[{\"key\":\"bar\",\"value\":{\"a\":-20.102}},{\"key\":\"baz\",\"value\":{\"a\":5}},{\"key\":\"foo\",\"value\":{\"a\":28}}]}}}`)\n\tsuite.assertQueryResult(m, \"{root{entries(count:1){value{a b}}}}\", `{\"data\":{\"root\":{\"entries\":[{\"value\":{\"a\":-20.102,\"b\":\"bar\"}}]}}}`)\n\tsuite.assertQueryResult(m, \"{root{entries(count:3){key}}}\", `{\"data\":{\"root\":{\"entries\":[{\"key\":\"bar\"},{\"key\":\"baz\"},{\"key\":\"foo\"}]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestRef() {\n\tr := suite.vs.WriteValue(types.Number(100))\n\n\tsuite.assertQueryResult(r, \"{root{targetValue}}\", `{\"data\":{\"root\":{\"targetValue\":100}}}`)\n\tsuite.assertQueryResult(r, \"{root{targetHash}}\", `{\"data\":{\"root\":{\"targetHash\":\"0123456789abcdefghijklmnopqrstuv\"}}}`)\n\tsuite.assertQueryResult(r, \"{root{targetValue targetHash}}\", `{\"data\":{\"root\":{\"targetHash\":\"0123456789abcdefghijklmnopqrstuv\",\"targetValue\":100}}}`)\n\n\tr = suite.vs.WriteValue(types.NewStruct(\"Foo\", types.StructData{\n\t\t\"a\": types.Number(28),\n\t\t\"b\": types.String(\"foo\"),\n\t}))\n\n\tsuite.assertQueryResult(r, \"{root{targetValue{a}}}\", `{\"data\":{\"root\":{\"targetValue\":{\"a\":28}}}}`)\n\tsuite.assertQueryResult(r, \"{root{targetValue{a b}}}\", `{\"data\":{\"root\":{\"targetValue\":{\"a\":28,\"b\":\"foo\"}}}}`)\n\n\tr = suite.vs.WriteValue(types.NewList(suite.vs, types.String(\"foo\"), types.String(\"bar\"), types.String(\"baz\")))\n\n\tsuite.assertQueryResult(r, \"{root{targetValue{values}}}\", `{\"data\":{\"root\":{\"targetValue\":{\"values\":[\"foo\",\"bar\",\"baz\"]}}}}`)\n\tsuite.assertQueryResult(r, \"{root{targetValue{values(at:1,count:2)}}}\", `{\"data\":{\"root\":{\"targetValue\":{\"values\":[\"bar\",\"baz\"]}}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestListOfUnionOfStructs() {\n\tlist := types.NewList(suite.vs,\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(28),\n\t\t\t\"b\": types.String(\"baz\"),\n\t\t}),\n\t\ttypes.NewStruct(\"Bar\", types.StructData{\n\t\t\t\"b\": types.String(\"bar\"),\n\t\t}),\n\t\ttypes.NewStruct(\"Baz\", types.StructData{\n\t\t\t\"c\": types.Bool(true),\n\t\t}),\n\t)\n\n\tsuite.assertQueryResult(list,\n\t\tfmt.Sprintf(\"{root{values{... on %s{a b} ... on %s{b} ... on %s{c}}}}\",\n\t\t\tGetTypeName(types.TypeOf(list.Get(0))),\n\t\t\tGetTypeName(types.TypeOf(list.Get(1))),\n\t\t\tGetTypeName(types.TypeOf(list.Get(2)))),\n\t\t`{\"data\":{\"root\":{\"values\":[{\"a\":28,\"b\":\"baz\"},{\"b\":\"bar\"},{\"c\":true}]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestListOfUnionOfStructsConflictingFieldTypes() {\n\tlist := types.NewList(suite.vs,\n\t\ttypes.NewStruct(\"Foo\", types.StructData{\n\t\t\t\"a\": types.Number(28),\n\t\t}),\n\t\ttypes.NewStruct(\"Bar\", types.StructData{\n\t\t\t\"a\": types.String(\"bar\"),\n\t\t}),\n\t\ttypes.NewStruct(\"Baz\", types.StructData{\n\t\t\t\"a\": types.Bool(true),\n\t\t}),\n\t)\n\n\tsuite.assertQueryResult(list,\n\t\tfmt.Sprintf(\"{root{values{... on %s{a} ... on %s{b: a} ... on %s{c: a}}}}\",\n\t\t\tGetTypeName(types.TypeOf(list.Get(0))),\n\t\t\tGetTypeName(types.TypeOf(list.Get(1))),\n\t\t\tGetTypeName(types.TypeOf(list.Get(2)))),\n\t\t`{\"data\":{\"root\":{\"values\":[{\"a\":28},{\"b\":\"bar\"},{\"c\":true}]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestListOfUnionOfScalars() {\n\tlist := types.NewList(suite.vs,\n\t\ttypes.Number(28),\n\t\ttypes.String(\"bar\"),\n\t\ttypes.Bool(true),\n\t)\n\n\tsuite.assertQueryResult(list, \"{root{values{... on BooleanValue{b: scalarValue} ... on StringValue{s: scalarValue} ... on NumberValue{n: scalarValue}}}}\", `{\"data\":{\"root\":{\"values\":[{\"n\":28},{\"s\":\"bar\"},{\"b\":true}]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestCyclicStructs() {\n\t// struct A {\n\t// a: \"aaa\"\n\t// b: Set(struct A {\n\t// \t a: \"bbb\"\n\t// \t b: Set()\n\t// })\n\t// }\n\n\ts1 := types.NewStruct(\"A\", types.StructData{\n\t\t\"a\": types.String(\"aaa\"),\n\t\t\"b\": types.NewSet(suite.vs,\n\t\t\ttypes.NewStruct(\"A\", types.StructData{\n\t\t\t\t\"a\": types.String(\"bbb\"),\n\t\t\t\t\"b\": types.NewSet(suite.vs),\n\t\t\t})),\n\t})\n\n\tsuite.assertQueryResult(s1, \"{root{a b{values{a}}}}\", `{\"data\":{\"root\":{\"a\":\"aaa\",\"b\":{\"values\":[{\"a\":\"bbb\"}]}}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestCyclicStructsWithUnion() {\n\t// struct A {\n\t// a: \"aaa\"\n\t// b: Struct A {\n\t// \t a: \"bbb\"\n\t// \t b: 42\n\t// })\n\t// }\n\n\t// struct A {\n\t// a: String,\n\t// b: Number | Cycle,\n\t// }\n\n\ts1 := types.NewStruct(\"A\", types.StructData{\n\t\t\"a\": types.String(\"aaa\"),\n\t\t\"b\": types.NewStruct(\"A\", types.StructData{\n\t\t\t\"a\": types.String(\"bbb\"),\n\t\t\t\"b\": types.Number(42),\n\t\t}),\n\t})\n\n\tsuite.assertQueryResult(s1,\n\t\t`{\n root{\n a\n b {\n a\n b {\n scalarValue\n }\n }\n }\n }\n `,\n\t\t`{\n \"data\": {\n \"root\": {\n \"a\": \"aaa\",\n \"b\": {\n \"a\": \"bbb\",\n \"b\": {\n \"scalarValue\": 42\n }\n }\n }\n }\n }`)\n\n\tsuite.assertQueryResult(s1,\n\t\tfmt.Sprintf(`{\n\t root{\n\t a\n\t b {\n\t ... on %s {\n\t a\n\t }\n\t }\n\t }\n\t }`, GetTypeName(types.TypeOf(s1))),\n\t\t`{\n\t \"data\": {\n\t \"root\": {\n\t \"a\": \"aaa\",\n\t \"b\": {\n\t \"a\": \"bbb\"\n\t }\n\t }\n\t }\n\t }`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestNestedCollection() {\n\tlist := types.NewList(suite.vs,\n\t\ttypes.NewSet(suite.vs,\n\t\t\ttypes.NewMap(suite.vs, types.Number(10), types.String(\"foo\")),\n\t\t\ttypes.NewMap(suite.vs, types.Number(20), types.String(\"bar\")),\n\t\t),\n\t\ttypes.NewSet(suite.vs,\n\t\t\ttypes.NewMap(suite.vs, types.Number(30), types.String(\"baz\")),\n\t\t\ttypes.NewMap(suite.vs, types.Number(40), types.String(\"bat\")),\n\t\t),\n\t)\n\n\tsuite.assertQueryResult(list, \"{root{size}}\", `{\"data\":{\"root\":{\"size\":2}}}`)\n\tsuite.assertQueryResult(list, \"{root{values(count:1){size}}}\", `{\"data\":{\"root\":{\"values\":[{\"size\":2}]}}}`)\n\tsuite.assertQueryResult(list, \"{root{values(at:1,count:1){values(count:1){entries{key value}}}}}\",\n\t\t`{\"data\":{\"root\":{\"values\":[{\"values\":[{\"entries\":[{\"key\":40,\"value\":\"bat\"}]}]}]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestLoFi() {\n\tb := types.NewBlob(suite.vs, bytes.NewBufferString(\"I am a blob\"))\n\n\tsuite.assertQueryResult(b, \"{root}\", `{\"data\":{\"root\":\"0123456789abcdefghijklmnopqrstuv\"}}`)\n\n\tt := types.StringType\n\tsuite.assertQueryResult(t, \"{root}\", `{\"data\":{\"root\":\"0123456789abcdefghijklmnopqrstuv\"}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestError() {\n\tbuff := &bytes.Buffer{}\n\tError(errors.New(\"Some error string\"), buff)\n\tsuite.Equal(buff.String(), `{\"data\":null,\"errors\":[{\"message\":\"Some error string\",\"locations\":null}]}\n`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestMapArgs() {\n\tfor _, entriesKey := range []string{\"elements\", \"entries\"} {\n\n\t\tm := types.NewMap(suite.vs,\n\t\t\ttypes.String(\"a\"), types.Number(1),\n\t\t\ttypes.String(\"c\"), types.Number(2),\n\t\t\ttypes.String(\"e\"), types.Number(3),\n\t\t\ttypes.String(\"g\"), types.Number(4),\n\t\t)\n\n\t\t// count\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(count:0){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(count:2){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":1},{\"value\":2}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(count:3){key}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\"},{\"key\":\"c\"},{\"key\":\"e\"}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(count: -1){key}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(count:5){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":1},{\"value\":2},{\"value\":3},{\"value\":4}]}}}`)\n\n\t\t// at\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:0){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":1},{\"value\":2},{\"value\":3},{\"value\":4}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:-1){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":1},{\"value\":2},{\"value\":3},{\"value\":4}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:2){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":3},{\"value\":4}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:5){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\n\t\t// at & count\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:0,count:2){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":1},{\"value\":2}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:-1,count:2){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":1},{\"value\":2}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:1,count:2){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":2},{\"value\":3}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:2,count:1){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":3}]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:2,count:0){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(m, \"{root{\"+entriesKey+\"(at:2,count:10){value}}}\", `{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":3},{\"value\":4}]}}}`)\n\n\t\t// key\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"e\"){key value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"e\",\"value\":3}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"g\"){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":4}]}}}`)\n\t\t// \"f\", no count/through so asking for exact match\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"f\"){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\t\t// \"x\" is larger than end\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"x\"){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\n\t\t// key & at\n\t\t// at is ignored when key is present\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"e\",at:2){key value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"e\",\"value\":3}]}}}`)\n\n\t\t// key & count\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"c\", count: 2){key value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"c\",\"value\":2},{\"key\":\"e\",\"value\":3}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"c\", count: 0){key value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"c\", count: -1){key value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"e\", count: 5){key value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"e\",\"value\":3},{\"key\":\"g\",\"value\":4}]}}}`)\n\n\t\t// through\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(through:\"c\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\"},{\"key\":\"c\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(through:\"b\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(through:\"0\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\n\t\t// key & through\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"c\", through:\"c\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"c\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"c\",through:\"e\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"c\"},{\"key\":\"e\"}]}}}`)\n\n\t\t// through & count\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(through:\"c\",count:1){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(through:\"b\",count:0){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(through:\"0\",count:10){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\n\t\t// at & through\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(at:0,through:\"a\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(at:1,through:\"e\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"c\"},{\"key\":\"e\"}]}}}`)\n\n\t\t// at & count & through\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(at:0,count:2,through:\"a\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(at:0,count:2,through:\"e\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\"},{\"key\":\"c\"}]}}}`)\n\n\t\t// key & count & through\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"c\",count:2,through:\"c\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"c\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(key:\"c\",count:2,through:\"g\"){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"c\"},{\"key\":\"e\"}]}}}`)\n\t}\n}\n\nfunc (suite *QueryGraphQLSuite) TestMapKeysArg() {\n\tfor _, entriesKey := range []string{\"elements\", \"entries\"} {\n\t\tm := types.NewMap(suite.vs,\n\t\t\ttypes.String(\"a\"), types.Number(1),\n\t\t\ttypes.String(\"c\"), types.Number(2),\n\t\t\ttypes.String(\"e\"), types.Number(3),\n\t\t\ttypes.String(\"g\"), types.Number(4),\n\t\t)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[\"c\",\"a\"]){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":2},{\"value\":1}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[]){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[]}}}`)\n\n\t\tm = types.NewMap(suite.vs,\n\t\t\ttypes.Number(1), types.String(\"a\"),\n\t\t\ttypes.Number(2), types.String(\"c\"),\n\t\t\ttypes.Number(3), types.String(\"e\"),\n\t\t\ttypes.Number(4), types.String(\"g\"),\n\t\t)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[4,1]){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":\"g\"},{\"value\":\"a\"}]}}}`)\n\n\t\t// Ignore other args\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[4,1],key:2){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":\"g\"},{\"value\":\"a\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[4,1],count:0){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":\"g\"},{\"value\":\"a\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[4,1],at:4){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":\"g\"},{\"value\":\"a\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[4,1],through:1){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":\"g\"},{\"value\":\"a\"}]}}}`)\n\t}\n}\n\nfunc (suite *QueryGraphQLSuite) TestSetArgs() {\n\tfor _, valuesKey := range []string{\"elements\", \"values\"} {\n\t\ts := types.NewSet(suite.vs,\n\t\t\ttypes.String(\"a\"),\n\t\t\ttypes.String(\"c\"),\n\t\t\ttypes.String(\"e\"),\n\t\t\ttypes.String(\"g\"),\n\t\t)\n\n\t\t// count\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(count:0)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"c\"]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(count:3)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"c\",\"e\"]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(count: -1)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(count:5)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"c\",\"e\",\"g\"]}}}`)\n\n\t\t// at\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:0)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"c\",\"e\",\"g\"]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:-1)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"c\",\"e\",\"g\"]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"e\",\"g\"]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:5)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\n\t\t// at & count\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:0,count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"c\"]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:-1,count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"c\"]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:1,count:2)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\",\"e\"]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:2,count:1)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"e\"]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:2,count:0)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(s, \"{root{\"+valuesKey+\"(at:2,count:10)}}\", `{\"data\":{\"root\":{\"`+valuesKey+`\":[\"e\",\"g\"]}}}`)\n\n\t\t// key\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"e\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"e\"]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"g\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"g\"]}}}`)\n\t\t// \"f\", no count/through so asking for exact match\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"f\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\t// \"x\" is larger than end\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"x\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\t// exact match\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"0\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\n\t\t// key & at\n\t\t// at is ignored when key is present\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"e\",at:2)}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"e\"]}}}`)\n\n\t\t// key & count\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"c\", count: 2)}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\",\"e\"]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"c\", count: 0)}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"c\", count: -1)}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"e\", count: 5)}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"e\",\"g\"]}}}`)\n\n\t\t// through\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(through:\"c\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"c\"]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(through:\"b\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\"]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(through:\"0\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\n\t\t// key & through\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"c\", through:\"c\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\"]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"c\",through:\"e\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\",\"e\"]}}}`)\n\n\t\t// through & count\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(through:\"c\",count:1)}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\"]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(through:\"b\",count:0)}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(through:\"0\",count:10)}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[]}}}`)\n\n\t\t// at & through\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(at:0,through:\"a\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\"]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(at:1,through:\"e\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\",\"e\"]}}}`)\n\n\t\t// at & count & through\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(at:0,count:2,through:\"a\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\"]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(at:0,count:2,through:\"e\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"a\",\"c\"]}}}`)\n\n\t\t// key & count & through\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"c\",count:2,through:\"c\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\"]}}}`)\n\t\tsuite.assertQueryResult(s, `{root{`+valuesKey+`(key:\"c\",count:2,through:\"g\")}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+valuesKey+`\":[\"c\",\"e\"]}}}`)\n\t}\n}\n\nfunc (suite *QueryGraphQLSuite) TestMapValues() {\n\tm := types.NewMap(suite.vs,\n\t\ttypes.String(\"a\"), types.Number(1),\n\t\ttypes.String(\"c\"), types.Number(2),\n\t\ttypes.String(\"e\"), types.Number(3),\n\t\ttypes.String(\"g\"), types.Number(4),\n\t)\n\n\tsuite.assertQueryResult(m, \"{root{values}}\", `{\"data\":{\"root\":{\"values\":[1,2,3,4]}}}`)\n\n\t// count\n\tsuite.assertQueryResult(m, \"{root{values(count:0)}}\", `{\"data\":{\"root\":{\"values\":[]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(count:2)}}\", `{\"data\":{\"root\":{\"values\":[1,2]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(count:3)}}\", `{\"data\":{\"root\":{\"values\":[1,2,3]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(count: -1)}}\", `{\"data\":{\"root\":{\"values\":[]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(count:5)}}\", `{\"data\":{\"root\":{\"values\":[1,2,3,4]}}}`)\n\n\t// at\n\tsuite.assertQueryResult(m, \"{root{values(at:0)}}\", `{\"data\":{\"root\":{\"values\":[1,2,3,4]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(at:-1)}}\", `{\"data\":{\"root\":{\"values\":[1,2,3,4]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(at:2)}}\", `{\"data\":{\"root\":{\"values\":[3,4]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(at:5)}}\", `{\"data\":{\"root\":{\"values\":[]}}}`)\n\n\t// at & count\n\tsuite.assertQueryResult(m, \"{root{values(at:0,count:2)}}\", `{\"data\":{\"root\":{\"values\":[1,2]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(at:-1,count:2)}}\", `{\"data\":{\"root\":{\"values\":[1,2]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(at:1,count:2)}}\", `{\"data\":{\"root\":{\"values\":[2,3]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(at:2,count:1)}}\", `{\"data\":{\"root\":{\"values\":[3]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(at:2,count:0)}}\", `{\"data\":{\"root\":{\"values\":[]}}}`)\n\tsuite.assertQueryResult(m, \"{root{values(at:2,count:10)}}\", `{\"data\":{\"root\":{\"values\":[3,4]}}}`)\n\n\t// key\n\tsuite.assertQueryResult(m, `{root{values(key:\"e\")}}`, `{\"data\":{\"root\":{\"values\":[3]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(key:\"g\")}}`, `{\"data\":{\"root\":{\"values\":[4]}}}`)\n\t// \"f\", no count/through so asking for exact match\n\tsuite.assertQueryResult(m, `{root{values(key:\"f\")}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\t// \"x\" is larger than end\n\tsuite.assertQueryResult(m, `{root{values(key:\"x\")}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\n\t// key & at\n\t// at is ignored when key is present\n\tsuite.assertQueryResult(m, `{root{values(key:\"e\",at:2)}}`, `{\"data\":{\"root\":{\"values\":[3]}}}`)\n\n\t// key & count\n\tsuite.assertQueryResult(m, `{root{values(key:\"c\",count:2)}}`, `{\"data\":{\"root\":{\"values\":[2,3]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(key:\"c\",count:0)}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(key:\"c\",count:-1)}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(key:\"e\",count:5)}}`, `{\"data\":{\"root\":{\"values\":[3,4]}}}`)\n\n\t// through\n\tsuite.assertQueryResult(m, `{root{values(through:\"c\")}}`, `{\"data\":{\"root\":{\"values\":[1,2]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(through:\"b\")}}`, `{\"data\":{\"root\":{\"values\":[1]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(through:\"0\")}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\n\t// key & through\n\tsuite.assertQueryResult(m, `{root{values(key:\"c\", through:\"c\")}}`, `{\"data\":{\"root\":{\"values\":[2]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(key:\"c\",through:\"e\")}}`, `{\"data\":{\"root\":{\"values\":[2,3]}}}`)\n\n\t// through & count\n\tsuite.assertQueryResult(m, `{root{values(through:\"c\",count:1)}}`, `{\"data\":{\"root\":{\"values\":[1]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(through:\"b\",count:0)}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(through:\"0\",count:10)}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\n\t// at & through\n\tsuite.assertQueryResult(m, `{root{values(at:0,through:\"a\")}}`, `{\"data\":{\"root\":{\"values\":[1]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(at:1,through:\"e\")}}`, `{\"data\":{\"root\":{\"values\":[2,3]}}}`)\n\n\t// at & count & through\n\tsuite.assertQueryResult(m, `{root{values(at:0,count:2,through:\"a\")}}`, `{\"data\":{\"root\":{\"values\":[1]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(at:0,count:2,through:\"e\")}}`, `{\"data\":{\"root\":{\"values\":[1,2]}}}`)\n\n\t// key & count & through\n\tsuite.assertQueryResult(m, `{root{values(key:\"c\",count:2,through:\"c\")}}`,\n\t\t`{\"data\":{\"root\":{\"values\":[2]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(key:\"c\",count:2,through:\"g\")}}`,\n\t\t`{\"data\":{\"root\":{\"values\":[2,3]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestMapKeys() {\n\tm := types.NewMap(suite.vs,\n\t\ttypes.String(\"a\"), types.Number(1),\n\t\ttypes.String(\"c\"), types.Number(2),\n\t\ttypes.String(\"e\"), types.Number(3),\n\t\ttypes.String(\"g\"), types.Number(4),\n\t)\n\n\tsuite.assertQueryResult(m, \"{root{keys}}\", `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\",\"e\",\"g\"]}}}`)\n\n\t// count\n\tsuite.assertQueryResult(m, \"{root{keys(count:0)}}\", `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(count:2)}}\", `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\"]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(count:3)}}\", `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\",\"e\"]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(count: -1)}}\", `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(count:5)}}\", `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\",\"e\",\"g\"]}}}`)\n\n\t// at\n\tsuite.assertQueryResult(m, \"{root{keys(at:0)}}\", `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\",\"e\",\"g\"]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(at:-1)}}\", `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\",\"e\",\"g\"]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(at:2)}}\", `{\"data\":{\"root\":{\"keys\":[\"e\",\"g\"]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(at:5)}}\", `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\n\t// at & count\n\tsuite.assertQueryResult(m, \"{root{keys(at:0,count:2)}}\", `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\"]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(at:-1,count:2)}}\", `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\"]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(at:1,count:2)}}\", `{\"data\":{\"root\":{\"keys\":[\"c\",\"e\"]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(at:2,count:1)}}\", `{\"data\":{\"root\":{\"keys\":[\"e\"]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(at:2,count:0)}}\", `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\tsuite.assertQueryResult(m, \"{root{keys(at:2,count:10)}}\", `{\"data\":{\"root\":{\"keys\":[\"e\",\"g\"]}}}`)\n\n\t// key\n\tsuite.assertQueryResult(m, `{root{keys(key:\"e\")}}`, `{\"data\":{\"root\":{\"keys\":[\"e\"]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(key:\"g\")}}`, `{\"data\":{\"root\":{\"keys\":[\"g\"]}}}`)\n\t// \"f\", no count/through so asking for exact match\n\tsuite.assertQueryResult(m, `{root{keys(key:\"f\")}}`, `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\t// \"x\" is larger than end\n\tsuite.assertQueryResult(m, `{root{keys(key:\"x\")}}`, `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\n\t// key & at\n\t// at is ignored when key is present\n\tsuite.assertQueryResult(m, `{root{keys(key:\"e\",at:2)}}`, `{\"data\":{\"root\":{\"keys\":[\"e\"]}}}`)\n\n\t// key & count\n\tsuite.assertQueryResult(m, `{root{keys(key:\"c\",count:2)}}`, `{\"data\":{\"root\":{\"keys\":[\"c\",\"e\"]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(key:\"c\",count:0)}}`, `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(key:\"c\",count:-1)}}`, `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(key:\"e\",count:5)}}`, `{\"data\":{\"root\":{\"keys\":[\"e\",\"g\"]}}}`)\n\n\t// through\n\tsuite.assertQueryResult(m, `{root{keys(through:\"c\")}}`, `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\"]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(through:\"b\")}}`, `{\"data\":{\"root\":{\"keys\":[\"a\"]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(through:\"0\")}}`, `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\n\t// key & through\n\tsuite.assertQueryResult(m, `{root{keys(key:\"c\", through:\"c\")}}`, `{\"data\":{\"root\":{\"keys\":[\"c\"]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(key:\"c\",through:\"e\")}}`, `{\"data\":{\"root\":{\"keys\":[\"c\",\"e\"]}}}`)\n\n\t// through & count\n\tsuite.assertQueryResult(m, `{root{keys(through:\"c\",count:1)}}`, `{\"data\":{\"root\":{\"keys\":[\"a\"]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(through:\"b\",count:0)}}`, `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(through:\"0\",count:10)}}`, `{\"data\":{\"root\":{\"keys\":[]}}}`)\n\n\t// at & through\n\tsuite.assertQueryResult(m, `{root{keys(at:0,through:\"a\")}}`, `{\"data\":{\"root\":{\"keys\":[\"a\"]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(at:1,through:\"e\")}}`, `{\"data\":{\"root\":{\"keys\":[\"c\",\"e\"]}}}`)\n\n\t// at & count & through\n\tsuite.assertQueryResult(m, `{root{keys(at:0,count:2,through:\"a\")}}`, `{\"data\":{\"root\":{\"keys\":[\"a\"]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(at:0,count:2,through:\"e\")}}`, `{\"data\":{\"root\":{\"keys\":[\"a\",\"c\"]}}}`)\n\n\t// key & count & through\n\tsuite.assertQueryResult(m, `{root{keys(key:\"c\",count:2,through:\"c\")}}`,\n\t\t`{\"data\":{\"root\":{\"keys\":[\"c\"]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(key:\"c\",count:2,through:\"g\")}}`,\n\t\t`{\"data\":{\"root\":{\"keys\":[\"c\",\"e\"]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestMapNullable() {\n\t// When selecting the result based on keys the values may be null.\n\tm := types.NewMap(suite.vs,\n\t\ttypes.String(\"a\"), types.Number(1),\n\t\ttypes.String(\"c\"), types.Number(2),\n\t)\n\n\tfor _, entriesKey := range []string{\"elements\", \"entries\"} {\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[\"a\",\"b\",\"c\"]){value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"value\":1},{\"value\":null},{\"value\":2}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[\"a\",\"b\",\"c\"]){key}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\"},{\"key\":\"b\"},{\"key\":\"c\"}]}}}`)\n\t\tsuite.assertQueryResult(m, `{root{`+entriesKey+`(keys:[\"a\",\"b\",\"c\"]){key value}}}`,\n\t\t\t`{\"data\":{\"root\":{\"`+entriesKey+`\":[{\"key\":\"a\",\"value\":1},{\"key\":\"b\",\"value\":null},{\"key\":\"c\",\"value\":2}]}}}`)\n\t}\n\tsuite.assertQueryResult(m, `{root{values(keys:[\"a\",\"b\",\"c\"])}}`,\n\t\t`{\"data\":{\"root\":{\"values\":[1,null,2]}}}`)\n\tsuite.assertQueryResult(m, `{root{keys(keys:[\"a\",\"b\",\"c\"])}}`,\n\t\t`{\"data\":{\"root\":{\"keys\":[\"a\",\"b\",\"c\"]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestStructWithOptionalField() {\n\ttm := NewTypeMap()\n\trootValue := types.NewStruct(\"\", types.StructData{\n\t\t\"n\": types.Number(42),\n\t})\n\trootType := NomsTypeToGraphQLType(types.MakeStructType(\"\",\n\t\ttypes.StructField{Name: \"n\", Type: types.NumberType, Optional: false},\n\t\ttypes.StructField{Name: \"s\", Type: types.StringType, Optional: true},\n\t), false, tm)\n\n\tqueryObj := graphql.NewObject(graphql.ObjectConfig{\n\t\tName: rootQueryKey,\n\t\tFields: graphql.Fields{\n\t\t\trootKey: &graphql.Field{\n\t\t\t\tType: rootType,\n\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\treturn MaybeGetScalar(rootValue), nil\n\t\t\t\t},\n\t\t\t},\n\t\t}})\n\n\tschemaConfig := graphql.SchemaConfig{Query: queryObj}\n\tschema, err := graphql.NewSchema(schemaConfig)\n\tsuite.NoError(err)\n\tctx := NewContext(suite.vs)\n\tquery := `{root{n s}}`\n\n\tr := graphql.Do(graphql.Params{\n\t\tSchema: schema,\n\t\tRequestString: query,\n\t\tContext: ctx,\n\t})\n\n\tsuite.Equal(map[string]interface{}{\"root\": map[string]interface{}{\"n\": float64(42), \"s\": nil}}, r.Data)\n}\n\nfunc (suite *QueryGraphQLSuite) TestMutationScalarArgs() {\n\ttest := func(query, expected string, nomsType *types.Type) {\n\t\ttc := NewTypeConverter()\n\t\tinType, err := tc.NomsTypeToGraphQLInputType(nomsType)\n\t\tsuite.NoError(err)\n\t\toutType := tc.NomsTypeToGraphQLType(nomsType)\n\t\tsuite.assertMutationTypes(query, expected, tc, inType, outType, func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\treturn p.Args[\"new\"], nil\n\t\t})\n\t}\n\n\ttest(`mutation {test(new: 123)}`, `{\"data\": {\"test\": 123}}`, types.NumberType)\n\ttest(`mutation {test(new: 0)}`, `{\"data\": {\"test\": 0}}`, types.NumberType)\n\n\ttest(`mutation {test(new: \"hi\")}`, `{\"data\": {\"test\": \"hi\"}}`, types.StringType)\n\ttest(`mutation {test(new: \"\")}`, `{\"data\": {\"test\": \"\"}}`, types.StringType)\n\n\ttest(`mutation {test(new: true)}`, `{\"data\": {\"test\": true}}`, types.BoolType)\n\ttest(`mutation {test(new: false)}`, `{\"data\": {\"test\": false}}`, types.BoolType)\n}\n\nfunc (suite *QueryGraphQLSuite) TestMutationWeirdosArgs() {\n\ttest := func(query, expected string, nomsType *types.Type) {\n\t\ttc := NewTypeConverter()\n\t\tinType, err := tc.NomsTypeToGraphQLInputType(nomsType)\n\t\tsuite.NoError(err)\n\t\toutType := graphql.String\n\t\tsuite.assertMutationTypes(query, expected, tc, inType, outType, func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\treturn p.Args[\"new\"], nil\n\t\t})\n\t}\n\n\ttest(`mutation {test(new: \"#abc\")}`, `{\"data\": {\"test\": \"#abc\"}}`, types.MakeRefType(types.NumberType))\n\ttest(`mutation {test(new: \"0123456789\")}`, `{\"data\": {\"test\": \"0123456789\"}}`, types.BlobType)\n}\n\nfunc (suite *QueryGraphQLSuite) assertMutationTypes(query, expected string, tc *TypeConverter, inType graphql.Input, outType graphql.Type, resolver graphql.FieldResolveFn) {\n\tbuf := &bytes.Buffer{}\n\troot := types.Number(0)\n\tschemaConfig := graphql.SchemaConfig{\n\t\tMutation: graphql.NewObject(graphql.ObjectConfig{\n\t\t\tName: \"Mutation\",\n\t\t\tFields: graphql.Fields{\n\t\t\t\t\"test\": &graphql.Field{\n\t\t\t\t\tType: outType,\n\t\t\t\t\tArgs: graphql.FieldConfigArgument{\n\t\t\t\t\t\t\"new\": &graphql.ArgumentConfig{\n\t\t\t\t\t\t\tType: inType,\n\t\t\t\t\t\t},\n\t\t\t\t\t},\n\t\t\t\t\tResolve: resolver,\n\t\t\t\t},\n\t\t\t},\n\t\t}),\n\t}\n\tqueryWithSchemaConfig(root, query, schemaConfig, suite.vs, tc, buf)\n\tsuite.JSONEq(expected, buf.String())\n}\n\nfunc (suite *QueryGraphQLSuite) TestMutationCollectionArgs() {\n\ttest := func(query, expected string, expectedArg interface{}, nomsType *types.Type) {\n\t\ttc := NewTypeConverter()\n\t\tinType, err := tc.NomsTypeToGraphQLInputType(nomsType)\n\t\tsuite.NoError(err)\n\t\toutType := graphql.Boolean\n\t\tsuite.assertMutationTypes(query, expected, tc, inType, outType, func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\tsuite.Equal(expectedArg, p.Args[\"new\"])\n\t\t\treturn true, nil\n\t\t})\n\t}\n\n\ttest(`mutation {test(new: [0, 1, 2, 3])}`, `{\"data\": {\"test\": true}}`, []interface{}{float64(0), float64(1), float64(2), float64(3)}, types.MakeListType(types.NumberType))\n\ttest(`mutation {test(new: [])}`, `{\"data\": {\"test\": true}}`, []interface{}{}, types.MakeListType(types.NumberType))\n\n\ttest(`mutation {test(new: [0, 1, 2, 3])}`, `{\"data\": {\"test\": true}}`, []interface{}{float64(0), float64(1), float64(2), float64(3)}, types.MakeSetType(types.NumberType))\n\ttest(`mutation {test(new: [])}`, `{\"data\": {\"test\": true}}`, []interface{}{}, types.MakeSetType(types.NumberType))\n\n\ttest(`mutation {\n test(new: [\n {\n key: 1,\n value: \"a\"\n }, {\n key: 2,\n value: \"b\"\n }\n ])\n }`, `{\"data\": {\"test\": true}}`, []interface{}{\n\t\tmap[string]interface{}{\"key\": float64(1), \"value\": \"a\"},\n\t\tmap[string]interface{}{\"key\": float64(2), \"value\": \"b\"},\n\t}, types.MakeMapType(types.NumberType, types.StringType))\n\ttest(`mutation {test(new: [])}`, `{\"data\": {\"test\": true}}`, []interface{}{}, types.MakeMapType(types.NumberType, types.StringType))\n\n\tst := types.MakeStructTypeFromFields(\"N\", types.FieldMap{\n\t\t\"f\": types.NumberType,\n\t\t\"b\": types.BoolType,\n\t\t\"s\": types.StringType,\n\t})\n\ttest(`mutation {test(new: {\n f: 42,\n b: true,\n s: \"hi\"\n })}`, `{\"data\": {\"test\": true}}`, map[string]interface{}{\"b\": true, \"f\": float64(42), \"s\": \"hi\"}, st)\n}\n\nfunc (suite *QueryGraphQLSuite) TestMapWithComplexKeys() {\n\tm := types.NewMap(suite.vs,\n\t\ttypes.NewList(suite.vs, types.String(\"a\")), types.Number(1),\n\t\ttypes.NewList(suite.vs, types.String(\"c\")), types.Number(2),\n\t\ttypes.NewList(suite.vs, types.String(\"e\")), types.Number(3),\n\t\ttypes.NewList(suite.vs, types.String(\"g\")), types.Number(4),\n\t)\n\n\tsuite.assertQueryResult(m, `{root{values(key: [\"e\"])}}`, `{\"data\":{\"root\":{\"values\":[3]}}}`)\n\tsuite.assertQueryResult(m, `{root{values(key: [])}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\n\t// The ordering here depends on the hash of the value...\n\tsuite.assertQueryResult(m, `{root{values(key: [\"a\"], through: [\"e\"])}}`, `{\"data\":{\"root\":{\"values\":[1, 2, 3]}}}`)\n\n\tsuite.assertQueryResult(m, `{root{values(keys: [[\"a\"],[\"b\"],[\"c\"]])}}`, `{\"data\":{\"root\":{\"values\":[1, null, 2]}}}`)\n\tsuite.assertQueryResult(m, `{\n root {\n keys(keys: [[\"a\"],[\"b\"],[\"c\"]]) {\n values\n }\n }\n }`, `{\"data\": {\n \"root\": {\n \"keys\": [\n {\"values\": [\"a\"]},\n {\"values\": [\"b\"]},\n {\"values\": [\"c\"]}\n ]\n }\n }}`)\n\n\tm2 := types.NewMap(suite.vs,\n\t\ttypes.NewStruct(\"\", types.StructData{\n\t\t\t\"n\": types.String(\"a\"),\n\t\t}), types.Number(1),\n\t\ttypes.NewStruct(\"\", types.StructData{\n\t\t\t\"n\": types.String(\"c\"),\n\t\t}), types.Number(2),\n\t\ttypes.NewStruct(\"\", types.StructData{\n\t\t\t\"n\": types.String(\"e\"),\n\t\t}), types.Number(3),\n\t\ttypes.NewStruct(\"\", types.StructData{\n\t\t\t\"n\": types.String(\"g\"),\n\t\t}), types.Number(4),\n\t)\n\tsuite.assertQueryResult(m2, `{root{values(key: {n: \"e\"})}}`, `{\"data\":{\"root\":{\"values\":[3]}}}`)\n\tsuite.assertQueryResult(m2, `{root{values(key: {n: \"x\"})}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\t// The order is based on hash\n\tsuite.assertQueryResult(m2, `{root{values(key: {n: \"g\"}, through: {n: \"c\"})}}`, `{\"data\":{\"root\":{\"values\":[4,2]}}}`)\n\tsuite.assertQueryResult(m2, `{root{values(keys: [{n: \"a\"}, {n: \"b\"}, {n: \"c\"}])}}`,\n\t\t`{\"data\":{\"root\":{\"values\":[1, null, 2]}}}`)\n\tsuite.assertQueryResult(m2, `{root{keys(keys: [{n: \"a\"}, {n: \"b\"}, {n: \"c\"}]) { n }}}`,\n\t\t`{\"data\":{\"root\":{\"keys\":[{\"n\": \"a\"}, {\"n\": \"b\"}, {\"n\": \"c\"}]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestSetWithComplexKeys() {\n\ts := types.NewSet(suite.vs,\n\t\ttypes.NewList(suite.vs, types.String(\"a\")),\n\t\ttypes.NewList(suite.vs, types.String(\"c\")),\n\t\ttypes.NewList(suite.vs, types.String(\"e\")),\n\t\ttypes.NewList(suite.vs, types.String(\"g\")),\n\t)\n\n\tsuite.assertQueryResult(s, `{root{values(key: [\"e\"]) { values }}}`,\n\t\t`{\"data\":{\"root\":{\"values\":[{\"values\":[\"e\"]}]}}}`)\n\tsuite.assertQueryResult(s, `{root{values(key: []) { values }}}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\n\t// The ordering here depends on the hash of the value...\n\tsuite.assertQueryResult(s, `{root{values(key: [\"g\"], through: [\"c\"]) { values }}}`,\n\t\t`{\"data\":{\"root\":{\"values\":[{\"values\":[\"g\"]},{\"values\":[\"a\"]},{\"values\":[\"c\"]}]}}}`)\n\n\ts2 := types.NewSet(suite.vs,\n\t\ttypes.NewStruct(\"\", types.StructData{\n\t\t\t\"n\": types.String(\"a\"),\n\t\t}),\n\t\ttypes.NewStruct(\"\", types.StructData{\n\t\t\t\"n\": types.String(\"c\"),\n\t\t}),\n\t\ttypes.NewStruct(\"\", types.StructData{\n\t\t\t\"n\": types.String(\"e\"),\n\t\t}),\n\t\ttypes.NewStruct(\"\", types.StructData{\n\t\t\t\"n\": types.String(\"g\"),\n\t\t}),\n\t)\n\n\tsuite.assertQueryResult(s2, `{root{values(key: {n: \"e\"}) { n } }}`,\n\t\t`{\"data\":{\"root\":{\"values\":[{\"n\": \"e\"}]}}}`)\n\tsuite.assertQueryResult(s2, `{root{values(key: {n: \"x\"}) { n } }}`, `{\"data\":{\"root\":{\"values\":[]}}}`)\n\t// The order is based on hash\n\tsuite.assertQueryResult(s2, `{root{values(key: {n: \"c\"}, through: {n: \"e\"}) { n }}}`,\n\t\t`{\"data\":{\"root\":{\"values\":[{\"n\": \"c\"}, {\"n\": \"e\"}]}}}`)\n}\n\nfunc (suite *QueryGraphQLSuite) TestInputToNomsValue() {\n\ttest := func(expected types.Value, val interface{}) {\n\t\tsuite.True(expected.Equals(InputToNomsValue(suite.vs, val, types.TypeOf(expected))))\n\t}\n\n\ttest(types.Number(42), int(42))\n\ttest(types.Number(0), int(0))\n\n\ttest(types.Number(1.23), float64(1.23))\n\ttest(types.Number(0), float64(0))\n\n\ttest(types.Bool(true), true)\n\ttest(types.Bool(false), false)\n\n\ttest(types.String(\"hi\"), \"hi\")\n\ttest(types.String(\"\"), \"\")\n\n\ttest(types.NewList(suite.vs, types.Number(42)), []interface{}{float64(42)})\n\ttest(types.NewList(suite.vs, types.Number(1), types.Number(2)), []interface{}{float64(1), float64(2)})\n\n\ttest(types.NewSet(suite.vs, types.Number(42)), []interface{}{float64(42)})\n\ttest(types.NewSet(suite.vs, types.Number(1), types.Number(2)), []interface{}{float64(1), float64(2)})\n\n\ttest(types.NewMap(suite.vs,\n\t\ttypes.String(\"a\"), types.Number(1),\n\t\ttypes.String(\"b\"), types.Number(2),\n\t), []interface{}{\n\t\tmap[string]interface{}{\"key\": \"a\", \"value\": 1},\n\t\tmap[string]interface{}{\"key\": \"b\", \"value\": 2},\n\t})\n\ttest(types.NewMap(suite.vs,\n\t\ttypes.NewList(suite.vs, types.String(\"a\")), types.Number(1),\n\t\ttypes.NewList(suite.vs, types.String(\"b\")), types.Number(2),\n\t), []interface{}{\n\t\tmap[string]interface{}{\"key\": []interface{}{\"a\"}, \"value\": 1},\n\t\tmap[string]interface{}{\"key\": []interface{}{\"b\"}, \"value\": 2},\n\t})\n\n\ttest(types.NewMap(suite.vs,\n\t\ttypes.NewStruct(\"S\", types.StructData{\"a\": types.Number(1)}), types.Number(11),\n\t\ttypes.NewStruct(\"S\", types.StructData{\"a\": types.Number(2)}), types.Number(22),\n\t), []interface{}{\n\t\tmap[string]interface{}{\"key\": map[string]interface{}{\"a\": float64(1)}, \"value\": 11},\n\t\tmap[string]interface{}{\"key\": map[string]interface{}{\"a\": float64(2)}, \"value\": 22},\n\t})\n\n\ttest(types.NewSet(suite.vs,\n\t\ttypes.NewStruct(\"S\", types.StructData{\"a\": types.Number(1)}),\n\t\ttypes.NewStruct(\"S\", types.StructData{\"a\": types.Number(2)}),\n\t), []interface{}{\n\t\tmap[string]interface{}{\"a\": float64(1)},\n\t\tmap[string]interface{}{\"a\": float64(2)},\n\t})\n\n\texpected := types.NewStruct(\"S\", types.StructData{\n\t\t\"x\": types.Number(42),\n\t})\n\texpectedType := types.MakeStructType(\"S\",\n\t\ttypes.StructField{Name: \"a\", Type: types.BoolType, Optional: true},\n\t\ttypes.StructField{Name: \"x\", Type: types.NumberType, Optional: false},\n\t)\n\tval := map[string]interface{}{\n\t\t\"x\": float64(42),\n\t}\n\tsuite.Equal(expected, InputToNomsValue(suite.vs, val, expectedType))\n\n\tval = map[string]interface{}{\n\t\t\"x\": float64(42),\n\t\t\"a\": nil,\n\t}\n\tsuite.Equal(expected, InputToNomsValue(suite.vs, val, expectedType))\n\n\tval = map[string]interface{}{\n\t\t\"x\": nil,\n\t}\n\tsuite.Panics(func() {\n\t\tInputToNomsValue(suite.vs, val, expectedType)\n\t})\n}\n\nfunc (suite *QueryGraphQLSuite) TestErrorsInInputType() {\n\tut := types.MakeUnionType(types.BoolType, types.NumberType)\n\n\ttest := func(t *types.Type) {\n\t\ttm := NewTypeMap()\n\t\t_, err := NomsTypeToGraphQLInputType(t, tm)\n\t\tsuite.Error(err)\n\t}\n\n\ttest(ut)\n\ttest(types.MakeListType(ut))\n\ttest(types.MakeSetType(ut))\n\ttest(types.MakeMapType(ut, types.BoolType))\n\ttest(types.MakeMapType(types.BoolType, ut))\n\ttest(types.MakeMapType(ut, ut))\n\ttest(types.MakeStructTypeFromFields(\"\", types.FieldMap{\"u\": ut}))\n\n\ttest(types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"l\": types.MakeListType(types.MakeCycleType(\"S\")),\n\t}))\n\ttest(types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"n\": types.NumberType,\n\t\t\"l\": types.MakeListType(types.MakeCycleType(\"S\")),\n\t}))\n}\n\nfunc (suite *QueryGraphQLSuite) TestVariables() {\n\ttest := func(rootValue types.Value, expected string, query string, vars map[string]interface{}) {\n\t\ttc := NewTypeConverter()\n\t\tctx := NewContext(suite.vs)\n\t\tschema, err := graphql.NewSchema(graphql.SchemaConfig{\n\t\t\tQuery: tc.NewRootQueryObject(rootValue),\n\t\t})\n\t\tsuite.NoError(err)\n\n\t\tr := graphql.Do(graphql.Params{\n\t\t\tSchema: schema,\n\t\t\tRequestString: query,\n\t\t\tContext: ctx,\n\t\t\tVariableValues: vars,\n\t\t})\n\t\tb, err := json.Marshal(r)\n\t\tsuite.NoError(err)\n\t\tsuite.JSONEq(expected, string(b))\n\t}\n\n\tv := types.NewList(suite.vs, types.Number(0), types.Number(1), types.Number(2), types.Number(3))\n\ttest(v, `{\"data\":{\"root\":{\"values\":[0,1,2,3]}}}`, `query Test($c: Int) { root { values(count: $c) } }`, nil)\n\ttest(v, `{\"data\":{\"root\":{\"values\":[0,1]}}}`, `query Test($c: Int) { root { values(count: $c) } }`, map[string]interface{}{\n\t\t\"c\": 2,\n\t})\n\n\tm := types.NewMap(suite.vs,\n\t\ttypes.String(\"a\"), types.Number(0),\n\t\ttypes.String(\"b\"), types.Number(1),\n\t\ttypes.String(\"c\"), types.Number(2),\n\t\ttypes.String(\"d\"), types.Number(3),\n\t)\n\ttest(m, `{\"data\":{\"root\":{\"values\":[1]}}}`, `query Test($k: String) { root { values(key: $k) } }`, map[string]interface{}{\n\t\t\"k\": \"b\",\n\t})\n\ttest(m, `{\"data\":{\"root\":{\"values\":[1, 2]}}}`, `query Test($k: String, $t: String) { root { values(key: $k, through: $t) } }`,\n\t\tmap[string]interface{}{\n\t\t\t\"k\": \"b\",\n\t\t\t\"t\": \"c\",\n\t\t})\n\ttest(m, `{\"data\":{\"root\":{\"values\":[0, 2]}}}`, `query Test($ks: [String!]!) { root { values(keys: $ks) } }`,\n\t\tmap[string]interface{}{\n\t\t\t\"ks\": []string{\"a\", \"c\"},\n\t\t})\n\n\tm2 := types.NewMap(suite.vs,\n\t\ttypes.NewStruct(\"S\", types.StructData{\"n\": types.String(\"a\")}), types.Number(0),\n\t\ttypes.NewStruct(\"S\", types.StructData{\"n\": types.String(\"b\")}), types.Number(1),\n\t\ttypes.NewStruct(\"S\", types.StructData{\"n\": types.String(\"c\")}), types.Number(2),\n\t\ttypes.NewStruct(\"S\", types.StructData{\"n\": types.String(\"d\")}), types.Number(3),\n\t)\n\tkeyType := types.TypeOf(m2).Desc.(types.CompoundDesc).ElemTypes[0]\n\tq := fmt.Sprintf(`query Test($k: %s) { root { values(key: $k) } }`, GetInputTypeName(keyType))\n\ttest(m2, `{\"data\":{\"root\":{\"values\":[1]}}}`, q, map[string]interface{}{\n\t\t\"k\": map[string]interface{}{\n\t\t\t\"n\": \"b\",\n\t\t},\n\t})\n\tq = fmt.Sprintf(`query Test($ks: [%s!]) { root { values(keys: $ks) } }`, GetInputTypeName(keyType))\n\ttest(m2, `{\"data\":{\"root\":{\"values\":[0, 3]}}}`, q, map[string]interface{}{\n\t\t\"ks\": []interface{}{\n\t\t\tmap[string]interface{}{\n\t\t\t\t\"n\": \"a\",\n\t\t\t},\n\t\t\tmap[string]interface{}{\n\t\t\t\t\"n\": \"d\",\n\t\t\t},\n\t\t},\n\t})\n\ttest(m2, `{\"data\":null,\"errors\":[{\"message\":\"Variable \\\"$ks\\\" got invalid value [{}].\\nIn element #1: In field \\\"n\\\": Expected \\\"String!\\\", found null.\",\"locations\":[{\"line\":1,\"column\":12}]}]}`,\n\t\tq,\n\t\tmap[string]interface{}{\n\t\t\t\"ks\": []interface{}{\n\t\t\t\tmap[string]interface{}{},\n\t\t\t},\n\t\t},\n\t)\n\ttest(m2, `{\"data\":null,\"errors\":[{\"message\":\"Variable \\\"$ks\\\" got invalid value [{\\\"m\\\":\\\"b\\\",\\\"n\\\":\\\"a\\\"}].\\nIn element #1: In field \\\"m\\\": Unknown field.\",\"locations\":[{\"line\":1,\"column\":12}]}]}`,\n\t\tq,\n\t\tmap[string]interface{}{\n\t\t\t\"ks\": []interface{}{\n\t\t\t\tmap[string]interface{}{\n\t\t\t\t\t\"n\": \"a\",\n\t\t\t\t\t\"m\": \"b\",\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t)\n\ttest(m2, `{\"data\":null,\"errors\":[{\"message\":\"Variable \\\"$ks\\\" got invalid value [{\\\"n\\\":null}].\\nIn element #1: In field \\\"n\\\": Expected \\\"String!\\\", found null.\",\"locations\":[{\"line\":1,\"column\":12}]}]}`,\n\t\tq,\n\t\tmap[string]interface{}{\n\t\t\t\"ks\": []interface{}{\n\t\t\t\tmap[string]interface{}{\n\t\t\t\t\t\"n\": nil,\n\t\t\t\t},\n\t\t\t},\n\t\t},\n\t)\n\ttest(m2, `{\"data\":null,\"errors\":[{\"message\":\"Variable \\\"$ks\\\" got invalid value [null].\\nIn element #1: Expected \\\"SInput_cgmdbo!\\\", found null.\",\"locations\":[{\"line\":1,\"column\":12}]}]}`,\n\t\tq,\n\t\tmap[string]interface{}{\n\t\t\t\"ks\": []interface{}{\n\t\t\t\tnil,\n\t\t\t},\n\t\t},\n\t)\n\n\tm3 := types.NewMap(suite.vs,\n\t\ttypes.NewMap(suite.vs, types.Number(0), types.String(\"zero\")), types.Bool(false),\n\t\ttypes.NewMap(suite.vs, types.Number(1), types.String(\"one\")), types.Bool(true),\n\t)\n\tkeyNomsType := types.TypeOf(m3).Desc.(types.CompoundDesc).ElemTypes[0]\n\ttc := NewTypeConverter()\n\tkeyGraphQLInputType, err := tc.NomsTypeToGraphQLInputType(keyNomsType)\n\tsuite.NoError(err)\n\tq = fmt.Sprintf(`query Test($k: %s!) { root { values(key: $k) } }`, keyGraphQLInputType.String())\n\ttest(m3, `{\"data\":{\"root\":{\"values\":[false]}}}`, q, map[string]interface{}{\n\t\t\"k\": []interface{}{\n\t\t\tmap[string]interface{}{\n\t\t\t\t\"key\": float64(0),\n\t\t\t\t\"value\": \"zero\",\n\t\t\t},\n\t\t},\n\t})\n\ttest(m3, `{\"data\":null,\"errors\":[{\"message\":\"Variable \\\"$k\\\" got invalid value [{\\\"key\\\":0}].\\nIn element #1: In field \\\"value\\\": Expected \\\"String!\\\", found null.\",\"locations\":[{\"line\":1,\"column\":12}]}]}`,\n\t\tq,\n\t\tmap[string]interface{}{\n\t\t\t\"k\": []interface{}{\n\t\t\t\tmap[string]interface{}{\n\t\t\t\t\t\"key\": float64(0),\n\t\t\t\t},\n\t\t\t},\n\t\t})\n\ttest(m3, `{\"data\":null,\"errors\":[{\"message\":\"Variable \\\"$k\\\" got invalid value [{\\\"key\\\":\\\"zero\\\"}].\\nIn element #1: In field \\\"key\\\": Expected type \\\"Float\\\", found \\\"zero\\\".\\nIn element #2: In field \\\"value\\\": Expected \\\"String!\\\", found null.\",\"locations\":[{\"line\":1,\"column\":12}]}]}`,\n\t\tq,\n\t\tmap[string]interface{}{\n\t\t\t\"k\": []interface{}{\n\t\t\t\tmap[string]interface{}{\n\t\t\t\t\t\"key\": \"zero\",\n\t\t\t\t},\n\t\t\t},\n\t\t})\n\ttest(m3, `{\"data\":null,\"errors\":[{\"message\":\"Variable \\\"$k\\\" got invalid value [{\\\"extra\\\":false,\\\"key\\\":0,\\\"value\\\":\\\"zero\\\"}].\\nIn element #1: In field \\\"extra\\\": Unknown field.\",\"locations\":[{\"line\":1,\"column\":12}]}]}`,\n\t\tq,\n\t\tmap[string]interface{}{\n\t\t\t\"k\": []interface{}{\n\t\t\t\tmap[string]interface{}{\n\t\t\t\t\t\"key\": float64(0),\n\t\t\t\t\t\"value\": \"zero\",\n\t\t\t\t\t\"extra\": false,\n\t\t\t\t},\n\t\t\t},\n\t\t})\n\ttest(m3, `{\"data\":null,\"errors\":[{\"message\":\"Variable \\\"$k\\\" got invalid value [null].\\nIn element #1: Expected \\\"NumberStringEntryInput!\\\", found null.\",\"locations\":[{\"line\":1,\"column\":12}]}]}`,\n\t\tq,\n\t\tmap[string]interface{}{\n\t\t\t\"k\": []interface{}{\n\t\t\t\tnil,\n\t\t\t},\n\t\t})\n}\n\nfunc (suite *QueryGraphQLSuite) TestNameFunc() {\n\ttest := func(tc *TypeConverter, rootValue types.Value, expected string, query string, vars map[string]interface{}) {\n\t\tctx := NewContext(suite.vs)\n\t\tschema, err := graphql.NewSchema(graphql.SchemaConfig{\n\t\t\tQuery: tc.NewRootQueryObject(rootValue),\n\t\t})\n\t\tsuite.NoError(err)\n\n\t\tr := graphql.Do(graphql.Params{\n\t\t\tSchema: schema,\n\t\t\tRequestString: query,\n\t\t\tContext: ctx,\n\t\t\tVariableValues: vars,\n\t\t})\n\n\t\tb, err := json.Marshal(r)\n\t\tsuite.NoError(err)\n\t\tsuite.JSONEq(expected, string(b))\n\t}\n\n\taVal := types.NewStruct(\"A\", types.StructData{\n\t\t\"a\": types.Number(1),\n\t})\n\tbVal := types.NewStruct(\"B\", types.StructData{\n\t\t\"b\": types.Number(2),\n\t})\n\n\tlist := types.NewList(suite.vs, aVal, bVal)\n\n\ttc := NewTypeConverter()\n\ttc.NameFunc = func(nomsType *types.Type, isInputType bool) string {\n\t\tif nomsType.Equals(types.TypeOf(aVal)) {\n\t\t\treturn \"A\"\n\t\t}\n\t\tif nomsType.Equals(types.TypeOf(bVal)) {\n\t\t\treturn \"BBB\"\n\t\t}\n\t\treturn DefaultNameFunc(nomsType, isInputType)\n\t}\n\n\tquery := `query {\n root {\n values {\n ... on A {\n a\n }\n ... on BBB {\n b\n }\n }\n }\n }`\n\texpected := `{\n \"data\": {\n \"root\": {\n \"values\": [\n {\"a\": 1},\n {\"b\": 2}\n ]\n }\n }\n }`\n\ttest(tc, list, expected, query, nil)\n\n\tset := types.NewSet(suite.vs, aVal,\n\t\ttypes.NewStruct(\"A\", types.StructData{\n\t\t\t\"a\": types.Number(2),\n\t\t}),\n\t\ttypes.NewStruct(\"A\", types.StructData{\n\t\t\t\"a\": types.Number(3),\n\t\t}),\n\t)\n\ttc = NewTypeConverter()\n\ttc.NameFunc = func(nomsType *types.Type, isInputType bool) string {\n\t\tif nomsType.Equals(types.TypeOf(aVal)) {\n\t\t\tif isInputType {\n\t\t\t\treturn \"AI\"\n\t\t\t}\n\t\t\treturn \"A\"\n\t\t}\n\t\treturn DefaultNameFunc(nomsType, isInputType)\n\t}\n\n\tquery = `query ($key: AI!) {\n root {\n values(key: $key) {\n a\n }\n }\n }`\n\texpected = `{\n \"data\": {\n \"root\": {\n \"values\": [\n {\"a\": 2}\n ]\n }\n }\n }`\n\ttest(tc, set, expected, query, map[string]interface{}{\n\t\t\"key\": map[string]interface{}{\"a\": 2},\n\t})\n}\n\nfunc TestGetListElementsWithSet(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\tv := types.NewSet(vs, types.Number(0), types.Number(1), types.Number(2))\n\tr := getListElements(vs, v, map[string]interface{}{})\n\tassert.Equal([]interface{}{float64(0), float64(1), float64(2)}, r)\n\n\tr = getListElements(vs, v, map[string]interface{}{\n\t\tatKey: 1,\n\t})\n\tassert.Equal([]interface{}{float64(1), float64(2)}, r)\n\n\tr = getListElements(vs, v, map[string]interface{}{\n\t\tcountKey: 2,\n\t})\n\tassert.Equal([]interface{}{float64(0), float64(1)}, r)\n}\n\nfunc TestNoErrorOnNonCyclicTypeRefsInputType(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype User struct {\n\t\tID string `noms:\"id\"`\n\t}\n\ttype Account struct {\n\t\tPendingUsers map[string]User\n\t\tUsers map[string]User\n\t}\n\n\tvar a Account\n\ttyp := marshal.MustMarshalType(a)\n\ttc := NewTypeConverter()\n\t_, err := tc.NomsTypeToGraphQLInputType(typ)\n\tassert.NoError(err)\n}\n\nfunc TestErrorOnCyclicTypeRefsInputType(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttype Node struct {\n\t\tChildren map[string]Node\n\t}\n\n\tvar n Node\n\ttyp := marshal.MustMarshalType(n)\n\ttc := NewTypeConverter()\n\t_, err := tc.NomsTypeToGraphQLInputType(typ)\n\tassert.Error(err)\n}\n" }, { "path": "go/ngql/types.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage ngql\n\nimport (\n\t\"context\"\n\t\"errors\"\n\t\"fmt\"\n\n\t\"strings\"\n\n\t\"github.com/attic-labs/graphql\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// TypeConverter provides functions to convert between Noms types and GraphQL\n// types.\ntype TypeConverter struct {\n\ttm TypeMap\n\tNameFunc NameFunc\n}\n\n// NewTypeConverter creates a new TypeConverter.\nfunc NewTypeConverter() *TypeConverter {\n\treturn &TypeConverter{\n\t\tTypeMap{},\n\t\tDefaultNameFunc,\n\t}\n}\n\n// NameFunc defines how to compute the GraphQL name for a Noms type.\ntype NameFunc func(nomsType *types.Type, isInputType bool) string\n\nfunc (tc *TypeConverter) getTypeName(nomsType *types.Type) string {\n\treturn tc.NameFunc(nomsType, false)\n}\n\nfunc (tc *TypeConverter) getInputTypeName(nomsType *types.Type) string {\n\treturn tc.NameFunc(nomsType, true)\n}\n\n// NomsTypeToGraphQLType creates a GraphQL type from a Noms type that knows how\n// to resolve the Noms values.\nfunc (tc *TypeConverter) NomsTypeToGraphQLType(nomsType *types.Type) graphql.Type {\n\treturn tc.nomsTypeToGraphQLType(nomsType, false)\n}\n\n// NomsTypeToGraphQLInputType creates a GraphQL input type from a Noms type.\n// Input types may not be unions or cyclic structs. If we encounter those\n// this returns an error.\nfunc (tc *TypeConverter) NomsTypeToGraphQLInputType(nomsType *types.Type) (graphql.Input, error) {\n\treturn tc.nomsTypeToGraphQLInputType(nomsType)\n}\n\n// TypeMap is used as a cache in NomsTypeToGraphQLType and\n// NomsTypeToGraphQLInputType.\ntype TypeMap map[typeMapKey]graphql.Type\n\ntype typeMapKey struct {\n\tname string\n\tboxedIfScalar bool\n}\n\n// NewTypeMap creates a new map that is used as a cache in\n// NomsTypeToGraphQLType and NomsTypeToGraphQLInputType.\nfunc NewTypeMap() *TypeMap {\n\treturn &TypeMap{}\n}\n\n// GraphQL has two type systems.\n// - One for output types which is used with resolvers to produce an output set.\n// - And another one for input types. Input types are used to verify that the\n// JSON like data passes as arguments are of the right type.\n// There is some overlap here. Scalars are the same and List can be used in\n// both.\n// The significant difference is graphql.Object (output) vs graphql.InputObject\n// Input types cannot be unions and input object types cannot contain cycles.\n\ntype graphQLTypeMode uint8\n\nconst (\n\tinputMode graphQLTypeMode = iota\n\toutputMode\n)\n\n// In terms of resolving a graph of data, there are three types of value:\n// scalars, lists and maps. During resolution, we are converting some noms\n// value to a graphql value. A getFieldFn will be invoked for a matching noms\n// type. Its job is to retrieve the sub-value from the noms type which is\n// mapped to a graphql map as a fieldname.\ntype getFieldFn func(v interface{}, fieldName string, ctx context.Context) types.Value\n\n// When a field name is resolved, it may take key:value arguments. A\n// getSubvaluesFn handles returning one or more *noms* values whose presence is\n// indicated by the provided arguments.\ntype getSubvaluesFn func(vrw types.ValueReadWriter, v types.Value, args map[string]interface{}) interface{}\n\n// GraphQL requires all memberTypes in a Union to be Structs, so when a noms\n// union contains a scalar, we represent it in that context as a \"boxed\" value.\n// E.g.\n// Boolean! =>\n// type BooleanValue {\n// scalarValue: Boolean!\n// }\nfunc (tc *TypeConverter) scalarToValue(nomsType *types.Type, scalarType graphql.Type) graphql.Type {\n\treturn graphql.NewObject(graphql.ObjectConfig{\n\t\tName: fmt.Sprintf(\"%sValue\", tc.getTypeName(nomsType)),\n\t\tFields: graphql.Fields{\n\t\t\tscalarValue: &graphql.Field{\n\t\t\t\tType: graphql.NewNonNull(scalarType),\n\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\treturn p.Source, nil // p.Source is already a go-native scalar type\n\t\t\t\t},\n\t\t\t},\n\t\t}})\n}\n\nfunc isScalar(nomsType *types.Type) bool {\n\tswitch nomsType {\n\tcase types.BoolType, types.NumberType, types.StringType:\n\t\treturn true\n\tdefault:\n\t\treturn false\n\t}\n}\n\n// NomsTypeToGraphQLType creates a GraphQL type from a Noms type that knows how\n// to resolve the Noms values.\nfunc NomsTypeToGraphQLType(nomsType *types.Type, boxedIfScalar bool, tm *TypeMap) graphql.Type {\n\ttc := TypeConverter{*tm, DefaultNameFunc}\n\treturn tc.nomsTypeToGraphQLType(nomsType, boxedIfScalar)\n}\n\nfunc (tc *TypeConverter) nomsTypeToGraphQLType(nomsType *types.Type, boxedIfScalar bool) graphql.Type {\n\tname := tc.getTypeName(nomsType)\n\tkey := typeMapKey{name, boxedIfScalar && isScalar(nomsType)}\n\tgqlType, ok := tc.tm[key]\n\tif ok {\n\t\treturn gqlType\n\t}\n\n\t// The graphql package has built in support for recursive types using\n\t// FieldsThunk which allows the inner type to refer to an outer type by\n\t// lazily initializing the fields.\n\tswitch nomsType.TargetKind() {\n\tcase types.NumberKind:\n\t\tgqlType = graphql.Float\n\t\tif boxedIfScalar {\n\t\t\tgqlType = tc.scalarToValue(nomsType, gqlType)\n\t\t}\n\n\tcase types.StringKind:\n\t\tgqlType = graphql.String\n\t\tif boxedIfScalar {\n\t\t\tgqlType = tc.scalarToValue(nomsType, gqlType)\n\t\t}\n\n\tcase types.BoolKind:\n\t\tgqlType = graphql.Boolean\n\t\tif boxedIfScalar {\n\t\t\tgqlType = tc.scalarToValue(nomsType, gqlType)\n\t\t}\n\n\tcase types.StructKind:\n\t\tgqlType = tc.structToGQLObject(nomsType)\n\n\tcase types.ListKind, types.SetKind:\n\t\tgqlType = tc.listAndSetToGraphQLObject(nomsType)\n\n\tcase types.MapKind:\n\t\tgqlType = tc.mapToGraphQLObject(nomsType)\n\n\tcase types.RefKind:\n\t\tgqlType = tc.refToGraphQLObject(nomsType)\n\n\tcase types.UnionKind:\n\t\tgqlType = tc.unionToGQLUnion(nomsType)\n\n\tcase types.BlobKind, types.ValueKind, types.TypeKind:\n\t\t// TODO: https://github.com/attic-labs/noms/issues/3155\n\t\tgqlType = graphql.String\n\n\tcase types.CycleKind:\n\t\tpanic(\"not reached\") // we should never attempt to create a schema for any unresolved cycle\n\n\tdefault:\n\t\tpanic(\"not reached\")\n\t}\n\n\ttc.tm[key] = gqlType\n\treturn gqlType\n}\n\n// NomsTypeToGraphQLInputType creates a GraphQL input type from a Noms type.\n// Input types may not be unions or cyclic structs. If we encounter those\n// this returns an error.\nfunc NomsTypeToGraphQLInputType(nomsType *types.Type, tm *TypeMap) (graphql.Input, error) {\n\ttc := TypeConverter{*tm, DefaultNameFunc}\n\treturn tc.nomsTypeToGraphQLInputType(nomsType)\n}\n\nfunc (tc *TypeConverter) nomsTypeToGraphQLInputType(nomsType *types.Type) (graphql.Input, error) {\n\t// GraphQL input types do not support cycles.\n\tif types.HasStructCycles(nomsType) {\n\t\treturn nil, errors.New(\"GraphQL input type cannot contain cycles\")\n\t}\n\n\tname := tc.getInputTypeName(nomsType)\n\tkey := typeMapKey{name, false}\n\tgqlType, ok := tc.tm[key]\n\tif ok {\n\t\treturn gqlType, nil\n\t}\n\n\tvar err error\n\tswitch nomsType.TargetKind() {\n\tcase types.NumberKind:\n\t\tgqlType = graphql.Float\n\n\tcase types.StringKind:\n\t\tgqlType = graphql.String\n\n\tcase types.BoolKind:\n\t\tgqlType = graphql.Boolean\n\n\tcase types.StructKind:\n\t\tgqlType, err = tc.structToGQLInputObject(nomsType)\n\n\tcase types.ListKind, types.SetKind:\n\t\tgqlType, err = tc.listAndSetToGraphQLInputObject(nomsType)\n\n\tcase types.MapKind:\n\t\tgqlType, err = tc.mapToGraphQLInputObject(nomsType)\n\n\tcase types.RefKind:\n\t\tgqlType = graphql.String\n\n\tcase types.UnionKind:\n\t\treturn nil, errors.New(\"GraphQL input type cannot contain unions\")\n\n\tcase types.BlobKind, types.ValueKind, types.TypeKind:\n\t\t// TODO: https://github.com/attic-labs/noms/issues/3155\n\t\tgqlType = graphql.String\n\n\tcase types.CycleKind:\n\t\tpanic(\"not reachable\") // This is handled at the top of nomsTypeToGraphQLInputType\n\n\tdefault:\n\t\tpanic(\"not reached\")\n\t}\n\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\ttc.tm[key] = gqlType\n\treturn gqlType, nil\n}\n\nfunc isEmptyNomsUnion(nomsType *types.Type) bool {\n\treturn nomsType.TargetKind() == types.UnionKind && len(nomsType.Desc.(types.CompoundDesc).ElemTypes) == 0\n}\n\n// Creates a union of structs type.\nfunc (tc *TypeConverter) unionToGQLUnion(nomsType *types.Type) *graphql.Union {\n\tnomsMemberTypes := nomsType.Desc.(types.CompoundDesc).ElemTypes\n\tmemberTypes := make([]*graphql.Object, len(nomsMemberTypes))\n\n\tfor i, nomsUnionType := range nomsMemberTypes {\n\t\t// Member types cannot be non-null and must be struct (graphl.Object)\n\t\tmemberTypes[i] = tc.nomsTypeToGraphQLType(nomsUnionType, true).(*graphql.Object)\n\t}\n\n\treturn graphql.NewUnion(graphql.UnionConfig{\n\t\tName: tc.getTypeName(nomsType),\n\t\tTypes: memberTypes,\n\t\tResolveType: func(p graphql.ResolveTypeParams) *graphql.Object {\n\t\t\tif v, ok := p.Value.(types.Value); ok {\n\t\t\t\t// We cannot just get the type of the value here. GraphQL requires\n\t\t\t\t// us to return one of the types in memberTypes.\n\t\t\t\tfor i, t := range nomsMemberTypes {\n\t\t\t\t\tif types.IsValueSubtypeOf(v, t) {\n\t\t\t\t\t\treturn memberTypes[i]\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\treturn nil\n\t\t\t}\n\n\t\t\tvar nomsType *types.Type\n\t\t\tswitch p.Value.(type) {\n\t\t\tcase float64:\n\t\t\t\tnomsType = types.NumberType\n\t\t\tcase string:\n\t\t\t\tnomsType = types.StringType\n\t\t\tcase bool:\n\t\t\t\tnomsType = types.BoolType\n\t\t\t}\n\t\t\treturn tc.nomsTypeToGraphQLType(nomsType, true).(*graphql.Object)\n\t\t},\n\t})\n}\n\nfunc (tc *TypeConverter) structToGQLObject(nomsType *types.Type) *graphql.Object {\n\treturn graphql.NewObject(graphql.ObjectConfig{\n\t\tName: tc.getTypeName(nomsType),\n\t\tFields: graphql.FieldsThunk(func() graphql.Fields {\n\t\t\tstructDesc := nomsType.Desc.(types.StructDesc)\n\t\t\tfields := graphql.Fields{\n\t\t\t\t\"hash\": &graphql.Field{\n\t\t\t\t\tType: graphql.NewNonNull(graphql.String),\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\treturn p.Source.(types.Struct).Hash().String(), nil\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t}\n\n\t\t\tstructDesc.IterFields(func(name string, nomsFieldType *types.Type, optional bool) {\n\t\t\t\tfieldType := tc.nomsTypeToGraphQLType(nomsFieldType, false)\n\t\t\t\tif !optional {\n\t\t\t\t\tfieldType = graphql.NewNonNull(fieldType)\n\t\t\t\t}\n\n\t\t\t\tfields[name] = &graphql.Field{\n\t\t\t\t\tType: fieldType,\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\tif field, ok := p.Source.(types.Struct).MaybeGet(name); ok {\n\t\t\t\t\t\t\treturn MaybeGetScalar(field), nil\n\t\t\t\t\t\t}\n\t\t\t\t\t\treturn nil, nil\n\t\t\t\t\t},\n\t\t\t\t}\n\t\t\t})\n\n\t\t\treturn fields\n\t\t}),\n\t})\n}\n\nfunc (tc *TypeConverter) listAndSetToGraphQLInputObject(nomsType *types.Type) (graphql.Input, error) {\n\tnomsValueType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\telemType, err := tc.nomsTypeToGraphQLInputType(nomsValueType)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn graphql.NewList(graphql.NewNonNull(elemType)), nil\n}\n\nfunc (tc *TypeConverter) mapToGraphQLInputObject(nomsType *types.Type) (graphql.Input, error) {\n\tnomsKeyType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\tnomsValueType := nomsType.Desc.(types.CompoundDesc).ElemTypes[1]\n\n\tkeyType, err := tc.nomsTypeToGraphQLInputType(nomsKeyType)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tvalueType, err := tc.nomsTypeToGraphQLInputType(nomsValueType)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tentryType := tc.mapEntryToGraphQLInputObject(keyType, valueType, nomsKeyType, nomsValueType)\n\treturn graphql.NewList(entryType), nil\n}\n\nfunc (tc *TypeConverter) structToGQLInputObject(nomsType *types.Type) (graphql.Input, error) {\n\tvar err error\n\trv := graphql.NewInputObject(graphql.InputObjectConfig{\n\t\tName: tc.getInputTypeName(nomsType),\n\t\tFields: graphql.InputObjectConfigFieldMapThunk(func() graphql.InputObjectConfigFieldMap {\n\t\t\tstructDesc := nomsType.Desc.(types.StructDesc)\n\t\t\tfields := make(graphql.InputObjectConfigFieldMap, structDesc.Len())\n\n\t\t\tstructDesc.IterFields(func(name string, nomsFieldType *types.Type, optional bool) {\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tvar fieldType graphql.Input\n\t\t\t\tfieldType, err = tc.nomsTypeToGraphQLInputType(nomsFieldType)\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tif !optional {\n\t\t\t\t\tfieldType = graphql.NewNonNull(fieldType)\n\t\t\t\t}\n\t\t\t\tfields[name] = &graphql.InputObjectFieldConfig{\n\t\t\t\t\tType: fieldType,\n\t\t\t\t}\n\t\t\t})\n\n\t\t\treturn fields\n\t\t}),\n\t})\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn rv, nil\n}\n\nvar listArgs = graphql.FieldConfigArgument{\n\tatKey: &graphql.ArgumentConfig{Type: graphql.Int},\n\tcountKey: &graphql.ArgumentConfig{Type: graphql.Int},\n}\n\nfunc getListElements(vrw types.ValueReadWriter, v types.Value, args map[string]interface{}) interface{} {\n\tl := v.(types.Collection)\n\tidx := 0\n\tcount := int(l.Len())\n\tend := count\n\n\tif at, ok := args[atKey].(int); ok {\n\t\tidx = at\n\t}\n\n\tif c, ok := args[countKey].(int); ok {\n\t\tcount = c\n\t}\n\n\t// Clamp ranges\n\tif count <= 0 || idx >= end {\n\t\treturn ([]interface{})(nil)\n\t}\n\tif idx < 0 {\n\t\tidx = 0\n\t}\n\tif idx+count > end {\n\t\tcount = end - idx\n\t}\n\n\tvalues := make([]interface{}, count)\n\n\tcols, offset := types.LoadLeafNodes([]types.Collection{l}, uint64(idx), uint64(idx+count))\n\n\t// Iterate the collections we got, skipping the first offset elements and bailing out\n\t// once we've filled values with count elements.\n\telementsSeen := uint64(0)\n\tmaybeAddElement := func(v types.Value) {\n\t\tif elementsSeen >= offset && elementsSeen-offset < uint64(count) {\n\t\t\tvalues[elementsSeen-offset] = MaybeGetScalar(v)\n\t\t}\n\t\telementsSeen++\n\t}\n\t// TODO: Use a cursor so we do not have to instantiate all values. @arv has a\n\t// change in the works that only creates Values as needed.\n\tfor _, c := range cols {\n\t\tv := c.(types.Value)\n\t\tv.WalkValues(maybeAddElement)\n\t\tif elementsSeen-offset >= uint64(count) {\n\t\t\tbreak\n\t\t}\n\t}\n\n\treturn values\n}\n\nfunc getSetElements(vrw types.ValueReadWriter, v types.Value, args map[string]interface{}) interface{} {\n\ts := v.(types.Set)\n\n\titer, nomsKey, nomsThrough, count, singleExactMatch := getCollectionArgs(vrw, s, args, iteratorFactory{\n\t\tIteratorFrom: func(from types.Value) interface{} {\n\t\t\treturn s.IteratorFrom(from)\n\t\t},\n\t\tIteratorAt: func(at uint64) interface{} {\n\t\t\treturn s.IteratorAt(at)\n\t\t},\n\t\tFirst: func() interface{} {\n\t\t\treturn &setFirstIterator{s: s}\n\t\t},\n\t})\n\n\tif count == 0 {\n\t\treturn ([]interface{})(nil)\n\t}\n\n\tsetIter := iter.(types.SetIterator)\n\tvalues := make([]interface{}, 0, count)\n\tfor i := uint64(0); i < count; i++ {\n\t\tv := setIter.Next()\n\t\tif v == nil {\n\t\t\tbreak\n\t\t}\n\t\tif singleExactMatch {\n\t\t\tif nomsKey.Equals(v) {\n\t\t\t\tvalues = append(values, MaybeGetScalar(v))\n\t\t\t}\n\t\t\tbreak\n\t\t}\n\n\t\tif nomsThrough != nil {\n\t\t\tif !nomsThrough.Less(v) {\n\t\t\t\tvalues = append(values, MaybeGetScalar(v))\n\t\t\t} else {\n\t\t\t\tbreak\n\t\t\t}\n\t\t} else {\n\t\t\tvalues = append(values, MaybeGetScalar(v))\n\t\t}\n\t}\n\n\treturn values\n}\n\nfunc getCollectionArgs(vrw types.ValueReadWriter, col types.Collection, args map[string]interface{}, factory iteratorFactory) (iter interface{}, nomsKey, nomsThrough types.Value, count uint64, singleExactMatch bool) {\n\ttyp := types.TypeOf(col)\n\tlength := col.Len()\n\tnomsKeyType := typ.Desc.(types.CompoundDesc).ElemTypes[0]\n\n\tif keys, ok := args[keysKey]; ok {\n\t\tslice := keys.([]interface{})\n\t\tnomsKeys := make(types.ValueSlice, len(slice))\n\t\tfor i, v := range slice {\n\t\t\tvar nomsValue types.Value\n\t\t\tnomsValue = InputToNomsValue(vrw, v, nomsKeyType)\n\t\t\tnomsKeys[i] = nomsValue\n\t\t}\n\t\tcount = uint64(len(slice))\n\t\titer = &mapIteratorForKeys{\n\t\t\tm: col.(types.Map),\n\t\t\tkeys: nomsKeys,\n\t\t}\n\t\treturn\n\t}\n\n\tnomsThrough = getThroughArg(vrw, nomsKeyType, args)\n\n\tcount, singleExactMatch = getCountArg(length, args)\n\n\tif key, ok := args[keyKey]; ok {\n\t\tnomsKey = InputToNomsValue(vrw, key, nomsKeyType)\n\t\titer = factory.IteratorFrom(nomsKey)\n\t} else if at, ok := args[atKey]; ok {\n\t\tidx := at.(int)\n\t\tif idx < 0 {\n\t\t\tidx = 0\n\t\t} else if uint64(idx) > length {\n\t\t\tcount = 0\n\t\t\treturn\n\t\t}\n\t\titer = factory.IteratorAt(uint64(idx))\n\t} else if count == 1 && !singleExactMatch {\n\t\t// no key, no at, no through, but a count:1\n\t\titer = factory.First()\n\t} else {\n\t\titer = factory.IteratorAt(0)\n\t}\n\n\treturn\n}\n\ntype mapAppender func(slice []interface{}, k, v types.Value) []interface{}\n\ntype mapiter interface {\n\tValid() bool\n\tEntry() (k, v types.Value)\n\tNext() bool\n}\n\nfunc getMapElements(vrw types.ValueReadWriter, v types.Value, args map[string]interface{}, app mapAppender) (interface{}, error) {\n\tm := v.(types.Map)\n\n\titer, nomsKey, nomsThrough, count, singleExactMatch := getCollectionArgs(vrw, m, args, iteratorFactory{\n\t\tIteratorFrom: func(from types.Value) interface{} {\n\t\t\treturn m.IteratorFrom(from)\n\t\t},\n\t\tIteratorAt: func(at uint64) interface{} {\n\t\t\treturn m.IteratorAt(at)\n\t\t},\n\t\tFirst: func() interface{} {\n\t\t\treturn &mapFirstIterator{m: &m}\n\t\t},\n\t})\n\n\tif count == 0 {\n\t\treturn ([]interface{})(nil), nil\n\t}\n\n\tmapIter := iter.(mapiter)\n\tvalues := make([]interface{}, 0, count)\n\tfor i := uint64(0); i < count; i++ {\n\t\tif !mapIter.Valid() {\n\t\t\tbreak\n\t\t}\n\n\t\tk, v := mapIter.Entry()\n\n\t\tif singleExactMatch {\n\t\t\tif nomsKey.Equals(k) {\n\t\t\t\tvalues = app(values, k, v)\n\t\t\t}\n\t\t\tbreak\n\t\t}\n\n\t\tif nomsThrough != nil {\n\t\t\tif !nomsThrough.Less(k) {\n\t\t\t\tvalues = app(values, k, v)\n\t\t\t} else {\n\t\t\t\tbreak\n\t\t\t}\n\t\t} else {\n\t\t\tvalues = app(values, k, v)\n\t\t}\n\n\t\tmapIter.Next()\n\t}\n\n\treturn values, nil\n}\n\nfunc getCountArg(count uint64, args map[string]interface{}) (c uint64, singleExactMatch bool) {\n\tif c, ok := args[countKey]; ok {\n\t\tc := c.(int)\n\t\tif c <= 0 {\n\t\t\treturn 0, false\n\t\t}\n\t\treturn uint64(c), false\n\t}\n\t// If we have key and no count/through we use count 1\n\t_, hasKey := args[keyKey]\n\t_, hasThrough := args[throughKey]\n\tif hasKey && !hasThrough {\n\t\treturn uint64(1), true\n\t}\n\n\treturn count, false\n}\n\nfunc getThroughArg(vrw types.ValueReadWriter, nomsKeyType *types.Type, args map[string]interface{}) types.Value {\n\tif through, ok := args[throughKey]; ok {\n\t\treturn InputToNomsValue(vrw, through, nomsKeyType)\n\t}\n\treturn nil\n}\n\ntype iteratorFactory struct {\n\tIteratorFrom func(from types.Value) interface{}\n\tIteratorAt func(at uint64) interface{}\n\tFirst func() interface{}\n}\n\ntype mapEntry struct {\n\tkey, value types.Value\n}\n\n// Map data must be returned as a list of key-value pairs. Each unique keyType:valueType is\n// represented as a graphql\n//\n// type Entry {\n//\t key: !\n//\t value: !\n// }\nfunc (tc *TypeConverter) mapEntryToGraphQLObject(keyType, valueType graphql.Type, nomsKeyType, nomsValueType *types.Type) graphql.Type {\n\treturn graphql.NewNonNull(graphql.NewObject(graphql.ObjectConfig{\n\t\tName: fmt.Sprintf(\"%s%sEntry\", tc.getTypeName(nomsKeyType), tc.getTypeName(nomsValueType)),\n\t\tFields: graphql.FieldsThunk(func() graphql.Fields {\n\t\t\treturn graphql.Fields{\n\t\t\t\tkeyKey: &graphql.Field{\n\t\t\t\t\tType: keyType,\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\tentry := p.Source.(mapEntry)\n\t\t\t\t\t\treturn MaybeGetScalar(entry.key), nil\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t\tvalueKey: &graphql.Field{\n\t\t\t\t\tType: valueType,\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\tentry := p.Source.(mapEntry)\n\t\t\t\t\t\treturn MaybeGetScalar(entry.value), nil\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t}\n\t\t}),\n\t}))\n}\n\nfunc (tc *TypeConverter) mapEntryToGraphQLInputObject(keyType, valueType graphql.Input, nomsKeyType, nomsValueType *types.Type) graphql.Input {\n\treturn graphql.NewNonNull(graphql.NewInputObject(graphql.InputObjectConfig{\n\t\tName: fmt.Sprintf(\"%s%sEntryInput\", tc.getInputTypeName(nomsKeyType), tc.getInputTypeName(nomsValueType)),\n\t\tFields: graphql.InputObjectConfigFieldMapThunk(func() graphql.InputObjectConfigFieldMap {\n\t\t\treturn graphql.InputObjectConfigFieldMap{\n\t\t\t\tkeyKey: &graphql.InputObjectFieldConfig{\n\t\t\t\t\tType: graphql.NewNonNull(keyType),\n\t\t\t\t},\n\t\t\t\tvalueKey: &graphql.InputObjectFieldConfig{\n\t\t\t\t\tType: graphql.NewNonNull(valueType),\n\t\t\t\t},\n\t\t\t}\n\t\t}),\n\t}))\n}\n\n// DefaultNameFunc returns the GraphQL type name for a Noms type.\nfunc DefaultNameFunc(nomsType *types.Type, isInputType bool) string {\n\tif isInputType {\n\t\treturn GetInputTypeName(nomsType)\n\t}\n\treturn GetTypeName(nomsType)\n}\n\n// GetTypeName provides a unique type name that is used by GraphQL.\nfunc GetTypeName(nomsType *types.Type) string {\n\treturn getTypeName(nomsType, \"\")\n}\n\n// GetInputTypeName returns a type name that is unique and useful for GraphQL\n// input types.\nfunc GetInputTypeName(nomsType *types.Type) string {\n\treturn getTypeName(nomsType, \"Input\")\n}\n\nfunc getTypeName(nomsType *types.Type, suffix string) string {\n\tswitch nomsType.TargetKind() {\n\tcase types.BoolKind:\n\t\treturn \"Boolean\"\n\n\tcase types.NumberKind:\n\t\treturn \"Number\"\n\n\tcase types.StringKind:\n\t\treturn \"String\"\n\n\tcase types.BlobKind:\n\t\treturn \"Blob\"\n\n\tcase types.ValueKind:\n\t\treturn \"Value\"\n\n\tcase types.ListKind:\n\t\tnomsValueType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\t\tif isEmptyNomsUnion(nomsValueType) {\n\t\t\treturn \"EmptyList\"\n\t\t}\n\t\treturn fmt.Sprintf(\"%sList%s\", GetTypeName(nomsValueType), suffix)\n\n\tcase types.MapKind:\n\t\tnomsKeyType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\t\tnomsValueType := nomsType.Desc.(types.CompoundDesc).ElemTypes[1]\n\t\tif isEmptyNomsUnion(nomsKeyType) {\n\t\t\td.Chk.True(isEmptyNomsUnion(nomsValueType))\n\t\t\treturn \"EmptyMap\"\n\t\t}\n\n\t\treturn fmt.Sprintf(\"%sTo%sMap%s\", GetTypeName(nomsKeyType), GetTypeName(nomsValueType), suffix)\n\n\tcase types.RefKind:\n\t\treturn fmt.Sprintf(\"%sRef%s\", GetTypeName(nomsType.Desc.(types.CompoundDesc).ElemTypes[0]), suffix)\n\n\tcase types.SetKind:\n\t\tnomsValueType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\t\tif isEmptyNomsUnion(nomsValueType) {\n\t\t\treturn \"EmptySet\"\n\t\t}\n\n\t\treturn fmt.Sprintf(\"%sSet%s\", GetTypeName(nomsValueType), suffix)\n\n\tcase types.StructKind:\n\t\t// GraphQL Name cannot start with a number.\n\t\t// GraphQL type names must be globally unique.\n\t\treturn fmt.Sprintf(\"%s%s_%s\", nomsType.Desc.(types.StructDesc).Name, suffix, nomsType.Hash().String()[:6])\n\n\tcase types.TypeKind:\n\t\t// GraphQL Name cannot start with a number.\n\t\t// TODO: https://github.com/attic-labs/noms/issues/3155\n\t\treturn fmt.Sprintf(\"Type%s_%s\", suffix, nomsType.Hash().String()[:6])\n\n\tcase types.UnionKind:\n\t\tunionMemberTypes := nomsType.Desc.(types.CompoundDesc).ElemTypes\n\t\tnames := make([]string, len(unionMemberTypes))\n\t\tfor i, unionMemberType := range unionMemberTypes {\n\t\t\tnames[i] = GetTypeName(unionMemberType)\n\t\t}\n\t\treturn strings.Join(names, \"Or\") + suffix\n\n\tcase types.CycleKind:\n\t\treturn \"Cycle\"\n\n\tdefault:\n\t\tpanic(fmt.Sprintf(\"(GetTypeName) not reached: %s\", nomsType.Describe()))\n\t}\n}\n\nfunc argsWithSize() graphql.Fields {\n\treturn graphql.Fields{\n\t\tsizeKey: &graphql.Field{\n\t\t\tType: graphql.Float,\n\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\tc := p.Source.(types.Collection)\n\t\t\t\treturn MaybeGetScalar(types.Number(c.Len())), nil\n\t\t\t},\n\t\t},\n\t}\n}\n\nfunc (tc *TypeConverter) listAndSetToGraphQLObject(nomsType *types.Type) *graphql.Object {\n\tnomsValueType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\tvar listType, valueType graphql.Type\n\tvar keyInputType graphql.Input\n\tvar keyInputError error\n\tif !isEmptyNomsUnion(nomsValueType) {\n\t\tvalueType = tc.nomsTypeToGraphQLType(nomsValueType, false)\n\t\tkeyInputType, keyInputError = tc.nomsTypeToGraphQLInputType(nomsValueType)\n\t\tlistType = graphql.NewNonNull(valueType)\n\t}\n\n\treturn graphql.NewObject(graphql.ObjectConfig{\n\t\tName: tc.getTypeName(nomsType),\n\t\tFields: graphql.FieldsThunk(func() graphql.Fields {\n\t\t\tfields := argsWithSize()\n\n\t\t\tif listType != nil {\n\t\t\t\tvar args graphql.FieldConfigArgument\n\t\t\t\tvar getSubvalues getSubvaluesFn\n\n\t\t\t\tswitch nomsType.TargetKind() {\n\t\t\t\tcase types.ListKind:\n\t\t\t\t\targs = listArgs\n\t\t\t\t\tgetSubvalues = getListElements\n\n\t\t\t\tcase types.SetKind:\n\t\t\t\t\targs = graphql.FieldConfigArgument{\n\t\t\t\t\t\tatKey: &graphql.ArgumentConfig{Type: graphql.Int},\n\t\t\t\t\t\tcountKey: &graphql.ArgumentConfig{Type: graphql.Int},\n\t\t\t\t\t}\n\t\t\t\t\tif keyInputError == nil {\n\t\t\t\t\t\targs[keyKey] = &graphql.ArgumentConfig{Type: keyInputType}\n\t\t\t\t\t\targs[throughKey] = &graphql.ArgumentConfig{Type: keyInputType}\n\t\t\t\t\t}\n\t\t\t\t\tgetSubvalues = getSetElements\n\t\t\t\t}\n\t\t\t\tvaluesField := &graphql.Field{\n\t\t\t\t\tType: graphql.NewList(listType),\n\t\t\t\t\tArgs: args,\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\tc := p.Source.(types.Collection)\n\t\t\t\t\t\tvrw := p.Context.Value(vrwKey).(types.ValueReadWriter)\n\t\t\t\t\t\treturn getSubvalues(vrw, c, p.Args), nil\n\t\t\t\t\t},\n\t\t\t\t}\n\t\t\t\tfields[valuesKey] = valuesField\n\t\t\t\tfields[elementsKey] = valuesField\n\t\t\t}\n\n\t\t\treturn fields\n\t\t}),\n\t})\n}\n\nfunc (tc *TypeConverter) mapToGraphQLObject(nomsType *types.Type) *graphql.Object {\n\treturn graphql.NewObject(graphql.ObjectConfig{\n\t\tName: tc.getTypeName(nomsType),\n\t\tFields: graphql.FieldsThunk(func() graphql.Fields {\n\t\t\tnomsKeyType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\t\t\tnomsValueType := nomsType.Desc.(types.CompoundDesc).ElemTypes[1]\n\t\t\tisEmptyMap := isEmptyNomsUnion(nomsKeyType) || isEmptyNomsUnion(nomsValueType)\n\n\t\t\tfields := argsWithSize()\n\n\t\t\tif !isEmptyMap {\n\t\t\t\tkeyType := tc.nomsTypeToGraphQLType(nomsKeyType, false)\n\t\t\t\tkeyInputType, keyInputError := tc.nomsTypeToGraphQLInputType(nomsKeyType)\n\t\t\t\tvalueType := tc.nomsTypeToGraphQLType(nomsValueType, false)\n\t\t\t\tentryType := tc.mapEntryToGraphQLObject(graphql.NewNonNull(keyType), valueType, nomsKeyType, nomsValueType)\n\n\t\t\t\targs := graphql.FieldConfigArgument{\n\t\t\t\t\tatKey: &graphql.ArgumentConfig{Type: graphql.Int},\n\t\t\t\t\tcountKey: &graphql.ArgumentConfig{Type: graphql.Int},\n\t\t\t\t}\n\t\t\t\tif keyInputError == nil {\n\t\t\t\t\targs[keyKey] = &graphql.ArgumentConfig{Type: keyInputType}\n\t\t\t\t\targs[keysKey] = &graphql.ArgumentConfig{Type: graphql.NewList(graphql.NewNonNull(keyInputType))}\n\t\t\t\t\targs[throughKey] = &graphql.ArgumentConfig{Type: keyInputType}\n\t\t\t\t}\n\n\t\t\t\tentriesField := &graphql.Field{\n\t\t\t\t\tType: graphql.NewList(entryType),\n\t\t\t\t\tArgs: args,\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\tc := p.Source.(types.Collection)\n\t\t\t\t\t\tvrw := p.Context.Value(vrwKey).(types.ValueReadWriter)\n\t\t\t\t\t\treturn getMapElements(vrw, c, p.Args, mapAppendEntry)\n\t\t\t\t\t},\n\t\t\t\t}\n\t\t\t\tfields[entriesKey] = entriesField\n\t\t\t\tfields[elementsKey] = entriesField\n\n\t\t\t\tfields[keysKey] = &graphql.Field{\n\t\t\t\t\tType: graphql.NewList(keyType),\n\t\t\t\t\tArgs: args,\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\tc := p.Source.(types.Collection)\n\t\t\t\t\t\tvrw := p.Context.Value(vrwKey).(types.ValueReadWriter)\n\t\t\t\t\t\treturn getMapElements(vrw, c, p.Args, mapAppendKey)\n\t\t\t\t\t},\n\t\t\t\t}\n\t\t\t\tfields[valuesKey] = &graphql.Field{\n\t\t\t\t\tType: graphql.NewList(valueType),\n\t\t\t\t\tArgs: args,\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\tc := p.Source.(types.Collection)\n\t\t\t\t\t\tvrw := p.Context.Value(vrwKey).(types.ValueReadWriter)\n\t\t\t\t\t\treturn getMapElements(vrw, c, p.Args, mapAppendValue)\n\t\t\t\t\t},\n\t\t\t\t}\n\t\t\t}\n\n\t\t\treturn fields\n\t\t}),\n\t})\n}\n\nfunc mapAppendKey(slice []interface{}, k, v types.Value) []interface{} {\n\treturn append(slice, MaybeGetScalar(k))\n}\n\nfunc mapAppendValue(slice []interface{}, k, v types.Value) []interface{} {\n\treturn append(slice, MaybeGetScalar(v))\n}\n\nfunc mapAppendEntry(slice []interface{}, k, v types.Value) []interface{} {\n\treturn append(slice, mapEntry{k, v})\n}\n\n// Refs are represented as structs:\n//\n// type Entry {\n//\t targetHash: String!\n//\t targetValue: !\n// }\nfunc (tc *TypeConverter) refToGraphQLObject(nomsType *types.Type) *graphql.Object {\n\treturn graphql.NewObject(graphql.ObjectConfig{\n\t\tName: tc.getTypeName(nomsType),\n\t\tFields: graphql.FieldsThunk(func() graphql.Fields {\n\t\t\tnomsTargetType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\t\t\ttargetType := tc.nomsTypeToGraphQLType(nomsTargetType, false)\n\n\t\t\treturn graphql.Fields{\n\t\t\t\ttargetHashKey: &graphql.Field{\n\t\t\t\t\tType: graphql.NewNonNull(graphql.String),\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\tr := p.Source.(types.Ref)\n\t\t\t\t\t\treturn MaybeGetScalar(types.String(r.TargetHash().String())), nil\n\t\t\t\t\t},\n\t\t\t\t},\n\n\t\t\t\ttargetValueKey: &graphql.Field{\n\t\t\t\t\tType: targetType,\n\t\t\t\t\tResolve: func(p graphql.ResolveParams) (interface{}, error) {\n\t\t\t\t\t\tr := p.Source.(types.Ref)\n\t\t\t\t\t\treturn MaybeGetScalar(r.TargetValue(p.Context.Value(vrwKey).(types.ValueReader))), nil\n\t\t\t\t\t},\n\t\t\t\t},\n\t\t\t}\n\t\t}),\n\t})\n}\n\nfunc MaybeGetScalar(v types.Value) interface{} {\n\tswitch v.(type) {\n\tcase types.Bool:\n\t\treturn bool(v.(types.Bool))\n\tcase types.Number:\n\t\treturn float64(v.(types.Number))\n\tcase types.String:\n\t\treturn string(v.(types.String))\n\tcase *types.Type, types.Blob:\n\t\t// TODO: https://github.com/attic-labs/noms/issues/3155\n\t\treturn v.Hash()\n\t}\n\n\treturn v\n}\n\n// InputToNomsValue converts a GraphQL input value (as used in arguments and\n// variables) to a Noms value.\nfunc InputToNomsValue(vrw types.ValueReadWriter, arg interface{}, nomsType *types.Type) types.Value {\n\tswitch nomsType.TargetKind() {\n\tcase types.BoolKind:\n\t\treturn types.Bool(arg.(bool))\n\tcase types.NumberKind:\n\t\tif i, ok := arg.(int); ok {\n\t\t\treturn types.Number(i)\n\t\t}\n\t\treturn types.Number(arg.(float64))\n\tcase types.StringKind:\n\t\treturn types.String(arg.(string))\n\tcase types.ListKind, types.SetKind:\n\t\telemType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\t\tsl := arg.([]interface{})\n\t\tvs := make(types.ValueSlice, len(sl))\n\t\tfor i, v := range sl {\n\t\t\tvs[i] = InputToNomsValue(vrw, v, elemType)\n\t\t}\n\t\tif nomsType.TargetKind() == types.ListKind {\n\t\t\treturn types.NewList(vrw, vs...)\n\t\t}\n\t\treturn types.NewSet(vrw, vs...)\n\tcase types.MapKind:\n\t\t// Maps are passed as [{key: K, value: V}, ...]\n\t\tkeyType := nomsType.Desc.(types.CompoundDesc).ElemTypes[0]\n\t\tvalType := nomsType.Desc.(types.CompoundDesc).ElemTypes[1]\n\t\tsl := arg.([]interface{})\n\t\tkvs := make(types.ValueSlice, 2*len(sl))\n\t\tfor i, v := range sl {\n\t\t\tv := v.(map[string]interface{})\n\t\t\tkvs[2*i] = InputToNomsValue(vrw, v[\"key\"], keyType)\n\t\t\tkvs[2*i+1] = InputToNomsValue(vrw, v[\"value\"], valType)\n\t\t}\n\t\treturn types.NewMap(vrw, kvs...)\n\tcase types.StructKind:\n\t\tdesc := nomsType.Desc.(types.StructDesc)\n\t\tdata := make(types.StructData, desc.Len())\n\t\tm := arg.(map[string]interface{})\n\t\tdesc.IterFields(func(name string, t *types.Type, optional bool) {\n\t\t\tif m[name] != nil || !optional {\n\t\t\t\tdata[name] = InputToNomsValue(vrw, m[name], t)\n\t\t\t}\n\t\t})\n\t\treturn types.NewStruct(desc.Name, data)\n\t}\n\tpanic(\"not yet implemented\")\n}\n\ntype mapIteratorForKeys struct {\n\tm types.Map\n\tkeys types.ValueSlice\n\tidx int\n}\n\nfunc (it *mapIteratorForKeys) Valid() bool {\n\treturn it.idx < len(it.keys)\n}\n\nfunc (it *mapIteratorForKeys) Entry() (k, v types.Value) {\n\tif it.idx >= len(it.keys) {\n\t\treturn\n\t}\n\tk = it.keys[it.idx]\n\tv = it.m.Get(k)\n\treturn\n}\n\nfunc (it *mapIteratorForKeys) Next() bool {\n\tit.idx++\n\treturn it.Valid()\n}\n\ntype setFirstIterator struct {\n\ts types.Set\n}\n\nfunc (it *setFirstIterator) Next() types.Value {\n\treturn it.s.First()\n}\n\nfunc (it *setFirstIterator) SkipTo(v types.Value) types.Value {\n\tpanic(\"not implemented\")\n}\n\ntype mapFirstIterator struct {\n\tm *types.Map\n}\n\nfunc (it *mapFirstIterator) Valid() bool {\n\treturn it.m != nil && !it.m.Empty()\n}\n\nfunc (it *mapFirstIterator) Entry() (types.Value, types.Value) {\n\tif it.m == nil {\n\t\treturn nil, nil\n\t}\n\tk, v := it.m.First()\n\tit.m = nil\n\treturn k, v\n}\n\nfunc (it *mapFirstIterator) Next() bool {\n\treturn false\n}\n" }, { "path": "go/nomdl/lexer.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nomdl\n\nimport (\n\t\"fmt\"\n\t\"text/scanner\"\n)\n\ntype lexer struct {\n\tscanner *scanner.Scanner\n\tpeekToken rune\n}\n\nfunc (lex *lexer) next() rune {\n\tif lex.peekToken != 0 {\n\t\ttok := lex.peekToken\n\t\tlex.peekToken = 0\n\t\treturn tok\n\t}\n\n\treturn lex.scanner.Scan()\n}\n\nfunc (lex *lexer) peek() rune {\n\tif lex.peekToken != 0 {\n\t\treturn lex.peekToken\n\t}\n\ttok := lex.scanner.Scan()\n\tlex.peekToken = tok\n\treturn tok\n}\n\nfunc (lex *lexer) pos() scanner.Position {\n\tif lex.peekToken != 0 {\n\t\tpanic(\"Cannot use pos after peek\")\n\t}\n\treturn lex.scanner.Pos()\n}\n\nfunc (lex *lexer) tokenText() string {\n\tif lex.peekToken != 0 {\n\t\tpanic(\"Cannot use tokenText after peek\")\n\t}\n\treturn lex.scanner.TokenText()\n}\n\nfunc (lex *lexer) eat(expected rune) rune {\n\ttok := lex.next()\n\tlex.check(expected, tok)\n\treturn tok\n}\n\nfunc (lex *lexer) eatIf(expected rune) bool {\n\ttok := lex.peek()\n\tif tok == expected {\n\t\tlex.next()\n\t\treturn true\n\t}\n\treturn false\n}\n\nfunc (lex *lexer) check(expected, actual rune) {\n\tif actual != expected {\n\t\tlex.tokenMismatch(expected, actual)\n\t}\n}\n\nfunc (lex *lexer) tokenMismatch(expected, actual rune) {\n\traiseSyntaxError(fmt.Sprintf(\"Unexpected token %s, expected %s\", scanner.TokenString(actual), scanner.TokenString(expected)), lex.pos())\n}\n\nfunc (lex *lexer) unexpectedToken(actual rune) {\n\traiseSyntaxError(fmt.Sprintf(\"Unexpected token %s\", scanner.TokenString(actual)), lex.pos())\n}\n\nfunc raiseSyntaxError(msg string, pos scanner.Position) {\n\tpanic(syntaxError{\n\t\tmsg: msg,\n\t\tpos: pos,\n\t})\n}\n\ntype syntaxError struct {\n\tmsg string\n\tpos scanner.Position\n}\n\nfunc (e syntaxError) Error() string {\n\treturn fmt.Sprintf(\"%s, %s\", e.msg, e.pos)\n}\n\nfunc catchSyntaxError(f func()) (errRes error) {\n\tdefer func() {\n\t\tif err := recover(); err != nil {\n\t\t\tif err, ok := err.(syntaxError); ok {\n\t\t\t\terrRes = err\n\t\t\t\treturn\n\t\t\t}\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\n\tf()\n\treturn\n}\n" }, { "path": "go/nomdl/parser.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nomdl\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"strconv\"\n\t\"strings\"\n\t\"text/scanner\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// Parser provides ways to parse Noms types.\ntype Parser struct {\n\tlex *lexer\n\tvrw types.ValueReadWriter\n}\n\n// ParserOptions allows passing options into New.\ntype ParserOptions struct {\n\t// Filename is the name of the file we are currently parsing.\n\tFilename string\n}\n\n// New creates a new Parser.\nfunc New(vrw types.ValueReadWriter, r io.Reader, options ParserOptions) *Parser {\n\ts := scanner.Scanner{}\n\ts.Init(r)\n\ts.Filename = options.Filename\n\ts.Mode = scanner.ScanIdents | scanner.ScanComments | scanner.SkipComments | scanner.ScanFloats | scanner.ScanStrings // | scanner.ScanRawStrings\n\ts.Error = func(s *scanner.Scanner, msg string) {}\n\tlex := lexer{scanner: &s}\n\treturn &Parser{&lex, vrw}\n}\n\n// ParseType parses a string describing a Noms type.\nfunc ParseType(code string) (typ *types.Type, err error) {\n\tp := New(nil, strings.NewReader(code), ParserOptions{})\n\terr = catchSyntaxError(func() {\n\t\ttyp = p.parseType()\n\t\tp.ensureAtEnd()\n\t})\n\treturn\n}\n\n// MustParseType parses a string describing a Noms type and panics if there\n// is an error.\nfunc MustParseType(code string) *types.Type {\n\ttyp, err := ParseType(code)\n\td.PanicIfError(err)\n\treturn typ\n}\n\n// Parse parses a string describing a Noms value.\nfunc Parse(vrw types.ValueReadWriter, code string) (v types.Value, err error) {\n\tp := New(vrw, strings.NewReader(code), ParserOptions{})\n\terr = catchSyntaxError(func() {\n\t\tv = p.parseValue()\n\t\tp.ensureAtEnd()\n\t})\n\treturn\n}\n\n// MustParse parses a string describing a Noms value and panics if there\n// is an error.\nfunc MustParse(vrw types.ValueReadWriter, code string) types.Value {\n\tv, err := Parse(vrw, code)\n\td.PanicIfError(err)\n\treturn v\n}\n\nfunc (p *Parser) ensureAtEnd() {\n\tp.lex.eat(scanner.EOF)\n}\n\n// Type :\n// TypeWithoutUnion (`|` TypeWithoutUnion)*\n//\n// TypeWithoutUnion :\n// `Blob`\n// `Bool`\n// `Number`\n// `String`\n// `Type`\n// `Value`\n// CycleType\n// ListType\n// MapType\n// RefType\n// SetType\n// StructType\n//\n// CycleType :\n// `Cycle` `<` StructName `>`\n//\n// ListType :\n// `List` `<` Type? `>`\n//\n// MapType :\n// `Map` `<` (Type `,` Type)? `>`\n//\n// RefType :\n// `Set` `<` Type `>`\n//\n// SetType :\n// `Set` `<` Type? `>`\n//\n// StructType :\n// `Struct` StructName? `{` StructTypeFields? `}`\n//\n// StructTypeFields :\n// StructTypeField\n// StructTypeField `,` StructTypeFields?\n//\n// StructName :\n// Ident\n//\n// StructTypeField :\n// StructFieldName `?`? `:` Type\n//\n// StructFieldName :\n// Ident\n\nfunc (p *Parser) parseType() *types.Type {\n\ttok := p.lex.eat(scanner.Ident)\n\treturn p.parseTypeWithToken(tok, p.lex.tokenText())\n}\n\nfunc (p *Parser) parseTypeWithToken(tok rune, tokenText string) *types.Type {\n\tt := p.parseSingleTypeWithToken(tok, tokenText)\n\ttok = p.lex.peek()\n\tif tok != '|' {\n\t\treturn t\n\t}\n\tunionTypes := []*types.Type{t}\n\n\tfor {\n\t\ttok = p.lex.peek()\n\t\tif tok == '|' {\n\t\t\tp.lex.next()\n\t\t} else {\n\t\t\tbreak\n\t\t}\n\t\tunionTypes = append(unionTypes, p.parseSingleType())\n\t}\n\treturn types.MakeUnionType(unionTypes...)\n}\n\nfunc (p *Parser) parseSingleType() *types.Type {\n\ttok := p.lex.eat(scanner.Ident)\n\treturn p.parseSingleTypeWithToken(tok, p.lex.tokenText())\n}\n\nfunc (p *Parser) parseSingleTypeWithToken(tok rune, tokenText string) *types.Type {\n\tswitch tokenText {\n\tcase \"Bool\":\n\t\treturn types.BoolType\n\tcase \"Blob\":\n\t\treturn types.BlobType\n\tcase \"Number\":\n\t\treturn types.NumberType\n\tcase \"String\":\n\t\treturn types.StringType\n\tcase \"Type\":\n\t\treturn types.TypeType\n\tcase \"Value\":\n\t\treturn types.ValueType\n\tcase \"Struct\":\n\t\treturn p.parseStructType()\n\tcase \"Map\":\n\t\treturn p.parseMapType()\n\tcase \"List\":\n\t\telemType := p.parseSingleElemType(true)\n\t\treturn types.MakeListType(elemType)\n\tcase \"Set\":\n\t\telemType := p.parseSingleElemType(true)\n\t\treturn types.MakeSetType(elemType)\n\tcase \"Ref\":\n\t\telemType := p.parseSingleElemType(false)\n\t\treturn types.MakeRefType(elemType)\n\tcase \"Cycle\":\n\t\treturn p.parseCycleType()\n\t}\n\n\tp.lex.unexpectedToken(tok)\n\treturn nil\n}\n\nfunc (p *Parser) parseStructType() *types.Type {\n\ttok := p.lex.next()\n\tname := \"\"\n\tif tok == scanner.Ident {\n\t\tname = p.lex.tokenText()\n\t\tp.lex.eat('{')\n\t} else {\n\t\tp.lex.check('{', tok)\n\t}\n\tfields := []types.StructField{}\n\n\tfor p.lex.peek() != '}' {\n\t\tp.lex.eat(scanner.Ident)\n\n\t\tfieldName := p.lex.tokenText()\n\t\toptional := p.lex.eatIf('?')\n\t\tp.lex.eat(':')\n\t\ttyp := p.parseType()\n\t\tfields = append(fields, types.StructField{\n\t\t\tName: fieldName,\n\t\t\tType: typ,\n\t\t\tOptional: optional,\n\t\t})\n\n\t\tif p.lex.eatIf(',') {\n\t\t\tcontinue\n\t\t}\n\n\t\tbreak\n\t}\n\tp.lex.eat('}')\n\treturn types.MakeStructType(name, fields...)\n}\n\nfunc (p *Parser) parseSingleElemType(allowEmptyUnion bool) *types.Type {\n\tp.lex.eat('<')\n\tif allowEmptyUnion && p.lex.eatIf('>') {\n\t\treturn types.MakeUnionType()\n\t}\n\telemType := p.parseType()\n\tp.lex.eat('>')\n\treturn elemType\n}\n\nfunc (p *Parser) parseCycleType() *types.Type {\n\tp.lex.eat('<')\n\tp.lex.eat(scanner.Ident)\n\tname := p.lex.tokenText()\n\tp.lex.eat('>')\n\treturn types.MakeCycleType(name)\n}\n\nfunc (p *Parser) parseMapType() *types.Type {\n\tvar keyType, valueType *types.Type\n\tp.lex.eat('<')\n\n\tif p.lex.eatIf('>') {\n\t\tkeyType = types.MakeUnionType()\n\t\tvalueType = keyType\n\t} else {\n\t\tkeyType = p.parseType()\n\t\tp.lex.eat(',')\n\t\tvalueType = p.parseType()\n\t\tp.lex.eat('>')\n\t}\n\treturn types.MakeMapType(keyType, valueType)\n}\n\n// Value :\n// Type\n// Bool\n// Number\n// String\n// List\n// Set\n// Map\n// Struct\n//\n// Bool :\n// `true`\n// `false`\n//\n// Number :\n// ...\n//\n// String :\n// ...\n//\n// List :\n// `[` Values? `]`\n//\n// Values :\n// Value\n// Value `,` Values?\n//\n// Set :\n// `set` `{` Values? `}`\n//\n// Map :\n// `map` `{` MapEntries? `}`\n//\n// MapEntries :\n// MapEntry\n// MapEntry `,` MapEntries?\n//\n// MapEntry :\n// Value `:` Value\n//\n// Struct :\n// `struct` StructName? `{` StructFields? `}`\n//\n// StructFields :\n// StructField\n// StructField `,` StructFields?\n//\n// StructField :\n// StructFieldName `:` Value\nfunc (p *Parser) parseValue() types.Value {\n\ttok := p.lex.next()\n\tswitch tok {\n\tcase scanner.Ident:\n\t\tswitch tokenText := p.lex.tokenText(); tokenText {\n\t\tcase \"true\":\n\t\t\treturn types.Bool(true)\n\t\tcase \"false\":\n\t\t\treturn types.Bool(false)\n\t\tcase \"set\":\n\t\t\treturn p.parseSet()\n\t\tcase \"map\":\n\t\t\treturn p.parseMap()\n\t\tcase \"struct\":\n\t\t\treturn p.parseStruct()\n\t\tcase \"blob\":\n\t\t\treturn p.parseBlob()\n\t\tdefault:\n\t\t\treturn p.parseTypeWithToken(tok, tokenText)\n\t\t}\n\tcase scanner.Float, scanner.Int:\n\t\tf := p.parseFloat()\n\t\treturn types.Number(f)\n\tcase '-':\n\t\tif !p.lex.eatIf(scanner.Float) {\n\t\t\tp.lex.eat(scanner.Int)\n\t\t}\n\t\tn := p.parseFloat()\n\t\treturn types.Number(-float64(n))\n\tcase '+':\n\t\tif !p.lex.eatIf(scanner.Float) {\n\t\t\tp.lex.eat(scanner.Int)\n\t\t}\n\t\treturn p.parseFloat()\n\tcase '[':\n\t\treturn p.parseList()\n\tcase scanner.String:\n\t\ts := p.lex.tokenText()\n\t\ts2, err := strconv.Unquote(s)\n\t\tif err != nil {\n\t\t\traiseSyntaxError(fmt.Sprintf(\"Invalid string %s\", s), p.lex.pos())\n\t\t}\n\t\treturn types.String(s2)\n\t}\n\n\tp.lex.unexpectedToken(tok)\n\n\tpanic(\"unreachable\")\n}\n\nfunc (p *Parser) parseFloat() types.Number {\n\ts := p.lex.tokenText()\n\tf, _ := strconv.ParseFloat(s, 64)\n\treturn types.Number(f)\n}\n\nfunc (p *Parser) parseList() types.List {\n\t// already swallowed '['\n\tle := types.NewList(p.vrw).Edit()\n\n\tfor p.lex.peek() != ']' {\n\t\tv := p.parseValue()\n\t\tle.Append(v)\n\n\t\tif p.lex.eatIf(',') {\n\t\t\tcontinue\n\t\t}\n\n\t\tbreak\n\t}\n\tp.lex.eat(']')\n\treturn le.List()\n}\n\nfunc (p *Parser) parseSet() types.Set {\n\t// already swallowed 'set'\n\tp.lex.eat('{')\n\tse := types.NewSet(p.vrw).Edit()\n\n\tfor p.lex.peek() != '}' {\n\t\tv := p.parseValue()\n\t\tse.Insert(v)\n\n\t\tif p.lex.eatIf(',') {\n\t\t\tcontinue\n\t\t}\n\n\t\tbreak\n\t}\n\tp.lex.eat('}')\n\treturn se.Set()\n}\n\nfunc (p *Parser) parseMap() types.Map {\n\t// already swallowed 'map'\n\tp.lex.eat('{')\n\tme := types.NewMap(p.vrw).Edit()\n\n\tfor p.lex.peek() != '}' {\n\t\tkey := p.parseValue()\n\n\t\tp.lex.eat(':')\n\t\tvalue := p.parseValue()\n\t\tme.Set(key, value)\n\n\t\tif p.lex.eatIf(',') {\n\t\t\tcontinue\n\t\t}\n\n\t\tbreak\n\t}\n\tp.lex.eat('}')\n\treturn me.Map()\n}\n\nfunc (p *Parser) blobString(s string) []byte {\n\traise := func() {\n\t\traiseSyntaxError(fmt.Sprintf(\"Invalid blob \\\"%s\\\"\", s), p.lex.pos())\n\t}\n\n\tif len(s)%2 != 0 {\n\t\traise()\n\t}\n\n\tvar buff bytes.Buffer\n\tfor i := 0; i < len(s); i += 2 {\n\t\tn, err := strconv.ParseUint(s[i:i+2], 16, 8)\n\t\tif err != nil {\n\t\t\traise()\n\t\t}\n\t\tbuff.WriteByte(uint8(n))\n\t}\n\treturn buff.Bytes()\n}\n\nfunc (p *Parser) parseBlob() types.Blob {\n\t// already swallowed 'blob'\n\tp.lex.eat('{')\n\tvar buff bytes.Buffer\n\n\tfor p.lex.peek() != '}' {\n\t\ttok := p.lex.next()\n\t\tswitch tok {\n\t\tcase scanner.Ident, scanner.Int:\n\t\t\ts := p.lex.tokenText()\n\t\t\tbuff.Write(p.blobString(s))\n\t\tdefault:\n\t\t\tp.lex.unexpectedToken(tok)\n\t\t}\n\n\t}\n\tp.lex.eat('}')\n\treturn types.NewBlob(p.vrw, bytes.NewReader(buff.Bytes()))\n}\n\nfunc (p *Parser) parseStruct() types.Struct {\n\t// already swallowed 'struct'\n\ttok := p.lex.next()\n\tname := \"\"\n\tif tok == scanner.Ident {\n\t\tname = p.lex.tokenText()\n\t\tp.lex.eat('{')\n\t} else {\n\t\tp.lex.check('{', tok)\n\t}\n\tdata := types.StructData{}\n\n\tfor p.lex.peek() != '}' {\n\t\tp.lex.eat(scanner.Ident)\n\n\t\tfieldName := p.lex.tokenText()\n\t\tp.lex.eat(':')\n\t\tv := p.parseValue()\n\t\tdata[fieldName] = v\n\n\t\tif p.lex.eatIf(',') {\n\t\t\tcontinue\n\t\t}\n\n\t\tbreak\n\t}\n\tp.lex.eat('}')\n\treturn types.NewStruct(name, data)\n}\n" }, { "path": "go/nomdl/parser_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage nomdl\n\nimport (\n\t\"bytes\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc newTestValueStore() *types.ValueStore {\n\tst := &chunks.TestStorage{}\n\treturn types.NewValueStore(st.NewView())\n}\n\nfunc assertParseType(t *testing.T, code string, expected *types.Type) {\n\tt.Run(code, func(t *testing.T) {\n\t\tactual, err := ParseType(code)\n\t\tassert.NoError(t, err)\n\t\tassert.True(t, expected.Equals(actual), \"Expected: %s, Actual: %s\", expected.Describe(), actual.Describe())\n\t})\n}\n\nfunc assertParse(t *testing.T, vrw types.ValueReadWriter, code string, expected types.Value) {\n\tt.Run(code, func(t *testing.T) {\n\t\tactual, err := Parse(vrw, code)\n\t\tif !assert.NoError(t, err) {\n\t\t\treturn\n\t\t}\n\t\tassert.True(t, expected.Equals(actual), \"Expected: %s, Actual: %s\", types.EncodedValue(expected), types.EncodedValue(actual))\n\t})\n}\n\nfunc assertParseError(t *testing.T, code, msg string) {\n\tt.Run(code, func(t *testing.T) {\n\t\tvrw := newTestValueStore()\n\t\tp := New(vrw, strings.NewReader(code), ParserOptions{\n\t\t\tFilename: \"example\",\n\t\t})\n\t\terr := catchSyntaxError(func() {\n\t\t\tp.parseValue()\n\t\t})\n\t\tif assert.Error(t, err) {\n\t\t\tassert.Equal(t, msg, err.Error())\n\t\t}\n\t})\n}\n\nfunc TestSimpleTypes(t *testing.T) {\n\tassertParseType(t, \"Blob\", types.BlobType)\n\tassertParseType(t, \"Bool\", types.BoolType)\n\tassertParseType(t, \"Number\", types.NumberType)\n\tassertParseType(t, \"String\", types.StringType)\n\tassertParseType(t, \"Value\", types.ValueType)\n\tassertParseType(t, \"Type\", types.TypeType)\n}\n\nfunc TestWhitespace(t *testing.T) {\n\tfor _, r := range \" \\t\\n\\r\" {\n\t\tassertParseType(t, string(r)+\"Blob\", types.BlobType)\n\t\tassertParseType(t, \"Blob\"+string(r), types.BlobType)\n\t}\n}\n\nfunc TestComments(t *testing.T) {\n\tassertParseType(t, \"/* */Blob\", types.BlobType)\n\tassertParseType(t, \"Blob/* */\", types.BlobType)\n\tassertParseType(t, \"Blob//\", types.BlobType)\n\tassertParseType(t, \"//\\nBlob\", types.BlobType)\n}\n\nfunc TestCompoundTypes(t *testing.T) {\n\tassertParseType(t, \"List<>\", types.MakeListType(types.MakeUnionType()))\n\tassertParseType(t, \"List\", types.MakeListType(types.BoolType))\n\tassertParseError(t, \"List\", `Unexpected token \",\", expected \">\", example:1:11`)\n\tassertParseError(t, \"List\", example:1:10`)\n\tassertParseError(t, \"List<\", `Unexpected token EOF, expected Ident, example:1:6`)\n\tassertParseError(t, \"List\", `Unexpected token EOF, expected \"<\", example:1:5`)\n\n\tassertParseType(t, \"Set<>\", types.MakeSetType(types.MakeUnionType()))\n\tassertParseType(t, \"Set\", types.MakeSetType(types.BoolType))\n\tassertParseError(t, \"Set\", `Unexpected token \",\", expected \">\", example:1:10`)\n\tassertParseError(t, \"Set\", example:1:9`)\n\tassertParseError(t, \"Set<\", `Unexpected token EOF, expected Ident, example:1:5`)\n\tassertParseError(t, \"Set\", `Unexpected token EOF, expected \"<\", example:1:4`)\n\n\tassertParseError(t, \"Ref<>\", `Unexpected token \">\", expected Ident, example:1:6`)\n\tassertParseType(t, \"Ref\", types.MakeRefType(types.BoolType))\n\tassertParseError(t, \"Ref\", `Unexpected token \",\", expected \">\", example:1:12`)\n\tassertParseError(t, \"Ref\", example:1:11`)\n\tassertParseError(t, \"Ref<\", `Unexpected token EOF, expected Ident, example:1:5`)\n\tassertParseError(t, \"Ref\", `Unexpected token EOF, expected \"<\", example:1:4`)\n\n\t// Cannot use Equals on unresolved cycles.\n\tct := MustParseType(\"Cycle\")\n\tassert.Equal(t, ct, types.MakeCycleType(\"Abc\"))\n\n\tassertParseError(t, \"Cycle<-123>\", `Unexpected token \"-\", expected Ident, example:1:8`)\n\tassertParseError(t, \"Cycle<12.3>\", `Unexpected token Float, expected Ident, example:1:11`)\n\tassertParseError(t, \"Cycle<>\", `Unexpected token \">\", expected Ident, example:1:8`)\n\tassertParseError(t, \"Cycle<\", `Unexpected token EOF, expected Ident, example:1:7`)\n\tassertParseError(t, \"Cycle\", `Unexpected token EOF, expected \"<\", example:1:6`)\n\n\tassertParseType(t, \"Map<>\", types.MakeMapType(types.MakeUnionType(), types.MakeUnionType()))\n\tassertParseType(t, \"Map\", types.MakeMapType(types.BoolType, types.StringType))\n\tassertParseError(t, \"Map\", `Unexpected token \">\", expected Ident, example:1:11`)\n\tassertParseError(t, \"Map<,Bool>\", `Unexpected token \",\", expected Ident, example:1:6`)\n\tassertParseError(t, \"Map<,>\", `Unexpected token \",\", expected Ident, example:1:6`)\n\tassertParseError(t, \"Map\", example:1:14`)\n\tassertParseError(t, \"Map\", types.MakeListType(types.MakeUnionType(types.BoolType, types.NumberType)))\n\tassertParseType(t, \"Map\",\n\t\ttypes.MakeMapType(\n\t\t\ttypes.MakeUnionType(types.BoolType, types.NumberType),\n\t\t\ttypes.MakeUnionType(types.BoolType, types.NumberType),\n\t\t),\n\t)\n\tassertParseType(t, `Struct S {\n x: Number | Bool\n }`, types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"x\": types.MakeUnionType(types.BoolType, types.NumberType),\n\t}))\n\tassertParseType(t, `Struct S {\n x: Number | Bool,\n y: String\n }`, types.MakeStructTypeFromFields(\"S\", types.FieldMap{\n\t\t\"x\": types.MakeUnionType(types.BoolType, types.NumberType),\n\t\t\"y\": types.StringType,\n\t}))\n\n\tassertParseError(t, \"Bool |\", \"Unexpected token EOF, expected Ident, example:1:7\")\n\tassertParseError(t, \"Bool | Number |\", \"Unexpected token EOF, expected Ident, example:1:16\")\n\tassertParseError(t, \"Bool | | \", `Unexpected token \"|\", expected Ident, example:1:9`)\n\tassertParseError(t, \"\", `Unexpected token EOF, example:1:1`)\n}\n\nfunc TestValuePrimitives(t *testing.T) {\n\tvs := newTestValueStore()\n\tassertParse(t, vs, \"Number\", types.NumberType)\n\tassertParse(t, vs, \"Number | String\", types.MakeUnionType(types.NumberType, types.StringType))\n\n\tassertParse(t, vs, \"true\", types.Bool(true))\n\tassertParse(t, vs, \"false\", types.Bool(false))\n\n\tassertParse(t, vs, \"0\", types.Number(0))\n\tassertParse(t, vs, \"1\", types.Number(1))\n\tassertParse(t, vs, \"1.1\", types.Number(1.1))\n\tassertParse(t, vs, \"1.1e1\", types.Number(1.1e1))\n\tassertParse(t, vs, \"1e1\", types.Number(1e1))\n\tassertParse(t, vs, \"1e-1\", types.Number(1e-1))\n\tassertParse(t, vs, \"1e+1\", types.Number(1e+1))\n\n\tassertParse(t, vs, \"+0\", types.Number(0))\n\tassertParse(t, vs, \"+1\", types.Number(1))\n\tassertParse(t, vs, \"+1.1\", types.Number(1.1))\n\tassertParse(t, vs, \"+1.1e1\", types.Number(1.1e1))\n\tassertParse(t, vs, \"+1e1\", types.Number(1e1))\n\tassertParse(t, vs, \"+1e-1\", types.Number(1e-1))\n\tassertParse(t, vs, \"+1e+1\", types.Number(1e+1))\n\n\tassertParse(t, vs, \"-0\", types.Number(-0))\n\tassertParse(t, vs, \"-1\", types.Number(-1))\n\tassertParse(t, vs, \"-1.1\", types.Number(-1.1))\n\tassertParse(t, vs, \"-1.1e1\", types.Number(-1.1e1))\n\tassertParse(t, vs, \"-1e1\", types.Number(-1e1))\n\tassertParse(t, vs, \"-1e-1\", types.Number(-1e-1))\n\tassertParse(t, vs, \"-1e+1\", types.Number(-1e+1))\n\n\tassertParse(t, vs, `\"a\"`, types.String(\"a\"))\n\tassertParse(t, vs, `\"\"`, types.String(\"\"))\n\tassertParse(t, vs, `\"\\\"\"`, types.String(\"\\\"\"))\n\tassertParseError(t, `\"\\\"`, \"Invalid string \\\"\\\\\\\", example:1:4\")\n\tassertParseError(t, `\"abc`, \"Invalid string \\\"abc, example:1:5\")\n\tassertParseError(t, `\"`, \"Invalid string \\\", example:1:2\")\n\tassertParseError(t, `\"\n\"`, \"Invalid string \\\"\\n, example:2:1\")\n\n\tassertParseError(t, \"`\", \"Unexpected token \\\"`\\\", example:1:2\")\n}\n\nfunc TestValueList(t *testing.T) {\n\tvs := newTestValueStore()\n\tassertParse(t, vs, \"[]\", types.NewList(vs))\n\n\tassertParse(t, vs, \"[42]\", types.NewList(vs, types.Number(42)))\n\tassertParse(t, vs, \"[42,]\", types.NewList(vs, types.Number(42)))\n\n\tassertParseError(t, \"[\", \"Unexpected token EOF, example:1:2\")\n\tassertParseError(t, \"[,\", \"Unexpected token \\\",\\\", example:1:3\")\n\tassertParseError(t, \"[42\", \"Unexpected token EOF, expected \\\"]\\\", example:1:4\")\n\tassertParseError(t, \"[42,\", \"Unexpected token EOF, example:1:5\")\n\tassertParseError(t, \"[,]\", \"Unexpected token \\\",\\\", example:1:3\")\n\n\tassertParse(t, vs, `[42,\n\t Bool,\n\t ]`, types.NewList(vs, types.Number(42), types.BoolType))\n\tassertParse(t, vs, `[42,\n\t Bool\n\t\t]`, types.NewList(vs, types.Number(42), types.BoolType))\n}\n\nfunc TestEmptyValuesInEditors(t *testing.T) {\n\tvs := newTestValueStore()\n\n\tassertParse(t, vs, \"[[]]\", types.NewList(vs, types.NewList(vs)))\n\tassertParse(t, vs, \"[set {}]\", types.NewList(vs, types.NewSet(vs)))\n\tassertParse(t, vs, \"[map {}]\", types.NewList(vs, types.NewMap(vs)))\n\n\tassertParse(t, vs, \"set {[]}\", types.NewSet(vs, types.NewList(vs)))\n\tassertParse(t, vs, \"set {set {}}\", types.NewSet(vs, types.NewSet(vs)))\n\tassertParse(t, vs, \"set {map {}}\", types.NewSet(vs, types.NewMap(vs)))\n\n\tassertParse(t, vs, \"map {map {}: map {}}\", types.NewMap(vs, types.NewMap(vs), types.NewMap(vs)))\n\tassertParse(t, vs, \"map {[]: []}\", types.NewMap(vs, types.NewList(vs), types.NewList(vs)))\n\tassertParse(t, vs, \"map {set {}: set {}}\", types.NewMap(vs, types.NewSet(vs), types.NewSet(vs)))\n}\n\nfunc TestValueSet(t *testing.T) {\n\tvs := newTestValueStore()\n\tassertParse(t, vs, \"set {}\", types.NewSet(vs))\n\n\tassertParse(t, vs, \"set {42}\", types.NewSet(vs, types.Number(42)))\n\tassertParse(t, vs, \"set {42,}\", types.NewSet(vs, types.Number(42)))\n\n\tassertParseError(t, \"set\", \"Unexpected token EOF, expected \\\"{\\\", example:1:4\")\n\tassertParseError(t, \"set {\", \"Unexpected token EOF, example:1:6\")\n\tassertParseError(t, \"set {,\", \"Unexpected token \\\",\\\", example:1:7\")\n\tassertParseError(t, \"set {42\", \"Unexpected token EOF, expected \\\"}\\\", example:1:8\")\n\tassertParseError(t, \"set {42,\", \"Unexpected token EOF, example:1:9\")\n\tassertParseError(t, \"set {,}\", \"Unexpected token \\\",\\\", example:1:7\")\n\n\tassertParse(t, vs, `set {42,\n Bool,\n }`, types.NewSet(vs, types.Number(42), types.BoolType))\n\tassertParse(t, vs, `set {42,\n Bool\n }`, types.NewSet(vs, types.Number(42), types.BoolType))\n}\n\nfunc TestValueMap(t *testing.T) {\n\tvs := newTestValueStore()\n\tassertParse(t, vs, \"map {}\", types.NewMap(vs))\n\n\tassertParse(t, vs, \"map {42: true}\", types.NewMap(vs, types.Number(42), types.Bool(true)))\n\tassertParse(t, vs, \"map {42: true,}\", types.NewMap(vs, types.Number(42), types.Bool(true)))\n\n\tassertParseError(t, \"map\", \"Unexpected token EOF, expected \\\"{\\\", example:1:4\")\n\tassertParseError(t, \"map {\", \"Unexpected token EOF, example:1:6\")\n\tassertParseError(t, \"map {,\", \"Unexpected token \\\",\\\", example:1:7\")\n\tassertParseError(t, \"map {42\", \"Unexpected token EOF, expected \\\":\\\", example:1:8\")\n\tassertParseError(t, \"map {42,\", \"Unexpected token \\\",\\\", expected \\\":\\\", example:1:9\")\n\tassertParseError(t, \"map {42:\", \"Unexpected token EOF, example:1:9\")\n\tassertParseError(t, \"map {42: true\", \"Unexpected token EOF, expected \\\"}\\\", example:1:14\")\n\tassertParseError(t, \"map {,}\", \"Unexpected token \\\",\\\", example:1:7\")\n\n\tassertParse(t, vs, `map {42:\n Bool,\n }`, types.NewMap(vs, types.Number(42), types.BoolType))\n\tassertParse(t, vs, `map {42:\n Bool\n }`, types.NewMap(vs, types.Number(42), types.BoolType))\n}\n\nfunc TestValueType(t *testing.T) {\n\tvs := newTestValueStore()\n\tassertParse(t, vs, \"Bool\", types.BoolType)\n\tassertParse(t, vs, \"Number\", types.NumberType)\n\tassertParse(t, vs, \"String\", types.StringType)\n}\n\nfunc TestValueStruct(t *testing.T) {\n\tvs := newTestValueStore()\n\tassertParse(t, vs, \"struct {}\", types.NewStruct(\"\", nil))\n\tassertParseError(t, \"struct\", \"Unexpected token EOF, expected \\\"{\\\", example:1:7\")\n\tassertParseError(t, \"struct {\", \"Unexpected token EOF, expected Ident, example:1:9\")\n\n\tassertParse(t, vs, \"struct name {}\", types.NewStruct(\"name\", nil))\n\tassertParseError(t, \"struct name\", \"Unexpected token EOF, expected \\\"{\\\", example:1:12\")\n\tassertParseError(t, \"struct name {\", \"Unexpected token EOF, expected Ident, example:1:14\")\n\n\tassertParse(t, vs, \"struct name {a: 42}\", types.NewStruct(\"name\", types.StructData{\"a\": types.Number(42)}))\n\tassertParse(t, vs, \"struct name {a: 42,}\", types.NewStruct(\"name\", types.StructData{\"a\": types.Number(42)}))\n\tassertParseError(t, \"struct name {a\", \"Unexpected token EOF, expected \\\":\\\", example:1:15\")\n\tassertParseError(t, \"struct name {a: \", \"Unexpected token EOF, example:1:17\")\n\tassertParseError(t, \"struct name {a,\", \"Unexpected token \\\",\\\", expected \\\":\\\", example:1:16\")\n\tassertParseError(t, \"struct name {a}\", \"Unexpected token \\\"}\\\", expected \\\":\\\", example:1:16\")\n\tassertParseError(t, \"struct name {a: 42\", \"Unexpected token EOF, expected \\\"}\\\", example:1:19\")\n\tassertParseError(t, \"struct name {a: 42,\", \"Unexpected token EOF, expected Ident, example:1:20\")\n\tassertParseError(t, \"struct name {a:}\", \"Unexpected token \\\"}\\\", example:1:17\")\n\n\tassertParse(t, vs, \"struct name {b: 42, a: true}\", types.NewStruct(\"name\", types.StructData{\"b\": types.Number(42), \"a\": types.Bool(true)}))\n\tassertParse(t, vs, `struct name {\n b: 42,\n a: true,\n }`, types.NewStruct(\"name\", types.StructData{\"b\": types.Number(42), \"a\": types.Bool(true)}))\n\n\tassertParse(t, vs, \"struct name {a: Struct {}}\", types.NewStruct(\"name\", types.StructData{\"a\": types.MakeStructType(\"\")}))\n}\n\nfunc TestValueBlob(t *testing.T) {\n\tvs := newTestValueStore()\n\n\ttest := func(code string, bs ...byte) {\n\t\tassertParse(t, vs, code, types.NewBlob(vs, bytes.NewBuffer(bs)))\n\t}\n\n\ttest(\"blob {}\")\n\ttest(\"blob {// comment\\n}\")\n\ttest(\"blob {10}\", 0x10)\n\ttest(\"blob {10/* comment */}\", 0x10)\n\ttest(\"blob {0000ff}\", 0, 0, 0xff)\n\ttest(\"blob {00 00 ff}\", 0, 0, 0xff)\n\ttest(\"blob { 00\\n00\\nff }\", 0, 0, 0xff)\n\ttest(\"blob { ffffffff ffffffff ffffffff ffffffff}\",\n\t\t0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,\n\t\t0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,\n\t)\n\ttest(\"blob { ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff}\",\n\t\t0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,\n\t\t0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,\n\t\t0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,\n\t\t0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,\n\t)\n\n\tassertParseError(t, \"blob\", \"Unexpected token EOF, expected \\\"{\\\", example:1:5\")\n\tassertParseError(t, \"blob {\", \"Unexpected token EOF, example:1:7\")\n\tassertParseError(t, \"blob { 00\", \"Unexpected token EOF, example:1:10\")\n\tassertParseError(t, \"blob {hh}\", \"Invalid blob \\\"hh\\\", example:1:9\")\n\tassertParseError(t, \"blob {0}\", \"Invalid blob \\\"0\\\", example:1:8\")\n\tassertParseError(t, \"blob {00 0}\", \"Invalid blob \\\"0\\\", example:1:11\")\n\tassertParseError(t, \"blob {ff 0 0}\", \"Invalid blob \\\"0\\\", example:1:11\")\n\n}\n\nfunc TestRoundTrips(t *testing.T) {\n\tvs := newTestValueStore()\n\n\ttest := func(v types.Value) {\n\t\tcode := types.EncodedValue(v)\n\t\tassertParse(t, vs, code, v)\n\t}\n\n\ttest(types.Number(0))\n\ttest(types.Number(42))\n\ttest(types.Number(-0))\n\ttest(types.Number(-42))\n\ttest(types.Number(0.05))\n\ttest(types.Number(-0.05))\n\ttest(types.Number(1e50))\n\ttest(types.Number(-1e50))\n\n\ttest(types.Bool(true))\n\ttest(types.Bool(false))\n\n\ttest(types.String(\"\"))\n\ttest(types.String(\"a\"))\n\ttest(types.String(\"\\\"\"))\n\ttest(types.String(\"'\"))\n\ttest(types.String(\"`\"))\n\n\ttest(types.NewEmptyBlob(vs))\n\ttest(types.NewBlob(vs, bytes.NewBufferString(\"abc\")))\n\n\ttest(types.NewList(vs))\n\ttest(types.NewList(vs, types.Number(42), types.Bool(true), types.String(\"abc\")))\n\n\ttest(types.NewSet(vs))\n\ttest(types.NewSet(vs, types.Number(42), types.Bool(true), types.String(\"abc\")))\n\n\ttest(types.NewMap(vs))\n\ttest(types.NewMap(vs, types.Number(42), types.Bool(true), types.String(\"abc\"), types.NewMap(vs)))\n\n\ttest(types.NewStruct(\"\", nil))\n\ttest(types.NewStruct(\"Number\", nil))\n\ttest(types.NewStruct(\"Number\", types.StructData{\n\t\t\"Number\": types.NumberType,\n\t}))\n\n\ttest(types.MakeStructType(\"S\", types.StructField{\n\t\tName: \"cycle\", Type: types.MakeCycleType(\"S\"), Optional: true,\n\t}))\n}\n" }, { "path": "go/perf/hash-perf-rig/README.md", "content": "This is a performance test rig for the two main types of hashing we do in NOMS - buzhash and sha1. There's also support for sha256, sha512, and blake2b hash functions for comparison.\n\nAs of May 9, these are the numbers I get on a macbook pro 3.1 GHz Intel Core i7.\n\n- no hashing : 3500 MB/s\n- sha1 only : 470 MB/s\n- sha256 only : 185 MB/s\n- sha512 only : 299 MB/s\n- blake2b only : 604 MB/s\n- bh only : 139 MB/s\n- sha1 and bh : 110 MB/s\n- sha256 and bh : 80 MB/s\n- sha512 and bh : 96 MB/s\n- blake2b and bh: 115 MB/s\n\nI think that in the no hashing case there is some compiler optimization going\non because I note that if all I do is add a loop that reads out bytes one by\none from the slice, it drops to 1000MB/s.\n\nOne outcome of this is that there's no sense going to sha256 - we should just\njump straight to sha512." }, { "path": "go/perf/hash-perf-rig/main.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"crypto/sha1\"\n\t\"crypto/sha256\"\n\t\"crypto/sha512\"\n\t\"fmt\"\n\t\"hash\"\n\t\"io\"\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/attic-labs/kingpin\"\n\t\"github.com/codahale/blake2\"\n\thumanize \"github.com/dustin/go-humanize\"\n\t\"github.com/kch42/buzhash\"\n)\n\nfunc main() {\n\tuseSHA := kingpin.Flag(\"use-sha\", \"=no hashing, 1=sha1, 256=sha256, 512=sha512, blake=blake2b\").String()\n\tuseBH := kingpin.Flag(\"use-bh\", \"whether we buzhash the bytes\").Bool()\n\tbigFile := kingpin.Arg(\"bigfile\", \"input file to chunk\").Required().String()\n\n\tkingpin.Parse()\n\n\tbh := buzhash.NewBuzHash(64 * 8)\n\tf, _ := os.Open(*bigFile)\n\tdefer f.Close()\n\tt0 := time.Now()\n\tbuf := make([]byte, 4*1024)\n\tl := uint64(0)\n\n\tvar h hash.Hash\n\tif *useSHA == \"1\" {\n\t\th = sha1.New()\n\t} else if *useSHA == \"256\" {\n\t\th = sha256.New()\n\t} else if *useSHA == \"512\" {\n\t\th = sha512.New()\n\t} else if *useSHA == \"blake\" {\n\t\th = blake2.NewBlake2B()\n\t}\n\n\tfor {\n\t\tn, err := f.Read(buf)\n\t\tl += uint64(n)\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\t\ts := buf[:n]\n\t\tif h != nil {\n\t\t\th.Write(s)\n\t\t}\n\t\tif *useBH {\n\t\t\tbh.Write(s)\n\t\t}\n\t}\n\n\tt1 := time.Now()\n\td := t1.Sub(t0)\n\tfmt.Printf(\"Read %s in %s (%s/s)\\n\", humanize.Bytes(l), d, humanize.Bytes(uint64(float64(l)/d.Seconds())))\n\tdigest := []byte{}\n\tif h != nil {\n\t\tfmt.Printf(\"%x\\n\", h.Sum(digest))\n\t}\n}\n" }, { "path": "go/perf/suite/suite.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package suite implements a performance test suite for Noms, intended for\n// measuring and reporting long running tests.\n//\n// Usage is similar to testify's suite:\n// 1. Define a test suite struct which inherits from suite.PerfSuite.\n// 2. Define methods on that struct that start with the word \"Test\", optionally\n// followed by digits, then followed a non-empty capitalized string.\n// 3. Call suite.Run with an instance of that struct.\n// 4. Run go test with the -perf flag.\n//\n// Flags:\n// -perf.mem Backs the database by a memory store, instead of nbs.\n// -perf.prefix Gives the dataset IDs for test results a prefix.\n// -perf.repeat Sets how many times tests are repeated (\"reps\").\n// -perf.run Only run tests that match a regex (case insensitive).\n// -perf.testdata Sets a custom path to the Noms testdata directory.\n//\n// PerfSuite also supports testify/suite style Setup/TearDown methods:\n// Setup/TearDownSuite is called exactly once.\n// Setup/TearDownRep is called for each repetition of the test runs, i.e. -perf.repeat times.\n// Setup/TearDownTest is called for every test.\n//\n// Test results are written to Noms, along with a dump of the environment they were recorded in.\n//\n// Test names are derived from that \"non-empty capitalized string\": \"Test\" is omitted because it's\n// redundant, and leading digits are omitted to allow for manual test ordering. For example:\n//\n// > cat ./samples/go/csv/csv-import/perf_test.go\n// type perfSuite {\n// suite.PerfSuite\n// }\n//\n// func (s *perfSuite) TestFoo() { ... }\n// func (s *perfSuite) TestZoo() { ... }\n// func (s *perfSuite) Test01Qux() { ... }\n// func (s *perfSuite) Test02Bar() { ... }\n//\n// func TestPerf(t *testing.T) {\n// suite.Run(\"csv-import\", t, &perfSuite{})\n// }\n//\n// > noms serve &\n// > go test -v ./samples/go/csv/... -perf http://localhost:8000 -perf.repeat 3\n// (perf) RUN(1/3) Test01Qux (recorded as \"Qux\")\n// (perf) PASS: Test01Qux (5s, paused 15s, total 20s)\n// (perf) RUN(1/3) Test02Bar (recorded as \"Bar\")\n// (perf) PASS: Test02Bar (15s, paused 2s, total 17s)\n// (perf) RUN(1/3) TestFoo (recorded as \"Foo\")\n// (perf) PASS: TestFoo (10s, paused 1s, total 11s)\n// (perf) RUN(1/3) TestZoo (recorded as \"Zoo\")\n// (perf) PASS: TestZoo (1s, paused 42s, total 43s)\n// ...\n//\n// > noms show http://localhost:8000::csv-import\n// {\n// environment: ...\n// tests: [{\n// \"Bar\": {elapsed: 15s, paused: 2s, total: 17s},\n// \"Foo\": {elapsed: 10s, paused: 1s, total: 11s},\n// \"Qux\": {elapsed: 5s, paused: 15s, total: 20s},\n// \"Zoo\": {elapsed: 1s, paused: 42s, total: 43s},\n// }, ...]\n// ...\n// }\npackage suite\n\nimport (\n\t\"bytes\"\n\t\"flag\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"os/exec\"\n\t\"path\"\n\t\"path/filepath\"\n\t\"reflect\"\n\t\"regexp\"\n\t\"strings\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/nbs\"\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/shirou/gopsutil/cpu\"\n\t\"github.com/shirou/gopsutil/disk\"\n\t\"github.com/shirou/gopsutil/host\"\n\t\"github.com/shirou/gopsutil/mem\"\n\t\"github.com/stretchr/testify/assert\"\n\ttestifySuite \"github.com/stretchr/testify/suite\"\n)\n\nvar (\n\tperfFlag = flag.String(\"perf\", \"\", \"The database to write perf tests to. If this isn't specified, perf tests are skipped. If you want a dry run, use \\\"mem\\\" as a database\")\n\tperfMemFlag = flag.Bool(\"perf.mem\", false, \"Back the test database by a memory store, not nbs. This will affect test timing, but it's provided in case you're low on disk space\")\n\tperfPrefixFlag = flag.String(\"perf.prefix\", \"\", `Prefix for the dataset IDs where results are written. For example, a prefix of \"foo/\" will write test datasets like \"foo/csv-import\" instead of just \"csv-import\"`)\n\tperfRepeatFlag = flag.Int(\"perf.repeat\", 1, \"The number of times to repeat each perf test\")\n\tperfRunFlag = flag.String(\"perf.run\", \"\", \"Only run perf tests that match a regular expression\")\n\tperfTestdataFlag = flag.String(\"perf.testdata\", \"\", \"Path to the noms testdata directory. By default this is ../testdata relative to the noms directory\")\n\ttestNamePattern = regexp.MustCompile(\"^Test[0-9]*([A-Z].*$)\")\n)\n\n// PerfSuite is the core of the perf testing suite. See package documentation for details.\ntype PerfSuite struct {\n\t// T is the testing.T instance set when the suite is passed into Run.\n\tT *testing.T\n\n\t// W is the io.Writer to write test output, which only outputs if the verbose flag is set.\n\tW io.Writer\n\n\t// AtticLabs is the path to the attic-labs directory (e.g. /path/to/go/src/github.com/attic-labs).\n\tAtticLabs string\n\n\t// Testdata is the path to the testdata directory - typically /path/to/go/src/github.com/attic-labs, but it can be overridden with the -perf.testdata flag.\n\tTestdata string\n\n\t// Database is a Noms database that tests can use for reading and writing. State is persisted across a single Run of a suite.\n\tDatabase datas.Database\n\n\t// DatabaseSpec is the Noms spec of Database (typically a localhost URL).\n\tDatabaseSpec string\n\n\ttempFiles []*os.File\n\ttempDirs []string\n\tpaused time.Duration\n\tdatasetID string\n}\n\n// SetupRepSuite has a SetupRep method, which runs every repetition of the test, i.e. -perf.repeat times in total.\ntype SetupRepSuite interface {\n\tSetupRep()\n}\n\n// TearDownRepSuite has a TearDownRep method, which runs every repetition of the test, i.e. -perf.repeat times in total.\ntype TearDownRepSuite interface {\n\tTearDownRep()\n}\n\ntype perfSuiteT interface {\n\tSuite() *PerfSuite\n}\n\ntype environment struct {\n\tDiskUsages map[string]disk.UsageStat\n\tCpus map[int]cpu.InfoStat\n\tMem mem.VirtualMemoryStat\n\tHost host.InfoStat\n\tPartitions map[string]disk.PartitionStat\n}\n\ntype timeInfo struct {\n\telapsed, paused, total time.Duration\n}\n\ntype testRep map[string]timeInfo\n\ntype nopWriter struct{}\n\nfunc (r nopWriter) Write(p []byte) (int, error) {\n\treturn len(p), nil\n}\n\n// Run runs suiteT and writes results to dataset datasetID in the database given by the -perf command line flag.\nfunc Run(datasetID string, t *testing.T, suiteT perfSuiteT) {\n\tassert := assert.New(t)\n\n\tif !assert.NotEqual(\"\", datasetID) {\n\t\treturn\n\t}\n\n\t// Piggy-back off the go test -v flag.\n\tverboseFlag := flag.Lookup(\"test.v\")\n\tassert.NotNil(verboseFlag)\n\tverbose := verboseFlag.Value.(flag.Getter).Get().(bool)\n\n\tif *perfFlag == \"\" {\n\t\tif verbose {\n\t\t\tfmt.Printf(\"(perf) Skipping %s, -perf flag not set\\n\", datasetID)\n\t\t}\n\t\treturn\n\t}\n\n\tsuite := suiteT.Suite()\n\tsuite.T = t\n\tif verbose {\n\t\tsuite.W = os.Stdout\n\t} else {\n\t\tsuite.W = nopWriter{}\n\t}\n\n\tgopath := os.Getenv(\"GOPATH\")\n\tif !assert.NotEmpty(gopath) {\n\t\treturn\n\t}\n\tsuite.AtticLabs = path.Join(gopath, \"src\", \"github.com\", \"attic-labs\")\n\tsuite.Testdata = *perfTestdataFlag\n\tif suite.Testdata == \"\" {\n\t\tsuite.Testdata = path.Join(suite.AtticLabs, \"testdata\")\n\t}\n\n\t// Clean up temporary directories/files last.\n\tdefer func() {\n\t\tfor _, f := range suite.tempFiles {\n\t\t\tos.Remove(f.Name())\n\t\t}\n\t\tfor _, d := range suite.tempDirs {\n\t\t\tos.RemoveAll(d)\n\t\t}\n\t}()\n\n\tsuite.datasetID = datasetID\n\n\t// This is the database the perf test results are written to.\n\tsp, err := spec.ForDatabase(*perfFlag)\n\tif !assert.NoError(err) {\n\t\treturn\n\t}\n\tdefer sp.Close()\n\n\t// List of test runs, each a map of test name => timing info.\n\ttestReps := make([]testRep, *perfRepeatFlag)\n\n\t// Note: the default value of perfRunFlag is \"\", which is actually a valid\n\t// regular expression that matches everything.\n\tperfRunRe, err := regexp.Compile(\"(?i)\" + *perfRunFlag)\n\tif !assert.NoError(err, `Invalid regular expression \"%s\"`, *perfRunFlag) {\n\t\treturn\n\t}\n\n\tdefer func() {\n\t\tdb := sp.GetDatabase()\n\n\t\treps := make([]types.Value, *perfRepeatFlag)\n\t\tfor i, rep := range testReps {\n\t\t\ttimesSlice := types.ValueSlice{}\n\t\t\tfor name, info := range rep {\n\t\t\t\ttimesSlice = append(timesSlice, types.String(name), types.NewStruct(\"\", types.StructData{\n\t\t\t\t\t\"elapsed\": types.Number(info.elapsed.Nanoseconds()),\n\t\t\t\t\t\"paused\": types.Number(info.paused.Nanoseconds()),\n\t\t\t\t\t\"total\": types.Number(info.total.Nanoseconds()),\n\t\t\t\t}))\n\t\t\t}\n\t\t\treps[i] = types.NewMap(db, timesSlice...)\n\t\t}\n\n\t\trecord := types.NewStruct(\"\", map[string]types.Value{\n\t\t\t\"environment\": suite.getEnvironment(db),\n\t\t\t\"nomsRevision\": types.String(suite.getGitHead(path.Join(suite.AtticLabs, \"noms\"))),\n\t\t\t\"testdataRevision\": types.String(suite.getGitHead(suite.Testdata)),\n\t\t\t\"reps\": types.NewList(db, reps...),\n\t\t})\n\n\t\tds := db.GetDataset(*perfPrefixFlag + datasetID)\n\t\t_, err := db.CommitValue(ds, record)\n\t\tassert.NoError(err)\n\t}()\n\n\tif t, ok := suiteT.(testifySuite.SetupAllSuite); ok {\n\t\tt.SetupSuite()\n\t}\n\n\tfor repIdx := 0; repIdx < *perfRepeatFlag; repIdx++ {\n\t\ttestReps[repIdx] = testRep{}\n\n\t\tserverHost, stopServerFn := suite.StartRemoteDatabase()\n\t\tsuite.DatabaseSpec = serverHost\n\t\tsuite.Database = datas.NewDatabase(datas.NewHTTPChunkStore(serverHost, \"\"))\n\t\tdefer suite.Database.Close()\n\n\t\tif t, ok := suiteT.(SetupRepSuite); ok {\n\t\t\tt.SetupRep()\n\t\t}\n\n\t\tfor t, mIdx := reflect.TypeOf(suiteT), 0; mIdx < t.NumMethod(); mIdx++ {\n\t\t\tm := t.Method(mIdx)\n\n\t\t\tparts := testNamePattern.FindStringSubmatch(m.Name)\n\t\t\tif parts == nil {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\trecordName := parts[1]\n\t\t\tif !perfRunRe.MatchString(recordName) && !perfRunRe.MatchString(m.Name) {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif _, ok := testReps[repIdx][recordName]; ok {\n\t\t\t\tassert.Fail(`Multiple tests are named \"%s\"`, recordName)\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif verbose {\n\t\t\t\tfmt.Printf(\"(perf) RUN(%d/%d) %s (as \\\"%s\\\")\\n\", repIdx+1, *perfRepeatFlag, m.Name, recordName)\n\t\t\t}\n\n\t\t\tif t, ok := suiteT.(testifySuite.SetupTestSuite); ok {\n\t\t\t\tt.SetupTest()\n\t\t\t}\n\n\t\t\tstart := time.Now()\n\t\t\tsuite.paused = 0\n\n\t\t\terr := callSafe(m.Name, m.Func, suiteT)\n\n\t\t\ttotal := time.Since(start)\n\t\t\telapsed := total - suite.paused\n\n\t\t\tif verbose && err == nil {\n\t\t\t\tfmt.Printf(\"(perf) PASS: %s (%s, paused for %s, total %s)\\n\", m.Name, elapsed, suite.paused, total)\n\t\t\t} else if err != nil {\n\t\t\t\tfmt.Printf(\"(perf) FAIL: %s (%s, paused for %s, total %s)\\n\", m.Name, elapsed, suite.paused, total)\n\t\t\t\tfmt.Println(err)\n\t\t\t}\n\n\t\t\ttestReps[repIdx][recordName] = timeInfo{elapsed, suite.paused, total}\n\n\t\t\tif t, ok := suiteT.(testifySuite.TearDownTestSuite); ok {\n\t\t\t\tt.TearDownTest()\n\t\t\t}\n\t\t}\n\n\t\tif t, ok := suiteT.(TearDownRepSuite); ok {\n\t\t\tt.TearDownRep()\n\t\t}\n\n\t\tstopServerFn()\n\t}\n\n\tif t, ok := suiteT.(testifySuite.TearDownAllSuite); ok {\n\t\tt.TearDownSuite()\n\t}\n}\n\nfunc (suite *PerfSuite) Suite() *PerfSuite {\n\treturn suite\n}\n\n// NewAssert returns the assert.Assertions instance for this test.\nfunc (suite *PerfSuite) NewAssert() *assert.Assertions {\n\treturn assert.New(suite.T)\n}\n\n// TempFile creates a temporary file, which will be automatically cleaned up by\n// the perf test suite. Files will be prefixed with the test's dataset ID\nfunc (suite *PerfSuite) TempFile() *os.File {\n\tf, err := ioutil.TempFile(\"\", suite.tempPrefix())\n\tassert.NoError(suite.T, err)\n\tsuite.tempFiles = append(suite.tempFiles, f)\n\treturn f\n}\n\n// TempDir creates a temporary directory, which will be automatically cleaned\n// up by the perf test suite. Directories will be prefixed with the test's\n// dataset ID.\nfunc (suite *PerfSuite) TempDir() string {\n\td, err := ioutil.TempDir(\"\", suite.tempPrefix())\n\tassert.NoError(suite.T, err)\n\tsuite.tempDirs = append(suite.tempDirs, d)\n\treturn d\n}\n\nfunc (suite *PerfSuite) tempPrefix() string {\n\tsep := fmt.Sprintf(\"%c\", os.PathSeparator)\n\treturn strings.Replace(fmt.Sprintf(\"perf.%s.\", suite.datasetID), sep, \".\", -1)\n}\n\n// Pause pauses the test timer while fn is executing. Useful for omitting long setup code (e.g. copying files) from the test elapsed time.\nfunc (suite *PerfSuite) Pause(fn func()) {\n\tstart := time.Now()\n\tfn()\n\tsuite.paused += time.Since(start)\n}\n\n// OpenGlob opens the concatenation of all files that match pattern, returned\n// as []io.Reader so it can be used immediately with io.MultiReader.\n//\n// Large CSV files in testdata are broken up into foo.a, foo.b, etc to get\n// around GitHub file size restrictions.\nfunc (suite *PerfSuite) OpenGlob(pattern ...string) []io.Reader {\n\tassert := suite.NewAssert()\n\n\tglob, err := filepath.Glob(path.Join(pattern...))\n\tassert.NoError(err)\n\n\tfiles := make([]io.Reader, len(glob))\n\tfor i, m := range glob {\n\t\tf, err := os.Open(m)\n\t\tassert.NoError(err)\n\t\tfiles[i] = f\n\t}\n\n\treturn files\n}\n\n// CloseGlob closes all of the files, designed to be used with OpenGlob.\nfunc (suite *PerfSuite) CloseGlob(files []io.Reader) {\n\tassert := suite.NewAssert()\n\tfor _, f := range files {\n\t\tassert.NoError(f.(*os.File).Close())\n\t}\n}\n\nfunc callSafe(name string, fun reflect.Value, args ...interface{}) error {\n\tfunArgs := make([]reflect.Value, len(args))\n\tfor i, arg := range args {\n\t\tfunArgs[i] = reflect.ValueOf(arg)\n\t}\n\treturn d.Try(func() {\n\t\tfun.Call(funArgs)\n\t})\n}\n\nfunc (suite *PerfSuite) getEnvironment(vrw types.ValueReadWriter) types.Value {\n\tassert := suite.NewAssert()\n\n\tenv := environment{\n\t\tDiskUsages: map[string]disk.UsageStat{},\n\t\tCpus: map[int]cpu.InfoStat{},\n\t\tPartitions: map[string]disk.PartitionStat{},\n\t}\n\n\tpartitions, err := disk.Partitions(false)\n\tassert.NoError(err)\n\tfor _, p := range partitions {\n\t\tusage, err := disk.Usage(p.Mountpoint)\n\t\tassert.NoError(err)\n\t\tenv.DiskUsages[p.Mountpoint] = *usage\n\t\tenv.Partitions[p.Device] = p\n\t}\n\n\tcpus, err := cpu.Info()\n\tassert.NoError(err)\n\tfor i, c := range cpus {\n\t\tenv.Cpus[i] = c\n\t}\n\n\tmem, err := mem.VirtualMemory()\n\tassert.NoError(err)\n\tenv.Mem = *mem\n\n\thostInfo, err := host.Info()\n\tassert.NoError(err)\n\tenv.Host = *hostInfo\n\n\tenvStruct, err := marshal.Marshal(vrw, env)\n\tassert.NoError(err)\n\treturn envStruct\n}\n\nfunc (suite *PerfSuite) getGitHead(dir string) string {\n\tstdout := &bytes.Buffer{}\n\tcmd := exec.Command(\"git\", \"rev-parse\", \"HEAD\")\n\tcmd.Stdout = stdout\n\tcmd.Dir = dir\n\tif err := cmd.Run(); err != nil {\n\t\treturn \"\"\n\t}\n\treturn strings.TrimSpace(stdout.String())\n}\n\n// StartRemoteDatabase creates a new remote database on an arbitrary free port,\n// running on a separate goroutine. Returns the hostname that that database was\n// started on, and a callback to run to shut down the server.\n//\n// If the -perf.mem flag is specified, the remote database is hosted in memory,\n// not on disk (in a temporary nbs directory).\n//\n// - Why not use a local database + memory store?\n// Firstly, because the spec would be \"mem\", and the spec library doesn't\n// know how to reuse stores.\n// Secondly, because it's an unrealistic performance measurement.\n//\n// - Why use a remote (HTTP) database?\n// It's more realistic to exercise the HTTP stack, even if it's just talking\n// over localhost.\n//\n// - Why provide an option for nbs vs memory underlying store?\n// Again, nbs is more realistic than memory, and in common cases disk\n// space > memory space.\n// However, on this developer's laptop, there is\n// actually very little disk space, and a lot of memory; plus making the\n// test run a little bit faster locally is nice.\nfunc (suite *PerfSuite) StartRemoteDatabase() (host string, stopFn func()) {\n\tvar chunkStore chunks.ChunkStore\n\tif *perfMemFlag {\n\t\tst := &chunks.MemoryStorage{}\n\t\tchunkStore = st.NewView()\n\t} else {\n\t\tdbDir := suite.TempDir()\n\t\tchunkStore = nbs.NewLocalStore(dbDir, 128*(1<<20))\n\t}\n\n\tserver := datas.NewRemoteDatabaseServer(chunkStore, \"0.0.0.0\", 0)\n\tportChan := make(chan int)\n\tserver.Ready = func() { portChan <- server.Port() }\n\tgo server.Run()\n\n\tport := <-portChan\n\thost = fmt.Sprintf(\"http://localhost:%d\", port)\n\tstopFn = func() { server.Stop() }\n\treturn\n}\n" }, { "path": "go/perf/suite/suite_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage suite\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype testSuite struct {\n\tPerfSuite\n\ttempFileName, tempDir string\n\tsetupTest, tearDownTest int\n\tsetupRep, tearDownRep int\n\tsetupSuite, tearDownSuite int\n\tfoo, bar, abc, def, nothing, testimate int\n}\n\nfunc (s *testSuite) TestNonEmptyPaths() {\n\tassert := s.NewAssert()\n\tassert.NotEqual(\"\", s.AtticLabs)\n\tassert.NotEqual(\"\", s.Testdata)\n\tassert.NotEqual(\"\", s.DatabaseSpec)\n}\n\nfunc (s *testSuite) TestDatabase() {\n\tassert := s.NewAssert()\n\tval := types.Bool(true)\n\tr := s.Database.WriteValue(val)\n\tassert.True(s.Database.ReadValue(r.TargetHash()).Equals(val))\n}\n\nfunc (s *testSuite) TestTempFile() {\n\ts.tempFileName = s.TempFile().Name()\n\ts.tempDir = s.TempDir()\n}\n\nfunc (s *testSuite) TestGlob() {\n\tassert := s.NewAssert()\n\tf := s.TempFile()\n\tf.Close()\n\n\tcreate := func(suffix string) {\n\t\tf, err := os.Create(f.Name() + suffix)\n\t\tassert.NoError(err)\n\t\tf.Close()\n\t}\n\n\tcreate(\"a\")\n\tcreate(\".a\")\n\tcreate(\".b\")\n\n\tglob := s.OpenGlob(f.Name() + \".*\")\n\tassert.Equal(2, len(glob))\n\tassert.Equal(f.Name()+\".a\", glob[0].(*os.File).Name())\n\tassert.Equal(f.Name()+\".b\", glob[1].(*os.File).Name())\n\n\ts.CloseGlob(glob)\n\tb := make([]byte, 16)\n\t_, err := glob[0].Read(b)\n\tassert.Error(err)\n\t_, err = glob[1].Read(b)\n\tassert.Error(err)\n}\n\nfunc (s *testSuite) TestPause() {\n\ts.Pause(func() {\n\t\ts.waitForSmidge()\n\t})\n}\n\nfunc (s *testSuite) TestFoo() {\n\ts.foo++\n\ts.waitForSmidge()\n}\n\nfunc (s *testSuite) TestBar() {\n\ts.bar++\n\ts.waitForSmidge()\n}\n\nfunc (s *testSuite) Test01Abc() {\n\ts.abc++\n\ts.waitForSmidge()\n}\n\nfunc (s *testSuite) Test02Def() {\n\ts.def++\n\ts.waitForSmidge()\n}\n\nfunc (s *testSuite) testNothing() {\n\ts.nothing++\n\ts.waitForSmidge()\n}\n\nfunc (s *testSuite) Testimate() {\n\ts.testimate++\n\ts.waitForSmidge()\n}\n\nfunc (s *testSuite) SetupTest() {\n\ts.setupTest++\n}\n\nfunc (s *testSuite) TearDownTest() {\n\ts.tearDownTest++\n}\n\nfunc (s *testSuite) SetupRep() {\n\ts.setupRep++\n}\n\nfunc (s *testSuite) TearDownRep() {\n\ts.tearDownRep++\n}\n\nfunc (s *testSuite) SetupSuite() {\n\ts.setupSuite++\n}\n\nfunc (s *testSuite) TearDownSuite() {\n\ts.tearDownSuite++\n}\n\nfunc (s *testSuite) waitForSmidge() {\n\t// Tests should call this to make sure the measurement shows up as > 0, not that it shows up as a millisecond.\n\t<-time.After(time.Millisecond)\n}\n\nfunc TestSuite(t *testing.T) {\n\trunTestSuite(t, false)\n}\n\nfunc TestSuiteWithMem(t *testing.T) {\n\tt.Skip(\"Flaky on Jenkins\")\n\trunTestSuite(t, true)\n}\n\nfunc runTestSuite(t *testing.T, mem bool) {\n\tassert := assert.New(t)\n\n\t// Write test results to our own temporary LDB database.\n\tldbDir, err := ioutil.TempDir(\"\", \"suite.TestSuite\")\n\tassert.NoError(err)\n\tdefer os.RemoveAll(ldbDir)\n\n\tflagVal, repeatFlagVal, memFlagVal := *perfFlag, *perfRepeatFlag, *perfMemFlag\n\t*perfFlag, *perfRepeatFlag, *perfMemFlag = ldbDir, 3, mem\n\tdefer func() {\n\t\t*perfFlag, *perfRepeatFlag, *perfMemFlag = flagVal, repeatFlagVal, memFlagVal\n\t}()\n\n\ts := &testSuite{}\n\tRun(\"ds\", t, s)\n\n\texpectedTests := []string{\n\t\t\"Abc\",\n\t\t\"Bar\",\n\t\t\"Database\",\n\t\t\"Def\",\n\t\t\"Foo\",\n\t\t\"Glob\",\n\t\t\"NonEmptyPaths\",\n\t\t\"Pause\",\n\t\t\"TempFile\",\n\t}\n\n\t// The temp file and dir should have been cleaned up.\n\t_, err = os.Stat(s.tempFileName)\n\tassert.NotNil(err)\n\t_, err = os.Stat(s.tempDir)\n\tassert.NotNil(err)\n\n\t// The correct number of Setup/TearDown calls should have been run.\n\tassert.Equal(1, s.setupSuite)\n\tassert.Equal(1, s.tearDownSuite)\n\tassert.Equal(*perfRepeatFlag, s.setupRep)\n\tassert.Equal(*perfRepeatFlag, s.tearDownRep)\n\tassert.Equal(*perfRepeatFlag*len(expectedTests), s.setupTest)\n\tassert.Equal(*perfRepeatFlag*len(expectedTests), s.tearDownTest)\n\n\t// The results should have been written to the \"ds\" dataset.\n\tsp, err := spec.ForDataset(ldbDir + \"::ds\")\n\tassert.NoError(err)\n\tdefer sp.Close()\n\thead := sp.GetDataset().HeadValue().(types.Struct)\n\n\t// These tests mostly assert that the structure of the results is correct. Specific values are hard.\n\n\tgetOrFail := func(s types.Struct, f string) types.Value {\n\t\tval, ok := s.MaybeGet(f)\n\t\tassert.True(ok)\n\t\treturn val\n\t}\n\n\tenv, ok := getOrFail(head, \"environment\").(types.Struct)\n\tassert.True(ok)\n\n\tgetOrFail(env, \"diskUsages\")\n\tgetOrFail(env, \"cpus\")\n\tgetOrFail(env, \"mem\")\n\tgetOrFail(env, \"host\")\n\tgetOrFail(env, \"partitions\")\n\n\t// Todo: re-enable this code once demo-server gets build without CodePipeline\n\t// This fails with CodePipeline because the source code is brought into\n\t// Jenkins as a zip file rather than as a git repo.\n\t//nomsRevision := getOrFail(head, \"nomsRevision\")\n\t//assert.True(ok)\n\t//assert.True(string(nomsRevision.(types.String)) != \"\")\n\t//getOrFail(head, \"testdataRevision\")\n\n\treps, ok := getOrFail(head, \"reps\").(types.List)\n\tassert.True(ok)\n\tassert.Equal(*perfRepeatFlag, int(reps.Len()))\n\n\treps.IterAll(func(rep types.Value, _ uint64) {\n\t\ti := 0\n\n\t\trep.(types.Map).IterAll(func(k, timesVal types.Value) {\n\t\t\tif assert.True(i < len(expectedTests)) {\n\t\t\t\tassert.Equal(expectedTests[i], string(k.(types.String)))\n\t\t\t}\n\n\t\t\ttimes := timesVal.(types.Struct)\n\t\t\tassert.True(getOrFail(times, \"elapsed\").(types.Number) > 0)\n\t\t\tassert.True(getOrFail(times, \"total\").(types.Number) > 0)\n\n\t\t\tpaused := getOrFail(times, \"paused\").(types.Number)\n\t\t\tif k == types.String(\"Pause\") {\n\t\t\t\tassert.True(paused > 0)\n\t\t\t} else {\n\t\t\t\tassert.True(paused == 0)\n\t\t\t}\n\n\t\t\ti++\n\t\t})\n\n\t\tassert.Equal(i, len(expectedTests))\n\t})\n}\n\nfunc TestPrefixFlag(t *testing.T) {\n\tt.Skip(\"Flaky on Jenkins\")\n\tassert := assert.New(t)\n\n\t// Write test results to a temporary database.\n\tldbDir, err := ioutil.TempDir(\"\", \"suite.TestSuite\")\n\tassert.NoError(err)\n\tdefer os.RemoveAll(ldbDir)\n\n\tflagVal, prefixFlagVal := *perfFlag, *perfPrefixFlag\n\t*perfFlag, *perfPrefixFlag = ldbDir, \"foo/\"\n\tdefer func() {\n\t\t*perfFlag, *perfPrefixFlag = flagVal, prefixFlagVal\n\t}()\n\n\tRun(\"my-prefix/test\", t, &PerfSuite{})\n\n\t// The results should have been written to \"foo/my-prefix/test\" not \"my-prefix/test\".\n\tsp, err := spec.ForDataset(ldbDir + \"::my-prefix/test\")\n\tassert.NoError(err)\n\tdefer sp.Close()\n\t_, ok := sp.GetDataset().MaybeHead()\n\tassert.False(ok)\n\n\tsp, err = spec.ForDataset(ldbDir + \"::foo/my-prefix/test\")\n\tassert.NoError(err)\n\tdefer sp.Close()\n\t_, ok = sp.GetDataset().HeadValue().(types.Struct)\n\tassert.True(ok)\n}\n\nfunc TestRunFlag(t *testing.T) {\n\tt.Skip(\"Flaky on Jenkins\")\n\tassert := assert.New(t)\n\n\ttype expect struct {\n\t\tfoo, bar, abc, def, nothing, testimate int\n\t}\n\n\trun := func(re string, exp expect) {\n\t\tflagVal, memFlagVal, runFlagVal := *perfFlag, *perfMemFlag, *perfRunFlag\n\t\t*perfFlag, *perfMemFlag, *perfRunFlag = \"mem\", true, re\n\t\tdefer func() {\n\t\t\t*perfFlag, *perfMemFlag, *perfRunFlag = flagVal, memFlagVal, runFlagVal\n\t\t}()\n\t\ts := testSuite{}\n\t\tRun(\"test\", t, &s)\n\t\tassert.Equal(exp, expect{s.foo, s.bar, s.abc, s.def, s.nothing, s.testimate})\n\t}\n\n\trun(\"\", expect{foo: 1, bar: 1, abc: 1, def: 1})\n\trun(\".\", expect{foo: 1, bar: 1, abc: 1, def: 1})\n\trun(\"test\", expect{foo: 1, bar: 1, abc: 1, def: 1})\n\trun(\"^test\", expect{foo: 1, bar: 1, abc: 1, def: 1})\n\trun(\"Test\", expect{foo: 1, bar: 1, abc: 1, def: 1})\n\trun(\"^Test\", expect{foo: 1, bar: 1, abc: 1, def: 1})\n\n\trun(\"f\", expect{foo: 1, def: 1})\n\trun(\"^f\", expect{foo: 1})\n\trun(\"testf\", expect{foo: 1})\n\trun(\"^testf\", expect{foo: 1})\n\trun(\"testF\", expect{foo: 1})\n\trun(\"^testF\", expect{foo: 1})\n\n\trun(\"F\", expect{foo: 1, def: 1})\n\trun(\"^F\", expect{foo: 1})\n\trun(\"Testf\", expect{foo: 1})\n\trun(\"^Testf\", expect{foo: 1})\n\trun(\"TestF\", expect{foo: 1})\n\trun(\"^TestF\", expect{foo: 1})\n\n\trun(\"ef\", expect{def: 1})\n\trun(\"def\", expect{def: 1})\n\trun(\"ddef\", expect{})\n\trun(\"testdef\", expect{})\n\trun(\"test01def\", expect{})\n\trun(\"test02def\", expect{def: 1})\n\trun(\"Test02def\", expect{def: 1})\n\trun(\"test02Def\", expect{def: 1})\n\trun(\"Test02Def\", expect{def: 1})\n\n\trun(\"z\", expect{})\n\trun(\"testz\", expect{})\n\trun(\"Testz\", expect{})\n\n\trun(\"[fa]\", expect{foo: 1, bar: 1, abc: 1, def: 1})\n\trun(\"[fb]\", expect{foo: 1, bar: 1, abc: 1, def: 1})\n\trun(\"[fc]\", expect{foo: 1, abc: 1, def: 1})\n\trun(\"test[fa]\", expect{foo: 1})\n\trun(\"test[fb]\", expect{foo: 1, bar: 1})\n\trun(\"test[fc]\", expect{foo: 1})\n\trun(\"Test[fa]\", expect{foo: 1})\n\trun(\"Test[fb]\", expect{foo: 1, bar: 1})\n\trun(\"Test[fc]\", expect{foo: 1})\n\n\trun(\"foo|bar\", expect{foo: 1, bar: 1})\n\trun(\"FOO|bar\", expect{foo: 1, bar: 1})\n\trun(\"Testfoo|bar\", expect{foo: 1, bar: 1})\n\trun(\"TestFOO|bar\", expect{foo: 1, bar: 1})\n\n\trun(\"Testfoo|Testbar\", expect{foo: 1, bar: 1})\n\trun(\"TestFOO|Testbar\", expect{foo: 1, bar: 1})\n\n\trun(\"footest\", expect{})\n\trun(\"nothing\", expect{})\n}\n" }, { "path": "go/sloppy/sloppy.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage sloppy\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nconst (\n\tmaxOffsetPOT = uint16(12)\n\tmaxTableSize = 1 << 14\n\tmaxLength = 1<<12 - 1\n\ttableMask = maxTableSize - 1\n\tshift = uint32(20)\n)\n\n// TODO: Make this configurable\nvar maxOffset = int(1< maxLength || src[sl.idx] != src[sl.matchOffset+sl.matchLength]) {\n\t\t\t// End Match\n\t\t\tif sl.maybeCopy(src) {\n\t\t\t\treturn // terminate if consumer has \"closed\"\n\t\t\t}\n\t\t}\n\n\t\t// Look for a match if we are beyond the first byte AND either there is no\n\t\t// match yet, OR we are matching, but fewer than 3 bytes have been\n\t\t// matched. The later condition allows for giving up to 2 bytes of a copy\n\t\t// in order to reference a \"closer\" sequence. Empirical tests on\n\t\t// structured data, suggests this reduces the average offset by ~2/3.\n\t\tif sl.idx > 0 && (!sl.matching || sl.matchLength < 3) {\n\t\t\tmatchPos := int(sl.table[nextHash&tableMask])\n\n\t\t\tif sl.idx > matchPos &&\n\t\t\t\tsrc[sl.idx] == src[matchPos] && // filter false positives\n\t\t\t\tsl.idx-matchPos <= maxOffset && // don't refer back beyond maxOffset\n\t\t\t\t(!sl.matching || matchPos >= sl.matchOffset+4) { // if we are \"rematching\", ensure the new match is at least 4 bytes closer\n\n\t\t\t\tif sl.matching {\n\t\t\t\t\t// We are dropping an existing match for a closer one. Emit the\n\t\t\t\t\t// matched bytes as literals\n\t\t\t\t\tif sl.dontCopy(src, sl.idx-sl.matchLength, sl.idx) {\n\t\t\t\t\t\treturn // terminate if consumer has \"closed\"\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\t// Begin a new match\n\t\t\t\tsl.matching = true\n\t\t\t\tsl.matchOffset = matchPos\n\t\t\t\tsl.matchLength = 0\n\t\t\t}\n\t\t}\n\n\t\t// Store new hashed offset\n\t\tsl.table[nextHash&tableMask] = uint32(sl.idx)\n\n\t\tif sl.matching {\n\t\t\tsl.matchLength++\n\t\t} else {\n\t\t\tif sl.enc.emitLiteral(src[sl.idx]) {\n\t\t\t\treturn // terminate if consumer has \"closed\"\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc (sl *Sloppy) Reset() {\n\tsl.idx = 0\n\tsl.matching = false\n\tsl.matchOffset = 0\n\tsl.matchLength = 0\n\tsl.table = [maxTableSize]uint32{}\n}\n\n// len >= 2^(2 + log2(maxOffset) - log2(maxOffset-off)). IOW, for the first 1/2\n// of the maxOffset, a copy must be >= 4. For 1/2 of what remains, a copy must\n// be >= 8, etc...\nfunc copyLongEnough(off, len uint16) bool {\n\td.PanicIfTrue(off == 0)\n\n\tp := uint16(0)\n\tx := (1 << maxOffsetPOT) - off\n\tfor x > 0 {\n\t\tx = x >> 1\n\t\tp++\n\t}\n\n\ti := maxOffsetPOT - p\n\tmin := 4\n\tfor i > 0 {\n\t\tmin = min << 1\n\t\ti--\n\t}\n\n\treturn int(len) >= min\n}\n\n// Emit matches bytes as literals.\nfunc (sl *Sloppy) dontCopy(src []byte, from, to int) bool {\n\tfor ; from < to; from++ {\n\t\tif sl.enc.emitLiteral(src[from]) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\n// Emit a copy if the length is sufficient for a given offset\nfunc (sl *Sloppy) maybeCopy(src []byte) bool {\n\toff, len := uint16(sl.idx-(sl.matchOffset+sl.matchLength)), uint16(sl.matchLength)\n\tsl.matching = false\n\tsl.matchOffset = 0\n\tsl.matchLength = 0\n\n\tif !copyLongEnough(off, len) {\n\t\treturn sl.dontCopy(src, sl.idx-int(len), sl.idx)\n\t}\n\n\treturn sl.enc.emitCopy(off, len)\n}\n\ntype encoder interface {\n\temitLiteral(b byte) bool\n\temitCopy(offset, length uint16) bool\n}\n\ntype binaryEncoder struct {\n\tf func(b byte) bool\n}\n\nfunc (be binaryEncoder) emitLiteral(b byte) bool {\n\treturn be.f(b)\n}\n\nfunc (be binaryEncoder) emitCopy(offset, length uint16) bool {\n\t// all copies are encoded as 3 bytes.\n\t// 12 bits for offset and 12 bits for length\n\n\t// 8 MSBits of offset\n\tif be.f(byte(offset >> 4)) {\n\t\treturn true\n\t}\n\n\t// 4 LSBits offset | 4 MSBits length\n\tif be.f(byte(offset<<4) | byte(length>>4)) {\n\t\treturn true\n\t}\n\n\t// 8 LSBits of length\n\tif be.f(byte(length)) {\n\t\treturn true\n\t}\n\n\treturn false\n}\n\nfunc fbhash(u uint32) uint32 {\n\treturn (u * 0x1e35a7bd) >> shift\n}\n\nfunc load32(b []byte, i int) uint32 {\n\tb = b[i : i+4 : len(b)] // Help the compiler eliminate bounds checks on the next line.\n\treturn uint32(b[0]) | uint32(b[1])<<8 | uint32(b[2])<<16 | uint32(b[3])<<24\n}\n" }, { "path": "go/sloppy/sloppy_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage sloppy\n\nimport (\n\t\"fmt\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/golang/snappy\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype testSloppyEncoder struct {\n\tcurrent []byte\n\tenc []interface{}\n}\n\nfunc (tt *testSloppyEncoder) emitLiteral(b byte) bool {\n\ttt.current = append(tt.current, b)\n\treturn false\n}\n\nfunc (tt *testSloppyEncoder) emitCopy(offset, length uint16) bool {\n\ttt.emitString()\n\ttt.enc = append(tt.enc, offset, length)\n\treturn false\n}\n\nfunc (tt *testSloppyEncoder) emitString() {\n\tif len(tt.current) > 0 {\n\t\ttt.enc = append(tt.enc, string(tt.current))\n\t\ttt.current = make([]byte, 0)\n\t}\n}\n\nfunc TestCopyLongEnough(t *testing.T) {\n\tassert.False(t, copyLongEnough(1, 3))\n\tassert.True(t, copyLongEnough(1, 4))\n\tassert.False(t, copyLongEnough(2047, 3))\n\tassert.True(t, copyLongEnough(2048, 4))\n\tassert.False(t, copyLongEnough(2048+1, 7))\n\tassert.True(t, copyLongEnough(2048+1, 8))\n\tassert.False(t, copyLongEnough(2048+1024+1, 15))\n\tassert.True(t, copyLongEnough(2048+1024+1, 16))\n\tassert.False(t, copyLongEnough(2048+1024+512+1, 31))\n\tassert.True(t, copyLongEnough(2048+1024+512+1, 32))\n\tassert.False(t, copyLongEnough(2048+1024+512+256+1, 63))\n\tassert.True(t, copyLongEnough(2048+1024+512+256+1, 64))\n\tassert.False(t, copyLongEnough(2048+1024+512+256+128+1, 127))\n\tassert.True(t, copyLongEnough(2048+1024+512+256+128+1, 128))\n\tassert.False(t, copyLongEnough(2048+1024+512+256+128+64+1, 255))\n\tassert.True(t, copyLongEnough(2048+1024+512+256+128+64+1, 256))\n\tassert.False(t, copyLongEnough(2048+1024+512+256+128+64+32+1, 511))\n\tassert.True(t, copyLongEnough(2048+1024+512+256+128+64+32+1, 512))\n\tassert.False(t, copyLongEnough(2048+1024+512+256+128+64+32+16+1, 1023))\n\tassert.True(t, copyLongEnough(2048+1024+512+256+128+64+32+16+1, 1024))\n\tassert.False(t, copyLongEnough(2048+1024+512+256+128+64+32+16+8+1, 2047))\n\tassert.True(t, copyLongEnough(2048+1024+512+256+128+64+32+16+8+1, 2048))\n\tassert.False(t, copyLongEnough(2048+1024+512+256+128+64+32+16+8+4+1, 4095))\n\tassert.True(t, copyLongEnough(2048+1024+512+256+128+64+32+16+8+4+1, 4096))\n}\n\nfunc TestSloppySimple(t *testing.T) {\n\tmo := maxOffset\n\tdefer func() {\n\t\tmaxOffset = mo\n\t}()\n\n\t// NOTE: Sloppy always buffers the last three bytes of input.\n\ttc := []struct {\n\t\tmax int\n\t\tpt string\n\t\tenc []interface{}\n\t}{\n\t\t// Match length not long enough\n\t\t{4, \"ababxxxx\", []interface{}{\"ababx\"}},\n\t\t// Trailing literal\n\t\t{4, \"01230123xxxx\", []interface{}{\"0123\", uint16(4), uint16(4), \"x\"}},\n\t\t// Match past current\n\t\t{4, \"012301230123xxxx\", []interface{}{\"0123\", uint16(4), uint16(8), \"x\"}},\n\t\t// Offset to most recent\n\t\t{6, \"ABCDxABCDyABCDzzzz\", []interface{}{\"ABCDx\", uint16(5), uint16(4), \"y\", uint16(5), uint16(4), \"z\"}},\n\t\t// Offset beyond max\n\t\t{5, \"0123x0123yy0123zzzz\", []interface{}{\"0123x\", uint16(5), uint16(4), \"yy0123z\"}},\n\t\t// Prefer closer match\n\t\t{128, \"23hello my friend, 12hello my friend, 01hello my friend, 23hello my friendxxxx\",\n\t\t\t[]interface{}{\"23hello my friend, 12\", uint16(19), uint16(17),\n\t\t\t\t\"01\", uint16(19), uint16(17),\n\t\t\t\t\"23\", uint16(19), uint16(15),\n\t\t\t\t\"x\",\n\t\t\t}},\n\t}\n\n\tfor i, c := range tc {\n\t\tt.Run(fmt.Sprintf(\"Case %d\", i), func(t *testing.T) {\n\t\t\ttt := &testSloppyEncoder{\n\t\t\t\t[]byte{},\n\t\t\t\t[]interface{}{},\n\t\t\t}\n\t\t\tmaxOffset = c.max\n\t\t\tsl := New(nil)\n\t\t\tsl.enc = tt\n\t\t\tsl.Update([]byte(c.pt))\n\t\t\ttt.emitString()\n\t\t\tassert.Equal(t, c.enc, tt.enc)\n\t\t})\n\t}\n}\n\nfunc TestSloppyContinuation(t *testing.T) {\n\tmo := maxOffset\n\tdefer func() {\n\t\tmaxOffset = mo\n\t}()\n\n\t// NOTE: Sloppy always buffers the last three bytes of input.\n\n\ttc := []struct {\n\t\tmax int\n\t\tpt, pt2 string\n\t\tenc []interface{}\n\t}{\n\t\t// Simple,\n\t\t{4, \"umborkbork\", \"umborkborkborkaaaa\", []interface{}{\"umbork\", uint16(4), uint16(8), \"a\"}},\n\t\t{8, \"umborkbork\", \"umborkborkxyzborkaaaa\", []interface{}{\"umbork\", uint16(4), uint16(4), \"xyz\", uint16(7), uint16(4), \"a\"}},\n\t\t// Resume indexing\n\t\t{8, \"x\", \"xABCDABCDxxxx\", []interface{}{\"xABCD\", uint16(4), uint16(4), \"x\"}},\n\t\t{8, \"xA\", \"xABCDABCDxxxx\", []interface{}{\"xABCD\", uint16(4), uint16(4), \"x\"}},\n\t\t{8, \"xAB\", \"xABCDABCDxxxx\", []interface{}{\"xABCD\", uint16(4), uint16(4), \"x\"}},\n\t\t{8, \"xABC\", \"xABCDABCDxxxx\", []interface{}{\"xABCD\", uint16(4), uint16(4), \"x\"}},\n\t\t{8, \"xABCD\", \"xABCDABCDxxxx\", []interface{}{\"xABCD\", uint16(4), uint16(4), \"x\"}},\n\t\t{8, \"xABCDA\", \"xABCDABCDxxxx\", []interface{}{\"xABCD\", uint16(4), uint16(4), \"x\"}},\n\t\t{8, \"xABCDAB\", \"xABCDABCDxxxx\", []interface{}{\"xABCD\", uint16(4), uint16(4), \"x\"}},\n\t\t{8, \"xABCDABC\", \"xABCDABCDxxxx\", []interface{}{\"xABCD\", uint16(4), uint16(4), \"x\"}},\n\t\t{8, \"xABCDABCD\", \"xABCDABCDxxxx\", []interface{}{\"xABCD\", uint16(4), uint16(4), \"x\"}},\n\t}\n\n\tfor i, c := range tc {\n\t\tt.Run(fmt.Sprintf(\"Case %d\", i), func(t *testing.T) {\n\t\t\ttt := &testSloppyEncoder{\n\t\t\t\t[]byte{},\n\t\t\t\t[]interface{}{},\n\t\t\t}\n\t\t\tmaxOffset = c.max\n\t\t\tsl := New(nil)\n\t\t\tsl.enc = tt\n\n\t\t\tsl.Update([]byte(c.pt))\n\t\t\tsl.Update([]byte(c.pt2))\n\t\t\ttt.emitString()\n\t\t\tassert.Equal(t, c.enc, tt.enc)\n\t\t})\n\t}\n}\n\nfunc TestBinaryEncoder(t *testing.T) {\n\tout := []byte{}\n\tf := func(b byte) bool {\n\t\tout = append(out, b)\n\t\treturn false\n\t}\n\n\tb := binaryEncoder{f}\n\tb.emitLiteral(5)\n\tassert.Equal(t, []byte{5}, out)\n\n\tout = out[0:0]\n\tb.emitCopy(0, 5)\n\tassert.Equal(t, []byte{0, 0, 5}, out)\n\n\tout = out[0:0]\n\tb.emitCopy(5, 5)\n\tassert.Equal(t, []byte{0, 80, 5}, out)\n\n\tout = out[0:0]\n\tb.emitCopy(4095, 4095)\n\tassert.Equal(t, []byte{255, 255, 255}, out)\n}\nfunc TestSnappyVsSloppy(t *testing.T) {\n\torig := []byte(aliceText)\n\tt1 := time.Now()\n\tsnappyComp := snappy.Encode(nil, orig)\n\tt2 := time.Now()\n\n\tsloppyBytes := int(0)\n\tf := func(b byte) bool {\n\t\tsloppyBytes++\n\t\treturn false\n\t}\n\tsl := New(f)\n\n\tt3 := time.Now()\n\tsl.Update(orig)\n\tt4 := time.Now()\n\n\t// Note: it's expected that sloppy doesn't compress as well as snappy - mainly\n\t// because it can only refer to copies 4k away, but also because it will\n\t// sacrifice a slightly longer copy in order to refer less far away.\n\tfmt.Println(\"Original\", len([]byte(aliceText)))\n\tfmt.Println(\"Snappy\", len(snappyComp), t2.Sub(t1))\n\tfmt.Println(\"Sloppy\", sloppyBytes, t4.Sub(t3))\n\tassert.Equal(t, int(43104), sloppyBytes)\n}\n\nvar aliceText = `Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, ‘and what is the use of a book,’ thought Alice ‘without pictures or conversations?’\nSo she was considering in her own mind (as well as she could, for the hot day made her feel very sleepy and stupid), whether the pleasure of making a daisy-chain would be worth the trouble of getting up and picking the daisies, when suddenly a White Rabbit with pink eyes ran close by her.\nThere was nothing so very remarkable in that; nor did Alice think it so very much out of the way to hear the Rabbit say to itself, ‘Oh dear! Oh dear! I shall be late!’ (when she thought it over afterwards, it occurred to her that she ought to have wondered at this, but at the time it all seemed quite natural); but when the Rabbit actually took a watch out of its waistcoat-pocket, and looked at it, and then hurried on, Alice started to her feet, for it flashed across her mind that she had never before seen a rabbit with either a waistcoat-pocket, or a watch to take out of it, and burning with curiosity, she ran across the field after it, and fortunately was just in time to see it pop down a large rabbit-hole under the hedge.\nIn another moment down went Alice after it, never once considering how in the world she was to get out again.\nThe rabbit-hole went straight on like a tunnel for some way, and then dipped suddenly down, so suddenly that Alice had not a moment to think about stopping herself before she found herself falling down a very deep well.\nEither the well was very deep, or she fell very slowly, for she had plenty of time as she went down to look about her and to wonder what was going to happen next. First, she tried to look down and make out what she was coming to, but it was too dark to see anything; then she looked at the sides of the well, and noticed that they were filled with cupboards and book-shelves; here and there she saw maps and pictures hung upon pegs. She took down a jar from one of the shelves as she passed; it was labelled ‘ORANGE MARMALADE’, but to her great disappointment it was empty: she did not like to drop the jar for fear of killing somebody, so managed to put it into one of the cupboards as she fell past it.\n‘Well!’ thought Alice to herself, ‘after such a fall as this, I shall think nothing of tumbling down stairs! How brave they’ll all think me at home! Why, I wouldn’t say anything about it, even if I fell off the top of the house!’ (Which was very likely true.)\nDown, down, down. Would the fall never come to an end! ‘I wonder how many miles I’ve fallen by this time?’ she said aloud. ‘I must be getting somewhere near the centre of the earth. Let me see: that would be four thousand miles down, I think—’ (for, you see, Alice had learnt several things of this sort in her lessons in the schoolroom, and though this was not a very good opportunity for showing off her knowledge, as there was no one to listen to her, still it was good practice to say it over) ‘—yes, that’s about the right distance—but then I wonder what Latitude or Longitude I’ve got to?’ (Alice had no idea what Latitude was, or Longitude either, but thought they were nice grand words to say.)\nPresently she began again. ‘I wonder if I shall fall right through the earth! How funny it’ll seem to come out among the people that walk with their heads downward! The Antipathies, I think—’ (she was rather glad there was no one listening, this time, as it didn’t sound at all the right word) ‘—but I shall have to ask them what the name of the country is, you know. Please, Ma’am, is this New Zealand or Australia?’ (and she tried to curtsey as she spoke—fancy curtseying as you’re falling through the air! Do you think you could manage it?) ‘And what an ignorant little girl she’ll think me for asking! No, it’ll never do to ask: perhaps I shall see it written up somewhere.’\nDown, down, down. There was nothing else to do, so Alice soon began talking again. ‘Dinah’ll miss me very much to-night, I should think!’ (Dinah was the cat.) ‘I hope they’ll remember her saucer of milk at tea-time. Dinah my dear! I wish you were down here with me! There are no mice in the air, I’m afraid, but you might catch a bat, and that’s very like a mouse, you know. But do cats eat bats, I wonder?’ And here Alice began to get rather sleepy, and went on saying to herself, in a dreamy sort of way, ‘Do cats eat bats? Do cats eat bats?’ and sometimes, ‘Do bats eat cats?’ for, you see, as she couldn’t answer either question, it didn’t much matter which way she put it. She felt that she was dozing off, and had just begun to dream that she was walking hand in hand with Dinah, and saying to her very earnestly, ‘Now, Dinah, tell me the truth: did you ever eat a bat?’ when suddenly, thump! thump! down she came upon a heap of sticks and dry leaves, and the fall was over.\nAlice was not a bit hurt, and she jumped up on to her feet in a moment: she looked up, but it was all dark overhead; before her was another long passage, and the White Rabbit was still in sight, hurrying down it. There was not a moment to be lost: away went Alice like the wind, and was just in time to hear it say, as it turned a corner, ‘Oh my ears and whiskers, how late it’s getting!’ She was close behind it when she turned the corner, but the Rabbit was no longer to be seen: she found herself in a long, low hall, which was lit up by a row of lamps hanging from the roof.\nThere were doors all round the hall, but they were all locked; and when Alice had been all the way down one side and up the other, trying every door, she walked sadly down the middle, wondering how she was ever to get out again.\nSuddenly she came upon a little three-legged table, all made of solid glass; there was nothing on it except a tiny golden key, and Alice’s first thought was that it might belong to one of the doors of the hall; but, alas! either the locks were too large, or the key was too small, but at any rate it would not open any of them. However, on the second time round, she came upon a low curtain she had not noticed before, and behind it was a little door about fifteen inches high: she tried the little golden key in the lock, and to her great delight it fitted!\nAlice opened the door and found that it led into a small passage, not much larger than a rat-hole: she knelt down and looked along the passage into the loveliest garden you ever saw. How she longed to get out of that dark hall, and wander about among those beds of bright flowers and those cool fountains, but she could not even get her head through the doorway; ‘and even if my head would go through,’ thought poor Alice, ‘it would be of very little use without my shoulders. Oh, how I wish I could shut up like a telescope! I think I could, if I only knew how to begin.’ For, you see, so many out-of-the-way things had happened lately, that Alice had begun to think that very few things indeed were really impossible.\nThere seemed to be no use in waiting by the little door, so she went back to the table, half hoping she might find another key on it, or at any rate a book of rules for shutting people up like telescopes: this time she found a little bottle on it, (‘which certainly was not here before,’ said Alice,) and round the neck of the bottle was a paper label, with the words ‘DRINK ME’ beautifully printed on it in large letters.\nIt was all very well to say ‘Drink me,’ but the wise little Alice was not going to do that in a hurry. ‘No, I’ll look first,’ she said, ‘and see whether it’s marked “poison” or not’; for she had read several nice little histories about children who had got burnt, and eaten up by wild beasts and other unpleasant things, all because they would not remember the simple rules their friends had taught them: such as, that a red-hot poker will burn you if you hold it too long; and that if you cut your finger very deeply with a knife, it usually bleeds; and she had never forgotten that, if you drink much from a bottle marked ‘poison,’ it is almost certain to disagree with you, sooner or later.\nHowever, this bottle was not marked ‘poison,’ so Alice ventured to taste it, and finding it very nice, (it had, in fact, a sort of mixed flavour of cherry-tart, custard, pine-apple, roast turkey, toffee, and hot buttered toast,) she very soon finished it off.\n * * * * * * *\n\n * * * * * *\n\n * * * * * * *\n‘What a curious feeling!’ said Alice; ‘I must be shutting up like a telescope.’\nAnd so it was indeed: she was now only ten inches high, and her face brightened up at the thought that she was now the right size for going through the little door into that lovely garden. First, however, she waited for a few minutes to see if she was going to shrink any further: she felt a little nervous about this; ‘for it might end, you know,’ said Alice to herself, ‘in my going out altogether, like a candle. I wonder what I should be like then?’ And she tried to fancy what the flame of a candle is like after the candle is blown out, for she could not remember ever having seen such a thing.\nAfter a while, finding that nothing more happened, she decided on going into the garden at once; but, alas for poor Alice! when she got to the door, she found she had forgotten the little golden key, and when she went back to the table for it, she found she could not possibly reach it: she could see it quite plainly through the glass, and she tried her best to climb up one of the legs of the table, but it was too slippery; and when she had tired herself out with trying, the poor little thing sat down and cried.\n‘Come, there’s no use in crying like that!’ said Alice to herself, rather sharply; ‘I advise you to leave off this minute!’ She generally gave herself very good advice, (though she very seldom followed it), and sometimes she scolded herself so severely as to bring tears into her eyes; and once she remembered trying to box her own ears for having cheated herself in a game of croquet she was playing against herself, for this curious child was very fond of pretending to be two people. ‘But it’s no use now,’ thought poor Alice, ‘to pretend to be two people! Why, there’s hardly enough of me left to make one respectable person!’\nSoon her eye fell on a little glass box that was lying under the table: she opened it, and found in it a very small cake, on which the words ‘EAT ME’ were beautifully marked in currants. ‘Well, I’ll eat it,’ said Alice, ‘and if it makes me grow larger, I can reach the key; and if it makes me grow smaller, I can creep under the door; so either way I’ll get into the garden, and I don’t care which happens!’\nShe ate a little bit, and said anxiously to herself, ‘Which way? Which way?’, holding her hand on the top of her head to feel which way it was growing, and she was quite surprised to find that she remained the same size: to be sure, this generally happens when one eats cake, but Alice had got so much into the way of expecting nothing but out-of-the-way things to happen, that it seemed quite dull and stupid for life to go on in the common way.\nSo she set to work, and very soon finished off the cake.\n * * * * * * *\n\n * * * * * *\n\n * * * * * * *\n\n\n\n\nCHAPTER II. The Pool of Tears\n\n‘Curiouser and curiouser!’ cried Alice (she was so much surprised, that for the moment she quite forgot how to speak good English); ‘now I’m opening out like the largest telescope that ever was! Good-bye, feet!’ (for when she looked down at her feet, they seemed to be almost out of sight, they were getting so far off). ‘Oh, my poor little feet, I wonder who will put on your shoes and stockings for you now, dears? I’m sure I shan’t be able! I shall be a great deal too far off to trouble myself about you: you must manage the best way you can;—but I must be kind to them,’ thought Alice, ‘or perhaps they won’t walk the way I want to go! Let me see: I’ll give them a new pair of boots every Christmas.’\nAnd she went on planning to herself how she would manage it. ‘They must go by the carrier,’ she thought; ‘and how funny it’ll seem, sending presents to one’s own feet! And how odd the directions will look!\n Alice’s Right Foot, Esq.\n Hearthrug,\n near The Fender,\n (with Alice’s love).\nOh dear, what nonsense I’m talking!’\nJust then her head struck against the roof of the hall: in fact she was now more than nine feet high, and she at once took up the little golden key and hurried off to the garden door.\nPoor Alice! It was as much as she could do, lying down on one side, to look through into the garden with one eye; but to get through was more hopeless than ever: she sat down and began to cry again.\n‘You ought to be ashamed of yourself,’ said Alice, ‘a great girl like you,’ (she might well say this), ‘to go on crying in this way! Stop this moment, I tell you!’ But she went on all the same, shedding gallons of tears, until there was a large pool all round her, about four inches deep and reaching half down the hall.\nAfter a time she heard a little pattering of feet in the distance, and she hastily dried her eyes to see what was coming. It was the White Rabbit returning, splendidly dressed, with a pair of white kid gloves in one hand and a large fan in the other: he came trotting along in a great hurry, muttering to himself as he came, ‘Oh! the Duchess, the Duchess! Oh! won’t she be savage if I’ve kept her waiting!’ Alice felt so desperate that she was ready to ask help of any one; so, when the Rabbit came near her, she began, in a low, timid voice, ‘If you please, sir—’ The Rabbit started violently, dropped the white kid gloves and the fan, and skurried away into the darkness as hard as he could go.\nAlice took up the fan and gloves, and, as the hall was very hot, she kept fanning herself all the time she went on talking: ‘Dear, dear! How queer everything is to-day! And yesterday things went on just as usual. I wonder if I’ve been changed in the night? Let me think: was I the same when I got up this morning? I almost think I can remember feeling a little different. But if I’m not the same, the next question is, Who in the world am I? Ah, that’s the great puzzle!’ And she began thinking over all the children she knew that were of the same age as herself, to see if she could have been changed for any of them.\n‘I’m sure I’m not Ada,’ she said, ‘for her hair goes in such long ringlets, and mine doesn’t go in ringlets at all; and I’m sure I can’t be Mabel, for I know all sorts of things, and she, oh! she knows such a very little! Besides, she’s she, and I’m I, and—oh dear, how puzzling it all is! I’ll try if I know all the things I used to know. Let me see: four times five is twelve, and four times six is thirteen, and four times seven is—oh dear! I shall never get to twenty at that rate! However, the Multiplication Table doesn’t signify: let’s try Geography. London is the capital of Paris, and Paris is the capital of Rome, and Rome—no, that’s all wrong, I’m certain! I must have been changed for Mabel! I’ll try and say “How doth the little—“’ and she crossed her hands on her lap as if she were saying lessons, and began to repeat it, but her voice sounded hoarse and strange, and the words did not come the same as they used to do:—\n ‘How doth the little crocodile\n Improve his shining tail,\n And pour the waters of the Nile\n On every golden scale!\n\n ‘How cheerfully he seems to grin,\n How neatly spread his claws,\n And welcome little fishes in\n With gently smiling jaws!’\n‘I’m sure those are not the right words,’ said poor Alice, and her eyes filled with tears again as she went on, ‘I must be Mabel after all, and I shall have to go and live in that poky little house, and have next to no toys to play with, and oh! ever so many lessons to learn! No, I’ve made up my mind about it; if I’m Mabel, I’ll stay down here! It’ll be no use their putting their heads down and saying “Come up again, dear!” I shall only look up and say “Who am I then? Tell me that first, and then, if I like being that person, I’ll come up: if not, I’ll stay down here till I’m somebody else”—but, oh dear!’ cried Alice, with a sudden burst of tears, ‘I do wish they would put their heads down! I am so very tired of being all alone here!’\nAs she said this she looked down at her hands, and was surprised to see that she had put on one of the Rabbit’s little white kid gloves while she was talking. ‘How can I have done that?’ she thought. ‘I must be growing small again.’ She got up and went to the table to measure herself by it, and found that, as nearly as she could guess, she was now about two feet high, and was going on shrinking rapidly: she soon found out that the cause of this was the fan she was holding, and she dropped it hastily, just in time to avoid shrinking away altogether.\n‘That was a narrow escape!’ said Alice, a good deal frightened at the sudden change, but very glad to find herself still in existence; ‘and now for the garden!’ and she ran with all speed back to the little door: but, alas! the little door was shut again, and the little golden key was lying on the glass table as before, ‘and things are worse than ever,’ thought the poor child, ‘for I never was so small as this before, never! And I declare it’s too bad, that it is!’\nAs she said these words her foot slipped, and in another moment, splash! she was up to her chin in salt water. Her first idea was that she had somehow fallen into the sea, ‘and in that case I can go back by railway,’ she said to herself. (Alice had been to the seaside once in her life, and had come to the general conclusion, that wherever you go to on the English coast you find a number of bathing machines in the sea, some children digging in the sand with wooden spades, then a row of lodging houses, and behind them a railway station.) However, she soon made out that she was in the pool of tears which she had wept when she was nine feet high.\n‘I wish I hadn’t cried so much!’ said Alice, as she swam about, trying to find her way out. ‘I shall be punished for it now, I suppose, by being drowned in my own tears! That will be a queer thing, to be sure! However, everything is queer to-day.’\nJust then she heard something splashing about in the pool a little way off, and she swam nearer to make out what it was: at first she thought it must be a walrus or hippopotamus, but then she remembered how small she was now, and she soon made out that it was only a mouse that had slipped in like herself.\n‘Would it be of any use, now,’ thought Alice, ‘to speak to this mouse? Everything is so out-of-the-way down here, that I should think very likely it can talk: at any rate, there’s no harm in trying.’ So she began: ‘O Mouse, do you know the way out of this pool? I am very tired of swimming about here, O Mouse!’ (Alice thought this must be the right way of speaking to a mouse: she had never done such a thing before, but she remembered having seen in her brother’s Latin Grammar, ‘A mouse—of a mouse—to a mouse—a mouse—O mouse!’) The Mouse looked at her rather inquisitively, and seemed to her to wink with one of its little eyes, but it said nothing.\n‘Perhaps it doesn’t understand English,’ thought Alice; ‘I daresay it’s a French mouse, come over with William the Conqueror.’ (For, with all her knowledge of history, Alice had no very clear notion how long ago anything had happened.) So she began again: ‘Ou est ma chatte?’ which was the first sentence in her French lesson-book. The Mouse gave a sudden leap out of the water, and seemed to quiver all over with fright. ‘Oh, I beg your pardon!’ cried Alice hastily, afraid that she had hurt the poor animal’s feelings. ‘I quite forgot you didn’t like cats.’\n‘Not like cats!’ cried the Mouse, in a shrill, passionate voice. ‘Would you like cats if you were me?’\n‘Well, perhaps not,’ said Alice in a soothing tone: ‘don’t be angry about it. And yet I wish I could show you our cat Dinah: I think you’d take a fancy to cats if you could only see her. She is such a dear quiet thing,’ Alice went on, half to herself, as she swam lazily about in the pool, ‘and she sits purring so nicely by the fire, licking her paws and washing her face—and she is such a nice soft thing to nurse—and she’s such a capital one for catching mice—oh, I beg your pardon!’ cried Alice again, for this time the Mouse was bristling all over, and she felt certain it must be really offended. ‘We won’t talk about her any more if you’d rather not.’\n‘We indeed!’ cried the Mouse, who was trembling down to the end of his tail. ‘As if I would talk on such a subject! Our family always hated cats: nasty, low, vulgar things! Don’t let me hear the name again!’\n‘I won’t indeed!’ said Alice, in a great hurry to change the subject of conversation. ‘Are you—are you fond—of—of dogs?’ The Mouse did not answer, so Alice went on eagerly: ‘There is such a nice little dog near our house I should like to show you! A little bright-eyed terrier, you know, with oh, such long curly brown hair! And it’ll fetch things when you throw them, and it’ll sit up and beg for its dinner, and all sorts of things—I can’t remember half of them—and it belongs to a farmer, you know, and he says it’s so useful, it’s worth a hundred pounds! He says it kills all the rats and—oh dear!’ cried Alice in a sorrowful tone, ‘I’m afraid I’ve offended it again!’ For the Mouse was swimming away from her as hard as it could go, and making quite a commotion in the pool as it went.\nSo she called softly after it, ‘Mouse dear! Do come back again, and we won’t talk about cats or dogs either, if you don’t like them!’ When the Mouse heard this, it turned round and swam slowly back to her: its face was quite pale (with passion, Alice thought), and it said in a low trembling voice, ‘Let us get to the shore, and then I’ll tell you my history, and you’ll understand why it is I hate cats and dogs.’\nIt was high time to go, for the pool was getting quite crowded with the birds and animals that had fallen into it: there were a Duck and a Dodo, a Lory and an Eaglet, and several other curious creatures. Alice led the way, and the whole party swam to the shore.\n\n\n\n\nCHAPTER III. A Caucus-Race and a Long Tale\n\nThey were indeed a queer-looking party that assembled on the bank—the birds with draggled feathers, the animals with their fur clinging close to them, and all dripping wet, cross, and uncomfortable.\nThe first question of course was, how to get dry again: they had a consultation about this, and after a few minutes it seemed quite natural to Alice to find herself talking familiarly with them, as if she had known them all her life. Indeed, she had quite a long argument with the Lory, who at last turned sulky, and would only say, ‘I am older than you, and must know better’; and this Alice would not allow without knowing how old it was, and, as the Lory positively refused to tell its age, there was no more to be said.\nAt last the Mouse, who seemed to be a person of authority among them, called out, ‘Sit down, all of you, and listen to me! I’ll soon make you dry enough!’ They all sat down at once, in a large ring, with the Mouse in the middle. Alice kept her eyes anxiously fixed on it, for she felt sure she would catch a bad cold if she did not get dry very soon.\n‘Ahem!’ said the Mouse with an important air, ‘are you all ready? This is the driest thing I know. Silence all round, if you please! “William the Conqueror, whose cause was favoured by the pope, was soon submitted to by the English, who wanted leaders, and had been of late much accustomed to usurpation and conquest. Edwin and Morcar, the earls of Mercia and Northumbria—“’\n‘Ugh!’ said the Lory, with a shiver.\n‘I beg your pardon!’ said the Mouse, frowning, but very politely: ‘Did you speak?’\n‘Not I!’ said the Lory hastily.\n‘I thought you did,’ said the Mouse. ‘—I proceed. “Edwin and Morcar, the earls of Mercia and Northumbria, declared for him: and even Stigand, the patriotic archbishop of Canterbury, found it advisable—“’\n‘Found what?’ said the Duck.\n‘Found it,’ the Mouse replied rather crossly: ‘of course you know what “it” means.’\n‘I know what “it” means well enough, when I find a thing,’ said the Duck: ‘it’s generally a frog or a worm. The question is, what did the archbishop find?’\nThe Mouse did not notice this question, but hurriedly went on, ‘“—found it advisable to go with Edgar Atheling to meet William and offer him the crown. William’s conduct at first was moderate. But the insolence of his Normans—” How are you getting on now, my dear?’ it continued, turning to Alice as it spoke.\n‘As wet as ever,’ said Alice in a melancholy tone: ‘it doesn’t seem to dry me at all.’\n‘In that case,’ said the Dodo solemnly, rising to its feet, ‘I move that the meeting adjourn, for the immediate adoption of more energetic remedies—’\n‘Speak English!’ said the Eaglet. ‘I don’t know the meaning of half those long words, and, what’s more, I don’t believe you do either!’ And the Eaglet bent down its head to hide a smile: some of the other birds tittered audibly.\n‘What I was going to say,’ said the Dodo in an offended tone, ‘was, that the best thing to get us dry would be a Caucus-race.’\n‘What is a Caucus-race?’ said Alice; not that she wanted much to know, but the Dodo had paused as if it thought that somebody ought to speak, and no one else seemed inclined to say anything.\n‘Why,’ said the Dodo, ‘the best way to explain it is to do it.’ (And, as you might like to try the thing yourself, some winter day, I will tell you how the Dodo managed it.)\nFirst it marked out a race-course, in a sort of circle, (‘the exact shape doesn’t matter,’ it said,) and then all the party were placed along the course, here and there. There was no ‘One, two, three, and away,’ but they began running when they liked, and left off when they liked, so that it was not easy to know when the race was over. However, when they had been running half an hour or so, and were quite dry again, the Dodo suddenly called out ‘The race is over!’ and they all crowded round it, panting, and asking, ‘But who has won?’\nThis question the Dodo could not answer without a great deal of thought, and it sat for a long time with one finger pressed upon its forehead (the position in which you usually see Shakespeare, in the pictures of him), while the rest waited in silence. At last the Dodo said, ‘Everybody has won, and all must have prizes.’\n‘But who is to give the prizes?’ quite a chorus of voices asked.\n‘Why, she, of course,’ said the Dodo, pointing to Alice with one finger; and the whole party at once crowded round her, calling out in a confused way, ‘Prizes! Prizes!’\nAlice had no idea what to do, and in despair she put her hand in her pocket, and pulled out a box of comfits, (luckily the salt water had not got into it), and handed them round as prizes. There was exactly one a-piece all round.\n‘But she must have a prize herself, you know,’ said the Mouse.\n‘Of course,’ the Dodo replied very gravely. ‘What else have you got in your pocket?’ he went on, turning to Alice.\n‘Only a thimble,’ said Alice sadly.\n‘Hand it over here,’ said the Dodo.\nThen they all crowded round her once more, while the Dodo solemnly presented the thimble, saying ‘We beg your acceptance of this elegant thimble’; and, when it had finished this short speech, they all cheered.\nAlice thought the whole thing very absurd, but they all looked so grave that she did not dare to laugh; and, as she could not think of anything to say, she simply bowed, and took the thimble, looking as solemn as she could.\nThe next thing was to eat the comfits: this caused some noise and confusion, as the large birds complained that they could not taste theirs, and the small ones choked and had to be patted on the back. However, it was over at last, and they sat down again in a ring, and begged the Mouse to tell them something more.\n‘You promised to tell me your history, you know,’ said Alice, ‘and why it is you hate—C and D,’ she added in a whisper, half afraid that it would be offended again.\n‘Mine is a long and a sad tale!’ said the Mouse, turning to Alice, and sighing.\n‘It is a long tail, certainly,’ said Alice, looking down with wonder at the Mouse’s tail; ‘but why do you call it sad?’ And she kept on puzzling about it while the Mouse was speaking, so that her idea of the tale was something like this:—\n ‘Fury said to a\n mouse, That he\n met in the\n house,\n “Let us\n both go to\n law: I will\n prosecute\n you.—Come,\n I’ll take no\n denial; We\n must have a\n trial: For\n really this\n morning I’ve\n nothing\n to do.”\n Said the\n mouse to the\n cur, “Such\n a trial,\n dear Sir,\n With\n no jury\n or judge,\n would be\n wasting\n our\n breath.”\n “I’ll be\n judge, I’ll\n be jury,”\n Said\n cunning\n old Fury:\n “I’ll\n try the\n whole\n cause,\n and\n condemn\n you\n to\n death.”’\n‘You are not attending!’ said the Mouse to Alice severely. ‘What are you thinking of?’\n‘I beg your pardon,’ said Alice very humbly: ‘you had got to the fifth bend, I think?’\n‘I had not!’ cried the Mouse, sharply and very angrily.\n‘A knot!’ said Alice, always ready to make herself useful, and looking anxiously about her. ‘Oh, do let me help to undo it!’\n‘I shall do nothing of the sort,’ said the Mouse, getting up and walking away. ‘You insult me by talking such nonsense!’\n‘I didn’t mean it!’ pleaded poor Alice. ‘But you’re so easily offended, you know!’\nThe Mouse only growled in reply.\n‘Please come back and finish your story!’ Alice called after it; and the others all joined in chorus, ‘Yes, please do!’ but the Mouse only shook its head impatiently, and walked a little quicker.\n‘What a pity it wouldn’t stay!’ sighed the Lory, as soon as it was quite out of sight; and an old Crab took the opportunity of saying to her daughter ‘Ah, my dear! Let this be a lesson to you never to lose your temper!’ ‘Hold your tongue, Ma!’ said the young Crab, a little snappishly. ‘You’re enough to try the patience of an oyster!’\n‘I wish I had our Dinah here, I know I do!’ said Alice aloud, addressing nobody in particular. ‘She’d soon fetch it back!’\n‘And who is Dinah, if I might venture to ask the question?’ said the Lory.\nAlice replied eagerly, for she was always ready to talk about her pet: ‘Dinah’s our cat. And she’s such a capital one for catching mice you can’t think! And oh, I wish you could see her after the birds! Why, she’ll eat a little bird as soon as look at it!’\nThis speech caused a remarkable sensation among the party. Some of the birds hurried off at once: one old Magpie began wrapping itself up very carefully, remarking, ‘I really must be getting home; the night-air doesn’t suit my throat!’ and a Canary called out in a trembling voice to its children, ‘Come away, my dears! It’s high time you were all in bed!’ On various pretexts they all moved off, and Alice was soon left alone.\n‘I wish I hadn’t mentioned Dinah!’ she said to herself in a melancholy tone. ‘Nobody seems to like her, down here, and I’m sure she’s the best cat in the world! Oh, my dear Dinah! I wonder if I shall ever see you any more!’ And here poor Alice began to cry again, for she felt very lonely and low-spirited. In a little while, however, she again heard a little pattering of footsteps in the distance, and she looked up eagerly, half hoping that the Mouse had changed his mind, and was coming back to finish his story.\n\n\n\n\nCHAPTER IV. The Rabbit Sends in a Little Bill\n\nIt was the White Rabbit, trotting slowly back again, and looking anxiously about as it went, as if it had lost something; and she heard it muttering to itself ‘The Duchess! The Duchess! Oh my dear paws! Oh my fur and whiskers! She’ll get me executed, as sure as ferrets are ferrets! Where can I have dropped them, I wonder?’ Alice guessed in a moment that it was looking for the fan and the pair of white kid gloves, and she very good-naturedly began hunting about for them, but they were nowhere to be seen—everything seemed to have changed since her swim in the pool, and the great hall, with the glass table and the little door, had vanished completely.\nVery soon the Rabbit noticed Alice, as she went hunting about, and called out to her in an angry tone, ‘Why, Mary Ann, what are you doing out here? Run home this moment, and fetch me a pair of gloves and a fan! Quick, now!’ And Alice was so much frightened that she ran off at once in the direction it pointed to, without trying to explain the mistake it had made.\n‘He took me for his housemaid,’ she said to herself as she ran. ‘How surprised he’ll be when he finds out who I am! But I’d better take him his fan and gloves—that is, if I can find them.’ As she said this, she came upon a neat little house, on the door of which was a bright brass plate with the name ‘W. RABBIT’ engraved upon it. She went in without knocking, and hurried upstairs, in great fear lest she should meet the real Mary Ann, and be turned out of the house before she had found the fan and gloves.\n‘How queer it seems,’ Alice said to herself, ‘to be going messages for a rabbit! I suppose Dinah’ll be sending me on messages next!’ And she began fancying the sort of thing that would happen: ‘“Miss Alice! Come here directly, and get ready for your walk!” “Coming in a minute, nurse! But I’ve got to see that the mouse doesn’t get out.” Only I don’t think,’ Alice went on, ‘that they’d let Dinah stop in the house if it began ordering people about like that!’\nBy this time she had found her way into a tidy little room with a table in the window, and on it (as she had hoped) a fan and two or three pairs of tiny white kid gloves: she took up the fan and a pair of the gloves, and was just going to leave the room, when her eye fell upon a little bottle that stood near the looking-glass. There was no label this time with the words ‘DRINK ME,’ but nevertheless she uncorked it and put it to her lips. ‘I know something interesting is sure to happen,’ she said to herself, ‘whenever I eat or drink anything; so I’ll just see what this bottle does. I do hope it’ll make me grow large again, for really I’m quite tired of being such a tiny little thing!’\nIt did so indeed, and much sooner than she had expected: before she had drunk half the bottle, she found her head pressing against the ceiling, and had to stoop to save her neck from being broken. She hastily put down the bottle, saying to herself ‘That’s quite enough—I hope I shan’t grow any more—As it is, I can’t get out at the door—I do wish I hadn’t drunk quite so much!’\nAlas! it was too late to wish that! She went on growing, and growing, and very soon had to kneel down on the floor: in another minute there was not even room for this, and she tried the effect of lying down with one elbow against the door, and the other arm curled round her head. Still she went on growing, and, as a last resource, she put one arm out of the window, and one foot up the chimney, and said to herself ‘Now I can do no more, whatever happens. What will become of me?’\nLuckily for Alice, the little magic bottle had now had its full effect, and she grew no larger: still it was very uncomfortable, and, as there seemed to be no sort of chance of her ever getting out of the room again, no wonder she felt unhappy.\n‘It was much pleasanter at home,’ thought poor Alice, ‘when one wasn’t always growing larger and smaller, and being ordered about by mice and rabbits. I almost wish I hadn’t gone down that rabbit-hole—and yet—and yet—it’s rather curious, you know, this sort of life! I do wonder what can have happened to me! When I used to read fairy-tales, I fancied that kind of thing never happened, and now here I am in the middle of one! There ought to be a book written about me, that there ought! And when I grow up, I’ll write one—but I’m grown up now,’ she added in a sorrowful tone; ‘at least there’s no room to grow up any more here.’\n‘But then,’ thought Alice, ‘shall I never get any older than I am now? That’ll be a comfort, one way—never to be an old woman—but then—always to have lessons to learn! Oh, I shouldn’t like that!’\n‘Oh, you foolish Alice!’ she answered herself. ‘How can you learn lessons in here? Why, there’s hardly room for you, and no room at all for any lesson-books!’\nAnd so she went on, taking first one side and then the other, and making quite a conversation of it altogether; but after a few minutes she heard a voice outside, and stopped to listen.\n‘Mary Ann! Mary Ann!’ said the voice. ‘Fetch me my gloves this moment!’ Then came a little pattering of feet on the stairs. Alice knew it was the Rabbit coming to look for her, and she trembled till she shook the house, quite forgetting that she was now about a thousand times as large as the Rabbit, and had no reason to be afraid of it.\nPresently the Rabbit came up to the door, and tried to open it; but, as the door opened inwards, and Alice’s elbow was pressed hard against it, that attempt proved a failure. Alice heard it say to itself ‘Then I’ll go round and get in at the window.’\n‘That you won’t’ thought Alice, and, after waiting till she fancied she heard the Rabbit just under the window, she suddenly spread out her hand, and made a snatch in the air. She did not get hold of anything, but she heard a little shriek and a fall, and a crash of broken glass, from which she concluded that it was just possible it had fallen into a cucumber-frame, or something of the sort.\nNext came an angry voice—the Rabbit’s—‘Pat! Pat! Where are you?’ And then a voice she had never heard before, ‘Sure then I’m here! Digging for apples, yer honour!’\n‘Digging for apples, indeed!’ said the Rabbit angrily. ‘Here! Come and help me out of this!’ (Sounds of more broken glass.)\n‘Now tell me, Pat, what’s that in the window?’\n‘Sure, it’s an arm, yer honour!’ (He pronounced it ‘arrum.’)\n‘An arm, you goose! Who ever saw one that size? Why, it fills the whole window!’\n‘Sure, it does, yer honour: but it’s an arm for all that.’\n‘Well, it’s got no business there, at any rate: go and take it away!’\nThere was a long silence after this, and Alice could only hear whispers now and then; such as, ‘Sure, I don’t like it, yer honour, at all, at all!’ ‘Do as I tell you, you coward!’ and at last she spread out her hand again, and made another snatch in the air. This time there were two little shrieks, and more sounds of broken glass. ‘What a number of cucumber-frames there must be!’ thought Alice. ‘I wonder what they’ll do next! As for pulling me out of the window, I only wish they could! I’m sure I don’t want to stay in here any longer!’\nShe waited for some time without hearing anything more: at last came a rumbling of little cartwheels, and the sound of a good many voices all talking together: she made out the words: ‘Where’s the other ladder?—Why, I hadn’t to bring but one; Bill’s got the other—Bill! fetch it here, lad!—Here, put ‘em up at this corner—No, tie ‘em together first—they don’t reach half high enough yet—Oh! they’ll do well enough; don’t be particular—Here, Bill! catch hold of this rope—Will the roof bear?—Mind that loose slate—Oh, it’s coming down! Heads below!’ (a loud crash)—‘Now, who did that?—It was Bill, I fancy—Who’s to go down the chimney?—Nay, I shan’t! You do it!—That I won’t, then!—Bill’s to go down—Here, Bill! the master says you’re to go down the chimney!’\n‘Oh! So Bill’s got to come down the chimney, has he?’ said Alice to herself. ‘Shy, they seem to put everything upon Bill! I wouldn’t be in Bill’s place for a good deal: this fireplace is narrow, to be sure; but I think I can kick a little!’\nShe drew her foot as far down the chimney as she could, and waited till she heard a little animal (she couldn’t guess of what sort it was) scratching and scrambling about in the chimney close above her: then, saying to herself ‘This is Bill,’ she gave one sharp kick, and waited to see what would happen next.\nThe first thing she heard was a general chorus of ‘There goes Bill!’ then the Rabbit’s voice along—‘Catch him, you by the hedge!’ then silence, and then another confusion of voices—‘Hold up his head—Brandy now—Don’t choke him—How was it, old fellow? What happened to you? Tell us all about it!’\nLast came a little feeble, squeaking voice, (‘That’s Bill,’ thought Alice,) ‘Well, I hardly know—No more, thank ye; I’m better now—but I’m a deal too flustered to tell you—all I know is, something comes at me like a Jack-in-the-box, and up I goes like a sky-rocket!’\n‘So you did, old fellow!’ said the others.\n‘We must burn the house down!’ said the Rabbit’s voice; and Alice called out as loud as she could, ‘If you do. I’ll set Dinah at you!’\nThere was a dead silence instantly, and Alice thought to herself, ‘I wonder what they will do next! If they had any sense, they’d take the roof off.’ After a minute or two, they began moving about again, and Alice heard the Rabbit say, ‘A barrowful will do, to begin with.’\n‘A barrowful of what?’ thought Alice; but she had not long to doubt, for the next moment a shower of little pebbles came rattling in at the window, and some of them hit her in the face. ‘I’ll put a stop to this,’ she said to herself, and shouted out, ‘You’d better not do that again!’ which produced another dead silence.\nAlice noticed with some surprise that the pebbles were all turning into little cakes as they lay on the floor, and a bright idea came into her head. ‘If I eat one of these cakes,’ she thought, ‘it’s sure to make some change in my size; and as it can’t possibly make me larger, it must make me smaller, I suppose.’\nSo she swallowed one of the cakes, and was delighted to find that she began shrinking directly. As soon as she was small enough to get through the door, she ran out of the house, and found quite a crowd of little animals and birds waiting outside. The poor little Lizard, Bill, was in the middle, being held up by two guinea-pigs, who were giving it something out of a bottle. They all made a rush at Alice the moment she appeared; but she ran off as hard as she could, and soon found herself safe in a thick wood.\n‘The first thing I’ve got to do,’ said Alice to herself, as she wandered about in the wood, ‘is to grow to my right size again; and the second thing is to find my way into that lovely garden. I think that will be the best plan.’\nIt sounded an excellent plan, no doubt, and very neatly and simply arranged; the only difficulty was, that she had not the smallest idea how to set about it; and while she was peering about anxiously among the trees, a little sharp bark just over her head made her look up in a great hurry.\nAn enormous puppy was looking down at her with large round eyes, and feebly stretching out one paw, trying to touch her. ‘Poor little thing!’ said Alice, in a coaxing tone, and she tried hard to whistle to it; but she was terribly frightened all the time at the thought that it might be hungry, in which case it would be very likely to eat her up in spite of all her coaxing.\nHardly knowing what she did, she picked up a little bit of stick, and held it out to the puppy; whereupon the puppy jumped into the air off all its feet at once, with a yelp of delight, and rushed at the stick, and made believe to worry it; then Alice dodged behind a great thistle, to keep herself from being run over; and the moment she appeared on the other side, the puppy made another rush at the stick, and tumbled head over heels in its hurry to get hold of it; then Alice, thinking it was very like having a game of play with a cart-horse, and expecting every moment to be trampled under its feet, ran round the thistle again; then the puppy began a series of short charges at the stick, running a very little way forwards each time and a long way back, and barking hoarsely all the while, till at last it sat down a good way off, panting, with its tongue hanging out of its mouth, and its great eyes half shut.\nThis seemed to Alice a good opportunity for making her escape; so she set off at once, and ran till she was quite tired and out of breath, and till the puppy’s bark sounded quite faint in the distance.\n‘And yet what a dear little puppy it was!’ said Alice, as she leant against a buttercup to rest herself, and fanned herself with one of the leaves: ‘I should have liked teaching it tricks very much, if—if I’d only been the right size to do it! Oh dear! I’d nearly forgotten that I’ve got to grow up again! Let me see—how is it to be managed? I suppose I ought to eat or drink something or other; but the great question is, what?’\nThe great question certainly was, what? Alice looked all round her at the flowers and the blades of grass, but she did not see anything that looked like the right thing to eat or drink under the circumstances. There was a large mushroom growing near her, about the same height as herself; and when she had looked under it, and on both sides of it, and behind it, it occurred to her that she might as well look and see what was on the top of it.\nShe stretched herself up on tiptoe, and peeped over the edge of the mushroom, and her eyes immediately met those of a large caterpillar, that was sitting on the top with its arms folded, quietly smoking a long hookah, and taking not the smallest notice of her or of anything else.\n\n\n\n\nCHAPTER V. Advice from a Caterpillar\n\nThe Caterpillar and Alice looked at each other for some time in silence: at last the Caterpillar took the hookah out of its mouth, and addressed her in a languid, sleepy voice.\n‘Who are you?’ said the Caterpillar.\nThis was not an encouraging opening for a conversation. Alice replied, rather shyly, ‘I—I hardly know, sir, just at present—at least I know who I was when I got up this morning, but I think I must have been changed several times since then.’\n‘What do you mean by that?’ said the Caterpillar sternly. ‘Explain yourself!’\n‘I can’t explain myself, I’m afraid, sir’ said Alice, ‘because I’m not myself, you see.’\n‘I don’t see,’ said the Caterpillar.\n‘I’m afraid I can’t put it more clearly,’ Alice replied very politely, ‘for I can’t understand it myself to begin with; and being so many different sizes in a day is very confusing.’\n‘It isn’t,’ said the Caterpillar.\n‘Well, perhaps you haven’t found it so yet,’ said Alice; ‘but when you have to turn into a chrysalis—you will some day, you know—and then after that into a butterfly, I should think you’ll feel it a little queer, won’t you?’\n‘Not a bit,’ said the Caterpillar.\n‘Well, perhaps your feelings may be different,’ said Alice; ‘all I know is, it would feel very queer to me.’\n‘You!’ said the Caterpillar contemptuously. ‘Who are you?’\nWhich brought them back again to the beginning of the conversation. Alice felt a little irritated at the Caterpillar’s making such very short remarks, and she drew herself up and said, very gravely, ‘I think, you ought to tell me who you are, first.’\n‘Why?’ said the Caterpillar.\nHere was another puzzling question; and as Alice could not think of any good reason, and as the Caterpillar seemed to be in a very unpleasant state of mind, she turned away.\n‘Come back!’ the Caterpillar called after her. ‘I’ve something important to say!’\nThis sounded promising, certainly: Alice turned and came back again.\n‘Keep your temper,’ said the Caterpillar.\n‘Is that all?’ said Alice, swallowing down her anger as well as she could.\n‘No,’ said the Caterpillar.\nAlice thought she might as well wait, as she had nothing else to do, and perhaps after all it might tell her something worth hearing. For some minutes it puffed away without speaking, but at last it unfolded its arms, took the hookah out of its mouth again, and said, ‘So you think you’re changed, do you?’\n‘I’m afraid I am, sir,’ said Alice; ‘I can’t remember things as I used—and I don’t keep the same size for ten minutes together!’\n‘Can’t remember what things?’ said the Caterpillar.\n‘Well, I’ve tried to say “How doth the little busy bee,” but it all came different!’ Alice replied in a very melancholy voice.\n‘Repeat, “You are old, Father William,”’ said the Caterpillar.\nAlice folded her hands, and began:—\n ‘You are old, Father William,’ the young man said,\n ‘And your hair has become very white;\n And yet you incessantly stand on your head—\n Do you think, at your age, it is right?’\n\n ‘In my youth,’ Father William replied to his son,\n ‘I feared it might injure the brain;\n But, now that I’m perfectly sure I have none,\n Why, I do it again and again.’\n\n ‘You are old,’ said the youth, ‘as I mentioned before,\n And have grown most uncommonly fat;\n Yet you turned a back-somersault in at the door—\n Pray, what is the reason of that?’\n\n ‘In my youth,’ said the sage, as he shook his grey locks,\n ‘I kept all my limbs very supple\n By the use of this ointment—one shilling the box—\n Allow me to sell you a couple?’\n\n ‘You are old,’ said the youth, ‘and your jaws are too weak\n For anything tougher than suet;\n Yet you finished the goose, with the bones and the beak—\n Pray how did you manage to do it?’\n\n ‘In my youth,’ said his father, ‘I took to the law,\n And argued each case with my wife;\n And the muscular strength, which it gave to my jaw,\n Has lasted the rest of my life.’\n\n ‘You are old,’ said the youth, ‘one would hardly suppose\n That your eye was as steady as ever;\n Yet you balanced an eel on the end of your nose—\n What made you so awfully clever?’\n\n ‘I have answered three questions, and that is enough,’\n Said his father; ‘don’t give yourself airs!\n Do you think I can listen all day to such stuff?\n Be off, or I’ll kick you down stairs!’\n‘That is not said right,’ said the Caterpillar.\n‘Not quite right, I’m afraid,’ said Alice, timidly; ‘some of the words have got altered.’\n‘It is wrong from beginning to end,’ said the Caterpillar decidedly, and there was silence for some minutes.\nThe Caterpillar was the first to speak.\n‘What size do you want to be?’ it asked.\n‘Oh, I’m not particular as to size,’ Alice hastily replied; ‘only one doesn’t like changing so often, you know.’\n‘I don’t know,’ said the Caterpillar.\nAlice said nothing: she had never been so much contradicted in her life before, and she felt that she was losing her temper.\n‘Are you content now?’ said the Caterpillar.\n‘Well, I should like to be a little larger, sir, if you wouldn’t mind,’ said Alice: ‘three inches is such a wretched height to be.’\n‘It is a very good height indeed!’ said the Caterpillar angrily, rearing itself upright as it spoke (it was exactly three inches high).\n‘But I’m not used to it!’ pleaded poor Alice in a piteous tone. And she thought of herself, ‘I wish the creatures wouldn’t be so easily offended!’\n‘You’ll get used to it in time,’ said the Caterpillar; and it put the hookah into its mouth and began smoking again.\nThis time Alice waited patiently until it chose to speak again. In a minute or two the Caterpillar took the hookah out of its mouth and yawned once or twice, and shook itself. Then it got down off the mushroom, and crawled away in the grass, merely remarking as it went, ‘One side will make you grow taller, and the other side will make you grow shorter.’\n‘One side of what? The other side of what?’ thought Alice to herself.\n‘Of the mushroom,’ said the Caterpillar, just as if she had asked it aloud; and in another moment it was out of sight.\nAlice remained looking thoughtfully at the mushroom for a minute, trying to make out which were the two sides of it; and as it was perfectly round, she found this a very difficult question. However, at last she stretched her arms round it as far as they would go, and broke off a bit of the edge with each hand.\n‘And now which is which?’ she said to herself, and nibbled a little of the right-hand bit to try the effect: the next moment she felt a violent blow underneath her chin: it had struck her foot!\nShe was a good deal frightened by this very sudden change, but she felt that there was no time to be lost, as she was shrinking rapidly; so she set to work at once to eat some of the other bit. Her chin was pressed so closely against her foot, that there was hardly room to open her mouth; but she did it at last, and managed to swallow a morsel of the lefthand bit.\n * * * * * * *\n\n * * * * * *\n\n * * * * * * *\n‘Come, my head’s free at last!’ said Alice in a tone of delight, which changed into alarm in another moment, when she found that her shoulders were nowhere to be found: all she could see, when she looked down, was an immense length of neck, which seemed to rise like a stalk out of a sea of green leaves that lay far below her.\n‘What can all that green stuff be?’ said Alice. ‘And where have my shoulders got to? And oh, my poor hands, how is it I can’t see you?’ She was moving them about as she spoke, but no result seemed to follow, except a little shaking among the distant green leaves.\nAs there seemed to be no chance of getting her hands up to her head, she tried to get her head down to them, and was delighted to find that her neck would bend about easily in any direction, like a serpent. She had just succeeded in curving it down into a graceful zigzag, and was going to dive in among the leaves, which she found to be nothing but the tops of the trees under which she had been wandering, when a sharp hiss made her draw back in a hurry: a large pigeon had flown into her face, and was beating her violently with its wings.\n‘Serpent!’ screamed the Pigeon.\n‘I’m not a serpent!’ said Alice indignantly. ‘Let me alone!’\n‘Serpent, I say again!’ repeated the Pigeon, but in a more subdued tone, and added with a kind of sob, ‘I’ve tried every way, and nothing seems to suit them!’\n‘I haven’t the least idea what you’re talking about,’ said Alice.\n‘I’ve tried the roots of trees, and I’ve tried banks, and I’ve tried hedges,’ the Pigeon went on, without attending to her; ‘but those serpents! There’s no pleasing them!’\nAlice was more and more puzzled, but she thought there was no use in saying anything more till the Pigeon had finished.\n‘As if it wasn’t trouble enough hatching the eggs,’ said the Pigeon; ‘but I must be on the look-out for serpents night and day! Why, I haven’t had a wink of sleep these three weeks!’\n‘I’m very sorry you’ve been annoyed,’ said Alice, who was beginning to see its meaning.\n‘And just as I’d taken the highest tree in the wood,’ continued the Pigeon, raising its voice to a shriek, ‘and just as I was thinking I should be free of them at last, they must needs come wriggling down from the sky! Ugh, Serpent!’\n‘But I’m not a serpent, I tell you!’ said Alice. ‘I’m a—I’m a—’\n‘Well! What are you?’ said the Pigeon. ‘I can see you’re trying to invent something!’\n‘I—I’m a little girl,’ said Alice, rather doubtfully, as she remembered the number of changes she had gone through that day.\n‘A likely story indeed!’ said the Pigeon in a tone of the deepest contempt. ‘I’ve seen a good many little girls in my time, but never one with such a neck as that! No, no! You’re a serpent; and there’s no use denying it. I suppose you’ll be telling me next that you never tasted an egg!’\n‘I have tasted eggs, certainly,’ said Alice, who was a very truthful child; ‘but little girls eat eggs quite as much as serpents do, you know.’\n‘I don’t believe it,’ said the Pigeon; ‘but if they do, why then they’re a kind of serpent, that’s all I can say.’\nThis was such a new idea to Alice, that she was quite silent for a minute or two, which gave the Pigeon the opportunity of adding, ‘You’re looking for eggs, I know that well enough; and what does it matter to me whether you’re a little girl or a serpent?’\n‘It matters a good deal to me,’ said Alice hastily; ‘but I’m not looking for eggs, as it happens; and if I was, I shouldn’t want yours: I don’t like them raw.’\n‘Well, be off, then!’ said the Pigeon in a sulky tone, as it settled down again into its nest. Alice crouched down among the trees as well as she could, for her neck kept getting entangled among the branches, and every now and then she had to stop and untwist it. After a while she remembered that she still held the pieces of mushroom in her hands, and she set to work very carefully, nibbling first at one and then at the other, and growing sometimes taller and sometimes shorter, until she had succeeded in bringing herself down to her usual height.\nIt was so long since she had been anything near the right size, that it felt quite strange at first; but she got used to it in a few minutes, and began talking to herself, as usual. ‘Come, there’s half my plan done now! How puzzling all these changes are! I’m never sure what I’m going to be, from one minute to another! However, I’ve got back to my right size: the next thing is, to get into that beautiful garden—how is that to be done, I wonder?’ As she said this, she came suddenly upon an open place, with a little house in it about four feet high. ‘Whoever lives there,’ thought Alice, ‘it’ll never do to come upon them this size: why, I should frighten them out of their wits!’ So she began nibbling at the righthand bit again, and did not venture to go near the house till she had brought herself down to nine inches high.\n\n\n\n\n`\n" }, { "path": "go/spec/absolute_path.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage spec\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"regexp\"\n\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\nvar datasetCapturePrefixRe = regexp.MustCompile(\"^(\" + datas.DatasetRe.String() + \")\")\n\n// AbsolutePath describes the location of a Value within a Noms database.\n//\n// To locate a value relative to some other value, see Path. To locate a value\n// globally, see Spec.\n//\n// For more on paths, absolute paths, and specs, see:\n// https://github.com/attic-labs/noms/blob/master/doc/spelling.md.\ntype AbsolutePath struct {\n\t// Dataset is the dataset this AbsolutePath is rooted at. Only one of\n\t// Dataset and Hash should be set.\n\tDataset string\n\t// Hash is the hash this AbsolutePath is rooted at. Only one of Dataset and\n\t// Hash should be set.\n\tHash hash.Hash\n\t// Path is the relative path from Dataset or Hash. This can be empty. In\n\t// that case, the AbsolutePath describes the value at either Dataset or\n\t// Hash.\n\tPath types.Path\n}\n\n// NewAbsolutePath attempts to parse 'str' and return an AbsolutePath.\nfunc NewAbsolutePath(str string) (AbsolutePath, error) {\n\tif len(str) == 0 {\n\t\treturn AbsolutePath{}, errors.New(\"Empty path\")\n\t}\n\n\tvar h hash.Hash\n\tvar dataset string\n\tvar pathStr string\n\n\tif str[0] == '#' {\n\t\ttail := str[1:]\n\t\tif len(tail) < hash.StringLen {\n\t\t\treturn AbsolutePath{}, errors.New(\"Invalid hash: \" + tail)\n\t\t}\n\n\t\thashStr := tail[:hash.StringLen]\n\t\tif h2, ok := hash.MaybeParse(hashStr); ok {\n\t\t\th = h2\n\t\t} else {\n\t\t\treturn AbsolutePath{}, errors.New(\"Invalid hash: \" + hashStr)\n\t\t}\n\n\t\tpathStr = tail[hash.StringLen:]\n\t} else {\n\t\tdatasetParts := datasetCapturePrefixRe.FindStringSubmatch(str)\n\t\tif datasetParts == nil {\n\t\t\treturn AbsolutePath{}, fmt.Errorf(\"Invalid dataset name: %s\", str)\n\t\t}\n\n\t\tdataset = datasetParts[1]\n\t\tpathStr = str[len(dataset):]\n\t}\n\n\tif len(pathStr) == 0 {\n\t\treturn AbsolutePath{Hash: h, Dataset: dataset}, nil\n\t}\n\n\tpath, err := types.ParsePath(pathStr)\n\tif err != nil {\n\t\treturn AbsolutePath{}, err\n\t}\n\n\treturn AbsolutePath{Hash: h, Dataset: dataset, Path: path}, nil\n}\n\n// Resolve returns the Value reachable by 'p' in 'db'.\nfunc (p AbsolutePath) Resolve(db datas.Database) (val types.Value) {\n\tif len(p.Dataset) > 0 {\n\t\tvar ok bool\n\t\tds := db.GetDataset(p.Dataset)\n\t\tif val, ok = ds.MaybeHead(); !ok {\n\t\t\tval = nil\n\t\t}\n\t} else if !p.Hash.IsEmpty() {\n\t\tval = db.ReadValue(p.Hash)\n\t} else {\n\t\tpanic(\"Unreachable\")\n\t}\n\n\tif val != nil && p.Path != nil {\n\t\tval = p.Path.Resolve(val, db)\n\t}\n\treturn\n}\n\nfunc (p AbsolutePath) IsEmpty() bool {\n\treturn p.Dataset == \"\" && p.Hash.IsEmpty()\n}\n\nfunc (p AbsolutePath) String() (str string) {\n\tif p.IsEmpty() {\n\t\treturn \"\"\n\t}\n\n\tif len(p.Dataset) > 0 {\n\t\tstr = p.Dataset\n\t} else if !p.Hash.IsEmpty() {\n\t\tstr = \"#\" + p.Hash.String()\n\t} else {\n\t\tpanic(\"Unreachable\")\n\t}\n\n\treturn str + p.Path.String()\n}\n\n// ReadAbsolutePaths attempts to parse each path in 'paths' and resolve them.\n// If any path fails to parse correctly or if any path can be resolved to an\n// existing Noms Value, then this function returns (nil, error).\nfunc ReadAbsolutePaths(db datas.Database, paths ...string) ([]types.Value, error) {\n\tr := make([]types.Value, 0, len(paths))\n\tfor _, ps := range paths {\n\t\tp, err := NewAbsolutePath(ps)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"Invalid input path '%s'\", ps)\n\t\t}\n\n\t\tv := p.Resolve(db)\n\t\tif v == nil {\n\t\t\treturn nil, fmt.Errorf(\"Input path '%s' does not exist in database\", ps)\n\t\t}\n\n\t\tr = append(r, v)\n\t}\n\treturn r, nil\n}\n" }, { "path": "go/spec/absolute_path_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage spec\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestAbsolutePathToAndFromString(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttest := func(str string) {\n\t\tp, err := NewAbsolutePath(str)\n\t\tassert.NoError(err)\n\t\tassert.Equal(str, p.String())\n\t}\n\n\th := types.Number(42).Hash() // arbitrary hash\n\ttest(fmt.Sprintf(\"foo.bar[#%s]\", h.String()))\n\ttest(fmt.Sprintf(\"#%s.bar[42]\", h.String()))\n}\n\nfunc TestAbsolutePaths(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\n\ts0, s1 := types.String(\"foo\"), types.String(\"bar\")\n\tlist := types.NewList(db, s0, s1)\n\temptySet := types.NewSet(db)\n\n\tdb.WriteValue(s0)\n\tdb.WriteValue(s1)\n\tdb.WriteValue(list)\n\tdb.WriteValue(emptySet)\n\n\tvar err error\n\tds := db.GetDataset(\"ds\")\n\tds, err = db.CommitValue(ds, list)\n\tassert.NoError(err)\n\thead := ds.Head()\n\n\tresolvesTo := func(exp types.Value, str string) {\n\t\tp, err := NewAbsolutePath(str)\n\t\tassert.NoError(err)\n\t\tact := p.Resolve(db)\n\t\tif exp == nil {\n\t\t\tassert.Nil(act)\n\t\t} else {\n\t\t\tassert.True(exp.Equals(act), \"%s Expected %s Actual %s\", str, types.EncodedValue(exp), types.EncodedValue(act))\n\t\t}\n\t}\n\n\tresolvesTo(head, \"ds\")\n\tresolvesTo(emptySet, \"ds.parents\")\n\tresolvesTo(list, \"ds.value\")\n\tresolvesTo(s0, \"ds.value[0]\")\n\tresolvesTo(s1, \"ds.value[1]\")\n\tresolvesTo(head, \"#\"+head.Hash().String())\n\tresolvesTo(list, \"#\"+list.Hash().String())\n\tresolvesTo(s0, \"#\"+s0.Hash().String())\n\tresolvesTo(s1, \"#\"+s1.Hash().String())\n\tresolvesTo(s0, \"#\"+list.Hash().String()+\"[0]\")\n\tresolvesTo(s1, \"#\"+list.Hash().String()+\"[1]\")\n\n\tresolvesTo(nil, \"foo\")\n\tresolvesTo(nil, \"foo.parents\")\n\tresolvesTo(nil, \"foo.value\")\n\tresolvesTo(nil, \"foo.value[0]\")\n\tresolvesTo(nil, \"#\"+types.String(\"baz\").Hash().String())\n\tresolvesTo(nil, \"#\"+types.String(\"baz\").Hash().String()+\"[0]\")\n}\n\nfunc TestReadAbsolutePaths(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\n\ts0, s1 := types.String(\"foo\"), types.String(\"bar\")\n\tlist := types.NewList(db, s0, s1)\n\n\tds := db.GetDataset(\"ds\")\n\t_, err := db.CommitValue(ds, list)\n\tassert.NoError(err)\n\n\tvals, err := ReadAbsolutePaths(db, \"ds.value[0]\", \"ds.value[1]\")\n\tassert.NoError(err)\n\n\tassert.Equal(2, len(vals))\n\tassert.Equal(\"foo\", string(vals[0].(types.String)))\n\tassert.Equal(\"bar\", string(vals[1].(types.String)))\n\n\tvals, err = ReadAbsolutePaths(db, \"!!#\")\n\tassert.Nil(vals)\n\tassert.Equal(\"Invalid input path '!!#'\", err.Error())\n\n\tvals, err = ReadAbsolutePaths(db, \"invalid.monkey\")\n\tassert.Nil(vals)\n\tassert.Equal(\"Input path 'invalid.monkey' does not exist in database\", err.Error())\n}\n\nfunc TestAbsolutePathParseErrors(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttest := func(path, errMsg string) {\n\t\tp, err := NewAbsolutePath(path)\n\t\tassert.Equal(AbsolutePath{}, p)\n\t\tassert.Error(err)\n\t\tassert.Equal(errMsg, err.Error())\n\t}\n\n\ttest(\"\", \"Empty path\")\n\ttest(\".foo\", \"Invalid dataset name: .foo\")\n\ttest(\".foo.bar.baz\", \"Invalid dataset name: .foo.bar.baz\")\n\ttest(\"#\", \"Invalid hash: \")\n\ttest(\"#abc\", \"Invalid hash: abc\")\n\tinvHash := strings.Repeat(\"z\", hash.StringLen)\n\ttest(\"#\"+invHash, \"Invalid hash: \"+invHash)\n}\n" }, { "path": "go/spec/commit_meta.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage spec\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\nconst CommitMetaDateFormat = time.RFC3339\n\nvar (\n\tcommitMetaDate string\n\tcommitMetaMessage string\n\tcommitMetaKeyValueStrings string\n\tcommitMetaKeyValuePaths string\n)\n\n// CreateCommitMetaStruct creates and returns a Noms struct suitable for use in CommitOptions.Meta.\n// It returns types.EmptyStruct and an error if any issues are encountered.\n// Database is used only if commitMetaKeyValuePaths are provided on the command line and values need to be resolved.\n// Date should be ISO 8601 format (see CommitMetaDateFormat), if empty the current date is used.\n// The values passed as command line arguments (if any) are merged with the values provided as function arguments.\nfunc CreateCommitMetaStruct(db datas.Database, date, message string, keyValueStrings map[string]string, keyValuePaths map[string]types.Value) (types.Struct, error) {\n\tmetaValues := types.StructData{}\n\n\tresolvePathFunc := func(path string) (types.Value, error) {\n\t\tabsPath, err := NewAbsolutePath(path)\n\t\tif err != nil {\n\t\t\treturn nil, errors.New(fmt.Sprintf(\"Bad path for meta-p: %s\", path))\n\t\t}\n\t\treturn absPath.Resolve(db), nil\n\t}\n\tparseMetaStrings := func(param string, resolveAsPaths bool) error {\n\t\tif param == \"\" {\n\t\t\treturn nil\n\t\t}\n\t\tms := strings.Split(param, \",\")\n\t\tfor _, m := range ms {\n\t\t\tkv := strings.Split(m, \"=\")\n\t\t\tif len(kv) != 2 {\n\t\t\t\treturn errors.New(fmt.Sprintf(\"Unable to parse meta value: %s\", m))\n\t\t\t}\n\t\t\tif !types.IsValidStructFieldName(kv[0]) {\n\t\t\t\treturn errors.New(fmt.Sprintf(\"Invalid meta key: %s\", kv[0]))\n\t\t\t}\n\t\t\tif resolveAsPaths {\n\t\t\t\tv, err := resolvePathFunc(kv[1])\n\t\t\t\tif err != nil {\n\t\t\t\t\treturn err\n\t\t\t\t}\n\t\t\t\tmetaValues[kv[0]] = v\n\t\t\t} else {\n\t\t\t\tmetaValues[kv[0]] = types.String(kv[1])\n\t\t\t}\n\t\t}\n\t\treturn nil\n\t}\n\n\tif err := parseMetaStrings(commitMetaKeyValueStrings, false); err != nil {\n\t\treturn types.EmptyStruct, err\n\t}\n\tif err := parseMetaStrings(commitMetaKeyValuePaths, true); err != nil {\n\t\treturn types.EmptyStruct, err\n\t}\n\n\tfor k, v := range keyValueStrings {\n\t\tif !types.IsValidStructFieldName(k) {\n\t\t\treturn types.EmptyStruct, errors.New(fmt.Sprintf(\"Invalid meta key: %s\", k))\n\t\t}\n\t\tmetaValues[k] = types.String(v)\n\t}\n\tfor k, v := range keyValuePaths {\n\t\tif !types.IsValidStructFieldName(k) {\n\t\t\treturn types.EmptyStruct, errors.New(fmt.Sprintf(\"Invalid meta key: %s\", k))\n\t\t}\n\t\tmetaValues[k] = v\n\t}\n\n\tif date == \"\" {\n\t\tdate = commitMetaDate\n\t}\n\tif date == \"\" {\n\t\tdate = time.Now().UTC().Format(CommitMetaDateFormat)\n\t} else {\n\t\t_, err := time.Parse(CommitMetaDateFormat, date)\n\t\tif err != nil {\n\t\t\treturn types.EmptyStruct, errors.New(fmt.Sprintf(\"Unable to parse date: %s, error: %s\", date, err))\n\t\t}\n\t}\n\tmetaValues[\"date\"] = types.String(date)\n\n\tif message != \"\" {\n\t\tmetaValues[\"message\"] = types.String(message)\n\t} else if commitMetaMessage != \"\" {\n\t\tmetaValues[\"message\"] = types.String(commitMetaMessage)\n\t}\n\treturn types.NewStruct(\"Meta\", metaValues), nil\n}\n" }, { "path": "go/spec/commit_meta_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage spec\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc isEmptyStruct(s types.Struct) bool {\n\treturn s.Equals(types.EmptyStruct)\n}\n\nfunc TestCreateCommitMetaStructBasic(t *testing.T) {\n\tassert := assert.New(t)\n\n\tmeta, err := CreateCommitMetaStruct(nil, \"\", \"\", nil, nil)\n\tassert.NoError(err)\n\tassert.False(isEmptyStruct(meta))\n\tassert.Equal(\"Struct Meta {\\n date: String,\\n}\", types.TypeOf(meta).Describe())\n}\n\nfunc TestCreateCommitMetaStructFromFlags(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsetCommitMetaFlags(time.Now().UTC().Format(CommitMetaDateFormat), \"this is a message\", \"k1=v1,k2=v2,k3=v3\")\n\tdefer resetCommitMetaFlags()\n\n\tmeta, err := CreateCommitMetaStruct(nil, \"\", \"\", nil, nil)\n\tassert.NoError(err)\n\tassert.Equal(\"Struct Meta {\\n date: String,\\n k1: String,\\n k2: String,\\n k3: String,\\n message: String,\\n}\",\n\t\ttypes.TypeOf(meta).Describe())\n\tassert.Equal(types.String(commitMetaDate), meta.Get(\"date\"))\n\tassert.Equal(types.String(commitMetaMessage), meta.Get(\"message\"))\n\tassert.Equal(types.String(\"v1\"), meta.Get(\"k1\"))\n\tassert.Equal(types.String(\"v2\"), meta.Get(\"k2\"))\n\tassert.Equal(types.String(\"v3\"), meta.Get(\"k3\"))\n}\n\nfunc TestCreateCommitMetaStructFromArgs(t *testing.T) {\n\tassert := assert.New(t)\n\n\tdateArg := time.Now().UTC().Format(CommitMetaDateFormat)\n\tmessageArg := \"this is a message\"\n\tkeyValueArg := map[string]string{\"k1\": \"v1\", \"k2\": \"v2\", \"k3\": \"v3\"}\n\tmeta, err := CreateCommitMetaStruct(nil, dateArg, messageArg, keyValueArg, nil)\n\tassert.NoError(err)\n\tassert.Equal(\"Struct Meta {\\n date: String,\\n k1: String,\\n k2: String,\\n k3: String,\\n message: String,\\n}\",\n\t\ttypes.TypeOf(meta).Describe())\n\tassert.Equal(types.String(dateArg), meta.Get(\"date\"))\n\tassert.Equal(types.String(messageArg), meta.Get(\"message\"))\n\tassert.Equal(types.String(\"v1\"), meta.Get(\"k1\"))\n\tassert.Equal(types.String(\"v2\"), meta.Get(\"k2\"))\n\tassert.Equal(types.String(\"v3\"), meta.Get(\"k3\"))\n}\n\nfunc TestCreateCommitMetaStructFromFlagsAndArgs(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsetCommitMetaFlags(time.Now().UTC().Format(CommitMetaDateFormat), \"this is a message\", \"k1=v1p1,k2=v2p2,k4=v4p4\")\n\tdefer resetCommitMetaFlags()\n\n\tdateArg := time.Now().UTC().Add(time.Hour * -24).Format(CommitMetaDateFormat)\n\tmessageArg := \"this is a message\"\n\tkeyValueArg := map[string]string{\"k1\": \"v1\", \"k2\": \"v2\", \"k3\": \"v3\"}\n\n\t// args passed in should win over the ones in the flags\n\tmeta, err := CreateCommitMetaStruct(nil, dateArg, messageArg, keyValueArg, nil)\n\tassert.NoError(err)\n\tassert.Equal(\"Struct Meta {\\n date: String,\\n k1: String,\\n k2: String,\\n k3: String,\\n k4: String,\\n message: String,\\n}\",\n\t\ttypes.TypeOf(meta).Describe())\n\tassert.Equal(types.String(dateArg), meta.Get(\"date\"))\n\tassert.Equal(types.String(messageArg), meta.Get(\"message\"))\n\tassert.Equal(types.String(\"v1\"), meta.Get(\"k1\"))\n\tassert.Equal(types.String(\"v2\"), meta.Get(\"k2\"))\n\tassert.Equal(types.String(\"v3\"), meta.Get(\"k3\"))\n\tassert.Equal(types.String(\"v4p4\"), meta.Get(\"k4\"))\n}\n\nfunc TestCreateCommitMetaStructBadDate(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttestBadDates := func(cliDateString, argDateString string) {\n\t\tsetCommitMetaFlags(cliDateString, \"\", \"\")\n\t\tdefer resetCommitMetaFlags()\n\n\t\tmeta, err := CreateCommitMetaStruct(nil, argDateString, \"\", nil, nil)\n\t\tassert.Error(err)\n\t\tassert.True(strings.HasPrefix(err.Error(), \"Unable to parse date: \"))\n\t\tassert.True(isEmptyStruct(meta))\n\t}\n\ttestBadDateMultipleWays := func(dateString string) {\n\t\ttestBadDates(dateString, \"\")\n\t\ttestBadDates(\"\", dateString)\n\t\ttestBadDates(dateString, dateString)\n\t}\n\n\ttestBadDateMultipleWays(time.Now().UTC().Format(\"Jan _2 15:04:05 2006\"))\n\ttestBadDateMultipleWays(time.Now().UTC().Format(\"Mon Jan _2 15:04:05 2006\"))\n\ttestBadDateMultipleWays(time.Now().UTC().Format(\"2006-01-02T15:04:05\"))\n}\n\nfunc TestCreateCommitMetaStructBadMetaStrings(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttestBadMetaSeparator := func(k, v, sep string) {\n\t\tsetCommitMetaFlags(\"\", \"\", fmt.Sprintf(\"%s%s%s\", k, sep, v))\n\t\tdefer resetCommitMetaFlags()\n\n\t\tmeta, err := CreateCommitMetaStruct(nil, \"\", \"\", nil, nil)\n\t\tassert.Error(err)\n\t\tassert.True(strings.HasPrefix(err.Error(), \"Unable to parse meta value: \"))\n\t\tassert.True(isEmptyStruct(meta))\n\t}\n\n\ttestBadMetaKeys := func(k, v string) {\n\t\ttestBadMetaSeparator(k, v, \":\")\n\t\ttestBadMetaSeparator(k, v, \"-\")\n\n\t\tsetCommitMetaFlags(\"\", \"\", fmt.Sprintf(\"%s=%s\", k, v))\n\n\t\tmeta, err := CreateCommitMetaStruct(nil, \"\", \"\", nil, nil)\n\t\tassert.Error(err)\n\t\tassert.True(strings.HasPrefix(err.Error(), \"Invalid meta key: \"))\n\t\tassert.True(isEmptyStruct(meta))\n\n\t\tresetCommitMetaFlags()\n\n\t\tmetaValues := map[string]string{k: v}\n\t\tmeta, err = CreateCommitMetaStruct(nil, \"\", \"\", metaValues, nil)\n\t\tassert.Error(err)\n\t\tassert.True(strings.HasPrefix(err.Error(), \"Invalid meta key: \"))\n\t\tassert.True(isEmptyStruct(meta))\n\t}\n\n\t// Valid names must start with `a-zA-Z` and after that `a-zA-Z0-9_`.\n\ttestBadMetaKeys(\"_name\", \"value\")\n\ttestBadMetaKeys(\"99problems\", \"now 100\")\n\ttestBadMetaKeys(\"one-hundred-bottles\", \"take one down\")\n\ttestBadMetaKeys(\"👀\", \"who watches the watchers?\")\n\ttestBadMetaKeys(\"key:\", \"value\")\n}\n\nfunc setCommitMetaFlags(date, message, kvStrings string) {\n\tcommitMetaDate = date\n\tcommitMetaMessage = message\n\tcommitMetaKeyValueStrings = kvStrings\n}\n\nfunc resetCommitMetaFlags() {\n\tsetCommitMetaFlags(\"\", \"\", \"\")\n}\n" }, { "path": "go/spec/spec.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package spec provides builders and parsers for spelling Noms databases,\n// datasets and values.\npackage spec\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"net/url\"\n\t\"os\"\n\t\"regexp\"\n\t\"strings\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/nbs\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/aws/aws-sdk-go/aws\"\n\t\"github.com/aws/aws-sdk-go/aws/session\"\n\t\"github.com/aws/aws-sdk-go/service/dynamodb\"\n\t\"github.com/aws/aws-sdk-go/service/s3\"\n)\n\nconst Separator = \"::\"\n\nvar datasetRe = regexp.MustCompile(\"^\" + datas.DatasetRe.String() + \"$\")\n\nvar GetAWSSession func() *session.Session = func() *session.Session {\n\treturn session.Must(session.NewSession(aws.NewConfig().WithRegion(\"us-west-2\")))\n}\n\ntype ProtocolImpl interface {\n\tNewChunkStore(sp Spec) (chunks.ChunkStore, error)\n\tNewDatabase(sp Spec) (datas.Database, error)\n}\n\nvar ExternalProtocols = map[string]ProtocolImpl{}\n\n// SpecOptions customize Spec behavior.\ntype SpecOptions struct {\n\t// Authorization token for requests. For example, if the database is HTTP\n\t// this will used for an `Authorization: Bearer ${authorization}` header.\n\tAuthorization string\n}\n\n// Spec locates a Noms database, dataset, or value globally. Spec caches\n// its database instance so it therefore does not reflect new commits in\n// the db, by (legacy) design.\ntype Spec struct {\n\t// Protocol is one of \"mem\", \"ldb\", \"http\", or \"https\".\n\tProtocol string\n\n\t// DatabaseName is the name of the Spec's database, which is the string after\n\t// \"protocol:\". http/https specs include their leading \"//\" characters.\n\tDatabaseName string\n\n\t// Options are the SpecOptions that the Spec was constructed with.\n\tOptions SpecOptions\n\n\t// Path is nil unless the spec was created with ForPath.\n\tPath AbsolutePath\n\n\t// db is lazily created, so it needs to be a pointer to a Database.\n\tdb *datas.Database\n}\n\nfunc newSpec(dbSpec string, opts SpecOptions) (Spec, error) {\n\tprotocol, dbName, err := parseDatabaseSpec(dbSpec)\n\tif err != nil {\n\t\treturn Spec{}, err\n\t}\n\n\treturn Spec{\n\t\tProtocol: protocol,\n\t\tDatabaseName: dbName,\n\t\tOptions: opts,\n\t\tdb: new(datas.Database),\n\t}, nil\n}\n\n// ForDatabase parses a spec for a Database.\nfunc ForDatabase(spec string) (Spec, error) {\n\treturn ForDatabaseOpts(spec, SpecOptions{})\n}\n\n// ForDatabaseOpts parses a spec for a Database.\nfunc ForDatabaseOpts(spec string, opts SpecOptions) (Spec, error) {\n\treturn newSpec(spec, opts)\n}\n\n// ForDataset parses a spec for a Dataset.\nfunc ForDataset(spec string) (Spec, error) {\n\treturn ForDatasetOpts(spec, SpecOptions{})\n}\n\n// ForDatasetOpts parses a spec for a Dataset.\nfunc ForDatasetOpts(spec string, opts SpecOptions) (Spec, error) {\n\tdbSpec, pathStr, err := splitDatabaseSpec(spec)\n\tif err != nil {\n\t\treturn Spec{}, err\n\t}\n\n\tsp, err := newSpec(dbSpec, opts)\n\tif err != nil {\n\t\treturn Spec{}, err\n\t}\n\n\tpath, err := NewAbsolutePath(pathStr)\n\tif err != nil {\n\t\treturn Spec{}, err\n\t}\n\n\tif path.Dataset == \"\" {\n\t\treturn Spec{}, errors.New(\"dataset name required for dataset spec\")\n\t}\n\n\tif !path.Path.IsEmpty() {\n\t\treturn Spec{}, errors.New(\"path is not allowed for dataset spec\")\n\t}\n\n\tsp.Path = path\n\treturn sp, nil\n}\n\n// ForPath parses a spec for a path to a Value.\nfunc ForPath(spec string) (Spec, error) {\n\treturn ForPathOpts(spec, SpecOptions{})\n}\n\n// ForPathOpts parses a spec for a path to a Value.\nfunc ForPathOpts(spec string, opts SpecOptions) (Spec, error) {\n\tdbSpec, pathStr, err := splitDatabaseSpec(spec)\n\tif err != nil {\n\t\treturn Spec{}, err\n\t}\n\n\tvar path AbsolutePath\n\tif pathStr != \"\" {\n\t\tpath, err = NewAbsolutePath(pathStr)\n\t\tif err != nil {\n\t\t\treturn Spec{}, err\n\t\t}\n\t}\n\n\tsp, err := newSpec(dbSpec, opts)\n\tif err != nil {\n\t\treturn Spec{}, err\n\t}\n\n\tsp.Path = path\n\treturn sp, nil\n}\n\nfunc (sp Spec) String() string {\n\ts := sp.Protocol\n\tif s != \"mem\" {\n\t\ts += \":\" + sp.DatabaseName\n\t}\n\tp := sp.Path.String()\n\tif p != \"\" {\n\t\ts += Separator + p\n\t}\n\treturn s\n}\n\n// GetDatabase returns the Database instance that this Spec's DatabaseName\n// describes. The same Database instance is returned every time, unless Close\n// is called. If the Spec is closed, it is re-opened with a new Database.\nfunc (sp Spec) GetDatabase() datas.Database {\n\tif *sp.db == nil {\n\t\t*sp.db = sp.createDatabase()\n\t}\n\treturn *sp.db\n}\n\n// GetDataset returns the current Dataset instance for this Spec's Database.\n// GetDataset is live, so if Commit is called on this Spec's Database later, a\n// new up-to-date Dataset will returned on the next call to GetDataset. If\n// this is not a Dataset spec, returns nil.\nfunc (sp Spec) GetDataset() (ds datas.Dataset) {\n\tif sp.Path.Dataset != \"\" {\n\t\tds = sp.GetDatabase().GetDataset(sp.Path.Dataset)\n\t}\n\treturn\n}\n\n// GetValue returns the Value at this Spec's Path within its Database, or nil\n// if this isn't a Path Spec or if that path isn't found.\nfunc (sp Spec) GetValue() (val types.Value) {\n\tif !sp.Path.IsEmpty() {\n\t\tval = sp.Path.Resolve(sp.GetDatabase())\n\t}\n\treturn\n}\n\n// Href treats the Protocol and DatabaseName as a URL, and returns its href.\n// For example, the spec http://example.com/path::ds returns\n// \"http://example.com/path\". If the Protocol is not \"http\" or \"http\", returns\n// an empty string.\nfunc (sp Spec) Href() string {\n\tswitch proto := sp.Protocol; proto {\n\tcase \"http\", \"https\", \"aws\":\n\t\treturn proto + \":\" + sp.DatabaseName\n\tdefault:\n\t\treturn \"\"\n\t}\n}\n\n// Pin returns a Spec in which the dataset component, if any, has been replaced\n// with the hash of the HEAD of that dataset. This \"pins\" the path to the state\n// of the database at the current moment in time. Returns itself if the\n// PathSpec is already \"pinned\".\nfunc (sp Spec) Pin() (Spec, bool) {\n\tvar ds datas.Dataset\n\n\tif !sp.Path.IsEmpty() {\n\t\tif !sp.Path.Hash.IsEmpty() {\n\t\t\t// Spec is already pinned.\n\t\t\treturn sp, true\n\t\t}\n\n\t\tds = sp.GetDatabase().GetDataset(sp.Path.Dataset)\n\t} else {\n\t\tds = sp.GetDataset()\n\t}\n\n\tcommit, ok := ds.MaybeHead()\n\tif !ok {\n\t\treturn Spec{}, false\n\t}\n\n\tr := sp\n\tr.Path.Hash = commit.Hash()\n\tr.Path.Dataset = \"\"\n\n\treturn r, true\n}\n\nfunc (sp Spec) Close() error {\n\tdb := *sp.db\n\tif db == nil {\n\t\treturn nil\n\t}\n\n\t*sp.db = nil\n\treturn db.Close()\n}\n\nfunc (sp Spec) createDatabase() datas.Database {\n\tswitch sp.Protocol {\n\tcase \"http\", \"https\", \"aws\", \"nbs\", \"mem\":\n\t\treturn datas.NewDatabase(sp.NewChunkStore())\n\tdefault:\n\t\timpl, ok := ExternalProtocols[sp.Protocol]\n\t\tif !ok {\n\t\t\td.PanicIfError(fmt.Errorf(\"Unknown protocol: %s\", sp.Protocol))\n\t\t}\n\t\tr, err := impl.NewDatabase(sp)\n\t\td.PanicIfError(err)\n\t\treturn r\n\t}\n}\n\n// NewChunkStore returns a new ChunkStore instance that this Spec's\n// DatabaseName describes. It's unusual to call this method, GetDatabase is\n// more useful.\nfunc (sp Spec) NewChunkStore() chunks.ChunkStore {\n\tswitch sp.Protocol {\n\tcase \"http\", \"https\":\n\t\treturn datas.NewHTTPChunkStore(sp.Href(), sp.Options.Authorization)\n\tcase \"aws\":\n\t\tparts := strings.SplitN(sp.DatabaseName, \"/\", 3) // table/bucket/ns\n\t\td.PanicIfFalse(len(parts) >= 3) // parse should have ensured this was true\n\t\tsess := GetAWSSession()\n\t\treturn nbs.NewAWSStore(parts[0], parts[2], parts[1], s3.New(sess), dynamodb.New(sess), 1<<28)\n\tcase \"nbs\":\n\t\tos.MkdirAll(sp.DatabaseName, 0777)\n\t\treturn nbs.NewLocalStore(sp.DatabaseName, 1<<28)\n\tcase \"mem\":\n\t\tstorage := &chunks.MemoryStorage{}\n\t\treturn storage.NewView()\n\tdefault:\n\t\timpl, ok := ExternalProtocols[sp.Protocol]\n\t\tif !ok {\n\t\t\td.PanicIfError(fmt.Errorf(\"Unknown protocol: %s\", sp.Protocol))\n\t\t}\n\t\tr, err := impl.NewChunkStore(sp)\n\t\td.PanicIfError(err)\n\t\treturn r\n\t}\n}\n\nfunc parseDatabaseSpec(spec string) (protocol, name string, err error) {\n\tif len(spec) == 0 {\n\t\terr = fmt.Errorf(\"Empty spec\")\n\t\treturn\n\t}\n\n\tparts := strings.SplitN(spec, \":\", 2) // [protocol] [, path]?\n\n\t// If there was no \":\" then this is either a mem spec, or a filesystem path.\n\t// This is ambiguous if the file system path is \"mem\" but that just means the\n\t// path needs to be explicitly \"nbs:mem\".\n\tif len(parts) == 1 {\n\t\tif spec == \"mem\" {\n\t\t\tprotocol = \"mem\"\n\t\t} else {\n\t\t\tprotocol, name = \"nbs\", spec\n\t\t}\n\t\treturn\n\t}\n\n\tif _, ok := ExternalProtocols[parts[0]]; ok {\n\t\tfmt.Println(\"found external spec\", parts[0])\n\t\tprotocol, name = parts[0], parts[1]\n\t\treturn\n\t}\n\n\tswitch parts[0] {\n\tcase \"nbs\":\n\t\tprotocol, name = parts[0], parts[1]\n\n\tcase \"aws\":\n\t\tp, n := parts[0], parts[1]\n\t\tpattern := regexp.MustCompile(\"^[^/]+/[^/]+/.*$\")\n\t\tif !pattern.MatchString(n) {\n\t\t\terr = errors.New(\"aws spec must match pattern aws:\" + pattern.String())\n\t\t}\n\t\tprotocol, name = p, n\n\t\treturn\n\n\tcase \"http\", \"https\":\n\t\tu, perr := url.Parse(spec)\n\t\tif perr != nil {\n\t\t\terr = perr\n\t\t} else if u.Host == \"\" {\n\t\t\terr = fmt.Errorf(\"%s has empty host\", spec)\n\t\t} else {\n\t\t\tprotocol, name = parts[0], parts[1]\n\t\t}\n\n\tcase \"mem\":\n\t\terr = fmt.Errorf(`In-memory database must be specified as \"mem\", not \"mem:\"`)\n\n\tdefault:\n\t\terr = fmt.Errorf(\"Invalid database protocol %s in %s\", protocol, spec)\n\t}\n\treturn\n}\n\nfunc splitDatabaseSpec(spec string) (string, string, error) {\n\tlastIdx := strings.LastIndex(spec, Separator)\n\tif lastIdx == -1 {\n\t\treturn \"\", \"\", fmt.Errorf(\"Missing %s after database in %s\", Separator, spec)\n\t}\n\n\treturn spec[:lastIdx], spec[lastIdx+len(Separator):], nil\n}\n" }, { "path": "go/spec/spec_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage spec\n\nimport (\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"os\"\n\t\"path\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/nbs\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestMemDatabaseSpec(t *testing.T) {\n\tassert := assert.New(t)\n\n\tspec, err := ForDatabase(\"mem\")\n\tassert.NoError(err)\n\tdefer spec.Close()\n\n\tassert.Equal(\"mem\", spec.Protocol)\n\tassert.Equal(\"\", spec.DatabaseName)\n\tassert.True(spec.Path.IsEmpty())\n\n\ts := types.String(\"hello\")\n\tdb := spec.GetDatabase()\n\tdb.WriteValue(s)\n\tassert.Equal(s, db.ReadValue(s.Hash()))\n}\n\nfunc TestMemDatasetSpec(t *testing.T) {\n\tassert := assert.New(t)\n\n\tspec, err := ForDataset(\"mem::test\")\n\tassert.NoError(err)\n\tdefer spec.Close()\n\n\tassert.Equal(\"mem\", spec.Protocol)\n\tassert.Equal(\"\", spec.DatabaseName)\n\tassert.Equal(\"test\", spec.Path.Dataset)\n\tassert.True(spec.Path.Path.IsEmpty())\n\n\tds := spec.GetDataset()\n\t_, ok := spec.GetDataset().MaybeHeadValue()\n\tassert.False(ok)\n\n\ts := types.String(\"hello\")\n\tds, err = spec.GetDatabase().CommitValue(ds, s)\n\tassert.NoError(err)\n\tassert.Equal(s, ds.HeadValue())\n}\n\nfunc TestMemHashPathSpec(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts := types.String(\"hello\")\n\n\tspec, err := ForPath(\"mem::#\" + s.Hash().String())\n\tassert.NoError(err)\n\tdefer spec.Close()\n\n\tassert.Equal(\"mem\", spec.Protocol)\n\tassert.Equal(\"\", spec.DatabaseName)\n\tassert.False(spec.Path.IsEmpty())\n\n\t// This is a reasonable check but it causes the next GetValue to return nil:\n\t// assert.Nil(spec.GetValue())\n\n\tspec.GetDatabase().WriteValue(s)\n\tassert.Equal(s, spec.GetValue())\n}\n\nfunc TestMemDatasetPathSpec(t *testing.T) {\n\tassert := assert.New(t)\n\n\tspec, err := ForPath(\"mem::test.value[0]\")\n\tassert.NoError(err)\n\tdefer spec.Close()\n\n\tassert.Equal(\"mem\", spec.Protocol)\n\tassert.Equal(\"\", spec.DatabaseName)\n\tassert.False(spec.Path.IsEmpty())\n\n\tassert.Nil(spec.GetValue())\n\n\tdb := spec.GetDatabase()\n\tds := db.GetDataset(\"test\")\n\t_, err = db.CommitValue(ds, types.NewList(db, types.Number(42)))\n\tassert.NoError(err)\n\n\tassert.Equal(types.Number(42), spec.GetValue())\n}\n\nfunc TestNBSDatabaseSpec(t *testing.T) {\n\tassert := assert.New(t)\n\n\trun := func(prefix string) {\n\t\ttmpDir, err := ioutil.TempDir(\"\", \"spec_test\")\n\t\tassert.NoError(err)\n\t\tdefer os.RemoveAll(tmpDir)\n\n\t\ts := types.String(\"string\")\n\n\t\t// Existing database in the database are read from the spec.\n\t\tstore1 := path.Join(tmpDir, \"store1\")\n\t\tos.Mkdir(store1, 0777)\n\t\tfunc() {\n\t\t\tdb := datas.NewDatabase(nbs.NewLocalStore(store1, 8*(1<<20)))\n\t\t\tdefer db.Close()\n\t\t\tr := db.WriteValue(s)\n\t\t\t_, err = db.CommitValue(db.GetDataset(\"datasetID\"), r)\n\t\t\tassert.NoError(err)\n\t\t}()\n\n\t\tspec1, err := ForDatabase(prefix + store1)\n\t\tassert.NoError(err)\n\t\tdefer spec1.Close()\n\n\t\tassert.Equal(\"nbs\", spec1.Protocol)\n\t\tassert.Equal(store1, spec1.DatabaseName)\n\n\t\tassert.Equal(s, spec1.GetDatabase().ReadValue(s.Hash()))\n\n\t\t// New databases can be created and read/written from.\n\t\tstore2 := path.Join(tmpDir, \"store2\")\n\t\tos.Mkdir(store2, 0777)\n\t\tspec2, err := ForDatabase(prefix + store2)\n\t\tassert.NoError(err)\n\t\tdefer spec2.Close()\n\n\t\tassert.Equal(\"nbs\", spec2.Protocol)\n\t\tassert.Equal(store2, spec2.DatabaseName)\n\n\t\tdb := spec2.GetDatabase()\n\t\tdb.WriteValue(s)\n\t\tr := db.WriteValue(s)\n\t\t_, err = db.CommitValue(db.GetDataset(\"datasetID\"), r)\n\t\tassert.NoError(err)\n\t\tassert.Equal(s, db.ReadValue(s.Hash()))\n\t}\n\n\trun(\"\")\n\trun(\"nbs:\")\n}\n\n// Skip LDB dataset and path tests: the database behaviour is tested in\n// TestLDBDatabaseSpec, TestMemDatasetSpec/TestMem*PathSpec cover general\n// dataset/path behaviour, and ForDataset/ForPath test LDB parsing.\n\nfunc TestCloseSpecWithoutOpen(t *testing.T) {\n\ts, err := ForDatabase(\"mem\")\n\tassert.NoError(t, err)\n\ts.Close()\n}\n\nfunc TestHref(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsp, _ := ForDatabase(\"http://localhost\")\n\tassert.Equal(\"http://localhost\", sp.Href())\n\tsp, _ = ForDatabase(\"http://localhost/foo/bar/baz\")\n\tassert.Equal(\"http://localhost/foo/bar/baz\", sp.Href())\n\tsp, _ = ForDatabase(\"https://my.example.com/foo/bar/baz\")\n\tassert.Equal(\"https://my.example.com/foo/bar/baz\", sp.Href())\n\tsp, _ = ForDataset(\"https://my.example.com/foo/bar/baz::myds\")\n\tassert.Equal(\"https://my.example.com/foo/bar/baz\", sp.Href())\n\tsp, _ = ForDataset(\"https://my.example.com:8080/foo/bar/baz::myds\")\n\tassert.Equal(\"https://my.example.com:8080/foo/bar/baz\", sp.Href())\n\tsp, _ = ForPath(\"https://my.example.com/foo/bar/baz::myds.my.path\")\n\tassert.Equal(\"https://my.example.com/foo/bar/baz\", sp.Href())\n\n\tsp, _ = ForDatabase(\"aws:table/bucket/ns\")\n\tassert.Equal(\"aws:table/bucket/ns\", sp.Href())\n\tsp, _ = ForDataset(\"aws:table/bucket/ns::myds\")\n\tassert.Equal(\"aws:table/bucket/ns\", sp.Href())\n\tsp, _ = ForPath(\"aws:table/bucket/ns::myds.my.path\")\n\tassert.Equal(\"aws:table/bucket/ns\", sp.Href())\n\n\tsp, err := ForPath(\"mem::myds.my.path\")\n\tassert.NoError(err)\n\tassert.Equal(\"\", sp.Href())\n}\n\nfunc TestForDatabase(t *testing.T) {\n\tassert := assert.New(t)\n\n\tbadSpecs := []string{\n\t\t\"mem:stuff\",\n\t\t\"mem::\",\n\t\t\"mem:\",\n\t\t\"http:\",\n\t\t\"http://\",\n\t\t\"http://%\",\n\t\t\"https:\",\n\t\t\"https://\",\n\t\t\"https://%\",\n\t\t\"http://::192.30.252.154\",\n\t\t\"http://0:0:0:0:0:ffff:c01e:fc9a\",\n\t\t\"http://::ffff:c01e:fc9a\",\n\t\t\"http://::ffff::1e::9a\",\n\t\t\"ldb:\",\n\t\t\"random:\",\n\t\t\"random:random\",\n\t\t\"/file/ba:d\",\n\t\t\"aws:\",\n\t\t\"aws:t\",\n\t\t\"aws:t/b\",\n\t\t\"aws://table/bucket/db\",\n\t}\n\n\tfor _, spec := range badSpecs {\n\t\t_, err := ForDatabase(spec)\n\t\tassert.Error(err, spec)\n\t}\n\n\ttmpDir, err := ioutil.TempDir(\"\", \"spec_test\")\n\tassert.NoError(err)\n\tdefer os.RemoveAll(tmpDir)\n\n\ttestCases := []struct {\n\t\tspec, protocol, databaseName, canonicalSpecIfAny string\n\t}{\n\t\t{\"http://localhost:8000\", \"http\", \"//localhost:8000\", \"\"},\n\t\t{\"http://localhost:8000/fff\", \"http\", \"//localhost:8000/fff\", \"\"},\n\t\t{\"https://local.attic.io/john/doe\", \"https\", \"//local.attic.io/john/doe\", \"\"},\n\t\t{\"mem\", \"mem\", \"\", \"\"},\n\t\t{tmpDir, \"nbs\", tmpDir, \"nbs:\" + tmpDir},\n\t\t{\"nbs:\" + tmpDir, \"nbs\", tmpDir, \"\"},\n\t\t{\"http://server.com/john/doe?access_token=jane\", \"http\", \"//server.com/john/doe?access_token=jane\", \"\"},\n\t\t{\"https://server.com/john/doe/?arg=2&qp1=true&access_token=jane\", \"https\", \"//server.com/john/doe/?arg=2&qp1=true&access_token=jane\", \"\"},\n\t\t{\"http://some/::/one\", \"http\", \"//some/::/one\", \"\"},\n\t\t{\"http://::1\", \"http\", \"//::1\", \"\"},\n\t\t{\"http://192.30.252.154\", \"http\", \"//192.30.252.154\", \"\"},\n\t\t{\"aws:table/bucket/db\", \"aws\", \"table/bucket/db\", \"\"},\n\t\t{\"aws:table/bucket/db/other/random/crap\", \"aws\", \"table/bucket/db/other/random/crap\", \"\"},\n\t}\n\n\tfor _, tc := range testCases {\n\t\tspec, err := ForDatabase(tc.spec)\n\t\tassert.NoError(err, tc.spec)\n\t\tdefer spec.Close()\n\n\t\tassert.Equal(tc.protocol, spec.Protocol)\n\t\tassert.Equal(tc.databaseName, spec.DatabaseName)\n\t\tassert.True(spec.Path.IsEmpty())\n\n\t\tif tc.canonicalSpecIfAny == \"\" {\n\t\t\tassert.Equal(tc.spec, spec.String())\n\t\t} else {\n\t\t\tassert.Equal(tc.canonicalSpecIfAny, spec.String())\n\t\t}\n\t}\n}\n\nfunc TestForDataset(t *testing.T) {\n\tassert := assert.New(t)\n\n\tbadSpecs := []string{\n\t\t\"mem\",\n\t\t\"mem:\",\n\t\t\"mem:::ds\",\n\t\t\"http\",\n\t\t\"http:\",\n\t\t\"http://foo\",\n\t\t\"monkey\",\n\t\t\"monkey:balls\",\n\t\t\"http:::dsname\",\n\t\t\"mem:/a/bogus/path:dsname\",\n\t\t\"http://localhost:8000/one\",\n\t\t\"http://::192.30.252.154::foo\",\n\t\t\"http://0:0:0:0:0:ffff:c01e:fc9a::foo\",\n\t\t\"http://::ffff::1e::9a::foo\",\n\t\t\"nbs:\",\n\t\t\"nbs:hello\",\n\t\t\"aws://table:bucket/db::ds\",\n\t\t\"aws:t::ds\",\n\t\t\"aws:t/b::ds\",\n\t\t\"aws://t/b/foo::ds\",\n\t\t\"mem::foo.value::ds\",\n\t}\n\n\tfor _, spec := range badSpecs {\n\t\t_, err := ForDataset(spec)\n\t\tassert.Error(err, spec)\n\t}\n\n\tinvalidDatasetNames := []string{\" \", \"\", \"$\", \"#\", \":\", \"\\n\", \"💩\"}\n\tfor _, s := range invalidDatasetNames {\n\t\t_, err := ForDataset(\"mem::\" + s)\n\t\tassert.Error(err)\n\t}\n\n\tvalidDatasetNames := []string{\"a\", \"Z\", \"0\", \"/\", \"-\", \"_\"}\n\tfor _, s := range validDatasetNames {\n\t\t_, err := ForDataset(\"mem::\" + s)\n\t\tassert.NoError(err)\n\t}\n\n\ttmpDir, err := ioutil.TempDir(\"\", \"spec_test\")\n\tassert.NoError(err)\n\tdefer os.RemoveAll(tmpDir)\n\n\ttestCases := []struct {\n\t\tspec, protocol, databaseName, datasetName, canonicalSpecIfAny string\n\t}{\n\t\t{\"http://localhost:8000::ds1\", \"http\", \"//localhost:8000\", \"ds1\", \"\"},\n\t\t{\"http://localhost:8000/john/doe/::ds2\", \"http\", \"//localhost:8000/john/doe/\", \"ds2\", \"\"},\n\t\t{\"https://local.attic.io/john/doe::ds3\", \"https\", \"//local.attic.io/john/doe\", \"ds3\", \"\"},\n\t\t{\"http://local.attic.io/john/doe::ds1\", \"http\", \"//local.attic.io/john/doe\", \"ds1\", \"\"},\n\t\t{\"nbs:\" + tmpDir + \"::ds/one\", \"nbs\", tmpDir, \"ds/one\", \"\"},\n\t\t{tmpDir + \"::ds/one\", \"nbs\", tmpDir, \"ds/one\", \"nbs:\" + tmpDir + \"::ds/one\"},\n\t\t{\"http://localhost:8000/john/doe?access_token=abc::ds/one\", \"http\", \"//localhost:8000/john/doe?access_token=abc\", \"ds/one\", \"\"},\n\t\t{\"https://localhost:8000?qp1=x&access_token=abc&qp2=y::ds/one\", \"https\", \"//localhost:8000?qp1=x&access_token=abc&qp2=y\", \"ds/one\", \"\"},\n\t\t{\"http://192.30.252.154::foo\", \"http\", \"//192.30.252.154\", \"foo\", \"\"},\n\t\t{\"http://::1::foo\", \"http\", \"//::1\", \"foo\", \"\"},\n\t\t{\"aws:table/bucket/db::ds\", \"aws\", \"table/bucket/db\", \"ds\", \"\"},\n\t}\n\n\tfor _, tc := range testCases {\n\t\tspec, err := ForDataset(tc.spec)\n\t\tassert.NoError(err, tc.spec)\n\t\tdefer spec.Close()\n\n\t\tassert.Equal(tc.protocol, spec.Protocol)\n\t\tassert.Equal(tc.databaseName, spec.DatabaseName)\n\t\tassert.Equal(tc.datasetName, spec.Path.Dataset)\n\n\t\tif tc.canonicalSpecIfAny == \"\" {\n\t\t\tassert.Equal(tc.spec, spec.String())\n\t\t} else {\n\t\t\tassert.Equal(tc.canonicalSpecIfAny, spec.String())\n\t\t}\n\t}\n}\n\nfunc TestForPath(t *testing.T) {\n\tassert := assert.New(t)\n\n\tbadSpecs := []string{\n\t\t\"mem::#\",\n\t\t\"mem::#s\",\n\t\t\"mem::#foobarbaz\",\n\t\t\"mem::#wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww\",\n\t\t\"http://::192.30.252.154::baz[42]\",\n\t\t\"http://0:0:0:0:0:ffff:c01e:fc9a::foo[42].bar\",\n\t\t\"http://::ffff:c01e:fc9a::foo.foo\",\n\t\t\"http://::ffff::1e::9a::hello[\\\"world\\\"]\",\n\t\t\"aws://table:bucket/db::foo.foo\",\n\t\t\"aws://table/db/foo.foo\",\n\t}\n\n\tfor _, bs := range badSpecs {\n\t\t_, err := ForPath(bs)\n\t\tassert.Error(err)\n\t}\n\n\ttmpDir, err := ioutil.TempDir(\"\", \"spec_test\")\n\tassert.NoError(err)\n\tdefer os.RemoveAll(tmpDir)\n\n\ttestCases := []struct {\n\t\tspec, protocol, databaseName, pathString, canonicalSpecIfAny string\n\t}{\n\t\t{\"http://local.attic.io/john/doe::#0123456789abcdefghijklmnopqrstuv\", \"http\", \"//local.attic.io/john/doe\", \"#0123456789abcdefghijklmnopqrstuv\", \"\"},\n\t\t{tmpDir + \"::#0123456789abcdefghijklmnopqrstuv\", \"nbs\", tmpDir, \"#0123456789abcdefghijklmnopqrstuv\", \"nbs:\" + tmpDir + \"::#0123456789abcdefghijklmnopqrstuv\"},\n\t\t{\"nbs:\" + tmpDir + \"::#0123456789abcdefghijklmnopqrstuv\", \"nbs\", tmpDir, \"#0123456789abcdefghijklmnopqrstuv\", \"\"},\n\t\t{\"mem::#0123456789abcdefghijklmnopqrstuv\", \"mem\", \"\", \"#0123456789abcdefghijklmnopqrstuv\", \"\"},\n\t\t{\"http://local.attic.io/john/doe::#0123456789abcdefghijklmnopqrstuv\", \"http\", \"//local.attic.io/john/doe\", \"#0123456789abcdefghijklmnopqrstuv\", \"\"},\n\t\t{\"http://localhost:8000/john/doe/::ds1\", \"http\", \"//localhost:8000/john/doe/\", \"ds1\", \"\"},\n\t\t{\"http://192.30.252.154::foo.bar\", \"http\", \"//192.30.252.154\", \"foo.bar\", \"\"},\n\t\t{\"http://::1::foo.bar.baz\", \"http\", \"//::1\", \"foo.bar.baz\", \"\"},\n\t\t{\"aws:table/bucket/db::foo.foo\", \"aws\", \"table/bucket/db\", \"foo.foo\", \"\"},\n\t}\n\n\tfor _, tc := range testCases {\n\t\tspec, err := ForPath(tc.spec)\n\t\tassert.NoError(err)\n\t\tdefer spec.Close()\n\n\t\tassert.Equal(tc.protocol, spec.Protocol)\n\t\tassert.Equal(tc.databaseName, spec.DatabaseName)\n\t\tassert.Equal(tc.pathString, spec.Path.String())\n\n\t\tif tc.canonicalSpecIfAny == \"\" {\n\t\t\tassert.Equal(tc.spec, spec.String())\n\t\t} else {\n\t\t\tassert.Equal(tc.canonicalSpecIfAny, spec.String())\n\t\t}\n\t}\n}\n\nfunc TestPinPathSpec(t *testing.T) {\n\tassert := assert.New(t)\n\n\tunpinned, err := ForPath(\"mem::foo.value\")\n\tassert.NoError(err)\n\tdefer unpinned.Close()\n\n\tdb := unpinned.GetDatabase()\n\tdb.CommitValue(db.GetDataset(\"foo\"), types.Number(42))\n\n\tpinned, ok := unpinned.Pin()\n\tassert.True(ok)\n\tdefer pinned.Close()\n\n\thead := db.GetDataset(\"foo\").Head()\n\n\tassert.Equal(head.Hash(), pinned.Path.Hash)\n\tassert.Equal(fmt.Sprintf(\"mem::#%s.value\", head.Hash().String()), pinned.String())\n\tassert.Equal(types.Number(42), pinned.GetValue())\n\tassert.Equal(types.Number(42), unpinned.GetValue())\n\n\tdb.CommitValue(db.GetDataset(\"foo\"), types.Number(43))\n\tassert.Equal(types.Number(42), pinned.GetValue())\n\tassert.Equal(types.Number(43), unpinned.GetValue())\n}\n\nfunc TestPinDatasetSpec(t *testing.T) {\n\tassert := assert.New(t)\n\n\tunpinned, err := ForDataset(\"mem::foo\")\n\tassert.NoError(err)\n\tdefer unpinned.Close()\n\n\tdb := unpinned.GetDatabase()\n\tdb.CommitValue(db.GetDataset(\"foo\"), types.Number(42))\n\n\tpinned, ok := unpinned.Pin()\n\tassert.True(ok)\n\tdefer pinned.Close()\n\n\thead := db.GetDataset(\"foo\").Head()\n\n\tcommitValue := func(val types.Value) types.Value {\n\t\treturn val.(types.Struct).Get(datas.ValueField)\n\t}\n\n\tassert.Equal(head.Hash(), pinned.Path.Hash)\n\tassert.Equal(fmt.Sprintf(\"mem::#%s\", head.Hash().String()), pinned.String())\n\tassert.Equal(types.Number(42), commitValue(pinned.GetValue()))\n\tassert.Equal(types.Number(42), unpinned.GetDataset().HeadValue())\n\n\tdb.CommitValue(db.GetDataset(\"foo\"), types.Number(43))\n\tassert.Equal(types.Number(42), commitValue(pinned.GetValue()))\n\tassert.Equal(types.Number(43), unpinned.GetDataset().HeadValue())\n}\n\nfunc TestAlreadyPinnedPathSpec(t *testing.T) {\n\tassert := assert.New(t)\n\n\tunpinned, err := ForPath(\"mem::#imgp9mp1h3b9nv0gna6mri53dlj9f4ql.value\")\n\tassert.NoError(err)\n\tpinned, ok := unpinned.Pin()\n\tassert.True(ok)\n\tassert.Equal(unpinned, pinned)\n}\n\nfunc TestMultipleSpecsSameNBS(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttmpDir, err := ioutil.TempDir(\"\", \"spec_test\")\n\tassert.NoError(err)\n\tdefer os.RemoveAll(tmpDir)\n\n\tspec1, err1 := ForDatabase(tmpDir)\n\tspec2, err2 := ForDatabase(tmpDir)\n\n\tassert.NoError(err1)\n\tassert.NoError(err2)\n\n\ts := types.String(\"hello\")\n\tdb := spec1.GetDatabase()\n\tr := db.WriteValue(s)\n\t_, err = db.CommitValue(db.GetDataset(\"datasetID\"), r)\n\tassert.NoError(err)\n\tassert.Equal(s, spec2.GetDatabase().ReadValue(s.Hash()))\n}\n\nfunc TestAcccessingInvalidSpec(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttest := func(spec string) {\n\t\tsp, err := ForDatabase(spec)\n\t\tassert.Error(err)\n\t\tassert.Equal(\"\", sp.Href())\n\t\tassert.Panics(func() { sp.GetDatabase() })\n\t\tassert.Panics(func() { sp.GetDatabase() })\n\t\tassert.Panics(func() { sp.NewChunkStore() })\n\t\tassert.Panics(func() { sp.NewChunkStore() })\n\t\tassert.Panics(func() { sp.Close() })\n\t\tassert.Panics(func() { sp.Close() })\n\t\t// Spec was created with ForDatabase, so dataset/path related functions\n\t\t// should just fail not panic.\n\t\t_, ok := sp.Pin()\n\t\tassert.False(ok)\n\t\tassert.Equal(datas.Dataset{}, sp.GetDataset())\n\t\tassert.Nil(sp.GetValue())\n\t}\n\n\ttest(\"\")\n\ttest(\"invalid:spec\")\n\ttest(\"💩:spec\")\n\ttest(\"http:\")\n\ttest(\"http:💩:\")\n}\n\ntype testProtocol struct {\n\tname string\n}\n\nfunc (t *testProtocol) NewChunkStore(sp Spec) (chunks.ChunkStore, error) {\n\tt.name = sp.DatabaseName\n\treturn chunks.NewMemoryStoreFactory().CreateStore(\"\"), nil\n}\nfunc (t *testProtocol) NewDatabase(sp Spec) (datas.Database, error) {\n\tt.name = sp.DatabaseName\n\tcs, err := t.NewChunkStore(sp)\n\td.PanicIfError(err)\n\treturn datas.NewDatabase(cs), nil\n}\n\nfunc TestExternalProtocol(t *testing.T) {\n\tassert := assert.New(t)\n\ttp := testProtocol{}\n\tExternalProtocols[\"test\"] = &tp\n\n\tsp, err := ForDataset(\"test:foo::bar\")\n\tassert.NoError(err)\n\tassert.Equal(\"test\", sp.Protocol)\n\tassert.Equal(\"foo\", sp.DatabaseName)\n\n\tcs := sp.NewChunkStore()\n\tassert.Equal(\"foo\", tp.name)\n\tc := chunks.NewChunk([]byte(\"hi!\"))\n\tcs.Put(c)\n\tassert.True(cs.Has(c.Hash()))\n\n\ttp.name = \"\"\n\tds := sp.GetDataset()\n\tassert.Equal(\"foo\", tp.name)\n\n\tds, err = ds.Database().CommitValue(ds, types.String(\"hi!\"))\n\td.PanicIfError(err)\n\n\tassert.True(types.String(\"hi!\").Equals(ds.HeadValue()))\n}\n\nfunc TestMkDirAll(t *testing.T) {\n\tassert := assert.New(t)\n\ttd, err := ioutil.TempDir(\"\", \"\")\n\tassert.NoError(err)\n\tp := path.Join(td, \"foo\", \"bar\", \"baz\")\n\tsp, err := ForDatabase(p)\n\tassert.NoError(err)\n\t_ = sp.NewChunkStore()\n}\n\nfunc TestNetworkError(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsvr := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tw.WriteHeader(http.StatusForbidden)\n\t\tw.Write([]byte(\"monkey\\n\"))\n\t}))\n\n\tsp, err := ForDatabase(svr.URL)\n\tassert.NoError(err)\n\terr = d.Try(func() {\n\t\tsp.GetDatabase()\n\t})\n\tassert.Equal(\"Unexpected response: Forbidden: monkey\", err.(d.WrappedError).Cause().Error())\n}\n" }, { "path": "go/spec/util.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage spec\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\nfunc CreateDatabaseSpecString(protocol, db string) string {\n\treturn Spec{Protocol: protocol, DatabaseName: db}.String()\n}\n\nfunc CreateValueSpecString(protocol, db, path string) string {\n\tp, err := NewAbsolutePath(path)\n\td.Chk.NoError(err)\n\treturn Spec{Protocol: protocol, DatabaseName: db, Path: p}.String()\n}\n\nfunc CreateHashSpecString(protocol, db string, h hash.Hash) string {\n\treturn Spec{Protocol: protocol, DatabaseName: db, Path: AbsolutePath{Hash: h}}.String()\n}\n" }, { "path": "go/types/blob.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"errors\"\n\t\"io\"\n\t\"sync\"\n\n\t\"runtime\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// Blob represents a list of Blobs.\ntype Blob struct {\n\tsequence\n}\n\nfunc newBlob(seq sequence) Blob {\n\treturn Blob{seq}\n}\n\nfunc NewEmptyBlob(vrw ValueReadWriter) Blob {\n\treturn Blob{newBlobLeafSequence(vrw, []byte{})}\n}\n\nfunc (b Blob) Edit() *BlobEditor {\n\treturn NewBlobEditor(b)\n}\n\n// ReadAt implements the ReaderAt interface. Eagerly loads requested byte-range from the blob p-tree.\nfunc (b Blob) ReadAt(p []byte, off int64) (n int, err error) {\n\t// TODO: Support negative off?\n\td.PanicIfTrue(off < 0)\n\n\tstartIdx := uint64(off)\n\tif startIdx >= b.Len() {\n\t\treturn 0, io.EOF\n\t}\n\n\tendIdx := startIdx + uint64(len(p))\n\tif endIdx > b.Len() {\n\t\tendIdx = b.Len()\n\t}\n\n\tif endIdx == b.Len() {\n\t\terr = io.EOF\n\t}\n\n\tif startIdx == endIdx {\n\t\treturn\n\t}\n\n\tleaves, localStart := LoadLeafNodes([]Collection{b}, startIdx, endIdx)\n\tendIdx = localStart + endIdx - startIdx\n\tstartIdx = localStart\n\n\tfor _, leaf := range leaves {\n\t\tbl := leaf.asSequence().(blobLeafSequence)\n\n\t\tlocalEnd := endIdx\n\t\tdata := bl.data()\n\t\tleafLength := uint64(len(data))\n\t\tif localEnd > leafLength {\n\t\t\tlocalEnd = leafLength\n\t\t}\n\t\tsrc := data[startIdx:localEnd]\n\n\t\tcopy(p[n:], src)\n\t\tn += len(src)\n\t\tendIdx -= localEnd\n\t\tstartIdx = 0\n\t}\n\n\treturn\n}\n\nfunc (b Blob) Reader() *BlobReader {\n\treturn &BlobReader{b, 0}\n}\n\nfunc (b Blob) Copy(w io.Writer) (n int64) {\n\treturn b.CopyReadAhead(w, 1<<23 /* 8MB */, 6)\n}\n\n// CopyReadAhead copies the entire contents of |b| to |w|, and attempts to stay\n// |concurrency| |chunkSize| blocks of bytes ahead of the last byte written to\n// |w|.\nfunc (b Blob) CopyReadAhead(w io.Writer, chunkSize uint64, concurrency int) (n int64) {\n\tbChan := make(chan chan []byte, concurrency)\n\n\tgo func() {\n\t\tfor idx, len := uint64(0), b.Len(); idx < len; {\n\t\t\tbc := make(chan []byte)\n\t\t\tbChan <- bc\n\n\t\t\tstart := idx\n\t\t\tblockLength := b.Len() - start\n\t\t\tif blockLength > chunkSize {\n\t\t\t\tblockLength = chunkSize\n\t\t\t}\n\t\t\tidx += blockLength\n\n\t\t\tgo func() {\n\t\t\t\tbuff := make([]byte, blockLength)\n\t\t\t\tb.ReadAt(buff, int64(start))\n\t\t\t\tbc <- buff\n\t\t\t}()\n\t\t}\n\t\tclose(bChan)\n\t}()\n\n\t// Ensure read-ahead goroutines can exit\n\tdefer func() {\n\t\tfor range bChan {\n\t\t}\n\t}()\n\n\tfor b := range bChan {\n\t\tln, err := w.Write(<-b)\n\t\tn += int64(ln)\n\t\tif err != nil {\n\t\t\treturn\n\t\t}\n\t}\n\treturn\n}\n\n// Concat returns a new Blob comprised of this joined with other. It only needs\n// to visit the rightmost prolly tree chunks of this Blob, and the leftmost\n// prolly tree chunks of other, so it's efficient.\nfunc (b Blob) Concat(other Blob) Blob {\n\tseq := concat(b.sequence, other.sequence, func(cur *sequenceCursor, vrw ValueReadWriter) *sequenceChunker {\n\t\treturn b.newChunker(cur, vrw)\n\t})\n\treturn newBlob(seq)\n}\n\nfunc (b Blob) newChunker(cur *sequenceCursor, vrw ValueReadWriter) *sequenceChunker {\n\treturn newSequenceChunker(cur, 0, vrw, makeBlobLeafChunkFn(vrw), newIndexedMetaSequenceChunkFn(BlobKind, vrw), hashValueByte)\n}\n\nfunc (b Blob) asSequence() sequence {\n\treturn b.sequence\n}\n\n// Value interface\nfunc (b Blob) Value() Value {\n\treturn b\n}\n\nfunc (b Blob) WalkValues(cb ValueCallback) {\n}\n\ntype BlobReader struct {\n\tb Blob\n\tpos int64\n}\n\nfunc (cbr *BlobReader) Read(p []byte) (n int, err error) {\n\tn, err = cbr.b.ReadAt(p, cbr.pos)\n\tcbr.pos += int64(n)\n\treturn\n}\n\nfunc (cbr *BlobReader) Seek(offset int64, whence int) (int64, error) {\n\tabs := int64(cbr.pos)\n\n\tswitch whence {\n\tcase 0:\n\t\tabs = offset\n\tcase 1:\n\t\tabs += offset\n\tcase 2:\n\t\tabs = int64(cbr.b.Len()) + offset\n\tdefault:\n\t\treturn 0, errors.New(\"Blob.Reader.Seek: invalid whence\")\n\t}\n\n\tif abs < 0 {\n\t\treturn 0, errors.New(\"Blob.Reader.Seek: negative position\")\n\t}\n\n\tcbr.pos = int64(abs)\n\treturn abs, nil\n}\n\nfunc makeBlobLeafChunkFn(vrw ValueReadWriter) makeChunkFn {\n\treturn func(level uint64, items []sequenceItem) (Collection, orderedKey, uint64) {\n\t\td.PanicIfFalse(level == 0)\n\t\tbuff := make([]byte, len(items))\n\n\t\tfor i, v := range items {\n\t\t\tbuff[i] = v.(byte)\n\t\t}\n\n\t\treturn chunkBlobLeaf(vrw, buff)\n\t}\n}\n\nfunc chunkBlobLeaf(vrw ValueReadWriter, buff []byte) (Collection, orderedKey, uint64) {\n\tblob := newBlob(newBlobLeafSequence(vrw, buff))\n\treturn blob, orderedKeyFromInt(len(buff)), uint64(len(buff))\n}\n\n// NewBlob creates a Blob by reading from every Reader in rs and\n// concatenating the result. NewBlob uses one goroutine per Reader.\nfunc NewBlob(vrw ValueReadWriter, rs ...io.Reader) Blob {\n\treturn readBlobsP(vrw, rs...)\n}\n\nfunc readBlobsP(vrw ValueReadWriter, rs ...io.Reader) Blob {\n\tswitch len(rs) {\n\tcase 0:\n\t\treturn NewEmptyBlob(vrw)\n\tcase 1:\n\t\treturn readBlob(rs[0], vrw)\n\t}\n\n\tblobs := make([]Blob, len(rs))\n\n\twg := &sync.WaitGroup{}\n\twg.Add(len(rs))\n\n\tfor i, r := range rs {\n\t\ti2, r2 := i, r\n\t\tgo func() {\n\t\t\tblobs[i2] = readBlob(r2, vrw)\n\t\t\twg.Done()\n\t\t}()\n\t}\n\n\twg.Wait()\n\n\tb := blobs[0]\n\tfor i := 1; i < len(blobs); i++ {\n\t\tb = b.Concat(blobs[i])\n\t}\n\treturn b\n}\n\nfunc readBlob(r io.Reader, vrw ValueReadWriter) Blob {\n\tsc := newEmptySequenceChunker(vrw, makeBlobLeafChunkFn(vrw), newIndexedMetaSequenceChunkFn(BlobKind, vrw), func(item sequenceItem, rv *rollingValueHasher) {\n\t\trv.HashByte(item.(byte))\n\t})\n\n\t// TODO: The code below is temporary. It's basically a custom leaf-level chunker for blobs. There are substational perf gains by doing it this way as it avoids the cost of boxing every single byte which is chunked.\n\tchunkBuff := [8192]byte{}\n\tchunkBytes := chunkBuff[:]\n\trv := newRollingValueHasher(0)\n\toffset := 0\n\taddByte := func(b byte) bool {\n\t\tif offset >= len(chunkBytes) {\n\t\t\ttmp := make([]byte, len(chunkBytes)*2)\n\t\t\tcopy(tmp, chunkBytes)\n\t\t\tchunkBytes = tmp\n\t\t}\n\t\tchunkBytes[offset] = b\n\t\toffset++\n\t\trv.HashByte(b)\n\t\treturn rv.crossedBoundary\n\t}\n\n\tmtChan := make(chan chan metaTuple, runtime.NumCPU())\n\n\tmakeChunk := func() {\n\t\trv.Reset()\n\t\tcp := make([]byte, offset)\n\t\tcopy(cp, chunkBytes[0:offset])\n\n\t\tch := make(chan metaTuple)\n\t\tmtChan <- ch\n\n\t\tgo func(ch chan metaTuple, cp []byte) {\n\t\t\tcol, key, numLeaves := chunkBlobLeaf(vrw, cp)\n\t\t\tch <- newMetaTuple(vrw.WriteValue(col), key, numLeaves)\n\t\t}(ch, cp)\n\n\t\toffset = 0\n\t}\n\n\tgo func() {\n\t\treadBuff := [8192]byte{}\n\t\tfor {\n\t\t\tn, err := r.Read(readBuff[:])\n\t\t\tfor i := 0; i < n; i++ {\n\t\t\t\tif addByte(readBuff[i]) {\n\t\t\t\t\tmakeChunk()\n\t\t\t\t}\n\t\t\t}\n\t\t\tif err != nil {\n\t\t\t\tif err != io.EOF {\n\t\t\t\t\tpanic(err)\n\t\t\t\t}\n\t\t\t\tif offset > 0 {\n\t\t\t\t\tmakeChunk()\n\t\t\t\t}\n\t\t\t\tclose(mtChan)\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}()\n\n\tfor ch := range mtChan {\n\t\tmt := <-ch\n\t\tif sc.parent == nil {\n\t\t\tsc.createParent()\n\t\t}\n\t\tsc.parent.Append(mt)\n\t}\n\n\treturn newBlob(sc.Done())\n}\n" }, { "path": "go/types/blob_editor.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"errors\"\n\n\t\"io\"\n\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype BlobEditor struct {\n\tb Blob\n\tedits *blobEdit\n\tpos int64\n}\n\nfunc NewBlobEditor(b Blob) *BlobEditor {\n\treturn &BlobEditor{b, nil, 0}\n}\n\nfunc (be *BlobEditor) Kind() NomsKind {\n\treturn BlobKind\n}\n\nfunc (be *BlobEditor) Value() Value {\n\treturn be.Blob()\n}\n\nfunc (be *BlobEditor) Blob() Blob {\n\tif be.edits == nil {\n\t\treturn be.b // no edits\n\t}\n\n\tseq := be.b.sequence\n\tvrw := seq.valueReadWriter()\n\n\tcurs := make([]chan *sequenceCursor, 0)\n\tfor edit := be.edits; edit != nil; edit = edit.next {\n\t\tedit := edit\n\n\t\t// TODO: Use ReadMany\n\t\tcc := make(chan *sequenceCursor, 1)\n\t\tcurs = append(curs, cc)\n\t\tgo func() {\n\t\t\tcc <- newCursorAtIndex(seq, edit.idx)\n\t\t}()\n\t}\n\n\tvar ch *sequenceChunker\n\tidx := 0\n\tfor edit := be.edits; edit != nil; edit = edit.next {\n\t\tcur := <-curs[idx]\n\t\tidx++\n\n\t\tif ch == nil {\n\t\t\tch = newSequenceChunker(cur, 0, vrw, makeBlobLeafChunkFn(vrw), newIndexedMetaSequenceChunkFn(BlobKind, vrw), hashValueByte)\n\t\t} else {\n\t\t\tch.advanceTo(cur)\n\t\t}\n\n\t\tdc := edit.removed\n\t\tfor dc > 0 {\n\t\t\tch.Skip()\n\t\t\tdc--\n\t\t}\n\n\t\tfor _, v := range edit.inserted {\n\t\t\tch.Append(v)\n\t\t}\n\t}\n\n\treturn newBlob(ch.Done())\n}\n\nfunc collapseBlobEdit(newEdit, edit *blobEdit) bool {\n\tif newEdit.idx+newEdit.removed < edit.idx ||\n\t\tedit.idx+uint64(len(edit.inserted)) < newEdit.idx {\n\t\treturn false\n\t}\n\n\tcollapsed := &blobEdit{}\n\n\tif newEdit.idx <= edit.idx {\n\t\tcollapsed.idx = newEdit.idx\n\n\t\toverlap := newEdit.removed - (edit.idx - newEdit.idx) // number of leading N values removed from edit.inserted\n\t\tif overlap < uint64(len(edit.inserted)) {\n\t\t\t// newEdit doesn't remove all of edit.inserted\n\t\t\tcollapsed.inserted = append(newEdit.inserted, edit.inserted[overlap:]...)\n\t\t\tcollapsed.removed = newEdit.removed + edit.removed - overlap\n\t\t} else {\n\t\t\t// newEdit removes all of edit.inserted\n\t\t\tcollapsed.inserted = newEdit.inserted\n\t\t\tcollapsed.removed = newEdit.removed + edit.removed - uint64(len(edit.inserted))\n\t\t}\n\t} else {\n\t\t// edit.idx < newEdit.idx\n\n\t\tcollapsed.idx = edit.idx\n\n\t\teditInsertedLen := uint64(len(edit.inserted))\n\t\tbeginEditRemovePoint := newEdit.idx - edit.idx\n\n\t\tif beginEditRemovePoint == editInsertedLen {\n\t\t\t// newEdit took place at the position immediately after the last element of edit.inserted\n\t\t\tcollapsed.inserted = append(edit.inserted, newEdit.inserted...)\n\t\t\tcollapsed.removed = edit.removed + newEdit.removed\n\t\t} else {\n\t\t\t// newEdit takes place within edit.inserted\n\t\t\tcollapsed.inserted = append(collapsed.inserted, edit.inserted[:beginEditRemovePoint]...)\n\t\t\tcollapsed.inserted = append(collapsed.inserted, newEdit.inserted...)\n\n\t\t\tendEditRemovePoint := beginEditRemovePoint + newEdit.removed\n\t\t\tif endEditRemovePoint < editInsertedLen {\n\t\t\t\t// elements of edit.inserted remain beyond newEdit.removed\n\t\t\t\tcollapsed.removed = edit.removed\n\t\t\t\tcollapsed.inserted = append(collapsed.inserted, edit.inserted[endEditRemovePoint:]...)\n\t\t\t} else {\n\t\t\t\tcollapsed.removed = edit.removed + endEditRemovePoint - editInsertedLen\n\t\t\t}\n\t\t}\n\t}\n\n\t*newEdit = *collapsed\n\treturn true\n}\n\nfunc (be *BlobEditor) Len() uint64 {\n\tdelta := int64(0)\n\tfor edit := be.edits; edit != nil; edit = edit.next {\n\t\tdelta += -int64(edit.removed) + int64(len(edit.inserted))\n\t}\n\n\treturn uint64(int64(be.b.Len()) + delta)\n}\n\nfunc (be *BlobEditor) Splice(idx uint64, deleteCount uint64, insert []byte) *BlobEditor {\n\tne := &blobEdit{idx, deleteCount, insert, nil}\n\n\tvar last *blobEdit\n\tedit := be.edits\n\n\tfor edit != nil {\n\t\tif collapseBlobEdit(ne, edit) {\n\t\t\tif last == nil {\n\t\t\t\tbe.edits = edit.next\n\t\t\t} else {\n\t\t\t\tlast.next = edit.next\n\t\t\t}\n\n\t\t\tedit = edit.next\n\t\t\tcontinue\n\t\t}\n\n\t\tif edit.idx > ne.idx {\n\t\t\tbreak\n\t\t}\n\n\t\tne.idx = adjustBlobIdx(ne.idx, edit)\n\t\tlast = edit\n\t\tedit = edit.next\n\t}\n\n\tif ne.removed == 0 && len(ne.inserted) == 0 {\n\t\treturn be // effectively removed 1 or more existing slices\n\t}\n\n\tif ne.idx > be.b.Len() {\n\t\td.Panic(\"Index Out Of Bounds\")\n\t}\n\tif ne.idx == be.b.Len() && ne.removed > 0 {\n\t\td.Panic(\"Index Out Of Bounds\")\n\t}\n\n\tif last == nil {\n\t\t// Insert |ne| in first position\n\t\tne.next = be.edits\n\t\tbe.edits = ne\n\t} else {\n\t\tne.next = last.next\n\t\tlast.next = ne\n\t}\n\n\treturn be\n}\n\nfunc (be *BlobEditor) Seek(offset int64, whence int) (int64, error) {\n\tabs := int64(be.pos)\n\n\tswitch whence {\n\tcase 0:\n\t\tabs = offset\n\tcase 1:\n\t\tabs += offset\n\tcase 2:\n\t\tabs = int64(be.Len()) + offset\n\tdefault:\n\t\treturn 0, errors.New(\"BlobEditor.Seek: invalid whence\")\n\t}\n\n\tif abs < 0 {\n\t\treturn 0, errors.New(\"BlobEditor.Seek: negative position\")\n\t}\n\n\tif uint64(abs) > be.Len() {\n\t\treturn 0, errors.New(\"BlobEditor.Seek: sparse blobs not supported\")\n\t}\n\n\tbe.pos = int64(abs)\n\treturn abs, nil\n}\n\nfunc (be *BlobEditor) Read(p []byte) (n int, err error) {\n\tstartIdx := uint64(be.pos)\n\tendIdx := startIdx + uint64(len(p))\n\tif endIdx > be.Len() {\n\t\tendIdx = be.Len()\n\t}\n\tn = int(endIdx - startIdx)\n\tif endIdx == be.Len() {\n\t\terr = io.EOF\n\t}\n\n\twg := &sync.WaitGroup{}\n\tasyncReadAt := func(length uint64) {\n\t\tidx := int64(startIdx)\n\t\tto := p[:length]\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tbe.b.ReadAt(to, idx)\n\t\t\twg.Done()\n\t\t}()\n\n\t\tstartIdx += length\n\t\tp = p[length:]\n\t}\n\n\tedit := be.edits\n\tfor edit != nil && startIdx < endIdx {\n\t\tif edit.idx > startIdx {\n\t\t\t// ReadAt the bytes before the current edit\n\t\t\tend := endIdx\n\t\t\tif endIdx > edit.idx {\n\t\t\t\tend = edit.idx\n\t\t\t}\n\n\t\t\tasyncReadAt(end - startIdx)\n\t\t\tcontinue\n\t\t}\n\n\t\tinsertedLength := uint64(len(edit.inserted))\n\t\tif edit.idx <= startIdx && startIdx < (edit.idx+insertedLength) {\n\t\t\t// Copy bytes within the current edit\n\t\t\tstart := startIdx - edit.idx\n\t\t\tend := endIdx - edit.idx\n\t\t\tif end > insertedLength {\n\t\t\t\tend = insertedLength\n\t\t\t}\n\n\t\t\tcopy(p, edit.inserted[start:end])\n\t\t\tp = p[end-start:]\n\t\t\tstartIdx += end - start\n\t\t\tcontinue\n\t\t}\n\n\t\tstartIdx = adjustBlobIdx(startIdx, edit)\n\t\tendIdx = adjustBlobIdx(endIdx, edit)\n\t\tedit = edit.next\n\t}\n\n\tif endIdx > startIdx {\n\t\t// ReadAt any bytes beyond the final edit\n\t\tasyncReadAt(endIdx - startIdx)\n\t}\n\n\twg.Wait()\n\treturn\n}\n\nfunc (be *BlobEditor) Write(p []byte) (n int, err error) {\n\tremoveCount := uint64(len(p))\n\tremaining := be.Len() - uint64(be.pos)\n\tif remaining < removeCount {\n\t\tremoveCount = remaining\n\t}\n\n\tbe.Splice(uint64(be.pos), removeCount, p)\n\treturn len(p), nil\n}\n\nfunc adjustBlobIdx(idx uint64, e *blobEdit) uint64 {\n\treturn idx + e.removed - uint64(len(e.inserted))\n}\n\ntype blobEdit struct {\n\tidx uint64\n\tremoved uint64\n\tinserted []byte\n\tnext *blobEdit\n}\n" }, { "path": "go/types/blob_editor_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"math/rand\"\n\t\"testing\"\n\n\t\"io/ioutil\"\n\n\t\"bytes\"\n\n\t\"io\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestBlobReadWriteFuzzer(t *testing.T) {\n\trounds := 1024\n\toperations := 512\n\tflushEvery := 16\n\tmaxInsertCount := uint64(64)\n\n\tts := &chunks.TestStorage{}\n\tcs := ts.NewView()\n\tvs := newValueStoreWithCacheAndPending(cs, 0, 0)\n\n\tr := rand.New(rand.NewSource(0))\n\tnextRandInt := func(from, to uint64) uint64 {\n\t\treturn from + uint64(float64(to-from)*r.Float64())\n\t}\n\n\tfor i := 0; i < rounds; i++ {\n\t\tb := NewBlob(vs)\n\n\t\tf, _ := ioutil.TempFile(\"\", \"buff\")\n\t\tbe := b.Edit()\n\n\t\tfor j := 0; j < operations; j++ {\n\t\t\tif j%2 == 1 {\n\t\t\t\t// random read\n\t\t\t\tidx := nextRandInt(0, be.Len())\n\t\t\t\tl := nextRandInt(0, be.Len()-idx)\n\t\t\t\tf.Seek(int64(idx), 0)\n\t\t\t\tbe.Seek(int64(idx), 0)\n\n\t\t\t\tex := make([]byte, l)\n\t\t\t\tac := make([]byte, l)\n\n\t\t\t\tf.Read(ex)\n\t\t\t\tbe.Read(ac)\n\t\t\t\tassert.True(t, bytes.Compare(ex, ac) == 0)\n\t\t\t} else {\n\t\t\t\t// randon write\n\t\t\t\tidx := nextRandInt(0, be.Len())\n\t\t\t\tf.Seek(int64(idx), 0)\n\t\t\t\tbe.Seek(int64(idx), 0)\n\n\t\t\t\tl := nextRandInt(0, maxInsertCount)\n\t\t\t\tdata, err := ioutil.ReadAll(&io.LimitedReader{R: r, N: int64(l)})\n\t\t\t\tassert.NoError(t, err)\n\t\t\t\tf.Write(data)\n\t\t\t\tbe.Write(data)\n\t\t\t}\n\t\t\tif j%flushEvery == 0 {\n\t\t\t\t// Flush\n\t\t\t\tb = be.Blob()\n\t\t\t\tbe = b.Edit()\n\t\t\t}\n\t\t}\n\n\t\tf.Sync()\n\t\tb = be.Blob()\n\n\t\tf.Seek(0, 0)\n\t\tinfo, err := f.Stat()\n\t\tassert.NoError(t, err)\n\t\tassert.True(t, uint64(info.Size()) == b.Len())\n\t\texpect, err := ioutil.ReadAll(f)\n\t\tassert.NoError(t, err)\n\n\t\tactual := make([]byte, b.Len())\n\t\tb.ReadAt(actual, 0)\n\n\t\tassert.True(t, bytes.Compare(expect, actual) == 0)\n\t}\n}\n" }, { "path": "go/types/blob_leaf_sequence.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport \"github.com/attic-labs/noms/go/d\"\n\ntype blobLeafSequence struct {\n\tleafSequence\n}\n\nfunc newBlobLeafSequence(vrw ValueReadWriter, data []byte) sequence {\n\td.PanicIfTrue(vrw == nil)\n\toffsets := make([]uint32, sequencePartValues+1)\n\tw := newBinaryNomsWriter()\n\toffsets[sequencePartKind] = w.offset\n\tBlobKind.writeTo(&w)\n\toffsets[sequencePartLevel] = w.offset\n\tw.writeCount(0) // level\n\toffsets[sequencePartCount] = w.offset\n\tcount := uint64(len(data))\n\tw.writeCount(count)\n\toffsets[sequencePartValues] = w.offset\n\tw.writeBytes(data)\n\treturn blobLeafSequence{newLeafSequence(vrw, w.data(), offsets, count)}\n}\n\nfunc (bl blobLeafSequence) writeTo(w nomsWriter) {\n\tw.writeRaw(bl.buff)\n}\n\n// sequence interface\n\nfunc (bl blobLeafSequence) data() []byte {\n\toffset := bl.offsets[sequencePartValues] - bl.offsets[sequencePartKind]\n\treturn bl.buff[offset:]\n}\n\nfunc (bl blobLeafSequence) getCompareFn(other sequence) compareFn {\n\toffsetStart := int(bl.offsets[sequencePartValues] - bl.offsets[sequencePartKind])\n\tobl := other.(blobLeafSequence)\n\totherOffsetStart := int(obl.offsets[sequencePartValues] - obl.offsets[sequencePartKind])\n\treturn func(idx, otherIdx int) bool {\n\t\treturn bl.buff[offsetStart+idx] == obl.buff[otherOffsetStart+otherIdx]\n\t}\n}\n\nfunc (bl blobLeafSequence) getItem(idx int) sequenceItem {\n\toffset := bl.offsets[sequencePartValues] - bl.offsets[sequencePartKind] + uint32(idx)\n\treturn bl.buff[offset]\n}\n\nfunc (bl blobLeafSequence) typeOf() *Type {\n\treturn BlobType\n}\n" }, { "path": "go/types/blob_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"math/rand\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc randomBuff(powOfTwo uint) []byte {\n\tlength := 1 << powOfTwo\n\trr := rand.New(rand.NewSource(int64(powOfTwo)))\n\tbuff := make([]byte, length)\n\trr.Read(buff)\n\treturn buff\n}\n\ntype blobTestSuite struct {\n\tcollectionTestSuite\n\tbuff []byte\n}\n\nfunc newBlobTestSuite(size uint, expectChunkCount int, expectPrependChunkDiff int, expectAppendChunkDiff int) *blobTestSuite {\n\tvrw := newTestValueStore()\n\n\tlength := 1 << size\n\tbuff := randomBuff(size)\n\tblob := NewBlob(vrw, bytes.NewReader(buff))\n\treturn &blobTestSuite{\n\t\tcollectionTestSuite: collectionTestSuite{\n\t\t\tcol: blob,\n\t\t\texpectType: BlobType,\n\t\t\texpectLen: uint64(length),\n\t\t\texpectChunkCount: expectChunkCount,\n\t\t\texpectPrependChunkDiff: expectPrependChunkDiff,\n\t\t\texpectAppendChunkDiff: expectAppendChunkDiff,\n\t\t\tvalidate: func(v2 Collection) bool {\n\t\t\t\tb2 := v2.(Blob)\n\t\t\t\toutBuff := &bytes.Buffer{}\n\t\t\t\tb2.Copy(outBuff)\n\t\t\t\treturn bytes.Compare(outBuff.Bytes(), buff) == 0\n\t\t\t},\n\t\t\tprependOne: func() Collection {\n\t\t\t\tdup := make([]byte, length+1)\n\t\t\t\tdup[0] = 0\n\t\t\t\tcopy(dup[1:], buff)\n\t\t\t\treturn NewBlob(vrw, bytes.NewReader(dup))\n\t\t\t},\n\t\t\tappendOne: func() Collection {\n\t\t\t\tdup := make([]byte, length+1)\n\t\t\t\tcopy(dup, buff)\n\t\t\t\tdup[len(dup)-1] = 0\n\t\t\t\treturn NewBlob(vrw, bytes.NewReader(dup))\n\t\t\t},\n\t\t},\n\t\tbuff: buff,\n\t}\n}\n\nfunc TestBlobSuite4K(t *testing.T) {\n\tsuite.Run(t, newBlobTestSuite(12, 2, 2, 2))\n}\n\nfunc TestBlobSuite64K(t *testing.T) {\n\tsuite.Run(t, newBlobTestSuite(16, 15, 2, 2))\n}\n\nfunc TestBlobSuite256K(t *testing.T) {\n\tsuite.Run(t, newBlobTestSuite(18, 64, 2, 2))\n}\n\nfunc TestBlobSuite1M(t *testing.T) {\n\tsuite.Run(t, newBlobTestSuite(20, 245, 2, 2))\n}\n\n// Checks the first 1/2 of the bytes, then 1/2 of the remainder, then 1/2 of the remainder, etc...\nfunc (suite *blobTestSuite) TestRandomRead() {\n\tbuffReader := bytes.NewReader(suite.buff)\n\tblobReader := suite.col.(Blob).Reader()\n\n\treadByteRange := func(r io.ReadSeeker, start, rel, count int64) []byte {\n\t\tbytes := make([]byte, count)\n\t\tn, err := r.Seek(start, 0)\n\t\tsuite.NoError(err)\n\t\tsuite.Equal(start, n)\n\t\tn2, err := r.Seek(rel, 1)\n\t\tsuite.NoError(err)\n\t\tsuite.Equal(start+rel, n2)\n\t\tn3, err := io.ReadFull(r, bytes)\n\t\tsuite.NoError(err)\n\t\tsuite.Equal(int(count), n3)\n\t\treturn bytes\n\t}\n\n\treadByteRangeFromEnd := func(r io.ReadSeeker, length, offset, count int64) []byte {\n\t\tbytes := make([]byte, count)\n\t\tn, err := r.Seek(offset, 2)\n\t\tsuite.NoError(err)\n\t\tsuite.Equal(length+offset, n)\n\t\tn2, err := io.ReadFull(r, bytes)\n\t\tsuite.NoError(err)\n\t\tsuite.Equal(int(count), n2)\n\t\treturn bytes\n\t}\n\n\tcheckByteRange := func(start, rel, count int64) {\n\t\texpect := readByteRange(buffReader, start, rel, count)\n\t\tactual := readByteRange(blobReader, start, rel, count)\n\t\tsuite.Equal(expect, actual)\n\t}\n\n\tcheckByteRangeFromEnd := func(length, offset, count int64) {\n\t\texpect := readByteRangeFromEnd(buffReader, length, offset, count)\n\t\tactual := readByteRangeFromEnd(blobReader, length, offset, count)\n\t\tsuite.Equal(expect, actual)\n\t}\n\n\tlength := int64(len(suite.buff))\n\tstart := int64(0)\n\tcount := int64(length / 2)\n\tfor count > 2 {\n\t\tcheckByteRange(start, 0, count)\n\t\tcheckByteRange(0, start, count)\n\t\tcheckByteRange(start/2, start-(start/2), count)\n\t\tcheckByteRangeFromEnd(length, start-length, count)\n\t\tstart = start + count\n\t\tcount = (length - start) / 2\n\t}\n}\n\ntype testReader struct {\n\treadCount int\n\tbuf *bytes.Buffer\n}\n\nfunc (r *testReader) Read(p []byte) (n int, err error) {\n\tr.readCount++\n\n\tswitch r.readCount {\n\tcase 1:\n\t\tfor i := 0; i < len(p); i++ {\n\t\t\tp[i] = 0x01\n\t\t}\n\t\tio.Copy(r.buf, bytes.NewReader(p))\n\t\treturn len(p), nil\n\tcase 2:\n\t\tp[0] = 0x02\n\t\tr.buf.WriteByte(p[0])\n\t\treturn 1, io.EOF\n\tdefault:\n\t\treturn 0, io.EOF\n\t}\n}\n\nfunc TestBlobFromReaderThatReturnsDataAndError(t *testing.T) {\n\t// See issue #264.\n\t// This tests the case of building a Blob from a reader who returns both data and an error for the final Read() call.\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\ttr := &testReader{buf: &bytes.Buffer{}}\n\n\tb := NewBlob(vrw, tr)\n\n\tactual := &bytes.Buffer{}\n\tio.Copy(actual, b.Reader())\n\n\tassert.True(bytes.Equal(actual.Bytes(), tr.buf.Bytes()))\n\tassert.Equal(byte(2), actual.Bytes()[len(actual.Bytes())-1])\n}\n\nfunc TestBlobSplice(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tblob := NewEmptyBlob(vrw)\n\tbuf := new(bytes.Buffer)\n\n\tblob = blob.Edit().Splice(0, 0, []byte(\"I'll do anything\")).Blob()\n\tbuf.Reset()\n\tbuf.ReadFrom(blob.Reader())\n\tassert.Equal(buf.String(), \"I'll do anything\")\n\n\tblob = blob.Edit().Splice(16, 0, []byte(\" for arv\")).Blob()\n\tbuf.Reset()\n\tbuf.ReadFrom(blob.Reader())\n\tassert.Equal(buf.String(), \"I'll do anything for arv\")\n\n\tblob = blob.Edit().Splice(0, 0, []byte(\"Yes, \")).Blob()\n\tbuf.Reset()\n\tbuf.ReadFrom(blob.Reader())\n\tassert.Equal(buf.String(), \"Yes, I'll do anything for arv\")\n\n\tblob = blob.Edit().Splice(5, 20, []byte(\"it's hard to satisfy\")).Blob()\n\tbuf.Reset()\n\tbuf.ReadFrom(blob.Reader())\n\tassert.Equal(buf.String(), \"Yes, it's hard to satisfy arv\")\n}\n\nfunc TestBlobConcat(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvs := newTestValueStore()\n\treload := func(b Blob) Blob {\n\t\treturn vs.ReadValue(vs.WriteValue(b).TargetHash()).(Blob)\n\t}\n\n\tsplit := func(b Blob, at int64) (Blob, Blob) {\n\t\tread1, read2 := b.Reader(), b.Reader()\n\t\tb1 := NewBlob(vs, &io.LimitedReader{R: read1, N: at})\n\t\tread2.Seek(at, 0)\n\t\tb2 := NewBlob(vs, read2)\n\t\treturn reload(b1), reload(b2)\n\t}\n\n\t// Random 1MB Blob.\n\t// Note that List.Concat is exhaustively tested, don't worry here.\n\tr := rand.New(rand.NewSource(0))\n\tb := NewBlob(vs, &io.LimitedReader{R: r, N: 1e6})\n\tb = reload(b)\n\n\tb1 := NewEmptyBlob(vs).Concat(b)\n\tassert.True(b.Equals(b1))\n\n\tb2 := b.Concat(NewEmptyBlob(vs))\n\tassert.True(b.Equals(b2))\n\n\tb3, b4 := split(b, 10)\n\tassert.True(b.Equals(b3.Concat(b4)))\n\n\tb5, b6 := split(b, 1e6-10)\n\tassert.True(b.Equals(b5.Concat(b6)))\n\n\tb7, b8 := split(b, 1e6/2)\n\tassert.True(b.Equals(b7.Concat(b8)))\n}\n\nfunc TestBlobNewParallel(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\treadAll := func(b Blob) []byte {\n\t\tdata, err := ioutil.ReadAll(b.Reader())\n\t\tassert.NoError(err)\n\t\treturn data\n\t}\n\n\tb := NewBlob(vrw)\n\tassert.True(b.Len() == 0)\n\n\tb = NewBlob(vrw, strings.NewReader(\"abc\"))\n\tassert.Equal(\"abc\", string(readAll(b)))\n\n\tb = NewBlob(vrw, strings.NewReader(\"abc\"), strings.NewReader(\"def\"))\n\tassert.Equal(\"abcdef\", string(readAll(b)))\n\n\tp, size := 100, 1024\n\tr := rand.New(rand.NewSource(0))\n\tdata := make([]byte, p*size)\n\t_, err := r.Read(data)\n\tassert.NoError(err)\n\n\treaders := make([]io.Reader, p)\n\tfor i := range readers {\n\t\treaders[i] = bytes.NewBuffer(data[i*size : (i+1)*size])\n\t}\n\n\tb = NewBlob(vrw, readers...)\n\tassert.Equal(data, readAll(b))\n}\n\nfunc TestStreamingParallelBlob(t *testing.T) {\n\tassert := assert.New(t)\n\n\tbuff := randomBuff(1 << 26 /* 64MB */)\n\tchunks := 4\n\treaders := make([]io.Reader, chunks)\n\tchunkSize := len(buff) / chunks\n\n\tfor i := 0; i < len(readers); i++ {\n\t\treaders[i] = bytes.NewReader(buff[i*chunkSize : (i+1)*chunkSize])\n\t}\n\n\tvs := newTestValueStore()\n\tblob := NewBlob(vs, readers...)\n\toutBuff := &bytes.Buffer{}\n\tblob.Copy(outBuff)\n\tassert.True(bytes.Compare(buff, outBuff.Bytes()) == 0)\n}\n" }, { "path": "go/types/bool.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n// Bool is a Noms Value wrapper around the primitive bool type.\ntype Bool bool\n\n// Value interface\nfunc (b Bool) Value() Value {\n\treturn b\n}\n\nfunc (b Bool) Equals(other Value) bool {\n\treturn b == other\n}\n\nfunc (b Bool) Less(other Value) bool {\n\tif b2, ok := other.(Bool); ok {\n\t\treturn !bool(b) && bool(b2)\n\t}\n\treturn true\n}\n\nfunc (b Bool) Hash() hash.Hash {\n\treturn getHash(b)\n}\n\nfunc (b Bool) WalkValues(cb ValueCallback) {\n}\n\nfunc (b Bool) WalkRefs(cb RefCallback) {\n}\n\nfunc (b Bool) typeOf() *Type {\n\treturn BoolType\n}\n\nfunc (b Bool) Kind() NomsKind {\n\treturn BoolKind\n}\n\nfunc (b Bool) valueReadWriter() ValueReadWriter {\n\treturn nil\n}\n\nfunc (b Bool) writeTo(w nomsWriter) {\n\tBoolKind.writeTo(w)\n\tw.writeBool(bool(b))\n}\n\nfunc (b Bool) valueBytes() []byte {\n\tif bool(b) {\n\t\treturn []byte{byte(BoolKind), 1}\n\t}\n\treturn []byte{byte(BoolKind), 0}\n}\n" }, { "path": "go/types/codec.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"encoding/binary\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\nconst initialBufferSize = 2048\n\ntype valueBytes interface {\n\tvalueBytes() []byte\n}\n\nfunc EncodeValue(v Value) chunks.Chunk {\n\tswitch v := v.(type) {\n\tcase valueBytes:\n\t\treturn chunks.NewChunk(v.valueBytes())\n\tcase *Type:\n\t\tw := newBinaryNomsWriter()\n\t\tv.writeTo(&w)\n\t\treturn chunks.NewChunk(w.data())\n\t}\n\n\tpanic(\"unreachable\")\n}\n\nfunc DecodeFromBytes(data []byte, vrw ValueReadWriter) Value {\n\tdec := newValueDecoder(data, vrw)\n\tv := dec.readValue()\n\td.PanicIfFalse(dec.pos() == uint32(len(data)))\n\treturn v\n}\n\nfunc decodeFromBytesWithValidation(data []byte, vrw ValueReadWriter) Value {\n\tr := binaryNomsReader{data, 0}\n\tdec := newValueDecoderWithValidation(r, vrw)\n\tv := dec.readValue()\n\td.PanicIfFalse(dec.pos() == uint32(len(data)))\n\treturn v\n}\n\n// DecodeValue decodes a value from a chunk source. It is an error to provide an empty chunk.\nfunc DecodeValue(c chunks.Chunk, vrw ValueReadWriter) Value {\n\td.PanicIfTrue(c.IsEmpty())\n\treturn DecodeFromBytes(c.Data(), vrw)\n}\n\ntype nomsWriter interface {\n\twriteBool(b bool)\n\twriteBytes(v []byte)\n\twriteCount(count uint64)\n\twriteHash(h hash.Hash)\n\twriteNumber(v Number)\n\twriteString(v string)\n\twriteUint8(v uint8)\n\n\twriteRaw(buff []byte)\n}\n\ntype binaryNomsReader struct {\n\tbuff []byte\n\toffset uint32\n}\n\nfunc (b *binaryNomsReader) pos() uint32 {\n\treturn b.offset\n}\n\nfunc (b *binaryNomsReader) readUint8() uint8 {\n\tv := uint8(b.buff[b.offset])\n\tb.offset++\n\treturn v\n}\n\nfunc (b *binaryNomsReader) peekUint8() uint8 {\n\treturn uint8(b.buff[b.offset])\n}\n\nfunc (b *binaryNomsReader) skipUint8() {\n\tb.offset++\n}\n\nfunc (b *binaryNomsReader) peekKind() NomsKind {\n\treturn NomsKind(b.peekUint8())\n}\n\nfunc (b *binaryNomsReader) readKind() NomsKind {\n\treturn NomsKind(b.readUint8())\n}\n\nfunc (b *binaryNomsReader) skipKind() {\n\tb.skipUint8()\n}\n\nfunc (b *binaryNomsReader) readCount() uint64 {\n\tv, count := binary.Uvarint(b.buff[b.offset:])\n\tb.offset += uint32(count)\n\treturn v\n}\n\nfunc (b *binaryNomsReader) skipCount() {\n\t_, count := binary.Uvarint(b.buff[b.offset:])\n\tb.offset += uint32(count)\n}\n\nfunc (b *binaryNomsReader) readNumber() Number {\n\t// b.assertCanRead(binary.MaxVarintLen64 * 2)\n\ti, count := binary.Varint(b.buff[b.offset:])\n\tb.offset += uint32(count)\n\texp, count2 := binary.Varint(b.buff[b.offset:])\n\tb.offset += uint32(count2)\n\treturn Number(fracExpToFloat(i, int(exp)))\n}\n\nfunc (b *binaryNomsReader) skipNumber() {\n\t_, count := binary.Varint(b.buff[b.offset:])\n\tb.offset += uint32(count)\n\t_, count2 := binary.Varint(b.buff[b.offset:])\n\tb.offset += uint32(count2)\n}\n\nfunc (b *binaryNomsReader) readBool() bool {\n\treturn b.readUint8() == 1\n}\n\nfunc (b *binaryNomsReader) skipBool() {\n\tb.skipUint8()\n}\n\nfunc (b *binaryNomsReader) readString() string {\n\tsize := uint32(b.readCount())\n\n\tv := string(b.buff[b.offset : b.offset+size])\n\tb.offset += size\n\treturn v\n}\n\nfunc (b *binaryNomsReader) skipString() {\n\tsize := uint32(b.readCount())\n\tb.offset += size\n}\n\nfunc (b *binaryNomsReader) readHash() hash.Hash {\n\th := hash.Hash{}\n\tcopy(h[:], b.buff[b.offset:b.offset+hash.ByteLen])\n\tb.offset += hash.ByteLen\n\treturn h\n}\n\nfunc (b *binaryNomsReader) skipHash() {\n\tb.offset += hash.ByteLen\n}\n\nfunc (b *binaryNomsReader) byteSlice(start, end uint32) []byte {\n\treturn b.buff[start:end]\n}\n\ntype binaryNomsWriter struct {\n\tbuff []byte\n\toffset uint32\n}\n\nfunc newBinaryNomsWriter() binaryNomsWriter {\n\treturn binaryNomsWriter{make([]byte, initialBufferSize), 0}\n}\n\nfunc (b *binaryNomsWriter) data() []byte {\n\treturn b.buff[0:b.offset]\n}\n\nfunc (b *binaryNomsWriter) reset() {\n\tb.offset = 0\n}\n\nfunc (b *binaryNomsWriter) ensureCapacity(n uint32) {\n\tlength := uint32(len(b.buff))\n\tif b.offset+n <= length {\n\t\treturn\n\t}\n\n\told := b.buff\n\n\tfor b.offset+n > length {\n\t\tlength = length * 2\n\t}\n\tb.buff = make([]byte, length, length)\n\n\tcopy(b.buff, old)\n}\n\nfunc (b *binaryNomsWriter) writeBytes(v []byte) {\n\tsize := uint32(len(v))\n\tb.ensureCapacity(size)\n\tcopy(b.buff[b.offset:], v)\n\tb.offset += size\n}\n\nfunc (b *binaryNomsWriter) writeUint8(v uint8) {\n\tb.ensureCapacity(1)\n\tb.buff[b.offset] = byte(v)\n\tb.offset++\n}\n\nfunc (b *binaryNomsWriter) writeCount(v uint64) {\n\tb.ensureCapacity(binary.MaxVarintLen64)\n\tcount := binary.PutUvarint(b.buff[b.offset:], v)\n\tb.offset += uint32(count)\n}\n\nfunc (b *binaryNomsWriter) writeNumber(v Number) {\n\tb.ensureCapacity(binary.MaxVarintLen64 * 2)\n\ti, exp := float64ToIntExp(float64(v))\n\tcount := binary.PutVarint(b.buff[b.offset:], i)\n\tb.offset += uint32(count)\n\tcount = binary.PutVarint(b.buff[b.offset:], int64(exp))\n\tb.offset += uint32(count)\n}\n\nfunc (b *binaryNomsWriter) writeBool(v bool) {\n\tif v {\n\t\tb.writeUint8(uint8(1))\n\t} else {\n\t\tb.writeUint8(uint8(0))\n\t}\n}\n\nfunc (b *binaryNomsWriter) writeString(v string) {\n\tsize := uint32(len(v))\n\tb.writeCount(uint64(size))\n\n\tb.ensureCapacity(size)\n\tcopy(b.buff[b.offset:], v)\n\tb.offset += size\n}\n\nfunc (b *binaryNomsWriter) writeHash(h hash.Hash) {\n\tb.ensureCapacity(hash.ByteLen)\n\tcopy(b.buff[b.offset:], h[:])\n\tb.offset += hash.ByteLen\n}\n\nfunc (b *binaryNomsWriter) writeRaw(buff []byte) {\n\tb.ensureCapacity(uint32(len(buff)))\n\tcopy(b.buff[b.offset:], buff)\n\tb.offset += uint32(len(buff))\n}\n" }, { "path": "go/types/codec_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestCodecWriteNumber(t *testing.T) {\n\ttest := func(f float64, exp []byte) {\n\t\tw := newBinaryNomsWriter()\n\t\tw.writeNumber(Number(f))\n\t\tassert.Equal(t, exp, w.data())\n\t}\n\n\t// We use zigzag encoding for the signed bit. For positive n we do 2*n and for negative we do 2*-n - 1\n\ttest(0, []byte{0, 0}) // 0 * 2 ** 0\n\n\ttest(1, []byte{1 * 2, 0}) // 1 * 2 ** 0\n\ttest(2, []byte{1 * 2, 1 * 2}) // 1 * 2 ** 1\n\ttest(-2, []byte{(1 * 2) - 1, 1 * 2}) // -1 * 2 ** 1\n\ttest(.5, []byte{1 * 2, 1*2 - 1}) // 1 * 2 ** -1\n\ttest(-.5, []byte{1*2 - 1, 1*2 - 1}) // -1 * 2 ** -1\n\ttest(.25, []byte{1 * 2, 2*2 - 1}) // 1 * 2 ** -2\n\ttest(3, []byte{3 * 2, 0}) // 0b11 * 2 ** 0\n\n\ttest(15, []byte{15 * 2, 0}) // 0b1111 * 2**0\n\ttest(256, []byte{1 * 2, 8 * 2}) // 1 * 2*8\n\ttest(-15, []byte{15*2 - 1, 0}) // -15 * 2*0\n}\n\nfunc TestCodecReadNumber(t *testing.T) {\n\ttest := func(data []byte, exp float64) {\n\t\tr := binaryNomsReader{buff: data}\n\t\tn := r.readNumber()\n\t\tassert.Equal(t, exp, float64(n))\n\t\tassert.Equal(t, len(data), int(r.offset))\n\t}\n\n\ttest([]byte{0, 0}, 0) // 0 * 2 ** 0\n\n\ttest([]byte{1 * 2, 0}, 1) // 1 * 2 ** 0\n\ttest([]byte{1 * 2, 1 * 2}, 2) // 1 * 2 ** 1\n\ttest([]byte{1*2 - 1, 1 + 1}, -2) // -1 * 2 ** 1\n\ttest([]byte{1 * 2, 1*2 - 1}, .5) // 1 * 2 ** -1\n\ttest([]byte{1*2 - 1, 1*2 - 1}, -.5) // -1 * 2 ** -1\n\ttest([]byte{1 * 2, 2*2 - 1}, .25) // 1 * 2 ** -2\n\ttest([]byte{3 * 2, 0}, 3) // 0b11 * 2 ** 0\n\n\ttest([]byte{15 * 2, 0}, 15) // 0b1111 * 2**0\n\ttest([]byte{1 * 2, 8 * 2}, 256) // 1 * 2*8\n\ttest([]byte{15*2 - 1, 0}, -15) // -15 * 2*0\n}\n" }, { "path": "go/types/collection.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\ntype Collection interface {\n\tValue\n\tEmpty() bool\n\tLen() uint64\n\tasSequence() sequence\n}\n" }, { "path": "go/types/collection_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport \"github.com/stretchr/testify/suite\"\n\ntype collectionTestSuite struct {\n\tsuite.Suite\n\tcol Collection\n\texpectType *Type\n\texpectLen uint64\n\texpectChunkCount int\n\texpectPrependChunkDiff int\n\texpectAppendChunkDiff int\n\tvalidate validateFn\n\tprependOne deltaFn\n\tappendOne deltaFn\n}\n\ntype validateFn func(v2 Collection) bool\ntype deltaFn func() Collection\n\nfunc (suite *collectionTestSuite) TestType() {\n\tsuite.True(suite.expectType.Equals(TypeOf(suite.col)))\n}\n\nfunc (suite *collectionTestSuite) TestLen() {\n\tsuite.Equal(suite.expectLen, suite.col.Len())\n\tsuite.Equal(suite.col.Empty(), suite.expectLen == 0)\n}\n\nfunc (suite *collectionTestSuite) TestEquals() {\n\tv2 := suite.col\n\tsuite.True(suite.col.Equals(v2))\n\tsuite.True(v2.Equals(suite.col))\n}\n\nfunc (suite *collectionTestSuite) TestChunkCountAndType() {\n\tsuite.Equal(suite.expectChunkCount, leafCount(suite.col), \"chunk count\")\n\trefType := MakeRefType(suite.expectType)\n\tsuite.col.WalkRefs(func(r Ref) {\n\t\tsuite.True(refType.Equals(TypeOf(r)))\n\t})\n}\n\nfunc (suite *collectionTestSuite) TestRoundTripAndValidate() {\n\tsuite.True(suite.validate(suite.col))\n}\n\nfunc (suite *collectionTestSuite) TestPrependChunkDiff() {\n\tv2 := suite.prependOne()\n\tsuite.Equal(suite.expectPrependChunkDiff, leafDiffCount(suite.col, v2), \"prepend count\")\n}\n\nfunc (suite *collectionTestSuite) TestAppendChunkDiff() {\n\tv2 := suite.appendOne()\n\tsuite.Equal(suite.expectAppendChunkDiff, leafDiffCount(suite.col, v2), \"append count\")\n}\n\nfunc deriveCollectionHeight(c Collection) uint64 {\n\treturn c.asSequence().treeLevel()\n}\n\nfunc getRefHeightOfCollection(c Collection) uint64 {\n\treturn c.asSequence().getItem(0).(metaTuple).ref().Height()\n}\n" }, { "path": "go/types/common_supertype.go", "content": "package types\n\nimport \"github.com/attic-labs/noms/go/d\"\n\n// ContainCommonSupertype returns true if it's possible to synthesize\n// a non-trivial (i.e. not empty) supertype from types |a| and |b|.\n//\n// It is useful for determining whether a subset of values can be extracted\n// from one object to produce another object.\n//\n// The rules for determining whether |a| and |b| intersect are:\n// - if either type is Value, return true\n// - if either type is Union, return true iff at least one variant of |a| intersects with one variant of |b|\n// - if |a| & |b| are not the same kind, return false\n// - else\n// - if both are structs, return true iff their names are equal or one name is \"\", they share a field name\n// and the type of that field intersects\n// - if both are refs, sets or lists, return true iff the element type intersects\n// - if both are maps, return true iff they have a key with the same type and value types that intersect\n// - else return true\nfunc ContainCommonSupertype(a, b *Type) bool {\n\t// Avoid cycles internally.\n\treturn containCommonSupertypeImpl(a, b, nil, nil)\n}\n\nfunc containCommonSupertypeImpl(a, b *Type, aVisited, bVisited []*Type) bool {\n\tif a.TargetKind() == ValueKind || b.TargetKind() == ValueKind {\n\t\treturn true\n\t}\n\tif a.TargetKind() == UnionKind || b.TargetKind() == UnionKind {\n\t\treturn unionsIntersect(a, b, aVisited, bVisited)\n\t}\n\tif a.TargetKind() != b.TargetKind() {\n\t\treturn false\n\t}\n\tswitch k := a.TargetKind(); k {\n\tcase StructKind:\n\t\treturn structsIntersect(a, b, aVisited, bVisited)\n\tcase ListKind, SetKind, RefKind:\n\t\treturn containersIntersect(k, a, b, aVisited, bVisited)\n\tcase MapKind:\n\t\treturn mapsIntersect(a, b, aVisited, bVisited)\n\tdefault:\n\t\treturn true\n\t}\n\n}\n\n// Checks for intersection between types that may be unions. If either or\n// both is a union, union, tests all types for intersection.\nfunc unionsIntersect(a, b *Type, aVisited, bVisited []*Type) bool {\n\taTypes, bTypes := typeList(a), typeList(b)\n\tfor _, t := range aTypes {\n\t\tfor _, u := range bTypes {\n\t\t\tif containCommonSupertypeImpl(t, u, aVisited, bVisited) {\n\t\t\t\treturn true\n\t\t\t}\n\t\t}\n\t}\n\treturn false\n}\n\n// if |t| is a union, returns all types represented; otherwise returns |t|\nfunc typeList(t *Type) typeSlice {\n\tif t.Desc.Kind() == UnionKind {\n\t\treturn t.Desc.(CompoundDesc).ElemTypes\n\t}\n\treturn typeSlice{t}\n}\n\nfunc containersIntersect(kind NomsKind, a, b *Type, aVisited, bVisited []*Type) bool {\n\td.Chk.True(kind == a.Desc.Kind() && kind == b.Desc.Kind())\n\treturn containCommonSupertypeImpl(a.Desc.(CompoundDesc).ElemTypes[0], b.Desc.(CompoundDesc).ElemTypes[0], aVisited, bVisited)\n}\n\nfunc mapsIntersect(a, b *Type, aVisited, bVisited []*Type) bool {\n\t// true if a and b are the same or (if either is a union) there is\n\t// common type between them.\n\thasCommonType := func(a, b *Type) bool {\n\t\taTypes, bTypes := typeList(a), typeList(b)\n\t\tfor _, t := range aTypes {\n\t\t\tfor _, u := range bTypes {\n\t\t\t\tif t.Equals(u) {\n\t\t\t\t\treturn true\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn false\n\t}\n\n\td.Chk.True(MapKind == a.Desc.Kind() && MapKind == b.Desc.Kind())\n\n\taDesc, bDesc := a.Desc.(CompoundDesc), b.Desc.(CompoundDesc)\n\tif !hasCommonType(aDesc.ElemTypes[0], bDesc.ElemTypes[0]) {\n\t\treturn false\n\t}\n\treturn containCommonSupertypeImpl(aDesc.ElemTypes[1], bDesc.ElemTypes[1], aVisited, bVisited)\n}\n\nfunc structsIntersect(a, b *Type, aVisited, bVisited []*Type) bool {\n\t_, aFound := indexOfType(a, aVisited)\n\t_, bFound := indexOfType(b, bVisited)\n\n\tif aFound && bFound {\n\t\treturn true\n\t}\n\n\td.Chk.True(StructKind == a.TargetKind() && StructKind == b.TargetKind())\n\taDesc := a.Desc.(StructDesc)\n\tbDesc := b.Desc.(StructDesc)\n\t// must be either the same name or one has no name\n\tif aDesc.Name != bDesc.Name && !(aDesc.Name == \"\" || bDesc.Name == \"\") {\n\t\treturn false\n\t}\n\tfor i, j := 0, 0; i < len(aDesc.fields) && j < len(bDesc.fields); {\n\t\taName, bName := aDesc.fields[i].Name, bDesc.fields[j].Name\n\t\tif aName < bName {\n\t\t\ti++\n\t\t} else if bName < aName {\n\t\t\tj++\n\t\t} else if !containCommonSupertypeImpl(aDesc.fields[i].Type, bDesc.fields[j].Type, append(aVisited, a), append(bVisited, b)) {\n\t\t\ti++\n\t\t\tj++\n\t\t} else {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n" }, { "path": "go/types/common_supertype_test.go", "content": "package types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestContainCommonSupertype(t *testing.T) {\n\tcases := []struct {\n\t\ta, b *Type\n\t\tout bool\n\t}{\n\t\t// bool & any -> true\n\t\t{ValueType, StringType, true},\n\t\t// ref & ref -> true\n\t\t{MakeRefType(BoolType), MakeRefType(BoolType), true},\n\t\t// ref & ref -> false\n\t\t{MakeRefType(NumberType), MakeRefType(StringType), false},\n\t\t// set & set -> true\n\t\t{MakeSetType(BoolType), MakeSetType(BoolType), true},\n\t\t// set & set -> false\n\t\t{MakeSetType(BoolType), MakeSetType(StringType), false},\n\t\t// list & list -> true\n\t\t{MakeListType(BlobType), MakeListType(BlobType), true},\n\t\t// list & list -> false\n\t\t{MakeListType(BlobType), MakeListType(StringType), false},\n\t\t// list & list -> true\n\t\t{MakeListType(MakeUnionType(BlobType, StringType, NumberType)), MakeListType(MakeUnionType(StringType, BoolType)), true},\n\t\t// list & list -> false\n\t\t{MakeListType(MakeUnionType(BlobType, StringType)), MakeListType(MakeUnionType(NumberType, BoolType)), false},\n\n\t\t// map & map -> true\n\t\t{MakeMapType(BoolType, BoolType), MakeMapType(BoolType, BoolType), true},\n\t\t// map & map -> false\n\t\t{MakeMapType(BoolType, BoolType), MakeMapType(BoolType, StringType), false},\n\t\t// map & map -> false\n\t\t{MakeMapType(BoolType, BoolType), MakeMapType(StringType, BoolType), false},\n\t\t// map & map -> false\n\t\t{MakeMapType(BoolType, BoolType), MakeMapType(StringType, BoolType), false},\n\t\t// map & map -> false\n\t\t{MakeMapType(MakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType}), BoolType),\n\t\t\tMakeMapType(MakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType, \"bar\": StringType}), BoolType), false},\n\t\t// map & map -> true\n\t\t{MakeMapType(MakeUnionType(StringType, BlobType), StringType),\n\t\t\tMakeMapType(MakeUnionType(NumberType, StringType), StringType), true},\n\t\t// map & map -> false\n\t\t{MakeMapType(MakeUnionType(BlobType, BoolType), StringType),\n\t\t\tMakeMapType(MakeUnionType(NumberType, StringType), StringType), false},\n\n\t\t// bool & string|bool|blob -> true\n\t\t{BoolType, MakeUnionType(StringType, BoolType, BlobType), true},\n\t\t// string|bool|blob & blob -> true\n\t\t{MakeUnionType(StringType, BoolType, BlobType), BlobType, true},\n\t\t// string|bool|blob & number|blob|string -> true\n\t\t{MakeUnionType(StringType, BoolType, BlobType), MakeUnionType(NumberType, BlobType, StringType), true},\n\n\t\t// struct{foo:bool} & struct{foo:bool} -> true\n\t\t{MakeStructTypeFromFields(\"\", FieldMap{\"foo\": BoolType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": BoolType}), true},\n\t\t// struct{foo:bool} & struct{foo:number} -> false\n\t\t{MakeStructTypeFromFields(\"\", FieldMap{\"foo\": BoolType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType}), false},\n\t\t// struct{foo:bool} & struct{foo:bool,bar:number} -> true\n\t\t{MakeStructTypeFromFields(\"\", FieldMap{\"foo\": BoolType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": BoolType, \"bar\": NumberType}), true},\n\t\t// struct{foo:ref} & struct{foo:ref} -> false\n\t\t{MakeStructTypeFromFields(\"\", FieldMap{\"foo\": MakeRefType(BoolType)}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": MakeRefType(NumberType)}), false},\n\t\t// struct{foo:ref} & struct{foo:ref} -> true\n\t\t{MakeStructTypeFromFields(\"\", FieldMap{\"foo\": MakeRefType(BoolType)}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": MakeRefType(MakeUnionType(NumberType, BoolType))}), true},\n\t\t// struct A{foo:bool} & struct A{foo:bool, baz:string} -> true\n\t\t{MakeStructTypeFromFields(\"A\", FieldMap{\"foo\": BoolType}),\n\t\t\tMakeStructTypeFromFields(\"A\", FieldMap{\"foo\": BoolType, \"baz\": StringType}), true},\n\n\t\t// struct A{foo:bool, stuff:set} & struct A{foo:bool, stuff:set} -> true\n\t\t{MakeStructTypeFromFields(\"A\", FieldMap{\"foo\": BoolType, \"stuff\": MakeSetType(MakeUnionType(StringType, BlobType))}),\n\t\t\tMakeStructTypeFromFields(\"A\", FieldMap{\"foo\": BoolType, \"stuff\": MakeSetType(StringType)}), true},\n\t\t// struct A{stuff:set} & struct A{foo:bool, stuff:set} -> false\n\t\t{MakeStructTypeFromFields(\"A\", FieldMap{\"foo\": BoolType, \"stuff\": MakeSetType(MakeUnionType(StringType, BlobType))}),\n\t\t\tMakeStructTypeFromFields(\"A\", FieldMap{\"stuff\": MakeSetType(NumberType)}), false},\n\n\t\t// struct A{foo:bool} & struct {foo:bool} -> true\n\t\t{MakeStructTypeFromFields(\"A\", FieldMap{\"foo\": BoolType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": BoolType}), true},\n\t\t// struct {foo:bool} & struct A{foo:bool} -> false\n\t\t{MakeStructTypeFromFields(\"\", FieldMap{\"foo\": BoolType}),\n\t\t\tMakeStructTypeFromFields(\"A\", FieldMap{\"foo\": BoolType}), true},\n\t\t// struct A{foo:bool} & struct B{foo:bool} -> false\n\t\t{MakeStructTypeFromFields(\"A\", FieldMap{\"foo\": BoolType}),\n\t\t\tMakeStructTypeFromFields(\"B\", FieldMap{\"foo\": BoolType}), false},\n\t\t// map & map -> true\n\t\t{MakeMapType(StringType, MakeStructTypeFromFields(\"A\", FieldMap{\"foo\": StringType})),\n\t\t\tMakeMapType(StringType, MakeStructTypeFromFields(\"A\", FieldMap{\"foo\": StringType, \"bar\": BoolType})), true},\n\n\t\t// struct{foo: string} & struct{foo: string|blob} -> true\n\t\t{MakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": MakeUnionType(StringType, BlobType)}), true},\n\n\t\t// struct{foo: string}|struct{foo: blob} & struct{foo: string|blob} -> true\n\t\t{MakeUnionType(\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": BlobType}),\n\t\t), MakeStructTypeFromFields(\"\", FieldMap{\"foo\": MakeUnionType(StringType, BlobType)}), true},\n\t\t// struct{foo: string}|struct{foo: blob} & struct{foo: number|bool} -> false\n\t\t{MakeUnionType(\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": BlobType}),\n\t\t), MakeStructTypeFromFields(\"\", FieldMap{\"foo\": MakeUnionType(NumberType, BoolType)}), false},\n\n\t\t// map & map -> true\n\t\t{\n\t\t\tMakeMapType(\n\t\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"x\": NumberType, \"y\": NumberType}),\n\t\t\t\tMakeStructTypeFromFields(\"A\", FieldMap{\"foo\": StringType})),\n\t\t\tMakeMapType(\n\t\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"x\": NumberType, \"y\": NumberType}),\n\t\t\t\tMakeStructTypeFromFields(\"A\", FieldMap{\"foo\": StringType, \"bar\": BoolType})),\n\t\t\ttrue,\n\t\t},\n\n\t\t// map & map -> true\n\t\t{\n\t\t\tMakeMapType(\n\t\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"x\": NumberType, \"y\": NumberType}),\n\t\t\t\tMakeStructTypeFromFields(\"A\", FieldMap{\"foo\": StringType})),\n\t\t\tMakeMapType(\n\t\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"x\": NumberType, \"y\": NumberType}),\n\t\t\t\tMakeStructTypeFromFields(\"A\", FieldMap{\"foo\": StringType, \"bar\": BoolType})),\n\t\t\ttrue,\n\t\t},\n\n\t\t// struct A{self:A} & struct A{self:A, foo:Number} -> true\n\t\t{MakeStructTypeFromFields(\"A\", FieldMap{\"self\": MakeCycleType(\"A\")}),\n\t\t\tMakeStructTypeFromFields(\"A\", FieldMap{\"self\": MakeCycleType(\"A\"), \"foo\": NumberType}), true},\n\n\t\t// struct{b:Bool} & struct{b?:Bool} -> true\n\t\t{\n\t\t\tMakeStructType(\"\", StructField{\"b\", BoolType, false}),\n\t\t\tMakeStructType(\"\", StructField{\"b\", BoolType, true}),\n\t\t\ttrue,\n\t\t},\n\n\t\t// struct{a?:Bool} & struct{b?:Bool} -> false\n\t\t{\n\t\t\tMakeStructType(\"\", StructField{\"a\", BoolType, true}),\n\t\t\tMakeStructType(\"\", StructField{\"b\", BoolType, true}),\n\t\t\tfalse,\n\t\t},\n\n\t\t// struct A {b: struct {a: Cycle}} & struct {b: Struct A {b: struct {b: Cycle}}} -> false\n\t\t{\n\t\t\tMakeStructType(\"A\",\n\t\t\t\tStructField{\"a\", MakeStructType(\"\",\n\t\t\t\t\tStructField{\"a\", MakeCycleType(\"A\"), false},\n\t\t\t\t), false},\n\t\t\t),\n\t\t\tMakeStructType(\"\",\n\t\t\t\tStructField{\"a\", MakeStructType(\"A\",\n\t\t\t\t\tStructField{\"a\", MakeStructType(\"\",\n\t\t\t\t\t\tStructField{\"a\", MakeCycleType(\"A\"), false},\n\t\t\t\t\t), false},\n\t\t\t\t), false},\n\t\t\t),\n\t\t\ttrue,\n\t\t},\n\t}\n\n\tfor i, c := range cases {\n\t\tact := ContainCommonSupertype(c.a, c.b)\n\t\tassert.Equal(t, c.out, act, \"Test case at position %d; \\n\\ta:%s\\n\\tb:%s\", i, c.a.Describe(), c.b.Describe())\n\t}\n}\n" }, { "path": "go/types/compare_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nvar prefix = []byte{0x01, 0x02, 0x03, 0x04}\n\nfunc TestCompareTotalOrdering(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\t// values in increasing order. Some of these are compared by ref so changing the serialization might change the ordering.\n\tvalues := []Value{\n\t\tBool(false), Bool(true),\n\t\tNumber(-10), Number(0), Number(10),\n\t\tString(\"a\"), String(\"b\"), String(\"c\"),\n\n\t\t// The order of these are done by the hash.\n\t\tNewSet(vrw, Number(0), Number(1), Number(2), Number(3)),\n\t\tBoolType,\n\n\t\t// Value - values cannot be value\n\t\t// Cycle - values cannot be cycle\n\t\t// Union - values cannot be unions\n\t}\n\n\tfor i, vi := range values {\n\t\tfor j, vj := range values {\n\t\t\tif i == j {\n\t\t\t\tassert.True(vi.Equals(vj))\n\t\t\t} else if i < j {\n\t\t\t\tx := vi.Less(vj)\n\t\t\t\tassert.True(x)\n\t\t\t} else {\n\t\t\t\tx := vi.Less(vj)\n\t\t\t\tassert.False(x)\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc TestCompareEmpties(t *testing.T) {\n\tassert := assert.New(t)\n\tcomp := opCacheComparer{}\n\tassert.Equal(-1, comp.Compare(prefix, append(prefix, 0xff)))\n\tassert.Equal(0, comp.Compare(prefix, prefix))\n\tassert.Equal(1, comp.Compare(append(prefix, 0xff), prefix))\n}\n\nfunc TestCompareDifferentPrimitiveTypes(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\tdefer vrw.Close()\n\n\tnums := ValueSlice{Number(1), Number(2), Number(3)}\n\twords := ValueSlice{String(\"k1\"), String(\"v1\")}\n\n\tblob := NewBlob(vrw, bytes.NewBuffer([]byte{1, 2, 3}))\n\tnList := NewList(vrw, nums...)\n\tnMap := NewMap(vrw, words...)\n\tnRef := NewRef(blob)\n\tnSet := NewSet(vrw, nums...)\n\tnStruct := NewStruct(\"teststruct\", map[string]Value{\"f1\": Number(1)})\n\n\tvals := ValueSlice{Bool(true), Number(19), String(\"hellow\"), blob, nList, nMap, nRef, nSet, nStruct}\n\tsort.Sort(vals)\n\n\tfor i, v1 := range vals {\n\t\tfor j, v2 := range vals {\n\t\t\tiBytes := [1024]byte{}\n\t\t\tjBytes := [1024]byte{}\n\t\t\tres := compareEncodedKey(encodeGraphKey(iBytes[:0], v1), encodeGraphKey(jBytes[:0], v2))\n\t\t\tassert.Equal(compareInts(i, j), res)\n\t\t}\n\t}\n}\n\nfunc TestComparePrimitives(t *testing.T) {\n\tassert := assert.New(t)\n\n\tbools := []Bool{false, true}\n\tfor i, v1 := range bools {\n\t\tfor j, v2 := range bools {\n\t\t\tres := compareEncodedNomsValues(encode(v1), encode(v2))\n\t\t\tassert.Equal(compareInts(i, j), res)\n\t\t}\n\t}\n\n\tnums := []Number{-1111.29, -23, 0, 4.2345, 298}\n\tfor i, v1 := range nums {\n\t\tfor j, v2 := range nums {\n\t\t\tres := compareEncodedNomsValues(encode(v1), encode(v2))\n\t\t\tassert.Equal(compareInts(i, j), res)\n\t\t}\n\t}\n\n\twords := []String{\"\", \"aaa\", \"another\", \"another1\"}\n\tfor i, v1 := range words {\n\t\tfor j, v2 := range words {\n\t\t\tres := compareEncodedNomsValues(encode(v1), encode(v2))\n\t\t\tassert.Equal(compareInts(i, j), res)\n\t\t}\n\t}\n}\n\nfunc TestCompareEncodedKeys(t *testing.T) {\n\tassert := assert.New(t)\n\tcomp := opCacheComparer{}\n\tvrw := newTestValueStore()\n\tdefer vrw.Close()\n\n\tk1 := ValueSlice{String(\"one\"), Number(3)}\n\tk2 := ValueSlice{String(\"one\"), Number(5)}\n\n\tbs1 := [initialBufferSize]byte{}\n\tbs2 := [initialBufferSize]byte{}\n\n\te1, _ := encodeKeys(bs1[:0], 0x01020304, MapKind, k1)\n\te2, _ := encodeKeys(bs2[:0], 0x01020304, MapKind, k2)\n\tassert.Equal(-1, comp.Compare(e1, e2))\n}\n\nfunc encode(v Value) []byte {\n\tw := &binaryNomsWriter{make([]byte, 128, 128), 0}\n\tv.writeTo(w)\n\treturn w.data()\n}\n\nfunc compareInts(i, j int) (res int) {\n\tif i < j {\n\t\tres = -1\n\t} else if i > j {\n\t\tres = 1\n\t}\n\treturn\n}\n" }, { "path": "go/types/edit_distance.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\n// ported from edit-distance.js, itself a port with minor modifications of\n// https://github.com/Polymer/observe-js/blob/master/src/observe.js#L1309.\n\nimport (\n\t\"fmt\"\n\t\"math\"\n)\n\nconst (\n\tDEFAULT_MAX_SPLICE_MATRIX_SIZE = 2e7\n\tSPLICE_UNASSIGNED = math.MaxUint64\n\n\tUNCHANGED = 0\n\tUPDATED = 1\n\tINSERTED = 2\n\tREMOVED = 3\n)\n\n// Read a Splice as \"at SpAt (in the previous state), SpRemoved elements were removed and SpAdded\n// elements were inserted, which can be found starting at SpFrom in the current state\"\ntype Splice struct {\n\tSpAt uint64\n\tSpRemoved uint64\n\tSpAdded uint64\n\tSpFrom uint64\n}\n\ntype EditDistanceEqualsFn func(prevIndex uint64, currentIndex uint64) bool\n\nfunc (s Splice) String() string {\n\treturn fmt.Sprintf(\"[%d, %d, %d, %d]\", s.SpAt, s.SpRemoved, s.SpAdded, s.SpFrom)\n}\n\nfunc uint64Min(a, b uint64) uint64 {\n\tif a < b {\n\t\treturn a\n\t}\n\treturn b\n}\n\nfunc uint64Min3(a, b, c uint64) uint64 {\n\tif a < b {\n\t\tif a < c {\n\t\t\treturn a\n\t\t}\n\t} else {\n\t\tif b < c {\n\t\t\treturn b\n\t\t}\n\t}\n\treturn c\n}\n\nfunc reverse(numbers []uint64) []uint64 {\n\tnewNumbers := make([]uint64, len(numbers))\n\tfor i := 0; i < len(numbers); i++ {\n\t\tnewNumbers[i] = numbers[len(numbers)-i-1]\n\t}\n\treturn newNumbers\n}\n\nfunc addSplice(splices []Splice, s Splice) []Splice {\n\tif s.SpFrom == SPLICE_UNASSIGNED {\n\t\ts.SpFrom = 0\n\t}\n\tsplices = append(splices, s)\n\treturn splices\n}\n\nfunc calcSplices(previousLength uint64, currentLength uint64, maxSpliceMatrixSize uint64, eqFn EditDistanceEqualsFn) []Splice {\n\tminLength := uint64Min(previousLength, currentLength)\n\tprefixCount := sharedPrefix(eqFn, minLength)\n\tsuffixCount := sharedSuffix(eqFn, previousLength, currentLength, minLength-prefixCount)\n\n\tpreviousStart := prefixCount\n\tcurrentStart := prefixCount\n\tpreviousEnd := previousLength - suffixCount\n\tcurrentEnd := currentLength - suffixCount\n\n\tif (currentEnd-currentStart) == 0 && (previousEnd-previousStart) == 0 {\n\t\treturn []Splice{}\n\t}\n\n\tif currentStart == currentEnd {\n\t\treturn []Splice{{previousStart, previousEnd - previousStart, 0, 0}}\n\t} else if previousStart == previousEnd {\n\t\treturn []Splice{{previousStart, 0, currentEnd - currentStart, currentStart}}\n\t}\n\n\tpreviousLength = previousEnd - previousStart\n\tcurrentLength = currentEnd - currentStart\n\n\tif previousLength*currentLength > maxSpliceMatrixSize {\n\t\treturn []Splice{{0, previousLength, currentLength, 0}}\n\t}\n\n\tsplices := make([]Splice, 0)\n\tdistances := calcEditDistances(eqFn, previousStart, previousLength, currentStart, currentLength)\n\tops := operationsFromEditDistances(distances)\n\n\tvar splice *Splice\n\tindex := currentStart\n\tpreviousIndex := previousStart\n\tfor i := 0; i < len(ops); i++ {\n\t\tswitch ops[i] {\n\t\tcase UNCHANGED:\n\t\t\tif splice != nil {\n\t\t\t\tsplices = addSplice(splices, *splice)\n\t\t\t\tsplice = nil\n\t\t\t}\n\n\t\t\tindex++\n\t\t\tpreviousIndex++\n\t\t\tbreak\n\t\tcase UPDATED:\n\t\t\tif splice == nil {\n\t\t\t\tsplice = &Splice{index, 0, 0, SPLICE_UNASSIGNED}\n\t\t\t}\n\n\t\t\tif splice.SpFrom == SPLICE_UNASSIGNED {\n\t\t\t\tsplice.SpFrom = previousIndex\n\t\t\t}\n\n\t\t\tsplice.SpRemoved++\n\t\t\tsplice.SpAdded++\n\t\t\tindex++\n\t\t\tpreviousIndex++\n\t\t\tbreak\n\t\tcase INSERTED:\n\t\t\tif splice == nil {\n\t\t\t\tsplice = &Splice{index, 0, 0, SPLICE_UNASSIGNED}\n\t\t\t}\n\n\t\t\tsplice.SpAdded++\n\t\t\tif splice.SpFrom == SPLICE_UNASSIGNED {\n\t\t\t\tsplice.SpFrom = previousIndex\n\t\t\t}\n\n\t\t\tpreviousIndex++\n\t\t\tbreak\n\t\tcase REMOVED:\n\t\t\tif splice == nil {\n\t\t\t\tsplice = &Splice{index, 0, 0, SPLICE_UNASSIGNED}\n\t\t\t}\n\n\t\t\tsplice.SpRemoved++\n\t\t\tindex++\n\t\t\tbreak\n\t\t}\n\t}\n\n\tif splice != nil {\n\t\tsplices = addSplice(splices, *splice)\n\t}\n\n\treturn splices\n}\n\nfunc calcEditDistances(eqFn EditDistanceEqualsFn, previousStart uint64, previousLen uint64,\n\tcurrentStart uint64, currentLen uint64) [][]uint64 {\n\t// \"Deletion\" columns\n\trowCount := previousLen + 1\n\tcolumnCount := currentLen + 1\n\n\t// see https://golang.org/doc/effective_go.html#two_dimensional_slices for below allocation optimization\n\tdistances := make([][]uint64, rowCount)\n\tdistance := make([]uint64, rowCount*columnCount)\n\tfor i := range distances {\n\t\tdistances[i], distance = distance[:columnCount], distance[columnCount:]\n\t}\n\n\t// \"Addition\" rows. Initialize null column.\n\tfor i := uint64(0); i < rowCount; i++ {\n\t\tdistances[i][0] = i\n\t}\n\n\t// Initialize null row\n\tfor j := uint64(0); j < columnCount; j++ {\n\t\tdistances[0][j] = j\n\t}\n\n\tfor i := uint64(1); i < rowCount; i++ {\n\t\tfor j := uint64(1); j < columnCount; j++ {\n\t\t\tif eqFn(previousStart+i-1, currentStart+j-1) {\n\t\t\t\tdistances[i][j] = distances[i-1][j-1]\n\t\t\t} else {\n\t\t\t\tnorth := distances[i-1][j] + 1\n\t\t\t\twest := distances[i][j-1] + 1\n\t\t\t\tdistances[i][j] = uint64Min(north, west)\n\t\t\t}\n\t\t}\n\t}\n\n\treturn distances\n}\n\nfunc operationsFromEditDistances(distances [][]uint64) []uint64 {\n\ti := len(distances) - 1\n\tj := len(distances[0]) - 1\n\tcurrent := distances[i][j]\n\tedits := make([]uint64, 0)\n\tfor i > 0 || j > 0 {\n\t\tif i == 0 {\n\t\t\tedits = append(edits, INSERTED)\n\t\t\tj--\n\t\t\tcontinue\n\t\t}\n\t\tif j == 0 {\n\t\t\tedits = append(edits, REMOVED)\n\t\t\ti--\n\t\t\tcontinue\n\t\t}\n\t\tnorthWest := distances[i-1][j-1]\n\t\twest := distances[i-1][j]\n\t\tnorth := distances[i][j-1]\n\n\t\tminValue := uint64Min3(west, north, northWest)\n\n\t\tif minValue == northWest {\n\t\t\tif northWest == current {\n\t\t\t\tedits = append(edits, UNCHANGED)\n\t\t\t} else {\n\t\t\t\tedits = append(edits, UPDATED)\n\t\t\t\tcurrent = northWest\n\t\t\t}\n\t\t\ti--\n\t\t\tj--\n\t\t} else if minValue == west {\n\t\t\tedits = append(edits, REMOVED)\n\t\t\ti--\n\t\t\tcurrent = west\n\t\t} else {\n\t\t\tedits = append(edits, INSERTED)\n\t\t\tj--\n\t\t\tcurrent = north\n\t\t}\n\t}\n\n\treturn reverse(edits)\n}\n\nfunc sharedPrefix(eqFn EditDistanceEqualsFn, searchLength uint64) uint64 {\n\tfor i := uint64(0); i < searchLength; i++ {\n\t\tif !eqFn(i, i) {\n\t\t\treturn i\n\t\t}\n\t}\n\n\treturn searchLength\n}\n\nfunc sharedSuffix(eqFn EditDistanceEqualsFn, previousLength uint64, currentLength uint64, searchLength uint64) uint64 {\n\tcount := uint64(0)\n\tpreviousLength--\n\tcurrentLength--\n\tfor count < searchLength && eqFn(previousLength, currentLength) {\n\t\tcount++\n\t\tpreviousLength--\n\t\tcurrentLength--\n\t}\n\n\treturn count\n}\n" }, { "path": "go/types/edit_distance_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc assertDiff(assert *assert.Assertions, last []uint64, current []uint64, expect []Splice) {\n\tactual := calcSplices(uint64(len(last)), uint64(len(current)), DEFAULT_MAX_SPLICE_MATRIX_SIZE,\n\t\tfunc(i uint64, j uint64) bool { return last[i] == current[j] })\n\tassert.Equal(expect, actual, \"splices are different: \\nexpect: %v\\nactual: %v\\n\", expect, actual)\n}\n\nfunc TestEditDistanceAppend(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2},\n\t\t[]uint64{0, 1, 2, 3, 4, 5},\n\t\t[]Splice{{3, 0, 3, 3}},\n\t)\n}\n\nfunc TestEditDistancePrepend(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{3, 4, 5, 6},\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6},\n\t\t[]Splice{{0, 0, 3, 0}},\n\t)\n}\n\nfunc TestEditDistanceChopFromEnd(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5},\n\t\t[]uint64{0, 1, 2},\n\t\t[]Splice{{3, 3, 0, 0}},\n\t)\n}\n\nfunc TestEditDistanceChopFromStart(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5},\n\t\t[]uint64{3, 4, 5},\n\t\t[]Splice{{0, 3, 0, 0}},\n\t)\n}\n\nfunc TestEditDistanceChopFromMiddle(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5},\n\t\t[]uint64{0, 5},\n\t\t[]Splice{{1, 4, 0, 0}},\n\t)\n}\n\nfunc TestEditDistanceA(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7, 8},\n\t\t[]uint64{0, 1, 2, 4, 5, 6, 8},\n\t\t[]Splice{\n\t\t\t{3, 1, 0, 0},\n\t\t\t{7, 1, 0, 0},\n\t\t},\n\t)\n}\n\nfunc TestEditDistanceRemoveABunch(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10},\n\t\t[]uint64{1, 2, 4, 5, 7, 8, 10},\n\t\t[]Splice{\n\t\t\t{0, 1, 0, 0},\n\t\t\t{3, 1, 0, 0},\n\t\t\t{6, 1, 0, 0},\n\t\t\t{9, 1, 0, 0},\n\t\t},\n\t)\n}\n\nfunc TestEditDistanceAddABunch(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10},\n\t\t[]uint64{0, 'a', 1, 2, 3, 'b', 'c', 'd', 4, 5, 6, 7, 'e', 8, 9, 'f', 10, 'g'},\n\t\t[]Splice{\n\t\t\t{1, 0, 1, 1},\n\t\t\t{4, 0, 3, 5},\n\t\t\t{8, 0, 1, 12},\n\t\t\t{10, 0, 1, 15},\n\t\t\t{11, 0, 1, 17},\n\t\t},\n\t)\n}\n\nfunc TestEditDistanceUpdateABunch(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10},\n\t\t[]uint64{'a', 1, 2, 'b', 'c', 'd', 6, 7, 'e', 9, 10},\n\t\t[]Splice{\n\t\t\t{0, 1, 1, 0},\n\t\t\t{3, 3, 3, 3},\n\t\t\t{8, 1, 1, 8},\n\t\t},\n\t)\n}\n\nfunc TestEditDistanceLeftOverlap(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10},\n\t\t[]uint64{0, 1, 2, 3, 'a', 'b', 8, 9, 10},\n\t\t[]Splice{\n\t\t\t{4, 4, 2, 4},\n\t\t},\n\t)\n}\n\nfunc TestEditDistanceRightOverlap(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10},\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 'a', 'b', 10},\n\t\t[]Splice{\n\t\t\t{6, 4, 2, 6},\n\t\t},\n\t)\n}\n\nfunc TestEditDistanceWithin(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10},\n\t\t[]uint64{0, 1, 2, 3, 'a', 'b', 10},\n\t\t[]Splice{\n\t\t\t{4, 6, 2, 4},\n\t\t},\n\t)\n}\n\nfunc TestEditDistanceWithout(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10},\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 'a', 'b', 'c', 'd', 8, 9, 10},\n\t\t[]Splice{\n\t\t\t{6, 2, 4, 6},\n\t\t},\n\t)\n}\n\nfunc TestEditDistanceMix1(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10},\n\t\t[]uint64{0, 'a', 1, 'b', 3, 'c', 4, 6, 7, 'e', 'f', 10},\n\t\t[]Splice{\n\t\t\t{1, 0, 1, 1},\n\t\t\t{2, 1, 1, 3},\n\t\t\t{4, 0, 1, 5},\n\t\t\t{5, 1, 0, 0},\n\t\t\t{8, 2, 2, 9},\n\t\t},\n\t)\n}\n\nfunc TestEditDistanceReverse(t *testing.T) {\n\tt.Parallel()\n\tassert := assert.New(t)\n\tassertDiff(assert,\n\t\t[]uint64{0, 1, 2, 3, 4, 5, 6, 7},\n\t\t[]uint64{7, 6, 5, 4, 3, 2, 1, 0},\n\t\t[]Splice{\n\t\t\t{0, 3, 4, 0},\n\t\t\t{4, 4, 3, 5},\n\t\t},\n\t)\n}\n" }, { "path": "go/types/encode_human_readable.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"strconv\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/util/writers\"\n\thumanize \"github.com/dustin/go-humanize\"\n)\n\n// Clients can register a 'commenter' to return a comment that will get appended\n// to the first line of encoded values. For example, the noms DateTime struct\n// normally gets encoded as follows:\n// lastRefresh: DateTime {\n// secSinceEpoch: 1.501801626877e+09,\n// }\n//\n// By registering a commenter that returns a nicely formatted date,\n// the struct will be coded with a comment:\n// lastRefresh: DateTime { // 2017-08-03T16:07:06-07:00\n// secSinceEpoch: 1.501801626877e+09,\n// }\n\n// Function type for commenter functions\ntype HRSCommenter interface {\n\tComment(Value) string\n}\n\nvar (\n\tcommenterRegistry = map[string]map[string]HRSCommenter{}\n\tregistryLock sync.RWMutex\n)\n\n// RegisterHRSCommenter is called to with three arguments:\n// typename: the name of the struct this function will be applied to\n// unique: an arbitrary string to differentiate functions that should be applied\n// to different structs that have the same name (e.g. two implementations of\n// the \"Employee\" type.\n// commenter: an interface with a 'Comment()' function that gets called for all\n// Values with this name. The function should verify the type of the Value\n// and, if appropriate, return a non-empty string to be appended as the comment\nfunc RegisterHRSCommenter(typename, unique string, commenter HRSCommenter) {\n\tregistryLock.Lock()\n\tdefer registryLock.Unlock()\n\tcommenters := commenterRegistry[typename]\n\tif commenters == nil {\n\t\tcommenters = map[string]HRSCommenter{}\n\t\tcommenterRegistry[typename] = commenters\n\t}\n\tcommenters[unique] = commenter\n}\n\n// UnregisterHRSCommenter will remove a commenter function for a specified\n// typename/unique string combination.\nfunc UnregisterHRSCommenter(typename, unique string) {\n\tregistryLock.Lock()\n\tdefer registryLock.Unlock()\n\tr := commenterRegistry[typename]\n\tif r == nil {\n\t\treturn\n\t}\n\tdelete(r, unique)\n}\n\n// GetHRSCommenters the map of 'unique' strings to HRSCommentFunc for\n// a specified typename.\nfunc GetHRSCommenters(typename string) []HRSCommenter {\n\tregistryLock.RLock()\n\tdefer registryLock.RUnlock()\n\t// need to copy this value so we can release the lock\n\tcommenters := []HRSCommenter{}\n\tfor _, f := range commenterRegistry[typename] {\n\t\tcommenters = append(commenters, f)\n\t}\n\treturn commenters\n}\n\n// Human Readable Serialization\ntype hrsWriter struct {\n\tind int\n\tw io.Writer\n\tlineLength int\n\tfloatFormat byte\n\terr error\n}\n\nfunc (w *hrsWriter) maybeWriteIndentation() {\n\tif w.lineLength == 0 {\n\t\tfor i := 0; i < w.ind && w.err == nil; i++ {\n\t\t\t_, w.err = io.WriteString(w.w, \" \")\n\t\t}\n\t\tw.lineLength = 2 * w.ind\n\t}\n}\n\nfunc (w *hrsWriter) write(s string) {\n\tif w.err != nil {\n\t\treturn\n\t}\n\tw.maybeWriteIndentation()\n\tvar n int\n\tn, w.err = io.WriteString(w.w, s)\n\tw.lineLength += n\n}\n\nfunc (w *hrsWriter) indent() {\n\tw.ind++\n}\n\nfunc (w *hrsWriter) outdent() {\n\tw.ind--\n}\n\nfunc (w *hrsWriter) newLine() {\n\tw.write(\"\\n\")\n\tw.lineLength = 0\n}\n\n// hexWriter is used to write blob byte data as \"00 01 ... 0f\\n10 11 ..\"\n// hexWriter is an io.Writer that writes to an underlying hrsWriter.\ntype hexWriter struct {\n\thrs *hrsWriter\n\tcount uint\n\tsizeWritten bool\n\tsize uint64\n}\n\nfunc (w *hexWriter) Write(p []byte) (n int, err error) {\n\tfor _, v := range p {\n\t\tif !w.sizeWritten && len(p) > 16 {\n\t\t\tw.hrs.write(\" // \")\n\t\t\tw.hrs.write(humanize.Bytes(w.size))\n\t\t\tw.sizeWritten = true\n\t\t\tw.hrs.indent()\n\t\t\tw.hrs.newLine()\n\t\t}\n\n\t\tif w.count == 16 {\n\t\t\tw.hrs.newLine()\n\t\t\tw.count = 0\n\t\t} else if w.count != 0 {\n\t\t\tw.hrs.write(\" \")\n\t\t}\n\t\tif v < 0x10 {\n\t\t\tw.hrs.write(\"0\")\n\t\t}\n\t\tw.hrs.write(strconv.FormatUint(uint64(v), 16))\n\t\tif w.hrs.err != nil {\n\t\t\terr = w.hrs.err\n\t\t\treturn\n\t\t}\n\t\tn++\n\t\tw.count++\n\t}\n\n\tif w.sizeWritten {\n\t\tw.hrs.outdent()\n\t\tw.hrs.newLine()\n\t}\n\n\treturn\n}\n\nfunc (w *hrsWriter) Write(v Value) {\n\tswitch v.Kind() {\n\tcase BoolKind:\n\t\tw.write(strconv.FormatBool(bool(v.(Bool))))\n\tcase NumberKind:\n\t\tw.write(strconv.FormatFloat(float64(v.(Number)), w.floatFormat, -1, 64))\n\n\tcase StringKind:\n\t\tw.write(strconv.Quote(string(v.(String))))\n\n\tcase BlobKind:\n\t\tw.write(\"blob {\")\n\t\tblob := v.(Blob)\n\t\tencoder := &hexWriter{hrs: w, size: blob.Len()}\n\t\t_, w.err = io.Copy(encoder, blob.Reader())\n\t\tw.write(\"}\")\n\n\tcase ListKind:\n\t\tw.write(\"[\")\n\t\tw.writeSize(v)\n\t\tw.indent()\n\t\tv.(List).Iter(func(v Value, i uint64) bool {\n\t\t\tif i == 0 {\n\t\t\t\tw.newLine()\n\t\t\t}\n\t\t\tw.Write(v)\n\t\t\tw.write(\",\")\n\t\t\tw.newLine()\n\t\t\treturn w.err != nil\n\t\t})\n\t\tw.outdent()\n\t\tw.write(\"]\")\n\n\tcase MapKind:\n\t\tw.write(\"map {\")\n\t\tw.writeSize(v)\n\t\tw.indent()\n\t\tif !v.(Map).Empty() {\n\t\t\tw.newLine()\n\t\t}\n\t\tv.(Map).Iter(func(key, val Value) bool {\n\t\t\tw.Write(key)\n\t\t\tw.write(\": \")\n\t\t\tw.Write(val)\n\t\t\tw.write(\",\")\n\t\t\tw.newLine()\n\t\t\treturn w.err != nil\n\t\t})\n\t\tw.outdent()\n\t\tw.write(\"}\")\n\n\tcase RefKind:\n\t\tw.write(\"#\")\n\t\tw.write(v.(Ref).TargetHash().String())\n\n\tcase SetKind:\n\t\tw.write(\"set {\")\n\t\tw.writeSize(v)\n\t\tw.indent()\n\t\tif !v.(Set).Empty() {\n\t\t\tw.newLine()\n\t\t}\n\t\tv.(Set).Iter(func(v Value) bool {\n\t\t\tw.Write(v)\n\t\t\tw.write(\",\")\n\t\t\tw.newLine()\n\t\t\treturn w.err != nil\n\t\t})\n\t\tw.outdent()\n\t\tw.write(\"}\")\n\n\tcase TypeKind:\n\t\tw.writeType(v.(*Type), map[*Type]struct{}{})\n\n\tcase StructKind:\n\t\tw.writeStruct(v.(Struct))\n\n\tdefault:\n\t\tpanic(\"unreachable\")\n\t}\n}\n\ntype hrsStructWriter struct {\n\t*hrsWriter\n\tv Struct\n}\n\nfunc (w hrsStructWriter) name(n string) {\n\tw.write(\"struct \")\n\tif n != \"\" {\n\t\tw.write(n)\n\t\tw.write(\" \")\n\t}\n\tw.write(\"{\")\n\tcommenters := GetHRSCommenters(n)\n\tfor _, commenter := range commenters {\n\t\tif comment := commenter.Comment(w.v); comment != \"\" {\n\t\t\tw.write(\" // \" + comment)\n\t\t\tbreak\n\t\t}\n\n\t}\n\tw.indent()\n}\n\nfunc (w hrsStructWriter) count(c uint64) {\n\tif c > 0 {\n\t\tw.newLine()\n\t}\n}\n\nfunc (w hrsStructWriter) fieldName(n string) {\n\tw.write(n)\n\tw.write(\": \")\n}\n\nfunc (w hrsStructWriter) fieldValue(v Value) {\n\tw.Write(v)\n\tw.write(\",\")\n\tw.newLine()\n}\n\nfunc (w hrsStructWriter) end() {\n\tw.outdent()\n\tw.write(\"}\")\n}\n\nfunc (w *hrsWriter) writeStruct(v Struct) {\n\tv.iterParts(hrsStructWriter{w, v})\n}\n\nfunc (w *hrsWriter) writeSize(v Value) {\n\tswitch v.Kind() {\n\tcase ListKind, MapKind, SetKind:\n\t\tl := v.(Collection).Len()\n\t\tif l < 4 {\n\t\t\treturn\n\t\t}\n\t\tw.write(fmt.Sprintf(\" // %s items\", humanize.Comma(int64(l))))\n\tdefault:\n\t\tpanic(\"unreachable\")\n\t}\n}\n\nfunc (w *hrsWriter) writeType(t *Type, seenStructs map[*Type]struct{}) {\n\tswitch t.TargetKind() {\n\tcase BlobKind, BoolKind, NumberKind, StringKind, TypeKind, ValueKind:\n\t\tw.write(t.TargetKind().String())\n\tcase ListKind, RefKind, SetKind, MapKind:\n\t\tw.write(t.TargetKind().String())\n\t\tw.write(\"<\")\n\t\tfor i, et := range t.Desc.(CompoundDesc).ElemTypes {\n\t\t\tif et.TargetKind() == UnionKind && len(et.Desc.(CompoundDesc).ElemTypes) == 0 {\n\t\t\t\t// If one of the element types is an empty union all the other element types must\n\t\t\t\t// also be empty union types.\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tif i != 0 {\n\t\t\t\tw.write(\", \")\n\t\t\t}\n\t\t\tw.writeType(et, seenStructs)\n\t\t\tif w.err != nil {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t\tw.write(\">\")\n\tcase UnionKind:\n\t\tfor i, et := range t.Desc.(CompoundDesc).ElemTypes {\n\t\t\tif i != 0 {\n\t\t\t\tw.write(\" | \")\n\t\t\t}\n\t\t\tw.writeType(et, seenStructs)\n\t\t\tif w.err != nil {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\tcase StructKind:\n\t\tw.writeStructType(t, seenStructs)\n\tcase CycleKind:\n\t\tname := string(t.Desc.(CycleDesc))\n\t\td.PanicIfTrue(name == \"\")\n\n\t\t// This can happen for types that have unresolved cyclic refs\n\t\tw.write(fmt.Sprintf(\"UnresolvedCycle<%s>\", name))\n\t\tif w.err != nil {\n\t\t\treturn\n\t\t}\n\tdefault:\n\t\tpanic(\"unreachable\")\n\t}\n}\n\nfunc (w *hrsWriter) writeStructType(t *Type, seenStructs map[*Type]struct{}) {\n\tname := t.Desc.(StructDesc).Name\n\tif _, ok := seenStructs[t]; ok {\n\t\tw.write(fmt.Sprintf(\"Cycle<%s>\", name))\n\t\treturn\n\t}\n\tseenStructs[t] = struct{}{}\n\n\tdesc := t.Desc.(StructDesc)\n\tw.write(\"Struct \")\n\tif desc.Name != \"\" {\n\t\tw.write(desc.Name + \" \")\n\t}\n\tw.write(\"{\")\n\tw.indent()\n\tif desc.Len() > 0 {\n\t\tw.newLine()\n\t}\n\tdesc.IterFields(func(name string, t *Type, optional bool) {\n\t\tw.write(name)\n\t\tif optional {\n\t\t\tw.write(\"?\")\n\t\t}\n\t\tw.write(\": \")\n\t\tw.writeType(t, seenStructs)\n\t\tw.write(\",\")\n\t\tw.newLine()\n\t})\n\tw.outdent()\n\tw.write(\"}\")\n}\n\nfunc encodedValueFormatMaxLines(v Value, floatFormat byte, maxLines uint32) string {\n\tvar buf bytes.Buffer\n\tmlw := &writers.MaxLineWriter{Dest: &buf, MaxLines: maxLines}\n\tw := &hrsWriter{w: mlw, floatFormat: floatFormat}\n\tw.Write(v)\n\tif w.err != nil {\n\t\td.Chk.IsType(writers.MaxLinesError{}, w.err, \"Unexpected error: %s\", w.err)\n\t}\n\treturn buf.String()\n}\n\nfunc encodedValueFormat(v Value, floatFormat byte) string {\n\tvar buf bytes.Buffer\n\tw := &hrsWriter{w: &buf, floatFormat: floatFormat}\n\tw.Write(v)\n\td.Chk.NoError(w.err)\n\treturn buf.String()\n}\n\nfunc EncodedIndexValue(v Value) string {\n\treturn encodedValueFormat(v, 'f')\n}\n\n// EncodedValue returns a string containing the serialization of a value.\nfunc EncodedValue(v Value) string {\n\treturn encodedValueFormat(v, 'g')\n}\n\n// EncodedValueMaxLines returns a string containing the serialization of a value.\n// The string is truncated at |maxLines|.\nfunc EncodedValueMaxLines(v Value, maxLines uint32) string {\n\treturn encodedValueFormatMaxLines(v, 'g', maxLines)\n}\n\n// WriteEncodedValue writes the serialization of a value\nfunc WriteEncodedValue(w io.Writer, v Value) error {\n\thrs := &hrsWriter{w: w, floatFormat: 'g'}\n\thrs.Write(v)\n\treturn hrs.err\n}\n\n// WriteEncodedValueMaxLines writes the serialization of a value. Writing will be\n// stopped and an error returned after |maxLines|.\nfunc WriteEncodedValueMaxLines(w io.Writer, v Value, maxLines uint32) error {\n\tmlw := &writers.MaxLineWriter{Dest: w, MaxLines: maxLines}\n\thrs := &hrsWriter{w: mlw, floatFormat: 'g'}\n\thrs.Write(v)\n\treturn hrs.err\n}\n" }, { "path": "go/types/encode_human_readable_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/util/test\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc assertWriteHRSEqual(t *testing.T, expected string, v Value) {\n\tassert := assert.New(t)\n\tvar buf bytes.Buffer\n\tw := &hrsWriter{w: &buf, floatFormat: 'g'}\n\tw.Write(v)\n\tassert.Equal(test.RemoveHashes(expected), test.RemoveHashes(buf.String()))\n}\n\nfunc TestWriteHumanReadablePrimitiveValues(t *testing.T) {\n\tassertWriteHRSEqual(t, \"true\", Bool(true))\n\tassertWriteHRSEqual(t, \"false\", Bool(false))\n\n\tassertWriteHRSEqual(t, \"0\", Number(0))\n\tassertWriteHRSEqual(t, \"42\", Number(42))\n\n\tassertWriteHRSEqual(t, \"-42\", Number(-42))\n\n\tassertWriteHRSEqual(t, \"3.1415926535\", Number(3.1415926535))\n\tassertWriteHRSEqual(t, \"314159.26535\", Number(3.1415926535e5))\n\tassertWriteHRSEqual(t, \"3.1415926535e+20\", Number(3.1415926535e20))\n\n\tassertWriteHRSEqual(t, `\"abc\"`, String(\"abc\"))\n\tassertWriteHRSEqual(t, `\" \"`, String(\" \"))\n\tassertWriteHRSEqual(t, `\"\\t\"`, String(\"\\t\"))\n\tassertWriteHRSEqual(t, `\"\\t\"`, String(\"\t\"))\n\tassertWriteHRSEqual(t, `\"\\n\"`, String(\"\\n\"))\n\tassertWriteHRSEqual(t, `\"\\n\"`, String(`\n`))\n\tassertWriteHRSEqual(t, `\"\\r\"`, String(\"\\r\"))\n\tassertWriteHRSEqual(t, `\"\\r\\n\"`, String(\"\\r\\n\"))\n\tassertWriteHRSEqual(t, `\"\\xff\"`, String(\"\\xff\"))\n\tassertWriteHRSEqual(t, `\"💩\"`, String(\"\\xf0\\x9f\\x92\\xa9\"))\n\tassertWriteHRSEqual(t, `\"💩\"`, String(\"💩\"))\n\tassertWriteHRSEqual(t, `\"\\a\"`, String(\"\\007\"))\n\tassertWriteHRSEqual(t, `\"☺\"`, String(\"\\u263a\"))\n}\n\nfunc TestWriteHumanReadableRef(t *testing.T) {\n\tvs := newTestValueStore()\n\n\tx := Number(42)\n\trv := vs.WriteValue(x)\n\tassertWriteHRSEqual(t, \"#0123456789abcdefghijklmnopqrstuv\", rv)\n}\n\nfunc TestWriteHumanReadableCollections(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tl := NewList(vrw, Number(0), Number(1), Number(2), Number(3))\n\tassertWriteHRSEqual(t, \"[ // 4 items\\n 0,\\n 1,\\n 2,\\n 3,\\n]\", l)\n\n\ts := NewSet(vrw, Number(0), Number(1), Number(2), Number(3))\n\tassertWriteHRSEqual(t, \"set { // 4 items\\n 0,\\n 1,\\n 2,\\n 3,\\n}\", s)\n\n\tm := NewMap(vrw, Number(0), Bool(false), Number(1), Bool(true))\n\tassertWriteHRSEqual(t, \"map {\\n 0: false,\\n 1: true,\\n}\", m)\n\n\tl2 := NewList(vrw)\n\tassertWriteHRSEqual(t, \"[]\", l2)\n\n\tl3 := NewList(vrw, Number(0))\n\tassertWriteHRSEqual(t, \"[\\n 0,\\n]\", l3)\n\n\tnums := make([]Value, 2000)\n\tfor i := range nums {\n\t\tnums[i] = Number(0)\n\t}\n\tl4 := NewList(vrw, nums...)\n\tassertWriteHRSEqual(t, \"[ // 2,000 items\\n\"+strings.Repeat(\" 0,\\n\", 2000)+\"]\", l4)\n}\n\nfunc TestWriteHumanReadableNested(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tl := NewList(vrw, Number(0), Number(1))\n\tl2 := NewList(vrw, Number(2), Number(3))\n\n\ts := NewSet(vrw, String(\"a\"), String(\"b\"))\n\ts2 := NewSet(vrw, String(\"c\"), String(\"d\"))\n\n\tm := NewMap(vrw, s, l, s2, l2)\n\tassertWriteHRSEqual(t, `map {\n set {\n \"c\",\n \"d\",\n }: [\n 2,\n 3,\n ],\n set {\n \"a\",\n \"b\",\n }: [\n 0,\n 1,\n ],\n}`, m)\n}\n\nfunc TestWriteHumanReadableStruct(t *testing.T) {\n\tstr := NewStruct(\"S1\", StructData{\n\t\t\"x\": Number(1),\n\t\t\"y\": Number(2),\n\t})\n\tassertWriteHRSEqual(t, \"struct S1 {\\n x: 1,\\n y: 2,\\n}\", str)\n}\n\nfunc TestWriteHumanReadableListOfStruct(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tstr1 := NewStruct(\"S3\", StructData{\n\t\t\"x\": Number(1),\n\t})\n\tstr2 := NewStruct(\"S3\", StructData{\n\t\t\"x\": Number(2),\n\t})\n\tstr3 := NewStruct(\"S3\", StructData{\n\t\t\"x\": Number(3),\n\t})\n\tl := NewList(vrw, str1, str2, str3)\n\tassertWriteHRSEqual(t, `[\n struct S3 {\n x: 1,\n },\n struct S3 {\n x: 2,\n },\n struct S3 {\n x: 3,\n },\n]`, l)\n}\n\nfunc TestWriteHumanReadableBlob(t *testing.T) {\n\tvrw := newTestValueStore()\n\tassertWriteHRSEqual(t, \"blob {}\", NewEmptyBlob(vrw))\n\n\tb1 := NewBlob(vrw, bytes.NewBuffer([]byte{0x01}))\n\tassertWriteHRSEqual(t, \"blob {01}\", b1)\n\n\tb2 := NewBlob(vrw, bytes.NewBuffer([]byte{0x01, 0x02}))\n\tassertWriteHRSEqual(t, \"blob {01 02}\", b2)\n\n\tb3 := NewBlob(vrw, bytes.NewBuffer([]byte{\n\t\t0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,\n\t\t0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,\n\t}))\n\tassertWriteHRSEqual(t, \"blob {00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f}\", b3)\n\n\tb4 := NewBlob(vrw, bytes.NewBuffer([]byte{\n\t\t0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,\n\t\t0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,\n\t\t0x10,\n\t}))\n\tassertWriteHRSEqual(t, \"blob { // 17 B\\n 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f\\n 10\\n}\", b4)\n\n\tbs := make([]byte, 256)\n\tfor i := range bs {\n\t\tbs[i] = byte(i)\n\t}\n\n\tb5 := NewBlob(vrw, bytes.NewBuffer(bs))\n\tassertWriteHRSEqual(t, \"blob { // 256 B\\n 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f\\n 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f\\n 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f\\n 30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f\\n 40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f\\n 50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f\\n 60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f\\n 70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f\\n 80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f\\n 90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f\\n a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af\\n b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf\\n c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf\\n d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df\\n e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef\\n f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff\\n}\", b5)\n\n\tb6 := NewBlob(vrw, bytes.NewBuffer(make([]byte, 16*100)))\n\trow := \" \" + strings.Repeat(\"00 \", 15) + \"00\\n\"\n\ts := strings.Repeat(row, 100)\n\tassertWriteHRSEqual(t, \"blob { // 1.6 kB\\n\"+s+\"}\", b6)\n}\n\nfunc TestWriteHumanReadableListOfBlob(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tb1 := NewBlob(vrw, bytes.NewBuffer([]byte{0x01}))\n\tb2 := NewBlob(vrw, bytes.NewBuffer([]byte{0x02}))\n\tb3 := NewBlob(vrw, bytes.NewBuffer([]byte{\n\t\t0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07,\n\t\t0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f,\n\t\t0x10,\n\t}))\n\tl := NewList(vrw, b1, NewEmptyBlob(vrw), b2, b3)\n\tassertWriteHRSEqual(t, \"[ // 4 items\\n blob {01},\\n blob {},\\n blob {02},\\n blob { // 17 B\\n 00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f\\n 10\\n },\\n]\", l)\n}\n\nfunc TestWriteHumanReadableType(t *testing.T) {\n\tassertWriteHRSEqual(t, \"Bool\", BoolType)\n\tassertWriteHRSEqual(t, \"Blob\", BlobType)\n\tassertWriteHRSEqual(t, \"String\", StringType)\n\tassertWriteHRSEqual(t, \"Number\", NumberType)\n\n\tassertWriteHRSEqual(t, \"List\", MakeListType(NumberType))\n\tassertWriteHRSEqual(t, \"Set\", MakeSetType(NumberType))\n\tassertWriteHRSEqual(t, \"Ref\", MakeRefType(NumberType))\n\tassertWriteHRSEqual(t, \"Map\", MakeMapType(NumberType, StringType))\n\tassertWriteHRSEqual(t, \"Number | String\", MakeUnionType(NumberType, StringType))\n\tassertWriteHRSEqual(t, \"Bool\", MakeUnionType(BoolType))\n\tassertWriteHRSEqual(t, \"\", MakeUnionType())\n\tassertWriteHRSEqual(t, \"List\", MakeListType(MakeUnionType(NumberType, StringType)))\n\tassertWriteHRSEqual(t, \"List<>\", MakeListType(MakeUnionType()))\n}\n\nfunc TestRecursiveStruct(t *testing.T) {\n\t// struct A {\n\t// b: A\n\t// c: List\n\t// d: struct D {\n\t// e: D\n\t// f: A\n\t// }\n\t// }\n\n\ta := MakeStructType(\"A\",\n\t\tStructField{\"b\", MakeCycleType(\"A\"), false},\n\t\tStructField{\"c\", MakeListType(MakeCycleType(\"A\")), false},\n\t\tStructField{\"d\", MakeStructType(\"D\",\n\t\t\tStructField{\"e\", MakeCycleType(\"D\"), false},\n\t\t\tStructField{\"f\", MakeCycleType(\"A\"), false},\n\t\t), false},\n\t)\n\n\tassertWriteHRSEqual(t, `Struct A {\n b: Cycle,\n c: List>,\n d: Struct D {\n e: Cycle,\n f: Cycle,\n },\n}`, a)\n\n\td, _ := a.Desc.(StructDesc).Field(\"d\")\n\n\tassertWriteHRSEqual(t, `Struct D {\n e: Cycle,\n f: Struct A {\n b: Cycle,\n c: List>,\n d: Cycle,\n },\n}`, d)\n}\n\nfunc TestUnresolvedRecursiveStruct(t *testing.T) {\n\t// struct A {\n\t// a: A\n\t// b: Cycle<1> (unresolved)\n\t// }\n\ta := MakeStructType(\"A\",\n\t\tStructField{\"a\", MakeCycleType(\"A\"), false},\n\t\tStructField{\"b\", MakeCycleType(\"X\"), false},\n\t)\n\n\tassertWriteHRSEqual(t, `Struct A {\n a: Cycle,\n b: UnresolvedCycle,\n}`, a)\n}\n\ntype errorWriter struct {\n\terr error\n}\n\nfunc (w *errorWriter) Write(p []byte) (int, error) {\n\treturn 0, w.err\n}\n\nfunc TestWriteHumanReadableWriterError(t *testing.T) {\n\tassert := assert.New(t)\n\terr := errors.New(\"test\")\n\tw := &errorWriter{err}\n\tassert.Equal(err, WriteEncodedValue(w, Number(42)))\n}\n\nfunc TestEmptyCollections(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\ta := MakeStructType(\"Nothing\")\n\tassertWriteHRSEqual(t, \"Struct Nothing {}\", a)\n\tb := NewStruct(\"Rien\", StructData{})\n\tassertWriteHRSEqual(t, \"struct Rien {}\", b)\n\tc := MakeMapType(BlobType, NumberType)\n\tassertWriteHRSEqual(t, \"Map\", c)\n\td := NewMap(vrw)\n\tassertWriteHRSEqual(t, \"map {}\", d)\n\te := MakeSetType(StringType)\n\tassertWriteHRSEqual(t, \"Set\", e)\n\tf := NewSet(vrw)\n\tassertWriteHRSEqual(t, \"set {}\", f)\n}\n\nfunc TestEncodedValueMaxLines(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tl1 := NewList(vrw, generateNumbersAsValues(11)...)\n\texpected := strings.Join(strings.SplitAfterN(EncodedValue(l1), \"\\n\", 6)[:5], \"\")\n\tassert.Equal(expected, EncodedValueMaxLines(l1, 5))\n\n\tbuf := bytes.Buffer{}\n\tWriteEncodedValueMaxLines(&buf, l1, 5)\n\tassert.Equal(expected, buf.String())\n}\n\nfunc TestWriteHumanReadableStructOptionalFields(t *testing.T) {\n\ttyp := MakeStructType(\"S1\",\n\t\tStructField{\"a\", BoolType, false},\n\t\tStructField{\"b\", BoolType, true})\n\tassertWriteHRSEqual(t, \"Struct S1 {\\n a: Bool,\\n b?: Bool,\\n}\", typ)\n}\n\ntype TestCommenter struct {\n\tprefix string\n\ttestType *Type\n}\n\nfunc (c TestCommenter) Comment(v Value) string {\n\tif !(v.typeOf().Equals(c.testType)) {\n\t\treturn \"\"\n\t}\n\treturn c.prefix + string(v.(Struct).Get(\"Name\").(String))\n}\n\nfunc TestRegisterCommenter(t *testing.T) {\n\ta := assert.New(t)\n\n\ttt := NewStruct(\"TestType1\", StructData{\"Name\": String(\"abc-123\")})\n\tnt := NewStruct(\"TestType2\", StructData{\"Name\": String(\"abc-123\")})\n\n\tRegisterHRSCommenter(\"TestType1\", \"mylib1\", TestCommenter{prefix: \"MyTest: \", testType: tt.typeOf()})\n\n\ts1 := EncodedValue(tt)\n\ta.True(strings.Contains(s1, \"// MyTest: abc-123\"))\n\ts1 = EncodedValue(nt)\n\ta.False(strings.Contains(s1, \"// MyTest: abc-123\"))\n\n\tRegisterHRSCommenter(\"TestType1\", \"mylib1\", TestCommenter{prefix: \"MyTest2: \", testType: tt.typeOf()})\n\ts1 = EncodedValue(tt)\n\ta.True(strings.Contains(s1, \"// MyTest2: abc-123\"))\n\n\tUnregisterHRSCommenter(\"TestType1\", \"mylib1\")\n\ts1 = EncodedValue(tt)\n\ta.False(strings.Contains(s1, \"// MyTest2: abc-123\"))\n}\n" }, { "path": "go/types/encoding_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"math\"\n\t\"strconv\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc toBinaryNomsReaderData(data []interface{}) []byte {\n\tw := newBinaryNomsWriter()\n\tfor _, v := range data {\n\t\tswitch v := v.(type) {\n\t\tcase uint8:\n\t\t\tw.writeUint8(v)\n\t\tcase string:\n\t\t\tw.writeString(v)\n\t\tcase Number:\n\t\t\tw.writeNumber(v)\n\t\tcase uint64:\n\t\t\tw.writeCount(v)\n\t\tcase bool:\n\t\t\tw.writeBool(v)\n\t\tcase hash.Hash:\n\t\t\tw.writeHash(v)\n\t\tcase []byte:\n\t\t\tw.writeCount(uint64(len(v)))\n\t\t\tw.writeBytes(v)\n\t\tcase NomsKind:\n\t\t\tw.writeUint8(uint8(v))\n\t\tdefault:\n\t\t\tpanic(\"unreachable\")\n\t\t}\n\t}\n\treturn w.data()\n}\n\nfunc assertEncoding(t *testing.T, expect []interface{}, v Value) {\n\texpectedAsByteSlice := toBinaryNomsReaderData(expect)\n\tvs := newTestValueStore()\n\tw := newBinaryNomsWriter()\n\tv.writeTo(&w)\n\tassert.EqualValues(t, expectedAsByteSlice, w.data())\n\n\tdec := newValueDecoder(expectedAsByteSlice, vs)\n\tv2 := dec.readValue()\n\tassert.True(t, v.Equals(v2))\n}\n\nfunc TestRoundTrips(t *testing.T) {\n\tvs := newTestValueStore()\n\n\tassertRoundTrips := func(v Value) {\n\t\tout := DecodeValue(EncodeValue(v), vs)\n\t\tassert.True(t, v.Equals(out))\n\t}\n\n\tassertRoundTrips(Bool(false))\n\tassertRoundTrips(Bool(true))\n\n\tassertRoundTrips(Number(0))\n\tassertRoundTrips(Number(-0))\n\tassertRoundTrips(Number(math.Copysign(0, -1)))\n\n\tintTest := []int64{1, 2, 3, 7, 15, 16, 17,\n\t\t127, 128, 129,\n\t\t254, 255, 256, 257,\n\t\t1023, 1024, 1025,\n\t\t2048, 4096, 8192, 32767, 32768, 65535, 65536,\n\t\t4294967295, 4294967296,\n\t\t9223372036854779,\n\t\t92233720368547760,\n\t}\n\tfor _, v := range intTest {\n\t\tf := float64(v)\n\t\tassertRoundTrips(Number(f))\n\t\tf = math.Copysign(f, -1)\n\t\tassertRoundTrips(Number(f))\n\t}\n\tfloatTest := []float64{1.01, 1.001, 1.0001, 1.00001, 1.000001, 100.01, 1000.000001, 122.411912027329, 0.42}\n\tfor _, f := range floatTest {\n\t\tassertRoundTrips(Number(f))\n\t\tf = math.Copysign(f, -1)\n\t\tassertRoundTrips(Number(f))\n\t}\n\n\t// JS Number.MAX_SAFE_INTEGER\n\tassertRoundTrips(Number(9007199254740991))\n\t// JS Number.MIN_SAFE_INTEGER\n\tassertRoundTrips(Number(-9007199254740991))\n\tassertRoundTrips(Number(math.MaxFloat64))\n\tassertRoundTrips(Number(math.Nextafter(1, 2) - 1))\n\n\tassertRoundTrips(String(\"\"))\n\tassertRoundTrips(String(\"foo\"))\n\tassertRoundTrips(String(\"AINT NO THANG\"))\n\tassertRoundTrips(String(\"💩\"))\n\n\tassertRoundTrips(NewStruct(\"\", StructData{\"a\": Bool(true), \"b\": String(\"foo\"), \"c\": Number(2.3)}))\n\n\tlistLeaf := newList(newListLeafSequence(vs, Number(4), Number(5), Number(6), Number(7)))\n\tassertRoundTrips(listLeaf)\n\n\tassertRoundTrips(newList(newListMetaSequence(1, []metaTuple{\n\t\tnewMetaTuple(NewRef(listLeaf), orderedKeyFromInt(10), 10),\n\t\tnewMetaTuple(NewRef(listLeaf), orderedKeyFromInt(20), 20),\n\t}, vs)))\n}\n\nfunc TestNonFiniteNumbers(tt *testing.T) {\n\tt := func(f float64, s string) {\n\t\tv := Number(f)\n\t\terr := d.Try(func() {\n\t\t\tEncodeValue(v)\n\t\t})\n\t\tassert.Error(tt, err)\n\t\tassert.Contains(tt, err.Error(), s)\n\t}\n\tt(math.NaN(), \"NaN is not a supported number\")\n\tt(math.Inf(1), \"+Inf is not a supported number\")\n\tt(math.Inf(-1), \"-Inf is not a supported number\")\n}\n\nfunc TestWritePrimitives(t *testing.T) {\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tBoolKind, true,\n\t\t},\n\t\tBool(true))\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tBoolKind, false,\n\t\t},\n\t\tBool(false))\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tNumberKind, Number(0),\n\t\t},\n\t\tNumber(0))\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tNumberKind, Number(1000000000000000000),\n\t\t},\n\t\tNumber(1e18))\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tNumberKind, Number(10000000000000000000),\n\t\t},\n\t\tNumber(1e19))\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tNumberKind, Number(1e+20),\n\t\t},\n\t\tNumber(1e20))\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tStringKind, \"hi\",\n\t\t},\n\t\tString(\"hi\"))\n}\n\nfunc TestWriteSimpleBlob(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tBlobKind, uint64(0), []byte{0x00, 0x01},\n\t\t},\n\t\tNewBlob(vrw, bytes.NewBuffer([]byte{0x00, 0x01})),\n\t)\n}\n\nfunc TestWriteList(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tListKind, uint64(0), uint64(4) /* len */, NumberKind, Number(0), NumberKind, Number(1), NumberKind, Number(2), NumberKind, Number(3),\n\t\t},\n\t\tNewList(vrw, Number(0), Number(1), Number(2), Number(3)),\n\t)\n}\n\nfunc TestWriteListOfList(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tListKind, uint64(0),\n\t\t\tuint64(2), // len\n\t\t\tListKind, uint64(0), uint64(1) /* len */, NumberKind, Number(0),\n\t\t\tListKind, uint64(0), uint64(3) /* len */, NumberKind, Number(1), NumberKind, Number(2), NumberKind, Number(3),\n\t\t},\n\t\tNewList(vrw, NewList(vrw, Number(0)), NewList(vrw, Number(1), Number(2), Number(3))),\n\t)\n}\n\nfunc TestWriteSet(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tSetKind, uint64(0), uint64(4), /* len */\n\t\t\tNumberKind, Number(0), NumberKind, Number(1), NumberKind, Number(2), NumberKind, Number(3),\n\t\t},\n\t\tNewSet(vrw, Number(3), Number(1), Number(2), Number(0)),\n\t)\n}\n\nfunc TestWriteSetOfSet(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tSetKind, uint64(0), uint64(2), // len\n\t\t\tSetKind, uint64(0), uint64(3) /* len */, NumberKind, Number(1), NumberKind, Number(2), NumberKind, Number(3),\n\t\t\tSetKind, uint64(0), uint64(1) /* len */, NumberKind, Number(0),\n\t\t},\n\t\tNewSet(vrw, NewSet(vrw, Number(0)), NewSet(vrw, Number(1), Number(2), Number(3))),\n\t)\n}\n\nfunc TestWriteMap(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tMapKind, uint64(0), uint64(2), /* len */\n\t\t\tStringKind, \"a\", BoolKind, false, StringKind, \"b\", BoolKind, true,\n\t\t},\n\t\tNewMap(vrw, String(\"a\"), Bool(false), String(\"b\"), Bool(true)),\n\t)\n}\n\nfunc TestWriteMapOfMap(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tMapKind, uint64(0), uint64(1), // len\n\t\t\tMapKind, uint64(0), uint64(1) /* len */, StringKind, \"a\", NumberKind, Number(0),\n\t\t\tSetKind, uint64(0), uint64(1) /* len */, BoolKind, true,\n\t\t},\n\t\tNewMap(vrw, NewMap(vrw, String(\"a\"), Number(0)), NewSet(vrw, Bool(true))),\n\t)\n}\n\nfunc TestWriteCompoundBlob(t *testing.T) {\n\tr1 := hash.Parse(\"00000000000000000000000000000001\")\n\tr2 := hash.Parse(\"00000000000000000000000000000002\")\n\tr3 := hash.Parse(\"00000000000000000000000000000003\")\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tBlobKind, uint64(1),\n\t\t\tuint64(3), // len\n\t\t\tRefKind, r1, BlobKind, uint64(11), NumberKind, Number(20), uint64(20),\n\t\t\tRefKind, r2, BlobKind, uint64(22), NumberKind, Number(40), uint64(40),\n\t\t\tRefKind, r3, BlobKind, uint64(33), NumberKind, Number(60), uint64(60),\n\t\t},\n\t\tnewBlob(newBlobMetaSequence(1, []metaTuple{\n\t\t\tnewMetaTuple(constructRef(r1, BlobType, 11), orderedKeyFromInt(20), 20),\n\t\t\tnewMetaTuple(constructRef(r2, BlobType, 22), orderedKeyFromInt(40), 40),\n\t\t\tnewMetaTuple(constructRef(r3, BlobType, 33), orderedKeyFromInt(60), 60),\n\t\t}, newTestValueStore())),\n\t)\n}\n\nfunc TestWriteEmptyStruct(t *testing.T) {\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tStructKind, \"S\", uint64(0), /* len */\n\t\t},\n\t\tNewStruct(\"S\", nil),\n\t)\n}\n\nfunc TestWriteStruct(t *testing.T) {\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tStructKind, \"S\", uint64(2), /* len */\n\t\t\t\"b\", BoolKind, true, \"x\", NumberKind, Number(42),\n\t\t},\n\t\tNewStruct(\"S\", StructData{\"x\": Number(42), \"b\": Bool(true)}),\n\t)\n}\n\nfunc TestWriteStructTooMuchData(t *testing.T) {\n\ts := NewStruct(\"S\", StructData{\"x\": Number(42), \"b\": Bool(true)})\n\tc := EncodeValue(s)\n\tdata := c.Data()\n\tbuff := make([]byte, len(data)+1)\n\tcopy(buff, data)\n\tbuff[len(data)] = 5 // Add a bogus extrabyte\n\tassert.Panics(t, func() {\n\t\tDecodeFromBytes(buff, nil)\n\t})\n}\n\nfunc TestWriteStructWithList(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\t// struct S {l: List}({l: [\"a\", \"b\"]})\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tStructKind, \"S\", uint64(1), /* len */\n\t\t\t\"l\", ListKind, uint64(0), uint64(2) /* len */, StringKind, \"a\", StringKind, \"b\",\n\t\t},\n\t\tNewStruct(\"S\", StructData{\"l\": NewList(vrw, String(\"a\"), String(\"b\"))}),\n\t)\n\n\t// struct S {l: List<>}({l: []})\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tStructKind, \"S\", uint64(1), /* len */\n\t\t\t\"l\", ListKind, uint64(0), uint64(0), /* len */\n\t\t},\n\t\tNewStruct(\"S\", StructData{\"l\": NewList(vrw)}),\n\t)\n}\n\nfunc TestWriteStructWithStruct(t *testing.T) {\n\t// struct S2 {\n\t// x: Number\n\t// }\n\t// struct S {\n\t// s: S2\n\t// }\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tStructKind, \"S\", uint64(1), // len\n\t\t\t\"s\", StructKind, \"S2\", uint64(1), /* len */\n\t\t\t\"x\", NumberKind, Number(42),\n\t\t},\n\t\t// {s: {x: 42}}\n\t\tNewStruct(\"S\", StructData{\"s\": NewStruct(\"S2\", StructData{\"x\": Number(42)})}),\n\t)\n}\n\nfunc TestWriteStructWithBlob(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tStructKind, \"S\", uint64(1), /* len */\n\t\t\t\"b\", BlobKind, uint64(0), []byte{0x00, 0x01},\n\t\t},\n\t\tNewStruct(\"S\", StructData{\"b\": NewBlob(vrw, bytes.NewBuffer([]byte{0x00, 0x01}))}),\n\t)\n}\n\nfunc TestWriteCompoundList(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tlist1 := newList(newListLeafSequence(vrw, Number(0)))\n\tlist2 := newList(newListLeafSequence(vrw, Number(1), Number(2), Number(3)))\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tListKind, uint64(1), uint64(2), // len,\n\t\t\tRefKind, list1.Hash(), ListKind, NumberKind, uint64(1), NumberKind, Number(1), uint64(1),\n\t\t\tRefKind, list2.Hash(), ListKind, NumberKind, uint64(1), NumberKind, Number(3), uint64(3),\n\t\t},\n\t\tnewList(newListMetaSequence(1, []metaTuple{\n\t\t\tnewMetaTuple(NewRef(list1), orderedKeyFromInt(1), 1),\n\t\t\tnewMetaTuple(NewRef(list2), orderedKeyFromInt(3), 3),\n\t\t}, nil)),\n\t)\n}\n\nfunc TestWriteCompoundSet(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tset1 := newSet(newSetLeafSequence(vrw, Number(0), Number(1)))\n\tset2 := newSet(newSetLeafSequence(vrw, Number(2), Number(3), Number(4)))\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tSetKind, uint64(1), uint64(2), // len,\n\t\t\tRefKind, set1.Hash(), SetKind, NumberKind, uint64(1), NumberKind, Number(1), uint64(2),\n\t\t\tRefKind, set2.Hash(), SetKind, NumberKind, uint64(1), NumberKind, Number(4), uint64(3),\n\t\t},\n\t\tnewSet(newSetMetaSequence(1, []metaTuple{\n\t\t\tnewMetaTuple(NewRef(set1), orderedKeyFromInt(1), 2),\n\t\t\tnewMetaTuple(NewRef(set2), orderedKeyFromInt(4), 3),\n\t\t}, vrw)),\n\t)\n}\n\nfunc TestWriteCompoundSetOfBlobs(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\t// Blobs are interesting because unlike the numbers used in TestWriteCompondSet, refs are sorted by their hashes, not their value.\n\tnewBlobOfInt := func(i int) Blob {\n\t\treturn NewBlob(vrw, strings.NewReader(strconv.Itoa(i)))\n\t}\n\n\tblob0 := newBlobOfInt(0)\n\tblob1 := newBlobOfInt(1)\n\tblob2 := newBlobOfInt(2)\n\tblob3 := newBlobOfInt(3)\n\tblob4 := newBlobOfInt(4)\n\n\tset1 := newSet(newSetLeafSequence(vrw, blob0, blob1))\n\tset2 := newSet(newSetLeafSequence(vrw, blob2, blob3, blob4))\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tSetKind, uint64(1), uint64(2), // len,\n\t\t\tRefKind, set1.Hash(), SetKind, BlobKind, uint64(1), hashKind, blob1.Hash(), uint64(2),\n\t\t\tRefKind, set2.Hash(), SetKind, BlobKind, uint64(1), hashKind, blob4.Hash(), uint64(3),\n\t\t},\n\t\tnewSet(newSetMetaSequence(1, []metaTuple{\n\t\t\tnewMetaTuple(NewRef(set1), newOrderedKey(blob1), 2),\n\t\t\tnewMetaTuple(NewRef(set2), newOrderedKey(blob4), 3),\n\t\t}, vrw)),\n\t)\n}\n\nfunc TestWriteListOfUnion(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t// Note that the order of members in a union is determined based on a hash computation; the particular ordering of Number, Bool, String was determined empirically. This must not change unless deliberately and explicitly revving the persistent format.\n\t\t[]interface{}{\n\t\t\tListKind, uint64(0),\n\t\t\tuint64(4) /* len */, StringKind, \"0\", NumberKind, Number(1), StringKind, \"2\", BoolKind, true,\n\t\t},\n\t\tNewList(vrw,\n\t\t\tString(\"0\"),\n\t\t\tNumber(1),\n\t\t\tString(\"2\"),\n\t\t\tBool(true),\n\t\t),\n\t)\n}\n\nfunc TestWriteListOfStruct(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tListKind, uint64(0), uint64(1), /* len */\n\t\t\tStructKind, \"S\", uint64(1) /* len */, \"x\", NumberKind, Number(42),\n\t\t},\n\t\tNewList(vrw, NewStruct(\"S\", StructData{\"x\": Number(42)})),\n\t)\n}\n\nfunc TestWriteListOfUnionWithType(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tstructType := MakeStructType(\"S\", StructField{\"x\", NumberType, false})\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tListKind, uint64(0), uint64(4), /* len */\n\t\t\tBoolKind, true,\n\t\t\tTypeKind, NumberKind,\n\t\t\tTypeKind, TypeKind,\n\t\t\tTypeKind, StructKind, \"S\", uint64(1) /* len */, \"x\", NumberKind, false,\n\t\t},\n\t\tNewList(vrw,\n\t\t\tBool(true),\n\t\t\tNumberType,\n\t\t\tTypeType,\n\t\t\tstructType,\n\t\t),\n\t)\n}\n\nfunc TestWriteRef(t *testing.T) {\n\tr := hash.Parse(\"0123456789abcdefghijklmnopqrstuv\")\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tRefKind, r, NumberKind, uint64(4),\n\t\t},\n\t\tconstructRef(r, NumberType, 4),\n\t)\n}\n\nfunc TestWriteListOfTypes(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tListKind, uint64(0), uint64(2), /* len */\n\t\t\tTypeKind, BoolKind, TypeKind, StringKind,\n\t\t},\n\t\tNewList(vrw, BoolType, StringType),\n\t)\n}\n\nfunc nomsTestWriteRecursiveStruct(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\t// struct A6 {\n\t// cs: List\n\t// v: Number\n\t// }\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tStructKind, \"A6\", uint64(2) /* len */, \"cs\", ListKind, CycleKind, uint64(0), \"v\", NumberKind,\n\t\t\tListKind, UnionKind, uint64(0) /* len */, false, uint64(0), /* len */\n\t\t\tNumberKind, Number(42),\n\t\t},\n\t\t// {v: 42, cs: [{v: 555, cs: []}]}\n\t\tNewStruct(\"A6\", StructData{\"cs\": NewList(vrw), \"v\": Number(42)}),\n\t)\n}\n\nfunc TestWriteUnionList(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tListKind, uint64(0), uint64(3), /* len */\n\t\t\tNumberKind, Number(23), StringKind, \"hi\", NumberKind, Number(42),\n\t\t},\n\t\tNewList(vrw, Number(23), String(\"hi\"), Number(42)),\n\t)\n}\n\nfunc TestWriteEmptyUnionList(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassertEncoding(t,\n\t\t[]interface{}{\n\t\t\tListKind, uint64(0), uint64(0), /* len */\n\t\t},\n\t\tNewList(vrw),\n\t)\n}\n\ntype bogusType int\n\nfunc (bg bogusType) Value() Value { return bg }\nfunc (bg bogusType) Equals(other Value) bool { return false }\nfunc (bg bogusType) Less(other Value) bool { return false }\nfunc (bg bogusType) Hash() hash.Hash { return hash.Hash{} }\nfunc (bg bogusType) WalkValues(cb ValueCallback) {}\nfunc (bg bogusType) WalkRefs(cb RefCallback) {}\nfunc (bg bogusType) Kind() NomsKind {\n\treturn CycleKind\n}\nfunc (bg bogusType) typeOf() *Type {\n\treturn MakeCycleType(\"ABC\")\n}\nfunc (bg bogusType) writeTo(nomsWriter) {\n\tpanic(\"abc\")\n}\n\nfunc TestBogusValueWithUnresolvedCycle(t *testing.T) {\n\tg := bogusType(1)\n\tassert.Panics(t, func() {\n\t\tEncodeValue(g)\n\t})\n}\n" }, { "path": "go/types/equals_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestValueEquals(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tvalues := []func() Value{\n\t\tfunc() Value { return Bool(false) },\n\t\tfunc() Value { return Bool(true) },\n\t\tfunc() Value { return Number(0) },\n\t\tfunc() Value { return Number(-1) },\n\t\tfunc() Value { return Number(1) },\n\t\tfunc() Value { return String(\"\") },\n\t\tfunc() Value { return String(\"hi\") },\n\t\tfunc() Value { return String(\"bye\") },\n\t\tfunc() Value {\n\t\t\treturn NewBlob(vrw, &bytes.Buffer{})\n\t\t},\n\t\tfunc() Value {\n\t\t\treturn NewBlob(vrw, bytes.NewBufferString(\"hi\"))\n\t\t},\n\t\tfunc() Value {\n\t\t\treturn NewBlob(vrw, bytes.NewBufferString(\"bye\"))\n\t\t},\n\t\tfunc() Value {\n\t\t\tb1 := NewBlob(vrw, bytes.NewBufferString(\"hi\"))\n\t\t\tb2 := NewBlob(vrw, bytes.NewBufferString(\"bye\"))\n\t\t\treturn newBlob(newBlobMetaSequence(1, []metaTuple{\n\t\t\t\tnewMetaTuple(NewRef(b1), orderedKeyFromInt(2), 2),\n\t\t\t\tnewMetaTuple(NewRef(b2), orderedKeyFromInt(5), 5),\n\t\t\t}, nil))\n\t\t},\n\t\tfunc() Value { return NewList(vrw) },\n\t\tfunc() Value { return NewList(vrw, String(\"foo\")) },\n\t\tfunc() Value { return NewList(vrw, String(\"bar\")) },\n\t\tfunc() Value { return NewMap(vrw) },\n\t\tfunc() Value { return NewMap(vrw, String(\"a\"), String(\"a\")) },\n\t\tfunc() Value { return NewSet(vrw) },\n\t\tfunc() Value { return NewSet(vrw, String(\"hi\")) },\n\n\t\tfunc() Value { return BoolType },\n\t\tfunc() Value { return StringType },\n\t\tfunc() Value { return MakeStructType(\"a\") },\n\t\tfunc() Value { return MakeStructType(\"b\") },\n\t\tfunc() Value { return MakeListType(BoolType) },\n\t\tfunc() Value { return MakeListType(NumberType) },\n\t\tfunc() Value { return MakeSetType(BoolType) },\n\t\tfunc() Value { return MakeSetType(NumberType) },\n\t\tfunc() Value { return MakeRefType(BoolType) },\n\t\tfunc() Value { return MakeRefType(NumberType) },\n\t\tfunc() Value {\n\t\t\treturn MakeMapType(BoolType, ValueType)\n\t\t},\n\t\tfunc() Value {\n\t\t\treturn MakeMapType(NumberType, ValueType)\n\t\t},\n\t}\n\n\tfor i, f1 := range values {\n\t\tfor j, f2 := range values {\n\t\t\tif i == j {\n\t\t\t\tassert.True(f1().Equals(f2()))\n\t\t\t} else {\n\t\t\t\tassert.False(f1().Equals(f2()))\n\t\t\t}\n\t\t}\n\t\tv := f1()\n\t\tif v != nil {\n\t\t\tr := NewRef(v)\n\t\t\tassert.False(r.Equals(v))\n\t\t\tassert.False(v.Equals(r))\n\t\t}\n\t}\n}\n" }, { "path": "go/types/get_hash.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport \"github.com/attic-labs/noms/go/hash\"\n\nvar getHashOverride func(v Value) hash.Hash\n\nfunc getHash(v Value) hash.Hash {\n\tif getHashOverride != nil {\n\t\treturn getHashOverride(v)\n\t}\n\treturn getHashNoOverride(v)\n}\n\nfunc getHashNoOverride(v Value) hash.Hash {\n\treturn EncodeValue(v).Hash()\n}\n\nfunc EnsureHash(h *hash.Hash, v Value) hash.Hash {\n\tif h.IsEmpty() {\n\t\t*h = getHash(v)\n\t}\n\treturn *h\n}\n" }, { "path": "go/types/graph_builder.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// GraphBuilder allows non-RAM-bound construction of a graph of nested Maps whose\n// leaf collections can be Lists, Sets, or Maps that contain any type of Noms\n// Values.\n//\n// Graphs are built by calling one of the GraphBuilder functions:\n// MapSet(graphKeys, key, value)\n// SetInsert(graphKeys, value)\n// ListAppend(graphKeys, value)\n//\n// GraphBuilder uses an opCache to store graph operations in leveldb and to be\n// able to read them out later in a way which ensures a total ordering of all\n// the nodes at each level of the graph. (See opcache.go for more info on how\n// that is done)\n//\n// GraphBuilder.Build() does the work of assembling the graph. Build() gets an\n// iterator for this graph from the opCache and uses it to iterate over all the\n// operations that have been stored for this graph. opCache ensures that the\n// operations are returned in optimal sorted order so that sequenceChunker can\n// most efficiently assemble the graph. Build() will ensure that there is a Map\n// object for each key in |graphKeys|. Any node that falls in the middle of the\n// graph must be a Map, although, intermediate nodes may have any element as keys\n// as long as the path formed by the graphKeys doesn't conflict.\n//\n// MapSet(), SetInsert(), and ListAppend() are threadsafe meaning they can safely\n// be called from different go routines. However, the semantics of ListAppend()\n// are such that the order of the list will be determined by which thread() calls\n// ListAppend first (this function call may be modified later to allow specification\n// of index or order).\n//\n// Build() should only be called once, after all the operations for the graph\n// have been stored. It is the caller's responsibility to make sure that all\n// calls to the mutation operations have completed before Build() is invoked.\n//\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype GraphBuilder struct {\n\topcStore opCacheStore\n\toc opCache\n\tvrw ValueReadWriter\n\tstack graphStack\n\tmutex sync.Mutex\n}\n\n// NewGraphBuilder returns an new GraphBuilder object.\nfunc NewGraphBuilder(vrw ValueReadWriter, rootKind NomsKind) *GraphBuilder {\n\treturn newGraphBuilder(vrw, newLdbOpCacheStore(vrw), rootKind)\n}\n\nfunc newGraphBuilder(vrw ValueReadWriter, opcStore opCacheStore, rootKind NomsKind) *GraphBuilder {\n\tb := &GraphBuilder{oc: opcStore.opCache(), opcStore: opcStore, vrw: vrw}\n\tb.pushNewKeyOnStack(String(\"ROOT\"), rootKind)\n\treturn b\n}\n\n// MapSet will add the key/value pair |k, v| to the map found by traversing\n// the graph using the |keys| slice. Intermediate maps referenced by |keys| are\n// created as necessary. This is threadsafe, may call from multiple go routines.\nfunc (b *GraphBuilder) MapSet(keys []Value, k Value, v Value) {\n\tif b.oc == nil {\n\t\td.Panic(\"Can't call MapSet() again after Build()\")\n\t}\n\tb.oc.GraphMapSet(keys, k, v)\n}\n\n// SetInsert will insert the value |v| into the set at path |keys|. Intermediate\n// maps referenced by |keys| are created as necessary. This is threadsafe, may\n// call from multiple go routines.\nfunc (b *GraphBuilder) SetInsert(keys []Value, v Value) {\n\tif b.oc == nil {\n\t\td.Panic(\"Can't call SetInsert() again after Build()\")\n\t}\n\tb.oc.GraphSetInsert(keys, v)\n}\n\n// ListAppend will append |v| to the list at path |p|. Intermediate\n// maps referenced by |keys| are created as necessary. This is threadsafe, may\n// call from multiple go routines, however append semantics are such that the\n// elements will be appended in order that functions are called, so order has\n// to be managed by caller.\nfunc (b *GraphBuilder) ListAppend(keys []Value, v Value) {\n\tif b.oc == nil {\n\t\td.Panic(\"Can't call ListAppend() again after Build()\")\n\t}\n\tb.oc.GraphListAppend(keys, v)\n}\n\ntype graphOpContainer struct {\n\tkeys []Value\n\tkind NomsKind\n\titem sequenceItem\n}\n\n// Build builds and returns the graph. This method should only be called after all\n// calls to the mutation operations (i.e. MapSet, SetInsert, and ListAppend)\n// have completed. It is the caller's responsibility to ensure that this is\n// the case. Build() will panic if called more than once on any GraphBuilder\n// object.\nfunc (b *GraphBuilder) Build() Value {\n\tvar opc opCache\n\tvar opcStore opCacheStore\n\n\tdefer func() {\n\t\topcStore.destroy()\n\t}()\n\n\t// Use function here to take advantage fo the deferred call to mutex.Unlock()\n\tfunc() {\n\t\tb.mutex.Lock()\n\t\tdefer b.mutex.Unlock()\n\n\t\tif b.oc == nil {\n\t\t\td.Panic(\"Can only call Build() once\")\n\t\t}\n\t\topcStore, opc = b.opcStore, b.oc\n\t\tb.opcStore, b.oc = nil, nil\n\t}()\n\n\titer := opc.NewIterator()\n\tdefer iter.Release()\n\n\t// start up a go routine that will do the reading from graphBuilder's private\n\t// ldb opCache.\n\tgraphOpChan := make(chan graphOpContainer, 512)\n\tgo func() {\n\t\tfor iter.Next() {\n\t\t\tkeys, kind, item := iter.GraphOp()\n\t\t\tcontainer := graphOpContainer{keys: keys, kind: kind, item: item}\n\t\t\tgraphOpChan <- container\n\t\t}\n\t\tclose(graphOpChan)\n\t}()\n\n\t// iterator returns keys, in sort order by array\n\tfor goc := range graphOpChan {\n\t\tkeys, kind, item := goc.keys, goc.kind, goc.item\n\n\t\t// Get index of first key that is different than what is on the stack\n\t\tidx := commonPrefixCount(b.stack, keys)\n\t\tif idx == -1 {\n\t\t\t// no keys have changed we're working on same coll as previous\n\t\t\t// iteration, just append to sequenceChunker at top of stack\n\t\t\tb.appendItemToCurrentTopOfStack(kind, item)\n\t\t\tcontinue\n\t\t}\n\n\t\t// Some keys that were in the last graphOp are no longer present\n\t\t// which indicates that we are finished adding to those cols. Pop\n\t\t// those keys from the stack. This will cause any popped cols to be\n\t\t// closed and added to their parents.\n\t\tfor idx < b.stack.lastIdx() {\n\t\t\tb.popKeyFromStack()\n\t\t}\n\n\t\t// We may have popped some keys off of the stack and are left with\n\t\t// an item to append to the stack of a previously existing key.\n\t\tif b.stack.lastIdx() == len(keys) {\n\t\t\tb.appendItemToCurrentTopOfStack(kind, item)\n\t\t}\n\n\t\t// Or we may have some new keys to add to the stack. Add those keys\n\t\t// and then append the item to the top element.\n\t\tfor b.stack.lastIdx() < len(keys) {\n\t\t\tif b.stack.lastIdx() < len(keys)-1 {\n\t\t\t\tb.pushNewKeyOnStack(keys[b.stack.lastIdx()], MapKind)\n\t\t\t} else {\n\t\t\t\tb.pushNewKeyOnStack(keys[b.stack.lastIdx()], kind)\n\t\t\t\tb.appendItemToCurrentTopOfStack(kind, item)\n\t\t\t}\n\t\t}\n\t}\n\n\t// We're done adding elements. Pop any intermediate keys off the stack and\n\t// fold their results into their parent map.\n\tfor b.stack.len() > 1 {\n\t\tb.popKeyFromStack()\n\t}\n\tres := b.stack.pop().done()\n\treturn res\n\n}\n\n// pushNewKeyOnStack() creates a new graphStackElem node and pushes it on the\n// stack. The new element contains the |key| and a new sequenceChunker that will\n// be appended to to build this node in the graph.\nfunc (b *GraphBuilder) pushNewKeyOnStack(key Value, kind NomsKind) {\n\tvar ch *sequenceChunker\n\tswitch kind {\n\tcase MapKind:\n\t\tch = newEmptyMapSequenceChunker(b.vrw)\n\tcase SetKind:\n\t\tch = newEmptySetSequenceChunker(b.vrw)\n\tcase ListKind:\n\t\tch = newEmptyListSequenceChunker(b.vrw)\n\tdefault:\n\t\tpanic(\"bad 'kind' value in GraphBuilder, newElem()\")\n\t}\n\tb.stack.push(&graphStackElem{key: key, kind: kind, ch: ch})\n}\n\n// popKeyFromStack() pops the last element off the stack, calls done() to\n// finish any sequenceChunking that is in progress, and then assigns the\n// finished collection it's parent map.\nfunc (b *GraphBuilder) popKeyFromStack() {\n\telem := b.stack.pop()\n\tcol := elem.done()\n\ttop := b.stack.top()\n\ttop.ch.Append(mapEntry{elem.key, col})\n}\n\n// appendItemToCurrentTopOfStack() adds the current item to the sequenceChunker\n// that's on the top of the stack.\nfunc (b *GraphBuilder) appendItemToCurrentTopOfStack(kind NomsKind, item sequenceItem) {\n\ttop := b.stack.top()\n\td.PanicIfTrue(top.kind != kind)\n\ttop.ch.Append(item)\n}\n\ntype graphStackElem struct {\n\tkey Value\n\tkind NomsKind\n\tch *sequenceChunker\n}\n\ntype graphStack struct {\n\telems []*graphStackElem\n}\n\nfunc (s *graphStack) push(e *graphStackElem) {\n\ts.elems = append(s.elems, e)\n}\n\nfunc (s *graphStack) pop() *graphStackElem {\n\tl := len(s.elems) - 1\n\telem := s.elems[l] // last element\n\ts.elems = s.elems[:l] // truncate last element\n\treturn elem\n}\n\nfunc (s *graphStack) top() *graphStackElem {\n\tl := len(s.elems) - 1\n\treturn s.elems[l] // last element\n}\n\nfunc (s *graphStack) len() int {\n\treturn len(s.elems)\n}\n\nfunc (s *graphStack) lastIdx() int {\n\treturn len(s.elems) - 1\n}\n\nfunc (s graphStack) String() string {\n\tbuf := bytes.Buffer{}\n\tfor i := len(s.elems) - 1; i >= 0; i-- {\n\t\tfmt.Fprintln(&buf, \"#:\", i, s.elems[i])\n\t}\n\treturn buf.String()\n}\n\n// done() creates the appropriate collection for this element and returns it\nfunc (e *graphStackElem) done() Collection {\n\tswitch e.kind {\n\tcase MapKind:\n\t\treturn newMap(e.ch.Done().(orderedSequence))\n\tcase SetKind:\n\t\treturn newSet(e.ch.Done().(orderedSequence))\n\tcase ListKind:\n\t\treturn newList(e.ch.Done())\n\t}\n\tpanic(\"unreachable\")\n}\n\n// Returns index of first element in keys that is different from stack. Note,\n// return value can be equal to len(keys) if there are more element in stack\n// than in keys\nfunc commonPrefixCount(stack graphStack, keys ValueSlice) int {\n\tminLen := len(keys)\n\t// don't consider the 'ROOT' stack element\n\telems := stack.elems[1:]\n\tif len(elems) < minLen {\n\t\tminLen = len(elems)\n\t}\n\n\tfor i := 0; i < minLen; i++ {\n\t\tif !elems[i].key.Equals(keys[i]) {\n\t\t\treturn i\n\t\t}\n\t}\n\n\tif len(keys) == len(elems) {\n\t\treturn -1\n\t}\n\treturn minLen\n}\n\nfunc (e *graphStackElem) String() string {\n\treturn fmt.Sprintf(\"key: %s, kind: %s, seq: %p\", EncodedValue(e.key), e.kind, e.ch)\n}\n" }, { "path": "go/types/graph_builder_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"fmt\"\n\t\"math/rand\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestGraphBuilderFindIndex(t *testing.T) {\n\tassert := assert.New(t)\n\n\telems := []*graphStackElem{\n\t\t{key: String(\"ROOT\")},\n\t\t{key: String(\"one\")},\n\t\t{key: String(\"two\")},\n\t\t{key: String(\"three\")},\n\t\t{key: String(\"four\")},\n\t}\n\n\ts := graphStack{elems: elems}\n\tassert.Equal(0, commonPrefixCount(s, []Value{String(\"zero\")}))\n\tassert.Equal(1, commonPrefixCount(s, []Value{String(\"one\"), String(\"zero\")}))\n\tassert.Equal(3, commonPrefixCount(s, []Value{String(\"one\"), String(\"two\"), String(\"three\")}))\n\tassert.Equal(-1, commonPrefixCount(s, []Value{String(\"one\"), String(\"two\"), String(\"three\"), String(\"four\")}))\n\tassert.Equal(4, commonPrefixCount(s, []Value{String(\"one\"), String(\"two\"), String(\"three\"), String(\"four\"), String(\"five\")}))\n\n\tvalues := []Value{String(\"one\"), String(\"two\"), String(\"three\"), String(\"four\")}\n\n\tassert.Equal(-1, commonPrefixCount(graphStack{elems: elems[:1]}, []Value{}))\n\tassert.Equal(0, commonPrefixCount(graphStack{elems: elems[:1]}, values))\n\tassert.Equal(1, commonPrefixCount(graphStack{elems: elems[:2]}, values))\n\tassert.Equal(3, commonPrefixCount(graphStack{elems: elems[:4]}, values))\n\tassert.Equal(-1, commonPrefixCount(graphStack{elems: elems}, values))\n\tassert.Equal(2, commonPrefixCount(graphStack{elems: elems[:5]}, values[:2]))\n}\n\ntype testGraphOp struct {\n\tkeys ValueSlice\n\tkind NomsKind\n\titem sequenceItem\n}\n\nfunc SafeEquals(v1, v2 Value) bool {\n\tif v1 == nil && v2 == nil {\n\t\treturn true\n\t}\n\tif v1 == nil || v2 == nil {\n\t\treturn false\n\t}\n\treturn v1.Equals(v2)\n}\n\nfunc TestGraphBuilderEncodeDecodeAsKey(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\tdefer vrw.Close()\n\n\tstruct1 := NewStruct(\"teststruct\", StructData{\n\t\t\"f1\": String(\"v1\"),\n\t\t\"f2\": String(\"v2\"),\n\t})\n\n\tkeys := ValueSlice{Bool(true), Number(19), String(\"think!\"), struct1}\n\tbyteBuf := [initialBufferSize]byte{}\n\tbs := byteBuf[:0]\n\tnumKeys := len(keys)\n\texpectedRes := ValueSlice{}\n\tfor _, k := range keys {\n\t\tif isKindOrderedByValue(k.Kind()) {\n\t\t\texpectedRes = append(expectedRes, k)\n\t\t} else {\n\t\t\texpectedRes = append(expectedRes, nil)\n\t\t}\n\t\tbs = encodeGraphKey(bs, k)\n\t}\n\tres := ValueSlice{}\n\tfor pos := 0; pos < numKeys; pos++ {\n\t\tvar k Value\n\t\tbs, k = decodeValue(bs, false, vrw)\n\t\tres = append(res, k)\n\t}\n\n\tassert.Equal(len(keys), len(res))\n\tfor i, origKey := range expectedRes {\n\t\tassert.True(SafeEquals(origKey, res[i]))\n\t}\n}\n\nfunc TestGraphBuilderEncodeDecodeAsValue(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\tdefer vrw.Close()\n\n\tstruct1 := NewStruct(\"teststruct\", StructData{\n\t\t\"f1\": String(\"v1\"),\n\t\t\"f2\": String(\"v2\"),\n\t})\n\n\tkeys := ValueSlice{Bool(true), Number(19), String(\"think!\"), struct1}\n\tbyteBuf := [initialBufferSize]byte{}\n\tbs := byteBuf[:0]\n\tnumKeys := len(keys)\n\tfor _, k := range keys {\n\t\tbs = encodeGraphValue(bs, k)\n\t}\n\tres := ValueSlice{}\n\tfor pos := 0; pos < numKeys; pos++ {\n\t\tvar k Value\n\t\tbs, k = decodeValue(bs, true, vrw)\n\t\tres = append(res, k)\n\t}\n\n\tassert.Equal(len(keys), len(res))\n\tfor i, origKey := range keys {\n\t\tassert.True(SafeEquals(origKey, res[i]))\n\t}\n}\n\nfunc TestGraphBuilderMapSetGraphOp(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\topcStore := newLdbOpCacheStore(vs)\n\topc := opcStore.opCache()\n\tdefer opcStore.destroy()\n\n\tstruct1 := NewStruct(\"teststruct\", StructData{\n\t\t\"f1\": String(\"v1\"),\n\t\t\"f2\": String(\"v2\"),\n\t})\n\tkeys := ValueSlice{Bool(true), Number(19), String(\"think!\"), struct1}\n\topc.GraphMapSet(keys, String(\"yo\"), Number(199))\n\titer := opc.NewIterator()\n\tassert.True(iter.Next())\n\n\tkeys1, kind, item := iter.GraphOp()\n\tassert.Equal(len(keys), len(keys1))\n\tassert.True(keys.Equals(keys1))\n\tassert.Equal(MapKind, kind)\n\tassert.IsType(mapEntry{}, item)\n\tme := item.(mapEntry)\n\tassert.True(String(\"yo\").Equals(me.key))\n\tassert.True(Number(199).Equals(me.value))\n\n\tassert.False(iter.Next())\n}\n\n// createTestMap() constructs a graph sized according to the |levels| and\n// |avgSize| parameters. The graph will contain nested maps with a\n// depth == |levels|, each map will contain |avgSize| elements of different\n// types.\nfunc createTestMap(vrw ValueReadWriter, levels, avgSize int, valGen func() Value) Map {\n\tsampleSize := func() int {\n\t\tsize := (int(rand.Int31()) % avgSize) + (avgSize / 2)\n\t\tif size < 2 {\n\t\t\treturn 2\n\t\t}\n\t\treturn size\n\t}\n\n\tgenLeaf := func() Value {\n\t\tnumElems := sampleSize()\n\t\telems := ValueSlice{}\n\t\tfor i := 0; i < numElems; i++ {\n\t\t\telems = append(elems, valGen())\n\t\t}\n\t\tswitch rand.Int31() % 3 {\n\t\tcase 0:\n\t\t\tif numElems%2 != 0 {\n\t\t\t\tnumElems--\n\t\t\t}\n\t\t\treturn NewMap(vrw, elems[:numElems]...)\n\t\tcase 1:\n\t\t\treturn NewSet(vrw, elems...)\n\t\tcase 2:\n\t\t\treturn NewList(vrw, elems...)\n\t\t}\n\t\tpanic(\"unreachable\")\n\t}\n\n\tvar genChildren func(lvl int) Map\n\tgenChildren = func(lvl int) Map {\n\t\tnumChildren := sampleSize()\n\t\tkvs := ValueSlice{}\n\t\tfor i := 0; i < numChildren; i++ {\n\t\t\tif lvl == levels {\n\t\t\t\tkvs = append(kvs, valGen(), genLeaf())\n\t\t\t} else {\n\t\t\t\t// Once in a while, throw in a non-collection value into the\n\t\t\t\t// middle of the graph\n\t\t\t\tif rand.Int31()%10 == 0 {\n\t\t\t\t\tkvs = append(kvs, valGen(), valGen())\n\t\t\t\t} else {\n\t\t\t\t\tkvs = append(kvs, valGen(), genChildren(lvl+1))\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\treturn NewMap(vrw, kvs...)\n\t}\n\n\treturn genChildren(0)\n}\n\n// valGen() creates a random String, Number, or Struct Value\nfunc valGen() Value {\n\tnum := rand.Int31() % 1000000\n\tswitch rand.Int31() % 4 {\n\tcase 0:\n\t\treturn String(fmt.Sprintf(\"%d\", num))\n\tcase 1:\n\t\treturn Number(num)\n\tcase 2:\n\t\treturn NewStruct(\"teststruct\", map[string]Value{\"f1\": Number(num)})\n\tcase 3:\n\t\treturn NewStruct(\"teststruct\", map[string]Value{\"f1\": String(fmt.Sprintf(\"%d\", num))})\n\t}\n\tpanic(\"unreachable\")\n}\n\n// dupSlice() duplicates a slice along with it's backing store.\nfunc dupSlice(s ValueSlice) ValueSlice {\n\tvs := make(ValueSlice, len(s))\n\tcopy(vs, s)\n\treturn vs\n}\n\nfunc shuffle(a []testGraphOp) {\n\tfor i := range a {\n\t\tj := rand.Intn(i + 1)\n\t\tif a[i].kind != ListKind && a[j].kind != ListKind {\n\t\t\ta[i], a[j] = a[j], a[i]\n\t\t}\n\t}\n}\n\n// See https://github.com/attic-labs/noms/issues/3840\nfunc TestGraphBuilderNestedMapSet(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\texpected := createTestMap(vs, 3, 4, valGen)\n\tb := NewGraphBuilder(vs, MapKind)\n\n\tops := []testGraphOp{}\n\n\tisNomsCollectionKind := func(kind NomsKind) bool {\n\t\treturn kind == MapKind || kind == SetKind || kind == ListKind\n\t}\n\tvar generateOps func(keys []Value, col Value)\n\tgenerateOps = func(keys []Value, col Value) {\n\t\tswitch c := col.(type) {\n\t\tcase Map:\n\t\t\tc.Iter(func(k, v Value) bool {\n\t\t\t\tif isNomsCollectionKind(v.Kind()) {\n\t\t\t\t\tnewKeys := append(keys, k)\n\t\t\t\t\tgenerateOps(newKeys, v)\n\t\t\t\t} else {\n\t\t\t\t\ttgo := testGraphOp{keys: dupSlice(keys), kind: MapKind, item: mapEntry{k, v}}\n\t\t\t\t\tops = append(ops, tgo)\n\t\t\t\t}\n\t\t\t\treturn false\n\t\t\t})\n\t\tcase List:\n\t\t\tc.Iter(func(v Value, idx uint64) bool {\n\t\t\t\ttgo := testGraphOp{keys: dupSlice(keys), kind: ListKind, item: v}\n\t\t\t\tops = append(ops, tgo)\n\t\t\t\treturn false\n\t\t\t})\n\t\tcase Set:\n\t\t\tc.Iter(func(v Value) bool {\n\t\t\t\ttgo := testGraphOp{keys: dupSlice(keys), kind: SetKind, item: v}\n\t\t\t\tops = append(ops, tgo)\n\t\t\t\treturn false\n\t\t\t})\n\t\t}\n\t}\n\tgenerateOps(nil, expected)\n\tshuffle(ops)\n\n\tfor _, op := range ops {\n\t\tswitch op.kind {\n\t\tcase MapKind:\n\t\t\tb.MapSet(op.keys, op.item.(mapEntry).key, op.item.(mapEntry).value)\n\t\tcase SetKind:\n\t\t\tb.SetInsert(op.keys, op.item.(Value))\n\t\tcase ListKind:\n\t\t\tb.ListAppend(op.keys, op.item.(Value))\n\t\t}\n\t}\n\n\tv := b.Build()\n\tassert.NotNil(v)\n\tassert.True(expected.Equals(v))\n}\n\nfunc ExampleGraphBuilder_Build() {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tgb := NewGraphBuilder(vs, MapKind)\n\tgb.SetInsert([]Value{String(\"parent\"), String(\"children\")}, String(\"John\"))\n\tgb.SetInsert([]Value{String(\"parent\"), String(\"children\")}, String(\"Mary\"))\n\tgb.SetInsert([]Value{String(\"parent\"), String(\"children\")}, String(\"Frieda\"))\n\tgb.MapSet([]Value{String(\"parent\"), String(\"ages\")}, String(\"Father\"), Number(42))\n\tgb.MapSet([]Value{String(\"parent\"), String(\"ages\")}, String(\"Mother\"), Number(44))\n\tgb.ListAppend([]Value{String(\"parent\"), String(\"chores\")}, String(\"Make dinner\"))\n\tgb.ListAppend([]Value{String(\"parent\"), String(\"chores\")}, String(\"Wash dishes\"))\n\tgb.ListAppend([]Value{String(\"parent\"), String(\"chores\")}, String(\"Make breakfast\"))\n\tgb.ListAppend([]Value{String(\"parent\"), String(\"chores\")}, String(\"Wash dishes\"))\n\tgb.MapSet([]Value{String(\"parent\")}, String(\"combinedAge\"), Number(86))\n\tm := gb.Build()\n\tfmt.Println(\"map:\", EncodedValue(m))\n}\n" }, { "path": "go/types/incremental_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc getTestVals(vrw ValueReadWriter) []Value {\n\treturn []Value{\n\t\tBool(true),\n\t\tNumber(1),\n\t\tString(\"hi\"),\n\t\tNewBlob(vrw, bytes.NewReader([]byte(\"hi\"))),\n\t\t// compoundBlob\n\t\tNewSet(vrw, String(\"hi\")),\n\t\tNewList(vrw, String(\"hi\")),\n\t\tNewMap(vrw, String(\"hi\"), String(\"hi\")),\n\t}\n}\n\nfunc isEncodedOutOfLine(v Value) int {\n\tswitch v.(type) {\n\tcase Ref:\n\t\treturn 1\n\t}\n\treturn 0\n}\n\nfunc TestIncrementalLoadList(t *testing.T) {\n\tassert := assert.New(t)\n\tts := &chunks.TestStorage{}\n\tcs := ts.NewView()\n\tvs := NewValueStore(cs)\n\n\texpected := NewList(vs, getTestVals(vs)...)\n\thash := vs.WriteValue(expected).TargetHash()\n\tvs.Commit(vs.Root(), vs.Root())\n\n\tactualVar := vs.ReadValue(hash)\n\tactual := actualVar.(List)\n\n\texpectedCount := cs.Reads\n\tassert.Equal(1, expectedCount)\n\t// There will be one read per chunk.\n\tchunkReads := make([]int, expected.Len())\n\tfor i := uint64(0); i < expected.Len(); i++ {\n\t\tv := actual.Get(i)\n\t\tassert.True(expected.Get(i).Equals(v))\n\n\t\texpectedCount += isEncodedOutOfLine(v)\n\t\tassert.Equal(expectedCount+chunkReads[i], cs.Reads)\n\n\t\t// Do it again to make sure multiple derefs don't do multiple loads.\n\t\t_ = actual.Get(i)\n\t\tassert.Equal(expectedCount+chunkReads[i], cs.Reads)\n\t}\n}\n\nfunc SkipTestIncrementalLoadSet(t *testing.T) {\n\tassert := assert.New(t)\n\tts := &chunks.TestStorage{}\n\tcs := ts.NewView()\n\tvs := NewValueStore(cs)\n\n\texpected := NewSet(vs, getTestVals(vs)...)\n\tref := vs.WriteValue(expected).TargetHash()\n\n\tactualVar := vs.ReadValue(ref)\n\tactual := actualVar.(Set)\n\n\texpectedCount := cs.Reads\n\tassert.Equal(1, expectedCount)\n\tactual.Iter(func(v Value) (stop bool) {\n\t\texpectedCount += isEncodedOutOfLine(v)\n\t\tassert.Equal(expectedCount, cs.Reads)\n\t\treturn\n\t})\n}\n\nfunc SkipTestIncrementalLoadMap(t *testing.T) {\n\tassert := assert.New(t)\n\tts := &chunks.TestStorage{}\n\tcs := ts.NewView()\n\tvs := NewValueStore(cs)\n\n\texpected := NewMap(vs, getTestVals(vs)...)\n\tref := vs.WriteValue(expected).TargetHash()\n\n\tactualVar := vs.ReadValue(ref)\n\tactual := actualVar.(Map)\n\n\texpectedCount := cs.Reads\n\tassert.Equal(1, expectedCount)\n\tactual.Iter(func(k, v Value) (stop bool) {\n\t\texpectedCount += isEncodedOutOfLine(k)\n\t\texpectedCount += isEncodedOutOfLine(v)\n\t\tassert.Equal(expectedCount, cs.Reads)\n\t\treturn\n\t})\n}\n\nfunc SkipTestIncrementalAddRef(t *testing.T) {\n\tassert := assert.New(t)\n\tts := &chunks.TestStorage{}\n\tcs := ts.NewView()\n\tvs := NewValueStore(cs)\n\n\texpectedItem := Number(42)\n\tref := vs.WriteValue(expectedItem)\n\n\texpected := NewList(vs, ref)\n\tref = vs.WriteValue(expected)\n\tactualVar := vs.ReadValue(ref.TargetHash())\n\n\tassert.Equal(1, cs.Reads)\n\tassert.True(expected.Equals(actualVar))\n\n\tactual := actualVar.(List)\n\tactualItem := actual.Get(0)\n\tassert.Equal(2, cs.Reads)\n\tassert.True(expectedItem.Equals(actualItem))\n\n\t// do it again to make sure caching works.\n\tactualItem = actual.Get(0)\n\tassert.Equal(2, cs.Reads)\n\tassert.True(expectedItem.Equals(actualItem))\n}\n" }, { "path": "go/types/indexed_sequence_diff.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nfunc sendSpliceChange(changes chan<- Splice, closeChan <-chan struct{}, splice Splice) bool {\n\tselect {\n\tcase changes <- splice:\n\tcase <-closeChan:\n\t\treturn false\n\t}\n\treturn true\n}\n\nfunc indexedSequenceDiff(last sequence, lastOffset uint64, current sequence, currentOffset uint64, changes chan<- Splice, closeChan <-chan struct{}, maxSpliceMatrixSize uint64) bool {\n\tif last.treeLevel() > current.treeLevel() {\n\t\tlastChild := last.getCompositeChildSequence(0, uint64(last.seqLen()))\n\t\treturn indexedSequenceDiff(lastChild, lastOffset, current, currentOffset, changes, closeChan, maxSpliceMatrixSize)\n\t}\n\n\tif current.treeLevel() > last.treeLevel() {\n\t\tcurrentChild := current.getCompositeChildSequence(0, uint64(current.seqLen()))\n\t\treturn indexedSequenceDiff(last, lastOffset, currentChild, currentOffset, changes, closeChan, maxSpliceMatrixSize)\n\t}\n\n\tcompareFn := last.getCompareFn(current)\n\tinitialSplices := calcSplices(uint64(last.seqLen()), uint64(current.seqLen()), maxSpliceMatrixSize,\n\t\tfunc(i uint64, j uint64) bool { return compareFn(int(i), int(j)) })\n\n\tfor _, splice := range initialSplices {\n\t\tif last.isLeaf() {\n\t\t\t// This is a leaf sequence, we can just report the splice, but it's indices must be offset.\n\t\t\tsplice.SpAt += lastOffset\n\t\t\tif splice.SpAdded > 0 {\n\t\t\t\tsplice.SpFrom += currentOffset\n\t\t\t}\n\n\t\t\tif !sendSpliceChange(changes, closeChan, splice) {\n\t\t\t\treturn false\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tif splice.SpRemoved == 0 || splice.SpAdded == 0 {\n\t\t\t// An entire subtree was removed at a meta level. We must do some math to map the splice from the meta level into the leaf coordinates.\n\t\t\tbeginRemoveIndex := uint64(0)\n\t\t\tif splice.SpAt > 0 {\n\t\t\t\tbeginRemoveIndex = last.cumulativeNumberOfLeaves(int(splice.SpAt) - 1)\n\t\t\t}\n\t\t\tendRemoveIndex := uint64(0)\n\t\t\tif splice.SpAt+splice.SpRemoved > 0 {\n\t\t\t\tendRemoveIndex = last.cumulativeNumberOfLeaves(int(splice.SpAt+splice.SpRemoved) - 1)\n\t\t\t}\n\t\t\tbeginAddIndex := uint64(0)\n\t\t\tif splice.SpFrom > 0 {\n\t\t\t\tbeginAddIndex = current.cumulativeNumberOfLeaves(int(splice.SpFrom) - 1)\n\t\t\t}\n\t\t\tendAddIndex := uint64(0)\n\t\t\tif splice.SpFrom+splice.SpAdded > 0 {\n\t\t\t\tendAddIndex = current.cumulativeNumberOfLeaves(int(splice.SpFrom+splice.SpAdded) - 1)\n\t\t\t}\n\n\t\t\tsplice.SpAt = lastOffset + beginRemoveIndex\n\t\t\tsplice.SpRemoved = endRemoveIndex - beginRemoveIndex\n\n\t\t\tsplice.SpAdded = endAddIndex - beginAddIndex\n\t\t\tif splice.SpAdded > 0 {\n\t\t\t\tsplice.SpFrom = currentOffset + beginAddIndex\n\t\t\t}\n\n\t\t\tif !sendSpliceChange(changes, closeChan, splice) {\n\t\t\t\treturn false\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\t// Meta sequence splice which includes removed & added sub-sequences. Must recurse down.\n\t\tlastChild := last.getCompositeChildSequence(splice.SpAt, splice.SpRemoved)\n\t\tcurrentChild := current.getCompositeChildSequence(splice.SpFrom, splice.SpAdded)\n\t\tlastChildOffset := lastOffset\n\t\tif splice.SpAt > 0 {\n\t\t\tlastChildOffset += last.cumulativeNumberOfLeaves(int(splice.SpAt) - 1)\n\t\t}\n\t\tcurrentChildOffset := currentOffset\n\t\tif splice.SpFrom > 0 {\n\t\t\tcurrentChildOffset += current.cumulativeNumberOfLeaves(int(splice.SpFrom) - 1)\n\t\t}\n\t\tif ok := indexedSequenceDiff(lastChild, lastChildOffset, currentChild, currentChildOffset, changes, closeChan, maxSpliceMatrixSize); !ok {\n\t\t\treturn false\n\t\t}\n\t}\n\n\treturn true\n}\n" }, { "path": "go/types/indexed_sequences.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\nfunc newListMetaSequence(level uint64, tuples []metaTuple, vrw ValueReadWriter) metaSequence {\n\treturn newMetaSequenceFromTuples(ListKind, level, tuples, vrw)\n}\n\nfunc newBlobMetaSequence(level uint64, tuples []metaTuple, vrw ValueReadWriter) metaSequence {\n\treturn newMetaSequenceFromTuples(BlobKind, level, tuples, vrw)\n}\n\n// advanceCursorToOffset advances the cursor as close as possible to idx\n//\n// If the cursor references a leaf sequence,\n// \tadvance to idx,\n// \tand return the number of values preceding the idx\n// If it references a meta-sequence,\n// \tadvance to the tuple containing idx,\n// \tand return the number of leaf values preceding this tuple\nfunc advanceCursorToOffset(cur *sequenceCursor, idx uint64) uint64 {\n\tseq := cur.seq\n\n\tif ms, ok := seq.(metaSequence); ok {\n\t\t// For a meta sequence, advance the cursor to the smallest position where idx < seq.cumulativeNumLeaves()\n\t\tcur.idx = 0\n\t\tcum := uint64(0)\n\n\t\tseqLen := ms.seqLen()\n\t\t// Advance the cursor to the meta-sequence tuple containing idx\n\t\tfor cur.idx < seqLen-1 {\n\t\t\tnumLeaves := ms.getNumLeavesAt(cur.idx)\n\t\t\tif uint64(idx) >= cum+numLeaves {\n\t\t\t\tcum += numLeaves\n\t\t\t\tcur.idx++\n\t\t\t} else {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\n\t\treturn cum // number of leaves sequences BEFORE cur.idx in meta sequence\n\t}\n\n\tseqLen := seq.seqLen()\n\tcur.idx = int(idx)\n\tif cur.idx > seqLen {\n\t\tcur.idx = seqLen\n\t}\n\treturn uint64(cur.idx)\n}\n\nfunc newIndexedMetaSequenceChunkFn(kind NomsKind, vrw ValueReadWriter) makeChunkFn {\n\treturn func(level uint64, items []sequenceItem) (Collection, orderedKey, uint64) {\n\t\ttuples := make([]metaTuple, len(items))\n\t\tnumLeaves := uint64(0)\n\n\t\tfor i, v := range items {\n\t\t\tmt := v.(metaTuple)\n\t\t\ttuples[i] = mt\n\t\t\tnumLeaves += mt.numLeaves()\n\t\t}\n\n\t\tvar col Collection\n\t\tif kind == ListKind {\n\t\t\tcol = newList(newListMetaSequence(level, tuples, vrw))\n\t\t} else {\n\t\t\td.PanicIfFalse(BlobKind == kind)\n\t\t\tcol = newBlob(newBlobMetaSequence(level, tuples, vrw))\n\t\t}\n\t\treturn col, orderedKeyFromSum(tuples), numLeaves\n\t}\n}\n\nfunc orderedKeyFromSum(msd []metaTuple) orderedKey {\n\tsum := uint64(0)\n\tfor _, mt := range msd {\n\t\tsum += mt.numLeaves()\n\t}\n\treturn orderedKeyFromUint64(sum)\n}\n\n// LoadLeafNodes loads the set of leaf nodes which contain the items\n// [startIdx -> endIdx). Returns the set of nodes and the offset within\n// the first sequence which corresponds to |startIdx|.\nfunc LoadLeafNodes(cols []Collection, startIdx, endIdx uint64) ([]Collection, uint64) {\n\tvrw := cols[0].asSequence().valueReadWriter()\n\td.PanicIfTrue(vrw == nil)\n\n\tif cols[0].asSequence().isLeaf() {\n\t\tfor _, c := range cols {\n\t\t\td.PanicIfFalse(c.asSequence().isLeaf())\n\t\t}\n\n\t\treturn cols, startIdx\n\t}\n\n\tlevel := cols[0].asSequence().treeLevel()\n\tchildTuples := []metaTuple{}\n\n\tcum := uint64(0)\n\tfor _, c := range cols {\n\t\ts := c.asSequence()\n\t\td.PanicIfFalse(s.treeLevel() == level)\n\t\tms := s.(metaSequence)\n\n\t\tfor _, mt := range ms.tuples() {\n\t\t\tnumLeaves := mt.numLeaves()\n\t\t\tif cum == 0 && numLeaves <= startIdx {\n\t\t\t\t// skip tuples whose items are < startIdx\n\t\t\t\tstartIdx -= numLeaves\n\t\t\t\tendIdx -= numLeaves\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tchildTuples = append(childTuples, mt)\n\t\t\tcum += numLeaves\n\t\t\tif cum >= endIdx {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}\n\n\ths := make(hash.HashSlice, len(childTuples))\n\tfor i, mt := range childTuples {\n\t\ths[i] = mt.ref().TargetHash()\n\t}\n\n\t// Fetch committed child sequences in a single batch\n\treadValues := vrw.ReadManyValues(hs)\n\n\tchildCols := make([]Collection, len(readValues))\n\tfor i, v := range readValues {\n\t\tchildCols[i] = v.(Collection)\n\t}\n\n\treturn LoadLeafNodes(childCols, startIdx, endIdx)\n}\n" }, { "path": "go/types/leaf_sequence.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"math\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype leafSequence struct {\n\tsequenceImpl\n}\n\nfunc newLeafSequence(vrw ValueReadWriter, buff []byte, offsets []uint32, len uint64) leafSequence {\n\treturn leafSequence{newSequenceImpl(vrw, buff, offsets, len)}\n}\n\nfunc newLeafSequenceFromValues(kind NomsKind, vrw ValueReadWriter, vs ...Value) leafSequence {\n\td.PanicIfTrue(vrw == nil)\n\tw := newBinaryNomsWriter()\n\toffsets := make([]uint32, len(vs)+sequencePartValues+1)\n\toffsets[sequencePartKind] = w.offset\n\tkind.writeTo(&w)\n\toffsets[sequencePartLevel] = w.offset\n\tw.writeCount(0) // level\n\toffsets[sequencePartCount] = w.offset\n\tcount := uint64(len(vs))\n\tw.writeCount(count)\n\toffsets[sequencePartValues] = w.offset\n\tfor i, v := range vs {\n\t\tv.writeTo(&w)\n\t\toffsets[i+sequencePartValues+1] = w.offset\n\t}\n\treturn newLeafSequence(vrw, w.data(), offsets, count)\n}\n\n// readLeafSequence reads the data provided by a decoder and moves the decoder forward.\nfunc readLeafSequence(dec *valueDecoder) leafSequence {\n\tstart := dec.pos()\n\toffsets, seqLen := skipLeafSequence(dec)\n\tend := dec.pos()\n\treturn newLeafSequence(dec.vrw, dec.byteSlice(start, end), offsets, seqLen)\n}\n\nfunc skipLeafSequence(dec *valueDecoder) ([]uint32, uint64) {\n\tkindPos := dec.pos()\n\tdec.skipKind()\n\tlevelPos := dec.pos()\n\tdec.skipCount() // level\n\tcountPos := dec.pos()\n\tcount := dec.readCount()\n\toffsets := make([]uint32, count+sequencePartValues+1)\n\toffsets[sequencePartKind] = kindPos\n\toffsets[sequencePartLevel] = levelPos\n\toffsets[sequencePartCount] = countPos\n\toffsets[sequencePartValues] = dec.pos()\n\tfor i := uint64(0); i < count; i++ {\n\t\tdec.skipValue()\n\t\toffsets[i+sequencePartValues+1] = dec.pos()\n\t}\n\treturn offsets, count\n}\n\nfunc (seq leafSequence) values() []Value {\n\treturn seq.valuesSlice(0, math.MaxUint64)\n}\n\nfunc (seq leafSequence) valuesSlice(from, to uint64) []Value {\n\tif len := seq.Len(); to > len {\n\t\tto = len\n\t}\n\n\tdec := seq.decoderSkipToIndex(int(from))\n\tvs := make([]Value, (to-from)*uint64(getValuesPerIdx(seq.Kind())))\n\tfor i := range vs {\n\t\tvs[i] = dec.readValue()\n\t}\n\treturn vs\n}\n\nfunc (seq leafSequence) getCompareFnHelper(other leafSequence) compareFn {\n\tdec := seq.decoder()\n\totherDec := other.decoder()\n\n\treturn func(idx, otherIdx int) bool {\n\t\tdec.offset = uint32(seq.getItemOffset(idx))\n\t\totherDec.offset = uint32(other.getItemOffset(otherIdx))\n\t\treturn dec.readValue().Equals(otherDec.readValue())\n\t}\n}\n\nfunc (seq leafSequence) getCompareFn(other sequence) compareFn {\n\tpanic(\"unreachable\")\n}\n\nfunc (seq leafSequence) typeOf() *Type {\n\tdec := seq.decoder()\n\tkind := dec.readKind()\n\tdec.skipCount() // level\n\tcount := dec.readCount()\n\tts := make(typeSlice, 0, count)\n\tvar lastType *Type\n\tfor i := uint64(0); i < count; i++ {\n\t\tif lastType != nil {\n\t\t\toffset := dec.offset\n\t\t\tif dec.isValueSameTypeForSure(lastType) {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tdec.offset = offset\n\t\t}\n\n\t\tlastType = dec.readTypeOfValue()\n\t\tts = append(ts, lastType)\n\t}\n\n\treturn makeCompoundType(kind, makeUnionType(ts...))\n}\n\nfunc (seq leafSequence) numLeaves() uint64 {\n\treturn seq.len\n}\n\nfunc (seq leafSequence) getChildSequence(idx int) sequence {\n\treturn nil\n}\n\nfunc (seq leafSequence) treeLevel() uint64 {\n\treturn 0\n}\n\nfunc (seq leafSequence) isLeaf() bool {\n\treturn true\n}\n\nfunc (seq leafSequence) cumulativeNumberOfLeaves(idx int) uint64 {\n\treturn uint64(idx) + 1\n}\n\nfunc (seq leafSequence) getCompositeChildSequence(start uint64, length uint64) sequence {\n\tpanic(\"getCompositeChildSequence called on a leaf sequence\")\n}\n\nfunc (seq leafSequence) getItem(idx int) sequenceItem {\n\tdec := seq.decoderSkipToIndex(idx)\n\treturn dec.readValue()\n}\n\nfunc getValuesPerIdx(kind NomsKind) int {\n\tif kind == MapKind {\n\t\treturn 2\n\t}\n\treturn 1\n}\n" }, { "path": "go/types/less.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype kindAndHash interface {\n\tKind() NomsKind\n\tHash() hash.Hash\n}\n\nfunc valueLess(v1, v2 kindAndHash) bool {\n\tswitch v2.Kind() {\n\tcase BoolKind, NumberKind, StringKind:\n\t\treturn false\n\tdefault:\n\t\treturn v1.Hash().Less(v2.Hash())\n\t}\n}\n" }, { "path": "go/types/list.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sync/atomic\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// List represents a list or an array of Noms values. A list can contain zero or more values of zero\n// or more types. The type of the list will reflect the type of the elements in the list. For\n// example:\n//\n// l := NewList(Number(1), Bool(true))\n// fmt.Println(l.Type().Describe())\n// // outputs List\n//\n// Lists, like all Noms values are immutable so the \"mutation\" methods return a new list.\ntype List struct {\n\tsequence\n}\n\nfunc newList(seq sequence) List {\n\treturn List{seq}\n}\n\n// NewList creates a new List where the type is computed from the elements in the list, populated\n// with values, chunking if and when needed.\nfunc NewList(vrw ValueReadWriter, values ...Value) List {\n\tch := newEmptyListSequenceChunker(vrw)\n\tfor _, v := range values {\n\t\tch.Append(v)\n\t}\n\treturn newList(ch.Done())\n}\n\n// NewStreamingList creates a new List, populated with values, chunking if and when needed. As\n// chunks are created, they're written to vrw -- including the root chunk of the list. Once the\n// caller has closed values, the caller can read the completed List from the returned channel.\nfunc NewStreamingList(vrw ValueReadWriter, values <-chan Value) <-chan List {\n\tout := make(chan List, 1)\n\tgo func() {\n\t\tdefer close(out)\n\t\tch := newEmptyListSequenceChunker(vrw)\n\t\tfor v := range values {\n\t\t\tch.Append(v)\n\t\t}\n\t\tout <- newList(ch.Done())\n\t}()\n\treturn out\n}\n\nfunc (l List) Edit() *ListEditor {\n\treturn NewListEditor(l)\n}\n\n// Collection interface\n\nfunc (l List) asSequence() sequence {\n\treturn l.sequence\n}\n\n// Value interface\nfunc (l List) Value() Value {\n\treturn l\n}\n\nfunc (l List) WalkValues(cb ValueCallback) {\n\titerAll(l, func(v Value, idx uint64) {\n\t\tcb(v)\n\t})\n}\n\n// Get returns the value at the given index. If this list has been chunked then this will have to\n// descend into the prolly-tree which leads to Get being O(depth).\nfunc (l List) Get(idx uint64) Value {\n\td.PanicIfFalse(idx < l.Len())\n\tcur := newCursorAtIndex(l.sequence, idx)\n\treturn cur.current().(Value)\n}\n\n// Concat returns a new List comprised of this joined with other. It only needs\n// to visit the rightmost prolly tree chunks of this List, and the leftmost\n// prolly tree chunks of other, so it's efficient.\nfunc (l List) Concat(other List) List {\n\tseq := concat(l.sequence, other.sequence, func(cur *sequenceCursor, vrw ValueReadWriter) *sequenceChunker {\n\t\treturn l.newChunker(cur, vrw)\n\t})\n\treturn newList(seq)\n}\n\n// Iter iterates over the list and calls f for every element in the list. If f returns true then the\n// iteration stops.\nfunc (l List) Iter(f func(v Value, index uint64) (stop bool)) {\n\tidx := uint64(0)\n\tcur := newCursorAtIndex(l.sequence, idx)\n\tcur.iter(func(v interface{}) bool {\n\t\tif f(v.(Value), uint64(idx)) {\n\t\t\treturn true\n\t\t}\n\t\tidx++\n\t\treturn false\n\t})\n}\n\nfunc (l List) IterRange(startIdx, endIdx uint64, f func(v Value, idx uint64)) {\n\tidx := uint64(startIdx)\n\tcb := func(v Value) {\n\t\tf(v, idx)\n\t\tidx++\n\t}\n\titerRange(l, startIdx, endIdx, cb)\n}\n\n// IterAll iterates over the list and calls f for every element in the list. Unlike Iter there is no\n// way to stop the iteration and all elements are visited.\nfunc (l List) IterAll(f func(v Value, index uint64)) {\n\titerAll(l, f)\n}\n\nfunc iterAll(col Collection, f func(v Value, index uint64)) {\n\tconcurrency := 6\n\tvcChan := make(chan chan Value, concurrency)\n\n\t// Target reading data in |targetBatchBytes| per thread. We don't know how\n\t// many bytes each value is, so update |estimatedNumValues| as data is read.\n\ttargetBatchBytes := 1 << 23 // 8MB\n\testimatedNumValues := uint64(1000)\n\n\tgo func() {\n\t\tfor idx, l := uint64(0), col.Len(); idx < l; {\n\t\t\tnumValues := atomic.LoadUint64(&estimatedNumValues)\n\n\t\t\tstart := idx\n\t\t\tblockLength := l - start\n\t\t\tif blockLength > numValues {\n\t\t\t\tblockLength = numValues\n\t\t\t}\n\t\t\tidx += blockLength\n\n\t\t\tvc := make(chan Value)\n\t\t\tvcChan <- vc\n\n\t\t\tgo func() {\n\t\t\t\tnumBytes := iterRange(col, start, start+blockLength, func(v Value) {\n\t\t\t\t\tvc <- v\n\t\t\t\t})\n\t\t\t\tclose(vc)\n\n\t\t\t\t// Adjust the estimated number of values to try to read\n\t\t\t\t// |targetBatchBytes| next time.\n\t\t\t\tif numValues == blockLength {\n\t\t\t\t\tscale := float64(targetBatchBytes) / float64(numBytes)\n\t\t\t\t\tatomic.StoreUint64(&estimatedNumValues, uint64(float64(numValues)*scale))\n\t\t\t\t}\n\t\t\t}()\n\t\t}\n\t\tclose(vcChan)\n\t}()\n\n\t// Ensure read-ahead goroutines can exit, because the `range` below might not\n\t// finish if an |f| callback panics.\n\tdefer func() {\n\t\tfor vc := range vcChan {\n\t\t\tclose(vc)\n\t\t}\n\t}()\n\n\ti := uint64(0)\n\tfor vc := range vcChan {\n\t\tfor v := range vc {\n\t\t\tf(v, i)\n\t\t\ti++\n\t\t}\n\t}\n}\n\nfunc iterRange(col Collection, startIdx, endIdx uint64, cb func(v Value)) (numBytes uint64) {\n\tl := col.Len()\n\td.PanicIfTrue(startIdx > endIdx || endIdx > l)\n\tif startIdx == endIdx {\n\t\treturn\n\t}\n\n\tleaves, localStart := LoadLeafNodes([]Collection{col}, startIdx, endIdx)\n\tendIdx = localStart + endIdx - startIdx\n\tstartIdx = localStart\n\tnumValues := 0\n\tvaluesPerIdx := uint64(getValuesPerIdx(col.Kind()))\n\n\tfor _, leaf := range leaves {\n\t\tseq := leaf.asSequence()\n\t\tvalues := seq.valuesSlice(startIdx, endIdx)\n\t\tnumValues += len(values)\n\n\t\tfor _, v := range values {\n\t\t\tcb(v)\n\t\t}\n\n\t\tendIdx = endIdx - uint64(len(values))/valuesPerIdx - startIdx\n\t\tstartIdx = 0\n\t\tnumBytes += uint64(len(seq.valueBytes())) // note: should really only include |values|\n\t}\n\treturn\n}\n\n// Iterator returns a ListIterator which can be used to iterate efficiently over a list.\nfunc (l List) Iterator() ListIterator {\n\treturn l.IteratorAt(0)\n}\n\n// IteratorAt returns a ListIterator starting at index. If index is out of bound the iterator will\n// have reached its end on creation.\nfunc (l List) IteratorAt(index uint64) ListIterator {\n\treturn ListIterator{\n\t\tnewCursorAtIndex(l.sequence, index),\n\t}\n}\n\n// Diff streams the diff from last to the current list to the changes channel. Caller can close\n// closeChan to cancel the diff operation.\nfunc (l List) Diff(last List, changes chan<- Splice, closeChan <-chan struct{}) {\n\tl.DiffWithLimit(last, changes, closeChan, DEFAULT_MAX_SPLICE_MATRIX_SIZE)\n}\n\n// DiffWithLimit streams the diff from last to the current list to the changes channel. Caller can\n// close closeChan to cancel the diff operation.\n// The maxSpliceMatrixSize determines the how big of an edit distance matrix we are willing to\n// compute versus just saying the thing changed.\nfunc (l List) DiffWithLimit(last List, changes chan<- Splice, closeChan <-chan struct{}, maxSpliceMatrixSize uint64) {\n\tif l.Equals(last) {\n\t\treturn\n\t}\n\tlLen, lastLen := l.Len(), last.Len()\n\tif lLen == 0 {\n\t\tchanges <- Splice{0, lastLen, 0, 0} // everything removed\n\t\treturn\n\t}\n\tif lastLen == 0 {\n\t\tchanges <- Splice{0, 0, lLen, 0} // everything added\n\t\treturn\n\t}\n\n\tindexedSequenceDiff(last.sequence, 0, l.sequence, 0, changes, closeChan, maxSpliceMatrixSize)\n}\n\nfunc (l List) newChunker(cur *sequenceCursor, vrw ValueReadWriter) *sequenceChunker {\n\treturn newSequenceChunker(cur, 0, vrw, makeListLeafChunkFn(vrw), newIndexedMetaSequenceChunkFn(ListKind, vrw), hashValueBytes)\n}\n\nfunc makeListLeafChunkFn(vrw ValueReadWriter) makeChunkFn {\n\treturn func(level uint64, items []sequenceItem) (Collection, orderedKey, uint64) {\n\t\td.PanicIfFalse(level == 0)\n\t\tvalues := make([]Value, len(items))\n\n\t\tfor i, v := range items {\n\t\t\tvalues[i] = v.(Value)\n\t\t}\n\n\t\tlist := newList(newListLeafSequence(vrw, values...))\n\t\treturn list, orderedKeyFromInt(len(values)), uint64(len(values))\n\t}\n}\n\nfunc newEmptyListSequenceChunker(vrw ValueReadWriter) *sequenceChunker {\n\treturn newEmptySequenceChunker(vrw, makeListLeafChunkFn(vrw), newIndexedMetaSequenceChunkFn(ListKind, vrw), hashValueBytes)\n}\n" }, { "path": "go/types/list_editor.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype ListEditor struct {\n\tl List\n\tedits *listEdit\n}\n\nfunc NewListEditor(l List) *ListEditor {\n\treturn &ListEditor{l, nil}\n}\n\nfunc (le *ListEditor) Kind() NomsKind {\n\treturn ListKind\n}\n\nfunc (le *ListEditor) Value() Value {\n\treturn le.List()\n}\n\nfunc (le *ListEditor) List() List {\n\tif le.edits == nil {\n\t\treturn le.l // no edits\n\t}\n\n\tseq := le.l.sequence\n\tvrw := seq.valueReadWriter()\n\n\tcursChan := make(chan chan *sequenceCursor)\n\tspliceChan := make(chan chan listEdit)\n\n\tgo func() {\n\t\tfor edit := le.edits; edit != nil; edit = edit.next {\n\t\t\tedit := edit\n\n\t\t\t// TODO: Use ReadMany\n\t\t\tcc := make(chan *sequenceCursor, 1)\n\t\t\tcursChan <- cc\n\n\t\t\tgo func() {\n\t\t\t\tcc <- newCursorAtIndex(seq, edit.idx)\n\t\t\t}()\n\n\t\t\tsc := make(chan listEdit, 1)\n\t\t\tspliceChan <- sc\n\n\t\t\twg := sync.WaitGroup{}\n\t\t\tsubEditors := false\n\t\t\tfor i, v := range edit.inserted {\n\t\t\t\tif _, ok := v.(Value); ok {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\tsubEditors = true\n\t\t\t\tidx, val := i, v\n\t\t\t\twg.Add(1)\n\t\t\t\tgo func() {\n\t\t\t\t\tedit.inserted[idx] = val.Value()\n\t\t\t\t\twg.Done()\n\t\t\t\t}()\n\t\t\t}\n\t\t\tif !subEditors {\n\t\t\t\tsc <- *edit\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tgo func() {\n\t\t\t\twg.Wait()\n\t\t\t\tsc <- *edit\n\t\t\t}()\n\t\t}\n\n\t\tclose(cursChan)\n\t\tclose(spliceChan)\n\t}()\n\n\tvar ch *sequenceChunker\n\tfor cc := range cursChan {\n\t\tcur := <-cc\n\t\tsp := <-<-spliceChan\n\n\t\tif ch == nil {\n\t\t\tch = newSequenceChunker(cur, 0, vrw, makeListLeafChunkFn(vrw), newIndexedMetaSequenceChunkFn(ListKind, vrw), hashValueBytes)\n\t\t} else {\n\t\t\tch.advanceTo(cur)\n\t\t}\n\n\t\tdc := sp.removed\n\t\tfor dc > 0 {\n\t\t\tch.Skip()\n\t\t\tdc--\n\t\t}\n\n\t\tfor _, v := range sp.inserted {\n\t\t\tch.Append(v)\n\t\t}\n\t}\n\n\treturn newList(ch.Done())\n}\n\nfunc collapseListEdit(newEdit, edit *listEdit) bool {\n\tif newEdit.idx+newEdit.removed < edit.idx ||\n\t\tedit.idx+uint64(len(edit.inserted)) < newEdit.idx {\n\t\treturn false\n\t}\n\n\tcollapsed := &listEdit{}\n\n\tif newEdit.idx <= edit.idx {\n\t\tcollapsed.idx = newEdit.idx\n\n\t\toverlap := newEdit.removed - (edit.idx - newEdit.idx) // number of leading N values removed from edit.inserted\n\t\tif overlap < uint64(len(edit.inserted)) {\n\t\t\t// newEdit doesn't remove all of edit.inserted\n\t\t\tcollapsed.inserted = append(newEdit.inserted, edit.inserted[overlap:]...)\n\t\t\tcollapsed.removed = newEdit.removed + edit.removed - overlap\n\t\t} else {\n\t\t\t// newEdit removes all of edit.inserted\n\t\t\tcollapsed.inserted = newEdit.inserted\n\t\t\tcollapsed.removed = newEdit.removed + edit.removed - uint64(len(edit.inserted))\n\t\t}\n\t} else {\n\t\t// edit.idx < newEdit.idx\n\n\t\tcollapsed.idx = edit.idx\n\n\t\teditInsertedLen := uint64(len(edit.inserted))\n\t\tbeginEditRemovePoint := newEdit.idx - edit.idx\n\n\t\tif beginEditRemovePoint == editInsertedLen {\n\t\t\t// newEdit took place at the position immediately after the last element of edit.inserted\n\t\t\tcollapsed.inserted = append(edit.inserted, newEdit.inserted...)\n\t\t\tcollapsed.removed = edit.removed + newEdit.removed\n\t\t} else {\n\t\t\t// newEdit takes place within edit.inserted\n\t\t\tcollapsed.inserted = append(collapsed.inserted, edit.inserted[:beginEditRemovePoint]...)\n\t\t\tcollapsed.inserted = append(collapsed.inserted, newEdit.inserted...)\n\n\t\t\tendEditRemovePoint := beginEditRemovePoint + newEdit.removed\n\t\t\tif endEditRemovePoint < editInsertedLen {\n\t\t\t\t// elements of edit.inserted remain beyond newEdit.removed\n\t\t\t\tcollapsed.removed = edit.removed\n\t\t\t\tcollapsed.inserted = append(collapsed.inserted, edit.inserted[endEditRemovePoint:]...)\n\t\t\t} else {\n\t\t\t\tcollapsed.removed = edit.removed + endEditRemovePoint - editInsertedLen\n\t\t\t}\n\t\t}\n\t}\n\n\t*newEdit = *collapsed\n\treturn true\n}\n\nfunc (le *ListEditor) Len() uint64 {\n\tdelta := int64(0)\n\tfor edit := le.edits; edit != nil; edit = edit.next {\n\t\tdelta += -int64(edit.removed) + int64(len(edit.inserted))\n\t}\n\n\treturn uint64(int64(le.l.Len()) + delta)\n}\n\nfunc (le *ListEditor) Splice(idx uint64, deleteCount uint64, vs ...Valuable) *ListEditor {\n\tfor _, sv := range vs {\n\t\td.PanicIfTrue(sv == nil)\n\t}\n\n\tne := &listEdit{idx, deleteCount, vs, nil}\n\n\tvar last *listEdit\n\tedit := le.edits\n\n\tfor edit != nil {\n\t\tif collapseListEdit(ne, edit) {\n\t\t\tif last == nil {\n\t\t\t\tle.edits = edit.next\n\t\t\t} else {\n\t\t\t\tlast.next = edit.next\n\t\t\t}\n\n\t\t\tedit = edit.next\n\t\t\tcontinue\n\t\t}\n\n\t\tif edit.idx > ne.idx {\n\t\t\tbreak\n\t\t}\n\n\t\tne.idx = adjustIdx(ne.idx, edit)\n\t\tlast = edit\n\t\tedit = edit.next\n\t}\n\n\tif ne.removed == 0 && len(ne.inserted) == 0 {\n\t\treturn le // effectively removed 1 or more existing slices\n\t}\n\n\tif ne.idx > le.l.Len() {\n\t\td.Panic(\"Index Out Of Bounds\")\n\t}\n\tif ne.idx == le.l.Len() && ne.removed > 0 {\n\t\td.Panic(\"Index Out Of Bounds\")\n\t}\n\n\tif last == nil {\n\t\t// Insert |ne| in first position\n\t\tne.next = le.edits\n\t\tle.edits = ne\n\t} else {\n\t\tne.next = last.next\n\t\tlast.next = ne\n\t}\n\n\treturn le\n}\n\nfunc (le *ListEditor) Set(idx uint64, v Valuable) *ListEditor {\n\treturn le.Splice(idx, 1, v)\n}\n\nfunc (le *ListEditor) Append(vs ...Valuable) *ListEditor {\n\treturn le.Splice(le.Len(), 0, vs...)\n}\n\nfunc (le *ListEditor) Insert(idx uint64, vs ...Valuable) *ListEditor {\n\treturn le.Splice(idx, 0, vs...)\n}\n\nfunc (le *ListEditor) Remove(start uint64, end uint64) *ListEditor {\n\td.PanicIfFalse(start <= end)\n\treturn le.Splice(start, end-start)\n}\n\nfunc (le *ListEditor) RemoveAt(idx uint64) *ListEditor {\n\treturn le.Splice(idx, 1)\n}\n\nfunc adjustIdx(idx uint64, e *listEdit) uint64 {\n\treturn idx + e.removed - uint64(len(e.inserted))\n}\n\nfunc (le *ListEditor) Get(idx uint64) Valuable {\n\tedit := le.edits\n\tfor edit != nil {\n\t\tif edit.idx > idx {\n\t\t\t// idx is before next splice\n\t\t\treturn le.l.Get(idx)\n\t\t}\n\n\t\tif edit.idx <= idx && idx < (edit.idx+uint64(len(edit.inserted))) {\n\t\t\t// idx is within the insert values of edit\n\t\t\treturn edit.inserted[idx-edit.idx]\n\t\t}\n\n\t\tidx = adjustIdx(idx, edit)\n\t\tedit = edit.next\n\t}\n\n\treturn le.l.Get(idx)\n}\n\ntype listEdit struct {\n\tidx uint64\n\tremoved uint64\n\tinserted []Valuable\n\tnext *listEdit\n}\n" }, { "path": "go/types/list_editor_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"math/rand\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc listOfInts(vrw ValueReadWriter, vals ...int) List {\n\tvs := ValueSlice{}\n\tfor _, v := range vals {\n\t\tvs = append(vs, Number(v))\n\t}\n\treturn NewList(vrw, vs...)\n}\n\nfunc testEditor(vrw ValueReadWriter, vals ...int) *ListEditor {\n\treturn NewListEditor(listOfInts(vrw, vals...))\n}\n\nfunc edit(le *ListEditor, idx, remove int, insert ...int) {\n\tvals := []Valuable{}\n\tfor _, v := range insert {\n\t\tvals = append(vals, Number(v))\n\t}\n\tle.Splice(uint64(idx), uint64(remove), vals...)\n}\n\nfunc assertState(t *testing.T, vrw ValueReadWriter, le *ListEditor, expectItems []int, expectEditCount int) {\n\tassert.Equal(t, uint64(len(expectItems)), le.Len())\n\n\tfor i, v := range expectItems {\n\t\tassert.Equal(t, Number(v), le.Get(uint64(i)))\n\t}\n\n\tactualEditCount := 0\n\tfor edit := le.edits; edit != nil; edit = edit.next {\n\t\tactualEditCount++\n\t}\n\n\tassert.Equal(t, expectEditCount, actualEditCount)\n\n\tassert.True(t, listOfInts(vrw, expectItems...).Equals(le.List()))\n}\n\nfunc TestListEditorBasic(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tt.Run(\"remove a few\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5)\n\t\tedit(le, 2, 2)\n\t\tassertState(t, vrw, le, []int{0, 1, 4, 5}, 1)\n\t})\n\n\tt.Run(\"insert a few\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5)\n\t\tedit(le, 2, 0, 9, 8, 7)\n\t\tassertState(t, vrw, le, []int{0, 1, 9, 8, 7, 2, 3, 4, 5}, 1)\n\t})\n\n\tt.Run(\"remove 2, insert 3\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5)\n\t\tedit(le, 2, 2, 9, 8, 7)\n\t\tassertState(t, vrw, le, []int{0, 1, 9, 8, 7, 4, 5}, 1)\n\t})\n\n\tt.Run(\"insert 2 twice\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5)\n\t\tedit(le, 2, 0, 9, 10)\n\t\tassertState(t, vrw, le, []int{0, 1, 9, 10, 2, 3, 4, 5}, 1)\n\t\tedit(le, 7, 0, 8, 9)\n\t\tassertState(t, vrw, le, []int{0, 1, 9, 10, 2, 3, 4, 8, 9, 5}, 2)\n\t})\n\n\tt.Run(\"remove 2 twice\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5, 6, 7)\n\t\tedit(le, 5, 2)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 3, 4, 7}, 1)\n\t\tedit(le, 1, 2)\n\t\tassertState(t, vrw, le, []int{0, 3, 4, 7}, 2)\n\t})\n}\n\nfunc TestCollapseSplices(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tt.Run(\"left adjacent\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5, 6, 7)\n\t\tedit(le, 4, 3)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 3, 7}, 1)\n\t\tedit(le, 1, 3)\n\t\tassertState(t, vrw, le, []int{0, 7}, 1)\n\t})\n\n\tt.Run(\"left adjacent 2\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5, 6, 7)\n\t\tedit(le, 4, 3, 0, 0)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 3, 0, 0, 7}, 1)\n\t\tedit(le, 1, 3, 5, 5)\n\t\tassertState(t, vrw, le, []int{0, 5, 5, 0, 0, 7}, 1)\n\t})\n\n\tt.Run(\"left consume\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5, 6, 7)\n\t\tedit(le, 2, 4)\n\t\tassertState(t, vrw, le, []int{0, 1, 6, 7}, 1)\n\t\tedit(le, 1, 2)\n\t\tassertState(t, vrw, le, []int{0, 7}, 1)\n\t})\n\n\tt.Run(\"left overlap \", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5)\n\t\tedit(le, 3, 2, 7, 8, 9)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 7, 8, 9, 5}, 1)\n\t\tedit(le, 0, 4)\n\t\tassertState(t, vrw, le, []int{8, 9, 5}, 1)\n\t})\n\n\tt.Run(\"undo 1\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5)\n\t\tedit(le, 2, 3)\n\t\tassertState(t, vrw, le, []int{0, 1, 5}, 1)\n\t\tedit(le, 2, 0, 2, 3, 4)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 3, 4, 5}, 1)\n\t})\n\n\tt.Run(\"undo 2\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5)\n\t\tedit(le, 2, 0, 9, 8, 7)\n\t\tassertState(t, vrw, le, []int{0, 1, 9, 8, 7, 2, 3, 4, 5}, 1)\n\t\tedit(le, 2, 3)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 3, 4, 5}, 0)\n\t})\n\n\tt.Run(\"splice middile of splice\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1)\n\t\tedit(le, 1, 0, 9, 8, 7, 6)\n\t\tassertState(t, vrw, le, []int{0, 9, 8, 7, 6, 1}, 1)\n\t\tedit(le, 2, 2)\n\t\tassertState(t, vrw, le, []int{0, 9, 6, 1}, 1)\n\t})\n}\n\nfunc TestFuzzFails(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tt.Run(\"Case 1\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24)\n\t\tedit(le, 23, 0, 0, 3, 2)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 0, 3, 2, 23, 24}, 1)\n\t\tedit(le, 5, 15, 1, 2, 9, 8)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 3, 4, 1, 2, 9, 8, 20, 21, 22, 0, 3, 2, 23, 24}, 2)\n\t\tedit(le, 4, 7, 7)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 3, 7, 22, 0, 3, 2, 23, 24}, 2)\n\t})\n\n\tt.Run(\"Case 2\", func(t *testing.T) {\n\t\tle := testEditor(vrw, 0, 1, 2, 3, 4, 5)\n\t\tedit(le, 5, 0, 1, 7, 5, 3, 13, 17)\n\t\tassertState(t, vrw, le, []int{0, 1, 2, 3, 4, 1, 7, 5, 3, 13, 17, 5}, 1)\n\t\tedit(le, 2, 2, 16, 5, 12, 5, 15, 0, 15, 15, 7)\n\t\tassertState(t, vrw, le, []int{0, 1, 16, 5, 12, 5, 15, 0, 15, 15, 7, 4, 1, 7, 5, 3, 13, 17, 5}, 2)\n\t\tedit(le, 8, 5, 4, 13)\n\t\tassertState(t, vrw, le, []int{0, 1, 16, 5, 12, 5, 15, 0, 4, 13, 7, 5, 3, 13, 17, 5}, 1)\n\t\tedit(le, 6, 2, 8, 2, 6, 3, 14, 6)\n\t\tassertState(t, vrw, le, []int{0, 1, 16, 5, 12, 5, 8, 2, 6, 3, 14, 6, 4, 13, 7, 5, 3, 13, 17, 5}, 1)\n\n\t})\n}\n\nfunc AsValuables(vs []Value) []Valuable {\n\tres := make([]Valuable, len(vs))\n\tfor i, v := range vs {\n\t\tres[i] = v\n\t}\n\treturn res\n}\n\nfunc TestListSpliceFuzzer(t *testing.T) {\n\tstartCount := 1000\n\trounds := 1000\n\tsplices := 100\n\tmaxInsertCount := uint64(50)\n\tmaxInt := uint64(100)\n\n\tvrw := newTestValueStore()\n\n\tr := rand.New(rand.NewSource(0))\n\n\tnextRandInt := func(from, to uint64) uint64 {\n\t\treturn from + uint64(float64(to-from)*r.Float64())\n\t}\n\n\tnextRandomSplice := func(len int) (idx, remove uint64, insert []Value) {\n\t\tidx = nextRandInt(0, uint64(len))\n\t\tremove = nextRandInt(0, uint64(len)-idx)\n\t\tinsCount := nextRandInt(0, maxInsertCount)\n\t\tfor i := uint64(0); i < insCount; i++ {\n\t\t\tinsert = append(insert, Number(nextRandInt(0, maxInt)))\n\t\t}\n\n\t\treturn\n\t}\n\n\tfor i := 0; i < rounds; i++ {\n\t\ttl := newTestList(startCount)\n\t\tle := tl.toList(vrw).Edit()\n\n\t\tfor j := 0; j < splices; j++ {\n\t\t\tidx, removed, insert := nextRandomSplice(len(tl))\n\t\t\ttl = tl.Splice(int(idx), int(removed), insert...)\n\t\t\tle.Splice(idx, removed, AsValuables(insert)...)\n\t\t}\n\t\texpect := tl.toList(vrw)\n\t\tactual := le.List()\n\t\tassert.True(t, expect.Equals(actual))\n\t}\n}\n" }, { "path": "go/types/list_iterator.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// ListIterator can be used to efficiently iterate through a Noms List.\ntype ListIterator struct {\n\tcursor *sequenceCursor\n}\n\n// Next returns subsequent Values from a List, starting with the index at which the iterator was\n// created. If there are no more Values, Next() returns nil.\nfunc (li ListIterator) Next() (out Value) {\n\tif li.cursor == nil {\n\t\td.Panic(\"Cannot use a nil ListIterator\")\n\t}\n\tif li.cursor.valid() {\n\t\tout = li.cursor.current().(Value)\n\t\tli.cursor.advance()\n\t}\n\treturn\n}\n" }, { "path": "go/types/list_iterator_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestListIterator(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tnumbers := append(generateNumbersAsValues(10), Number(20), Number(25))\n\tl := NewList(vrw, numbers...)\n\ti := l.Iterator()\n\tvs := iterToSlice(i)\n\tassert.True(vs.Equals(numbers), \"Expected: %v != actual: %v\", numbers, vs)\n\n\ti = l.IteratorAt(3)\n\tvs = iterToSlice(i)\n\tassert.True(vs.Equals(numbers[3:]), \"Expected: %v != actual: %v\", numbers, vs)\n\n\ti = l.IteratorAt(l.Len())\n\tassert.Nil(i.Next())\n}\n" }, { "path": "go/types/list_leaf_sequence.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\ntype listLeafSequence struct {\n\tleafSequence\n}\n\nfunc newListLeafSequence(vrw ValueReadWriter, vs ...Value) sequence {\n\treturn listLeafSequence{newLeafSequenceFromValues(ListKind, vrw, vs...)}\n}\n\n// sequence interface\n\nfunc (ll listLeafSequence) getCompareFn(other sequence) compareFn {\n\treturn ll.getCompareFnHelper(other.(listLeafSequence).leafSequence)\n}\n" }, { "path": "go/types/list_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"math/rand\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst testListSize = 5000\n\ntype testList ValueSlice\n\nfunc (tl testList) AsValuables() []Valuable {\n\tvs := make([]Valuable, len(tl))\n\tfor i, v := range tl {\n\t\tvs[i] = v\n\t}\n\treturn vs\n}\n\nfunc (tl testList) Set(idx int, v Value) (res testList) {\n\treturn tl.Splice(idx, 1, v)\n}\n\nfunc (tl testList) Insert(idx int, vs ...Value) testList {\n\treturn tl.Splice(idx, 0, vs...)\n}\n\nfunc (tl testList) Remove(start, end int) testList {\n\treturn tl.Splice(start, end-start)\n}\n\nfunc (tl testList) RemoveAt(idx int) testList {\n\treturn tl.Splice(idx, 1)\n}\n\nfunc (tl testList) Splice(idx int, remove int, insert ...Value) (res testList) {\n\tres = append(res, tl[:idx]...)\n\tres = append(res, insert...)\n\tres = append(res, tl[idx+remove:]...)\n\treturn\n}\n\nfunc (tl testList) Diff(last testList) []Splice {\n\t// Note: this could be use tl.toList/last.toList and then tlList.Diff(lastList)\n\t// but the purpose of this method is to be redundant.\n\treturn calcSplices(uint64(len(last)), uint64(len(tl)), DEFAULT_MAX_SPLICE_MATRIX_SIZE,\n\t\tfunc(i uint64, j uint64) bool { return last[i] == tl[j] })\n}\n\nfunc (tl testList) toList(vrw ValueReadWriter) List {\n\treturn NewList(vrw, tl...)\n}\n\nfunc newTestList(length int) testList {\n\treturn generateNumbersAsValues(length)\n}\n\nfunc newTestListFromList(list List) testList {\n\ttl := testList{}\n\tlist.IterAll(func(v Value, idx uint64) {\n\t\ttl = append(tl, v)\n\t})\n\treturn tl\n}\n\nfunc validateList(t *testing.T, vrw ValueReadWriter, l List, values ValueSlice) {\n\tassert.True(t, l.Equals(NewList(vrw, values...)))\n\tout := ValueSlice{}\n\tl.IterAll(func(v Value, idx uint64) {\n\t\tout = append(out, v)\n\t})\n\tassert.True(t, out.Equals(values))\n}\n\ntype listTestSuite struct {\n\tcollectionTestSuite\n\telems testList\n}\n\nfunc newListTestSuite(size uint, expectChunkCount int, expectPrependChunkDiff int, expectAppendChunkDiff int) *listTestSuite {\n\tvrw := newTestValueStore()\n\n\tlength := 1 << size\n\telems := newTestList(length)\n\ttr := MakeListType(NumberType)\n\tlist := NewList(vrw, elems...)\n\treturn &listTestSuite{\n\t\tcollectionTestSuite: collectionTestSuite{\n\t\t\tcol: list,\n\t\t\texpectType: tr,\n\t\t\texpectLen: uint64(length),\n\t\t\texpectChunkCount: expectChunkCount,\n\t\t\texpectPrependChunkDiff: expectPrependChunkDiff,\n\t\t\texpectAppendChunkDiff: expectAppendChunkDiff,\n\t\t\tvalidate: func(v2 Collection) bool {\n\t\t\t\tl2 := v2.(List)\n\t\t\t\tout := ValueSlice{}\n\t\t\t\tl2.IterAll(func(v Value, index uint64) {\n\t\t\t\t\tout = append(out, v)\n\t\t\t\t})\n\t\t\t\treturn ValueSlice(elems).Equals(out)\n\t\t\t},\n\t\t\tprependOne: func() Collection {\n\t\t\t\tdup := make([]Value, length+1)\n\t\t\t\tdup[0] = Number(0)\n\t\t\t\tcopy(dup[1:], elems)\n\t\t\t\treturn NewList(vrw, dup...)\n\t\t\t},\n\t\t\tappendOne: func() Collection {\n\t\t\t\tdup := make([]Value, length+1)\n\t\t\t\tcopy(dup, elems)\n\t\t\t\tdup[len(dup)-1] = Number(0)\n\t\t\t\treturn NewList(vrw, dup...)\n\t\t\t},\n\t\t},\n\t\telems: elems,\n\t}\n}\n\nfunc (suite *listTestSuite) TestGet() {\n\tlist := suite.col.(List)\n\tfor i := 0; i < len(suite.elems); i++ {\n\t\tsuite.True(suite.elems[i].Equals(list.Get(uint64(i))))\n\t}\n\tsuite.Equal(suite.expectLen, list.Len())\n}\n\nfunc (suite *listTestSuite) TestIter() {\n\tlist := suite.col.(List)\n\texpectIdx := uint64(0)\n\tendAt := suite.expectLen / 2\n\tlist.Iter(func(v Value, idx uint64) bool {\n\t\tsuite.Equal(expectIdx, idx)\n\t\texpectIdx++\n\t\tsuite.Equal(suite.elems[idx], v)\n\t\treturn expectIdx == endAt\n\t})\n\n\tsuite.Equal(endAt, expectIdx)\n}\n\nfunc (suite *listTestSuite) TestIterRange() {\n\tlist := suite.col.(List)\n\n\tfor s := uint64(0); s < 6; s++ {\n\t\tbatchSize := list.Len() / (2 << s)\n\t\texpectIdx := uint64(0)\n\t\tfor i := uint64(0); i < list.Len(); i += batchSize {\n\t\t\tlist.IterRange(i, i+batchSize, func(v Value, idx uint64) {\n\t\t\t\tsuite.Equal(expectIdx, idx)\n\t\t\t\texpectIdx++\n\t\t\t\tsuite.Equal(suite.elems[idx], v)\n\t\t\t})\n\t\t}\n\t}\n}\n\nfunc TestListSuite4K(t *testing.T) {\n\tsuite.Run(t, newListTestSuite(12, 8, 2, 2))\n}\n\nfunc TestListSuite8K(t *testing.T) {\n\tsuite.Run(t, newListTestSuite(14, 22, 2, 2))\n}\n\nfunc TestListInsert(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\n\ttl := newTestList(1024)\n\tlist := tl.toList(vrw)\n\n\tfor i := 0; i < len(tl); i += 16 {\n\t\ttl = tl.Insert(i, Number(i))\n\t\tlist = list.Edit().Insert(uint64(i), Number(i)).List()\n\t}\n\n\tassert.True(tl.toList(vrw).Equals(list))\n}\n\nfunc TestListRemove(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\n\ttl := newTestList(1024)\n\tlist := tl.toList(vrw)\n\n\tfor i := len(tl) - 16; i >= 0; i -= 16 {\n\t\ttl = tl.Remove(i, i+4)\n\t\tlist = list.Edit().Remove(uint64(i), uint64(i+4)).List()\n\t}\n\n\tassert.True(tl.toList(vrw).Equals(list))\n}\n\nfunc TestListRemoveAt(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tl0 := NewList(vrw)\n\tl0 = l0.Edit().Append(Bool(false), Bool(true)).List()\n\tl1 := l0.Edit().RemoveAt(1).List()\n\tassert.True(NewList(vrw, Bool(false)).Equals(l1))\n\tl1 = l1.Edit().RemoveAt(0).List()\n\tassert.True(NewList(vrw).Equals(l1))\n\n\tassert.Panics(func() {\n\t\tl1.Edit().RemoveAt(0).List()\n\t})\n}\n\nfunc getTestListLen() uint64 {\n\treturn uint64(64) * 50\n}\n\nfunc getTestList() testList {\n\treturn getTestListWithLen(int(getTestListLen()))\n}\n\nfunc getTestListWithLen(length int) testList {\n\ts := rand.NewSource(42)\n\tvalues := make([]Value, length)\n\tfor i := 0; i < length; i++ {\n\t\tvalues[i] = Number(s.Int63() & 0xff)\n\t}\n\n\treturn values\n}\n\nfunc getTestListUnique() testList {\n\tlength := int(getTestListLen())\n\ts := rand.NewSource(42)\n\tuniques := map[int64]bool{}\n\tfor len(uniques) < length {\n\t\tuniques[s.Int63()] = true\n\t}\n\tvalues := make([]Value, 0, length)\n\tfor k := range uniques {\n\t\tvalues = append(values, Number(k))\n\t}\n\treturn values\n}\n\nfunc testListFromNomsList(list List) testList {\n\tsimple := make(testList, list.Len())\n\tlist.IterAll(func(v Value, offset uint64) {\n\t\tsimple[offset] = v\n\t})\n\treturn simple\n}\n\nfunc TestStreamingListCreation(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tsimpleList := getTestList()\n\n\tcl := NewList(vs, simpleList...)\n\tvalueChan := make(chan Value)\n\tlistChan := NewStreamingList(vs, valueChan)\n\tfor _, v := range simpleList {\n\t\tvalueChan <- v\n\t}\n\tclose(valueChan)\n\tsl := <-listChan\n\tassert.True(cl.Equals(sl))\n\tcl.Iter(func(v Value, idx uint64) (done bool) {\n\t\tdone = !assert.True(v.Equals(sl.Get(idx)))\n\t\treturn\n\t})\n}\n\nfunc TestListAppend(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tnewList := func(items testList) List {\n\t\treturn NewList(vrw, items...)\n\t}\n\n\tlistToSimple := func(cl List) (simple testList) {\n\t\tcl.IterAll(func(v Value, offset uint64) {\n\t\t\tsimple = append(simple, v)\n\t\t})\n\t\treturn\n\t}\n\n\tcl := newList(getTestList())\n\tcl2 := cl.Edit().Append(Number(42)).List()\n\tcl3 := cl2.Edit().Append(Number(43)).List()\n\tcl4 := cl3.Edit().Append(getTestList().AsValuables()...).List()\n\tcl5 := cl4.Edit().Append(Number(44), Number(45)).List()\n\tcl6 := cl5.Edit().Append(getTestList().AsValuables()...).List()\n\n\texpected := getTestList()\n\tassert.Equal(expected, listToSimple(cl))\n\tassert.Equal(getTestListLen(), cl.Len())\n\tassert.True(newList(expected).Equals(cl))\n\n\texpected = append(expected, Number(42))\n\tassert.Equal(expected, listToSimple(cl2))\n\tassert.Equal(getTestListLen()+1, cl2.Len())\n\tassert.True(newList(expected).Equals(cl2))\n\n\texpected = append(expected, Number(43))\n\tassert.Equal(expected, listToSimple(cl3))\n\tassert.Equal(getTestListLen()+2, cl3.Len())\n\tassert.True(newList(expected).Equals(cl3))\n\n\texpected = append(expected, getTestList()...)\n\tassert.Equal(expected, listToSimple(cl4))\n\tassert.Equal(2*getTestListLen()+2, cl4.Len())\n\tassert.True(newList(expected).Equals(cl4))\n\n\texpected = append(expected, Number(44), Number(45))\n\tassert.Equal(expected, listToSimple(cl5))\n\tassert.Equal(2*getTestListLen()+4, cl5.Len())\n\tassert.True(newList(expected).Equals(cl5))\n\n\texpected = append(expected, getTestList()...)\n\tassert.Equal(expected, listToSimple(cl6))\n\tassert.Equal(3*getTestListLen()+4, cl6.Len())\n\tassert.True(newList(expected).Equals(cl6))\n}\n\nfunc TestListValidateInsertAscending(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tvalues := generateNumbersAsValues(1000)\n\n\ts := NewList(vrw)\n\tfor i, v := range values {\n\t\ts = s.Edit().Insert(uint64(i), v).List()\n\t\tvalidateList(t, vrw, s, values[0:i+1])\n\t}\n}\n\nfunc TestListValidateInsertAtZero(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tvalues := generateNumbersAsValues(1000)\n\ts := NewList(vrw)\n\tcount := len(values)\n\tfor count > 0 {\n\t\tcount--\n\t\tv := values[count]\n\t\ts = s.Edit().Insert(uint64(0), v).List()\n\t\tvalidateList(t, vrw, s, values[count:])\n\t}\n}\n\nfunc TestListInsertNothing(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tcl := getTestList().toList(vrw)\n\n\tassert.True(cl.Equals(cl.Edit().Insert(0).List()))\n\tfor i := uint64(1); i < getTestListLen(); i *= 2 {\n\t\tassert.True(cl.Equals(cl.Edit().Insert(i).List()))\n\t}\n\tassert.True(cl.Equals(cl.Edit().Insert(cl.Len() - 1).List()))\n\tassert.True(cl.Equals(cl.Edit().Insert(cl.Len()).List()))\n}\n\nfunc TestListInsertStart(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tcl := getTestList().toList(vrw)\n\tcl2 := cl.Edit().Insert(0, Number(42)).List()\n\tcl3 := cl2.Edit().Insert(0, Number(43)).List()\n\tcl4 := cl3.Edit().Insert(0, getTestList().AsValuables()...).List()\n\tcl5 := cl4.Edit().Insert(0, Number(44), Number(45)).List()\n\tcl6 := cl5.Edit().Insert(0, getTestList().AsValuables()...).List()\n\n\texpected := getTestList()\n\tassert.Equal(expected, testListFromNomsList(cl))\n\tassert.Equal(getTestListLen(), cl.Len())\n\tassert.True(expected.toList(vrw).Equals(cl))\n\n\texpected = expected.Insert(0, Number(42))\n\tassert.Equal(expected, testListFromNomsList(cl2))\n\tassert.Equal(getTestListLen()+1, cl2.Len())\n\tassert.True(expected.toList(vrw).Equals(cl2))\n\n\texpected = expected.Insert(0, Number(43))\n\tassert.Equal(expected, testListFromNomsList(cl3))\n\tassert.Equal(getTestListLen()+2, cl3.Len())\n\tassert.True(expected.toList(vrw).Equals(cl3))\n\n\texpected = expected.Insert(0, getTestList()...)\n\tassert.Equal(expected, testListFromNomsList(cl4))\n\tassert.Equal(2*getTestListLen()+2, cl4.Len())\n\tassert.True(expected.toList(vrw).Equals(cl4))\n\n\texpected = expected.Insert(0, Number(44), Number(45))\n\tassert.Equal(expected, testListFromNomsList(cl5))\n\tassert.Equal(2*getTestListLen()+4, cl5.Len())\n\tassert.True(expected.toList(vrw).Equals(cl5))\n\n\texpected = expected.Insert(0, getTestList()...)\n\tassert.Equal(expected, testListFromNomsList(cl6))\n\tassert.Equal(3*getTestListLen()+4, cl6.Len())\n\tassert.True(expected.toList(vrw).Equals(cl6))\n}\n\nfunc TestListInsertMiddle(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tcl := getTestList().toList(vrw)\n\tcl2 := cl.Edit().Insert(100, Number(42)).List()\n\tcl3 := cl2.Edit().Insert(200, Number(43)).List()\n\tcl4 := cl3.Edit().Insert(300, getTestList().AsValuables()...).List()\n\tcl5 := cl4.Edit().Insert(400, Number(44), Number(45)).List()\n\tcl6 := cl5.Edit().Insert(500, getTestList().AsValuables()...).List()\n\tcl7 := cl6.Edit().Insert(600, Number(100)).List()\n\n\texpected := getTestList()\n\tassert.Equal(expected, testListFromNomsList(cl))\n\tassert.Equal(getTestListLen(), cl.Len())\n\tassert.True(expected.toList(vrw).Equals(cl))\n\n\texpected = expected.Insert(100, Number(42))\n\tassert.Equal(expected, testListFromNomsList(cl2))\n\tassert.Equal(getTestListLen()+1, cl2.Len())\n\tassert.True(expected.toList(vrw).Equals(cl2))\n\n\texpected = expected.Insert(200, Number(43))\n\tassert.Equal(expected, testListFromNomsList(cl3))\n\tassert.Equal(getTestListLen()+2, cl3.Len())\n\tassert.True(expected.toList(vrw).Equals(cl3))\n\n\texpected = expected.Insert(300, getTestList()...)\n\tassert.Equal(expected, testListFromNomsList(cl4))\n\tassert.Equal(2*getTestListLen()+2, cl4.Len())\n\tassert.True(expected.toList(vrw).Equals(cl4))\n\n\texpected = expected.Insert(400, Number(44), Number(45))\n\tassert.Equal(expected, testListFromNomsList(cl5))\n\tassert.Equal(2*getTestListLen()+4, cl5.Len())\n\tassert.True(expected.toList(vrw).Equals(cl5))\n\n\texpected = expected.Insert(500, getTestList()...)\n\tassert.Equal(expected, testListFromNomsList(cl6))\n\tassert.Equal(3*getTestListLen()+4, cl6.Len())\n\tassert.True(expected.toList(vrw).Equals(cl6))\n\n\texpected = expected.Insert(600, Number(100))\n\tassert.Equal(expected, testListFromNomsList(cl7))\n\tassert.Equal(3*getTestListLen()+5, cl7.Len())\n\tassert.True(expected.toList(vrw).Equals(cl7))\n}\n\nfunc TestListInsertRanges(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\ttestList := getTestList()\n\twhole := testList.toList(vrw)\n\n\t// Compare list equality. Increment by 256 (16^2) because each iteration requires building a new list, which is slow.\n\tfor incr, i := 256, 0; i < len(testList)-incr; i += incr {\n\t\tfor window := 1; window <= incr; window *= 16 {\n\t\t\ttestListPart := testList.Remove(i, i+window)\n\t\t\tactual := testListPart.toList(vrw).Edit().Insert(uint64(i), testList[i:i+window].AsValuables()...).List()\n\t\t\tassert.Equal(whole.Len(), actual.Len())\n\t\t\tassert.True(whole.Equals(actual))\n\t\t}\n\t}\n\n\t// Compare list length, which doesn't require building a new list every iteration, so the increment can be smaller.\n\tfor incr, i := 10, 0; i < len(testList); i += incr {\n\t\tassert.Equal(len(testList)+incr, int(whole.Edit().Insert(uint64(i), testList[0:incr].AsValuables()...).List().Len()))\n\t}\n}\n\nfunc TestListRemoveNothing(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tcl := getTestList().toList(vrw)\n\n\tassert.True(cl.Equals(cl.Edit().Remove(0, 0).List()))\n\tfor i := uint64(1); i < getTestListLen(); i *= 2 {\n\t\tassert.True(cl.Equals(cl.Edit().Remove(i, i).List()))\n\t}\n\tassert.True(cl.Equals(cl.Edit().Remove(cl.Len()-1, cl.Len()-1).List()))\n\tassert.True(cl.Equals(cl.Edit().Remove(cl.Len(), cl.Len()).List()))\n}\n\nfunc TestListRemoveEverything(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tcl := getTestList().toList(vrw).Edit().Remove(0, getTestListLen()).List()\n\n\tassert.True(NewList(vrw).Equals(cl))\n\tassert.Equal(0, int(cl.Len()))\n}\n\nfunc TestListRemoveAtMiddle(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tcl := getTestList().toList(vrw)\n\tcl2 := cl.Edit().RemoveAt(100).List()\n\tcl3 := cl2.Edit().RemoveAt(200).List()\n\n\texpected := getTestList()\n\tassert.Equal(expected, testListFromNomsList(cl))\n\tassert.Equal(getTestListLen(), cl.Len())\n\tassert.True(expected.toList(vrw).Equals(cl))\n\n\texpected = expected.RemoveAt(100)\n\tassert.Equal(expected, testListFromNomsList(cl2))\n\tassert.Equal(getTestListLen()-1, cl2.Len())\n\tassert.True(expected.toList(vrw).Equals(cl2))\n\n\texpected = expected.RemoveAt(200)\n\tassert.Equal(expected, testListFromNomsList(cl3))\n\tassert.Equal(getTestListLen()-2, cl3.Len())\n\tassert.True(expected.toList(vrw).Equals(cl3))\n}\n\nfunc TestListRemoveRanges(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\ttestList := getTestList()\n\twhole := testList.toList(vrw)\n\n\t// Compare list equality. Increment by 256 (16^2) because each iteration requires building a new list, which is slow.\n\tfor incr, i := 256, 0; i < len(testList)-incr; i += incr {\n\t\tfor window := 1; window <= incr; window *= 16 {\n\t\t\ttestListPart := testList.Remove(i, i+window)\n\t\t\texpected := testListPart.toList(vrw)\n\t\t\tactual := whole.Edit().Remove(uint64(i), uint64(i+window)).List()\n\t\t\tassert.Equal(expected.Len(), actual.Len())\n\t\t\tassert.True(expected.Equals(actual))\n\t\t}\n\t}\n\n\t// Compare list length, which doesn't require building a new list every iteration, so the increment can be smaller.\n\tfor incr, i := 10, 0; i < len(testList)-incr; i += incr {\n\t\tassert.Equal(len(testList)-incr, int(whole.Edit().Remove(uint64(i), uint64(i+incr)).List().Len()))\n\t}\n}\n\nfunc TestListRemoveAtEnd(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\ttl := getTestListWithLen(testListSize / 10)\n\tcl := tl.toList(vrw)\n\n\tfor i := len(tl) - 1; i >= 0; i-- {\n\t\tcl = cl.Edit().Remove(uint64(i), uint64(i+1)).List()\n\t\texpect := tl[0:i].toList(vrw)\n\t\tassert.True(expect.Equals(cl))\n\t}\n}\n\nfunc TestListSet(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\ttestList := getTestList()\n\tcl := testList.toList(vrw)\n\n\ttestIdx := func(idx int, testEquality bool) {\n\t\tnewVal := Number(-1) // Test values are never < 0\n\t\tcl2 := cl.Edit().Set(uint64(idx), newVal).List()\n\t\tassert.False(cl.Equals(cl2))\n\t\tif testEquality {\n\t\t\tassert.True(testList.Set(idx, newVal).toList(vrw).Equals(cl2))\n\t\t}\n\t}\n\n\t// Compare list equality. Increment by 100 because each iteration requires building a new list, which is slow, but always test the last index.\n\tfor incr, i := 100, 0; i < len(testList); i += incr {\n\t\ttestIdx(i, true)\n\t}\n\ttestIdx(len(testList)-1, true)\n\n\t// Compare list unequality, which doesn't require building a new list every iteration, so the increment can be smaller.\n\tfor incr, i := 10, 0; i < len(testList); i += incr {\n\t\ttestIdx(i, false)\n\t}\n}\n\nfunc TestListFirstNNumbers(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tnums := generateNumbersAsValues(testListSize)\n\tNewList(vrw, nums...)\n}\n\nfunc TestListRefOfStructFirstNNumbers(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tvrw := newTestValueStore()\n\n\tnums := generateNumbersAsRefOfStructs(vrw, testListSize)\n\tNewList(vrw, nums...)\n}\n\nfunc TestListModifyAfterRead(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvs := newTestValueStore()\n\n\tlist := getTestList().toList(vs)\n\t// Drop chunk values.\n\tlist = vs.ReadValue(vs.WriteValue(list).TargetHash()).(List)\n\t// Modify/query. Once upon a time this would crash.\n\tllen := list.Len()\n\tz := list.Get(0)\n\tlist = list.Edit().RemoveAt(0).List()\n\tassert.Equal(llen-1, list.Len())\n\tlist = list.Edit().Append(z).List()\n\tassert.Equal(llen, list.Len())\n}\n\nfunc accumulateDiffSplices(l1, l2 List) (diff []Splice) {\n\tdiffChan := make(chan Splice)\n\tgo func() {\n\t\tl1.Diff(l2, diffChan, nil)\n\t\tclose(diffChan)\n\t}()\n\tfor splice := range diffChan {\n\t\tdiff = append(diff, splice)\n\t}\n\treturn\n}\n\nfunc accumulateDiffSplicesWithLimit(l1, l2 List, maxSpliceMatrixSize uint64) (diff []Splice) {\n\tdiffChan := make(chan Splice)\n\tgo func() {\n\t\tl1.DiffWithLimit(l2, diffChan, nil, maxSpliceMatrixSize)\n\t\tclose(diffChan)\n\t}()\n\tfor splice := range diffChan {\n\t\tdiff = append(diff, splice)\n\t}\n\treturn diff\n}\n\nfunc TestListDiffIdentical(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums := generateNumbersAsValues(5)\n\tl1 := NewList(vrw, nums...)\n\tl2 := NewList(vrw, nums...)\n\n\tdiff1 := accumulateDiffSplices(l1, l2)\n\tdiff2 := accumulateDiffSplices(l2, l1)\n\n\tassert.Equal(0, len(diff1))\n\tassert.Equal(0, len(diff2))\n}\n\nfunc TestListDiffVersusEmpty(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums1 := generateNumbersAsValues(5)\n\tl1 := NewList(vrw, nums1...)\n\tl2 := NewList(vrw)\n\n\tdiff1 := accumulateDiffSplices(l1, l2)\n\tdiff2 := accumulateDiffSplices(l2, l1)\n\n\tassert.Equal(len(diff2), len(diff1))\n\tdiffExpected := []Splice{\n\t\t{0, 0, 5, 0},\n\t}\n\tassert.Equal(diffExpected, diff1, \"expected diff is wrong\")\n}\n\nfunc TestListDiffReverse(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums1 := generateNumbersAsValues(5000)\n\tnums2 := reverseValues(nums1)\n\tl1 := NewList(vrw, nums1...)\n\tl2 := NewList(vrw, nums2...)\n\n\tdiff1 := accumulateDiffSplices(l1, l2)\n\tdiff2 := accumulateDiffSplices(l2, l1)\n\n\tdiffExpected := []Splice{\n\t\t{0, 5000, 5000, 0},\n\t}\n\tassert.Equal(diffExpected, diff1, \"expected diff is wrong\")\n\tassert.Equal(diffExpected, diff2, \"expected diff is wrong\")\n}\n\nfunc TestListDiffReverseWithLargerLimit(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums1 := generateNumbersAsValues(5000)\n\tnums2 := reverseValues(nums1)\n\n\tl1 := NewList(vrw, nums1...)\n\tl2 := NewList(vrw, nums2...)\n\n\tdiff1 := accumulateDiffSplicesWithLimit(l1, l2, 27e6)\n\tdiff2 := accumulateDiffSplicesWithLimit(l2, l1, 27e6)\n\n\tassert.Equal(len(diff2), len(diff1))\n\tdiffExpected := []Splice{\n\t\t{0, 2499, 2500, 0},\n\t\t{2500, 2500, 2499, 2501},\n\t}\n\tassert.Equal(diffExpected, diff1, \"expected diff is wrong\")\n\tassert.Equal(diffExpected, diff2, \"expected diff is wrong\")\n}\n\nfunc TestListDiffRemove5x100(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums1 := generateNumbersAsValues(5000)\n\tnums2 := generateNumbersAsValues(5000)\n\tfor count := 5; count > 0; count-- {\n\t\tnums2 = spliceValues(nums2, (count-1)*1000, 100)\n\t}\n\tl1 := NewList(vrw, nums1...)\n\tl2 := NewList(vrw, nums2...)\n\n\tdiff1 := accumulateDiffSplices(l1, l2)\n\tdiff2 := accumulateDiffSplices(l2, l1)\n\n\tassert.Equal(len(diff1), len(diff2))\n\tdiff2Expected := []Splice{\n\t\t{0, 100, 0, 0},\n\t\t{1000, 100, 0, 0},\n\t\t{2000, 100, 0, 0},\n\t\t{3000, 100, 0, 0},\n\t\t{4000, 100, 0, 0},\n\t}\n\tassert.Equal(diff2Expected, diff2, \"expected diff is wrong\")\n}\n\nfunc TestListDiffAdd5x5(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums1 := generateNumbersAsValues(5000)\n\tnums2 := generateNumbersAsValues(5000)\n\tfor count := 5; count > 0; count-- {\n\t\tnums2 = spliceValues(nums2, (count-1)*1000, 0, Number(0), Number(1), Number(2), Number(3), Number(4))\n\t}\n\tl1 := NewList(vrw, nums1...)\n\tl2 := NewList(vrw, nums2...)\n\n\tdiff1 := accumulateDiffSplices(l1, l2)\n\tdiff2 := accumulateDiffSplices(l2, l1)\n\n\tassert.Equal(len(diff1), len(diff2))\n\tdiff2Expected := []Splice{\n\t\t{5, 0, 5, 5},\n\t\t{1000, 0, 5, 1005},\n\t\t{2000, 0, 5, 2010},\n\t\t{3000, 0, 5, 3015},\n\t\t{4000, 0, 5, 4020},\n\t}\n\tassert.Equal(diff2Expected, diff2, \"expected diff is wrong\")\n}\n\nfunc TestListDiffReplaceReverse5x100(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums1 := generateNumbersAsValues(5000)\n\tnums2 := generateNumbersAsValues(5000)\n\tfor count := 5; count > 0; count-- {\n\t\tout := reverseValues(nums2[(count-1)*1000 : (count-1)*1000+100])\n\t\tnums2 = spliceValues(nums2, (count-1)*1000, 100, out...)\n\t}\n\tl1 := NewList(vrw, nums1...)\n\tl2 := NewList(vrw, nums2...)\n\tdiff := accumulateDiffSplices(l2, l1)\n\n\tdiffExpected := []Splice{\n\t\t{0, 49, 50, 0},\n\t\t{50, 50, 49, 51},\n\t\t{1000, 49, 50, 1000},\n\t\t{1050, 50, 49, 1051},\n\t\t{2000, 49, 50, 2000},\n\t\t{2050, 50, 49, 2051},\n\t\t{3000, 49, 50, 3000},\n\t\t{3050, 50, 49, 3051},\n\t\t{4000, 49, 50, 4000},\n\t\t{4050, 50, 49, 4051},\n\t}\n\tassert.Equal(diffExpected, diff, \"expected diff is wrong\")\n}\n\nfunc TestListDiffString1(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums1 := []Value{String(\"one\"), String(\"two\"), String(\"three\")}\n\tnums2 := []Value{String(\"one\"), String(\"two\"), String(\"three\")}\n\tl1 := NewList(vrw, nums1...)\n\tl2 := NewList(vrw, nums2...)\n\tdiff := accumulateDiffSplices(l2, l1)\n\n\tassert.Equal(0, len(diff))\n}\n\nfunc TestListDiffString2(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums1 := []Value{String(\"one\"), String(\"two\"), String(\"three\")}\n\tnums2 := []Value{String(\"one\"), String(\"two\"), String(\"three\"), String(\"four\")}\n\tl1 := NewList(vrw, nums1...)\n\tl2 := NewList(vrw, nums2...)\n\tdiff := accumulateDiffSplices(l2, l1)\n\n\tdiffExpected := []Splice{\n\t\t{3, 0, 1, 3},\n\t}\n\tassert.Equal(diffExpected, diff, \"expected diff is wrong\")\n}\n\nfunc TestListDiffString3(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tnums1 := []Value{String(\"one\"), String(\"two\"), String(\"three\")}\n\tnums2 := []Value{String(\"one\"), String(\"two\"), String(\"four\")}\n\tl1 := NewList(vrw, nums1...)\n\tl2 := NewList(vrw, nums2...)\n\tdiff := accumulateDiffSplices(l2, l1)\n\n\tdiffExpected := []Splice{\n\t\t{2, 1, 1, 2},\n\t}\n\tassert.Equal(diffExpected, diff, \"expected diff is wrong\")\n}\n\nfunc TestListDiffLargeWithSameMiddle(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tassert := assert.New(t)\n\n\tstorage := &chunks.TestStorage{}\n\n\tcs1 := storage.NewView()\n\tvs1 := NewValueStore(cs1)\n\tnums1 := generateNumbersAsValues(4000)\n\tl1 := NewList(vs1, nums1...)\n\thash1 := vs1.WriteValue(l1).TargetHash()\n\tvs1.Commit(vs1.Root(), vs1.Root())\n\n\trefList1 := vs1.ReadValue(hash1).(List)\n\n\tcs2 := storage.NewView()\n\tvs2 := NewValueStore(cs2)\n\tnums2 := generateNumbersAsValuesFromToBy(5, 3550, 1)\n\tl2 := NewList(vs2, nums2...)\n\thash2 := vs2.WriteValue(l2).TargetHash()\n\tvs2.Commit(vs1.Root(), vs1.Root())\n\trefList2 := vs2.ReadValue(hash2).(List)\n\n\t// diff lists without value store\n\tdiff1 := accumulateDiffSplices(l2, l1)\n\tassert.Equal(2, len(diff1))\n\n\t// diff lists from value stores\n\tdiff2 := accumulateDiffSplices(refList2, refList1)\n\tassert.Equal(2, len(diff2))\n\n\t// diff without and with value store should be same\n\tassert.Equal(diff1, diff2)\n\n\t// should only read/write a \"small & reasonably sized portion of the total\"\n\tassert.Equal(9, cs1.Writes)\n\tassert.Equal(3, cs1.Reads)\n\tassert.Equal(9, cs2.Writes)\n\tassert.Equal(3, cs2.Reads)\n}\n\nfunc TestListDiffAllValuesInSequenceRemoved(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tnewSequenceMetaTuple := func(vs ...Value) metaTuple {\n\t\tseq := newListLeafSequence(vrw, vs...)\n\t\tlist := newList(seq)\n\t\treturn newMetaTuple(vrw.WriteValue(list), orderedKeyFromInt(len(vs)), uint64(len(vs)))\n\t}\n\n\tm1 := newSequenceMetaTuple(Number(1), Number(2), Number(3))\n\tm2 := newSequenceMetaTuple(Number(4), Number(5), Number(6), Number(7), Number(8))\n\tm3 := newSequenceMetaTuple(Number(9), Number(10), Number(11), Number(12), Number(13), Number(14), Number(15))\n\n\tl1 := newList(newListMetaSequence(1, []metaTuple{m1, m3}, vrw)) // [1, 2, 3][9, 10, 11, 12, 13, 14, 15]\n\tl2 := newList(newListMetaSequence(1, []metaTuple{m1, m2, m3}, vrw)) // [1, 2, 3][4, 5, 6, 7, 8][9, 10, 11, 12, 13, 14, 15]\n\n\tdiff := accumulateDiffSplices(l2, l1)\n\n\texpected := []Splice{\n\t\t{3, 0, 5, 3},\n\t}\n\n\tassert.Equal(expected, diff)\n}\n\nfunc TestListTypeAfterMutations(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\n\ttest := func(n int, c interface{}) {\n\t\tvrw := newTestValueStore()\n\t\tvalues := generateNumbersAsValues(n)\n\n\t\tl := NewList(vrw, values...)\n\t\tassert.Equal(l.Len(), uint64(n))\n\t\tassert.IsType(c, l.asSequence())\n\t\tassert.True(TypeOf(l).Equals(MakeListType(NumberType)))\n\n\t\tl = l.Edit().Append(String(\"a\")).List()\n\t\tassert.Equal(l.Len(), uint64(n+1))\n\t\tassert.IsType(c, l.asSequence())\n\t\tassert.True(TypeOf(l).Equals(MakeListType(MakeUnionType(NumberType, StringType))))\n\n\t\tl = l.Edit().Splice(l.Len()-1, 1).List()\n\t\tassert.Equal(l.Len(), uint64(n))\n\t\tassert.IsType(c, l.asSequence())\n\t\tassert.True(TypeOf(l).Equals(MakeListType(NumberType)))\n\t}\n\n\ttest(15, listLeafSequence{})\n\ttest(1500, metaSequence{})\n}\n\nfunc TestListRemoveLastWhenNotLoaded(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvs := newTestValueStore()\n\treload := func(l List) List {\n\t\treturn vs.ReadValue(vs.WriteValue(l).TargetHash()).(List)\n\t}\n\n\ttl := newTestList(1024)\n\tnl := tl.toList(vs)\n\n\tfor len(tl) > 0 {\n\t\ttl = tl[:len(tl)-1]\n\t\tnl = reload(nl.Edit().RemoveAt(uint64(len(tl))).List())\n\t\tassert.True(tl.toList(vs).Equals(nl))\n\t}\n}\n\nfunc TestListConcat(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvs := newTestValueStore()\n\treload := func(vs *ValueStore, l List) List {\n\t\treturn vs.ReadValue(vs.WriteValue(l).TargetHash()).(List)\n\t}\n\n\trun := func(seed int64, size, from, to, by int) {\n\t\tr := rand.New(rand.NewSource(seed))\n\n\t\tlistSlice := make(testList, size)\n\t\tfor i := range listSlice {\n\t\t\tlistSlice[i] = Number(r.Intn(size))\n\t\t}\n\n\t\tlist := listSlice.toList(vs)\n\n\t\tfor i := from; i < to; i += by {\n\t\t\tfst := reload(vs, listSlice[:i].toList(vs))\n\t\t\tsnd := reload(vs, listSlice[i:].toList(vs))\n\t\t\tactual := fst.Concat(snd)\n\t\t\tassert.True(list.Equals(actual),\n\t\t\t\t\"fail at %d/%d (with expected length %d, actual %d)\", i, size, list.Len(), actual.Len())\n\t\t}\n\t}\n\n\trun(0, 10, 0, 10, 1)\n\n\trun(1, 100, 0, 100, 1)\n\n\trun(2, 1000, 0, 1000, 10)\n\trun(3, 1000, 0, 100, 1)\n\trun(4, 1000, 900, 1000, 1)\n\n\trun(5, 1e4, 0, 1e4, 100)\n\trun(6, 1e4, 0, 1000, 10)\n\trun(7, 1e4, 1e4-1000, 1e4, 10)\n}\n\nfunc TestListConcatDifferentTypes(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tfst := generateNumbersAsValuesFromToBy(0, testListSize/2, 1)\n\tsnd := generateNumbersAsStructsFromToBy(testListSize/2, testListSize, 1)\n\n\tvar whole ValueSlice\n\twhole = append(whole, fst...)\n\twhole = append(whole, snd...)\n\n\tconcat := NewList(vrw, fst...).Concat(NewList(vrw, snd...))\n\tassert.True(NewList(vrw, whole...).Equals(concat))\n}\n\nfunc TestListWithStructShouldHaveOptionalFields(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tlist := NewList(vrw,\n\t\tNewStruct(\"Foo\", StructData{\n\t\t\t\"a\": Number(1),\n\t\t}),\n\t\tNewStruct(\"Foo\", StructData{\n\t\t\t\"a\": Number(2),\n\t\t\t\"b\": String(\"bar\"),\n\t\t}),\n\t)\n\tassert.True(\n\t\tMakeListType(MakeStructType(\"Foo\",\n\t\t\tStructField{\"a\", NumberType, false},\n\t\t\tStructField{\"b\", StringType, true},\n\t\t),\n\t\t).Equals(TypeOf(list)))\n}\n\nfunc TestListWithNil(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassert.Panics(t, func() {\n\t\tNewList(vrw, nil)\n\t})\n\tassert.Panics(t, func() {\n\t\tNewList(vrw, Number(42), nil)\n\t})\n}\n\nfunc TestListOfListsDoesNotWriteRoots(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tl1 := NewList(vrw, String(\"a\"), String(\"b\"))\n\tl2 := NewList(vrw, String(\"c\"), String(\"d\"))\n\tl3 := NewList(vrw, l1, l2)\n\n\tassert.Nil(vrw.ReadValue(l1.Hash()))\n\tassert.Nil(vrw.ReadValue(l2.Hash()))\n\tassert.Nil(vrw.ReadValue(l3.Hash()))\n\n\tvrw.WriteValue(l3)\n\tassert.Nil(vrw.ReadValue(l1.Hash()))\n\tassert.Nil(vrw.ReadValue(l2.Hash()))\n}\n" }, { "path": "go/types/make_type.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nfunc MakePrimitiveType(k NomsKind) *Type {\n\tswitch k {\n\tcase BoolKind:\n\t\treturn BoolType\n\tcase NumberKind:\n\t\treturn NumberType\n\tcase StringKind:\n\t\treturn StringType\n\tcase BlobKind:\n\t\treturn BlobType\n\tcase ValueKind:\n\t\treturn ValueType\n\tcase TypeKind:\n\t\treturn TypeType\n\t}\n\td.Chk.Fail(\"invalid NomsKind: %d\", k)\n\treturn nil\n}\n\n// MakeUnionType creates a new union type unless the elemTypes can be folded into a single non union type.\nfunc MakeUnionType(elemTypes ...*Type) *Type {\n\treturn simplifyType(makeUnionType(elemTypes...), false)\n}\n\nfunc MakeListType(elemType *Type) *Type {\n\treturn simplifyType(makeCompoundType(ListKind, elemType), false)\n}\n\nfunc MakeSetType(elemType *Type) *Type {\n\treturn simplifyType(makeCompoundType(SetKind, elemType), false)\n}\n\nfunc MakeRefType(elemType *Type) *Type {\n\treturn simplifyType(makeCompoundType(RefKind, elemType), false)\n}\n\nfunc MakeMapType(keyType, valType *Type) *Type {\n\treturn simplifyType(makeCompoundType(MapKind, keyType, valType), false)\n}\n\nfunc MakeStructType(name string, fields ...StructField) *Type {\n\tfs := structTypeFields(fields)\n\tsort.Sort(fs)\n\treturn simplifyType(makeStructType(name, fs), false)\n}\n\n// MakeUnionTypeIntersectStructs is a bit of strange function. It creates a\n// simplified union type except for structs, where it creates interesection\n// types.\n// This function will go away so do not use it!\nfunc MakeUnionTypeIntersectStructs(elemTypes ...*Type) *Type {\n\treturn simplifyType(makeUnionType(elemTypes...), true)\n}\n\nfunc MakeCycleType(name string) *Type {\n\td.PanicIfTrue(name == \"\")\n\treturn newType(CycleDesc(name))\n}\n\nfunc makePrimitiveType(k NomsKind) *Type {\n\treturn newType(PrimitiveDesc(k))\n}\n\nvar BoolType = makePrimitiveType(BoolKind)\nvar NumberType = makePrimitiveType(NumberKind)\nvar StringType = makePrimitiveType(StringKind)\nvar BlobType = makePrimitiveType(BlobKind)\nvar TypeType = makePrimitiveType(TypeKind)\nvar ValueType = makePrimitiveType(ValueKind)\n\nfunc makeCompoundType(kind NomsKind, elemTypes ...*Type) *Type {\n\treturn newType(CompoundDesc{kind, elemTypes})\n}\n\nfunc makeUnionType(elemTypes ...*Type) *Type {\n\tif len(elemTypes) == 1 {\n\t\treturn elemTypes[0]\n\t}\n\treturn makeCompoundType(UnionKind, elemTypes...)\n}\n\nfunc makeStructTypeQuickly(name string, fields structTypeFields) *Type {\n\treturn newType(StructDesc{name, fields})\n}\n\nfunc makeStructType(name string, fields structTypeFields) *Type {\n\tverifyStructName(name)\n\tverifyFields(fields)\n\treturn makeStructTypeQuickly(name, fields)\n}\n\ntype FieldMap map[string]*Type\n\nfunc MakeStructTypeFromFields(name string, fields FieldMap) *Type {\n\tfs := make(structTypeFields, len(fields))\n\ti := 0\n\tfor k, v := range fields {\n\t\tfs[i] = StructField{k, v, false}\n\t\ti++\n\t}\n\tsort.Sort(fs)\n\treturn simplifyType(makeStructType(name, fs), false)\n}\n\n// StructField describes a field in a struct type.\ntype StructField struct {\n\tName string\n\tType *Type\n\tOptional bool\n}\n\ntype structTypeFields []StructField\n\nfunc (s structTypeFields) Len() int { return len(s) }\nfunc (s structTypeFields) Swap(i, j int) { s[i], s[j] = s[j], s[i] }\nfunc (s structTypeFields) Less(i, j int) bool { return s[i].Name < s[j].Name }\n" }, { "path": "go/types/map.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"fmt\"\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype Map struct {\n\torderedSequence\n}\n\nfunc newMap(seq orderedSequence) Map {\n\treturn Map{seq}\n}\n\nfunc mapHashValueBytes(item sequenceItem, rv *rollingValueHasher) {\n\tentry := item.(mapEntry)\n\thashValueBytes(entry.key, rv)\n\thashValueBytes(entry.value, rv)\n}\n\nfunc NewMap(vrw ValueReadWriter, kv ...Value) Map {\n\tentries := buildMapData(kv)\n\tch := newEmptyMapSequenceChunker(vrw)\n\n\tfor _, entry := range entries {\n\t\tch.Append(entry)\n\t}\n\n\treturn newMap(ch.Done().(orderedSequence))\n}\n\n// NewStreamingMap takes an input channel of values and returns a output\n// channel that will produce a finished Map. Values sent to the input channel\n// must be alternating keys and values. (e.g. k1, v1, k2, v2...). Moreover keys\n// need to be added to the channel in Noms sortorder, adding key values to the\n// input channel out of order will result in a panic. Once the input channel is\n// closed by the caller, a finished Map will be sent to the output channel. See\n// graph_builder.go for building collections with values that are not in order.\nfunc NewStreamingMap(vrw ValueReadWriter, kvs <-chan Value) <-chan Map {\n\td.PanicIfTrue(vrw == nil)\n\treturn newStreamingMap(vrw, kvs, func(vrw ValueReadWriter, kvs <-chan Value, outChan chan<- Map) {\n\t\tgo readMapInput(vrw, kvs, outChan)\n\t})\n}\n\ntype streamingMapReadFunc func(vrw ValueReadWriter, kvs <-chan Value, outChan chan<- Map)\n\nfunc newStreamingMap(vrw ValueReadWriter, kvs <-chan Value, readFunc streamingMapReadFunc) <-chan Map {\n\toutChan := make(chan Map, 1)\n\treadFunc(vrw, kvs, outChan)\n\treturn outChan\n}\n\nfunc readMapInput(vrw ValueReadWriter, kvs <-chan Value, outChan chan<- Map) {\n\tdefer close(outChan)\n\tch := newEmptyMapSequenceChunker(vrw)\n\tvar lastK Value\n\tnextIsKey := true\n\tvar k Value\n\tfor v := range kvs {\n\t\td.PanicIfTrue(v == nil)\n\t\tif nextIsKey {\n\t\t\tk = v\n\t\t\td.PanicIfFalse(lastK == nil || lastK.Less(k))\n\t\t\tlastK = k\n\t\t\tnextIsKey = false\n\t\t\tcontinue\n\t\t}\n\t\tch.Append(mapEntry{key: k, value: v})\n\t\tnextIsKey = true\n\t}\n\toutChan <- newMap(ch.Done().(orderedSequence))\n}\n\n// Diff computes the diff from |last| to |m| using the top-down algorithm,\n// which completes as fast as possible while taking longer to return early\n// results than left-to-right.\nfunc (m Map) Diff(last Map, changes chan<- ValueChanged, closeChan <-chan struct{}) {\n\tif m.Equals(last) {\n\t\treturn\n\t}\n\torderedSequenceDiffTopDown(last.orderedSequence, m.orderedSequence, changes, closeChan)\n}\n\n// DiffHybrid computes the diff from |last| to |m| using a hybrid algorithm\n// which balances returning results early vs completing quickly, if possible.\nfunc (m Map) DiffHybrid(last Map, changes chan<- ValueChanged, closeChan <-chan struct{}) {\n\tif m.Equals(last) {\n\t\treturn\n\t}\n\torderedSequenceDiffBest(last.orderedSequence, m.orderedSequence, changes, closeChan)\n}\n\n// DiffLeftRight computes the diff from |last| to |m| using a left-to-right\n// streaming approach, optimised for returning results early, but not\n// completing quickly.\nfunc (m Map) DiffLeftRight(last Map, changes chan<- ValueChanged, closeChan <-chan struct{}) {\n\tif m.Equals(last) {\n\t\treturn\n\t}\n\torderedSequenceDiffLeftRight(last.orderedSequence, m.orderedSequence, changes, closeChan)\n}\n\n// Collection interface\n\nfunc (m Map) asSequence() sequence {\n\treturn m.orderedSequence\n}\n\n// Value interface\nfunc (m Map) Value() Value {\n\treturn m\n}\n\nfunc (m Map) WalkValues(cb ValueCallback) {\n\titerAll(m, func(v Value, idx uint64) {\n\t\tcb(v)\n\t})\n\treturn\n}\n\nfunc (m Map) firstOrLast(last bool) (Value, Value) {\n\tcur := newCursorAt(m.orderedSequence, emptyKey, false, last)\n\tif !cur.valid() {\n\t\treturn nil, nil\n\t}\n\tentry := cur.current().(mapEntry)\n\treturn entry.key, entry.value\n}\n\nfunc (m Map) First() (Value, Value) {\n\treturn m.firstOrLast(false)\n}\n\nfunc (m Map) Last() (Value, Value) {\n\treturn m.firstOrLast(true)\n}\n\nfunc (m Map) At(idx uint64) (key, value Value) {\n\tif idx >= m.Len() {\n\t\tpanic(fmt.Errorf(\"Out of bounds: %d >= %d\", idx, m.Len()))\n\t}\n\n\tcur := newCursorAtIndex(m.orderedSequence, idx)\n\tentry := cur.current().(mapEntry)\n\treturn entry.key, entry.value\n}\n\nfunc (m Map) MaybeGet(key Value) (v Value, ok bool) {\n\tcur := newCursorAtValue(m.orderedSequence, key, false, false)\n\tif !cur.valid() {\n\t\treturn nil, false\n\t}\n\tentry := cur.current().(mapEntry)\n\tif !entry.key.Equals(key) {\n\t\treturn nil, false\n\t}\n\n\treturn entry.value, true\n}\n\nfunc (m Map) Has(key Value) bool {\n\tcur := newCursorAtValue(m.orderedSequence, key, false, false)\n\tif !cur.valid() {\n\t\treturn false\n\t}\n\tentry := cur.current().(mapEntry)\n\treturn entry.key.Equals(key)\n}\n\nfunc (m Map) Get(key Value) Value {\n\tv, _ := m.MaybeGet(key)\n\treturn v\n}\n\ntype mapIterCallback func(key, value Value) (stop bool)\n\nfunc (m Map) Iter(cb mapIterCallback) {\n\tcur := newCursorAt(m.orderedSequence, emptyKey, false, false)\n\tcur.iter(func(v interface{}) bool {\n\t\tentry := v.(mapEntry)\n\t\treturn cb(entry.key, entry.value)\n\t})\n}\n\n// Any returns true if cb() return true for any of the items in the map.\nfunc (m Map) Any(cb func(k, v Value) bool) (yep bool) {\n\tm.Iter(func(k, v Value) bool {\n\t\tif cb(k, v) {\n\t\t\tyep = true\n\t\t\treturn true\n\t\t}\n\t\treturn false\n\t})\n\treturn\n}\n\nfunc (m Map) Iterator() *MapIterator {\n\treturn m.IteratorAt(0)\n}\n\nfunc (m Map) IteratorAt(pos uint64) *MapIterator {\n\treturn &MapIterator{\n\t\tcursor: newCursorAtIndex(m.orderedSequence, pos),\n\t}\n}\n\nfunc (m Map) IteratorFrom(key Value) *MapIterator {\n\treturn &MapIterator{\n\t\tcursor: newCursorAtValue(m.orderedSequence, key, false, false),\n\t}\n}\n\ntype mapIterAllCallback func(key, value Value)\n\nfunc (m Map) IterAll(cb mapIterAllCallback) {\n\tvar k Value\n\titerAll(m, func(v Value, idx uint64) {\n\t\tif k != nil {\n\t\t\tcb(k, v)\n\t\t\tk = nil\n\t\t} else {\n\t\t\tk = v\n\t\t}\n\t})\n\td.PanicIfFalse(k == nil)\n}\n\nfunc (m Map) IterFrom(start Value, cb mapIterCallback) {\n\tcur := newCursorAtValue(m.orderedSequence, start, false, false)\n\tcur.iter(func(v interface{}) bool {\n\t\tentry := v.(mapEntry)\n\t\treturn cb(entry.key, entry.value)\n\t})\n}\n\nfunc (m Map) Edit() *MapEditor {\n\treturn NewMapEditor(m)\n}\n\nfunc buildMapData(values []Value) mapEntrySlice {\n\tif len(values) == 0 {\n\t\treturn mapEntrySlice{}\n\t}\n\n\tif len(values)%2 != 0 {\n\t\td.Panic(\"Must specify even number of key/value pairs\")\n\t}\n\tkvs := make(mapEntrySlice, len(values)/2)\n\n\tfor i := 0; i < len(values); i += 2 {\n\t\td.PanicIfTrue(values[i] == nil)\n\t\td.PanicIfTrue(values[i+1] == nil)\n\t\tentry := mapEntry{values[i], values[i+1]}\n\t\tkvs[i/2] = entry\n\t}\n\n\tuniqueSorted := make(mapEntrySlice, 0, len(kvs))\n\tsort.Stable(kvs)\n\tlast := kvs[0]\n\tfor i := 1; i < len(kvs); i++ {\n\t\tkv := kvs[i]\n\t\tif !kv.key.Equals(last.key) {\n\t\t\tuniqueSorted = append(uniqueSorted, last)\n\t\t}\n\n\t\tlast = kv\n\t}\n\n\treturn append(uniqueSorted, last)\n}\n\nfunc makeMapLeafChunkFn(vrw ValueReadWriter) makeChunkFn {\n\treturn func(level uint64, items []sequenceItem) (Collection, orderedKey, uint64) {\n\t\td.PanicIfFalse(level == 0)\n\t\tmapData := make([]mapEntry, len(items), len(items))\n\n\t\tvar lastKey Value\n\t\tfor i, v := range items {\n\t\t\tentry := v.(mapEntry)\n\t\t\td.PanicIfFalse(lastKey == nil || lastKey.Less(entry.key))\n\t\t\tlastKey = entry.key\n\t\t\tmapData[i] = entry\n\t\t}\n\n\t\tm := newMap(newMapLeafSequence(vrw, mapData...))\n\t\tvar key orderedKey\n\t\tif len(mapData) > 0 {\n\t\t\tkey = newOrderedKey(mapData[len(mapData)-1].key)\n\t\t}\n\t\treturn m, key, uint64(len(items))\n\t}\n}\n\nfunc newEmptyMapSequenceChunker(vrw ValueReadWriter) *sequenceChunker {\n\treturn newEmptySequenceChunker(vrw, makeMapLeafChunkFn(vrw), newOrderedMetaSequenceChunkFn(MapKind, vrw), mapHashValueBytes)\n}\n" }, { "path": "go/types/map_editor.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// MapEditor allows for efficient editing of Map-typed prolly trees. Edits\n// are buffered to memory and can be applied via Build(), which returns a new\n// Map. Prior to Build(), Get() & Has() will return the value that the resulting\n// Map would return if it were built immediately prior to the respective call.\n// Note: The implementation biases performance towards a usage which applies\n// edits in key-order.\ntype MapEditor struct {\n\tm Map\n\tedits mapEditSlice // edits may contain duplicate key values, in which case, the last edit of a given key is used\n\tnormalized bool\n}\n\nfunc NewMapEditor(m Map) *MapEditor {\n\treturn &MapEditor{m, mapEditSlice{}, true}\n}\n\nfunc (me *MapEditor) Kind() NomsKind {\n\treturn MapKind\n}\n\nfunc (me *MapEditor) Value() Value {\n\treturn me.Map()\n}\n\nfunc (me *MapEditor) Map() Map {\n\tif len(me.edits) == 0 {\n\t\treturn me.m // no edits\n\t}\n\n\tseq := me.m.orderedSequence\n\tvrw := seq.valueReadWriter()\n\n\tme.normalize()\n\n\tcursChan := make(chan chan *sequenceCursor)\n\tkvsChan := make(chan chan mapEntry)\n\n\tgo func() {\n\t\tfor i, edit := range me.edits {\n\t\t\tif i+1 < len(me.edits) && me.edits[i+1].key.Equals(edit.key) {\n\t\t\t\tcontinue // next edit supercedes this one\n\t\t\t}\n\n\t\t\tedit := edit\n\n\t\t\t// TODO: Use ReadMany\n\t\t\tcc := make(chan *sequenceCursor, 1)\n\t\t\tcursChan <- cc\n\n\t\t\tgo func() {\n\t\t\t\tcc <- newCursorAtValue(seq, edit.key, true, false)\n\t\t\t}()\n\n\t\t\tkvc := make(chan mapEntry, 1)\n\t\t\tkvsChan <- kvc\n\n\t\t\tif edit.value == nil {\n\t\t\t\tkvc <- mapEntry{edit.key, nil}\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif v, ok := edit.value.(Value); ok {\n\t\t\t\tkvc <- mapEntry{edit.key, v}\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tgo func() {\n\t\t\t\tsv := edit.value.Value()\n\t\t\t\tkvc <- mapEntry{edit.key, sv}\n\t\t\t}()\n\t\t}\n\n\t\tclose(cursChan)\n\t\tclose(kvsChan)\n\t}()\n\n\tvar ch *sequenceChunker\n\tfor cc := range cursChan {\n\t\tcur := <-cc\n\t\tkv := <-<-kvsChan\n\n\t\tvar existingValue Value\n\t\tif cur.idx < cur.seq.seqLen() {\n\t\t\tckv := cur.current().(mapEntry)\n\t\t\tif ckv.key.Equals(kv.key) {\n\t\t\t\texistingValue = ckv.value\n\t\t\t}\n\t\t}\n\n\t\tif existingValue == nil && kv.value == nil {\n\t\t\tcontinue // already non-present\n\t\t}\n\n\t\tif existingValue != nil && kv.value != nil && existingValue.Equals(kv.value) {\n\t\t\tcontinue // same value\n\t\t}\n\n\t\tif ch == nil {\n\t\t\tch = newSequenceChunker(cur, 0, vrw, makeMapLeafChunkFn(vrw), newOrderedMetaSequenceChunkFn(MapKind, vrw), mapHashValueBytes)\n\t\t} else {\n\t\t\tch.advanceTo(cur)\n\t\t}\n\n\t\tif existingValue != nil {\n\t\t\tch.Skip()\n\t\t}\n\t\tif kv.value != nil {\n\t\t\tch.Append(kv)\n\t\t}\n\t}\n\n\tif ch == nil {\n\t\treturn me.m // no edits required application\n\t}\n\n\treturn newMap(ch.Done().(orderedSequence))\n}\n\nfunc (me *MapEditor) Set(k Value, v Valuable) *MapEditor {\n\td.PanicIfTrue(v == nil)\n\tme.set(k, v)\n\treturn me\n}\n\nfunc (me *MapEditor) SetM(kv ...Valuable) *MapEditor {\n\td.PanicIfFalse(len(kv)%2 == 0)\n\n\tfor i := 0; i < len(kv); i += 2 {\n\t\tme.Set(kv[i].(Value), kv[i+1])\n\t}\n\treturn me\n}\n\nfunc (me *MapEditor) Remove(k Value) *MapEditor {\n\tme.set(k, nil)\n\treturn me\n}\n\nfunc (me *MapEditor) Get(k Value) Valuable {\n\tif idx, found := me.findEdit(k); found {\n\t\treturn me.edits[idx].value\n\t}\n\n\treturn me.m.Get(k)\n}\n\nfunc (me *MapEditor) Has(k Value) bool {\n\tif idx, found := me.findEdit(k); found {\n\t\treturn me.edits[idx].value != nil\n\t}\n\n\treturn me.m.Has(k)\n}\n\nfunc (me *MapEditor) set(k Value, v Valuable) {\n\tif len(me.edits) == 0 {\n\t\tme.edits = append(me.edits, mapEdit{k, v})\n\t\treturn\n\t}\n\n\tfinal := me.edits[len(me.edits)-1]\n\tif final.key.Equals(k) {\n\t\tme.edits[len(me.edits)-1] = mapEdit{k, v}\n\t\treturn // update the last edit\n\t}\n\n\tme.edits = append(me.edits, mapEdit{k, v})\n\n\tif me.normalized && final.key.Less(k) {\n\t\t// fast-path: edits take place in key-order\n\t\treturn\n\t}\n\n\t// de-normalize\n\tme.normalized = false\n}\n\n// Find the edit position of the last edit for a given key\nfunc (me *MapEditor) findEdit(k Value) (idx int, found bool) {\n\tme.normalize()\n\n\tidx = sort.Search(len(me.edits), func(i int) bool {\n\t\treturn !me.edits[i].key.Less(k)\n\t})\n\n\tif idx == len(me.edits) {\n\t\treturn\n\t}\n\n\tif !me.edits[idx].key.Equals(k) {\n\t\treturn\n\t}\n\n\t// advance to final edit position where kv.key == k\n\tfor idx < len(me.edits) && me.edits[idx].key.Equals(k) {\n\t\tidx++\n\t}\n\tidx--\n\n\tfound = true\n\treturn\n}\n\nfunc (me *MapEditor) normalize() {\n\tif me.normalized {\n\t\treturn\n\t}\n\n\tsort.Stable(me.edits)\n\t// TODO: GC duplicate keys over some threshold of collectable memory?\n\tme.normalized = true\n}\n\ntype mapEdit struct {\n\tkey Value\n\tvalue Valuable\n}\n\ntype mapEditSlice []mapEdit\n\nfunc (mes mapEditSlice) Len() int { return len(mes) }\nfunc (mes mapEditSlice) Swap(i, j int) { mes[i], mes[j] = mes[j], mes[i] }\nfunc (mes mapEditSlice) Less(i, j int) bool { return mes[i].key.Less(mes[j].key) }\n" }, { "path": "go/types/map_iterator.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\n// MapIterator can efficiently iterate through a Noms Map.\ntype MapIterator struct {\n\tcursor *sequenceCursor\n\tcurrentKey Value\n\tcurrentValue Value\n}\n\nfunc (mi *MapIterator) Valid() bool {\n\treturn mi.cursor.valid()\n}\n\nfunc (mi *MapIterator) Entry() (k Value, v Value) {\n\treturn mi.Key(), mi.Value()\n}\n\nfunc (mi *MapIterator) Key() Value {\n\tif !mi.cursor.valid() {\n\t\treturn nil\n\t}\n\treturn mi.cursor.current().(mapEntry).key\n}\n\nfunc (mi *MapIterator) Value() Value {\n\tif !mi.cursor.valid() {\n\t\treturn nil\n\t}\n\treturn mi.cursor.current().(mapEntry).value\n}\n\nfunc (mi *MapIterator) Position() uint64 {\n\tif !mi.cursor.valid() {\n\t\treturn 0\n\t}\n\treturn uint64(mi.cursor.idx)\n}\n\n// Prev returns the previous entry from the Map. If there is no previous entry, Prev() returns nils.\nfunc (mi *MapIterator) Prev() bool {\n\tif !mi.cursor.valid() {\n\t\treturn false\n\t}\n\treturn mi.cursor.retreat()\n}\n\n// Next returns the subsequent entries from the Map, starting with the entry at which the iterator\n// was created. If there are no more entries, Next() returns nils.\nfunc (mi *MapIterator) Next() bool {\n\tif !mi.cursor.valid() {\n\t\treturn false\n\t}\n\treturn mi.cursor.advance()\n}\n" }, { "path": "go/types/map_iterator_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"fmt\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestMapIterator(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tme := NewMap(vrw).Edit()\n\tfor i := 0; i < 5; i++ {\n\t\tme.Set(String(string(byte(65+i))), Number(i))\n\t}\n\n\tm := me.Map()\n\n\ttc := []struct {\n\t\treverse bool\n\t\titer bool\n\t\titerAt uint64\n\t\titerFrom string\n\t\texpected []string\n\t}{\n\t\t{false, true, 0, \"\", []string{\"A\", \"B\", \"C\", \"D\", \"E\"}},\n\t\t{false, false, 0, \"\", []string{\"A\", \"B\", \"C\", \"D\", \"E\"}},\n\t\t{false, false, 2, \"\", []string{\"C\", \"D\", \"E\"}},\n\t\t{false, false, 4, \"\", []string{\"E\"}},\n\t\t{false, false, 5, \"\", []string{}},\n\t\t{false, false, 0, \"A\", []string{\"A\", \"B\", \"C\", \"D\", \"E\"}},\n\t\t{false, false, 0, \"C\", []string{\"C\", \"D\", \"E\"}},\n\t\t{false, false, 4, \"E\", []string{\"E\"}},\n\t\t{false, false, 0, \"AA\", []string{\"B\", \"C\", \"D\", \"E\"}},\n\t\t{false, false, 0, \"F\", []string{}},\n\t\t{true, false, 0, \"\", []string{}},\n\t\t{true, true, 0, \"\", []string{}},\n\t\t{true, false, 2, \"\", []string{\"C\", \"B\", \"A\"}},\n\t\t{true, false, 4, \"\", []string{\"E\", \"D\", \"C\", \"B\", \"A\"}},\n\t\t{true, false, 5, \"\", []string{}},\n\t\t{true, false, 0, \"A\", []string{\"A\"}},\n\t\t{true, false, 0, \"C\", []string{\"C\", \"B\", \"A\"}},\n\t\t{true, false, 0, \"E\", []string{\"E\", \"D\", \"C\", \"B\", \"A\"}},\n\t\t{true, false, 0, \"AA\", []string{\"B\", \"A\"}},\n\t\t{true, false, 0, \"F\", []string{}},\n\t}\n\n\tfor i, t := range tc {\n\t\tlbl := fmt.Sprintf(\"test case %d\", i)\n\t\tvar it *MapIterator\n\t\tif t.iter {\n\t\t\tit = m.Iterator()\n\t\t} else if t.iterFrom != \"\" {\n\t\t\tit = m.IteratorFrom(String(t.iterFrom))\n\t\t} else {\n\t\t\tit = m.IteratorAt(t.iterAt)\n\t\t}\n\t\tfor i, e := range t.expected {\n\t\t\tlbl := fmt.Sprintf(\"%s: iteration %d\", lbl, i)\n\t\t\tassert.True(it.Valid(), lbl)\n\n\t\t\tassert.Equal(e, string(it.Key().(String)), lbl)\n\t\t\tassert.True(m.Get(it.Key()).Equals(it.Value()), lbl)\n\n\t\t\tk, v := it.Entry()\n\t\t\tassert.Equal(e, string(k.(String)), lbl)\n\t\t\tassert.True(m.Get(it.Key()).Equals(v), lbl)\n\n\t\t\tassert.True(m.Get(it.Key()).Equals(Number(it.Position())), lbl)\n\n\t\t\tvar last bool\n\t\t\tif t.reverse {\n\t\t\t\tlast = it.Prev()\n\t\t\t} else {\n\t\t\t\tlast = it.Next()\n\t\t\t}\n\t\t\tassert.Equal(i < len(t.expected)-1, last, lbl)\n\t\t\tassert.Equal(i < len(t.expected)-1, it.Valid(), lbl)\n\t\t}\n\t}\n}\n" }, { "path": "go/types/map_leaf_sequence.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype mapLeafSequence struct {\n\tleafSequence\n}\n\ntype mapEntry struct {\n\tkey Value\n\tvalue Value\n}\n\nfunc (entry mapEntry) writeTo(w nomsWriter) {\n\tentry.key.writeTo(w)\n\tentry.value.writeTo(w)\n}\n\nfunc readMapEntry(r *valueDecoder) mapEntry {\n\treturn mapEntry{r.readValue(), r.readValue()}\n}\n\nfunc (entry mapEntry) equals(other mapEntry) bool {\n\treturn entry.key.Equals(other.key) && entry.value.Equals(other.value)\n}\n\ntype mapEntrySlice []mapEntry\n\nfunc (mes mapEntrySlice) Len() int { return len(mes) }\nfunc (mes mapEntrySlice) Swap(i, j int) { mes[i], mes[j] = mes[j], mes[i] }\nfunc (mes mapEntrySlice) Less(i, j int) bool { return mes[i].key.Less(mes[j].key) }\nfunc (mes mapEntrySlice) Equals(other mapEntrySlice) bool {\n\tif mes.Len() != other.Len() {\n\t\treturn false\n\t}\n\n\tfor i, v := range mes {\n\t\tif !v.equals(other[i]) {\n\t\t\treturn false\n\t\t}\n\t}\n\n\treturn true\n}\n\nfunc newMapLeafSequence(vrw ValueReadWriter, data ...mapEntry) orderedSequence {\n\td.PanicIfTrue(vrw == nil)\n\toffsets := make([]uint32, len(data)+sequencePartValues+1)\n\tw := newBinaryNomsWriter()\n\toffsets[sequencePartKind] = w.offset\n\tMapKind.writeTo(&w)\n\toffsets[sequencePartLevel] = w.offset\n\tw.writeCount(0) // level\n\toffsets[sequencePartCount] = w.offset\n\tcount := uint64(len(data))\n\tw.writeCount(count)\n\toffsets[sequencePartValues] = w.offset\n\tfor i, me := range data {\n\t\tme.writeTo(&w)\n\t\toffsets[i+sequencePartValues+1] = w.offset\n\t}\n\treturn mapLeafSequence{newLeafSequence(vrw, w.data(), offsets, count)}\n}\n\nfunc (ml mapLeafSequence) writeTo(w nomsWriter) {\n\tw.writeRaw(ml.buff)\n}\n\n// sequence interface\n\nfunc (ml mapLeafSequence) getItem(idx int) sequenceItem {\n\tdec := ml.decoderSkipToIndex(idx)\n\treturn readMapEntry(&dec)\n}\n\nfunc (ml mapLeafSequence) WalkRefs(cb RefCallback) {\n\twalkRefs(ml.valueBytes(), cb)\n}\n\nfunc (ml mapLeafSequence) entries() mapEntrySlice {\n\tdec, count := ml.decoderSkipToValues()\n\tentries := make(mapEntrySlice, count)\n\tfor i := uint64(0); i < count; i++ {\n\t\tentries[i] = mapEntry{dec.readValue(), dec.readValue()}\n\t}\n\treturn entries\n}\n\nfunc (ml mapLeafSequence) getCompareFn(other sequence) compareFn {\n\tdec1 := ml.decoder()\n\tml2 := other.(mapLeafSequence)\n\tdec2 := ml2.decoder()\n\treturn func(idx, otherIdx int) bool {\n\t\tdec1.offset = uint32(ml.getItemOffset(idx))\n\t\tdec2.offset = uint32(ml2.getItemOffset(otherIdx))\n\t\tk1 := dec1.readValue()\n\t\tk2 := dec2.readValue()\n\t\tif !k1.Equals(k2) {\n\t\t\treturn false\n\t\t}\n\t\tv1 := dec1.readValue()\n\t\tv2 := dec2.readValue()\n\t\treturn v1.Equals(v2)\n\t}\n}\n\nfunc (ml mapLeafSequence) typeOf() *Type {\n\tdec, count := ml.decoderSkipToValues()\n\tkts := make(typeSlice, 0, count)\n\tvts := make(typeSlice, 0, count)\n\tvar lastKeyType, lastValueType *Type\n\tfor i := uint64(0); i < count; i++ {\n\t\tif lastKeyType != nil && lastValueType != nil {\n\t\t\toffset := dec.offset\n\t\t\tif dec.isValueSameTypeForSure(lastKeyType) && dec.isValueSameTypeForSure(lastValueType) {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tdec.offset = offset\n\n\t\t}\n\n\t\tlastKeyType = dec.readTypeOfValue()\n\t\tkts = append(kts, lastKeyType)\n\t\tlastValueType = dec.readTypeOfValue()\n\t\tvts = append(vts, lastValueType)\n\t}\n\n\treturn makeCompoundType(MapKind, makeUnionType(kts...), makeUnionType(vts...))\n}\n\n// orderedSequence interface\n\nfunc (ml mapLeafSequence) decoderSkipToIndex(idx int) valueDecoder {\n\toffset := ml.getItemOffset(idx)\n\treturn ml.decoderAtOffset(offset)\n}\n\nfunc (ml mapLeafSequence) getKey(idx int) orderedKey {\n\tdec := ml.decoderSkipToIndex(idx)\n\treturn newOrderedKey(dec.readValue())\n}\n\nfunc (ml mapLeafSequence) search(key orderedKey) int {\n\treturn sort.Search(int(ml.Len()), func(i int) bool {\n\t\treturn !ml.getKey(i).Less(key)\n\t})\n}\n\nfunc (ml mapLeafSequence) getValue(idx int) Value {\n\tdec := ml.decoderSkipToIndex(idx)\n\tdec.skipValue()\n\treturn dec.readValue()\n}\n" }, { "path": "go/types/map_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"math/rand\"\n\t\"sort\"\n\t\"sync\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst testMapSize = 8000\n\ntype genValueFn func(i int) Value\n\ntype testMap struct {\n\tentries mapEntrySlice\n\tknownBadKey Value\n}\n\nfunc (tm testMap) SetValue(i int, v Value) testMap {\n\tentries := make([]mapEntry, 0, len(tm.entries))\n\tentries = append(entries, tm.entries...)\n\tentries[i].value = v\n\treturn testMap{entries, tm.knownBadKey}\n}\n\nfunc (tm testMap) Remove(from, to int) testMap {\n\tentries := make([]mapEntry, 0, len(tm.entries)-(to-from))\n\tentries = append(entries, tm.entries[:from]...)\n\tentries = append(entries, tm.entries[to:]...)\n\treturn testMap{entries, tm.knownBadKey}\n}\n\nfunc (tm testMap) MaybeGet(key Value) (v Value, ok bool) {\n\tfor _, entry := range tm.entries {\n\t\tif entry.key.Equals(key) {\n\t\t\treturn entry.value, true\n\t\t}\n\t}\n\treturn nil, false\n}\n\nfunc (tm testMap) Diff(last testMap) (added []Value, removed []Value, modified []Value) {\n\t// Note: this could be use tm.toMap/last.toMap and then tmMap.Diff(lastMap) but the\n\t// purpose of this method is to be redundant.\n\tif len(tm.entries) == 0 && len(last.entries) == 0 {\n\t\treturn // nothing changed\n\t}\n\tif len(tm.entries) == 0 {\n\t\t// everything removed\n\t\tfor _, entry := range last.entries {\n\t\t\tremoved = append(removed, entry.key)\n\t\t}\n\t\treturn\n\t}\n\tif len(last.entries) == 0 {\n\t\t// everything added\n\t\tfor _, entry := range tm.entries {\n\t\t\tadded = append(added, entry.key)\n\t\t}\n\t\treturn\n\t}\n\n\tfor _, entry := range tm.entries {\n\t\totherValue, exists := last.MaybeGet(entry.key)\n\t\tif !exists {\n\t\t\tadded = append(added, entry.key)\n\t\t} else if !entry.value.Equals(otherValue) {\n\t\t\tmodified = append(modified, entry.key)\n\t\t}\n\t}\n\tfor _, entry := range last.entries {\n\t\t_, exists := tm.MaybeGet(entry.key)\n\t\tif !exists {\n\t\t\tremoved = append(removed, entry.key)\n\t\t}\n\t}\n\treturn\n}\n\nfunc (tm testMap) toMap(vrw ValueReadWriter) Map {\n\tkeyvals := []Value{}\n\tfor _, entry := range tm.entries {\n\t\tkeyvals = append(keyvals, entry.key, entry.value)\n\t}\n\treturn NewMap(vrw, keyvals...)\n}\n\nfunc toValuable(vs ValueSlice) []Valuable {\n\tvb := make([]Valuable, len(vs))\n\tfor i, v := range vs {\n\t\tvb[i] = v\n\t}\n\treturn vb\n}\n\nfunc (tm testMap) Flatten(from, to int) []Value {\n\tflat := make([]Value, 0, len(tm.entries)*2)\n\tfor _, entry := range tm.entries[from:to] {\n\t\tflat = append(flat, entry.key)\n\t\tflat = append(flat, entry.value)\n\t}\n\treturn flat\n}\n\nfunc (tm testMap) FlattenAll() []Value {\n\treturn tm.Flatten(0, len(tm.entries))\n}\n\nfunc newSortedTestMap(length int, gen genValueFn) testMap {\n\tkeys := make(ValueSlice, 0, length)\n\tfor i := 0; i < length; i++ {\n\t\tkeys = append(keys, gen(i))\n\t}\n\n\tsort.Sort(keys)\n\n\tentries := make([]mapEntry, 0, len(keys))\n\tfor i, k := range keys {\n\t\tentries = append(entries, mapEntry{k, Number(i * 2)})\n\t}\n\n\treturn testMap{entries, Number(length + 2)}\n}\n\nfunc newTestMapFromMap(m Map) testMap {\n\tentries := make([]mapEntry, 0, m.Len())\n\tm.IterAll(func(key, value Value) {\n\t\tentries = append(entries, mapEntry{key, value})\n\t})\n\treturn testMap{entries, Number(-0)}\n}\n\nfunc newRandomTestMap(length int, gen genValueFn) testMap {\n\ts := rand.NewSource(4242)\n\tused := map[int]bool{}\n\n\tmask := int(0xffffff)\n\tentries := make([]mapEntry, 0, length)\n\tfor len(entries) < length {\n\t\tv := int(s.Int63()) & mask\n\t\tif _, ok := used[v]; !ok {\n\t\t\tentry := mapEntry{gen(v), gen(v * 2)}\n\t\t\tentries = append(entries, entry)\n\t\t\tused[v] = true\n\t\t}\n\t}\n\n\treturn testMap{entries, gen(mask + 1)}\n}\n\nfunc validateMap(t *testing.T, vrw ValueReadWriter, m Map, entries mapEntrySlice) {\n\ttm := testMap{entries: entries}\n\tassert.True(t, m.Equals(tm.toMap(vrw)))\n\n\tout := mapEntrySlice{}\n\tm.IterAll(func(k Value, v Value) {\n\t\tout = append(out, mapEntry{k, v})\n\t})\n\n\tassert.True(t, out.Equals(entries))\n}\n\ntype mapTestSuite struct {\n\tcollectionTestSuite\n\telems testMap\n}\n\nfunc newMapTestSuite(size uint, expectChunkCount int, expectPrependChunkDiff int, expectAppendChunkDiff int, gen genValueFn) *mapTestSuite {\n\tvrw := newTestValueStore()\n\n\tlength := 1 << size\n\tkeyType := TypeOf(gen(0))\n\telems := newSortedTestMap(length, gen)\n\ttr := MakeMapType(keyType, NumberType)\n\ttmap := NewMap(vrw, elems.FlattenAll()...)\n\treturn &mapTestSuite{\n\t\tcollectionTestSuite: collectionTestSuite{\n\t\t\tcol: tmap,\n\t\t\texpectType: tr,\n\t\t\texpectLen: uint64(length),\n\t\t\texpectChunkCount: expectChunkCount,\n\t\t\texpectPrependChunkDiff: expectPrependChunkDiff,\n\t\t\texpectAppendChunkDiff: expectAppendChunkDiff,\n\t\t\tvalidate: func(v2 Collection) bool {\n\t\t\t\tif v2.Len() != uint64(elems.entries.Len()) {\n\t\t\t\t\tfmt.Println(\"lengths not equal:\", v2.Len(), elems.entries.Len())\n\t\t\t\t\treturn false\n\t\t\t\t}\n\t\t\t\tl2 := v2.(Map)\n\t\t\t\tidx := uint64(0)\n\t\t\t\tl2.Iter(func(key, value Value) (stop bool) {\n\t\t\t\t\tentry := elems.entries[idx]\n\t\t\t\t\tif !key.Equals(entry.key) {\n\t\t\t\t\t\tfmt.Printf(\"%d: %s (%s)\\n!=\\n%s (%s)\\n\", idx, EncodedValue(key), key.Hash(), EncodedValue(entry.key), entry.key.Hash())\n\t\t\t\t\t\tstop = true\n\t\t\t\t\t}\n\t\t\t\t\tif !value.Equals(entry.value) {\n\t\t\t\t\t\tfmt.Printf(\"%s (%s) !=\\n%s (%s)\\n\", EncodedValue(value), value.Hash(), EncodedValue(entry.value), entry.value.Hash())\n\t\t\t\t\t\tstop = true\n\t\t\t\t\t}\n\t\t\t\t\tidx++\n\t\t\t\t\treturn\n\t\t\t\t})\n\t\t\t\treturn idx == v2.Len()\n\t\t\t},\n\t\t\tprependOne: func() Collection {\n\t\t\t\tdup := make([]mapEntry, length+1)\n\t\t\t\tdup[0] = mapEntry{Number(-1), Number(-2)}\n\t\t\t\tcopy(dup[1:], elems.entries)\n\t\t\t\tflat := []Value{}\n\t\t\t\tfor _, entry := range dup {\n\t\t\t\t\tflat = append(flat, entry.key, entry.value)\n\t\t\t\t}\n\t\t\t\treturn NewMap(vrw, flat...)\n\t\t\t},\n\t\t\tappendOne: func() Collection {\n\t\t\t\tdup := make([]mapEntry, length+1)\n\t\t\t\tcopy(dup, elems.entries)\n\t\t\t\tdup[len(dup)-1] = mapEntry{Number(length*2 + 1), Number((length*2 + 1) * 2)}\n\t\t\t\tflat := []Value{}\n\t\t\t\tfor _, entry := range dup {\n\t\t\t\t\tflat = append(flat, entry.key, entry.value)\n\t\t\t\t}\n\t\t\t\treturn NewMap(vrw, flat...)\n\t\t\t},\n\t\t},\n\t\telems: elems,\n\t}\n}\n\nfunc (suite *mapTestSuite) createStreamingMap(vs *ValueStore) Map {\n\tkvChan := make(chan Value)\n\tmapChan := NewStreamingMap(vs, kvChan)\n\tfor _, entry := range suite.elems.entries {\n\t\tkvChan <- entry.key\n\t\tkvChan <- entry.value\n\t}\n\tclose(kvChan)\n\treturn <-mapChan\n}\n\nfunc (suite *mapTestSuite) TestStreamingMap() {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\tm := suite.createStreamingMap(vs)\n\tsuite.True(suite.validate(m), \"map not valid\")\n}\n\nfunc (suite *mapTestSuite) TestStreamingMapOrder() {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tentries := make(mapEntrySlice, len(suite.elems.entries))\n\tcopy(entries, suite.elems.entries)\n\tentries[0], entries[1] = entries[1], entries[0]\n\n\tkvChan := make(chan Value, len(entries)*2)\n\tfor _, e := range entries {\n\t\tkvChan <- e.key\n\t\tkvChan <- e.value\n\t}\n\tclose(kvChan)\n\n\treadInput := func(vrw ValueReadWriter, kvs <-chan Value, outChan chan<- Map) {\n\t\treadMapInput(vrw, kvs, outChan)\n\t}\n\n\ttestFunc := func() {\n\t\toutChan := newStreamingMap(vs, kvChan, readInput)\n\t\t<-outChan\n\t}\n\n\tsuite.Panics(testFunc)\n}\n\nfunc (suite *mapTestSuite) TestStreamingMap2() {\n\twg := sync.WaitGroup{}\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\twg.Add(2)\n\tvar m1, m2 Map\n\tgo func() {\n\t\tm1 = suite.createStreamingMap(vs)\n\t\twg.Done()\n\t}()\n\tgo func() {\n\t\tm2 = suite.createStreamingMap(vs)\n\t\twg.Done()\n\t}()\n\twg.Wait()\n\tsuite.True(suite.validate(m1), \"map 'm1' not valid\")\n\tsuite.True(suite.validate(m2), \"map 'm2' not valid\")\n}\n\nfunc TestMapSuite4K(t *testing.T) {\n\tsuite.Run(t, newMapTestSuite(12, 4, 2, 2, newNumber))\n}\n\nfunc TestMapSuite4KStructs(t *testing.T) {\n\tsuite.Run(t, newMapTestSuite(12, 11, 2, 2, newNumberStruct))\n}\n\nfunc newNumber(i int) Value {\n\treturn Number(i)\n}\n\nfunc newNumberStruct(i int) Value {\n\treturn NewStruct(\"\", StructData{\"n\": Number(i)})\n}\n\nfunc getTestNativeOrderMap(scale int, vrw ValueReadWriter) testMap {\n\treturn newRandomTestMap(64*scale, newNumber)\n}\n\nfunc getTestRefValueOrderMap(scale int, vrw ValueReadWriter) testMap {\n\treturn newRandomTestMap(64*scale, newNumber)\n}\n\nfunc getTestRefToNativeOrderMap(scale int, vrw ValueReadWriter) testMap {\n\treturn newRandomTestMap(64*scale, func(i int) Value {\n\t\treturn vrw.WriteValue(Number(i))\n\t})\n}\n\nfunc getTestRefToValueOrderMap(scale int, vrw ValueReadWriter) testMap {\n\treturn newRandomTestMap(64*scale, func(i int) Value {\n\t\treturn vrw.WriteValue(NewSet(vrw, Number(i)))\n\t})\n}\n\nfunc accumulateMapDiffChanges(m1, m2 Map) (added []Value, removed []Value, modified []Value) {\n\tchanges := make(chan ValueChanged)\n\tgo func() {\n\t\tm1.Diff(m2, changes, nil)\n\t\tclose(changes)\n\t}()\n\tfor change := range changes {\n\t\tif change.ChangeType == DiffChangeAdded {\n\t\t\tadded = append(added, change.Key)\n\t\t} else if change.ChangeType == DiffChangeRemoved {\n\t\t\tremoved = append(removed, change.Key)\n\t\t} else {\n\t\t\tmodified = append(modified, change.Key)\n\t\t}\n\t}\n\treturn\n}\n\nfunc diffMapTest(assert *assert.Assertions, m1 Map, m2 Map, numAddsExpected int, numRemovesExpected int, numModifiedExpected int) (added []Value, removed []Value, modified []Value) {\n\tadded, removed, modified = accumulateMapDiffChanges(m1, m2)\n\tassert.Equal(numAddsExpected, len(added), \"num added is not as expected\")\n\tassert.Equal(numRemovesExpected, len(removed), \"num removed is not as expected\")\n\tassert.Equal(numModifiedExpected, len(modified), \"num modified is not as expected\")\n\n\ttm1 := newTestMapFromMap(m1)\n\ttm2 := newTestMapFromMap(m2)\n\ttmAdded, tmRemoved, tmModified := tm1.Diff(tm2)\n\tassert.Equal(numAddsExpected, len(tmAdded), \"num added is not as expected\")\n\tassert.Equal(numRemovesExpected, len(tmRemoved), \"num removed is not as expected\")\n\tassert.Equal(numModifiedExpected, len(tmModified), \"num modified is not as expected\")\n\n\tassert.Equal(added, tmAdded, \"map added != tmMap added\")\n\tassert.Equal(removed, tmRemoved, \"map removed != tmMap removed\")\n\tassert.Equal(modified, tmModified, \"map modified != tmMap modified\")\n\treturn\n}\n\nfunc TestMapDiff(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\ttestMap1 := newRandomTestMap(64*2, newNumber)\n\ttestMap2 := newRandomTestMap(64*2, newNumber)\n\ttestMapAdded, testMapRemoved, testMapModified := testMap1.Diff(testMap2)\n\tmap1 := testMap1.toMap(vrw)\n\tmap2 := testMap2.toMap(vrw)\n\n\tmapDiffAdded, mapDiffRemoved, mapDiffModified := accumulateMapDiffChanges(map1, map2)\n\tassert.Equal(t, testMapAdded, mapDiffAdded, \"testMap.diff != map.diff\")\n\tassert.Equal(t, testMapRemoved, mapDiffRemoved, \"testMap.diff != map.diff\")\n\tassert.Equal(t, testMapModified, mapDiffModified, \"testMap.diff != map.diff\")\n}\n\nfunc TestMapMutationReadWriteCount(t *testing.T) {\n\t// This test is a sanity check that we are reading a \"reasonable\" number of\n\t// sequences while mutating maps.\n\t// TODO: We are currently un-reasonable.\n\ttemp := MakeStructTemplate(\"Foo\", []string{\"Bool\", \"Number\", \"String1\", \"String2\"})\n\n\tnewLargeStruct := func(i int) Value {\n\t\treturn temp.NewStruct([]Value{\n\t\t\tBool(i%2 == 0),\n\t\t\tNumber(i),\n\t\t\tString(fmt.Sprintf(\"I AM A REALLY REALY REALL SUPER CALIFRAGILISTICLY CRAZY-ASSED LONGTASTIC String %d\", i)),\n\t\t\tString(fmt.Sprintf(\"I am a bit shorter and also more chill: %d\", i)),\n\t\t})\n\t}\n\n\tts := &chunks.TestStorage{}\n\tcs := ts.NewView()\n\tvs := newValueStoreWithCacheAndPending(cs, 0, 0)\n\n\tme := NewMap(vs).Edit()\n\tfor i := 0; i < 10000; i++ {\n\t\tme.Set(Number(i), newLargeStruct(i))\n\t}\n\tm := me.Map()\n\tr := vs.WriteValue(m)\n\tvs.Commit(vs.Root(), vs.Root())\n\tm = r.TargetValue(vs).(Map)\n\n\tevery := 100\n\n\tme = m.Edit()\n\tfor i := 0; i < 10000; i++ {\n\t\tif i%every == 0 {\n\t\t\tk := Number(i)\n\t\t\ts := me.Get(Number(i)).(Struct)\n\t\t\ts = s.Set(\"Number\", Number(float64(s.Get(\"Number\").(Number))+1))\n\t\t\tme.Set(k, s)\n\t\t}\n\t\ti++\n\t}\n\n\tcs.Writes = 0\n\tcs.Reads = 0\n\n\tm = me.Map()\n\n\tvs.Commit(vs.Root(), vs.Root())\n\n\tassert.Equal(t, uint64(3), NewRef(m).Height())\n\tassert.Equal(t, 105, cs.Reads)\n\tassert.Equal(t, 62, cs.Writes)\n}\n\nfunc TestMapInfiniteChunkBug(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tkeyLen := chunkWindow + 1\n\n\tbuff := &bytes.Buffer{}\n\tfor i := uint32(0); i < keyLen; i++ {\n\t\tbuff.WriteString(\"s\")\n\t}\n\n\tprefix := buff.String()\n\n\tme := NewMap(vrw).Edit()\n\n\tfor i := 0; i < 10000; i++ {\n\t\tme.Set(String(prefix+fmt.Sprintf(\"%d\", i)), Number(i))\n\t}\n\n\tme.Map()\n}\n\nfunc TestNewMap(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tm := NewMap(vrw)\n\tassert.Equal(uint64(0), m.Len())\n\tm = NewMap(vrw, String(\"foo1\"), String(\"bar1\"), String(\"foo2\"), String(\"bar2\"))\n\tassert.Equal(uint64(2), m.Len())\n\tassert.True(String(\"bar1\").Equals(m.Get(String(\"foo1\"))))\n\tassert.True(String(\"bar2\").Equals(m.Get(String(\"foo2\"))))\n}\n\nfunc TestMapUniqueKeysString(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\tl := []Value{\n\t\tString(\"hello\"), String(\"world\"),\n\t\tString(\"foo\"), String(\"bar\"),\n\t\tString(\"bar\"), String(\"foo\"),\n\t\tString(\"hello\"), String(\"foo\"),\n\t}\n\tm := NewMap(vrw, l...)\n\tassert.Equal(uint64(3), m.Len())\n\tassert.True(String(\"foo\").Equals(m.Get(String(\"hello\"))))\n}\n\nfunc TestMapUniqueKeysNumber(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tl := []Value{\n\t\tNumber(4), Number(1),\n\t\tNumber(0), Number(2),\n\t\tNumber(1), Number(2),\n\t\tNumber(3), Number(4),\n\t\tNumber(1), Number(5),\n\t}\n\tm := NewMap(vrw, l...)\n\tassert.Equal(uint64(4), m.Len())\n\tassert.True(Number(5).Equals(m.Get(Number(1))))\n}\n\ntype toTestMapFunc func(scale int, vrw ValueReadWriter) testMap\n\nfunc TestMapHas(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\n\tdoTest := func(toTestMap toTestMapFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\ttm := toTestMap(scale, vrw)\n\t\tm := tm.toMap(vrw)\n\t\tm2 := vrw.ReadValue(vrw.WriteValue(m).TargetHash()).(Map)\n\t\tfor _, entry := range tm.entries {\n\t\t\tk, v := entry.key, entry.value\n\t\t\tassert.True(m.Has(k))\n\t\t\tassert.True(m.Get(k).Equals(v))\n\t\t\tassert.True(m2.Has(k))\n\t\t\tassert.True(m2.Get(k).Equals(v))\n\t\t}\n\t\tdiffMapTest(assert, m, m2, 0, 0, 0)\n\t}\n\n\tdoTest(getTestNativeOrderMap, 16)\n\tdoTest(getTestRefValueOrderMap, 2)\n\tdoTest(getTestRefToNativeOrderMap, 2)\n\tdoTest(getTestRefToValueOrderMap, 2)\n}\n\nfunc TestMapRemoveMasksUnderlyingMap(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tk := String(\"foo\")\n\tme := NewMap(vrw, k, String(\"bar\")).Edit()\n\tme.Remove(k)\n\tassert.False(me.Has(k))\n\tassert.Nil(me.Get(k))\n\n\ttwo := Number(2)\n\tme.Set(two, two)\n\tme.Remove(two)\n\tassert.False(me.Has(two))\n\tassert.Nil(me.Get(two))\n\n\tme2 := NewMap(vrw).Edit()\n\tthree := Number(3)\n\tme.Set(three, me2)\n\tme.Remove(three)\n\tassert.False(me.Has(three))\n\tassert.Nil(me.Get(three))\n}\n\nfunc TestMapHasRemove(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tme := NewMap(vrw).Edit()\n\tbothHave := func(k Value) bool {\n\t\tmeHas := me.Has(k)\n\t\tmHas := me.Map().Has(k)\n\t\tassert.Equal(meHas, mHas)\n\t\treturn meHas\n\t}\n\n\tassert.False(bothHave(String(\"a\")))\n\n\tme.Set(String(\"a\"), String(\"a\"))\n\tassert.True(bothHave(String(\"a\")))\n\n\tme.Remove(String(\"a\"))\n\tassert.False(bothHave(String(\"a\")))\n\n\tme.Set(String(\"a\"), String(\"a\"))\n\tassert.True(bothHave(String(\"a\")))\n\n\tme.Set(String(\"a\"), String(\"a\"))\n\tassert.True(bothHave(String(\"a\")))\n\n\t// In-order insertions\n\tme.Set(String(\"b\"), String(\"b\"))\n\tme.Set(String(\"c\"), String(\"c\"))\n\tassert.True(bothHave(String(\"a\")))\n\tassert.True(bothHave(String(\"b\")))\n\tassert.True(bothHave(String(\"c\")))\n\n\t// Out-of-order insertions\n\tme.Set(String(\"z\"), String(\"z\"))\n\tme.Set(String(\"y\"), String(\"y\"))\n\tassert.True(bothHave(String(\"z\")))\n\tassert.True(bothHave(String(\"y\")))\n\tassert.True(bothHave(String(\"a\")))\n\tassert.True(bothHave(String(\"b\")))\n\tassert.True(bothHave(String(\"c\")))\n\n\t// Removals\n\tme.Remove(String(\"z\")).Remove(String(\"y\")).Remove(String(\"a\")).Remove(String(\"b\")).Remove(String(\"c\")).Remove(String(\"never-inserted\"))\n\tassert.False(bothHave(String(\"z\")))\n\tassert.False(bothHave(String(\"y\")))\n\tassert.False(bothHave(String(\"a\")))\n\tassert.False(bothHave(String(\"b\")))\n\tassert.False(bothHave(String(\"c\")))\n\n\tm := me.Map()\n\tassert.True(m.Len() == 0)\n}\n\nfunc TestMapRemove(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\n\tdoTest := func(incr int, toTestMap toTestMapFunc, scale int) {\n\t\tvs := newTestValueStore()\n\t\ttm := toTestMap(scale, vs)\n\t\twhole := tm.toMap(vs)\n\t\trun := func(i int) {\n\t\t\texpected := tm.Remove(i, i+1).toMap(vs)\n\t\t\tactual := whole.Edit().Remove(tm.entries[i].key).Map()\n\t\t\tassert.Equal(expected.Len(), actual.Len())\n\t\t\tassert.True(expected.Equals(actual))\n\t\t\tdiffMapTest(assert, expected, actual, 0, 0, 0)\n\t\t}\n\t\tfor i := 0; i < len(tm.entries); i += incr {\n\t\t\trun(i)\n\t\t}\n\t\trun(len(tm.entries) - 1)\n\t}\n\n\tdoTest(128, getTestNativeOrderMap, 32)\n\tdoTest(64, getTestRefValueOrderMap, 4)\n\tdoTest(64, getTestRefToNativeOrderMap, 4)\n\tdoTest(64, getTestRefToValueOrderMap, 4)\n}\n\nfunc TestMapRemoveNonexistentKey(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\ttm := getTestNativeOrderMap(2, vrw)\n\toriginal := tm.toMap(vrw)\n\tactual := original.Edit().Remove(Number(-1)).Map() // rand.Int63 returns non-negative numbers.\n\n\tassert.Equal(original.Len(), actual.Len())\n\tassert.True(original.Equals(actual))\n}\n\nfunc TestMapFirst(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tm1 := NewMap(vrw)\n\tk, v := m1.First()\n\tassert.Nil(k)\n\tassert.Nil(v)\n\n\tm1 = m1.Edit().Set(String(\"foo\"), String(\"bar\")).Set(String(\"hot\"), String(\"dog\")).Map()\n\tak, av := m1.First()\n\tvar ek, ev Value\n\n\tm1.Iter(func(k, v Value) (stop bool) {\n\t\tek, ev = k, v\n\t\treturn true\n\t})\n\n\tassert.True(ek.Equals(ak))\n\tassert.True(ev.Equals(av))\n}\n\nfunc TestMapFirst2(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tdoTest := func(toTestMap toTestMapFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\ttm := toTestMap(scale, vrw)\n\t\tm := tm.toMap(vrw)\n\t\tsort.Stable(tm.entries)\n\t\tactualKey, actualValue := m.First()\n\t\tassert.True(tm.entries[0].key.Equals(actualKey))\n\t\tassert.True(tm.entries[0].value.Equals(actualValue))\n\t}\n\n\tdoTest(getTestNativeOrderMap, 16)\n\tdoTest(getTestRefValueOrderMap, 2)\n\tdoTest(getTestRefToNativeOrderMap, 2)\n\tdoTest(getTestRefToValueOrderMap, 2)\n}\n\nfunc TestMapLast(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tm1 := NewMap(vrw)\n\tk, v := m1.First()\n\tassert.Nil(k)\n\tassert.Nil(v)\n\n\tm1 = m1.Edit().Set(String(\"foo\"), String(\"bar\")).Set(String(\"hot\"), String(\"dog\")).Map()\n\tak, av := m1.Last()\n\tvar ek, ev Value\n\n\tm1.Iter(func(k, v Value) (stop bool) {\n\t\tek, ev = k, v\n\t\treturn false\n\t})\n\n\tassert.True(ek.Equals(ak))\n\tassert.True(ev.Equals(av))\n}\n\nfunc TestMapLast2(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tdoTest := func(toTestMap toTestMapFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\ttm := toTestMap(scale, vrw)\n\t\tm := tm.toMap(vrw)\n\t\tsort.Stable(tm.entries)\n\t\tactualKey, actualValue := m.Last()\n\t\tassert.True(tm.entries[len(tm.entries)-1].key.Equals(actualKey))\n\t\tassert.True(tm.entries[len(tm.entries)-1].value.Equals(actualValue))\n\t}\n\n\tdoTest(getTestNativeOrderMap, 16)\n\tdoTest(getTestRefValueOrderMap, 2)\n\tdoTest(getTestRefToNativeOrderMap, 2)\n\tdoTest(getTestRefToValueOrderMap, 2)\n}\n\nfunc TestMapSetGet(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tme := NewMap(vrw).Edit()\n\tbothAre := func(k Value) Value {\n\t\tmeV := me.Get(k)\n\t\tmV := me.Map().Get(k)\n\t\tassert.True((meV == nil && mV == nil) || meV.(Value).Equals(mV))\n\t\treturn mV\n\t}\n\n\tassert.Nil(bothAre(String(\"a\")))\n\n\tme.Set(String(\"a\"), Number(42))\n\tassert.True(Number(42).Equals(bothAre(String(\"a\"))))\n\n\tme.Set(String(\"a\"), Number(43))\n\tassert.True(Number(43).Equals(bothAre(String(\"a\"))))\n\n\tme.Remove(String(\"a\"))\n\tassert.Nil(bothAre(String(\"a\")))\n\n\t// in-order insertions\n\tme.Set(String(\"b\"), Number(43))\n\tme.Set(String(\"c\"), Number(44))\n\n\tassert.True(Number(43).Equals(bothAre(String(\"b\"))))\n\tassert.True(Number(44).Equals(bothAre(String(\"c\"))))\n\n\t// out-of-order insertions\n\tme.Set(String(\"z\"), Number(0))\n\tme.Set(String(\"y\"), Number(1))\n\n\tassert.True(Number(0).Equals(bothAre(String(\"z\"))))\n\tassert.True(Number(1).Equals(bothAre(String(\"y\"))))\n\n\t// removals\n\tme.Remove(String(\"z\"))\n\tme.Remove(String(\"a\"))\n\tme.Remove(String(\"y\"))\n\tme.Remove(String(\"b\"))\n\tme.Remove(String(\"c\"))\n\n\tassert.Nil(bothAre(String(\"a\")))\n\tassert.Nil(bothAre(String(\"b\")))\n\tassert.Nil(bothAre(String(\"c\")))\n\tassert.Nil(bothAre(String(\"y\")))\n\tassert.Nil(bothAre(String(\"z\")))\n\tassert.Nil(bothAre(String(\"never-inserted\")))\n\n\tm := me.Map()\n\tassert.True(m.Len() == 0)\n}\n\nfunc validateMapInsertion(t *testing.T, tm testMap) {\n\tvrw := newTestValueStore()\n\n\tallMe := NewMap(vrw).Edit()\n\tincrMe := NewMap(vrw).Edit()\n\n\tfor i, entry := range tm.entries {\n\t\tallMe.Set(entry.key, entry.value)\n\t\tincrMe.Set(entry.key, entry.value)\n\n\t\tm1 := allMe.Map()\n\t\tm2 := incrMe.Map()\n\n\t\tvalidateMap(t, vrw, m1, tm.entries[0:i+1])\n\t\tvalidateMap(t, vrw, m2, tm.entries[0:i+1])\n\n\t\tincrMe = m2.Edit()\n\t}\n}\n\nfunc TestMapValidateInsertAscending(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvalidateMapInsertion(t, newSortedTestMap(300, newNumber))\n}\n\nfunc TestMapSet(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\n\tdoTest := func(incr, offset int, toTestMap toTestMapFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\ttm := toTestMap(scale, vrw)\n\t\texpected := tm.toMap(vrw)\n\t\trun := func(from, to int) {\n\t\t\tactual := tm.Remove(from, to).toMap(vrw).Edit().SetM(toValuable(tm.Flatten(from, to))...).Map()\n\t\t\tassert.Equal(expected.Len(), actual.Len())\n\t\t\tassert.True(expected.Equals(actual))\n\t\t\tdiffMapTest(assert, expected, actual, 0, 0, 0)\n\t\t}\n\t\tfor i := 0; i < len(tm.entries)-offset; i += incr {\n\t\t\trun(i, i+offset)\n\t\t}\n\t\trun(len(tm.entries)-offset, len(tm.entries))\n\t}\n\n\tdoTest(18, 3, getTestNativeOrderMap, 9)\n\tdoTest(128, 1, getTestNativeOrderMap, 32)\n\tdoTest(64, 1, getTestRefValueOrderMap, 4)\n\tdoTest(64, 1, getTestRefToNativeOrderMap, 4)\n\tdoTest(64, 1, getTestRefToValueOrderMap, 4)\n}\n\nfunc TestMapSetM(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tm1 := NewMap(vrw)\n\tm2 := m1.Edit().SetM().Map()\n\tassert.True(m1.Equals(m2))\n\tm3 := m2.Edit().SetM(String(\"foo\"), String(\"bar\"), String(\"hot\"), String(\"dog\")).Map()\n\tassert.Equal(uint64(2), m3.Len())\n\tassert.True(String(\"bar\").Equals(m3.Get(String(\"foo\"))))\n\tassert.True(String(\"dog\").Equals(m3.Get(String(\"hot\"))))\n\tm4 := m3.Edit().SetM(String(\"mon\"), String(\"key\")).Map()\n\tassert.Equal(uint64(2), m3.Len())\n\tassert.Equal(uint64(3), m4.Len())\n}\n\nfunc TestMapSetExistingKeyToNewValue(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\tassert := assert.New(t)\n\n\ttm := getTestNativeOrderMap(2, vrw)\n\toriginal := tm.toMap(vrw)\n\n\texpectedWorking := tm\n\tactual := original\n\tfor i, entry := range tm.entries {\n\t\tnewValue := Number(int64(entry.value.(Number)) + 1)\n\t\texpectedWorking = expectedWorking.SetValue(i, newValue)\n\t\tactual = actual.Edit().Set(entry.key, newValue).Map()\n\t}\n\n\texpected := expectedWorking.toMap(vrw)\n\tassert.Equal(expected.Len(), actual.Len())\n\tassert.True(expected.Equals(actual))\n\tassert.False(original.Equals(actual))\n\tdiffMapTest(assert, expected, actual, 0, 0, 0)\n}\n\n// BUG 98\nfunc TestMapDuplicateSet(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tm1 := NewMap(vrw, Bool(true), Bool(true), Number(42), Number(42), Number(42), Number(42))\n\tassert.Equal(uint64(2), m1.Len())\n}\n\nfunc TestMapMaybeGet(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\n\tdoTest := func(toTestMap toTestMapFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\ttm := toTestMap(scale, vrw)\n\t\tm := tm.toMap(vrw)\n\t\tfor _, entry := range tm.entries {\n\t\t\tv, ok := m.MaybeGet(entry.key)\n\t\t\tif assert.True(ok, \"%v should have been in the map!\", entry.key) {\n\t\t\t\tassert.True(v.Equals(entry.value), \"%v != %v\", v, entry.value)\n\t\t\t}\n\t\t}\n\t\t_, ok := m.MaybeGet(tm.knownBadKey)\n\t\tassert.False(ok, \"m should not contain %v\", tm.knownBadKey)\n\t}\n\n\tdoTest(getTestNativeOrderMap, 2)\n\tdoTest(getTestRefValueOrderMap, 2)\n\tdoTest(getTestRefToNativeOrderMap, 2)\n\tdoTest(getTestRefToValueOrderMap, 2)\n}\n\nfunc TestMapIter(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tm := NewMap(vrw)\n\n\ttype entry struct {\n\t\tkey Value\n\t\tvalue Value\n\t}\n\n\ttype resultList []entry\n\tresults := resultList{}\n\tgot := func(key, val Value) bool {\n\t\tfor _, r := range results {\n\t\t\tif key.Equals(r.key) && val.Equals(r.value) {\n\t\t\t\treturn true\n\t\t\t}\n\t\t}\n\t\treturn false\n\t}\n\n\tstop := false\n\tcb := func(k, v Value) bool {\n\t\tresults = append(results, entry{k, v})\n\t\treturn stop\n\t}\n\n\tm.Iter(cb)\n\tassert.Equal(0, len(results))\n\n\tm = m.Edit().Set(String(\"a\"), Number(0)).Set(String(\"b\"), Number(1)).Map()\n\tm.Iter(cb)\n\tassert.Equal(2, len(results))\n\tassert.True(got(String(\"a\"), Number(0)))\n\tassert.True(got(String(\"b\"), Number(1)))\n\n\tresults = resultList{}\n\tstop = true\n\tm.Iter(cb)\n\tassert.Equal(1, len(results))\n\t// Iteration order not guaranteed, but it has to be one of these.\n\tassert.True(got(String(\"a\"), Number(0)) || got(String(\"b\"), Number(1)))\n}\n\nfunc TestMapIter2(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tdoTest := func(toTestMap toTestMapFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\ttm := toTestMap(scale, vrw)\n\t\tm := tm.toMap(vrw)\n\t\tsort.Sort(tm.entries)\n\t\tidx := uint64(0)\n\t\tendAt := uint64(64)\n\n\t\tm.Iter(func(k, v Value) (done bool) {\n\t\t\tassert.True(tm.entries[idx].key.Equals(k))\n\t\t\tassert.True(tm.entries[idx].value.Equals(v))\n\t\t\tif idx == endAt {\n\t\t\t\tdone = true\n\t\t\t}\n\t\t\tidx++\n\t\t\treturn\n\t\t})\n\n\t\tassert.Equal(endAt, idx-1)\n\t}\n\n\tdoTest(getTestNativeOrderMap, 16)\n\tdoTest(getTestRefValueOrderMap, 2)\n\tdoTest(getTestRefToNativeOrderMap, 2)\n\tdoTest(getTestRefToValueOrderMap, 2)\n}\n\nfunc TestMapAny(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tp := func(k, v Value) bool {\n\t\treturn k.Equals(String(\"foo\")) && v.Equals(String(\"bar\"))\n\t}\n\n\tassert.False(NewMap(vrw).Any(p))\n\tassert.False(NewMap(vrw, String(\"foo\"), String(\"baz\")).Any(p))\n\tassert.True(NewMap(vrw, String(\"foo\"), String(\"bar\")).Any(p))\n}\n\nfunc TestMapIterAll(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\n\tdoTest := func(toTestMap toTestMapFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\ttm := toTestMap(scale, vrw)\n\t\tm := tm.toMap(vrw)\n\t\tsort.Sort(tm.entries)\n\t\tidx := uint64(0)\n\n\t\tm.IterAll(func(k, v Value) {\n\t\t\tassert.True(tm.entries[idx].key.Equals(k))\n\t\t\tassert.True(tm.entries[idx].value.Equals(v))\n\t\t\tidx++\n\t\t})\n\t}\n\n\tdoTest(getTestNativeOrderMap, 16)\n\tdoTest(getTestRefValueOrderMap, 2)\n\tdoTest(getTestRefToNativeOrderMap, 2)\n\tdoTest(getTestRefToValueOrderMap, 2)\n}\n\nfunc TestMapEquals(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tm1 := NewMap(vrw)\n\tm2 := m1\n\tm3 := NewMap(vrw)\n\n\tassert.True(m1.Equals(m2))\n\tassert.True(m2.Equals(m1))\n\tassert.True(m3.Equals(m2))\n\tassert.True(m2.Equals(m3))\n\tdiffMapTest(assert, m1, m2, 0, 0, 0)\n\tdiffMapTest(assert, m1, m3, 0, 0, 0)\n\tdiffMapTest(assert, m2, m1, 0, 0, 0)\n\tdiffMapTest(assert, m2, m3, 0, 0, 0)\n\tdiffMapTest(assert, m3, m1, 0, 0, 0)\n\tdiffMapTest(assert, m3, m2, 0, 0, 0)\n\n\tm1 = NewMap(vrw, String(\"foo\"), Number(0.0), String(\"bar\"), NewList(vrw))\n\tm2 = m2.Edit().Set(String(\"foo\"), Number(0.0)).Set(String(\"bar\"), NewList(vrw)).Map()\n\tassert.True(m1.Equals(m2))\n\tassert.True(m2.Equals(m1))\n\tassert.False(m2.Equals(m3))\n\tassert.False(m3.Equals(m2))\n\tdiffMapTest(assert, m1, m2, 0, 0, 0)\n\tdiffMapTest(assert, m1, m3, 2, 0, 0)\n\tdiffMapTest(assert, m2, m1, 0, 0, 0)\n\tdiffMapTest(assert, m2, m3, 2, 0, 0)\n\tdiffMapTest(assert, m3, m1, 0, 2, 0)\n\tdiffMapTest(assert, m3, m2, 0, 2, 0)\n}\n\nfunc TestMapNotStringKeys(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tb1 := NewBlob(vrw, bytes.NewBufferString(\"blob1\"))\n\tb2 := NewBlob(vrw, bytes.NewBufferString(\"blob2\"))\n\tl := []Value{\n\t\tBool(true), String(\"true\"),\n\t\tBool(false), String(\"false\"),\n\t\tNumber(1), String(\"Number: 1\"),\n\t\tNumber(0), String(\"Number: 0\"),\n\t\tb1, String(\"blob1\"),\n\t\tb2, String(\"blob2\"),\n\t\tNewList(vrw), String(\"empty list\"),\n\t\tNewList(vrw, NewList(vrw)), String(\"list of list\"),\n\t\tNewMap(vrw), String(\"empty map\"),\n\t\tNewMap(vrw, NewMap(vrw), NewMap(vrw)), String(\"map of map/map\"),\n\t\tNewSet(vrw), String(\"empty set\"),\n\t\tNewSet(vrw, NewSet(vrw)), String(\"map of set/set\"),\n\t}\n\tm1 := NewMap(vrw, l...)\n\tassert.Equal(uint64(12), m1.Len())\n\tfor i := 0; i < len(l); i += 2 {\n\t\tassert.True(m1.Get(l[i]).Equals(l[i+1]))\n\t}\n\tassert.Nil(m1.Get(Number(42)))\n}\n\nfunc testMapOrder(assert *assert.Assertions, vrw ValueReadWriter, keyType, valueType *Type, tuples []Value, expectOrdering []Value) {\n\tm := NewMap(vrw, tuples...)\n\ti := 0\n\tm.IterAll(func(key, value Value) {\n\t\tassert.Equal(expectOrdering[i].Hash().String(), key.Hash().String())\n\t\ti++\n\t})\n}\n\nfunc TestMapOrdering(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvrw := newTestValueStore()\n\n\ttestMapOrder(assert, vrw,\n\t\tStringType, StringType,\n\t\t[]Value{\n\t\t\tString(\"a\"), String(\"unused\"),\n\t\t\tString(\"z\"), String(\"unused\"),\n\t\t\tString(\"b\"), String(\"unused\"),\n\t\t\tString(\"y\"), String(\"unused\"),\n\t\t\tString(\"c\"), String(\"unused\"),\n\t\t\tString(\"x\"), String(\"unused\"),\n\t\t},\n\t\t[]Value{\n\t\t\tString(\"a\"),\n\t\t\tString(\"b\"),\n\t\t\tString(\"c\"),\n\t\t\tString(\"x\"),\n\t\t\tString(\"y\"),\n\t\t\tString(\"z\"),\n\t\t},\n\t)\n\n\ttestMapOrder(assert, vrw,\n\t\tNumberType, StringType,\n\t\t[]Value{\n\t\t\tNumber(0), String(\"unused\"),\n\t\t\tNumber(1000), String(\"unused\"),\n\t\t\tNumber(1), String(\"unused\"),\n\t\t\tNumber(100), String(\"unused\"),\n\t\t\tNumber(2), String(\"unused\"),\n\t\t\tNumber(10), String(\"unused\"),\n\t\t},\n\t\t[]Value{\n\t\t\tNumber(0),\n\t\t\tNumber(1),\n\t\t\tNumber(2),\n\t\t\tNumber(10),\n\t\t\tNumber(100),\n\t\t\tNumber(1000),\n\t\t},\n\t)\n\n\ttestMapOrder(assert, vrw,\n\t\tNumberType, StringType,\n\t\t[]Value{\n\t\t\tNumber(0), String(\"unused\"),\n\t\t\tNumber(-30), String(\"unused\"),\n\t\t\tNumber(25), String(\"unused\"),\n\t\t\tNumber(1002), String(\"unused\"),\n\t\t\tNumber(-5050), String(\"unused\"),\n\t\t\tNumber(23), String(\"unused\"),\n\t\t},\n\t\t[]Value{\n\t\t\tNumber(-5050),\n\t\t\tNumber(-30),\n\t\t\tNumber(0),\n\t\t\tNumber(23),\n\t\t\tNumber(25),\n\t\t\tNumber(1002),\n\t\t},\n\t)\n\n\ttestMapOrder(assert, vrw,\n\t\tNumberType, StringType,\n\t\t[]Value{\n\t\t\tNumber(0.0001), String(\"unused\"),\n\t\t\tNumber(0.000001), String(\"unused\"),\n\t\t\tNumber(1), String(\"unused\"),\n\t\t\tNumber(25.01e3), String(\"unused\"),\n\t\t\tNumber(-32.231123e5), String(\"unused\"),\n\t\t\tNumber(23), String(\"unused\"),\n\t\t},\n\t\t[]Value{\n\t\t\tNumber(-32.231123e5),\n\t\t\tNumber(0.000001),\n\t\t\tNumber(0.0001),\n\t\t\tNumber(1),\n\t\t\tNumber(23),\n\t\t\tNumber(25.01e3),\n\t\t},\n\t)\n\n\ttestMapOrder(assert, vrw,\n\t\tValueType, StringType,\n\t\t[]Value{\n\t\t\tString(\"a\"), String(\"unused\"),\n\t\t\tString(\"z\"), String(\"unused\"),\n\t\t\tString(\"b\"), String(\"unused\"),\n\t\t\tString(\"y\"), String(\"unused\"),\n\t\t\tString(\"c\"), String(\"unused\"),\n\t\t\tString(\"x\"), String(\"unused\"),\n\t\t},\n\t\t[]Value{\n\t\t\tString(\"a\"),\n\t\t\tString(\"b\"),\n\t\t\tString(\"c\"),\n\t\t\tString(\"x\"),\n\t\t\tString(\"y\"),\n\t\t\tString(\"z\"),\n\t\t},\n\t)\n\n\ttestMapOrder(assert, vrw,\n\t\tBoolType, StringType,\n\t\t[]Value{\n\t\t\tBool(true), String(\"unused\"),\n\t\t\tBool(false), String(\"unused\"),\n\t\t},\n\t\t[]Value{\n\t\t\tBool(false),\n\t\t\tBool(true),\n\t\t},\n\t)\n}\n\nfunc TestMapEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tme := NewMap(vrw).Edit()\n\tempty := func() bool {\n\t\treturn me.Map().Empty()\n\t}\n\n\tassert.True(empty())\n\tme.Set(Bool(false), String(\"hi\"))\n\tassert.False(empty())\n\tme.Set(NewList(vrw), NewMap(vrw))\n\tassert.False(empty())\n}\n\nfunc TestMapType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\temptyMapType := MakeMapType(MakeUnionType(), MakeUnionType())\n\tm := NewMap(vrw)\n\tassert.True(TypeOf(m).Equals(emptyMapType))\n\n\tm2 := m.Edit().Remove(String(\"B\")).Map()\n\tassert.True(emptyMapType.Equals(TypeOf(m2)))\n\n\ttr := MakeMapType(StringType, NumberType)\n\tm2 = m.Edit().Set(String(\"A\"), Number(1)).Map()\n\tassert.True(tr.Equals(TypeOf(m2)))\n\n\tm2 = m.Edit().Set(String(\"B\"), Number(2)).Set(String(\"C\"), Number(2)).Map()\n\tassert.True(tr.Equals(TypeOf(m2)))\n\n\tm3 := m2.Edit().Set(String(\"A\"), Bool(true)).Map()\n\tassert.True(MakeMapType(StringType, MakeUnionType(BoolType, NumberType)).Equals(TypeOf(m3)), TypeOf(m3).Describe())\n\tm4 := m3.Edit().Set(Bool(true), Number(1)).Map()\n\tassert.True(MakeMapType(MakeUnionType(BoolType, StringType), MakeUnionType(BoolType, NumberType)).Equals(TypeOf(m4)))\n}\n\nfunc TestMapChunks(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tl1 := NewMap(vrw, Number(0), Number(1))\n\tc1 := getChunks(l1)\n\tassert.Len(c1, 0)\n\n\tl2 := NewMap(vrw, NewRef(Number(0)), Number(1))\n\tc2 := getChunks(l2)\n\tassert.Len(c2, 1)\n\n\tl3 := NewMap(vrw, Number(0), NewRef(Number(1)))\n\tc3 := getChunks(l3)\n\tassert.Len(c3, 1)\n}\n\nfunc TestMapFirstNNumbers(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tkvs := []Value{}\n\tfor i := 0; i < testMapSize; i++ {\n\t\tkvs = append(kvs, Number(i), Number(i+1))\n\t}\n\n\tm := NewMap(vrw, kvs...)\n\tassert.Equal(deriveCollectionHeight(m), getRefHeightOfCollection(m))\n}\n\nfunc TestMapRefOfStructFirstNNumbers(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tkvs := []Value{}\n\tfor i := 0; i < testMapSize; i++ {\n\t\tk := vs.WriteValue(NewStruct(\"num\", StructData{\"n\": Number(i)}))\n\t\tv := vs.WriteValue(NewStruct(\"num\", StructData{\"n\": Number(i + 1)}))\n\t\tassert.NotNil(k)\n\t\tassert.NotNil(v)\n\t\tkvs = append(kvs, k, v)\n\t}\n\n\tm := NewMap(vs, kvs...)\n\t// height + 1 because the leaves are Ref values (with height 1).\n\tassert.Equal(deriveCollectionHeight(m)+1, getRefHeightOfCollection(m))\n}\n\nfunc TestMapModifyAfterRead(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvs := newTestValueStore()\n\tm := getTestNativeOrderMap(2, vs).toMap(vs)\n\t// Drop chunk values.\n\tm = vs.ReadValue(vs.WriteValue(m).TargetHash()).(Map)\n\t// Modify/query. Once upon a time this would crash.\n\tfst, fstval := m.First()\n\tm = m.Edit().Remove(fst).Map()\n\tassert.False(m.Has(fst))\n\n\tfst2, _ := m.First()\n\tassert.True(m.Has(fst2))\n\n\tm = m.Edit().Set(fst, fstval).Map()\n\tassert.True(m.Has(fst))\n}\n\nfunc TestMapTypeAfterMutations(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\ttest := func(n int, c interface{}) {\n\t\tvalues := make([]Value, 2*n)\n\t\tfor i := 0; i < n; i++ {\n\t\t\tvalues[2*i] = Number(i)\n\t\t\tvalues[2*i+1] = Number(i)\n\t\t}\n\n\t\tm := NewMap(vrw, values...)\n\t\tassert.Equal(m.Len(), uint64(n))\n\t\tassert.IsType(c, m.asSequence())\n\t\tassert.True(TypeOf(m).Equals(MakeMapType(NumberType, NumberType)))\n\n\t\tm = m.Edit().Set(String(\"a\"), String(\"a\")).Map()\n\t\tassert.Equal(m.Len(), uint64(n+1))\n\t\tassert.IsType(c, m.asSequence())\n\t\tassert.True(TypeOf(m).Equals(MakeMapType(MakeUnionType(NumberType, StringType), MakeUnionType(NumberType, StringType))))\n\n\t\tm = m.Edit().Remove(String(\"a\")).Map()\n\t\tassert.Equal(m.Len(), uint64(n))\n\t\tassert.IsType(c, m.asSequence())\n\t\tassert.True(TypeOf(m).Equals(MakeMapType(NumberType, NumberType)))\n\t}\n\n\ttest(10, mapLeafSequence{})\n\ttest(8000, metaSequence{})\n}\n\nfunc TestCompoundMapWithValuesOfEveryType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tv := Number(42)\n\tkvs := []Value{\n\t\t// Values\n\t\tBool(true), v,\n\t\tNumber(0), v,\n\t\tString(\"hello\"), v,\n\t\tNewBlob(vrw, bytes.NewBufferString(\"buf\")), v,\n\t\tNewSet(vrw, Bool(true)), v,\n\t\tNewList(vrw, Bool(true)), v,\n\t\tNewMap(vrw, Bool(true), Number(0)), v,\n\t\tNewStruct(\"\", StructData{\"field\": Bool(true)}), v,\n\t\t// Refs of values\n\t\tNewRef(Bool(true)), v,\n\t\tNewRef(Number(0)), v,\n\t\tNewRef(String(\"hello\")), v,\n\t\tNewRef(NewBlob(vrw, bytes.NewBufferString(\"buf\"))), v,\n\t\tNewRef(NewSet(vrw, Bool(true))), v,\n\t\tNewRef(NewList(vrw, Bool(true))), v,\n\t\tNewRef(NewMap(vrw, Bool(true), Number(0))), v,\n\t\tNewRef(NewStruct(\"\", StructData{\"field\": Bool(true)})), v,\n\t}\n\n\tm := NewMap(vrw, kvs...)\n\tfor i := 1; m.asSequence().isLeaf(); i++ {\n\t\tk := Number(i)\n\t\tkvs = append(kvs, k, v)\n\t\tm = m.Edit().Set(k, v).Map()\n\t}\n\n\tassert.Equal(len(kvs)/2, int(m.Len()))\n\tfk, fv := m.First()\n\tassert.True(bool(fk.(Bool)))\n\tassert.True(v.Equals(fv))\n\n\tfor i, keyOrValue := range kvs {\n\t\tif i%2 == 0 {\n\t\t\tassert.True(m.Has(keyOrValue))\n\t\t\tassert.True(v.Equals(m.Get(keyOrValue)))\n\t\t} else {\n\t\t\tassert.True(v.Equals(keyOrValue))\n\t\t}\n\t}\n\n\tfor len(kvs) > 0 {\n\t\tk := kvs[0]\n\t\tkvs = kvs[2:]\n\t\tm = m.Edit().Remove(k).Map()\n\t\tassert.False(m.Has(k))\n\t\tassert.Equal(len(kvs)/2, int(m.Len()))\n\t}\n}\n\nfunc TestMapRemoveLastWhenNotLoaded(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvs := newTestValueStore()\n\treload := func(m Map) Map {\n\t\treturn vs.ReadValue(vs.WriteValue(m).TargetHash()).(Map)\n\t}\n\n\ttm := getTestNativeOrderMap(4, vs)\n\tnm := tm.toMap(vs)\n\n\tfor len(tm.entries) > 0 {\n\t\tentr := tm.entries\n\t\tlast := entr[len(entr)-1]\n\t\tentr = entr[:len(entr)-1]\n\t\ttm.entries = entr\n\t\tnm = reload(nm.Edit().Remove(last.key).Map())\n\t\tassert.True(tm.toMap(vs).Equals(nm))\n\t}\n}\n\nfunc TestMapIterFrom(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\ttest := func(m Map, start, end Value) ValueSlice {\n\t\tres := ValueSlice{}\n\t\tm.IterFrom(start, func(k, v Value) bool {\n\t\t\tif end.Less(k) {\n\t\t\t\treturn true\n\t\t\t}\n\t\t\tres = append(res, k, v)\n\t\t\treturn false\n\t\t})\n\t\treturn res\n\t}\n\n\tkvs := generateNumbersAsValuesFromToBy(-50, 50, 1)\n\tm1 := NewMap(vrw, kvs...)\n\tassert.True(kvs.Equals(test(m1, nil, Number(1000))))\n\tassert.True(kvs.Equals(test(m1, Number(-1000), Number(1000))))\n\tassert.True(kvs.Equals(test(m1, Number(-50), Number(1000))))\n\tassert.True(kvs[2:].Equals(test(m1, Number(-49), Number(1000))))\n\tassert.True(kvs[2:].Equals(test(m1, Number(-48), Number(1000))))\n\tassert.True(kvs[4:].Equals(test(m1, Number(-47), Number(1000))))\n\tassert.True(kvs[98:].Equals(test(m1, Number(48), Number(1000))))\n\tassert.True(kvs[0:0].Equals(test(m1, Number(100), Number(1000))))\n\tassert.True(kvs[50:60].Equals(test(m1, Number(0), Number(8))))\n}\n\nfunc TestMapAt(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tvalues := []Value{Bool(false), Number(42), String(\"a\"), String(\"b\"), String(\"c\"), String(\"d\")}\n\tm := NewMap(vrw, values...)\n\n\tfor i := 0; i < len(values); i += 2 {\n\t\tk, v := m.At(uint64(i / 2))\n\t\tassert.Equal(values[i], k)\n\t\tassert.Equal(values[i+1], v)\n\t}\n\n\tassert.Panics(func() {\n\t\tm.At(42)\n\t})\n}\n\nfunc TestMapWithStructShouldHaveOptionalFields(t *testing.T) {\n\tassert := assert.New(t)\n\tvrw := newTestValueStore()\n\n\tlist := NewMap(vrw,\n\t\tString(\"one\"),\n\t\tNewStruct(\"Foo\", StructData{\n\t\t\t\"a\": Number(1),\n\t\t}),\n\t\tString(\"two\"),\n\t\tNewStruct(\"Foo\", StructData{\n\t\t\t\"a\": Number(2),\n\t\t\t\"b\": String(\"bar\"),\n\t\t}),\n\t)\n\tassert.True(\n\t\tMakeMapType(StringType,\n\t\t\tMakeStructType(\"Foo\",\n\t\t\t\tStructField{\"a\", NumberType, false},\n\t\t\t\tStructField{\"b\", StringType, true},\n\t\t\t),\n\t\t).Equals(TypeOf(list)))\n\n\t// transpose\n\tlist = NewMap(vrw,\n\t\tNewStruct(\"Foo\", StructData{\n\t\t\t\"a\": Number(1),\n\t\t}),\n\t\tString(\"one\"),\n\t\tNewStruct(\"Foo\", StructData{\n\t\t\t\"a\": Number(2),\n\t\t\t\"b\": String(\"bar\"),\n\t\t}),\n\t\tString(\"two\"),\n\t)\n\tassert.True(\n\t\tMakeMapType(\n\t\t\tMakeStructType(\"Foo\",\n\t\t\t\tStructField{\"a\", NumberType, false},\n\t\t\t\tStructField{\"b\", StringType, true},\n\t\t\t),\n\t\t\tStringType,\n\t\t).Equals(TypeOf(list)))\n\n}\n\nfunc TestMapWithNil(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tassert.Panics(t, func() {\n\t\tNewMap(nil, Number(42))\n\t})\n\tassert.Panics(t, func() {\n\t\tNewSet(vrw, Number(42), nil)\n\t})\n\tassert.Panics(t, func() {\n\t\tNewMap(vrw, String(\"a\"), String(\"b\"), nil, Number(42))\n\t})\n\tassert.Panics(t, func() {\n\t\tNewSet(vrw, String(\"a\"), String(\"b\"), Number(42), nil)\n\t})\n}\n\nfunc TestNestedEditing(t *testing.T) {\n\tvrw := newTestValueStore()\n\n\tme0 := NewMap(vrw).Edit()\n\n\t// m.a.a\n\tme1a := NewMap(vrw).Edit()\n\tme0.Set(String(\"a\"), me1a)\n\tse2a := NewSet(vrw).Edit()\n\tme1a.Set(String(\"a\"), se2a)\n\tse2a.Insert(String(\"a\"))\n\n\t// m.b.b\n\tme1b := NewMap(vrw).Edit()\n\tme0.Set(String(\"b\"), me1b)\n\tse2b := NewSet(vrw).Edit()\n\tme1b.Set(String(\"b\"), se2b)\n\tse2b.Insert(String(\"b\"))\n\n\tmOut := me0.Map()\n\tassert.True(t, mOut.Equals(NewMap(vrw,\n\t\tString(\"a\"), NewMap(vrw,\n\t\t\tString(\"a\"), NewSet(vrw, String(\"a\")),\n\t\t),\n\t\tString(\"b\"), NewMap(vrw,\n\t\t\tString(\"b\"), NewSet(vrw, String(\"b\")),\n\t\t),\n\t)))\n\n\tse2a.Remove(String(\"a\")).Insert(String(\"aa\"))\n\tse2b.Remove(String(\"b\")).Insert(String(\"bb\"))\n\n\tmOut = me0.Map()\n\tassert.True(t, mOut.Equals(NewMap(vrw,\n\t\tString(\"a\"), NewMap(vrw,\n\t\t\tString(\"a\"), NewSet(vrw, String(\"aa\")),\n\t\t),\n\t\tString(\"b\"), NewMap(vrw,\n\t\t\tString(\"b\"), NewSet(vrw, String(\"bb\")),\n\t\t),\n\t)))\n\n\tse2a.Remove(String(\"aa\"))\n\tse2b.Remove(String(\"bb\"))\n\n\tmOut = me0.Map()\n\tfmt.Println(EncodedValue(mOut))\n\tassert.True(t, mOut.Equals(NewMap(vrw,\n\t\tString(\"a\"), NewMap(vrw, String(\"a\"), NewSet(vrw)),\n\t\tString(\"b\"), NewMap(vrw, String(\"b\"), NewSet(vrw)),\n\t))) // do not remove empty\n}\n" }, { "path": "go/types/meta_sequence.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\nconst (\n\tobjectWindowSize = 8\n\torderedSequenceWindowSize = 1\n\tobjectPattern = uint32(1<<6 - 1) // Average size of 64 elements\n)\n\nvar emptyKey = orderedKey{}\n\nfunc newMetaTuple(ref Ref, key orderedKey, numLeaves uint64) metaTuple {\n\td.PanicIfTrue(ref.buff == nil)\n\tw := newBinaryNomsWriter()\n\tvar offsets [metaTuplePartNumLeaves + 1]uint32\n\toffsets[metaTuplePartRef] = w.offset\n\tref.writeTo(&w)\n\toffsets[metaTuplePartKey] = w.offset\n\tkey.writeTo(&w)\n\toffsets[metaTuplePartNumLeaves] = w.offset\n\tw.writeCount(numLeaves)\n\treturn metaTuple{w.data(), offsets}\n}\n\n// metaTuple is a node in a Prolly Tree, consisting of data in the node (either tree leaves or other metaSequences), and a Value annotation for exploring the tree (e.g. the largest item if this an ordered sequence).\ntype metaTuple struct {\n\tbuff []byte\n\toffsets [metaTuplePartNumLeaves + 1]uint32\n}\n\nconst (\n\tmetaTuplePartRef = 0\n\tmetaTuplePartKey = 1\n\tmetaTuplePartNumLeaves = 2\n)\n\nfunc (mt metaTuple) decoderAtPart(part uint32) valueDecoder {\n\toffset := mt.offsets[part] - mt.offsets[metaTuplePartRef]\n\treturn newValueDecoder(mt.buff[offset:], nil)\n}\n\nfunc (mt metaTuple) ref() Ref {\n\tdec := mt.decoderAtPart(metaTuplePartRef)\n\treturn dec.readRef()\n}\n\nfunc (mt metaTuple) key() orderedKey {\n\tdec := mt.decoderAtPart(metaTuplePartKey)\n\treturn dec.readOrderedKey()\n}\n\nfunc (mt metaTuple) numLeaves() uint64 {\n\tdec := mt.decoderAtPart(metaTuplePartNumLeaves)\n\treturn dec.readCount()\n}\n\nfunc (mt metaTuple) getChildSequence(vr ValueReader) sequence {\n\treturn mt.ref().TargetValue(vr).(Collection).asSequence()\n}\n\nfunc (mt metaTuple) writeTo(w nomsWriter) {\n\tw.writeRaw(mt.buff)\n}\n\n// orderedKey is a key in a Prolly Tree level, which is a metaTuple in a metaSequence, or a value in a leaf sequence.\n// |v| may be nil or |h| may be empty, but not both.\ntype orderedKey struct {\n\tisOrderedByValue bool\n\tv Value\n\th hash.Hash\n}\n\nfunc newOrderedKey(v Value) orderedKey {\n\tif isKindOrderedByValue(v.Kind()) {\n\t\treturn orderedKey{true, v, hash.Hash{}}\n\t}\n\treturn orderedKey{false, v, v.Hash()}\n}\n\nfunc orderedKeyFromHash(h hash.Hash) orderedKey {\n\treturn orderedKey{false, nil, h}\n}\n\nfunc orderedKeyFromInt(n int) orderedKey {\n\treturn newOrderedKey(Number(n))\n}\n\nfunc orderedKeyFromUint64(n uint64) orderedKey {\n\treturn newOrderedKey(Number(n))\n}\n\nfunc (key orderedKey) Less(mk2 orderedKey) bool {\n\tswitch {\n\tcase key.isOrderedByValue && mk2.isOrderedByValue:\n\t\treturn key.v.Less(mk2.v)\n\tcase key.isOrderedByValue:\n\t\treturn true\n\tcase mk2.isOrderedByValue:\n\t\treturn false\n\tdefault:\n\t\td.PanicIfTrue(key.h.IsEmpty() || mk2.h.IsEmpty())\n\t\treturn key.h.Less(mk2.h)\n\t}\n}\n\nfunc (key orderedKey) writeTo(w nomsWriter) {\n\tif !key.isOrderedByValue {\n\t\td.PanicIfTrue(key != emptyKey && key.h.IsEmpty())\n\t\thashKind.writeTo(w)\n\t\tw.writeHash(key.h)\n\t} else {\n\t\tkey.v.writeTo(w)\n\t}\n}\n\ntype metaSequence struct {\n\tsequenceImpl\n}\n\nfunc newMetaSequence(vrw ValueReadWriter, buff []byte, offsets []uint32, len uint64) metaSequence {\n\treturn metaSequence{newSequenceImpl(vrw, buff, offsets, len)}\n}\n\n// readLeafSequence reads the data provided by a decoder and moves the decoder forward.\nfunc readMetaSequence(dec *valueDecoder) metaSequence {\n\tstart := dec.pos()\n\toffsets, len := skipMetaSequence(dec)\n\tend := dec.pos()\n\treturn newMetaSequence(dec.vrw, dec.byteSlice(start, end), offsets, len)\n}\n\nfunc skipMetaSequence(dec *valueDecoder) ([]uint32, uint64) {\n\tkindPos := dec.pos()\n\tdec.skipKind()\n\tlevelPos := dec.pos()\n\tdec.skipCount() // level\n\tcountPos := dec.pos()\n\tcount := dec.readCount()\n\tvaluesPos := dec.pos()\n\toffsets := make([]uint32, count+sequencePartValues+1)\n\toffsets[sequencePartKind] = kindPos\n\toffsets[sequencePartLevel] = levelPos\n\toffsets[sequencePartCount] = countPos\n\toffsets[sequencePartValues] = valuesPos\n\tlength := uint64(0)\n\tfor i := uint64(0); i < count; i++ {\n\t\tdec.skipValue() // ref\n\t\tdec.skipValue() // v\n\t\tlength += dec.readCount() // numLeaves\n\t\toffsets[i+sequencePartValues+1] = dec.pos()\n\t}\n\treturn offsets, length\n}\n\nfunc newMetaSequenceFromTuples(kind NomsKind, level uint64, tuples []metaTuple, vrw ValueReadWriter) metaSequence {\n\td.PanicIfFalse(level > 0)\n\tw := newBinaryNomsWriter()\n\toffsets := make([]uint32, len(tuples)+sequencePartValues+1)\n\toffsets[sequencePartKind] = w.offset\n\tkind.writeTo(&w)\n\toffsets[sequencePartLevel] = w.offset\n\tw.writeCount(level)\n\toffsets[sequencePartCount] = w.offset\n\tw.writeCount(uint64(len(tuples)))\n\toffsets[sequencePartValues] = w.offset\n\tlength := uint64(0)\n\tfor i, mt := range tuples {\n\t\tlength += mt.numLeaves()\n\t\tmt.writeTo(&w)\n\t\toffsets[i+sequencePartValues+1] = w.offset\n\t}\n\treturn newMetaSequence(vrw, w.data(), offsets, length)\n}\n\nfunc (ms metaSequence) tuples() []metaTuple {\n\tdec, count := ms.decoderSkipToValues()\n\ttuples := make([]metaTuple, count)\n\tfor i := uint64(0); i < count; i++ {\n\t\ttuples[i] = ms.readTuple(&dec)\n\t}\n\treturn tuples\n}\n\nfunc (ms metaSequence) getKey(idx int) orderedKey {\n\tdec := ms.decoderSkipToIndex(idx)\n\tdec.skipValue() // ref\n\treturn dec.readOrderedKey()\n}\n\nfunc (ms metaSequence) search(key orderedKey) int {\n\treturn sort.Search(ms.seqLen(), func(i int) bool {\n\t\treturn !ms.getKey(i).Less(key)\n\t})\n}\n\nfunc (ms metaSequence) cumulativeNumberOfLeaves(idx int) uint64 {\n\tcum := uint64(0)\n\tdec, _ := ms.decoderSkipToValues()\n\tfor i := 0; i <= idx; i++ {\n\t\tdec.skipValue() // ref\n\t\tdec.skipValue() // v\n\t\tcum += dec.readCount()\n\t}\n\treturn cum\n}\n\nfunc (ms metaSequence) getCompareFn(other sequence) compareFn {\n\tdec := ms.decoder()\n\toms := other.(metaSequence)\n\totherDec := oms.decoder()\n\treturn func(idx, otherIdx int) bool {\n\t\treturn ms.getRefAt(&dec, idx).TargetHash() == oms.getRefAt(&otherDec, otherIdx).TargetHash()\n\t}\n}\n\nfunc (ms metaSequence) readTuple(dec *valueDecoder) metaTuple {\n\tvar offsets [metaTuplePartNumLeaves + 1]uint32\n\tstart := dec.offset\n\toffsets[metaTuplePartRef] = start\n\tdec.skipRef()\n\toffsets[metaTuplePartKey] = dec.offset\n\tdec.skipOrderedKey()\n\toffsets[metaTuplePartNumLeaves] = dec.offset\n\tdec.skipCount()\n\tend := dec.offset\n\treturn metaTuple{dec.byteSlice(start, end), offsets}\n}\n\nfunc (ms metaSequence) getRefAt(dec *valueDecoder, idx int) Ref {\n\tdec.offset = uint32(ms.getItemOffset(idx))\n\treturn dec.readRef()\n}\n\nfunc (ms metaSequence) getNumLeavesAt(idx int) uint64 {\n\tdec := ms.decoderSkipToIndex(idx)\n\tdec.skipValue()\n\tdec.skipOrderedKey()\n\treturn dec.readCount()\n}\n\n// sequence interface\nfunc (ms metaSequence) getItem(idx int) sequenceItem {\n\tdec := ms.decoderSkipToIndex(idx)\n\treturn ms.readTuple(&dec)\n}\n\nfunc (ms metaSequence) valuesSlice(from, to uint64) []Value {\n\tpanic(\"meta sequence\")\n}\n\nfunc (ms metaSequence) typeOf() *Type {\n\tdec, count := ms.decoderSkipToValues()\n\tts := make(typeSlice, 0, count)\n\tvar lastRef Ref\n\tfor i := uint64(0); i < count; i++ {\n\t\tref := dec.readRef()\n\t\tif lastRef.IsZeroValue() || !lastRef.isSameTargetType(ref) {\n\t\t\tlastRef = ref\n\t\t\tt := ref.TargetType()\n\t\t\tts = append(ts, t)\n\t\t}\n\n\t\tdec.skipOrderedKey() // key\n\t\tdec.skipCount() // numLeaves\n\t}\n\n\treturn makeUnionType(ts...)\n}\n\nfunc (ms metaSequence) numLeaves() uint64 {\n\treturn ms.len\n}\n\nfunc (ms metaSequence) treeLevel() uint64 {\n\tdec := ms.decoderAtPart(sequencePartLevel)\n\treturn dec.readCount()\n}\n\nfunc (ms metaSequence) isLeaf() bool {\n\td.PanicIfTrue(ms.treeLevel() == 0)\n\treturn false\n}\n\n// metaSequence interface\nfunc (ms metaSequence) getChildSequence(idx int) sequence {\n\tmt := ms.getItem(idx).(metaTuple)\n\t// TODO: IsZeroValue?\n\tif mt.buff == nil {\n\t\treturn nil\n\t}\n\treturn mt.getChildSequence(ms.vrw)\n}\n\n// Returns the sequences pointed to by all items[i], s.t. start <= i < end, and returns the\n// concatentation as one long composite sequence\nfunc (ms metaSequence) getCompositeChildSequence(start uint64, length uint64) sequence {\n\tlevel := ms.treeLevel()\n\td.PanicIfFalse(level > 0)\n\tif length == 0 {\n\t\treturn emptySequence{level - 1}\n\t}\n\n\toutput := ms.getChildren(start, start+length)\n\n\tif level > 1 {\n\t\tvar metaItems []metaTuple\n\t\tfor _, seq := range output {\n\t\t\tmetaItems = append(metaItems, seq.(metaSequence).tuples()...)\n\t\t}\n\t\treturn newMetaSequenceFromTuples(ms.Kind(), level-1, metaItems, ms.vrw)\n\t}\n\n\tswitch ms.Kind() {\n\tcase ListKind:\n\t\tvar valueItems []Value\n\t\tfor _, seq := range output {\n\t\t\tvalueItems = append(valueItems, seq.(listLeafSequence).values()...)\n\t\t}\n\t\treturn newListLeafSequence(ms.vrw, valueItems...)\n\tcase MapKind:\n\t\tvar valueItems []mapEntry\n\t\tfor _, seq := range output {\n\t\t\tvalueItems = append(valueItems, seq.(mapLeafSequence).entries()...)\n\t\t}\n\t\treturn newMapLeafSequence(ms.vrw, valueItems...)\n\tcase SetKind:\n\t\tvar valueItems []Value\n\t\tfor _, seq := range output {\n\t\t\tvalueItems = append(valueItems, seq.(setLeafSequence).values()...)\n\t\t}\n\t\treturn newSetLeafSequence(ms.vrw, valueItems...)\n\t}\n\n\tpanic(\"unreachable\")\n}\n\n// fetches child sequences from start (inclusive) to end (exclusive).\nfunc (ms metaSequence) getChildren(start, end uint64) (seqs []sequence) {\n\td.Chk.True(end <= uint64(ms.seqLen()))\n\td.Chk.True(start <= end)\n\n\tseqs = make([]sequence, end-start)\n\ths := make(hash.HashSlice, len(seqs))\n\n\tdec := ms.decoder()\n\n\tfor i := start; i < end; i++ {\n\t\ths[i-start] = ms.getRefAt(&dec, int(i)).TargetHash()\n\t}\n\n\tif len(hs) == 0 {\n\t\treturn // can occur with ptree that is fully uncommitted\n\t}\n\n\t// Fetch committed child sequences in a single batch\n\treadValues := ms.vrw.ReadManyValues(hs)\n\tfor i, v := range readValues {\n\t\tseqs[i] = v.(Collection).asSequence()\n\t}\n\n\treturn\n}\n\nfunc metaHashValueBytes(item sequenceItem, rv *rollingValueHasher) {\n\trv.hashBytes(item.(metaTuple).buff)\n}\n\ntype emptySequence struct {\n\tlevel uint64\n}\n\nfunc (es emptySequence) getItem(idx int) sequenceItem {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) seqLen() int {\n\treturn 0\n}\n\nfunc (es emptySequence) numLeaves() uint64 {\n\treturn 0\n}\n\nfunc (es emptySequence) valueReadWriter() ValueReadWriter {\n\treturn nil\n}\n\nfunc (es emptySequence) WalkRefs(cb RefCallback) {\n}\n\nfunc (es emptySequence) getCompareFn(other sequence) compareFn {\n\treturn func(idx, otherIdx int) bool { panic(\"empty sequence\") }\n}\n\nfunc (es emptySequence) getKey(idx int) orderedKey {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) search(key orderedKey) int {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) getValue(idx int) Value {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) cumulativeNumberOfLeaves(idx int) uint64 {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) getChildSequence(i int) sequence {\n\treturn nil\n}\n\nfunc (es emptySequence) Kind() NomsKind {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) typeOf() *Type {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) getCompositeChildSequence(start uint64, length uint64) sequence {\n\td.PanicIfFalse(es.level > 0)\n\td.PanicIfFalse(start == 0)\n\td.PanicIfFalse(length == 0)\n\treturn emptySequence{es.level - 1}\n}\n\nfunc (es emptySequence) treeLevel() uint64 {\n\treturn es.level\n}\n\nfunc (es emptySequence) isLeaf() bool {\n\treturn es.level == 0\n}\n\nfunc (es emptySequence) Hash() hash.Hash {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) Equals(other Value) bool {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) Less(other Value) bool {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) valueBytes() []byte {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) valuesSlice(from, to uint64) []Value {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) writeTo(nomsWriter) {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) Empty() bool {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) Len() uint64 {\n\tpanic(\"empty sequence\")\n}\n\nfunc (es emptySequence) asValueImpl() valueImpl {\n\tpanic(\"empty sequence\")\n}\n" }, { "path": "go/types/noms_kind.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\n// NomsKind allows a TypeDesc to indicate what kind of type is described.\ntype NomsKind uint8\n\n// All supported kinds of Noms types are enumerated here.\n// The ordering of these (especially Bool, Number and String) is important for ordering of values.\nconst (\n\tBoolKind NomsKind = iota\n\tNumberKind\n\tStringKind\n\tBlobKind\n\tValueKind\n\tListKind\n\tMapKind\n\tRefKind\n\tSetKind\n\n\t// Keep StructKind and CycleKind together.\n\tStructKind\n\tCycleKind\n\n\tTypeKind\n\tUnionKind\n\n\t// Internal to decoder\n\thashKind\n)\n\nvar KindToString = map[NomsKind]string{\n\tBlobKind: \"Blob\",\n\tBoolKind: \"Bool\",\n\tCycleKind: \"Cycle\",\n\tListKind: \"List\",\n\tMapKind: \"Map\",\n\tNumberKind: \"Number\",\n\tRefKind: \"Ref\",\n\tSetKind: \"Set\",\n\tStructKind: \"Struct\",\n\tStringKind: \"String\",\n\tTypeKind: \"Type\",\n\tUnionKind: \"Union\",\n\tValueKind: \"Value\",\n}\n\n// String returns the name of the kind.\nfunc (k NomsKind) String() string {\n\treturn KindToString[k]\n}\n\n// IsPrimitiveKind returns true if k represents a Noms primitive type, which excludes collections (List, Map, Set), Refs, Structs, Symbolic and Unresolved types.\nfunc IsPrimitiveKind(k NomsKind) bool {\n\tswitch k {\n\tcase BoolKind, NumberKind, StringKind, BlobKind, ValueKind, TypeKind:\n\t\treturn true\n\tdefault:\n\t\treturn false\n\t}\n}\n\n// isKindOrderedByValue determines if a value is ordered by its value instead of its hash.\nfunc isKindOrderedByValue(k NomsKind) bool {\n\treturn k <= StringKind\n}\n\nfunc (k NomsKind) writeTo(w nomsWriter) {\n\tw.writeUint8(uint8(k))\n}\n" }, { "path": "go/types/number.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"encoding/binary\"\n\t\"math\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n// Number is a Noms Value wrapper around the primitive float64 type.\ntype Number float64\n\n// Value interface\nfunc (v Number) Value() Value {\n\treturn v\n}\n\nfunc (v Number) Equals(other Value) bool {\n\treturn v == other\n}\n\nfunc (v Number) Less(other Value) bool {\n\tif v2, ok := other.(Number); ok {\n\t\treturn v < v2\n\t}\n\treturn NumberKind < other.Kind()\n}\n\nfunc (v Number) Hash() hash.Hash {\n\treturn getHash(v)\n}\n\nfunc (v Number) WalkValues(cb ValueCallback) {\n}\n\nfunc (v Number) WalkRefs(cb RefCallback) {\n}\n\nfunc (v Number) typeOf() *Type {\n\treturn NumberType\n}\n\nfunc (v Number) Kind() NomsKind {\n\treturn NumberKind\n}\n\nfunc (v Number) valueReadWriter() ValueReadWriter {\n\treturn nil\n}\n\nfunc (v Number) writeTo(w nomsWriter) {\n\tNumberKind.writeTo(w)\n\tf := float64(v)\n\tif math.IsNaN(f) || math.IsInf(f, 0) {\n\t\td.Panic(\"%f is not a supported number\", f)\n\t}\n\tw.writeNumber(v)\n}\n\nfunc (v Number) valueBytes() []byte {\n\t// We know the size of the buffer here so allocate it once.\n\t// NumberKind, int (Varint), exp (Varint)\n\tbuff := make([]byte, 1+2*binary.MaxVarintLen64)\n\tw := binaryNomsWriter{buff, 0}\n\tv.writeTo(&w)\n\treturn buff[:w.offset]\n}\n" }, { "path": "go/types/number_util.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport \"math\"\n\nfunc float64IsInt(f float64) bool {\n\treturn math.Trunc(f) == f\n}\n\n// convert float64 to int64 where f == i * 2^exp\nfunc float64ToIntExp(f float64) (int64, int) {\n\tif f == 0 {\n\t\treturn 0, 0\n\t}\n\n\tisNegative := math.Signbit(f)\n\tf = math.Abs(f)\n\n\tfrac, exp := math.Frexp(f)\n\t// frac is [.5, 1)\n\t// Move frac up until it is an integer.\n\tfor !float64IsInt(frac) {\n\t\tfrac *= 2\n\t\texp--\n\t}\n\n\tif isNegative {\n\t\tfrac *= -1\n\t}\n\n\treturn int64(frac), exp\n}\n\n// fracExpToFloat returns frac * 2 ** exp\nfunc fracExpToFloat(frac int64, exp int) float64 {\n\treturn float64(frac) * math.Pow(2, float64(exp))\n}\n" }, { "path": "go/types/opcache.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// opCache stores build operations on a graph of nested Maps whose leaves can\n// in turn be Set, Map, or List collections containing any Noms Value.\n// OpCacheIterator returns operations in sorted order.\n//\n// OpCache uses a special encoding of the information supplied by the MapSet(),\n// ListAppend(), or SetInsert() operation stored in the ldbKey combined with\n// custom ldb Comparer object implemented in opcache_compare.go to make this\n// happen.\n//\n// Ldb keys are encoded byte arrays that contain the following information:\n// 4-bytes -- uint32 in BigEndian order which identifies this key/value\n// as belonging to a particular graph\n// 1-byte -- a NomsKind value that represents the collection type that is\n// being acted on. This will either be MapKind, SetKind, or ListKind.\n// 1-byte -- uint8 representing the number of NomsValues encoded in this key\n//\n// After this 6-byte header, there is a section of bytes for each value encoded\n// into the key. Each value has a 1-byte prefix:\n// 1-byte -- a NomsKind value that represents the type of value that is\n// being encoded.\n// The 1-byte NomsKind value determines what follows, if this value is\n// BoolKind, NumberKind, or StringKind, the rest of the bytes are:\n// 4-bytes -- uint32 length of the Value serialization\n// n-bytes -- the serialized value\n// If the NomsKind byte has any other value, it is followed by:\n// 20-bytes -- digest of Value's hash\n//\n// Whenever the value is encoded as a hash digest in the ldbKey, it's actual value\n// needs to get stored in the ldbValue. (More about this later)\n//\n// There are 3 operation types on opCache: MapSet(), SetInsert(), and ListAppend().\n// Each one stores slightly different things in the ldbKey.\n// MapSet() -- stores each graphKey and the key to the final Map\n// ValueSet() -- stores each graphKey and the Value being inserted into the set\n// ListAppend() -- stores each graphKey and a Number() containing an uint64 value\n// that is shared across all collections and lists which is incremented each time\n// ListAppend() is called.\n//\n// The ldbValue also stores different information for each mutation operation. An\n// ldbValue has a 1-byte uint8 header that is the number of values that are encoded\n// into it.\n// 1-byte -- uint8 indicating number of values encoded into this byte array\n// Then for each encoded value it contains:\n// 4-byte -- uint32 indicating length of value serialization\n// n-bytes -- the serialized value\n//\n// The ldbValue contains the following values for each type of mutation:\n// MapSet() -- stores any graphKeys that were encoded as a hash digest in\n// the ldbKey. The mapKey if it was encoded as a hash digest in the ldbKey\n// and the value being set in the map.\n// SetInsert() -- stores any graphKeys that were encoded as a hash digest in\n// the ldbKey. The value being inserted into the set if it was encoded into the\n// ldbKey as a hash digest.\n// ListAppend() -- stores any graphKeys that were encoded as a hash digest in the\n// ldbKey. The value being appended to the list.\n//\n\npackage types\n\nimport (\n\t\"encoding/binary\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"sync/atomic\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/syndtr/goleveldb/leveldb\"\n\tldbIterator \"github.com/syndtr/goleveldb/leveldb/iterator\"\n\t\"github.com/syndtr/goleveldb/leveldb/opt\"\n\t\"github.com/syndtr/goleveldb/leveldb/util\"\n)\n\nconst uint32Size = 4\n\ntype opCacheStore interface {\n\topCache() opCache\n\tdestroy() error\n}\n\ntype opCache interface {\n\t// This method can be called from multiple go routines.\n\tGraphMapSet(keys ValueSlice, mapKey Value, mapVal Value)\n\n\t// This method can be called from multiple go routines.\n\tGraphSetInsert(keys ValueSlice, val Value)\n\n\t// This method can be called from multiple go routines, however items will\n\t// be appended to the list based on the order that routines execute\n\t// this method.\n\tGraphListAppend(keys ValueSlice, val Value)\n\n\tNewIterator() opCacheIterator\n}\n\ntype opCacheIterator interface {\n\tGraphOp() (ValueSlice, NomsKind, sequenceItem)\n\tNext() bool\n\tRelease()\n}\n\ntype ldbOpCacheStore struct {\n\tldb *leveldb.DB\n\tdbDir string\n\tcollectionId uint32\n\tvrw ValueReadWriter\n}\n\ntype ldbOpCache struct {\n\tvrw ValueReadWriter\n\tcolId uint32\n\tlistIdx int64\n\tldb *leveldb.DB\n}\n\ntype ldbOpCacheIterator struct {\n\titer ldbIterator.Iterator\n\tvrw ValueReadWriter\n}\n\nfunc newLdbOpCacheStore(vrw ValueReadWriter) *ldbOpCacheStore {\n\tdir, err := ioutil.TempDir(\"\", \"\")\n\td.Chk.NoError(err)\n\tdb, err := leveldb.OpenFile(dir, &opt.Options{\n\t\tCompression: opt.NoCompression,\n\t\tComparer: opCacheComparer{},\n\t\tOpenFilesCacheCapacity: 24,\n\t\t// This data does not have to be durable. LDB is acting as temporary\n\t\t// storage that can be larger than main memory.\n\t\tNoSync: true,\n\t\tWriteBuffer: 1 << 27, // 128MiB\n\t})\n\td.Chk.NoError(err, \"opening put cache in %s\", dir)\n\treturn &ldbOpCacheStore{ldb: db, dbDir: dir, vrw: vrw}\n}\n\nfunc (store *ldbOpCacheStore) destroy() error {\n\td.Chk.NoError(store.ldb.Close())\n\treturn os.RemoveAll(store.dbDir)\n}\n\nfunc (store *ldbOpCacheStore) opCache() opCache {\n\tcolId := atomic.AddUint32(&store.collectionId, 1)\n\treturn &ldbOpCache{vrw: store.vrw, colId: colId, ldb: store.ldb}\n}\n\n// insertLdbOp encodes allKeys into the ldb key. Bool, Number, and String values\n// are encoded directly into the ldb key bytes. All other types are encoded as\n// their Hash() digest. Their actual value is then stored in ldb value.\nfunc (opc *ldbOpCache) insertLdbOp(allKeys ValueSlice, opKind NomsKind, val Value) {\n\tif len(allKeys) > 0x00FF {\n\t\td.Panic(\"Number of keys in GraphMapSet exceeds max of 256\")\n\t}\n\tldbKeyBytes := [initialBufferSize]byte{}\n\tldbValBytes := [initialBufferSize]byte{}\n\n\tldbKey, valuesToEncode := encodeKeys(ldbKeyBytes[:0], opc.colId, opKind, allKeys)\n\n\t// val may be nil when dealing with sets, since the val is the key.\n\tif val != nil {\n\t\tvaluesToEncode = append(valuesToEncode, val)\n\t}\n\tldbVal := encodeValues(ldbValBytes[:0], valuesToEncode)\n\n\terr := opc.ldb.Put(ldbKey, ldbVal, nil)\n\td.Chk.NoError(err)\n}\n\nfunc (opc *ldbOpCache) GraphMapSet(graphKeys ValueSlice, mapKey, mapVal Value) {\n\tallKeys := append(graphKeys, mapKey)\n\topc.insertLdbOp(allKeys, MapKind, mapVal)\n}\n\nfunc (opc *ldbOpCache) GraphSetInsert(graphKeys ValueSlice, val Value) {\n\tallKeys := append(graphKeys, val)\n\topc.insertLdbOp(allKeys, SetKind, val)\n}\n\nfunc (opc *ldbOpCache) GraphListAppend(graphKeys ValueSlice, val Value) {\n\tidx := atomic.AddInt64(&opc.listIdx, 1)\n\tallKeys := append(graphKeys, Number(idx))\n\topc.insertLdbOp(allKeys, ListKind, val)\n}\n\nfunc (i *ldbOpCacheIterator) GraphOp() (ValueSlice, NomsKind, sequenceItem) {\n\tldbKey := i.iter.Key()\n\tldbVal := i.iter.Value()\n\n\t// skip over 4 bytes of colId and get opKind, and numKeys from bytes 4 & 5\n\topKind := NomsKind(ldbKey[4])\n\tnumKeys := uint8(ldbKey[5])\n\tldbKey = ldbKey[6:]\n\n\t// Call decodeValue for each encoded graphKey. nil will be appended to\n\t// graphKeys for any keys that were encoded as hash digests.\n\tgraphKeys := ValueSlice{}\n\tfor pos := uint8(0); pos < numKeys; pos++ {\n\t\tvar gk Value\n\t\tldbKey, gk = decodeValue(ldbKey, false, i.vrw)\n\t\tgraphKeys = append(graphKeys, gk)\n\t}\n\n\t// Get the number of values whose value was encoded in ldbVal\n\tnumEncodedValues := uint8(ldbVal[0])\n\tldbVal = ldbVal[1:]\n\n\t// Call decodeValue for each non-primitive key stored in ldbVal. Replace\n\t// the nil value in graphKeys with the new decodedValue.\n\tvalues := ValueSlice{}\n\tfor pos := uint8(0); pos < numEncodedValues; pos++ {\n\t\tvar gk Value\n\t\tldbVal, gk = decodeValue(ldbVal, true, i.vrw)\n\t\tvalues = append(values, gk)\n\t}\n\n\t// Fold in any non-primitive key values that were stored in ldbVal\n\tpos := 0\n\tfor idx, k1 := range graphKeys {\n\t\tif k1 == nil {\n\t\t\tgraphKeys[idx] = values[pos]\n\t\t\tpos++\n\t\t}\n\t}\n\n\t// Remove the last key in graphKeys. The last key in graphKeys is the\n\t// mapkey for Maps, the item for Sets, and the index for Lists.\n\tkey := graphKeys[len(graphKeys)-1]\n\tgraphKeys = graphKeys[:len(graphKeys)-1]\n\n\tvar item sequenceItem\n\tswitch opKind {\n\tcase MapKind:\n\t\tval := values[len(values)-1]\n\t\titem = mapEntry{key, val}\n\tcase SetKind:\n\t\titem = key\n\tcase ListKind:\n\t\titem = values[len(values)-1]\n\t}\n\n\treturn graphKeys, opKind, item\n}\n\nfunc (opc *ldbOpCache) NewIterator() opCacheIterator {\n\tprefix := [4]byte{}\n\tbinary.BigEndian.PutUint32(prefix[:], opc.colId)\n\treturn &ldbOpCacheIterator{iter: opc.ldb.NewIterator(util.BytesPrefix(prefix[:]), nil), vrw: opc.vrw}\n}\n\nfunc (i *ldbOpCacheIterator) Next() bool {\n\treturn i.iter.Next()\n}\n\nfunc (i *ldbOpCacheIterator) Release() {\n\ti.iter.Release()\n}\n\n// encodeKeys() serializes a list of keys to the byte slice |bs|.\nfunc encodeKeys(bs []byte, colId uint32, opKind NomsKind, keys []Value) ([]byte, []Value) {\n\t// All ldb keys start with a 4-byte collection id that serves as a namespace\n\t// that keeps them separate from other collections.\n\tidHolder := [4]byte{}\n\tidHolderSlice := idHolder[:4]\n\tbinary.BigEndian.PutUint32(idHolderSlice, colId)\n\tbs = append(bs, idHolderSlice...)\n\n\t// bs[4] is a NomsKind value which represents the type of leaf\n\t// collection being operated on (i.e. MapKind, SetKind, or ListKind)\n\t// bs[5] is a single uint8 value representing the number of keys\n\t// encoded in the ldb key.\n\tbs = append(bs, byte(opKind), byte(len(keys)))\n\n\tvaluesToEncode := ValueSlice{}\n\tfor _, gk := range keys {\n\t\tbs = encodeGraphKey(bs, gk)\n\t\tif !isKindOrderedByValue(gk.Kind()) {\n\t\t\tvaluesToEncode = append(valuesToEncode, gk)\n\t\t}\n\t}\n\treturn bs, valuesToEncode\n}\n\nfunc encodeValues(bs []byte, valuesToEncode []Value) []byte {\n\t// Encode allValues into the ldbVal byte slice.\n\tbs = append(bs, uint8(len(valuesToEncode)))\n\tfor _, k := range valuesToEncode {\n\t\tbs = encodeGraphValue(bs, k)\n\t}\n\treturn bs\n}\n\nfunc encodeGraphKey(bs []byte, v Value) []byte {\n\treturn encodeForGraph(bs, v, false)\n}\n\nfunc encodeGraphValue(bs []byte, v Value) []byte {\n\treturn encodeForGraph(bs, v, true)\n}\n\nfunc encodeForGraph(bs []byte, v Value, asValue bool) []byte {\n\t// Note: encToSlice() and append() will both grow the backing store of |bs|\n\t// as necessary. Always call them when writing to |bs|.\n\tif asValue || isKindOrderedByValue(v.Kind()) {\n\t\t// if we're encoding value, then put:\n\t\t// noms-kind(1-byte), serialization-len(4-bytes), serialization(n-bytes)\n\t\tbuf := [initialBufferSize]byte{}\n\t\tuint32buf := [4]byte{}\n\t\tencodedVal := encToSlice(v, buf[:])\n\t\tbinary.BigEndian.PutUint32(uint32buf[:], uint32(len(encodedVal)))\n\t\tbs = append(bs, uint8(v.Kind()))\n\t\tbs = append(bs, uint32buf[:]...)\n\t\tbs = append(bs, encodedVal...)\n\t} else {\n\t\t// if we're encoding hash values, we know the length, so we can leave that out\n\t\tbs = append(bs, uint8(v.Kind()))\n\t\th := v.Hash()\n\t\tbs = append(bs, h[:]...)\n\t}\n\treturn bs\n}\n\nfunc decodeValue(bs []byte, asValue bool, vrw ValueReadWriter) ([]byte, Value) {\n\tkind := NomsKind(bs[0])\n\tvar v Value\n\tif asValue || isKindOrderedByValue(kind) {\n\t\tencodedLen := binary.BigEndian.Uint32(bs[1:5])\n\t\t// The bytes in bs gets reused by LDB. The data of a chunk must\n\t\t// never change since we are backing the values by this data.\n\t\tdata := make([]byte, encodedLen)\n\t\tcopy(data, bs[5:5+encodedLen])\n\t\tv = DecodeFromBytes(data, vrw)\n\t\treturn bs[5+encodedLen:], v\n\t}\n\treturn bs[1+hash.ByteLen:], nil\n}\n\n// Note that, if 'v' are prolly trees, any in-memory child chunks will be written to vw at this time.\nfunc encToSlice(v Value, initBuf []byte) []byte {\n\t// TODO: Are there enough calls to this that it's worth re-using a nomsWriter?\n\tw := &binaryNomsWriter{initBuf, 0}\n\tv.writeTo(w)\n\treturn w.data()\n}\n" }, { "path": "go/types/opcache_compare.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"encoding/binary\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype opCacheComparer struct{}\n\nfunc (opCacheComparer) Compare(a, b []byte) int {\n\tif res := bytes.Compare(a[:uint32Size], b[:uint32Size]); res != 0 {\n\t\treturn res\n\t}\n\treturn compareEncodedKeys(a[uint32Size:], b[uint32Size:])\n}\n\nfunc (opCacheComparer) Name() string {\n\treturn \"noms.OpCacheComparator\"\n}\n\nfunc (opCacheComparer) Successor(dst, b []byte) []byte {\n\treturn nil\n}\n\nfunc (opCacheComparer) Separator(dst, a, b []byte) []byte {\n\treturn nil\n}\n\nfunc compareEncodedKeys(a, b []byte) int {\n\tif compared, res := compareEmpties(a, b); compared {\n\t\treturn res\n\t}\n\n\t// keys are encoded as either values:\n\t// nomsKind(1-byte) + serialized len(4-bytes) + serialized value(n-bytes)\n\t// or digests:\n\t// nomsKind(1-byte) + digest(hash.Bytelen-bytes)\n\tsplitAfterFirstKey := func(bs []byte) ([]byte, []byte) {\n\t\tkeyLen := 1 + hash.ByteLen\n\t\tif isKindOrderedByValue(NomsKind(bs[0])) {\n\t\t\tl := int(binary.BigEndian.Uint32(bs[1:5]))\n\t\t\tkeyLen = 1 + uint32Size + l\n\t\t}\n\t\treturn bs[:keyLen], bs[keyLen:]\n\t}\n\n\t// a[0] and b[0] represent NomsKind of leafNode being operated on\n\t// a[1] and b[1] are the number of keys encoded in this byte slice\n\tnumAGraphKeys, numBGraphKeys := a[1], b[1]\n\tminNumKeys := minByte(numAGraphKeys, numBGraphKeys)\n\n\ta, b = a[2:], b[2:]\n\tcres := 0\n\tfor pos := 0; pos < int(minNumKeys) && cres == 0; pos++ {\n\t\taKey, aRest := splitAfterFirstKey(a)\n\t\tbKey, bRest := splitAfterFirstKey(b)\n\t\tcres = compareEncodedKey(aKey, bKey)\n\t\ta, b = aRest, bRest\n\t}\n\n\tif cres == 0 {\n\t\tif numAGraphKeys < numBGraphKeys {\n\t\t\treturn -1\n\t\t}\n\t\tif numAGraphKeys > numBGraphKeys {\n\t\t\treturn 1\n\t\t}\n\t}\n\treturn cres\n}\n\n// compareEncodedKey accepts two byte slices that each contain a number of\n// encoded keys. It extracts the first key in each slice and returns the result\n// of comparing them.\nfunc compareEncodedKey(a, b []byte) int {\n\t// keys that are orderd by value are encoded as:\n\t// NomsKind(1-byte) + length(4-bytes) + encoding(n-bytes)\n\t// keys that are not ordred by value are encoded as\n\t// NomsKind(1-byte) + hash digest(20-bytes)\n\n\taKind, bKind := NomsKind(a[0]), NomsKind(b[0])\n\tif !isKindOrderedByValue(aKind) && !isKindOrderedByValue(bKind) {\n\t\ta, b := a[1:], b[1:]\n\t\td.PanicIfFalse(len(a) == hash.ByteLen && len(b) == hash.ByteLen)\n\t\tres := bytes.Compare(a, b)\n\t\tif res == 0 && aKind != bKind {\n\t\t\td.Panic(\"Values of different kinds with the same hash. Whaa??\")\n\t\t}\n\t\treturn res\n\t}\n\n\t// Now, we know that at least one of a and b is ordered by value. So if the\n\t// kinds are different, we can sort just by comparing them.\n\tif res := compareKinds(aKind, bKind); res != 0 {\n\t\treturn res\n\t}\n\n\t// Now we know that we are comparing two values that are both Bools, Numbers,\n\t// or Strings. Extract their length and create slices that just contain their\n\t// Noms encodings.\n\tlenA := binary.BigEndian.Uint32(a[1:5])\n\tlenB := binary.BigEndian.Uint32(b[1:5])\n\n\t// create a1, b1 slices that just contain encoding\n\ta1, b1 := a[1+uint32Size:1+uint32Size+lenA], b[1+uint32Size:1+uint32Size+lenB]\n\n\treturn compareEncodedNomsValues(a1, b1)\n}\n\n// compareEncodedNomsValues compares two slices. Each slice contains a first\n// byte that holds the nomsKind of the original key and an encoding for that key.\n// This method relies on knowledge about how bytes are arranged in a Noms\n// encoding and makes use of that for companing values efficiently.\nfunc compareEncodedNomsValues(a, b []byte) int {\n\tif compared, res := compareEmpties(a, b); compared {\n\t\treturn res\n\t}\n\taKind, bKind := NomsKind(a[0]), NomsKind(b[0])\n\tif aKind != bKind {\n\t\td.Panic(\"compareEncodedNomsValues, aKind: %s != bKind: %s\", aKind, bKind)\n\t}\n\n\tswitch aKind {\n\tcase BoolKind:\n\t\treturn bytes.Compare(a, b)\n\tcase NumberKind:\n\t\treader := binaryNomsReader{a[1:], 0}\n\t\taNum := reader.readNumber()\n\t\treader.buff, reader.offset = b[1:], 0\n\t\tbNum := reader.readNumber()\n\t\tif aNum == bNum {\n\t\t\treturn 0\n\t\t}\n\t\tif aNum < bNum {\n\t\t\treturn -1\n\t\t}\n\t\treturn 1\n\tcase StringKind:\n\t\t// Skip past uvarint-encoded string length\n\t\t_, aCount := binary.Uvarint(a[1:])\n\t\t_, bCount := binary.Uvarint(b[1:])\n\t\tres := bytes.Compare(a[1+aCount:], b[1+bCount:])\n\t\treturn res\n\t}\n\tpanic(\"unreachable\")\n}\n\nfunc compareEmpties(a, b []byte) (bool, int) {\n\taLen, bLen := len(a), len(b)\n\tif aLen > 0 && bLen > 0 {\n\t\treturn false, 0\n\t}\n\tif aLen == 0 {\n\t\tif bLen == 0 {\n\t\t\treturn true, 0\n\t\t}\n\t\treturn true, -1\n\t}\n\treturn true, 1\n}\n\nfunc compareKinds(aKind, bKind NomsKind) (res int) {\n\tif aKind < bKind {\n\t\tres = -1\n\t} else if aKind > bKind {\n\t\tres = 1\n\t}\n\treturn\n}\n\nfunc minByte(a, b byte) byte {\n\tif a < b {\n\t\treturn a\n\t}\n\treturn b\n}\n" }, { "path": "go/types/opcache_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc TestOpCache(t *testing.T) {\n\tsuite.Run(t, &OpCacheSuite{})\n}\n\ntype OpCacheSuite struct {\n\tsuite.Suite\n\tvs *ValueStore\n}\n\nfunc (suite *OpCacheSuite) SetupTest() {\n\tsuite.vs = newTestValueStore()\n}\n\nfunc (suite *OpCacheSuite) TearDownTest() {\n\tsuite.vs.Close()\n}\n\nfunc (suite *OpCacheSuite) TestMapSet() {\n\tvs := suite.vs\n\topCacheStore := newLdbOpCacheStore(vs)\n\toc := opCacheStore.opCache()\n\tdefer opCacheStore.destroy()\n\n\tentries := mapEntrySlice{\n\t\t{NewList(vs, Number(8), Number(0)), String(\"ahoy\")},\n\t\t{String(\"A key\"), NewBlob(vs, bytes.NewBufferString(\"A value\"))},\n\t\t{Number(1), Bool(true)},\n\t\t{Bool(false), Number(1)},\n\t\t{NewBlob(vs, bytes.NewBuffer([]byte{0xff, 0, 0})), NewMap(vs)},\n\t\t{Bool(true), Number(42)},\n\t\t{NewStruct(\"thing1\", StructData{\"a\": Number(7)}), Number(42)},\n\t\t{String(\"struct\"), NewStruct(\"thing2\", nil)},\n\t\t{Number(42), String(\"other\")},\n\t}\n\tfor _, entry := range entries {\n\t\toc.GraphMapSet(nil, entry.key, entry.value)\n\t}\n\tsort.Sort(entries)\n\n\titerated := mapEntrySlice{}\n\titer := oc.NewIterator()\n\tdefer iter.Release()\n\tfor iter.Next() {\n\t\tkeys, kind, item := iter.GraphOp()\n\t\td.Chk.Empty(keys)\n\t\td.Chk.Equal(MapKind, kind)\n\t\titerated = append(iterated, item.(mapEntry))\n\t}\n\tsuite.True(entries.Equals(iterated))\n}\n\nfunc (suite *OpCacheSuite) TestSetInsert() {\n\tvs := suite.vs\n\topCacheStore := newLdbOpCacheStore(vs)\n\toc := opCacheStore.opCache()\n\tdefer opCacheStore.destroy()\n\n\tentries := ValueSlice{\n\t\tNewList(vs, Number(8), Number(0)),\n\t\tString(\"ahoy\"),\n\t\tNewBlob(vs, bytes.NewBufferString(\"A value\")),\n\t\tNumber(1),\n\t\tBool(true),\n\t\tBool(false),\n\t\tNewBlob(vs, bytes.NewBuffer([]byte{0xff, 0, 0})),\n\t\tNewMap(vs),\n\t\tNumber(42),\n\t\tNewStruct(\"thing1\", StructData{\"a\": Number(7)}),\n\t\tString(\"struct\"),\n\t\tNewStruct(\"thing2\", nil),\n\t\tString(\"other\"),\n\t}\n\tfor _, entry := range entries {\n\t\toc.GraphSetInsert(nil, entry)\n\t}\n\tsort.Sort(entries)\n\n\titerated := ValueSlice{}\n\titer := oc.NewIterator()\n\tdefer iter.Release()\n\tfor iter.Next() {\n\t\tkeys, kind, item := iter.GraphOp()\n\t\td.Chk.Empty(keys)\n\t\td.Chk.Equal(SetKind, kind)\n\t\titerated = append(iterated, item.(Value))\n\t}\n\tsuite.True(entries.Equals(iterated))\n}\n\nfunc (suite *OpCacheSuite) TestListAppend() {\n\tvs := suite.vs\n\topCacheStore := newLdbOpCacheStore(vs)\n\toc := opCacheStore.opCache()\n\tdefer opCacheStore.destroy()\n\n\tentries := ValueSlice{\n\t\tNewList(vs, Number(8), Number(0)),\n\t\tString(\"ahoy\"),\n\t\tNewBlob(vs, bytes.NewBufferString(\"A value\")),\n\t\tNumber(1),\n\t\tBool(true),\n\t\tBool(false),\n\t\tNewBlob(vs, bytes.NewBuffer([]byte{0xff, 0, 0})),\n\t\tNewMap(vs),\n\t\tNumber(42),\n\t\tNewStruct(\"thing1\", StructData{\"a\": Number(7)}),\n\t\tString(\"struct\"),\n\t\tNewStruct(\"thing2\", nil),\n\t\tString(\"other\"),\n\t}\n\tfor _, entry := range entries {\n\t\toc.GraphListAppend(nil, entry)\n\t}\n\n\titerated := ValueSlice{}\n\titer := oc.NewIterator()\n\tdefer iter.Release()\n\tfor iter.Next() {\n\t\tkeys, kind, item := iter.GraphOp()\n\t\td.Chk.Empty(keys)\n\t\td.Chk.Equal(ListKind, kind)\n\t\titerated = append(iterated, item.(Value))\n\t}\n\tsuite.True(entries.Equals(iterated))\n}\n" }, { "path": "go/types/ordered_sequences.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype orderedSequence interface {\n\tsequence\n\tgetKey(idx int) orderedKey\n\tsearch(key orderedKey) int\n}\n\nfunc newSetMetaSequence(level uint64, tuples []metaTuple, vrw ValueReadWriter) metaSequence {\n\treturn newMetaSequenceFromTuples(SetKind, level, tuples, vrw)\n}\n\nfunc newMapMetaSequence(level uint64, tuples []metaTuple, vrw ValueReadWriter) metaSequence {\n\treturn newMetaSequenceFromTuples(MapKind, level, tuples, vrw)\n}\n\nfunc newCursorAtValue(seq orderedSequence, val Value, forInsertion bool, last bool) *sequenceCursor {\n\tvar key orderedKey\n\tif val != nil {\n\t\tkey = newOrderedKey(val)\n\t}\n\treturn newCursorAt(seq, key, forInsertion, last)\n}\n\nfunc newCursorAt(seq orderedSequence, key orderedKey, forInsertion bool, last bool) *sequenceCursor {\n\tvar cur *sequenceCursor\n\tfor {\n\t\tidx := 0\n\t\tif last {\n\t\t\tidx = -1\n\t\t}\n\t\tcur = newSequenceCursor(cur, seq, idx)\n\t\tif key != emptyKey {\n\t\t\tif !seekTo(cur, key, forInsertion && !seq.isLeaf()) {\n\t\t\t\treturn cur\n\t\t\t}\n\t\t}\n\n\t\tcs := cur.getChildSequence()\n\t\tif cs == nil {\n\t\t\tbreak\n\t\t}\n\t\tseq = cs.(orderedSequence)\n\t}\n\td.PanicIfFalse(cur != nil)\n\treturn cur\n}\n\nfunc seekTo(cur *sequenceCursor, key orderedKey, lastPositionIfNotFound bool) bool {\n\tseq := cur.seq.(orderedSequence)\n\n\t// Find smallest idx in seq where key(idx) >= key\n\tcur.idx = seq.search(key)\n\tseqLen := seq.seqLen()\n\tif cur.idx == seqLen && lastPositionIfNotFound {\n\t\td.PanicIfFalse(cur.idx > 0)\n\t\tcur.idx--\n\t}\n\n\treturn cur.idx < seqLen\n}\n\n// Gets the key used for ordering the sequence at current index.\nfunc getCurrentKey(cur *sequenceCursor) orderedKey {\n\tseq, ok := cur.seq.(orderedSequence)\n\tif !ok {\n\t\td.Panic(\"need an ordered sequence here\")\n\t}\n\treturn seq.getKey(cur.idx)\n}\n\nfunc getMapValue(cur *sequenceCursor) Value {\n\tif ml, ok := cur.seq.(mapLeafSequence); ok {\n\t\treturn ml.getValue(cur.idx)\n\t}\n\n\treturn nil\n}\n\n// If |vw| is not nil, chunks will be eagerly written as they're created. Otherwise they are\n// written when the root is written.\nfunc newOrderedMetaSequenceChunkFn(kind NomsKind, vrw ValueReadWriter) makeChunkFn {\n\treturn func(level uint64, items []sequenceItem) (Collection, orderedKey, uint64) {\n\t\ttuples := make([]metaTuple, len(items))\n\t\tnumLeaves := uint64(0)\n\n\t\tvar lastKey orderedKey\n\t\tfor i, v := range items {\n\t\t\tmt := v.(metaTuple)\n\t\t\tkey := mt.key()\n\t\t\td.PanicIfFalse(lastKey == emptyKey || lastKey.Less(key))\n\t\t\tlastKey = key\n\t\t\ttuples[i] = mt // chunk is written when the root sequence is written\n\t\t\tnumLeaves += mt.numLeaves()\n\t\t}\n\n\t\tvar col Collection\n\t\tif kind == SetKind {\n\t\t\tcol = newSet(newSetMetaSequence(level, tuples, vrw))\n\t\t} else {\n\t\t\td.PanicIfFalse(MapKind == kind)\n\t\t\tcol = newMap(newMapMetaSequence(level, tuples, vrw))\n\t\t}\n\n\t\treturn col, tuples[len(tuples)-1].key(), numLeaves\n\t}\n}\n" }, { "path": "go/types/ordered_sequences_diff.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/util/functions\"\n)\n\ntype DiffChangeType uint8\n\nconst (\n\tDiffChangeAdded DiffChangeType = iota\n\tDiffChangeRemoved\n\tDiffChangeModified\n)\n\ntype ValueChanged struct {\n\tChangeType DiffChangeType\n\tKey, OldValue, NewValue Value\n}\n\nfunc sendChange(changes chan<- ValueChanged, stopChan <-chan struct{}, change ValueChanged) bool {\n\tselect {\n\tcase changes <- change:\n\t\treturn true\n\tcase <-stopChan:\n\t\treturn false\n\t}\n}\n\n// Streams the diff from |last| to |current| into |changes|, using both left-right and top-down approach in parallel.\n// The left-right diff is expected to return results earlier, whereas the top-down approach is faster overall. This \"best\" algorithm runs both:\n// - early results from left-right are sent to |changes|.\n// - if/when top-down catches up, left-right is stopped and the rest of the changes are streamed from top-down.\nfunc orderedSequenceDiffBest(last orderedSequence, current orderedSequence, changes chan<- ValueChanged, stopChan <-chan struct{}) bool {\n\tlrChanges := make(chan ValueChanged)\n\ttdChanges := make(chan ValueChanged)\n\t// Give the stop channels a buffer size of 1 so that they won't block (see below).\n\tlrStopChan := make(chan struct{}, 1)\n\ttdStopChan := make(chan struct{}, 1)\n\n\t// Ensure all diff functions have finished doing work by the time this function returns, otherwise database reads might cause deadlock - e.g. https://github.com/attic-labs/noms/issues/2165.\n\twg := &sync.WaitGroup{}\n\n\tdefer func() {\n\t\t// Stop diffing. The left-right or top-down diff might have already finished, but sending to the stop channels won't block due to the buffer.\n\t\tlrStopChan <- struct{}{}\n\t\ttdStopChan <- struct{}{}\n\t\twg.Wait()\n\t}()\n\n\twg.Add(2)\n\tgo func() {\n\t\tdefer wg.Done()\n\t\torderedSequenceDiffLeftRight(last, current, lrChanges, lrStopChan)\n\t\tclose(lrChanges)\n\t}()\n\tgo func() {\n\t\tdefer wg.Done()\n\t\torderedSequenceDiffTopDown(last, current, tdChanges, tdStopChan)\n\t\tclose(tdChanges)\n\t}()\n\n\t// Stream left-right changes while the top-down diff algorithm catches up.\n\tvar lrChangeCount, tdChangeCount int\n\n\tfor multiplexing := true; multiplexing; {\n\t\tselect {\n\t\tcase <-stopChan:\n\t\t\treturn false\n\t\tcase c, ok := <-lrChanges:\n\t\t\tif !ok {\n\t\t\t\t// Left-right diff completed.\n\t\t\t\treturn true\n\t\t\t}\n\t\t\tlrChangeCount++\n\t\t\tif !sendChange(changes, stopChan, c) {\n\t\t\t\treturn false\n\t\t\t}\n\t\tcase c, ok := <-tdChanges:\n\t\t\tif !ok {\n\t\t\t\t// Top-down diff completed.\n\t\t\t\treturn true\n\t\t\t}\n\t\t\ttdChangeCount++\n\t\t\tif tdChangeCount > lrChangeCount {\n\t\t\t\t// Top-down diff has overtaken left-right diff.\n\t\t\t\tif !sendChange(changes, stopChan, c) {\n\t\t\t\t\treturn false\n\t\t\t\t}\n\t\t\t\tlrStopChan <- struct{}{}\n\t\t\t\tmultiplexing = false\n\t\t\t}\n\t\t}\n\t}\n\n\tfor c := range tdChanges {\n\t\tif !sendChange(changes, stopChan, c) {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\n// Streams the diff from |last| to |current| into |changes|, using a top-down approach.\n// Top-down is parallel and efficiently returns the complete diff, but compared to left-right it's slow to start streaming changes.\nfunc orderedSequenceDiffTopDown(last orderedSequence, current orderedSequence, changes chan<- ValueChanged, stopChan <-chan struct{}) bool {\n\treturn orderedSequenceDiffInternalNodes(last, current, changes, stopChan)\n}\n\n// TODO - something other than the literal edit-distance, which is way too much cpu work for this case - https://github.com/attic-labs/noms/issues/2027\nfunc orderedSequenceDiffInternalNodes(last orderedSequence, current orderedSequence, changes chan<- ValueChanged, stopChan <-chan struct{}) bool {\n\tif last.treeLevel() > current.treeLevel() {\n\t\tlastChild := last.getCompositeChildSequence(0, uint64(last.seqLen())).(orderedSequence)\n\t\treturn orderedSequenceDiffInternalNodes(lastChild, current, changes, stopChan)\n\t}\n\n\tif current.treeLevel() > last.treeLevel() {\n\t\tcurrentChild := current.getCompositeChildSequence(0, uint64(current.seqLen())).(orderedSequence)\n\t\treturn orderedSequenceDiffInternalNodes(last, currentChild, changes, stopChan)\n\t}\n\n\tif last.isLeaf() && current.isLeaf() {\n\t\treturn orderedSequenceDiffLeftRight(last, current, changes, stopChan)\n\t}\n\n\tcompareFn := last.getCompareFn(current)\n\tinitialSplices := calcSplices(uint64(last.seqLen()), uint64(current.seqLen()), DEFAULT_MAX_SPLICE_MATRIX_SIZE,\n\t\tfunc(i uint64, j uint64) bool { return compareFn(int(i), int(j)) })\n\n\tfor _, splice := range initialSplices {\n\t\tvar lastChild, currentChild orderedSequence\n\t\tfunctions.All(\n\t\t\tfunc() {\n\t\t\t\tlastChild = last.getCompositeChildSequence(splice.SpAt, splice.SpRemoved).(orderedSequence)\n\t\t\t},\n\t\t\tfunc() {\n\t\t\t\tcurrentChild = current.getCompositeChildSequence(splice.SpFrom, splice.SpAdded).(orderedSequence)\n\t\t\t},\n\t\t)\n\t\tif ok := orderedSequenceDiffInternalNodes(lastChild, currentChild, changes, stopChan); !ok {\n\t\t\treturn false\n\t\t}\n\t}\n\n\treturn true\n}\n\n// Streams the diff from |last| to |current| into |changes|, using a left-right approach.\n// Left-right immediately descends to the first change and starts streaming changes, but compared to top-down it's serial and much slower to calculate the full diff.\nfunc orderedSequenceDiffLeftRight(last orderedSequence, current orderedSequence, changes chan<- ValueChanged, stopChan <-chan struct{}) bool {\n\tlastCur := newCursorAt(last, emptyKey, false, false)\n\tcurrentCur := newCursorAt(current, emptyKey, false, false)\n\n\tfor lastCur.valid() && currentCur.valid() {\n\t\tfastForward(lastCur, currentCur)\n\n\t\tfor lastCur.valid() && currentCur.valid() &&\n\t\t\t!lastCur.seq.getCompareFn(currentCur.seq)(lastCur.idx, currentCur.idx) {\n\t\t\tlastKey := getCurrentKey(lastCur)\n\t\t\tcurrentKey := getCurrentKey(currentCur)\n\t\t\tif currentKey.Less(lastKey) {\n\t\t\t\tif !sendChange(changes, stopChan, ValueChanged{DiffChangeAdded, currentKey.v, nil, getMapValue(currentCur)}) {\n\t\t\t\t\treturn false\n\t\t\t\t}\n\t\t\t\tcurrentCur.advance()\n\t\t\t} else if lastKey.Less(currentKey) {\n\t\t\t\tif !sendChange(changes, stopChan, ValueChanged{DiffChangeRemoved, lastKey.v, getMapValue(lastCur), nil}) {\n\t\t\t\t\treturn false\n\t\t\t\t}\n\t\t\t\tlastCur.advance()\n\t\t\t} else {\n\t\t\t\tif !sendChange(changes, stopChan, ValueChanged{DiffChangeModified, lastKey.v, getMapValue(lastCur), getMapValue(currentCur)}) {\n\t\t\t\t\treturn false\n\t\t\t\t}\n\t\t\t\tlastCur.advance()\n\t\t\t\tcurrentCur.advance()\n\t\t\t}\n\t\t}\n\t}\n\n\tfor lastCur.valid() {\n\t\tif !sendChange(changes, stopChan, ValueChanged{DiffChangeRemoved, getCurrentKey(lastCur).v, getMapValue(lastCur), nil}) {\n\t\t\treturn false\n\t\t}\n\t\tlastCur.advance()\n\t}\n\tfor currentCur.valid() {\n\t\tif !sendChange(changes, stopChan, ValueChanged{DiffChangeAdded, getCurrentKey(currentCur).v, nil, getMapValue(currentCur)}) {\n\t\t\treturn false\n\t\t}\n\t\tcurrentCur.advance()\n\t}\n\n\treturn true\n}\n\n/**\n * Advances |a| and |b| past their common sequence of equal values.\n */\nfunc fastForward(a *sequenceCursor, b *sequenceCursor) {\n\tif a.valid() && b.valid() {\n\t\tdoFastForward(true, a, b)\n\t}\n}\n\nfunc syncWithIdx(cur *sequenceCursor, hasMore bool, allowPastEnd bool) {\n\tcur.sync()\n\tif hasMore {\n\t\tcur.idx = 0\n\t} else if allowPastEnd {\n\t\tcur.idx = cur.length()\n\t} else {\n\t\tcur.idx = cur.length() - 1\n\t}\n}\n\n/*\n * Returns an array matching |a| and |b| respectively to whether that cursor has more values.\n */\nfunc doFastForward(allowPastEnd bool, a *sequenceCursor, b *sequenceCursor) (aHasMore bool, bHasMore bool) {\n\td.PanicIfFalse(a.valid())\n\td.PanicIfFalse(b.valid())\n\taHasMore = true\n\tbHasMore = true\n\n\tfor aHasMore && bHasMore && isCurrentEqual(a, b) {\n\t\tif nil != a.parent && nil != b.parent && isCurrentEqual(a.parent, b.parent) {\n\t\t\t// Key optimisation: if the sequences have common parents, then entire chunks can be\n\t\t\t// fast-forwarded without reading unnecessary data.\n\t\t\taHasMore, bHasMore = doFastForward(false, a.parent, b.parent)\n\t\t\tsyncWithIdx(a, aHasMore, allowPastEnd)\n\t\t\tsyncWithIdx(b, bHasMore, allowPastEnd)\n\t\t} else {\n\t\t\taHasMore = a.advanceMaybeAllowPastEnd(allowPastEnd)\n\t\t\tbHasMore = b.advanceMaybeAllowPastEnd(allowPastEnd)\n\t\t}\n\t}\n\treturn aHasMore, bHasMore\n}\n\nfunc isCurrentEqual(a *sequenceCursor, b *sequenceCursor) bool {\n\treturn a.seq.getCompareFn(b.seq)(a.idx, b.idx)\n}\n" }, { "path": "go/types/ordered_sequences_diff_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst (\n\tlengthOfNumbersTest = 1000\n)\n\ntype diffFn func(last orderedSequence, current orderedSequence, changes chan<- ValueChanged, closeChan <-chan struct{}) bool\n\ntype diffTestSuite struct {\n\tsuite.Suite\n\tfrom1, to1, by1 int\n\tfrom2, to2, by2 int\n\tnumAddsExpected int\n\tnumRemovesExpected int\n\tnumModifiedExpected int\n\tadded ValueSlice\n\tremoved ValueSlice\n\tmodified ValueSlice\n}\n\nfunc newDiffTestSuite(from1, to1, by1, from2, to2, by2, numAddsExpected, numRemovesExpected, numModifiedExpected int) *diffTestSuite {\n\treturn &diffTestSuite{\n\t\tfrom1: from1, to1: to1, by1: by1,\n\t\tfrom2: from2, to2: to2, by2: by2,\n\t\tnumAddsExpected: numAddsExpected,\n\t\tnumRemovesExpected: numRemovesExpected,\n\t\tnumModifiedExpected: numModifiedExpected,\n\t}\n}\n\nfunc accumulateOrderedSequenceDiffChanges(o1, o2 orderedSequence, df diffFn) (added []Value, removed []Value, modified []Value) {\n\tchanges := make(chan ValueChanged)\n\tcloseChan := make(chan struct{})\n\tgo func() {\n\t\tdf(o1, o2, changes, closeChan)\n\t\tclose(changes)\n\t}()\n\tfor change := range changes {\n\t\tif change.ChangeType == DiffChangeAdded {\n\t\t\tadded = append(added, change.Key)\n\t\t} else if change.ChangeType == DiffChangeRemoved {\n\t\t\tremoved = append(removed, change.Key)\n\t\t} else {\n\t\t\tmodified = append(modified, change.Key)\n\t\t}\n\t}\n\treturn\n}\n\nfunc (suite *diffTestSuite) TestDiff() {\n\tvs := newTestValueStore()\n\n\ttype valFn func(int, int, int) ValueSlice\n\ttype colFn func([]Value) Collection\n\n\tnotNil := func(vs []Value) bool {\n\t\tfor _, v := range vs {\n\t\t\tif v == nil {\n\t\t\t\treturn false\n\t\t\t}\n\t\t}\n\t\treturn true\n\t}\n\n\trunTestDf := func(name string, vf valFn, cf colFn, df diffFn) {\n\t\tcol1 := cf(vf(suite.from1, suite.to1, suite.by1))\n\t\tcol2 := cf(vf(suite.from2, suite.to2, suite.by2))\n\t\tsuite.added, suite.removed, suite.modified = accumulateOrderedSequenceDiffChanges(\n\t\t\tcol1.asSequence().(orderedSequence),\n\t\t\tcol2.asSequence().(orderedSequence),\n\t\t\tdf)\n\t\tsuite.Equal(suite.numAddsExpected, len(suite.added), \"test %s: num added is not as expected\", name)\n\t\tsuite.Equal(suite.numRemovesExpected, len(suite.removed), \"test %s: num removed is not as expected\", name)\n\t\tsuite.Equal(suite.numModifiedExpected, len(suite.modified), \"test %s: num modified is not as expected\", name)\n\t\tsuite.True(notNil(suite.added), \"test %s: added has nil values\", name)\n\t\tsuite.True(notNil(suite.removed), \"test %s: removed has nil values\", name)\n\t\tsuite.True(notNil(suite.modified), \"test %s: modified has nil values\", name)\n\t}\n\n\trunTest := func(name string, vf valFn, cf colFn) {\n\t\trunTestDf(name, vf, cf, orderedSequenceDiffTopDown)\n\t\trunTestDf(name, vf, cf, orderedSequenceDiffLeftRight)\n\t\trunTestDf(name, vf, cf, orderedSequenceDiffBest)\n\t}\n\n\tnewSetAsCol := func(vals []Value) Collection { return NewSet(vs, vals...) }\n\tnewMapAsCol := func(vals []Value) Collection { return NewMap(vs, vals...) }\n\n\trw := func(col Collection) Collection {\n\t\th := vs.WriteValue(col).TargetHash()\n\t\tvs.Commit(vs.Root(), vs.Root())\n\t\treturn vs.ReadValue(h).(Collection)\n\t}\n\tnewSetAsColRw := func(vs []Value) Collection { return rw(newSetAsCol(vs)) }\n\tnewMapAsColRw := func(vs []Value) Collection { return rw(newMapAsCol(vs)) }\n\n\trunTest(\"set of numbers\", generateNumbersAsValuesFromToBy, newSetAsCol)\n\trunTest(\"set of numbers (rw)\", generateNumbersAsValuesFromToBy, newSetAsColRw)\n\trunTest(\"set of structs\", generateNumbersAsStructsFromToBy, newSetAsCol)\n\trunTest(\"set of structs (rw)\", generateNumbersAsStructsFromToBy, newSetAsColRw)\n\n\tsuite.to1 *= 2\n\tsuite.to2 *= 2\n\trunTest(\"map of numbers\", generateNumbersAsValuesFromToBy, newMapAsCol)\n\trunTest(\"map of structs\", generateNumbersAsStructsFromToBy, newMapAsColRw)\n\trunTest(\"map of numbers (rw)\", generateNumbersAsValuesFromToBy, newMapAsCol)\n\trunTest(\"map of structs (rw)\", generateNumbersAsStructsFromToBy, newMapAsColRw)\n}\n\nfunc TestOrderedSequencesIdentical(t *testing.T) {\n\tts := newDiffTestSuite(\n\t\t0, lengthOfNumbersTest, 1,\n\t\t0, lengthOfNumbersTest, 1,\n\t\t0, 0, 0)\n\tsuite.Run(t, ts)\n}\n\nfunc TestOrderedSequencesSubset(t *testing.T) {\n\tts1 := newDiffTestSuite(\n\t\t0, lengthOfNumbersTest, 1,\n\t\t0, lengthOfNumbersTest/2, 1,\n\t\t0, lengthOfNumbersTest/2, 0)\n\tts2 := newDiffTestSuite(\n\t\t0, lengthOfNumbersTest/2, 1,\n\t\t0, lengthOfNumbersTest, 1,\n\t\tlengthOfNumbersTest/2, 0, 0)\n\tsuite.Run(t, ts1)\n\tsuite.Run(t, ts2)\n\tts1.True(ts1.added.Equals(ts2.removed), \"added and removed in reverse order diff\")\n\tts1.True(ts1.removed.Equals(ts2.added), \"removed and added in reverse order diff\")\n}\n\nfunc TestOrderedSequencesDisjoint(t *testing.T) {\n\tts1 := newDiffTestSuite(\n\t\t0, lengthOfNumbersTest, 2,\n\t\t1, lengthOfNumbersTest, 2,\n\t\tlengthOfNumbersTest/2, lengthOfNumbersTest/2, 0)\n\tts2 := newDiffTestSuite(\n\t\t1, lengthOfNumbersTest, 2,\n\t\t0, lengthOfNumbersTest, 2,\n\t\tlengthOfNumbersTest/2, lengthOfNumbersTest/2, 0)\n\tsuite.Run(t, ts1)\n\tsuite.Run(t, ts2)\n\tts1.True(ts1.added.Equals(ts2.removed), \"added and removed in disjoint diff\")\n\tts1.True(ts1.removed.Equals(ts2.added), \"removed and added in disjoint diff\")\n}\n\nfunc TestOrderedSequencesDiffCloseWithoutReading(t *testing.T) {\n\tvs := newTestValueStore()\n\n\trunTest := func(df diffFn) {\n\t\ts1 := NewSet(vs).orderedSequence\n\t\t// A single item should be enough, but generate lots anyway.\n\t\ts2 := NewSet(vs, generateNumbersAsValuesFromToBy(0, 1000, 1)...).orderedSequence\n\n\t\tchangeChan := make(chan ValueChanged)\n\t\tcloseChan := make(chan struct{})\n\t\tstopChan := make(chan struct{})\n\n\t\tgo func() {\n\t\t\tdf(s1, s2, changeChan, closeChan)\n\t\t\tstopChan <- struct{}{}\n\t\t}()\n\n\t\tcloseChan <- struct{}{}\n\t\t<-stopChan\n\t}\n\n\trunTest(orderedSequenceDiffBest)\n\trunTest(orderedSequenceDiffLeftRight)\n\trunTest(orderedSequenceDiffTopDown)\n}\n\nfunc TestOrderedSequenceDiffWithMetaNodeGap(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvrw := newTestValueStore()\n\n\tnewSetSequenceMt := func(v ...Value) metaTuple {\n\t\tseq := newSetLeafSequence(vrw, v...)\n\t\tset := newSet(seq)\n\t\treturn newMetaTuple(vrw.WriteValue(set), newOrderedKey(v[len(v)-1]), uint64(len(v)))\n\t}\n\n\tm1 := newSetSequenceMt(Number(1), Number(2))\n\tm2 := newSetSequenceMt(Number(3), Number(4))\n\tm3 := newSetSequenceMt(Number(5), Number(6))\n\ts1 := newSetMetaSequence(1, []metaTuple{m1, m3}, vrw)\n\ts2 := newSetMetaSequence(1, []metaTuple{m1, m2, m3}, vrw)\n\n\trunTest := func(df diffFn) {\n\t\tchanges := make(chan ValueChanged)\n\t\tgo func() {\n\t\t\tdf(s1, s2, changes, nil)\n\t\t\tchanges <- ValueChanged{}\n\t\t\tdf(s2, s1, changes, nil)\n\t\t\tclose(changes)\n\t\t}()\n\n\t\texpected := []ValueChanged{\n\t\t\t{DiffChangeAdded, Number(3), nil, nil},\n\t\t\t{DiffChangeAdded, Number(4), nil, nil},\n\t\t\t{},\n\t\t\t{DiffChangeRemoved, Number(3), nil, nil},\n\t\t\t{DiffChangeRemoved, Number(4), nil, nil},\n\t\t}\n\n\t\ti := 0\n\t\tfor c := range changes {\n\t\t\tassert.Equal(expected[i], c)\n\t\t\ti++\n\t\t}\n\t\tassert.Equal(len(expected), i)\n\t}\n\n\trunTest(orderedSequenceDiffBest)\n\trunTest(orderedSequenceDiffLeftRight)\n\trunTest(orderedSequenceDiffTopDown)\n}\n" }, { "path": "go/types/path.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"fmt\"\n\t\"math\"\n\t\"regexp\"\n\t\"strconv\"\n\t\"strings\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n// For an annotation like @type, 1st capture group is the annotation.\n// For @at(42), 1st capture group is the annotation and 3rd is the parameter.\n// Note, @at() is valid under this regexp, code should deal with the error.\nvar annotationRe = regexp.MustCompile(`^([a-z]+)(\\(([\\w\\-\"']*)\\))?`)\n\n// A Path locates a value in Noms relative to some other value. For locating\n// values absolutely within a database, see AbsolutePath. To locate values\n// globally, see Spec.\n//\n// For more details, see:\n// https://github.com/attic-labs/noms/blob/master/doc/spelling.md.\ntype Path []PathPart\n\ntype PathPart interface {\n\tResolve(v Value, vr ValueReader) Value\n\tString() string\n}\n\n// ParsePath parses str into a Path, or returns an error if parsing failed.\nfunc ParsePath(str string) (Path, error) {\n\tif str == \"\" {\n\t\treturn Path{}, errors.New(\"Empty path\")\n\t}\n\treturn constructPath(Path{}, str)\n}\n\n// MustParsePath parses str into a Path, or panics if parsing failed.\nfunc MustParsePath(str string) Path {\n\tp, err := ParsePath(str)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\treturn p\n}\n\ntype keyIndexable interface {\n\tsetIntoKey(v bool) keyIndexable\n}\n\nfunc constructPath(p Path, str string) (Path, error) {\n\tif len(str) == 0 {\n\t\treturn p, nil\n\t}\n\n\top, tail := str[0], str[1:]\n\n\tswitch op {\n\tcase '.':\n\t\tidx := fieldNameComponentRe.FindIndex([]byte(tail))\n\t\tif idx == nil {\n\t\t\treturn Path{}, errors.New(\"Invalid field: \" + tail)\n\t\t}\n\t\tp = append(p, FieldPath{tail[:idx[1]]})\n\t\treturn constructPath(p, tail[idx[1]:])\n\n\tcase '[':\n\t\tif len(tail) == 0 {\n\t\t\treturn Path{}, errors.New(\"Path ends in [\")\n\t\t}\n\n\t\tidx, h, rem, err := ParsePathIndex(tail)\n\t\tif err != nil {\n\t\t\treturn Path{}, err\n\t\t}\n\t\tif !strings.HasPrefix(rem, \"]\") {\n\t\t\treturn Path{}, errors.New(\"[ is missing closing ]\")\n\t\t}\n\t\td.PanicIfTrue(idx == nil && h.IsEmpty())\n\t\td.PanicIfTrue(idx != nil && !h.IsEmpty())\n\n\t\tif idx != nil {\n\t\t\tp = append(p, NewIndexPath(idx))\n\t\t} else {\n\t\t\tp = append(p, NewHashIndexPath(h))\n\t\t}\n\t\treturn constructPath(p, rem[1:])\n\n\tcase '@':\n\t\tann, hasArg, arg, rem := getAnnotation(tail)\n\n\t\tswitch ann {\n\t\tcase \"at\":\n\t\t\tif arg == \"\" {\n\t\t\t\treturn Path{}, fmt.Errorf(\"@at annotation requires a position argument\")\n\t\t\t}\n\t\t\tidx, err := strconv.ParseInt(arg, 10, 64)\n\t\t\tif err != nil {\n\t\t\t\treturn Path{}, fmt.Errorf(\"Invalid position: %s\", arg)\n\t\t\t}\n\t\t\treturn constructPath(append(p, NewAtAnnotation(idx)), rem)\n\n\t\tcase \"key\":\n\t\t\tif hasArg {\n\t\t\t\treturn Path{}, fmt.Errorf(\"@key annotation does not support arguments\")\n\t\t\t}\n\t\t\tif len(p) == 0 {\n\t\t\t\treturn Path{}, fmt.Errorf(\"Cannot use @key annotation at beginning of path\")\n\t\t\t}\n\t\t\tlastPart := p[len(p)-1]\n\t\t\tif ki, ok := lastPart.(keyIndexable); ok {\n\t\t\t\tp[len(p)-1] = ki.setIntoKey(true).(PathPart)\n\t\t\t\treturn constructPath(p, rem)\n\t\t\t}\n\t\t\treturn Path{}, fmt.Errorf(\"Cannot use @key annotation on: %s\", lastPart.String())\n\n\t\tcase \"target\":\n\t\t\tif hasArg {\n\t\t\t\treturn Path{}, fmt.Errorf(\"@target annotation does not support arguments\")\n\t\t\t}\n\t\t\treturn constructPath(append(p, TargetAnnotation{}), rem)\n\n\t\tcase \"type\":\n\t\t\tif hasArg {\n\t\t\t\treturn Path{}, fmt.Errorf(\"@type annotation does not support arguments\")\n\t\t\t}\n\t\t\treturn constructPath(append(p, TypeAnnotation{}), rem)\n\n\t\tdefault:\n\t\t\treturn Path{}, fmt.Errorf(\"Unsupported annotation: @%s\", ann)\n\t\t}\n\n\tcase ']':\n\t\treturn Path{}, errors.New(\"] is missing opening [\")\n\n\tdefault:\n\t\treturn Path{}, fmt.Errorf(\"Invalid operator: %c\", op)\n\t}\n}\n\n// Resolve resolves a path relative to some value.\n// A ValueReader is required to resolve paths that contain the @target annotation.\nfunc (p Path) Resolve(v Value, vr ValueReader) (resolved Value) {\n\tresolved = v\n\tfor _, part := range p {\n\t\tif resolved == nil {\n\t\t\tbreak\n\t\t}\n\t\tresolved = part.Resolve(resolved, vr)\n\t}\n\n\treturn\n}\n\nfunc (p Path) Equals(o Path) bool {\n\tif len(p) != len(o) {\n\t\treturn false\n\t}\n\tfor i, pp := range p {\n\t\tif pp != o[i] {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\n// Append makes a copy of a p and appends the PathPart 'pp' to it.\nfunc (p Path) Append(pp PathPart) Path {\n\tp1 := make(Path, len(p), len(p)+1)\n\tcopy(p1, p)\n\treturn append(p1, pp)\n}\n\nfunc (p Path) String() string {\n\tstrs := make([]string, 0, len(p))\n\tfor _, part := range p {\n\t\tstrs = append(strs, part.String())\n\t}\n\treturn strings.Join(strs, \"\")\n}\n\nfunc (p Path) IsEmpty() bool {\n\treturn len(p) == 0\n}\n\n// FieldPath references Struct field values by name.\ntype FieldPath struct {\n\t// The name of the field, e.g. `.Name`.\n\tName string\n}\n\nfunc NewFieldPath(name string) FieldPath {\n\treturn FieldPath{name}\n}\n\nfunc (fp FieldPath) Resolve(v Value, vr ValueReader) Value {\n\tswitch v := v.(type) {\n\tcase Struct:\n\t\tif sv, ok := v.MaybeGet(fp.Name); ok {\n\t\t\treturn sv\n\t\t}\n\tcase *Type:\n\t\tif desc, ok := v.Desc.(StructDesc); ok {\n\t\t\tif df, _ := desc.Field(fp.Name); df != nil {\n\t\t\t\treturn df\n\t\t\t}\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc (fp FieldPath) String() string {\n\treturn fmt.Sprintf(\".%s\", fp.Name)\n}\n\n// IndexPath ndexes into Maps and Lists by key or index.\ntype IndexPath struct {\n\t// The value of the index, e.g. `[42]` or `[\"value\"]`. If Index is a negative\n\t// number and the path is resolved in a List, it means index from the back.\n\tIndex Value\n\t// Whether this index should resolve to the key of a map, given by a `@key`\n\t// annotation. Typically IntoKey is false, and indices would resolve to the\n\t// values. E.g. given `{a: 42}` then `[\"a\"]` resolves to `42`. If IntoKey is\n\t// true, then it resolves to `\"a\"`. For IndexPath this isn't particularly\n\t// useful - it's mostly provided for consistency with HashIndexPath - but\n\t// note that given `{a: 42}` then `[\"b\"]` resolves to nil, not `\"b\"`.\n\tIntoKey bool\n}\n\nfunc NewIndexPath(idx Value) IndexPath {\n\treturn newIndexPath(idx, false)\n}\n\nfunc NewIndexIntoKeyPath(idx Value) IndexPath {\n\treturn newIndexPath(idx, true)\n}\n\nfunc ValueCanBePathIndex(v Value) bool {\n\tk := v.Kind()\n\treturn k == StringKind || k == BoolKind || k == NumberKind\n}\n\nfunc newIndexPath(idx Value, intoKey bool) IndexPath {\n\td.PanicIfFalse(ValueCanBePathIndex(idx))\n\treturn IndexPath{idx, intoKey}\n}\n\nfunc (ip IndexPath) Resolve(v Value, vr ValueReader) Value {\n\tseqIndex := func(getter func(i uint64) Value) Value {\n\t\tn, ok := ip.Index.(Number)\n\t\tif !ok {\n\t\t\treturn nil\n\t\t}\n\t\tf := float64(n)\n\t\tif f != math.Trunc(f) {\n\t\t\treturn nil\n\t\t}\n\t\tai, ok := getAbsoluteIndex(v, int64(f))\n\t\tif !ok {\n\t\t\treturn nil\n\t\t}\n\t\tif ip.IntoKey {\n\t\t\treturn Number(ai)\n\t\t}\n\t\treturn getter(ai)\n\t}\n\n\tswitch v := v.(type) {\n\tcase List:\n\t\treturn seqIndex(func(i uint64) Value { return v.Get(i) })\n\tcase *Type:\n\t\tif cd, ok := v.Desc.(CompoundDesc); ok {\n\t\t\treturn seqIndex(func(i uint64) Value { return cd.ElemTypes[i] })\n\t\t}\n\tcase Map:\n\t\tif !ip.IntoKey {\n\t\t\treturn v.Get(ip.Index)\n\t\t}\n\t\tif v.Has(ip.Index) {\n\t\t\treturn ip.Index\n\t\t}\n\t}\n\n\treturn nil\n}\n\nfunc (ip IndexPath) String() (str string) {\n\tstr = fmt.Sprintf(\"[%s]\", EncodedIndexValue(ip.Index))\n\tif ip.IntoKey {\n\t\tstr += \"@key\"\n\t}\n\treturn\n}\n\nfunc (ip IndexPath) setIntoKey(v bool) keyIndexable {\n\tip.IntoKey = v\n\treturn ip\n}\n\n// Indexes into Maps by the hash of a key, or a Set by the hash of a value.\ntype HashIndexPath struct {\n\t// The hash of the key or value to search for. Maps and Set are ordered, so\n\t// this in O(log(size)).\n\tHash hash.Hash\n\t// Whether this index should resolve to the key of a map, given by a `@key`\n\t// annotation. Typically IntoKey is false, and indices would resolve to the\n\t// values. E.g. given `{a: 42}` and if the hash of `\"a\"` is `#abcd`, then\n\t// `[#abcd]` resolves to `42`. If IntoKey is true, then it resolves to `\"a\"`.\n\t// This is useful for when Map keys aren't primitive values, e.g. a struct,\n\t// since struct literals can't be spelled using a Path.\n\tIntoKey bool\n}\n\nfunc NewHashIndexPath(h hash.Hash) HashIndexPath {\n\treturn newHashIndexPath(h, false)\n}\n\nfunc NewHashIndexIntoKeyPath(h hash.Hash) HashIndexPath {\n\treturn newHashIndexPath(h, true)\n}\n\nfunc newHashIndexPath(h hash.Hash, intoKey bool) HashIndexPath {\n\td.PanicIfTrue(h.IsEmpty())\n\treturn HashIndexPath{h, intoKey}\n}\n\nfunc (hip HashIndexPath) Resolve(v Value, vr ValueReader) (res Value) {\n\tvar seq orderedSequence\n\tvar getCurrentValue func(cur *sequenceCursor) Value\n\n\tswitch v := v.(type) {\n\tcase Set:\n\t\t// Unclear what the behavior should be if |hip.IntoKey| is true, but ignoring it for sets is arguably correct.\n\t\tseq = v.orderedSequence\n\t\tgetCurrentValue = func(cur *sequenceCursor) Value { return cur.current().(Value) }\n\tcase Map:\n\t\tseq = v.orderedSequence\n\t\tif hip.IntoKey {\n\t\t\tgetCurrentValue = func(cur *sequenceCursor) Value { return cur.current().(mapEntry).key }\n\t\t} else {\n\t\t\tgetCurrentValue = func(cur *sequenceCursor) Value { return cur.current().(mapEntry).value }\n\t\t}\n\tdefault:\n\t\treturn nil\n\t}\n\n\tcur := newCursorAt(seq, orderedKeyFromHash(hip.Hash), false, false)\n\tif !cur.valid() {\n\t\treturn nil\n\t}\n\n\tif getCurrentKey(cur).h != hip.Hash {\n\t\treturn nil\n\t}\n\n\treturn getCurrentValue(cur)\n}\n\nfunc (hip HashIndexPath) String() (str string) {\n\tstr = fmt.Sprintf(\"[#%s]\", hip.Hash.String())\n\tif hip.IntoKey {\n\t\tstr += \"@key\"\n\t}\n\treturn\n}\n\nfunc (hip HashIndexPath) setIntoKey(v bool) keyIndexable {\n\thip.IntoKey = v\n\treturn hip\n}\n\n// Parse a Noms value from the path index syntax.\n// 4 -> types.Number\n// \"4\" -> types.String\n// true|false -> types.Boolean\n// # -> hash.Hash\nfunc ParsePathIndex(str string) (idx Value, h hash.Hash, rem string, err error) {\nSwitch:\n\tswitch str[0] {\n\tcase '\"':\n\t\t// String is complicated because ] might be quoted, and \" or \\ might be escaped.\n\t\tstringBuf := bytes.Buffer{}\n\t\ti := 1\n\n\t\tfor ; i < len(str); i++ {\n\t\t\tc := str[i]\n\t\t\tif c == '\"' {\n\t\t\t\ti++\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tif c == '\\\\' && i < len(str)-1 {\n\t\t\t\ti++\n\t\t\t\tc = str[i]\n\t\t\t\tif c != '\\\\' && c != '\"' {\n\t\t\t\t\terr = errors.New(`Only \" and \\ can be escaped`)\n\t\t\t\t\tbreak Switch\n\t\t\t\t}\n\t\t\t}\n\t\t\tstringBuf.WriteByte(c)\n\t\t}\n\n\t\tidx = String(stringBuf.String())\n\t\trem = str[i:]\n\n\tdefault:\n\t\tidxStr := str\n\t\tsepIdx := strings.Index(str, \"]\")\n\t\tif sepIdx >= 0 {\n\t\t\tidxStr = str[:sepIdx]\n\t\t\trem = str[sepIdx:]\n\t\t}\n\t\tif len(idxStr) == 0 {\n\t\t\terr = errors.New(\"Empty index value\")\n\t\t} else if idxStr[0] == '#' {\n\t\t\thashStr := idxStr[1:]\n\t\t\th, _ = hash.MaybeParse(hashStr)\n\t\t\tif h.IsEmpty() {\n\t\t\t\terr = errors.New(\"Invalid hash: \" + hashStr)\n\t\t\t}\n\t\t} else if idxStr == \"true\" {\n\t\t\tidx = Bool(true)\n\t\t} else if idxStr == \"false\" {\n\t\t\tidx = Bool(false)\n\t\t} else if i, err2 := strconv.ParseFloat(idxStr, 64); err2 == nil {\n\t\t\t// Should we be more strict here? ParseFloat allows leading and trailing dots, and exponents.\n\t\t\tidx = Number(i)\n\t\t} else {\n\t\t\terr = errors.New(\"Invalid index: \" + idxStr)\n\t\t}\n\t}\n\n\treturn\n}\n\n// TypeAnnotation is a PathPart annotation to resolve to the type of the value\n// it's resolved in.\ntype TypeAnnotation struct {\n}\n\nfunc (ann TypeAnnotation) Resolve(v Value, vr ValueReader) Value {\n\treturn TypeOf(v)\n}\n\nfunc (ann TypeAnnotation) String() string {\n\treturn \"@type\"\n}\n\n// TargetAnnotation is a PathPart annotation to resolve to the targetValue of the Ref it is resolved on.\ntype TargetAnnotation struct {\n}\n\nfunc (ann TargetAnnotation) Resolve(v Value, vr ValueReader) Value {\n\tif vr == nil {\n\t\td.Panic(\"@target annotation requires a database to resolve against\")\n\t}\n\tif r, ok := v.(Ref); ok {\n\t\treturn r.TargetValue(vr)\n\t} else {\n\t\treturn nil\n\t}\n}\n\nfunc (ann TargetAnnotation) String() string {\n\treturn \"@target\"\n}\n\n// AtAnnotation is a PathPart annotation that gets the value of a collection at\n// a position, rather than a key. This is equivalent to IndexPath for lists,\n// but different for sets and maps.\ntype AtAnnotation struct {\n\t// Index is the position to resolve at. If negative, it means an index\n\t// relative to the end of the collection.\n\tIndex int64\n\t// IntoKey see IndexPath.IntoKey.\n\tIntoKey bool\n}\n\nfunc NewAtAnnotation(idx int64) AtAnnotation {\n\treturn AtAnnotation{idx, false}\n}\n\nfunc NewAtAnnotationIntoKeyPath(idx int64) AtAnnotation {\n\treturn AtAnnotation{idx, true}\n}\n\nfunc (ann AtAnnotation) Resolve(v Value, vr ValueReader) Value {\n\tai, ok := getAbsoluteIndex(v, ann.Index)\n\tif !ok {\n\t\treturn nil\n\t}\n\n\tswitch v := v.(type) {\n\tcase List:\n\t\tif !ann.IntoKey {\n\t\t\treturn v.Get(ai)\n\t\t}\n\tcase Set:\n\t\treturn v.At(ai)\n\tcase Map:\n\t\tk, mapv := v.At(ai)\n\t\tif ann.IntoKey {\n\t\t\treturn k\n\t\t}\n\t\treturn mapv\n\tcase *Type:\n\t\tif cd, ok := v.Desc.(CompoundDesc); ok {\n\t\t\treturn cd.ElemTypes[ai]\n\t\t}\n\t}\n\n\treturn nil\n}\n\nfunc (ann AtAnnotation) String() (str string) {\n\tstr = fmt.Sprintf(\"@at(%d)\", ann.Index)\n\tif ann.IntoKey {\n\t\tstr += \"@key\"\n\t}\n\treturn\n}\n\nfunc (ann AtAnnotation) setIntoKey(v bool) keyIndexable {\n\tann.IntoKey = v\n\treturn ann\n}\n\nfunc getAnnotation(str string) (ann string, hasArg bool, arg, rem string) {\n\tparts := annotationRe.FindStringSubmatch(str)\n\tif parts == nil {\n\t\treturn\n\t}\n\n\tann = parts[1]\n\thasArg = parts[2] != \"\"\n\targ = parts[3]\n\trem = str[len(parts[0]):]\n\treturn\n}\n\nfunc getAbsoluteIndex(v Value, relIdx int64) (absIdx uint64, ok bool) {\n\tvar l uint64\n\tswitch v := v.(type) {\n\tcase Collection:\n\t\tl = v.Len()\n\tcase *Type:\n\t\tif cd, cdOK := v.Desc.(CompoundDesc); cdOK {\n\t\t\tl = uint64(len(cd.ElemTypes))\n\t\t} else {\n\t\t\treturn\n\t\t}\n\tdefault:\n\t\treturn\n\t}\n\n\tif relIdx < 0 {\n\t\tif uint64(-relIdx) > l {\n\t\t\treturn\n\t\t}\n\t\tabsIdx = l - uint64(-relIdx)\n\t} else {\n\t\tif uint64(relIdx) >= l {\n\t\t\treturn\n\t\t}\n\t\tabsIdx = uint64(relIdx)\n\t}\n\n\tok = true\n\treturn\n}\n" }, { "path": "go/types/path_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc hashIdx(v Value) string {\n\treturn fmt.Sprintf(\"[#%s]\", v.Hash().String())\n}\n\nfunc assertResolvesTo(assert *assert.Assertions, expect, ref Value, str string) {\n\tassertResolvesToWithVR(assert, expect, ref, str, nil)\n}\n\nfunc assertResolvesToWithVR(assert *assert.Assertions, expect, ref Value, str string, vr ValueReader) {\n\tp, err := ParsePath(str)\n\tassert.NoError(err)\n\tactual := p.Resolve(ref, vr)\n\tif expect == nil {\n\t\tif actual != nil {\n\t\t\tassert.Fail(\"\", \"Expected nil, but got %s\", EncodedValue(actual))\n\t\t}\n\t} else if actual == nil {\n\t\tassert.Fail(\"\", \"Expected %s, but got nil\", EncodedValue(expect))\n\t} else {\n\t\tassert.True(expect.Equals(actual), \"Expected %s, but got %s\", EncodedValue(expect), EncodedValue(actual))\n\t}\n}\n\nfunc TestPathStruct(t *testing.T) {\n\tassert := assert.New(t)\n\n\tv := NewStruct(\"\", StructData{\n\t\t\"foo\": String(\"foo\"),\n\t\t\"bar\": Bool(false),\n\t\t\"baz\": Number(203),\n\t})\n\n\tassertResolvesTo(assert, String(\"foo\"), v, `.foo`)\n\tassertResolvesTo(assert, Bool(false), v, `.bar`)\n\tassertResolvesTo(assert, Number(203), v, `.baz`)\n\tassertResolvesTo(assert, nil, v, `.notHere`)\n\n\tv2 := NewStruct(\"\", StructData{\n\t\t\"v1\": v,\n\t})\n\n\tassertResolvesTo(assert, String(\"foo\"), v2, `.v1.foo`)\n\tassertResolvesTo(assert, Bool(false), v2, `.v1.bar`)\n\tassertResolvesTo(assert, Number(203), v2, `.v1.baz`)\n\tassertResolvesTo(assert, nil, v2, `.v1.notHere`)\n\tassertResolvesTo(assert, nil, v2, `.notHere.v1`)\n}\n\nfunc TestPathStructType(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttyp := MakeStructType(\"MyStruct\",\n\t\tStructField{Name: \"foo\", Type: StringType},\n\t\tStructField{Name: \"bar\", Type: BoolType},\n\t\tStructField{Name: \"baz\", Type: NumberType},\n\t)\n\n\tassertResolvesTo(assert, StringType, typ, `.foo`)\n\tassertResolvesTo(assert, BoolType, typ, `.bar`)\n\tassertResolvesTo(assert, NumberType, typ, `.baz`)\n\tassertResolvesTo(assert, nil, typ, `.notHere`)\n\n\ttyp2 := MakeStructType(\"\",\n\t\tStructField{Name: \"typ\", Type: typ},\n\t)\n\n\tassertResolvesTo(assert, typ, typ2, `.typ`)\n\tassertResolvesTo(assert, StringType, typ2, `.typ.foo`)\n\tassertResolvesTo(assert, BoolType, typ2, `.typ.bar`)\n\tassertResolvesTo(assert, NumberType, typ2, `.typ.baz`)\n\tassertResolvesTo(assert, nil, typ2, `.typ.notHere`)\n\tassertResolvesTo(assert, nil, typ2, `.notHere.typ`)\n}\n\nfunc TestPathIndex(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tvar v Value\n\tresolvesTo := func(expVal, expKey Value, str string) {\n\t\tassertResolvesTo(assert, expVal, v, str)\n\t\tassertResolvesTo(assert, expKey, v, str+\"@key\")\n\t}\n\n\tv = NewList(vs, Number(1), Number(3), String(\"foo\"), Bool(false))\n\n\tresolvesTo(Number(1), Number(0), \"[0]\")\n\tresolvesTo(Number(3), Number(1), \"[1]\")\n\tresolvesTo(String(\"foo\"), Number(2), \"[2]\")\n\tresolvesTo(Bool(false), Number(3), \"[3]\")\n\tresolvesTo(nil, nil, \"[4]\")\n\tresolvesTo(nil, nil, \"[-5]\")\n\tresolvesTo(Number(1), Number(0), \"[-4]\")\n\tresolvesTo(Number(3), Number(1), \"[-3]\")\n\tresolvesTo(String(\"foo\"), Number(2), \"[-2]\")\n\tresolvesTo(Bool(false), Number(3), \"[-1]\")\n\n\tv = NewMap(vs,\n\t\tBool(false), Number(23),\n\t\tNumber(1), String(\"foo\"),\n\t\tNumber(2.3), Number(4.5),\n\t\tString(\"two\"), String(\"bar\"),\n\t)\n\n\tresolvesTo(String(\"foo\"), Number(1), \"[1]\")\n\tresolvesTo(String(\"bar\"), String(\"two\"), `[\"two\"]`)\n\tresolvesTo(Number(23), Bool(false), \"[false]\")\n\tresolvesTo(Number(4.5), Number(2.3), \"[2.3]\")\n\tresolvesTo(nil, nil, \"[4]\")\n}\n\nfunc TestPathIndexType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tst := MakeSetType(NumberType)\n\tlt := MakeListType(st)\n\tmt := MakeMapType(st, lt)\n\tut := MakeUnionType(lt, mt, st)\n\n\tassertResolvesTo(assert, NumberType, st, \"[0]\")\n\tassertResolvesTo(assert, NumberType, st, \"[-1]\")\n\tassertResolvesTo(assert, NumberType, st, \"@at(0)\")\n\tassertResolvesTo(assert, nil, st, \"[1]\")\n\tassertResolvesTo(assert, nil, st, \"[-2]\")\n\n\tassertResolvesTo(assert, st, lt, \"[0]\")\n\tassertResolvesTo(assert, st, lt, \"[-1]\")\n\tassertResolvesTo(assert, NumberType, lt, \"[0][0]\")\n\tassertResolvesTo(assert, NumberType, lt, \"@at(0)@at(0)\")\n\tassertResolvesTo(assert, nil, lt, \"[1]\")\n\tassertResolvesTo(assert, nil, lt, \"[-2]\")\n\n\tassertResolvesTo(assert, st, mt, \"[0]\")\n\tassertResolvesTo(assert, st, mt, \"[-2]\")\n\tassertResolvesTo(assert, lt, mt, \"[1]\")\n\tassertResolvesTo(assert, lt, mt, \"[-1]\")\n\tassertResolvesTo(assert, NumberType, mt, \"[1][0][0]\")\n\tassertResolvesTo(assert, NumberType, mt, \"@at(1)@at(0)@at(0)\")\n\tassertResolvesTo(assert, nil, mt, \"[2]\")\n\tassertResolvesTo(assert, nil, mt, \"[-3]\")\n\n\tassertResolvesTo(assert, lt, ut, \"[0]\")\n\tassertResolvesTo(assert, lt, ut, \"[-3]\")\n\tassertResolvesTo(assert, mt, ut, \"[1]\")\n\tassertResolvesTo(assert, mt, ut, \"[-2]\")\n\tassertResolvesTo(assert, st, ut, \"[2]\")\n\tassertResolvesTo(assert, st, ut, \"[-1]\")\n\tassertResolvesTo(assert, NumberType, ut, \"[1][1][0][0]\")\n\tassertResolvesTo(assert, NumberType, ut, \"@at(1)@at(1)@at(0)@at(0)\")\n\tassertResolvesTo(assert, nil, ut, \"[3]\")\n\tassertResolvesTo(assert, nil, ut, \"[-4]\")\n}\n\nfunc TestPathHashIndex(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tb := Bool(true)\n\tbr := NewRef(b)\n\ti := Number(0)\n\tstr := String(\"foo\")\n\tl := NewList(vs, b, i, str)\n\tlr := NewRef(l)\n\tm := NewMap(vs,\n\t\tb, br,\n\t\tbr, i,\n\t\ti, str,\n\t\tl, lr,\n\t\tlr, b,\n\t)\n\ts := NewSet(vs, b, br, i, str, l, lr)\n\n\tresolvesTo := func(col, key, expVal, expKey Value) {\n\t\tassertResolvesTo(assert, expVal, col, hashIdx(key))\n\t\tassertResolvesTo(assert, expKey, col, hashIdx(key)+\"@key\")\n\t}\n\n\t// Primitives are only addressable by their values.\n\tresolvesTo(m, b, nil, nil)\n\tresolvesTo(m, i, nil, nil)\n\tresolvesTo(m, str, nil, nil)\n\tresolvesTo(s, b, nil, nil)\n\tresolvesTo(s, i, nil, nil)\n\tresolvesTo(s, str, nil, nil)\n\n\t// Other values are only addressable by their hashes.\n\tresolvesTo(m, br, i, br)\n\tresolvesTo(m, l, lr, l)\n\tresolvesTo(m, lr, b, lr)\n\tresolvesTo(s, br, br, br)\n\tresolvesTo(s, l, l, l)\n\tresolvesTo(s, lr, lr, lr)\n\n\t// Lists cannot be addressed by hashes, obviously.\n\tresolvesTo(l, i, nil, nil)\n}\n\nfunc TestPathHashIndexOfSingletonCollection(t *testing.T) {\n\t// This test is to make sure we don't accidentally return |b| if it's the only element.\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tresolvesToNil := func(col, val Value) {\n\t\tassertResolvesTo(assert, nil, col, hashIdx(val))\n\t}\n\n\tb := Bool(true)\n\tresolvesToNil(NewMap(vs, b, b), b)\n\tresolvesToNil(NewSet(vs, b), b)\n}\n\nfunc TestPathMulti(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tm1 := NewMap(vs,\n\t\tString(\"a\"), String(\"foo\"),\n\t\tString(\"b\"), String(\"bar\"),\n\t\tString(\"c\"), String(\"car\"),\n\t)\n\n\tm2 := NewMap(vs,\n\t\tBool(false), String(\"earth\"),\n\t\tString(\"d\"), String(\"dar\"),\n\t\tm1, String(\"fire\"),\n\t)\n\n\tl := NewList(vs, m1, m2)\n\n\ts := NewStruct(\"\", StructData{\n\t\t\"foo\": l,\n\t})\n\n\tassertResolvesTo(assert, l, s, `.foo`)\n\tassertResolvesTo(assert, m1, s, `.foo[0]`)\n\tassertResolvesTo(assert, String(\"foo\"), s, `.foo[0][\"a\"]`)\n\tassertResolvesTo(assert, String(\"bar\"), s, `.foo[0][\"b\"]`)\n\tassertResolvesTo(assert, String(\"car\"), s, `.foo[0][\"c\"]`)\n\tassertResolvesTo(assert, String(\"foo\"), s, `.foo[0]@at(0)`)\n\tassertResolvesTo(assert, String(\"bar\"), s, `.foo[0]@at(1)`)\n\tassertResolvesTo(assert, String(\"car\"), s, `.foo[0]@at(2)`)\n\tassertResolvesTo(assert, nil, s, `.foo[0][\"x\"]`)\n\tassertResolvesTo(assert, nil, s, `.foo[0]@at(3)`)\n\tassertResolvesTo(assert, nil, s, `.foo[2][\"c\"]`)\n\tassertResolvesTo(assert, nil, s, `.notHere[0][\"c\"]`)\n\tassertResolvesTo(assert, m2, s, `.foo[1]`)\n\tassertResolvesTo(assert, String(\"dar\"), s, `.foo[1][\"d\"]`)\n\tassertResolvesTo(assert, String(\"earth\"), s, `.foo[1][false]`)\n\tassertResolvesTo(assert, String(\"fire\"), s, fmt.Sprintf(`.foo[1]%s`, hashIdx(m1)))\n\tassertResolvesTo(assert, m1, s, fmt.Sprintf(`.foo[1]%s@key`, hashIdx(m1)))\n\tassertResolvesTo(assert, String(\"car\"), s, fmt.Sprintf(`.foo[1]%s@key[\"c\"]`, hashIdx(m1)))\n\tassertResolvesTo(assert, String(\"fire\"), s, `.foo[1]@at(2)`)\n\tassertResolvesTo(assert, m1, s, `.foo[1]@at(2)@key`)\n\tassertResolvesTo(assert, String(\"car\"), s, `.foo[1]@at(2)@key@at(2)`)\n\tassertResolvesTo(assert, String(\"fire\"), s, `.foo[1]@at(-1)`)\n\tassertResolvesTo(assert, m1, s, `.foo[1]@at(-1)@key`)\n\tassertResolvesTo(assert, String(\"car\"), s, `.foo[1]@at(-1)@key@at(-1)`)\n}\n\nfunc TestPathParseSuccess(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttest := func(str string) {\n\t\tp, err := ParsePath(str)\n\t\tassert.NoError(err)\n\t\texpectStr := str\n\t\tswitch expectStr { // Human readable serialization special cases.\n\t\tcase \"[1e4]\":\n\t\t\texpectStr = \"[10000]\"\n\t\tcase \"[1.]\":\n\t\t\texpectStr = \"[1]\"\n\t\tcase \"[\\\"line\\nbreak\\rreturn\\\"]\":\n\t\t\texpectStr = `[\"line\\nbreak\\rreturn\"]`\n\t\t}\n\t\tassert.Equal(expectStr, p.String())\n\t}\n\n\th := Number(42).Hash() // arbitrary hash\n\n\ttest(\".foo\")\n\ttest(\".foo@type\")\n\ttest(\".Q\")\n\ttest(\".QQ\")\n\ttest(\"[true]\")\n\ttest(\"[true]@type\")\n\ttest(\"[false]\")\n\ttest(\"[false]@key\")\n\ttest(\"[false]@key@type\")\n\ttest(\"[false]@key@type@at(42)\")\n\ttest(\"[42]\")\n\ttest(\"[42]@key\")\n\ttest(\"[42]@at(-101)\")\n\ttest(\"[1e4]\")\n\ttest(\"[1.]\")\n\ttest(\"[1.345]\")\n\ttest(`[\"\"]`)\n\ttest(`[\"42\"]`)\n\ttest(`[\"42\"]@key`)\n\ttest(\"[\\\"line\\nbreak\\rreturn\\\"]\")\n\ttest(`[\"qu\\\\ote\\\"\"]`)\n\ttest(`[\"π\"]`)\n\ttest(`[\"[[br][]acke]]ts\"]`)\n\ttest(`[\"xπy✌z\"]`)\n\ttest(`[\"ಠ_ಠ\"]`)\n\ttest(`[\"0\"][\"1\"][\"100\"]`)\n\ttest(\".foo[0].bar[4.5][false]\")\n\ttest(fmt.Sprintf(\".foo[#%s]\", h.String()))\n\ttest(fmt.Sprintf(\".bar[#%s]@key\", h.String()))\n}\n\nfunc TestPathParseErrors(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttest := func(str, expectError string) {\n\t\tp, err := ParsePath(str)\n\t\tassert.Equal(Path{}, p)\n\t\tif err != nil {\n\t\t\tassert.Equal(expectError, err.Error())\n\t\t} else {\n\t\t\tassert.Fail(\"Expected \" + expectError)\n\t\t}\n\t}\n\n\ttest(\"\", \"Empty path\")\n\ttest(\".\", \"Invalid field: \")\n\ttest(\"[\", \"Path ends in [\")\n\ttest(\"]\", \"] is missing opening [\")\n\ttest(\".#\", \"Invalid field: #\")\n\ttest(\". \", \"Invalid field: \")\n\ttest(\". invalid.field\", \"Invalid field: invalid.field\")\n\ttest(\".foo.\", \"Invalid field: \")\n\ttest(\".foo.#invalid.field\", \"Invalid field: #invalid.field\")\n\ttest(\".foo!\", \"Invalid operator: !\")\n\ttest(\".foo!bar\", \"Invalid operator: !\")\n\ttest(\".foo#\", \"Invalid operator: #\")\n\ttest(\".foo#bar\", \"Invalid operator: #\")\n\ttest(\".foo[\", \"Path ends in [\")\n\ttest(\".foo[.bar\", \"Invalid index: .bar\")\n\ttest(\".foo]\", \"] is missing opening [\")\n\ttest(\".foo].bar\", \"] is missing opening [\")\n\ttest(\".foo[]\", \"Empty index value\")\n\ttest(\".foo[[]\", \"Invalid index: [\")\n\ttest(\".foo[[]]\", \"Invalid index: [\")\n\ttest(\".foo[42.1.2]\", \"Invalid index: 42.1.2\")\n\ttest(\".foo[1f4]\", \"Invalid index: 1f4\")\n\ttest(\".foo[hello]\", \"Invalid index: hello\")\n\ttest(\".foo['hello']\", \"Invalid index: 'hello'\")\n\ttest(`.foo[\\]`, `Invalid index: \\`)\n\ttest(`.foo[\\\\]`, `Invalid index: \\\\`)\n\ttest(`.foo[\"hello]`, \"[ is missing closing ]\")\n\ttest(`.foo[\"hello`, \"[ is missing closing ]\")\n\ttest(`.foo[\"hello\"`, \"[ is missing closing ]\")\n\ttest(`.foo[\"`, \"[ is missing closing ]\")\n\ttest(`.foo[\"\\`, \"[ is missing closing ]\")\n\ttest(`.foo[\"]`, \"[ is missing closing ]\")\n\ttest(\".foo[#]\", \"Invalid hash: \")\n\ttest(\".foo[#invalid]\", \"Invalid hash: invalid\")\n\ttest(`.foo[\"hello\\nworld\"]`, `Only \" and \\ can be escaped`)\n\ttest(\".foo[42]bar\", \"Invalid operator: b\")\n\ttest(\"#foo\", \"Invalid operator: #\")\n\ttest(\"!foo\", \"Invalid operator: !\")\n\ttest(\"@foo\", \"Unsupported annotation: @foo\")\n\ttest(\"@key\", \"Cannot use @key annotation at beginning of path\")\n\ttest(\".foo@key\", \"Cannot use @key annotation on: .foo\")\n\ttest(\".foo@key()\", \"@key annotation does not support arguments\")\n\ttest(\".foo@key(42)\", \"@key annotation does not support arguments\")\n\ttest(\".foo@type()\", \"@type annotation does not support arguments\")\n\ttest(\".foo@type(42)\", \"@type annotation does not support arguments\")\n\ttest(\".foo@at\", \"@at annotation requires a position argument\")\n\ttest(\".foo@at()\", \"@at annotation requires a position argument\")\n\ttest(\".foo@at(\", \"@at annotation requires a position argument\")\n\ttest(\".foo@at(42\", \"@at annotation requires a position argument\")\n\ttest(fmt.Sprintf(\".foo[#%s]@soup\", hash.Of([]byte{42}).String()), \"Unsupported annotation: @soup\")\n}\n\nfunc TestPathEquals(t *testing.T) {\n\tassert := assert.New(t)\n\tequalPaths := []string{\n\t\t`[1]`,\n\t\t`[\"one\"]`,\n\t\t`.two.three`,\n\t\t`[\"yo\"]@key`,\n\t}\n\tnotEqualPaths := [][]string{\n\t\t{`[1]`, `[2]`},\n\t\t{`[\"one\"]`, `[\"two\"]`},\n\t\t{`.two.three`, `.two.four`},\n\t\t{`[\"yo\"]@key`, `[\"yo\"]`},\n\t}\n\n\tassert.True(Path{}.Equals(Path{}))\n\tfor _, s := range equalPaths {\n\t\tp, err := ParsePath(s)\n\t\tassert.NoError(err)\n\t\tassert.True(p.Equals(p))\n\t}\n\n\tsimple, err := ParsePath(`[\"one\"].two`)\n\tassert.NoError(err)\n\tassert.False(Path{}.Equals(simple))\n\tfor _, a := range notEqualPaths {\n\t\ts0, s1 := a[0], a[1]\n\t\tp0, err := ParsePath(s0)\n\t\tassert.NoError(err)\n\t\tp1, err := ParsePath(s1)\n\t\tassert.NoError(err)\n\t\tassert.False(p0.Equals(p1))\n\t}\n}\n\nfunc TestPathCanBePathIndex(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tassert.True(ValueCanBePathIndex(Bool(true)))\n\tassert.True(ValueCanBePathIndex(Number(5)))\n\tassert.True(ValueCanBePathIndex(String(\"yes\")))\n\n\tassert.False(ValueCanBePathIndex(NewRef(String(\"yes\"))))\n\tassert.False(ValueCanBePathIndex(NewBlob(vs, bytes.NewReader([]byte(\"yes\")))))\n}\n\nfunc TestCopyPath(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttestCases := []string{\n\t\t``,\n\t\t`[\"key\"]`,\n\t\t`[\"key\"].field1`,\n\t\t`[\"key\"]@key.field1`,\n\t}\n\n\tfor _, s1 := range testCases {\n\t\texpected, err := ParsePath(s1 + `[\"anIndex\"]`)\n\t\tassert.NoError(err)\n\t\tvar p Path\n\t\tif s1 != \"\" {\n\t\t\tp, err = ParsePath(s1)\n\t\t}\n\t\tassert.NoError(err)\n\t\tp1 := p.Append(NewIndexPath(String(\"anIndex\")))\n\t\tif len(p) > 0 {\n\t\t\tp[0] = expected[1] // if p1 really is a copy, this shouldn't be noticed\n\t\t}\n\t\tassert.Equal(expected, p1)\n\t}\n}\n\nfunc TestMustParsePath(t *testing.T) {\n\tfor _, good := range []string{\".good\", \"[\\\"good\\\"]\"} {\n\t\tassert.NotNil(t, MustParsePath(good))\n\t}\n\tfor _, bad := range []string{\"\", \"bad\", \"[bad]\", \"!\", \"💩\"} {\n\t\tassert.Panics(t, func() { MustParsePath(bad) })\n\t}\n}\n\nfunc TestPathType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tm := NewMap(vs,\n\t\tString(\"string\"), String(\"foo\"),\n\t\tString(\"bool\"), Bool(false),\n\t\tString(\"number\"), Number(42),\n\t\tString(\"List\"), NewList(vs, Number(42), String(\"foo\")),\n\t\tString(\"Map\"), NewMap(vs, Bool(true), Bool(false)))\n\n\tm.IterAll(func(k, cv Value) {\n\t\tks := k.(String)\n\t\tassertResolvesTo(assert, TypeOf(cv), m, fmt.Sprintf(\"[\\\"%s\\\"]@type\", ks))\n\t})\n\n\tassertResolvesTo(assert, StringType, m, `[\"string\"]@key@type`)\n\tassertResolvesTo(assert, TypeOf(m), m, `@type`)\n\ts := NewStruct(\"\", StructData{\n\t\t\"str\": String(\"foo\"),\n\t\t\"num\": Number(42),\n\t})\n\tassertResolvesTo(assert, TypeOf(s.Get(\"str\")), s, \".str@type\")\n\tassertResolvesTo(assert, TypeOf(s.Get(\"num\")), s, \".num@type\")\n}\n\nfunc TestPathTarget(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts := NewStruct(\"\", StructData{\n\t\t\"foo\": String(\"bar\"),\n\t})\n\tvs := newTestValueStore()\n\tr := vs.WriteValue(s)\n\ts2 := NewStruct(\"\", StructData{\n\t\t\"ref\": r,\n\t})\n\n\tassertResolvesToWithVR(assert, nil, String(\"notref\"), `@target`, vs)\n\tassertResolvesToWithVR(assert, s, r, `@target`, vs)\n\tassertResolvesToWithVR(assert, String(\"bar\"), r, `@target.foo`, vs)\n\tassertResolvesToWithVR(assert, String(\"bar\"), s2, `.ref@target.foo`, vs)\n}\n\nfunc TestPathAtAnnotation(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tvar v Value\n\tresolvesTo := func(expVal, expKey Value, str string) {\n\t\tassertResolvesTo(assert, expVal, v, str)\n\t\tassertResolvesTo(assert, expKey, v, str+\"@key\")\n\t}\n\n\tv = NewList(vs, Number(1), Number(3), String(\"foo\"), Bool(false))\n\n\tresolvesTo(Number(1), nil, \"@at(0)\")\n\tresolvesTo(Number(3), nil, \"@at(1)\")\n\tresolvesTo(String(\"foo\"), nil, \"@at(2)\")\n\tresolvesTo(Bool(false), nil, \"@at(3)\")\n\tresolvesTo(nil, nil, \"@at(4)\")\n\tresolvesTo(nil, nil, \"@at(-5)\")\n\tresolvesTo(Number(1), nil, \"@at(-4)\")\n\tresolvesTo(Number(3), nil, \"@at(-3)\")\n\tresolvesTo(String(\"foo\"), nil, \"@at(-2)\")\n\tresolvesTo(Bool(false), nil, \"@at(-1)\")\n\n\tv = NewSet(vs,\n\t\tBool(false),\n\t\tNumber(1),\n\t\tNumber(2.3),\n\t\tString(\"two\"),\n\t)\n\n\tresolvesTo(Bool(false), Bool(false), \"@at(0)\")\n\tresolvesTo(Number(1), Number(1), \"@at(1)\")\n\tresolvesTo(Number(2.3), Number(2.3), \"@at(2)\")\n\tresolvesTo(String(\"two\"), String(\"two\"), `@at(3)`)\n\tresolvesTo(nil, nil, \"@at(4)\")\n\tresolvesTo(nil, nil, \"@at(-5)\")\n\tresolvesTo(Bool(false), Bool(false), \"@at(-4)\")\n\tresolvesTo(Number(1), Number(1), \"@at(-3)\")\n\tresolvesTo(Number(2.3), Number(2.3), \"@at(-2)\")\n\tresolvesTo(String(\"two\"), String(\"two\"), `@at(-1)`)\n\n\tv = NewMap(vs,\n\t\tBool(false), Number(23),\n\t\tNumber(1), String(\"foo\"),\n\t\tNumber(2.3), Number(4.5),\n\t\tString(\"two\"), String(\"bar\"),\n\t)\n\n\tresolvesTo(Number(23), Bool(false), \"@at(0)\")\n\tresolvesTo(String(\"foo\"), Number(1), \"@at(1)\")\n\tresolvesTo(Number(4.5), Number(2.3), \"@at(2)\")\n\tresolvesTo(String(\"bar\"), String(\"two\"), `@at(3)`)\n\tresolvesTo(nil, nil, \"@at(4)\")\n\tresolvesTo(nil, nil, \"@at(-5)\")\n\tresolvesTo(Number(23), Bool(false), \"@at(-4)\")\n\tresolvesTo(String(\"foo\"), Number(1), \"@at(-3)\")\n\tresolvesTo(Number(4.5), Number(2.3), \"@at(-2)\")\n\tresolvesTo(String(\"bar\"), String(\"two\"), `@at(-1)`)\n}\n" }, { "path": "go/types/perf/dummy.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage perf\n\n// go build fails if there are _test.go but no other go files in a directory.\n" }, { "path": "go/types/perf/perf_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage perf\n\nimport (\n\t\"io\"\n\t\"io/ioutil\"\n\t\"math/rand\"\n\t\"os\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/perf/suite\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype perfSuite struct {\n\tsuite.PerfSuite\n\tr *rand.Rand\n\tds string\n}\n\nfunc (s *perfSuite) SetupSuite() {\n\ts.r = rand.New(rand.NewSource(0))\n}\n\nfunc (s *perfSuite) Test01BuildList10mNumbers() {\n\tassert := s.NewAssert()\n\tin := make(chan types.Value, 16)\n\tout := types.NewStreamingList(s.Database, in)\n\n\tfor i := 0; i < 1e7; i++ {\n\t\tin <- types.Number(s.r.Int63())\n\t}\n\tclose(in)\n\n\tds := s.Database.GetDataset(\"BuildList10mNumbers\")\n\n\tvar err error\n\tds, err = s.Database.CommitValue(ds, <-out)\n\n\tassert.NoError(err)\n\ts.Database = ds.Database()\n}\n\nfunc (s *perfSuite) Test02BuildList10mStructs() {\n\tassert := s.NewAssert()\n\tin := make(chan types.Value, 16)\n\tout := types.NewStreamingList(s.Database, in)\n\n\tfor i := 0; i < 1e7; i++ {\n\t\tin <- types.NewStruct(\"\", types.StructData{\n\t\t\t\"number\": types.Number(s.r.Int63()),\n\t\t})\n\t}\n\tclose(in)\n\n\tds := s.Database.GetDataset(\"BuildList10mStructs\")\n\n\tvar err error\n\tds, err = s.Database.CommitValue(ds, <-out)\n\n\tassert.NoError(err)\n\ts.Database = ds.Database()\n}\n\nfunc (s *perfSuite) Test03Read10mNumbers() {\n\ts.headList(\"BuildList10mNumbers\").IterAll(func(v types.Value, index uint64) {})\n}\n\nfunc (s *perfSuite) Test04Read10mStructs() {\n\ts.headList(\"BuildList10mStructs\").IterAll(func(v types.Value, index uint64) {})\n}\n\nfunc (s *perfSuite) Test05Concat10mValues2kTimes() {\n\tassert := s.NewAssert()\n\n\tlast := func(v types.List) types.Value {\n\t\treturn v.Get(v.Len() - 1)\n\t}\n\n\tl1 := s.headList(\"BuildList10mNumbers\")\n\tl2 := s.headList(\"BuildList10mStructs\")\n\tl1Len, l2Len := l1.Len(), l2.Len()\n\tl1Last, l2Last := last(l1), last(l2)\n\n\tl3 := types.NewList(s.Database)\n\tfor i := uint64(0); i < 1e3; i++ { // 1k iterations * 2 concat ops = 2k times\n\t\t// Include some basic sanity checks.\n\t\tl3 = l3.Concat(l1)\n\t\tassert.True(l1Last.Equals(last(l3)))\n\t\tassert.Equal(i*(l1Len+l2Len)+l1Len, l3.Len())\n\t\tl3 = l3.Concat(l2)\n\t\tassert.True(l2Last.Equals(last(l3)))\n\t\tassert.Equal((i+1)*(l1Len+l2Len), l3.Len())\n\t}\n\n\tds := s.Database.GetDataset(\"Concat10mValues2kTimes\")\n\tvar err error\n\tds, err = s.Database.CommitValue(ds, l3)\n\n\tassert.NoError(err)\n\ts.Database = ds.Database()\n}\n\nfunc (s *perfSuite) TestBuild500megBlobFromFilesP1() {\n\ts.testBuild500megBlob(1)\n}\n\nfunc (s *perfSuite) TestBuild500megBlobFromFilesP2() {\n\ts.testBuild500megBlob(2)\n}\n\nfunc (s *perfSuite) TestBuild500megBlobFromFilesP8() {\n\ts.testBuild500megBlob(8)\n}\n\nfunc (s *perfSuite) TestBuild500megBlobFromFilesP64() {\n\t// Note: can't have too many files open.\n\ts.testBuild500megBlob(64)\n}\n\nfunc (s *perfSuite) testBuild500megBlob(p int) {\n\tassert := s.NewAssert()\n\tsize := int(5e8)\n\n\treaders := make([]io.Reader, p)\n\tdefer func() {\n\t\tfor _, r := range readers {\n\t\t\tf := r.(*os.File)\n\t\t\terr := f.Close()\n\t\t\tassert.NoError(err)\n\t\t\terr = os.Remove(f.Name())\n\t\t\tassert.NoError(err)\n\t\t}\n\t}()\n\n\ts.Pause(func() {\n\t\tfor i := range readers {\n\t\t\tf, err := ioutil.TempFile(\"\", \"testBuildBlob\")\n\t\t\tassert.NoError(err)\n\t\t\t_, err = f.Write(s.randomBytes(int64(i), size/p))\n\t\t\tassert.NoError(err)\n\t\t\terr = f.Close()\n\t\t\tassert.NoError(err)\n\t\t\tf, err = os.Open(f.Name())\n\t\t\tassert.NoError(err)\n\t\t\treaders[i] = f\n\t\t}\n\t})\n\n\tb := types.NewBlob(s.Database, readers...)\n\tassert.Equal(uint64(size), b.Len())\n}\n\nfunc (s *perfSuite) randomBytes(seed int64, size int) []byte {\n\tr := rand.New(rand.NewSource(seed))\n\trandBytes := make([]byte, size)\n\t_, err := r.Read(randBytes)\n\tassert.NoError(s.T, err)\n\treturn randBytes\n}\n\nfunc (s *perfSuite) headList(dsName string) types.List {\n\tds := s.Database.GetDataset(dsName)\n\treturn ds.HeadValue().(types.List)\n}\n\nfunc TestPerf(t *testing.T) {\n\tsuite.Run(\"types\", t, &perfSuite{})\n}\n" }, { "path": "go/types/primitives_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestPrimitives(t *testing.T) {\n\tdata := []Value{\n\t\tBool(true), Bool(false),\n\t\tNumber(0), Number(-1),\n\t\tNumber(-0.1), Number(0.1),\n\t}\n\n\tfor i := range data {\n\t\tfor j := range data {\n\t\t\tif i == j {\n\t\t\t\tassert.True(t, data[i].Equals(data[j]), \"Expected value to equal self at index %d\", i)\n\t\t\t} else {\n\t\t\t\tassert.False(t, data[i].Equals(data[j]), \"Expected values at indices %d and %d to not equal\", i, j)\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc TestPrimitivesType(t *testing.T) {\n\tdata := []struct {\n\t\tv Value\n\t\tk NomsKind\n\t}{\n\t\t{Bool(false), BoolKind},\n\t\t{Number(0), NumberKind},\n\t}\n\n\tfor _, d := range data {\n\t\tassert.True(t, TypeOf(d.v).Equals(MakePrimitiveType(d.k)))\n\t}\n}\n" }, { "path": "go/types/ref.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype Ref struct {\n\tvalueImpl\n}\n\ntype refPart uint32\n\nconst (\n\trefPartKind refPart = iota\n\trefPartTargetHash\n\trefPartTargetType\n\trefPartHeight\n\trefPartEnd\n)\n\nfunc NewRef(v Value) Ref {\n\treturn constructRef(v.Hash(), TypeOf(v), maxChunkHeight(v)+1)\n}\n\n// ToRefOfValue returns a new Ref that points to the same target as |r|, but\n// with the type 'Ref'.\nfunc ToRefOfValue(r Ref) Ref {\n\treturn constructRef(r.TargetHash(), ValueType, r.Height())\n}\n\nfunc constructRef(targetHash hash.Hash, targetType *Type, height uint64) Ref {\n\tw := newBinaryNomsWriter()\n\n\toffsets := make([]uint32, refPartEnd)\n\toffsets[refPartKind] = w.offset\n\tRefKind.writeTo(&w)\n\toffsets[refPartTargetHash] = w.offset\n\tw.writeHash(targetHash)\n\toffsets[refPartTargetType] = w.offset\n\ttargetType.writeToAsType(&w, map[string]*Type{})\n\toffsets[refPartHeight] = w.offset\n\tw.writeCount(height)\n\n\treturn Ref{valueImpl{nil, w.data(), offsets}}\n}\n\nfunc writeRefPartsTo(w nomsWriter, targetHash hash.Hash, targetType *Type, height uint64) {\n\tRefKind.writeTo(w)\n\tw.writeHash(targetHash)\n\ttargetType.writeToAsType(w, map[string]*Type{})\n\tw.writeCount(height)\n}\n\n// readRef reads the data provided by a reader and moves the reader forward.\nfunc readRef(dec *typedBinaryNomsReader) Ref {\n\tstart := dec.pos()\n\toffsets := skipRef(dec)\n\tend := dec.pos()\n\treturn Ref{valueImpl{nil, dec.byteSlice(start, end), offsets}}\n}\n\n// skipRef moves the reader forward, past the data representing the Ref, and returns the offsets of the component parts.\nfunc skipRef(dec *typedBinaryNomsReader) []uint32 {\n\toffsets := make([]uint32, refPartEnd)\n\toffsets[refPartKind] = dec.pos()\n\tdec.skipKind()\n\toffsets[refPartTargetHash] = dec.pos()\n\tdec.skipHash() // targetHash\n\toffsets[refPartTargetType] = dec.pos()\n\tdec.skipType() // targetType\n\toffsets[refPartHeight] = dec.pos()\n\tdec.skipCount() // height\n\treturn offsets\n}\n\nfunc maxChunkHeight(v Value) (max uint64) {\n\tv.WalkRefs(func(r Ref) {\n\t\tif height := r.Height(); height > max {\n\t\t\tmax = height\n\t\t}\n\t})\n\treturn\n}\n\nfunc (r Ref) offsetAtPart(part refPart) uint32 {\n\treturn r.offsets[part] - r.offsets[refPartKind]\n}\n\nfunc (r Ref) decoderAtPart(part refPart) valueDecoder {\n\toffset := r.offsetAtPart(part)\n\treturn newValueDecoder(r.buff[offset:], nil)\n}\n\nfunc (r Ref) TargetHash() hash.Hash {\n\tdec := r.decoderAtPart(refPartTargetHash)\n\treturn dec.readHash()\n}\n\nfunc (r Ref) Height() uint64 {\n\tdec := r.decoderAtPart(refPartHeight)\n\treturn dec.readCount()\n}\n\nfunc (r Ref) TargetValue(vr ValueReader) Value {\n\treturn vr.ReadValue(r.TargetHash())\n}\n\nfunc (r Ref) TargetType() *Type {\n\tdec := r.decoderAtPart(refPartTargetType)\n\treturn dec.readType()\n}\n\n// Value interface\nfunc (r Ref) Value() Value {\n\treturn r\n}\n\nfunc (r Ref) WalkValues(cb ValueCallback) {\n}\n\nfunc (r Ref) typeOf() *Type {\n\treturn makeCompoundType(RefKind, r.TargetType())\n}\n\nfunc (r Ref) isSameTargetType(other Ref) bool {\n\ttargetTypeBytes := r.buff[r.offsetAtPart(refPartTargetType):r.offsetAtPart(refPartHeight)]\n\totherTargetTypeBytes := other.buff[other.offsetAtPart(refPartTargetType):other.offsetAtPart(refPartHeight)]\n\treturn bytes.Equal(targetTypeBytes, otherTargetTypeBytes)\n}\n" }, { "path": "go/types/ref_heap.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n// RefByHeight implements sort.Interface to order by increasing HeightOrder(). It uses increasing order because this causes repeated pushes and pops of the 'tallest' Refs to re-use memory, avoiding reallocations.\n// We might consider making this a firmer abstraction boundary as a part of BUG 2182\ntype RefByHeight []Ref\n\nfunc (h RefByHeight) Len() int {\n\treturn len(h)\n}\n\nfunc (h RefByHeight) Less(i, j int) bool {\n\treturn !HeightOrder(h[i], h[j])\n}\n\nfunc (h RefByHeight) Swap(i, j int) {\n\th[i], h[j] = h[j], h[i]\n}\n\nfunc (h *RefByHeight) PushBack(r Ref) {\n\t*h = append(*h, r)\n}\n\nfunc (h *RefByHeight) PopBack() Ref {\n\told := *h\n\tn := len(old)\n\tx := old[n-1]\n\t*h = old[0 : n-1]\n\treturn x\n}\n\n// DropIndices takes a slice of integer indices into h and splices out the Refs at those indices.\nfunc (h *RefByHeight) DropIndices(indices []int) {\n\tsort.Ints(indices)\n\told := *h\n\tnumIdx := len(indices)\n\tfor i, j := 0, 0; i < old.Len(); i++ {\n\t\tif len(indices) > 0 && i == indices[0] {\n\t\t\tindices = indices[1:]\n\t\t\tcontinue\n\t\t}\n\t\tif i != j {\n\t\t\told[j] = old[i]\n\t\t}\n\t\tj++\n\t}\n\t*h = old[:old.Len()-numIdx]\n}\n\nfunc (h *RefByHeight) Unique() {\n\tseen := hash.HashSet{}\n\tresult := make(RefByHeight, 0, cap(*h))\n\tfor _, r := range *h {\n\t\ttarget := r.TargetHash()\n\t\tif !seen.Has(target) {\n\t\t\tresult = append(result, r)\n\t\t}\n\t\tseen.Insert(target)\n\t}\n\t*h = result\n}\n\n// PopRefsOfHeight pops off and returns all refs r in h for which r.Height() == height.\nfunc (h *RefByHeight) PopRefsOfHeight(height uint64) (refs RefSlice) {\n\tfor h.MaxHeight() == height {\n\t\tr := h.PopBack()\n\t\trefs = append(refs, r)\n\t}\n\treturn\n}\n\n// MaxHeight returns the height of the 'tallest' Ref in h.\nfunc (h RefByHeight) MaxHeight() uint64 {\n\tif h.Empty() {\n\t\treturn 0\n\t}\n\treturn h.PeekEnd().Height()\n}\n\nfunc (h RefByHeight) Empty() bool {\n\treturn h.Len() == 0\n}\n\n// PeekEnd returns, but does not Pop the tallest Ref in h.\nfunc (h RefByHeight) PeekEnd() (head Ref) {\n\treturn h.PeekAt(h.Len() - 1)\n}\n\n// PeekAt returns, but does not remove, the Ref at h[idx]. If the index is out of range, returns the empty Ref.\nfunc (h RefByHeight) PeekAt(idx int) (peek Ref) {\n\tif idx >= 0 && idx < h.Len() {\n\t\tpeek = h[idx]\n\t}\n\treturn\n}\n\n// HeightOrder returns true if a is 'higher than' b, generally if its ref-height is greater. If the two are of the same height, fall back to sorting by TargetHash.\nfunc HeightOrder(a, b Ref) bool {\n\tif a.Height() == b.Height() {\n\t\treturn a.TargetHash().Less(b.TargetHash())\n\t}\n\t// > because we want the larger heights to be at the start of the queue.\n\treturn a.Height() > b.Height()\n\n}\n\n// RefSlice implements sort.Interface to order by target ref.\ntype RefSlice []Ref\n\nfunc (s RefSlice) Len() int {\n\treturn len(s)\n}\n\nfunc (s RefSlice) Less(i, j int) bool {\n\treturn s[i].TargetHash().Less(s[j].TargetHash())\n}\n\nfunc (s RefSlice) Swap(i, j int) {\n\ts[i], s[j] = s[j], s[i]\n}\n" }, { "path": "go/types/ref_heap_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestRefByHeight(t *testing.T) {\n\tunique := 0\n\tnewRefWithHeight := func(height uint64) Ref {\n\t\tv := Number(unique)\n\t\tunique++\n\t\treturn constructRef(v.Hash(), NumberType, height)\n\t}\n\n\tassert := assert.New(t)\n\n\th := RefByHeight{}\n\n\tr1 := newRefWithHeight(1)\n\tr2 := newRefWithHeight(2)\n\tr3 := newRefWithHeight(3)\n\tr4 := newRefWithHeight(2)\n\n\th.PushBack(r1)\n\tassert.Equal(r1, h.PeekEnd())\n\tassert.Equal(1, len(h))\n\n\th.PushBack(r3)\n\tsort.Sort(&h)\n\tassert.Equal(r3, h.PeekEnd())\n\tassert.Equal(2, len(h))\n\n\th.PushBack(r2)\n\tsort.Sort(&h)\n\tassert.Equal(r3, h.PeekEnd())\n\tassert.Equal(3, len(h))\n\n\th.PushBack(r4)\n\tsort.Sort(&h)\n\tassert.Equal(r3, h.PeekEnd())\n\tassert.Equal(4, len(h))\n\n\texpectedSecond, expectedThird := func() (Ref, Ref) {\n\t\tif r2.TargetHash().Less(r4.TargetHash()) {\n\t\t\treturn r2, r4\n\t\t}\n\t\treturn r4, r2\n\t}()\n\n\tassert.Equal(r3, h.PopBack())\n\tassert.Equal(expectedSecond, h.PeekEnd())\n\tassert.Equal(3, len(h))\n\n\tassert.Equal(expectedSecond, h.PopBack())\n\tassert.Equal(expectedThird, h.PeekEnd())\n\tassert.Equal(2, len(h))\n\n\tassert.Equal(expectedThird, h.PopBack())\n\tassert.Equal(r1, h.PeekEnd())\n\tassert.Equal(1, len(h))\n\n\tassert.Equal(r1, h.PopBack())\n\tassert.Equal(0, len(h))\n}\n\nfunc TestDropIndices(t *testing.T) {\n\th := &RefByHeight{}\n\tfor i := 0; i < 10; i++ {\n\t\th.PushBack(NewRef(Number(i)))\n\t}\n\tsort.Sort(h)\n\n\ttoDrop := []int{2, 4, 7}\n\texpected := RefSlice{h.PeekAt(2), h.PeekAt(4), h.PeekAt(7)}\n\th.DropIndices(toDrop)\n\tassert.Len(t, *h, 7)\n\tfor i, dropped := range expected {\n\t\tassert.NotContains(t, *h, dropped, \"Should not contain %d\", toDrop[i])\n\t}\n}\n\nfunc TestPopRefsOfHeight(t *testing.T) {\n\th := &RefByHeight{}\n\tfor i, n := range []int{6, 3, 6, 6, 2} {\n\t\tr := constructRef(Number(i).Hash(), NumberType, uint64(n))\n\t\th.PushBack(r)\n\t}\n\tsort.Sort(h)\n\n\texpected := RefSlice{h.PeekAt(4), h.PeekAt(3), h.PeekAt(2)}\n\trefs := h.PopRefsOfHeight(h.MaxHeight())\n\tassert.Len(t, *h, 2)\n\tassert.Len(t, refs, 3)\n\tfor _, popped := range expected {\n\t\tassert.NotContains(t, *h, popped, \"Should not contain ref of height 6\")\n\t}\n}\n" }, { "path": "go/types/ref_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestRefInList(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tl := NewList(vs)\n\tr := NewRef(l)\n\tl = l.Edit().Append(r).List()\n\tr2 := l.Get(0)\n\tassert.True(r.Equals(r2))\n}\n\nfunc TestRefInSet(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs)\n\tr := NewRef(s)\n\ts = s.Edit().Insert(r).Set()\n\tr2 := s.First()\n\tassert.True(r.Equals(r2))\n}\n\nfunc TestRefInMap(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tm := NewMap(vs)\n\tr := NewRef(m)\n\tm = m.Edit().Set(Number(0), r).Set(r, Number(1)).Map()\n\tr2 := m.Get(Number(0))\n\tassert.True(r.Equals(r2))\n\n\ti := m.Get(r)\n\tassert.Equal(int32(1), int32(i.(Number)))\n}\n\nfunc TestRefChunks(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tl := NewList(vs)\n\tr := NewRef(l)\n\tassert.Len(getChunks(r), 1)\n\tassert.Equal(r, getChunks(r)[0])\n}\n" }, { "path": "go/types/rolling_value_hasher.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/sloppy\"\n\n\t\"github.com/kch42/buzhash\"\n)\n\nconst (\n\tdefaultChunkPattern = uint32(1<<12 - 1) // Avg Chunk Size of 4k\n\n\t// The window size to use for computing the rolling hash. This is way more than necessary assuming random data (two bytes would be sufficient with a target chunk size of 4k). The benefit of a larger window is it allows for better distribution on input with lower entropy. At a target chunk size of 4k, any given byte changing has roughly a 1.5% chance of affecting an existing boundary, which seems like an acceptable trade-off. The choice of a prime number provides better distribution for repeating input.\n\tchunkWindow = uint32(67)\n\tmaxChunkSize = 1 << 24 // TODO: Remove when https://github.com/attic-labs/noms/issues/3743 is fixed.\n)\n\n// Only set by tests\nvar (\n\tchunkPattern = defaultChunkPattern\n\tchunkConfigMu = &sync.Mutex{}\n)\n\nfunc chunkingConfig() (pattern, window uint32) {\n\tchunkConfigMu.Lock()\n\tdefer chunkConfigMu.Unlock()\n\treturn chunkPattern, chunkWindow\n}\n\nfunc smallTestChunks() {\n\tchunkConfigMu.Lock()\n\tdefer chunkConfigMu.Unlock()\n\tchunkPattern = uint32(1<<8 - 1) // Avg Chunk Size of 256 bytes\n}\n\nfunc normalProductionChunks() {\n\tchunkConfigMu.Lock()\n\tdefer chunkConfigMu.Unlock()\n\tchunkPattern = defaultChunkPattern\n}\n\ntype rollingValueHasher struct {\n\tbw binaryNomsWriter\n\tbz *buzhash.BuzHash\n\tcrossedBoundary bool\n\tpattern, window uint32\n\tsalt byte\n\tsl *sloppy.Sloppy\n}\n\nfunc hashValueBytes(item sequenceItem, rv *rollingValueHasher) {\n\trv.HashValue(item.(Value))\n}\n\nfunc hashValueByte(item sequenceItem, rv *rollingValueHasher) {\n\trv.HashByte(item.(byte))\n}\n\nfunc newRollingValueHasher(salt byte) *rollingValueHasher {\n\tpattern, window := chunkingConfig()\n\tw := newBinaryNomsWriter()\n\n\trv := &rollingValueHasher{\n\t\tbw: w,\n\t\tbz: buzhash.NewBuzHash(window),\n\t\tpattern: pattern,\n\t\twindow: window,\n\t\tsalt: salt,\n\t}\n\n\trv.sl = sloppy.New(rv.HashByte)\n\n\treturn rv\n}\n\nfunc (rv *rollingValueHasher) HashByte(b byte) bool {\n\tif !rv.crossedBoundary {\n\t\trv.bz.HashByte(b ^ rv.salt)\n\t\trv.crossedBoundary = (rv.bz.Sum32()&rv.pattern == rv.pattern)\n\t\tif rv.bw.offset > maxChunkSize {\n\t\t\trv.crossedBoundary = true\n\t\t}\n\t}\n\treturn rv.crossedBoundary\n}\n\nfunc (rv *rollingValueHasher) Reset() {\n\trv.crossedBoundary = false\n\trv.bz = buzhash.NewBuzHash(rv.window)\n\trv.bw.reset()\n\trv.sl.Reset()\n}\n\nfunc (rv *rollingValueHasher) HashValue(v Value) {\n\tv.writeTo(&rv.bw)\n\trv.sl.Update(rv.bw.data())\n}\n\nfunc (rv *rollingValueHasher) hashBytes(buff []byte) {\n\trv.bw.writeRaw(buff)\n\trv.sl.Update(rv.bw.data())\n}\n" }, { "path": "go/types/rungen.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\n//go:generate go run gen/main.go\n" }, { "path": "go/types/sequence.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype sequenceItem interface{}\n\ntype compareFn func(x int, y int) bool\n\ntype sequence interface {\n\tasValueImpl() valueImpl\n\tcumulativeNumberOfLeaves(idx int) uint64\n\tEmpty() bool\n\tEquals(other Value) bool\n\tgetChildSequence(idx int) sequence\n\tgetCompareFn(other sequence) compareFn\n\tgetCompositeChildSequence(start uint64, length uint64) sequence\n\tgetItem(idx int) sequenceItem\n\tHash() hash.Hash\n\tisLeaf() bool\n\tKind() NomsKind\n\tLen() uint64\n\tLess(other Value) bool\n\tnumLeaves() uint64\n\tseqLen() int\n\ttreeLevel() uint64\n\ttypeOf() *Type\n\tvalueBytes() []byte\n\tvalueReadWriter() ValueReadWriter\n\tvaluesSlice(from, to uint64) []Value\n\tWalkRefs(cb RefCallback)\n\twriteTo(nomsWriter)\n}\n\nconst (\n\tsequencePartKind = 0\n\tsequencePartLevel = 1\n\tsequencePartCount = 2\n\tsequencePartValues = 3\n)\n\ntype sequenceImpl struct {\n\tvalueImpl\n\tlen uint64\n}\n\nfunc newSequenceImpl(vrw ValueReadWriter, buff []byte, offsets []uint32, len uint64) sequenceImpl {\n\treturn sequenceImpl{valueImpl{vrw, buff, offsets}, len}\n}\n\nfunc (seq sequenceImpl) decoderSkipToValues() (valueDecoder, uint64) {\n\tdec := seq.decoderAtPart(sequencePartCount)\n\tcount := dec.readCount()\n\treturn dec, count\n}\n\nfunc (seq sequenceImpl) decoderAtPart(part uint32) valueDecoder {\n\toffset := seq.offsets[part] - seq.offsets[sequencePartKind]\n\treturn newValueDecoder(seq.buff[offset:], seq.vrw)\n}\n\nfunc (seq sequenceImpl) Empty() bool {\n\treturn seq.Len() == 0\n}\n\nfunc (seq sequenceImpl) Len() uint64 {\n\treturn seq.len\n}\n\nfunc (seq sequenceImpl) seqLen() int {\n\t_, count := seq.decoderSkipToValues()\n\treturn int(count)\n}\n\nfunc (seq sequenceImpl) getItemOffset(idx int) int {\n\t// kind, level, count, elements...\n\t// 0 1 2 3 n+1\n\td.PanicIfTrue(idx+sequencePartValues+1 > len(seq.offsets))\n\treturn int(seq.offsets[idx+sequencePartValues] - seq.offsets[sequencePartKind])\n}\n\nfunc (seq sequenceImpl) decoderSkipToIndex(idx int) valueDecoder {\n\toffset := seq.getItemOffset(idx)\n\treturn seq.decoderAtOffset(offset)\n}\n" }, { "path": "go/types/sequence_chunker.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport \"github.com/attic-labs/noms/go/d\"\n\ntype hashValueBytesFn func(item sequenceItem, rv *rollingValueHasher)\n\ntype sequenceChunker struct {\n\tcur *sequenceCursor\n\tlevel uint64\n\tvrw ValueReadWriter\n\tparent *sequenceChunker\n\tcurrent []sequenceItem\n\tmakeChunk, parentMakeChunk makeChunkFn\n\tisLeaf bool\n\thashValueBytes hashValueBytesFn\n\trv *rollingValueHasher\n\tdone bool\n\tunwrittenCol Collection\n}\n\n// makeChunkFn takes a sequence of items to chunk, and returns the result of chunking those items, a tuple of a reference to that chunk which can itself be chunked + its underlying value.\ntype makeChunkFn func(level uint64, values []sequenceItem) (Collection, orderedKey, uint64)\n\nfunc newEmptySequenceChunker(vrw ValueReadWriter, makeChunk, parentMakeChunk makeChunkFn, hashValueBytes hashValueBytesFn) *sequenceChunker {\n\treturn newSequenceChunker(nil, uint64(0), vrw, makeChunk, parentMakeChunk, hashValueBytes)\n}\n\nfunc newSequenceChunker(cur *sequenceCursor, level uint64, vrw ValueReadWriter, makeChunk, parentMakeChunk makeChunkFn, hashValueBytes hashValueBytesFn) *sequenceChunker {\n\td.PanicIfFalse(makeChunk != nil)\n\td.PanicIfFalse(parentMakeChunk != nil)\n\td.PanicIfFalse(hashValueBytes != nil)\n\td.PanicIfTrue(vrw == nil)\n\n\t// |cur| will be nil if this is a new sequence, implying this is a new tree, or the tree has grown in height relative to its original chunked form.\n\n\tsc := &sequenceChunker{\n\t\tcur,\n\t\tlevel,\n\t\tvrw,\n\t\tnil,\n\t\tmake([]sequenceItem, 0, 1<<10),\n\t\tmakeChunk, parentMakeChunk,\n\t\ttrue,\n\t\thashValueBytes,\n\t\tnewRollingValueHasher(byte(level % 256)),\n\t\tfalse,\n\t\tnil,\n\t}\n\n\tif cur != nil {\n\t\tsc.resume()\n\t}\n\n\treturn sc\n}\n\nfunc (sc *sequenceChunker) resume() {\n\tif sc.cur.parent != nil && sc.parent == nil {\n\t\tsc.createParent()\n\t}\n\n\tidx := sc.cur.idx\n\n\t// Walk backwards to the start of the existing chunk.\n\tfor sc.cur.indexInChunk() > 0 && sc.cur.retreatMaybeAllowBeforeStart(false) {\n\t}\n\n\tfor ; sc.cur.idx < idx; sc.cur.advance() {\n\t\tsc.Append(sc.cur.current())\n\t}\n}\n\n// advanceTo advances the sequenceChunker to the next \"spine\" at which\n// modifications to the prolly-tree should take place\nfunc (sc *sequenceChunker) advanceTo(next *sequenceCursor) {\n\t// There are four basic situations which must be handled when advancing to a\n\t// new chunking position:\n\t//\n\t// Case (1): |sc.cur| and |next| are exactly aligned. In this case, there's\n\t// nothing to do. Just assign sc.cur = next.\n\t//\n\t// Case (2): |sc.cur| is \"ahead\" of |next|. This can only have resulted from\n\t// advancing of a lower level causing |sc.cur| to advance. In this\n\t// case, we advance |next| until the cursors are aligned and then\n\t// process as if Case (1):\n\t//\n\t// Case (3+4): |sc.cur| is \"behind\" |next|, we must consume elements in\n\t// |sc.cur| until either:\n\t//\n\t// Case (3): |sc.cur| aligns with |next|. In this case, we just assign\n\t// sc.cur = next.\n\t// Case (4): A boundary is encountered which is aligned with a boundary\n\t// in the previous state. This is the critical case, as is allows\n\t// us to skip over large parts of the tree. In this case, we align\n\t// parent chunkers then sc.resume() at |next|\n\n\tfor sc.cur.compare(next) > 0 {\n\t\tnext.advance() // Case (2)\n\t}\n\n\t// If neither loop above and below are entered, it is Case (1). If the loop\n\t// below is entered but Case (4) isn't reached, then it is Case (3).\n\treachedNext := true\n\tfor sc.cur.compare(next) < 0 {\n\t\tif sc.Append(sc.cur.current()) && sc.cur.atLastItem() {\n\t\t\tif sc.cur.parent != nil {\n\n\t\t\t\tif sc.cur.parent.compare(next.parent) < 0 {\n\t\t\t\t\t// Case (4): We stopped consuming items on this level before entering\n\t\t\t\t\t// the sequence referenced by |next|\n\t\t\t\t\treachedNext = false\n\t\t\t\t}\n\n\t\t\t\t// Note: Logically, what is happening here is that we are consuming the\n\t\t\t\t// item at the current level. Logically, we'd call sc.cur.advance(),\n\t\t\t\t// but that would force loading of the next sequence, which we don't\n\t\t\t\t// need for any reason, so instead we advance the parent and take care\n\t\t\t\t// not to allow it to step outside the sequence.\n\t\t\t\tsc.cur.parent.advanceMaybeAllowPastEnd(false)\n\n\t\t\t\t// Invalidate this cursor, since it is now inconsistent with its parent\n\t\t\t\tsc.cur.parent = nil\n\t\t\t\tsc.cur.seq = nil\n\t\t\t}\n\n\t\t\tbreak\n\t\t}\n\n\t\tsc.cur.advance()\n\t}\n\n\tif sc.parent != nil && next.parent != nil {\n\t\tsc.parent.advanceTo(next.parent)\n\t}\n\n\tsc.cur = next\n\tif !reachedNext {\n\t\tsc.resume() // Case (4)\n\t}\n}\n\nfunc (sc *sequenceChunker) Append(item sequenceItem) bool {\n\td.PanicIfTrue(item == nil)\n\tsc.current = append(sc.current, item)\n\tsc.hashValueBytes(item, sc.rv)\n\tif sc.rv.crossedBoundary {\n\t\tsc.handleChunkBoundary()\n\t\treturn true\n\t}\n\treturn false\n}\n\nfunc (sc *sequenceChunker) Skip() {\n\tsc.cur.advance()\n}\n\nfunc (sc *sequenceChunker) createParent() {\n\td.PanicIfFalse(sc.parent == nil)\n\tvar parent *sequenceCursor\n\tif sc.cur != nil && sc.cur.parent != nil {\n\t\t// Clone the parent cursor because otherwise calling cur.advance() will affect our parent - and vice versa - in surprising ways. Instead, Skip moves forward our parent's cursor if we advance across a boundary.\n\t\tparent = sc.cur.parent\n\t}\n\tsc.parent = newSequenceChunker(parent, sc.level+1, sc.vrw, sc.parentMakeChunk, sc.parentMakeChunk, metaHashValueBytes)\n\tsc.parent.isLeaf = false\n\n\tif sc.unwrittenCol != nil {\n\t\t// There is an unwritten collection, but this chunker now has a parent, so\n\t\t// write it. See createSequence().\n\t\tsc.vrw.WriteValue(sc.unwrittenCol)\n\t\tsc.unwrittenCol = nil\n\t}\n}\n\n// createSequence creates a sequence from the current items in |sc.current|,\n// clears the current items, then returns the new sequence and a metaTuple that\n// points to it.\n//\n// If |write| is true then the sequence is eagerly written, or if false it's\n// manually constructed and stored in |sc.unwrittenCol| to possibly write later\n// in createParent(). This is to hopefully avoid unnecessarily writing the root\n// chunk (for example, the sequence may be stored inline).\n//\n// There is a catch: in the rare case that the root chunk is actually not the\n// canonical root of the sequence (see Done()), then we will have ended up\n// unnecessarily writing a chunk - the canonical root. However, this is a fair\n// tradeoff for simplicity of the chunking algorithm.\nfunc (sc *sequenceChunker) createSequence(write bool) (sequence, metaTuple) {\n\tcol, key, numLeaves := sc.makeChunk(sc.level, sc.current)\n\n\t// |sc.makeChunk| copies |sc.current| so it's safe to re-use the memory.\n\tsc.current = sc.current[:0]\n\n\tvar ref Ref\n\tif write {\n\t\tref = sc.vrw.WriteValue(col)\n\t} else {\n\t\tref = NewRef(col)\n\t\tsc.unwrittenCol = col\n\t}\n\n\tmt := newMetaTuple(ref, key, numLeaves)\n\treturn col.asSequence(), mt\n}\n\nfunc (sc *sequenceChunker) handleChunkBoundary() {\n\td.PanicIfFalse(len(sc.current) > 0)\n\tsc.rv.Reset()\n\tif sc.parent == nil {\n\t\tsc.createParent()\n\t}\n\t_, mt := sc.createSequence(true)\n\tsc.parent.Append(mt)\n}\n\n// Returns true if this chunker or any of its parents have any pending items in their |current| slice.\nfunc (sc *sequenceChunker) anyPending() bool {\n\tif len(sc.current) > 0 {\n\t\treturn true\n\t}\n\n\tif sc.parent != nil {\n\t\treturn sc.parent.anyPending()\n\t}\n\n\treturn false\n}\n\n// Returns the root sequence of the resulting tree. The logic here is subtle, but hopefully correct and understandable. See comments inline.\nfunc (sc *sequenceChunker) Done() sequence {\n\td.PanicIfTrue(sc.done)\n\tsc.done = true\n\n\tif sc.cur != nil {\n\t\tsc.finalizeCursor()\n\t}\n\n\t// There is pending content above us, so we must push any remaining items from this level up and allow some parent to find the root of the resulting tree.\n\tif sc.parent != nil && sc.parent.anyPending() {\n\t\tif len(sc.current) > 0 {\n\t\t\t// If there are items in |current| at this point, they represent the final items of the sequence which occurred beyond the previous *explicit* chunk boundary. The end of input of a sequence is considered an *implicit* boundary.\n\t\t\tsc.handleChunkBoundary()\n\t\t}\n\n\t\treturn sc.parent.Done()\n\t}\n\n\t// At this point, we know this chunker contains, in |current| every item at this level of the resulting tree. To see this, consider that there are two ways a chunker can enter items into its |current|: (1) as the result of resume() with the cursor on anything other than the first item in the sequence, and (2) as a result of a child chunker hitting an explicit chunk boundary during either Append() or finalize(). The only way there can be no items in some parent chunker's |current| is if this chunker began with cursor within its first existing chunk (and thus all parents resume()'d with a cursor on their first item) and continued through all sebsequent items without creating any explicit chunk boundaries (and thus never sent any items up to a parent as a result of chunking). Therefore, this chunker's current must contain all items within the current sequence.\n\n\t// This level must represent *a* root of the tree, but it is possibly non-canonical. There are three cases to consider:\n\n\t// (1) This is \"leaf\" chunker and thus produced tree of depth 1 which contains exactly one chunk (never hit a boundary), or (2) This in an internal node of the tree which contains multiple references to child nodes. In either case, this is the canonical root of the tree.\n\tif sc.isLeaf || len(sc.current) > 1 {\n\t\tseq, _ := sc.createSequence(false)\n\t\treturn seq\n\t}\n\n\t// (3) This is an internal node of the tree which contains a single reference to a child node. This can occur if a non-leaf chunker happens to chunk on the first item (metaTuple) appended. In this case, this is the root of the tree, but it is *not* canonical and we must walk down until we find cases (1) or (2), above.\n\td.PanicIfFalse(!sc.isLeaf && len(sc.current) == 1)\n\tmt := sc.current[0].(metaTuple)\n\n\tfor {\n\t\tchild := mt.getChildSequence(sc.vrw)\n\t\tif _, ok := child.(metaSequence); !ok || child.seqLen() > 1 {\n\t\t\treturn child\n\t\t}\n\n\t\tmt = child.getItem(0).(metaTuple)\n\t}\n}\n\n// If we are mutating an existing sequence, appending subsequent items in the sequence until we reach a pre-existing chunk boundary or the end of the sequence.\nfunc (sc *sequenceChunker) finalizeCursor() {\n\tfor ; sc.cur.valid(); sc.cur.advance() {\n\t\tif sc.Append(sc.cur.current()) && sc.cur.atLastItem() {\n\t\t\tbreak // boundary occurred at same place in old & new sequence\n\t\t}\n\t}\n\n\tif sc.cur.parent != nil {\n\t\tsc.cur.parent.advance()\n\n\t\t// Invalidate this cursor, since it is now inconsistent with its parent\n\t\tsc.cur.parent = nil\n\t\tsc.cur.seq = nil\n\t}\n}\n" }, { "path": "go/types/sequence_concat.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\ntype newSequenceChunkerFn func(cur *sequenceCursor, vrw ValueReadWriter) *sequenceChunker\n\nfunc concat(fst, snd sequence, newSequenceChunker newSequenceChunkerFn) sequence {\n\tif fst.numLeaves() == 0 {\n\t\treturn snd\n\t}\n\tif snd.numLeaves() == 0 {\n\t\treturn fst\n\t}\n\n\t// concat works by tricking the sequenceChunker into resuming chunking at a\n\t// cursor to the end of fst, then finalizing chunking to the start of snd - by\n\t// swapping fst cursors for snd cursors in the middle of chunking.\n\tvrw := fst.valueReadWriter()\n\tchunker := newSequenceChunker(newCursorAtIndex(fst, fst.numLeaves()), vrw)\n\n\tfor cur, ch := newCursorAtIndex(snd, 0), chunker; ch != nil; ch = ch.parent {\n\t\t// Note that if snd is shallower than fst, then higher chunkers will have\n\t\t// their cursors set to nil. This has the effect of \"dropping\" the final\n\t\t// item in each of those sequences.\n\t\tch.cur = cur\n\t\tif cur != nil {\n\t\t\tcur = cur.parent\n\t\t\tif cur != nil && ch.parent == nil {\n\t\t\t\t// If fst is shallower than snd, its cur will have a parent whereas the\n\t\t\t\t// chunker to snd won't. In that case, create a parent for fst.\n\t\t\t\tch.createParent()\n\t\t\t}\n\t\t}\n\t}\n\n\treturn chunker.Done()\n}\n" }, { "path": "go/types/sequence_cursor.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport \"github.com/attic-labs/noms/go/d\"\nimport \"fmt\"\n\n// sequenceCursor explores a tree of sequence items.\ntype sequenceCursor struct {\n\tparent *sequenceCursor\n\tseq sequence\n\tidx int\n\tseqLen int\n}\n\n// newSequenceCursor creates a cursor on seq positioned at idx.\n// If idx < 0, count backward from the end of seq.\nfunc newSequenceCursor(parent *sequenceCursor, seq sequence, idx int) *sequenceCursor {\n\td.PanicIfTrue(seq == nil)\n\tseqLen := seq.seqLen()\n\tif idx < 0 {\n\t\tidx += seqLen\n\t\td.PanicIfFalse(idx >= 0)\n\t}\n\n\treturn &sequenceCursor{parent, seq, idx, seqLen}\n}\n\nfunc (cur *sequenceCursor) length() int {\n\treturn cur.seqLen\n}\n\nfunc (cur *sequenceCursor) getItem(idx int) sequenceItem {\n\treturn cur.seq.getItem(idx)\n}\n\n// sync loads the sequence that the cursor index points to.\n// It's called whenever the cursor advances/retreats to a different chunk.\nfunc (cur *sequenceCursor) sync() {\n\td.PanicIfFalse(cur.parent != nil)\n\tcur.seq = cur.parent.getChildSequence()\n\tcur.seqLen = cur.seq.seqLen()\n}\n\n// getChildSequence retrieves the child at the current cursor position.\nfunc (cur *sequenceCursor) getChildSequence() sequence {\n\treturn cur.seq.getChildSequence(cur.idx)\n}\n\n// current returns the value at the current cursor position\nfunc (cur *sequenceCursor) current() sequenceItem {\n\td.PanicIfFalse(cur.valid())\n\treturn cur.getItem(cur.idx)\n}\n\nfunc (cur *sequenceCursor) valid() bool {\n\treturn cur.idx >= 0 && cur.idx < cur.length()\n}\n\nfunc (cur *sequenceCursor) indexInChunk() int {\n\treturn cur.idx\n}\n\nfunc (cur *sequenceCursor) atLastItem() bool {\n\treturn cur.idx == cur.length()-1\n}\n\nfunc (cur *sequenceCursor) advance() bool {\n\treturn cur.advanceMaybeAllowPastEnd(true)\n}\n\nfunc (cur *sequenceCursor) advanceMaybeAllowPastEnd(allowPastEnd bool) bool {\n\tif cur.idx < cur.length()-1 {\n\t\tcur.idx++\n\t\treturn true\n\t}\n\tif cur.idx == cur.length() {\n\t\treturn false\n\t}\n\tif cur.parent != nil && cur.parent.advanceMaybeAllowPastEnd(false) {\n\t\t// at end of current leaf chunk and there are more\n\t\tcur.sync()\n\t\tcur.idx = 0\n\t\treturn true\n\t}\n\tif allowPastEnd {\n\t\tcur.idx++\n\t}\n\treturn false\n}\n\nfunc (cur *sequenceCursor) retreat() bool {\n\treturn cur.retreatMaybeAllowBeforeStart(true)\n}\n\nfunc (cur *sequenceCursor) retreatMaybeAllowBeforeStart(allowBeforeStart bool) bool {\n\tif cur.idx > 0 {\n\t\tcur.idx--\n\t\treturn true\n\t}\n\tif cur.idx == -1 {\n\t\treturn false\n\t}\n\td.PanicIfFalse(0 == cur.idx)\n\tif cur.parent != nil && cur.parent.retreatMaybeAllowBeforeStart(false) {\n\t\tcur.sync()\n\t\tcur.idx = cur.length() - 1\n\t\treturn true\n\t}\n\tif allowBeforeStart {\n\t\tcur.idx--\n\t}\n\treturn false\n}\n\n// clone creates a copy of the cursor\nfunc (cur *sequenceCursor) clone() *sequenceCursor {\n\tvar parent *sequenceCursor\n\tif cur.parent != nil {\n\t\tparent = cur.parent.clone()\n\t}\n\tcl := newSequenceCursor(parent, cur.seq, cur.idx)\n\treturn cl\n}\n\ntype cursorIterCallback func(item interface{}) bool\n\nfunc (cur *sequenceCursor) String() string {\n\tif cur.parent == nil {\n\t\treturn fmt.Sprintf(\"%s (%d): %d\", newMap(cur.seq.(orderedSequence)).Hash().String(), cur.seq.seqLen(), cur.idx)\n\t}\n\n\treturn fmt.Sprintf(\"%s (%d): %d -- %s\", newMap(cur.seq.(orderedSequence)).Hash().String(), cur.seq.seqLen(), cur.idx, cur.parent.String())\n}\n\nfunc (cur *sequenceCursor) compare(other *sequenceCursor) int {\n\tif cur.parent != nil {\n\t\td.PanicIfFalse(other.parent != nil)\n\t\tp := cur.parent.compare(other.parent)\n\t\tif p != 0 {\n\t\t\treturn p\n\t\t}\n\n\t}\n\n\t// TODO: It'd be nice here to assert that the two sequences are the same\n\t// but there isn't a good way to that at this point because the containing\n\t// collection of the sequence isn't available.\n\td.PanicIfFalse(cur.seq.seqLen() == other.seq.seqLen())\n\treturn cur.idx - other.idx\n}\n\n// iter iterates forward from the current position\nfunc (cur *sequenceCursor) iter(cb cursorIterCallback) {\n\tfor cur.valid() && !cb(cur.getItem(cur.idx)) {\n\t\tcur.advance()\n\t}\n}\n\n// newCursorAtIndex creates a new cursor over seq positioned at idx.\n//\n// Implemented by searching down the tree to the leaf sequence containing idx. Each\n// sequence cursor includes a back pointer to its parent so that it can follow the path\n// to the next leaf chunk when the cursor exhausts the entries in the current chunk.\nfunc newCursorAtIndex(seq sequence, idx uint64) *sequenceCursor {\n\tvar cur *sequenceCursor\n\tfor {\n\t\tcur = newSequenceCursor(cur, seq, 0)\n\t\tidx = idx - advanceCursorToOffset(cur, idx)\n\t\tcs := cur.getChildSequence()\n\t\tif cs == nil {\n\t\t\tbreak\n\t\t}\n\t\tseq = cs\n\t}\n\td.PanicIfTrue(cur == nil)\n\treturn cur\n}\n" }, { "path": "go/types/sequence_cursor_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype testSequence struct {\n\titems []interface{}\n}\n\n// sequence interface\nfunc (ts testSequence) getItem(idx int) sequenceItem {\n\treturn ts.items[idx]\n}\n\nfunc (ts testSequence) seqLen() int {\n\treturn len(ts.items)\n}\n\nfunc (ts testSequence) numLeaves() uint64 {\n\treturn uint64(len(ts.items))\n}\n\nfunc (ts testSequence) cumulativeNumberOfLeaves(idx int) uint64 {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) getCompositeChildSequence(start uint64, length uint64) sequence {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) treeLevel() uint64 {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) Kind() NomsKind {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) getCompareFn(other sequence) compareFn {\n\tobl := other.(testSequence)\n\treturn func(idx, otherIdx int) bool {\n\t\treturn ts.items[idx] == obl.items[otherIdx]\n\t}\n}\n\nfunc (ts testSequence) valueReadWriter() ValueReadWriter {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) writeTo(nomsWriter) {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) getChildSequence(idx int) sequence {\n\tchild := ts.items[idx]\n\treturn testSequence{child.([]interface{})}\n}\n\nfunc (ts testSequence) isLeaf() bool {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) Equals(other Value) bool {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) valueBytes() []byte {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) valuesSlice(from, to uint64) []Value {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) Less(other Value) bool {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) Hash() hash.Hash {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) WalkValues(cb ValueCallback) {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) WalkRefs(cb RefCallback) {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) typeOf() *Type {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) Len() uint64 {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) Empty() bool {\n\tpanic(\"not reached\")\n}\n\nfunc (ts testSequence) asValueImpl() valueImpl {\n\tpanic(\"not reached\")\n}\n\nfunc newTestSequenceCursor(items []interface{}) *sequenceCursor {\n\tparent := newSequenceCursor(nil, testSequence{items}, 0)\n\titems = items[0].([]interface{})\n\treturn newSequenceCursor(parent, testSequence{items}, 0)\n}\n\n// TODO: Convert all tests to use newTestSequenceCursor3.\nfunc newTestSequenceCursor3(items []interface{}) *sequenceCursor {\n\ttop := newSequenceCursor(nil, testSequence{items}, 0)\n\titems = items[0].([]interface{})\n\tmiddle := newSequenceCursor(top, testSequence{items}, 0)\n\titems = items[0].([]interface{})\n\treturn newSequenceCursor(middle, testSequence{items}, 0)\n}\n\nfunc TestTestCursor(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvar cur *sequenceCursor\n\treset := func() {\n\t\tcur = newTestSequenceCursor([]interface{}{[]interface{}{100, 101}, []interface{}{102}})\n\t}\n\texpect := func(expectIdx, expectParentIdx int, expectOk bool, expectVal sequenceItem) {\n\t\tassert.Equal(expectIdx, cur.indexInChunk())\n\t\tassert.Equal(expectParentIdx, cur.parent.indexInChunk())\n\t\tassert.Equal(expectOk, cur.valid())\n\t\tif cur.valid() {\n\t\t\tassert.Equal(expectVal, cur.current())\n\t\t}\n\t}\n\n\t// Test retreating past the start.\n\treset()\n\texpect(0, 0, true, sequenceItem(100))\n\tassert.False(cur.retreat())\n\texpect(-1, 0, false, nil)\n\tassert.False(cur.retreat())\n\texpect(-1, 0, false, nil)\n\n\t// Test retreating past the start, then advanding past the end.\n\treset()\n\tassert.False(cur.retreat())\n\tassert.True(cur.advance())\n\texpect(0, 0, true, sequenceItem(100))\n\tassert.True(cur.advance())\n\texpect(1, 0, true, sequenceItem(101))\n\tassert.True(cur.advance())\n\texpect(0, 1, true, sequenceItem(102))\n\tassert.False(cur.advance())\n\texpect(1, 1, false, nil)\n\tassert.False(cur.advance())\n\texpect(1, 1, false, nil)\n\n\t// Test advancing past the end.\n\treset()\n\tassert.True(cur.advance())\n\texpect(1, 0, true, sequenceItem(101))\n\tassert.True(cur.retreat())\n\texpect(0, 0, true, sequenceItem(100))\n\tassert.False(cur.retreat())\n\texpect(-1, 0, false, nil)\n\tassert.False(cur.retreat())\n\texpect(-1, 0, false, nil)\n\n\t// Test advancing past the end, then retreating past the start.\n\treset()\n\tassert.True(cur.advance())\n\tassert.True(cur.advance())\n\texpect(0, 1, true, sequenceItem(102))\n\tassert.False(cur.advance())\n\texpect(1, 1, false, nil)\n\tassert.False(cur.advance())\n\texpect(1, 1, false, nil)\n\tassert.True(cur.retreat())\n\texpect(0, 1, true, sequenceItem(102))\n\tassert.True(cur.retreat())\n\texpect(1, 0, true, sequenceItem(101))\n\tassert.True(cur.retreat())\n\texpect(0, 0, true, sequenceItem(100))\n\tassert.False(cur.retreat())\n\texpect(-1, 0, false, nil)\n\tassert.False(cur.retreat())\n\texpect(-1, 0, false, nil)\n}\n" }, { "path": "go/types/set.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"fmt\"\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype Set struct {\n\torderedSequence\n}\n\nfunc newSet(seq orderedSequence) Set {\n\treturn Set{seq}\n}\n\nfunc NewSet(vrw ValueReadWriter, v ...Value) Set {\n\tdata := buildSetData(v)\n\tch := newEmptySetSequenceChunker(vrw)\n\n\tfor _, v := range data {\n\t\tch.Append(v)\n\t}\n\n\treturn newSet(ch.Done().(orderedSequence))\n}\n\n// NewStreamingSet takes an input channel of values and returns a output\n// channel that will produce a finished Set. Values that are sent to the input\n// channel must be in Noms sortorder, adding values to the input channel\n// out of order will result in a panic. Once the input channel is closed\n// by the caller, a finished Set will be sent to the output channel. See\n// graph_builder.go for building collections with values that are not in order.\nfunc NewStreamingSet(vrw ValueReadWriter, vChan <-chan Value) <-chan Set {\n\treturn newStreamingSet(vrw, vChan, func(vrw ValueReadWriter, vChan <-chan Value, outChan chan<- Set) {\n\t\tgo readSetInput(vrw, vChan, outChan)\n\t})\n}\n\ntype streamingSetReadFunc func(vrw ValueReadWriter, vChan <-chan Value, outChan chan<- Set)\n\nfunc newStreamingSet(vrw ValueReadWriter, vChan <-chan Value, readFunc streamingSetReadFunc) <-chan Set {\n\td.PanicIfTrue(vrw == nil)\n\toutChan := make(chan Set, 1)\n\treadFunc(vrw, vChan, outChan)\n\treturn outChan\n}\n\nfunc readSetInput(vrw ValueReadWriter, vChan <-chan Value, outChan chan<- Set) {\n\tdefer close(outChan)\n\tch := newEmptySetSequenceChunker(vrw)\n\tvar lastV Value\n\tfor v := range vChan {\n\t\td.PanicIfTrue(v == nil)\n\t\tif lastV != nil {\n\t\t\td.PanicIfFalse(lastV.Less(v))\n\t\t}\n\t\tlastV = v\n\t\tch.Append(v)\n\t}\n\toutChan <- newSet(ch.Done().(orderedSequence))\n}\n\n// Diff computes the diff from |last| to |m| using the top-down algorithm,\n// which completes as fast as possible while taking longer to return early\n// results than left-to-right.\nfunc (s Set) Diff(last Set, changes chan<- ValueChanged, closeChan <-chan struct{}) {\n\tif s.Equals(last) {\n\t\treturn\n\t}\n\torderedSequenceDiffTopDown(last.orderedSequence, s.orderedSequence, changes, closeChan)\n}\n\n// DiffHybrid computes the diff from |last| to |s| using a hybrid algorithm\n// which balances returning results early vs completing quickly, if possible.\nfunc (s Set) DiffHybrid(last Set, changes chan<- ValueChanged, closeChan <-chan struct{}) {\n\tif s.Equals(last) {\n\t\treturn\n\t}\n\torderedSequenceDiffBest(last.orderedSequence, s.orderedSequence, changes, closeChan)\n}\n\n// DiffLeftRight computes the diff from |last| to |s| using a left-to-right\n// streaming approach, optimised for returning results early, but not\n// completing quickly.\nfunc (s Set) DiffLeftRight(last Set, changes chan<- ValueChanged, closeChan <-chan struct{}) {\n\tif s.Equals(last) {\n\t\treturn\n\t}\n\torderedSequenceDiffLeftRight(last.orderedSequence, s.orderedSequence, changes, closeChan)\n}\n\nfunc (s Set) asSequence() sequence {\n\treturn s.orderedSequence\n}\n\n// Value interface\nfunc (s Set) Value() Value {\n\treturn s\n}\n\nfunc (s Set) WalkValues(cb ValueCallback) {\n\titerAll(s, func(v Value, idx uint64) {\n\t\tcb(v)\n\t})\n}\n\nfunc (s Set) First() Value {\n\tcur := newCursorAt(s.orderedSequence, emptyKey, false, false)\n\tif !cur.valid() {\n\t\treturn nil\n\t}\n\treturn cur.current().(Value)\n}\n\nfunc (s Set) At(idx uint64) Value {\n\tif idx >= s.Len() {\n\t\tpanic(fmt.Errorf(\"Out of bounds: %d >= %d\", idx, s.Len()))\n\t}\n\n\tcur := newCursorAtIndex(s.orderedSequence, idx)\n\treturn cur.current().(Value)\n}\n\nfunc (s Set) Has(v Value) bool {\n\tcur := newCursorAtValue(s.orderedSequence, v, false, false)\n\treturn cur.valid() && cur.current().(Value).Equals(v)\n}\n\ntype setIterCallback func(v Value) bool\n\nfunc (s Set) Iter(cb setIterCallback) {\n\tcur := newCursorAt(s.orderedSequence, emptyKey, false, false)\n\tcur.iter(func(v interface{}) bool {\n\t\treturn cb(v.(Value))\n\t})\n}\n\ntype setIterAllCallback func(v Value)\n\nfunc (s Set) IterAll(cb setIterAllCallback) {\n\titerAll(s, func(v Value, idx uint64) {\n\t\tcb(v)\n\t})\n}\n\nfunc (s Set) Iterator() SetIterator {\n\treturn s.IteratorAt(0)\n}\n\nfunc (s Set) IteratorAt(idx uint64) SetIterator {\n\treturn &setIterator{\n\t\tcursor: newCursorAtIndex(s.orderedSequence, idx),\n\t\ts: s,\n\t}\n}\n\nfunc (s Set) IteratorFrom(val Value) SetIterator {\n\treturn &setIterator{\n\t\tcursor: newCursorAtValue(s.orderedSequence, val, false, false),\n\t\ts: s,\n\t}\n}\n\nfunc (s Set) Edit() *SetEditor {\n\treturn NewSetEditor(s)\n}\n\nfunc buildSetData(values ValueSlice) ValueSlice {\n\tif len(values) == 0 {\n\t\treturn ValueSlice{}\n\t}\n\n\tuniqueSorted := make(ValueSlice, 0, len(values))\n\tsort.Stable(values)\n\tlast := values[0]\n\tfor i := 1; i < len(values); i++ {\n\t\tv := values[i]\n\t\tif !v.Equals(last) {\n\t\t\tuniqueSorted = append(uniqueSorted, last)\n\t\t}\n\t\tlast = v\n\t}\n\n\treturn append(uniqueSorted, last)\n}\n\nfunc makeSetLeafChunkFn(vrw ValueReadWriter) makeChunkFn {\n\treturn func(level uint64, items []sequenceItem) (Collection, orderedKey, uint64) {\n\t\td.PanicIfFalse(level == 0)\n\t\tsetData := make([]Value, len(items), len(items))\n\n\t\tvar lastValue Value\n\t\tfor i, item := range items {\n\t\t\tv := item.(Value)\n\t\t\td.PanicIfFalse(lastValue == nil || lastValue.Less(v))\n\t\t\tlastValue = v\n\t\t\tsetData[i] = v\n\t\t}\n\n\t\tset := newSet(newSetLeafSequence(vrw, setData...))\n\t\tvar key orderedKey\n\t\tif len(setData) > 0 {\n\t\t\tkey = newOrderedKey(setData[len(setData)-1])\n\t\t}\n\n\t\treturn set, key, uint64(len(items))\n\t}\n}\n\nfunc newEmptySetSequenceChunker(vrw ValueReadWriter) *sequenceChunker {\n\treturn newEmptySequenceChunker(vrw, makeSetLeafChunkFn(vrw), newOrderedMetaSequenceChunkFn(SetKind, vrw), hashValueBytes)\n}\n" }, { "path": "go/types/set_editor.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// SetEditor allows for efficient editing of Set-typed prolly trees. Edits\n// are buffered to memory and can be applied via Build(), which returns a new\n// Set. Prior to Build(), Get() & Has() will return the value that the resulting\n// Set would return if it were built immediately prior to the respective call.\n// Note: The implementation biases performance towards a usage which applies\n// edits in key-order.\ntype SetEditor struct {\n\ts Set\n\tedits setEditSlice // edits may contain duplicate values, in which case, the last edit of a given key is used\n\tnormalized bool\n}\n\nfunc NewSetEditor(s Set) *SetEditor {\n\treturn &SetEditor{s, setEditSlice{}, true}\n}\n\nfunc (se *SetEditor) Kind() NomsKind {\n\treturn SetKind\n}\n\nfunc (se *SetEditor) Value() Value {\n\treturn se.Set()\n}\n\nfunc (se *SetEditor) Set() Set {\n\tif len(se.edits) == 0 {\n\t\treturn se.s // no edits\n\t}\n\n\tseq := se.s.orderedSequence\n\tvrw := seq.valueReadWriter()\n\n\tse.normalize()\n\n\tcursChan := make(chan chan *sequenceCursor)\n\teditChan := make(chan setEdit)\n\n\tgo func() {\n\t\tfor i, edit := range se.edits {\n\t\t\tif i+1 < len(se.edits) && se.edits[i+1].value.Equals(edit.value) {\n\t\t\t\tcontinue // next edit supercedes this one\n\t\t\t}\n\n\t\t\tedit := edit\n\n\t\t\t// Load cursor. TODO: Use ReadMany\n\t\t\tcc := make(chan *sequenceCursor, 1)\n\t\t\tcursChan <- cc\n\n\t\t\tgo func() {\n\t\t\t\tcc <- newCursorAtValue(seq, edit.value, true, false)\n\t\t\t}()\n\n\t\t\teditChan <- edit\n\t\t}\n\t\tclose(cursChan)\n\t\tclose(editChan)\n\t}()\n\n\tvar ch *sequenceChunker\n\tfor cc := range cursChan {\n\t\tcur := <-cc\n\t\tedit := <-editChan\n\n\t\texists := false\n\t\tif cur.idx < cur.seq.seqLen() {\n\t\t\tv := cur.current().(Value)\n\t\t\tif v.Equals(edit.value) {\n\t\t\t\texists = true\n\t\t\t}\n\t\t}\n\n\t\tif exists && edit.insert {\n\t\t\tcontinue // already present\n\t\t}\n\n\t\tif !exists && !edit.insert {\n\t\t\tcontinue // already non-present\n\t\t}\n\n\t\tif ch == nil {\n\t\t\tch = newSequenceChunker(cur, 0, vrw, makeSetLeafChunkFn(vrw), newOrderedMetaSequenceChunkFn(SetKind, vrw), hashValueBytes)\n\t\t} else {\n\t\t\tch.advanceTo(cur)\n\t\t}\n\n\t\tif edit.insert {\n\t\t\tch.Append(edit.value)\n\t\t} else {\n\t\t\tch.Skip()\n\t\t}\n\t}\n\n\tif ch == nil {\n\t\treturn se.s // no edits required application\n\t}\n\n\treturn newSet(ch.Done().(orderedSequence))\n}\n\nfunc (se *SetEditor) Insert(vs ...Value) *SetEditor {\n\tsort.Stable(ValueSlice(vs))\n\tfor _, v := range vs {\n\t\td.PanicIfTrue(v == nil)\n\t\tse.edit(v, true)\n\t}\n\treturn se\n}\n\nfunc (se *SetEditor) Remove(vs ...Value) *SetEditor {\n\tsort.Stable(ValueSlice(vs))\n\tfor _, v := range vs {\n\t\td.PanicIfTrue(v == nil)\n\t\tse.edit(v, false)\n\t}\n\treturn se\n}\n\nfunc (se *SetEditor) Has(v Value) bool {\n\tif idx, found := se.findEdit(v); found {\n\t\treturn se.edits[idx].insert\n\t}\n\n\treturn se.s.Has(v)\n}\n\nfunc (se *SetEditor) edit(v Value, insert bool) {\n\tif len(se.edits) == 0 {\n\t\tse.edits = append(se.edits, setEdit{v, insert})\n\t\treturn\n\t}\n\n\tfinal := se.edits[len(se.edits)-1]\n\tif final.value.Equals(v) {\n\t\tse.edits[len(se.edits)-1] = setEdit{v, insert}\n\t\treturn // update the last edit\n\t}\n\n\tse.edits = append(se.edits, setEdit{v, insert})\n\n\tif se.normalized && final.value.Less(v) {\n\t\t// fast-path: edits take place in key-order\n\t\treturn\n\t}\n\n\t// de-normalize\n\tse.normalized = false\n}\n\n// Find the edit position of the last edit for a given key\nfunc (se *SetEditor) findEdit(v Value) (idx int, found bool) {\n\tse.normalize()\n\n\tidx = sort.Search(len(se.edits), func(i int) bool {\n\t\treturn !se.edits[i].value.Less(v)\n\t})\n\n\tif idx == len(se.edits) {\n\t\treturn\n\t}\n\n\tif !se.edits[idx].value.Equals(v) {\n\t\treturn\n\t}\n\n\t// advance to final edit position where kv.key == k\n\tfor idx < len(se.edits) && se.edits[idx].value.Equals(v) {\n\t\tidx++\n\t}\n\tidx--\n\n\tfound = true\n\treturn\n}\n\nfunc (se *SetEditor) normalize() {\n\tif se.normalized {\n\t\treturn\n\t}\n\n\tsort.Stable(se.edits)\n\t// TODO: GC duplicate keys over some threshold of collectable memory?\n\tse.normalized = true\n}\n\ntype setEdit struct {\n\tvalue Value\n\tinsert bool\n}\n\ntype setEditSlice []setEdit\n\nfunc (ses setEditSlice) Len() int { return len(ses) }\nfunc (ses setEditSlice) Swap(i, j int) { ses[i], ses[j] = ses[j], ses[i] }\nfunc (ses setEditSlice) Less(i, j int) bool { return ses[i].value.Less(ses[j].value) }\n" }, { "path": "go/types/set_iterator.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// SetIterator defines methods that can be used to efficiently iterate through a set in 'Noms-defined'\n// sorted order.\ntype SetIterator interface {\n\t// Next returns subsequent values from a set. It returns nil, when no objects remain.\n\tNext() Value\n\n\t// SkipTo(v) advances to and returns the next value in the iterator >= v.\n\t// Note: if the iterator has already returned the value being skipped to, it will return the next\n\t// value (just as if Next() was called). For example, given the following set:\n\t// s = Set{ 0, 3, 6, 9, 12, 15, 18 }\n\t// An iterator on the set would return:\n\t// i := s.Iterator()\n\t// i.Next() return 0\n\t// i.SkipTo(4) -- returns 6\n\t// i.skipTo(3) -- returns 9 (this is the next value in the iterator >= 3)\n\t// i.skipTo(12) -- returns 12\n\t// i.skipTo(12) -- return 15 (this is the next value in the iterator >= 12)\n\t// i.skipTo(20) -- returns nil\n\t// If there are no values left in the iterator that are >= v,\n\t// the iterator will skip to the end of the sequence and return nil.\n\tSkipTo(v Value) Value\n}\n\ntype setIterator struct {\n\ts Set\n\tcursor *sequenceCursor\n\tcurrentValue Value\n}\n\nfunc (si *setIterator) Next() Value {\n\tif si.cursor.valid() {\n\t\tsi.currentValue = si.cursor.current().(Value)\n\t\tsi.cursor.advance()\n\t} else {\n\t\tsi.currentValue = nil\n\t}\n\treturn si.currentValue\n}\n\nfunc (si *setIterator) SkipTo(v Value) Value {\n\td.PanicIfTrue(v == nil)\n\tif si.cursor.valid() {\n\t\tif compareValue(v, si.currentValue) <= 0 {\n\t\t\treturn si.Next()\n\t\t}\n\n\t\tsi.cursor = newCursorAtValue(si.s.orderedSequence, v, true, false)\n\t\tif si.cursor.valid() {\n\t\t\tsi.currentValue = si.cursor.current().(Value)\n\t\t\tsi.cursor.advance()\n\t\t} else {\n\t\t\tsi.currentValue = nil\n\t\t}\n\t} else {\n\t\tsi.currentValue = nil\n\t}\n\treturn si.currentValue\n}\n\n// iterState contains iterator and it's current value\ntype iterState struct {\n\ti SetIterator\n\tv Value\n}\n\nfunc (st *iterState) Next() Value {\n\tif st.v == nil {\n\t\treturn nil\n\t}\n\tv := st.v\n\tst.v = st.i.Next()\n\treturn v\n}\n\nfunc (st *iterState) SkipTo(v Value) Value {\n\tif st.v == nil || v == nil {\n\t\tst.v = nil\n\t\treturn nil\n\t}\n\tst.v = st.i.SkipTo(v)\n\treturn st.v\n}\n\n// UnionIterator combines the results from two other iterators. The values from Next() are returned in\n// noms-defined order with all duplicates removed.\ntype UnionIterator struct {\n\taState iterState\n\tbState iterState\n}\n\n// NewUnionIterator creates a union iterator from two other SetIterators.\nfunc NewUnionIterator(iterA, iterB SetIterator) SetIterator {\n\td.PanicIfTrue(iterA == nil)\n\td.PanicIfTrue(iterB == nil)\n\ta := iterState{i: iterA, v: iterA.Next()}\n\tb := iterState{i: iterB, v: iterB.Next()}\n\treturn &UnionIterator{aState: a, bState: b}\n}\n\nfunc (u *UnionIterator) Next() Value {\n\tswitch compareValue(u.aState.v, u.bState.v) {\n\tcase -1:\n\t\treturn u.aState.Next()\n\tcase 0:\n\t\tu.aState.Next()\n\t\treturn u.bState.Next()\n\tcase 1:\n\t\treturn u.bState.Next()\n\t}\n\tpanic(\"Unreachable\")\n}\n\nfunc (u *UnionIterator) SkipTo(v Value) Value {\n\td.PanicIfTrue(v == nil)\n\tdidAdvance := false\n\tif compareValue(u.aState.v, v) < 0 {\n\t\tdidAdvance = true\n\t\tu.aState.SkipTo(v)\n\t}\n\tif compareValue(u.bState.v, v) < 0 {\n\t\tdidAdvance = true\n\t\tu.bState.SkipTo(v)\n\t}\n\tif !didAdvance {\n\t\treturn u.Next()\n\t}\n\tswitch compareValue(u.aState.v, u.bState.v) {\n\tcase -1:\n\t\treturn u.aState.Next()\n\tcase 0:\n\t\tu.aState.Next()\n\t\treturn u.bState.Next()\n\tcase 1:\n\t\treturn u.bState.Next()\n\t}\n\tpanic(\"Unreachable\")\n}\n\n// IntersectionIterator only returns values that are returned in both of its child iterators.\n// The values from Next() are returned in noms-defined order with all duplicates removed.\ntype IntersectionIterator struct {\n\taState iterState\n\tbState iterState\n}\n\n// NewIntersectionIterator creates a intersect iterator from two other SetIterators.\nfunc NewIntersectionIterator(iterA, iterB SetIterator) SetIterator {\n\td.PanicIfTrue(iterA == nil)\n\td.PanicIfTrue(iterB == nil)\n\ta := iterState{i: iterA, v: iterA.Next()}\n\tb := iterState{i: iterB, v: iterB.Next()}\n\treturn &IntersectionIterator{aState: a, bState: b}\n}\n\nfunc (i *IntersectionIterator) Next() Value {\n\tfor cont := true; cont; {\n\t\tswitch compareValue(i.aState.v, i.bState.v) {\n\t\tcase -1:\n\t\t\ti.aState.SkipTo(i.bState.v)\n\t\tcase 0:\n\t\t\tcont = false\n\t\tcase 1:\n\t\t\ti.bState.SkipTo(i.aState.v)\n\t\t}\n\t}\n\t// we only get here if aState and bState are equal\n\tres := i.aState.v\n\ti.aState.Next()\n\ti.bState.Next()\n\treturn res\n}\n\nfunc (i *IntersectionIterator) SkipTo(v Value) Value {\n\td.PanicIfTrue(v == nil)\n\tif compareValue(v, i.aState.v) >= 0 {\n\t\ti.aState.SkipTo(v)\n\t}\n\tif compareValue(v, i.bState.v) >= 0 {\n\t\ti.bState.SkipTo(v)\n\t}\n\treturn i.Next()\n}\n\n// considers nil max value, return -1 if v1 < v2, 0 if v1 == v2, 1 if v1 > v2\nfunc compareValue(v1, v2 Value) int {\n\tif v1 == nil && v2 == nil {\n\t\treturn 0\n\t}\n\tif v2 == nil || (v1 != nil && v1.Less(v2)) {\n\t\treturn -1\n\t}\n\tif v1 == nil || (v2 != nil && v2.Less(v1)) {\n\t\treturn 1\n\t}\n\treturn 0\n}\n" }, { "path": "go/types/set_iterator_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"math\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestSetIterator(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tnumbers := append(generateNumbersAsValues(10), Number(20), Number(25))\n\ts := NewSet(vs, numbers...)\n\ti := s.Iterator()\n\tvals := iterToSlice(i)\n\tassert.True(vals.Equals(numbers), \"Expected: %v != actual: %v\", numbers, vs)\n\n\ti = s.Iterator()\n\tassert.Panics(func() { i.SkipTo(nil) })\n\tassert.Equal(Number(0), i.SkipTo(Number(-20)))\n\tassert.Equal(Number(2), i.SkipTo(Number(2)))\n\tassert.Equal(Number(3), i.SkipTo(Number(-20)))\n\tassert.Equal(Number(5), i.SkipTo(Number(5)))\n\tassert.Equal(Number(6), i.Next())\n\tassert.Equal(Number(7), i.SkipTo(Number(6)))\n\tassert.Equal(Number(20), i.SkipTo(Number(15)))\n\tassert.Nil(i.SkipTo(Number(30)))\n\tassert.Nil(i.SkipTo(Number(30)))\n\tassert.Nil(i.SkipTo(Number(1)))\n\n\ti = s.Iterator()\n\tassert.Equal(Number(0), i.Next())\n\tassert.Equal(Number(1), i.Next())\n\tassert.Equal(Number(3), i.SkipTo(Number(3)))\n\tassert.Equal(Number(4), i.Next())\n\n\tempty := NewSet(vs)\n\tassert.Nil(empty.Iterator().Next())\n\tassert.Nil(empty.Iterator().SkipTo(Number(-30)))\n\n\tsingle := NewSet(vs, Number(42)).Iterator()\n\tassert.Equal(Number(42), single.SkipTo(Number(42)))\n\tassert.Equal(nil, single.SkipTo(Number(42)))\n\n\tsingle = NewSet(vs, Number(42)).Iterator()\n\tassert.Equal(Number(42), single.SkipTo(Number(42)))\n\tassert.Equal(nil, single.Next())\n\n\tsingle = NewSet(vs, Number(42)).Iterator()\n\tassert.Equal(Number(42), single.SkipTo(Number(21)))\n}\n\nfunc TestSetIteratorAt(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tnumbers := append(generateNumbersAsValues(5), Number(10))\n\ts := NewSet(vs, numbers...)\n\ti := s.IteratorAt(0)\n\tvals := iterToSlice(i)\n\tassert.True(vals.Equals(numbers), \"Expected: %v != actual: %v\", numbers, vs)\n\n\ti = s.IteratorAt(2)\n\tvals = iterToSlice(i)\n\tassert.True(vals.Equals(numbers[2:]), \"Expected: %v != actual: %v\", numbers[2:], vs)\n\n\ti = s.IteratorAt(10)\n\tvals = iterToSlice(i)\n\tassert.True(vals.Equals(nil), \"Expected: %v != actual: %v\", nil, vs)\n}\n\nfunc TestSetIteratorFrom(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tnumbers := append(generateNumbersAsValues(5), Number(10), Number(20))\n\ts := NewSet(vs, numbers...)\n\ti := s.IteratorFrom(Number(0))\n\tvals := iterToSlice(i)\n\tassert.True(vals.Equals(numbers), \"Expected: %v != actual: %v\", numbers, vs)\n\n\ti = s.IteratorFrom(Number(2))\n\tvals = iterToSlice(i)\n\tassert.True(vals.Equals(numbers[2:]), \"Expected: %v != actual: %v\", numbers[2:], vs)\n\n\ti = s.IteratorFrom(Number(10))\n\tvals = iterToSlice(i)\n\tassert.True(vals.Equals(ValueSlice{Number(10), Number(20)}), \"Expected: %v != actual: %v\", nil, vs)\n\n\ti = s.IteratorFrom(Number(20))\n\tvals = iterToSlice(i)\n\tassert.True(vals.Equals(ValueSlice{Number(20)}), \"Expected: %v != actual: %v\", nil, vs)\n\n\ti = s.IteratorFrom(Number(100))\n\tvals = iterToSlice(i)\n\tassert.True(vals.Equals(nil), \"Expected: %v != actual: %v\", nil, vs)\n\n\t// Not present. Starts at next larger.\n\ti = s.IteratorFrom(Number(15))\n\tvals = iterToSlice(i)\n\tassert.True(vals.Equals(ValueSlice{Number(20)}), \"Expected: %v != actual: %v\", nil, vs)\n}\n\nfunc TestUnionIterator(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tset1 := NewSet(vs, generateNumbersAsValuesFromToBy(0, 10, 1)...)\n\tset2 := NewSet(vs, generateNumbersAsValuesFromToBy(5, 15, 1)...)\n\tset3 := NewSet(vs, generateNumbersAsValuesFromToBy(10, 20, 1)...)\n\tset4 := NewSet(vs, generateNumbersAsValuesFromToBy(15, 25, 1)...)\n\n\tui1 := NewUnionIterator(set1.Iterator(), set2.Iterator())\n\tvals := iterToSlice(ui1)\n\texpectedRes := generateNumbersAsValues(15)\n\tassert.True(vals.Equals(expectedRes), \"Expected: %v != actual: %v\", expectedRes, vs)\n\n\tui1 = NewUnionIterator(set1.Iterator(), set4.Iterator())\n\tui2 := NewUnionIterator(set3.Iterator(), set2.Iterator())\n\tui3 := NewUnionIterator(ui1, ui2)\n\tvals = iterToSlice(ui3)\n\texpectedRes = generateNumbersAsValues(25)\n\tassert.True(vals.Equals(expectedRes), \"Expected: %v != actual: %v\", expectedRes, vs)\n\n\tui1 = NewUnionIterator(set1.Iterator(), set4.Iterator())\n\tui2 = NewUnionIterator(set3.Iterator(), set2.Iterator())\n\tui3 = NewUnionIterator(ui1, ui2)\n\n\tassert.Panics(func() { ui3.SkipTo(nil) })\n\tassert.Equal(Number(0), ui3.SkipTo(Number(-5)))\n\tassert.Equal(Number(5), ui3.SkipTo(Number(5)))\n\tassert.Equal(Number(8), ui3.SkipTo(Number(8)))\n\tassert.Equal(Number(9), ui3.SkipTo(Number(8)))\n\tassert.Equal(Number(10), ui3.SkipTo(Number(8)))\n\tassert.Equal(Number(11), ui3.SkipTo(Number(7)))\n\tassert.Equal(Number(12), ui3.Next())\n\tassert.Equal(Number(15), ui3.SkipTo(Number(15)))\n\tassert.Equal(Number(24), ui3.SkipTo(Number(24)))\n\tassert.Nil(ui3.SkipTo(Number(25)))\n\n\tsingleElemSet := NewSet(vs, Number(4))\n\temptySet := NewSet(vs)\n\n\tui10 := NewUnionIterator(singleElemSet.Iterator(), singleElemSet.Iterator())\n\tui20 := NewUnionIterator(emptySet.Iterator(), emptySet.Iterator())\n\tui30 := NewUnionIterator(ui10, ui20)\n\tvals = iterToSlice(ui30)\n\texpectedRes = ValueSlice{Number(4)}\n\tassert.True(vals.Equals(expectedRes), \"%v != %v\\n\", expectedRes, vs)\n}\n\nfunc TestIntersectionIterator(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tbyTwos := NewSet(vs, generateNumbersAsValuesFromToBy(0, 200, 2)...)\n\tbyThrees := NewSet(vs, generateNumbersAsValuesFromToBy(0, 200, 3)...)\n\tbyFives := NewSet(vs, generateNumbersAsValuesFromToBy(0, 200, 5)...)\n\n\ti1 := NewIntersectionIterator(byTwos.Iterator(), byThrees.Iterator())\n\tvals := iterToSlice(i1)\n\texpectedRes := generateNumbersAsValuesFromToBy(0, 200, 6)\n\tassert.True(vals.Equals(expectedRes), \"Expected: %v != actual: %v\", expectedRes, vs)\n\n\tit1 := NewIntersectionIterator(byTwos.Iterator(), byThrees.Iterator())\n\tit2 := NewIntersectionIterator(it1, byFives.Iterator())\n\tvals = iterToSlice(it2)\n\texpectedRes = generateNumbersAsValuesFromToBy(0, 200, 30)\n\tassert.True(vals.Equals(expectedRes), \"Expected: %v != actual: %v\", expectedRes, vs)\n\n\tit1 = NewIntersectionIterator(byThrees.Iterator(), byFives.Iterator())\n\tit2 = NewIntersectionIterator(it1, byTwos.Iterator())\n\n\tassert.Panics(func() { it2.SkipTo(nil) })\n\tassert.Equal(Number(30), it2.SkipTo(Number(5)))\n\tassert.Equal(Number(60), it2.SkipTo(Number(60)))\n\tassert.Equal(Number(90), it2.SkipTo(Number(5)))\n\tassert.Equal(Number(120), it2.Next())\n\tassert.Equal(Number(150), it2.SkipTo(Number(150)))\n\tassert.Nil(it2.SkipTo(Number(40000)))\n}\n\nfunc TestCombinationIterator(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tbyTwos := NewSet(vs, generateNumbersAsValuesFromToBy(0, 70, 2)...)\n\tbyThrees := NewSet(vs, generateNumbersAsValuesFromToBy(0, 70, 3)...)\n\tbyFives := NewSet(vs, generateNumbersAsValuesFromToBy(0, 70, 5)...)\n\tbySevens := NewSet(vs, generateNumbersAsValuesFromToBy(0, 70, 7)...)\n\n\tit1 := NewIntersectionIterator(byTwos.Iterator(), bySevens.Iterator())\n\tit2 := NewIntersectionIterator(byFives.Iterator(), byThrees.Iterator())\n\tut1 := NewUnionIterator(it1, it2)\n\tvals := iterToSlice(ut1)\n\texpectedRes := intsToValueSlice(0, 14, 15, 28, 30, 42, 45, 56, 60)\n\tassert.True(vals.Equals(expectedRes), \"Expected: %v != actual: %v\", expectedRes, vs)\n\n\tut1 = NewUnionIterator(byTwos.Iterator(), bySevens.Iterator())\n\tit2 = NewIntersectionIterator(byFives.Iterator(), byThrees.Iterator())\n\tut2 := NewIntersectionIterator(ut1, it2)\n\tvals = iterToSlice(ut2)\n\texpectedRes = intsToValueSlice(0, 30, 60)\n\tassert.True(vals.Equals(expectedRes), \"Expected: %v != actual: %v\", expectedRes, vs)\n}\n\ntype UnionTestIterator struct {\n\t*UnionIterator\n\tcntr *int\n}\n\nfunc (ui *UnionTestIterator) Next() Value {\n\t*ui.cntr++\n\treturn ui.UnionIterator.Next()\n}\n\nfunc (ui *UnionTestIterator) SkipTo(v Value) Value {\n\t*ui.cntr++\n\treturn ui.UnionIterator.SkipTo(v)\n}\n\nfunc NewUnionTestIterator(i1, i2 SetIterator, cntr *int) SetIterator {\n\tui := NewUnionIterator(i1, i2).(*UnionIterator)\n\treturn &UnionTestIterator{ui, cntr}\n}\n\n// When a binary tree of union operators is built on top of a list of sets, the complexity to\n// retrieve all of the elements in sorted order should be Log(N) * M where N is the number of sets func init() {\n// the list and M is the total number of elements in all of the sets.\nfunc TestUnionComplexity(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tnumSets := 256\n\tnumElemsPerSet := 1000\n\tlogNumSets := int(math.Ceil(math.Log2(float64(numSets))))\n\ttotalElems := numSets * numElemsPerSet\n\texpectedMax := logNumSets*totalElems + numSets\n\n\tcallCount1 := 0\n\titer := iterize(createSetsWithDistinctNumbers(vs, numSets, numElemsPerSet), NewUnionTestIterator, &callCount1)\n\tvals := iterToSlice(iter)\n\texpected := generateNumbersAsValueSlice(numSets * numElemsPerSet)\n\tassert.True(expected.Equals(vals), \"expected: %v != actual: %v\", expected, vals)\n\tassert.True(expectedMax > callCount1, \"callCount: %d exceeds expectedMax: %d\", callCount1, expectedMax)\n\n\tcallCount2 := 0\n\titer = iterize(createSetsWithSameNumbers(vs, numSets, numElemsPerSet), NewUnionTestIterator, &callCount2)\n\tvals = iterToSlice(iter)\n\texpected = generateNumbersAsValueSlice(numElemsPerSet)\n\tassert.True(expected.Equals(vals), \"expected: %v != actual: %v\", expected, vals)\n\tassert.True(expectedMax > callCount2, \"callCount: %d exceeds expectedMax: %d\", callCount2, expectedMax)\n}\n\ntype IntersectionTestIterator struct {\n\t*IntersectionIterator\n\tcntr *int\n}\n\nfunc (i *IntersectionTestIterator) Next() Value {\n\t*i.cntr++\n\treturn i.IntersectionIterator.Next()\n}\n\nfunc (i *IntersectionTestIterator) SkipTo(v Value) Value {\n\t*i.cntr++\n\treturn i.IntersectionIterator.SkipTo(v)\n}\n\nfunc NewIntersectionTestIterator(i1, i2 SetIterator, cntr *int) SetIterator {\n\tui := NewIntersectionIterator(i1, i2).(*IntersectionIterator)\n\treturn &IntersectionTestIterator{ui, cntr}\n}\n\n// When a binary tree of intersection operators is built on top of a list of sets, the complexity to\n// retrieve all of the elements in sorted order should be Log(N) * M where N is the number of sets func init() {\n// the list and M is the total number of elements in all of the sets.\nfunc TestIntersectComplexity(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\tnumSets := 256\n\tnumElemsPerSet := 1000\n\tlogNumSets := int(math.Ceil(math.Log2(float64(numSets))))\n\ttotalElems := numSets * numElemsPerSet\n\texpectedMax := logNumSets*totalElems + numSets\n\n\tcallCount1 := 0\n\titer := iterize(createSetsWithDistinctNumbers(vs, numSets, numElemsPerSet), NewIntersectionTestIterator, &callCount1)\n\tvals := iterToSlice(iter)\n\texpected := ValueSlice{}\n\tassert.True(expected.Equals(vals), \"expected: %v != actual: %v\", expected, vals)\n\tassert.True(expectedMax > callCount1, \"callCount: %d exceeds expectedMax: %d\", callCount1, expectedMax)\n\n\tcallCount2 := 0\n\titer = iterize(createSetsWithSameNumbers(vs, numSets, numElemsPerSet), NewIntersectionTestIterator, &callCount2)\n\tvals = iterToSlice(iter)\n\texpected = generateNumbersAsValueSlice(numElemsPerSet)\n\tassert.True(expected.Equals(vals), \"expected: %v != actual: %v\", expected, vals)\n\tassert.True(expectedMax > callCount2, \"callCount: %d exceeds expectedMax: %d\", callCount2, expectedMax)\n}\n\nfunc createSetsWithDistinctNumbers(vrw ValueReadWriter, numSets, numElemsPerSet int) []SetIterator {\n\titerSlice := []SetIterator{}\n\n\tfor i := 0; i < numSets; i++ {\n\t\tvals := ValueSlice{}\n\t\tfor j := 0; j < numElemsPerSet; j++ {\n\t\t\tvals = append(vals, Number(i+(numSets*j)))\n\t\t}\n\t\ts := NewSet(vrw, vals...)\n\t\titerSlice = append(iterSlice, s.Iterator())\n\t}\n\treturn iterSlice\n}\n\nfunc createSetsWithSameNumbers(vrw ValueReadWriter, numSets, numElemsPerSet int) []SetIterator {\n\tvs := ValueSlice{}\n\tfor j := 0; j < numElemsPerSet; j++ {\n\t\tvs = append(vs, Number(j))\n\t}\n\titerSlice := []SetIterator{}\n\tfor i := 0; i < numSets; i++ {\n\t\titerSlice = append(iterSlice, NewSet(vrw, vs...).Iterator())\n\t}\n\treturn iterSlice\n}\n\ntype newIterFunc func(i1, i2 SetIterator, cntr *int) SetIterator\n\n// Iterize calls itself recursively to build a binary tree of iterators over the original set.\nfunc iterize(iters []SetIterator, newIter newIterFunc, cntr *int) SetIterator {\n\tif len(iters) == 0 {\n\t\treturn nil\n\t}\n\tif len(iters) <= 1 {\n\t\treturn iters[0]\n\t}\n\tvar iter0 SetIterator\n\tnewIters := []SetIterator{}\n\tfor i, iter := range iters {\n\t\tif i%2 == 0 {\n\t\t\titer0 = iter\n\t\t} else {\n\t\t\tni := newIter(iter0, iter, cntr)\n\t\t\tnewIters = append(newIters, ni)\n\t\t\titer0 = nil\n\t\t}\n\t}\n\tif iter0 != nil {\n\t\tnewIters = append(newIters, iter0)\n\t}\n\treturn iterize(newIters, newIter, cntr)\n}\n" }, { "path": "go/types/set_leaf_sequence.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport \"sort\"\n\ntype setLeafSequence struct {\n\tleafSequence\n}\n\nfunc newSetLeafSequence(vrw ValueReadWriter, vs ...Value) orderedSequence {\n\treturn setLeafSequence{newLeafSequenceFromValues(SetKind, vrw, vs...)}\n}\n\nfunc (sl setLeafSequence) getCompareFn(other sequence) compareFn {\n\treturn sl.getCompareFnHelper(other.(setLeafSequence).leafSequence)\n}\n\n// orderedSequence interface\n\nfunc (sl setLeafSequence) getKey(idx int) orderedKey {\n\treturn newOrderedKey(sl.getItem(idx).(Value))\n}\n\nfunc (sl setLeafSequence) search(key orderedKey) int {\n\treturn sort.Search(int(sl.Len()), func(i int) bool {\n\t\treturn !sl.getKey(i).Less(key)\n\t})\n}\n" }, { "path": "go/types/set_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"math/rand\"\n\t\"sort\"\n\t\"sync\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst testSetSize = 5000\n\ntype testSet ValueSlice\n\ntype toTestSetFunc func(scale int, vrw ValueReadWriter) testSet\n\nfunc (ts testSet) Remove(from, to int) testSet {\n\tvalues := make(testSet, 0, len(ts)-(to-from))\n\tvalues = append(values, ts[:from]...)\n\tvalues = append(values, ts[to:]...)\n\treturn values\n}\n\nfunc (ts testSet) Has(key Value) bool {\n\tfor _, entry := range ts {\n\t\tif entry.Equals(key) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc (ts testSet) Diff(last testSet) (added []Value, removed []Value) {\n\t// Note: this could be use ts.toSet/last.toSet and then tsSet.Diff(lastSet) but the\n\t// purpose of this method is to be redundant.\n\tif len(ts) == 0 && len(last) == 0 {\n\t\treturn // nothing changed\n\t}\n\tif len(ts) == 0 {\n\t\t// everything removed\n\t\tfor _, entry := range last {\n\t\t\tremoved = append(removed, entry)\n\t\t}\n\t\treturn\n\t}\n\tif len(last) == 0 {\n\t\t// everything added\n\t\tfor _, entry := range ts {\n\t\t\tadded = append(added, entry)\n\t\t}\n\t\treturn\n\t}\n\tfor _, entry := range ts {\n\t\tif !last.Has(entry) {\n\t\t\tadded = append(added, entry)\n\t\t}\n\t}\n\tfor _, entry := range last {\n\t\tif !ts.Has(entry) {\n\t\t\tremoved = append(removed, entry)\n\t\t}\n\t}\n\treturn\n}\n\nfunc (ts testSet) toSet(vrw ValueReadWriter) Set {\n\treturn NewSet(vrw, ts...)\n}\n\nfunc newSortedTestSet(length int, gen genValueFn) (values testSet) {\n\tfor i := 0; i < length; i++ {\n\t\tvalues = append(values, gen(i))\n\t}\n\treturn\n}\n\nfunc newTestSetFromSet(s Set) testSet {\n\tvalues := make([]Value, 0, s.Len())\n\ts.IterAll(func(v Value) {\n\t\tvalues = append(values, v)\n\t})\n\treturn values\n}\n\nfunc newRandomTestSet(length int, gen genValueFn) testSet {\n\ts := rand.NewSource(4242)\n\tused := map[int]bool{}\n\n\tvar values []Value\n\tfor len(values) < length {\n\t\tv := int(s.Int63()) & 0xffffff\n\t\tif _, ok := used[v]; !ok {\n\t\t\tvalues = append(values, gen(v))\n\t\t\tused[v] = true\n\t\t}\n\t}\n\n\treturn values\n}\n\nfunc validateSet(t *testing.T, vrw ValueReadWriter, s Set, values ValueSlice) {\n\tassert.True(t, s.Equals(NewSet(vrw, values...)))\n\tout := ValueSlice{}\n\ts.IterAll(func(v Value) {\n\t\tout = append(out, v)\n\t})\n\tassert.True(t, out.Equals(values))\n}\n\ntype setTestSuite struct {\n\tcollectionTestSuite\n\telems testSet\n}\n\nfunc newSetTestSuite(size uint, expectChunkCount int, expectPrependChunkDiff int, expectAppendChunkDiff int, gen genValueFn) *setTestSuite {\n\tvs := newTestValueStore()\n\n\tlength := 1 << size\n\telemType := TypeOf(gen(0))\n\telems := newSortedTestSet(length, gen)\n\ttr := MakeSetType(elemType)\n\tset := NewSet(vs, elems...)\n\treturn &setTestSuite{\n\t\tcollectionTestSuite: collectionTestSuite{\n\t\t\tcol: set,\n\t\t\texpectType: tr,\n\t\t\texpectLen: uint64(length),\n\t\t\texpectChunkCount: expectChunkCount,\n\t\t\texpectPrependChunkDiff: expectPrependChunkDiff,\n\t\t\texpectAppendChunkDiff: expectAppendChunkDiff,\n\t\t\tvalidate: func(v2 Collection) bool {\n\t\t\t\tl2 := v2.(Set)\n\t\t\t\tout := ValueSlice{}\n\t\t\t\tl2.IterAll(func(v Value) {\n\t\t\t\t\tout = append(out, v)\n\t\t\t\t})\n\t\t\t\texp := ValueSlice(elems)\n\t\t\t\trv := exp.Equals(out)\n\t\t\t\tif !rv {\n\t\t\t\t\tprintBadCollections(exp, out)\n\t\t\t\t}\n\t\t\t\treturn rv\n\t\t\t},\n\t\t\tprependOne: func() Collection {\n\t\t\t\tdup := make([]Value, length+1)\n\t\t\t\tdup[0] = Number(-1)\n\t\t\t\tcopy(dup[1:], elems)\n\t\t\t\treturn NewSet(vs, dup...)\n\t\t\t},\n\t\t\tappendOne: func() Collection {\n\t\t\t\tdup := make([]Value, length+1)\n\t\t\t\tcopy(dup, elems)\n\t\t\t\tdup[len(dup)-1] = Number(length + 1)\n\t\t\t\treturn NewSet(vs, dup...)\n\t\t\t},\n\t\t},\n\t\telems: elems,\n\t}\n}\n\nvar mutex sync.Mutex\n\nfunc printBadCollections(expected, actual ValueSlice) {\n\tmutex.Lock()\n\tdefer mutex.Unlock()\n\tfmt.Println(\"expected:\", expected)\n\tfmt.Println(\"actual:\", actual)\n}\n\nfunc (suite *setTestSuite) createStreamingSet(vs *ValueStore) Set {\n\tvChan := make(chan Value)\n\tsetChan := NewStreamingSet(vs, vChan)\n\tfor _, entry := range suite.elems {\n\t\tvChan <- entry\n\t}\n\tclose(vChan)\n\treturn <-setChan\n}\n\nfunc (suite *setTestSuite) TestStreamingSet() {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\ts := suite.createStreamingSet(vs)\n\tsuite.True(suite.validate(s))\n}\n\nfunc (suite *setTestSuite) TestStreamingSetOrder() {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\telems := make(testSet, len(suite.elems))\n\tcopy(elems, suite.elems)\n\telems[0], elems[1] = elems[1], elems[0]\n\tvChan := make(chan Value, len(elems))\n\tfor _, e := range elems {\n\t\tvChan <- e\n\t}\n\tclose(vChan)\n\n\treadInput := func(vrw ValueReadWriter, vChan <-chan Value, outChan chan<- Set) {\n\t\treadSetInput(vrw, vChan, outChan)\n\t}\n\n\ttestFunc := func() {\n\t\toutChan := newStreamingSet(vs, vChan, readInput)\n\t\t<-outChan\n\t}\n\tsuite.Panics(testFunc)\n}\n\nfunc (suite *setTestSuite) TestStreamingSet2() {\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\twg := sync.WaitGroup{}\n\twg.Add(2)\n\tvar s1, s2 Set\n\tgo func() {\n\t\ts1 = suite.createStreamingSet(vs)\n\t\twg.Done()\n\t}()\n\tgo func() {\n\t\ts2 = suite.createStreamingSet(vs)\n\t\twg.Done()\n\t}()\n\twg.Wait()\n\tsuite.True(suite.validate(s1))\n\tsuite.True(suite.validate(s2))\n}\n\nfunc TestSetSuite4K(t *testing.T) {\n\tsuite.Run(t, newSetTestSuite(12, 8, 2, 2, newNumber))\n}\n\nfunc TestSetSuite4KStructs(t *testing.T) {\n\tsuite.Run(t, newSetTestSuite(12, 9, 2, 2, newNumberStruct))\n}\n\nfunc getTestNativeOrderSet(scale int, vrw ValueReadWriter) testSet {\n\treturn newRandomTestSet(64*scale, newNumber)\n}\n\nfunc getTestRefValueOrderSet(scale int, vrw ValueReadWriter) testSet {\n\treturn newRandomTestSet(64*scale, newNumber)\n}\n\nfunc getTestRefToNativeOrderSet(scale int, vrw ValueReadWriter) testSet {\n\treturn newRandomTestSet(64*scale, func(v int) Value {\n\t\treturn vrw.WriteValue(Number(v))\n\t})\n}\n\nfunc getTestRefToValueOrderSet(scale int, vrw ValueReadWriter) testSet {\n\treturn newRandomTestSet(64*scale, func(v int) Value {\n\t\treturn vrw.WriteValue(NewSet(vrw, Number(v)))\n\t})\n}\n\nfunc accumulateSetDiffChanges(s1, s2 Set) (added []Value, removed []Value) {\n\tchanges := make(chan ValueChanged)\n\tgo func() {\n\t\ts1.Diff(s2, changes, nil)\n\t\tclose(changes)\n\t}()\n\tfor change := range changes {\n\t\tif change.ChangeType == DiffChangeAdded {\n\t\t\tadded = append(added, change.Key)\n\t\t} else if change.ChangeType == DiffChangeRemoved {\n\t\t\tremoved = append(removed, change.Key)\n\t\t}\n\t}\n\treturn\n}\n\nfunc diffSetTest(assert *assert.Assertions, s1 Set, s2 Set, numAddsExpected int, numRemovesExpected int) (added []Value, removed []Value) {\n\tadded, removed = accumulateSetDiffChanges(s1, s2)\n\tassert.Equal(numAddsExpected, len(added), \"num added is not as expected\")\n\tassert.Equal(numRemovesExpected, len(removed), \"num removed is not as expected\")\n\n\tts1 := newTestSetFromSet(s1)\n\tts2 := newTestSetFromSet(s2)\n\ttsAdded, tsRemoved := ts1.Diff(ts2)\n\tassert.Equal(numAddsExpected, len(tsAdded), \"num added is not as expected\")\n\tassert.Equal(numRemovesExpected, len(tsRemoved), \"num removed is not as expected\")\n\n\tassert.Equal(added, tsAdded, \"set added != tsSet added\")\n\tassert.Equal(removed, tsRemoved, \"set removed != tsSet removed\")\n\treturn\n}\n\nfunc TestNewSet(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs)\n\tassert.True(MakeSetType(MakeUnionType()).Equals(TypeOf(s)))\n\tassert.Equal(uint64(0), s.Len())\n\n\ts = NewSet(vs, Number(0))\n\tassert.True(MakeSetType(NumberType).Equals(TypeOf(s)))\n\n\ts = NewSet(vs)\n\tassert.IsType(MakeSetType(NumberType), TypeOf(s))\n\n\ts2 := s.Edit().Remove(Number(1)).Set()\n\tassert.IsType(TypeOf(s), TypeOf(s2))\n}\n\nfunc TestSetLen(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\ts0 := NewSet(vs)\n\tassert.Equal(uint64(0), s0.Len())\n\ts1 := NewSet(vs, Bool(true), Number(1), String(\"hi\"))\n\tassert.Equal(uint64(3), s1.Len())\n\tdiffSetTest(assert, s0, s1, 0, 3)\n\tdiffSetTest(assert, s1, s0, 3, 0)\n\n\ts2 := s1.Edit().Insert(Bool(false)).Set()\n\tassert.Equal(uint64(4), s2.Len())\n\tdiffSetTest(assert, s0, s2, 0, 4)\n\tdiffSetTest(assert, s2, s0, 4, 0)\n\tdiffSetTest(assert, s1, s2, 0, 1)\n\tdiffSetTest(assert, s2, s1, 1, 0)\n\n\ts3 := s2.Edit().Remove(Bool(true)).Set()\n\tassert.Equal(uint64(3), s3.Len())\n\tdiffSetTest(assert, s2, s3, 1, 0)\n\tdiffSetTest(assert, s3, s2, 0, 1)\n}\n\nfunc TestSetEmpty(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs)\n\tassert.True(s.Empty())\n\tassert.Equal(uint64(0), s.Len())\n}\n\nfunc TestSetEmptyInsert(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs)\n\tassert.True(s.Empty())\n\ts = s.Edit().Insert(Bool(false)).Set()\n\tassert.False(s.Empty())\n\tassert.Equal(uint64(1), s.Len())\n}\n\nfunc TestSetEmptyInsertRemove(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs)\n\tassert.True(s.Empty())\n\ts = s.Edit().Insert(Bool(false)).Set()\n\tassert.False(s.Empty())\n\tassert.Equal(uint64(1), s.Len())\n\ts = s.Edit().Remove(Bool(false)).Set()\n\tassert.True(s.Empty())\n\tassert.Equal(uint64(0), s.Len())\n}\n\n// BUG 98\nfunc TestSetDuplicateInsert(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts1 := NewSet(vs, Bool(true), Number(42), Number(42))\n\tassert.Equal(uint64(2), s1.Len())\n}\n\nfunc TestSetUniqueKeysString(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts1 := NewSet(vs, String(\"hello\"), String(\"world\"), String(\"hello\"))\n\tassert.Equal(uint64(2), s1.Len())\n\tassert.True(s1.Has(String(\"hello\")))\n\tassert.True(s1.Has(String(\"world\")))\n\tassert.False(s1.Has(String(\"foo\")))\n}\n\nfunc TestSetUniqueKeysNumber(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts1 := NewSet(vs, Number(4), Number(1), Number(0), Number(0), Number(1), Number(3))\n\tassert.Equal(uint64(4), s1.Len())\n\tassert.True(s1.Has(Number(4)))\n\tassert.True(s1.Has(Number(1)))\n\tassert.True(s1.Has(Number(0)))\n\tassert.True(s1.Has(Number(3)))\n\tassert.False(s1.Has(Number(2)))\n}\n\nfunc TestSetHas(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts1 := NewSet(vs, Bool(true), Number(1), String(\"hi\"))\n\tassert.True(s1.Has(Bool(true)))\n\tassert.False(s1.Has(Bool(false)))\n\tassert.True(s1.Has(Number(1)))\n\tassert.False(s1.Has(Number(0)))\n\tassert.True(s1.Has(String(\"hi\")))\n\tassert.False(s1.Has(String(\"ho\")))\n\n\ts2 := s1.Edit().Insert(Bool(false)).Set()\n\tassert.True(s2.Has(Bool(false)))\n\tassert.True(s2.Has(Bool(true)))\n\n\tassert.True(s1.Has(Bool(true)))\n\tassert.False(s1.Has(Bool(false)))\n}\n\nfunc TestSetHas2(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\n\tdoTest := func(toTestSet toTestSetFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\tts := toTestSet(scale, vrw)\n\t\tset := ts.toSet(vrw)\n\t\tset2 := vrw.ReadValue(vrw.WriteValue(set).TargetHash()).(Set)\n\t\tfor _, v := range ts {\n\t\t\tassert.True(set.Has(v))\n\t\t\tassert.True(set2.Has(v))\n\t\t}\n\t\tdiffSetTest(assert, set, set2, 0, 0)\n\t}\n\n\tdoTest(getTestNativeOrderSet, 16)\n\tdoTest(getTestRefValueOrderSet, 2)\n\tdoTest(getTestRefToNativeOrderSet, 2)\n\tdoTest(getTestRefToValueOrderSet, 2)\n}\n\nfunc validateSetInsertion(t *testing.T, vrw ValueReadWriter, values ValueSlice) {\n\ts := NewSet(vrw)\n\tfor i, v := range values {\n\t\ts = s.Edit().Insert(v).Set()\n\t\tvalidateSet(t, vrw, s, values[0:i+1])\n\t}\n}\n\nfunc TestSetValidateInsertAscending(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvs := newTestValueStore()\n\n\tvalidateSetInsertion(t, vs, generateNumbersAsValues(300))\n}\n\nfunc TestSetInsert(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs)\n\tv1 := Bool(false)\n\tv2 := Bool(true)\n\tv3 := Number(0)\n\n\tassert.False(s.Has(v1))\n\ts = s.Edit().Insert(v1).Set()\n\tassert.True(s.Has(v1))\n\ts = s.Edit().Insert(v2).Set()\n\tassert.True(s.Has(v1))\n\tassert.True(s.Has(v2))\n\ts2 := s.Edit().Insert(v3).Set()\n\tassert.True(s.Has(v1))\n\tassert.True(s.Has(v2))\n\tassert.False(s.Has(v3))\n\tassert.True(s2.Has(v1))\n\tassert.True(s2.Has(v2))\n\tassert.True(s2.Has(v3))\n}\n\nfunc TestSetInsert2(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\n\tdoTest := func(incr, offset int, toTestSet toTestSetFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\tts := toTestSet(scale, vrw)\n\t\texpected := ts.toSet(vrw)\n\t\trun := func(from, to int) {\n\t\t\tactual := ts.Remove(from, to).toSet(vrw).Edit().Insert(ts[from:to]...).Set()\n\t\t\tassert.Equal(expected.Len(), actual.Len())\n\t\t\tassert.True(expected.Equals(actual))\n\t\t\tdiffSetTest(assert, expected, actual, 0, 0)\n\t\t}\n\t\tfor i := 0; i < len(ts)-offset; i += incr {\n\t\t\trun(i, i+offset)\n\t\t}\n\t\trun(len(ts)-offset, len(ts))\n\t}\n\n\tdoTest(18, 3, getTestNativeOrderSet, 9)\n\tdoTest(64, 1, getTestNativeOrderSet, 32)\n\tdoTest(32, 1, getTestRefValueOrderSet, 4)\n\tdoTest(32, 1, getTestRefToNativeOrderSet, 4)\n\tdoTest(32, 1, getTestRefToValueOrderSet, 4)\n}\n\nfunc TestSetInsertExistingValue(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvs := newTestValueStore()\n\n\tts := getTestNativeOrderSet(2, vs)\n\toriginal := ts.toSet(vs)\n\tactual := original.Edit().Insert(ts[0]).Set()\n\n\tassert.Equal(original.Len(), actual.Len())\n\tassert.True(original.Equals(actual))\n}\n\nfunc TestSetRemove(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tv1 := Bool(false)\n\tv2 := Bool(true)\n\tv3 := Number(0)\n\ts := NewSet(vs, v1, v2, v3)\n\tassert.True(s.Has(v1))\n\tassert.True(s.Has(v2))\n\tassert.True(s.Has(v3))\n\ts = s.Edit().Remove(v1).Set()\n\tassert.False(s.Has(v1))\n\tassert.True(s.Has(v2))\n\tassert.True(s.Has(v3))\n\ts2 := s.Edit().Remove(v2).Set()\n\tassert.False(s.Has(v1))\n\tassert.True(s.Has(v2))\n\tassert.True(s.Has(v3))\n\tassert.False(s2.Has(v1))\n\tassert.False(s2.Has(v2))\n\tassert.True(s2.Has(v3))\n}\n\nfunc TestSetRemove2(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\n\tdoTest := func(incr, offset int, toTestSet toTestSetFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\tts := toTestSet(scale, vrw)\n\t\twhole := ts.toSet(vrw)\n\t\trun := func(from, to int) {\n\t\t\texpected := ts.Remove(from, to).toSet(vrw)\n\t\t\tactual := whole.Edit().Remove(ts[from:to]...).Set()\n\t\t\tassert.Equal(expected.Len(), actual.Len())\n\t\t\tassert.True(expected.Equals(actual))\n\t\t\tdiffSetTest(assert, expected, actual, 0, 0)\n\t\t}\n\t\tfor i := 0; i < len(ts)-offset; i += incr {\n\t\t\trun(i, i+offset)\n\t\t}\n\t\trun(len(ts)-offset, len(ts))\n\t}\n\n\tdoTest(18, 3, getTestNativeOrderSet, 9)\n\tdoTest(64, 1, getTestNativeOrderSet, 32)\n\tdoTest(32, 1, getTestRefValueOrderSet, 4)\n\tdoTest(32, 1, getTestRefToNativeOrderSet, 4)\n\tdoTest(32, 1, getTestRefToValueOrderSet, 4)\n}\n\nfunc TestSetRemoveNonexistentValue(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tts := getTestNativeOrderSet(2, vs)\n\toriginal := ts.toSet(vs)\n\tactual := original.Edit().Remove(Number(-1)).Set() // rand.Int63 returns non-negative values.\n\n\tassert.Equal(original.Len(), actual.Len())\n\tassert.True(original.Equals(actual))\n}\n\nfunc TestSetFirst(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs)\n\tassert.Nil(s.First())\n\ts = s.Edit().Insert(Number(1)).Set()\n\tassert.NotNil(s.First())\n\ts = s.Edit().Insert(Number(2)).Set()\n\tassert.NotNil(s.First())\n\ts2 := s.Edit().Remove(Number(1)).Set()\n\tassert.NotNil(s2.First())\n\ts2 = s2.Edit().Remove(Number(2)).Set()\n\tassert.Nil(s2.First())\n}\n\nfunc TestSetOfStruct(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\telems := []Value{}\n\tfor i := 0; i < 200; i++ {\n\t\telems = append(elems, NewStruct(\"S1\", StructData{\"o\": Number(i)}))\n\t}\n\n\ts := NewSet(vs, elems...)\n\tfor i := 0; i < 200; i++ {\n\t\tassert.True(s.Has(elems[i]))\n\t}\n}\n\nfunc TestSetIter(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs, Number(0), Number(1), Number(2), Number(3), Number(4))\n\tacc := NewSet(vs)\n\ts.Iter(func(v Value) bool {\n\t\t_, ok := v.(Number)\n\t\tassert.True(ok)\n\t\tacc = acc.Edit().Insert(v).Set()\n\t\treturn false\n\t})\n\tassert.True(s.Equals(acc))\n\n\tacc = NewSet(vs)\n\ts.Iter(func(v Value) bool {\n\t\treturn true\n\t})\n\tassert.True(acc.Empty())\n}\n\nfunc TestSetIter2(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tdoTest := func(toTestSet toTestSetFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\tts := toTestSet(scale, vrw)\n\t\tset := ts.toSet(vrw)\n\t\tsort.Sort(ValueSlice(ts))\n\t\tidx := uint64(0)\n\t\tendAt := uint64(64)\n\n\t\tset.Iter(func(v Value) (done bool) {\n\t\t\tassert.True(ts[idx].Equals(v))\n\t\t\tif idx == endAt {\n\t\t\t\tdone = true\n\t\t\t}\n\t\t\tidx++\n\t\t\treturn\n\t\t})\n\n\t\tassert.Equal(endAt, idx-1)\n\t}\n\n\tdoTest(getTestNativeOrderSet, 16)\n\tdoTest(getTestRefValueOrderSet, 2)\n\tdoTest(getTestRefToNativeOrderSet, 2)\n\tdoTest(getTestRefToValueOrderSet, 2)\n}\n\nfunc TestSetIterAll(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs, Number(0), Number(1), Number(2), Number(3), Number(4))\n\tacc := NewSet(vs)\n\ts.IterAll(func(v Value) {\n\t\t_, ok := v.(Number)\n\t\tassert.True(ok)\n\t\tacc = acc.Edit().Insert(v).Set()\n\t})\n\tassert.True(s.Equals(acc))\n}\n\nfunc TestSetIterAll2(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tdoTest := func(toTestSet toTestSetFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\tts := toTestSet(scale, vrw)\n\t\tset := ts.toSet(vrw)\n\t\tsort.Sort(ValueSlice(ts))\n\t\tidx := uint64(0)\n\n\t\tset.IterAll(func(v Value) {\n\t\t\tassert.True(ts[idx].Equals(v))\n\t\t\tidx++\n\t\t})\n\t}\n\n\tdoTest(getTestNativeOrderSet, 16)\n\tdoTest(getTestRefValueOrderSet, 2)\n\tdoTest(getTestRefToNativeOrderSet, 2)\n\tdoTest(getTestRefToValueOrderSet, 2)\n}\n\nfunc testSetOrder(assert *assert.Assertions, vrw ValueReadWriter, valueType *Type, value []Value, expectOrdering []Value) {\n\tm := NewSet(vrw, value...)\n\ti := 0\n\tm.IterAll(func(value Value) {\n\t\tassert.Equal(expectOrdering[i].Hash().String(), value.Hash().String())\n\t\ti++\n\t})\n}\n\nfunc TestSetOrdering(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ttestSetOrder(assert, vs,\n\t\tStringType,\n\t\t[]Value{\n\t\t\tString(\"a\"),\n\t\t\tString(\"z\"),\n\t\t\tString(\"b\"),\n\t\t\tString(\"y\"),\n\t\t\tString(\"c\"),\n\t\t\tString(\"x\"),\n\t\t},\n\t\t[]Value{\n\t\t\tString(\"a\"),\n\t\t\tString(\"b\"),\n\t\t\tString(\"c\"),\n\t\t\tString(\"x\"),\n\t\t\tString(\"y\"),\n\t\t\tString(\"z\"),\n\t\t},\n\t)\n\n\ttestSetOrder(assert, vs,\n\t\tNumberType,\n\t\t[]Value{\n\t\t\tNumber(0),\n\t\t\tNumber(1000),\n\t\t\tNumber(1),\n\t\t\tNumber(100),\n\t\t\tNumber(2),\n\t\t\tNumber(10),\n\t\t},\n\t\t[]Value{\n\t\t\tNumber(0),\n\t\t\tNumber(1),\n\t\t\tNumber(2),\n\t\t\tNumber(10),\n\t\t\tNumber(100),\n\t\t\tNumber(1000),\n\t\t},\n\t)\n\n\ttestSetOrder(assert, vs,\n\t\tNumberType,\n\t\t[]Value{\n\t\t\tNumber(0),\n\t\t\tNumber(-30),\n\t\t\tNumber(25),\n\t\t\tNumber(1002),\n\t\t\tNumber(-5050),\n\t\t\tNumber(23),\n\t\t},\n\t\t[]Value{\n\t\t\tNumber(-5050),\n\t\t\tNumber(-30),\n\t\t\tNumber(0),\n\t\t\tNumber(23),\n\t\t\tNumber(25),\n\t\t\tNumber(1002),\n\t\t},\n\t)\n\n\ttestSetOrder(assert, vs,\n\t\tNumberType,\n\t\t[]Value{\n\t\t\tNumber(0.0001),\n\t\t\tNumber(0.000001),\n\t\t\tNumber(1),\n\t\t\tNumber(25.01e3),\n\t\t\tNumber(-32.231123e5),\n\t\t\tNumber(23),\n\t\t},\n\t\t[]Value{\n\t\t\tNumber(-32.231123e5),\n\t\t\tNumber(0.000001),\n\t\t\tNumber(0.0001),\n\t\t\tNumber(1),\n\t\t\tNumber(23),\n\t\t\tNumber(25.01e3),\n\t\t},\n\t)\n\n\ttestSetOrder(assert, vs,\n\t\tValueType,\n\t\t[]Value{\n\t\t\tString(\"a\"),\n\t\t\tString(\"z\"),\n\t\t\tString(\"b\"),\n\t\t\tString(\"y\"),\n\t\t\tString(\"c\"),\n\t\t\tString(\"x\"),\n\t\t},\n\t\t// Ordered by value\n\t\t[]Value{\n\t\t\tString(\"a\"),\n\t\t\tString(\"b\"),\n\t\t\tString(\"c\"),\n\t\t\tString(\"x\"),\n\t\t\tString(\"y\"),\n\t\t\tString(\"z\"),\n\t\t},\n\t)\n\n\ttestSetOrder(assert, vs,\n\t\tBoolType,\n\t\t[]Value{\n\t\t\tBool(true),\n\t\t\tBool(false),\n\t\t},\n\t\t// Ordered by value\n\t\t[]Value{\n\t\t\tBool(false),\n\t\t\tBool(true),\n\t\t},\n\t)\n}\n\nfunc TestSetType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\n\ts := NewSet(vs)\n\tassert.True(TypeOf(s).Equals(MakeSetType(MakeUnionType())))\n\n\ts = NewSet(vs, Number(0))\n\tassert.True(TypeOf(s).Equals(MakeSetType(NumberType)))\n\n\ts2 := s.Edit().Remove(Number(1)).Set()\n\tassert.True(TypeOf(s2).Equals(MakeSetType(NumberType)))\n\n\ts2 = s.Edit().Insert(Number(0), Number(1)).Set()\n\tassert.True(TypeOf(s).Equals(TypeOf(s2)))\n\n\ts3 := s.Edit().Insert(Bool(true)).Set()\n\tassert.True(TypeOf(s3).Equals(MakeSetType(MakeUnionType(BoolType, NumberType))))\n\ts4 := s.Edit().Insert(Number(3), Bool(true)).Set()\n\tassert.True(TypeOf(s4).Equals(MakeSetType(MakeUnionType(BoolType, NumberType))))\n}\n\nfunc TestSetChunks(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tl1 := NewSet(vs, Number(0))\n\tc1 := getChunks(l1)\n\tassert.Len(c1, 0)\n\n\tl2 := NewSet(vs, NewRef(Number(0)))\n\tc2 := getChunks(l2)\n\tassert.Len(c2, 1)\n}\n\nfunc TestSetChunks2(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tdoTest := func(toTestSet toTestSetFunc, scale int) {\n\t\tvrw := newTestValueStore()\n\t\tts := toTestSet(scale, vrw)\n\t\tset := ts.toSet(vrw)\n\t\tset2chunks := getChunks(vrw.ReadValue(vrw.WriteValue(set).TargetHash()))\n\t\tfor i, r := range getChunks(set) {\n\t\t\tassert.True(TypeOf(r).Equals(TypeOf(set2chunks[i])), \"%s != %s\", TypeOf(r).Describe(), TypeOf(set2chunks[i]).Describe())\n\t\t}\n\t}\n\n\tdoTest(getTestNativeOrderSet, 16)\n\tdoTest(getTestRefValueOrderSet, 2)\n\tdoTest(getTestRefToNativeOrderSet, 2)\n\tdoTest(getTestRefToValueOrderSet, 2)\n}\n\nfunc TestSetFirstNNumbers(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tnums := generateNumbersAsValues(testSetSize)\n\ts := NewSet(vs, nums...)\n\tassert.Equal(deriveCollectionHeight(s), getRefHeightOfCollection(s))\n}\n\nfunc TestSetRefOfStructFirstNNumbers(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"Skipping test in short mode.\")\n\t}\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tnums := generateNumbersAsRefOfStructs(vs, testSetSize)\n\ts := NewSet(vs, nums...)\n\t// height + 1 because the leaves are Ref values (with height 1).\n\tassert.Equal(deriveCollectionHeight(s)+1, getRefHeightOfCollection(s))\n}\n\nfunc TestSetModifyAfterRead(t *testing.T) {\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\tset := getTestNativeOrderSet(2, vs).toSet(vs)\n\t// Drop chunk values.\n\tset = vs.ReadValue(vs.WriteValue(set).TargetHash()).(Set)\n\t// Modify/query. Once upon a time this would crash.\n\tfst := set.First()\n\tset = set.Edit().Remove(fst).Set()\n\tassert.False(set.Has(fst))\n\tassert.True(set.Has(set.First()))\n\tset = set.Edit().Insert(fst).Set()\n\tassert.True(set.Has(fst))\n}\n\nfunc TestSetTypeAfterMutations(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\ttest := func(n int, c interface{}) {\n\t\tvs := newTestValueStore()\n\t\tvalues := generateNumbersAsValues(n)\n\n\t\ts := NewSet(vs, values...)\n\t\tassert.Equal(s.Len(), uint64(n))\n\t\tassert.IsType(c, s.asSequence())\n\t\tassert.True(TypeOf(s).Equals(MakeSetType(NumberType)))\n\n\t\ts = s.Edit().Insert(String(\"a\")).Set()\n\t\tassert.Equal(s.Len(), uint64(n+1))\n\t\tassert.IsType(c, s.asSequence())\n\t\tassert.True(TypeOf(s).Equals(MakeSetType(MakeUnionType(NumberType, StringType))))\n\n\t\ts = s.Edit().Remove(String(\"a\")).Set()\n\t\tassert.Equal(s.Len(), uint64(n))\n\t\tassert.IsType(c, s.asSequence())\n\t\tassert.True(TypeOf(s).Equals(MakeSetType(NumberType)))\n\t}\n\n\ttest(10, setLeafSequence{})\n\ttest(2000, metaSequence{})\n}\n\nfunc TestChunkedSetWithValuesOfEveryType(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvals := []Value{\n\t\t// Values\n\t\tBool(true),\n\t\tNumber(0),\n\t\tString(\"hello\"),\n\t\tNewBlob(vs, bytes.NewBufferString(\"buf\")),\n\t\tNewSet(vs, Bool(true)),\n\t\tNewList(vs, Bool(true)),\n\t\tNewMap(vs, Bool(true), Number(0)),\n\t\tNewStruct(\"\", StructData{\"field\": Bool(true)}),\n\t\t// Refs of values\n\t\tNewRef(Bool(true)),\n\t\tNewRef(Number(0)),\n\t\tNewRef(String(\"hello\")),\n\t\tNewRef(NewBlob(vs, bytes.NewBufferString(\"buf\"))),\n\t\tNewRef(NewSet(vs, Bool(true))),\n\t\tNewRef(NewList(vs, Bool(true))),\n\t\tNewRef(NewMap(vs, Bool(true), Number(0))),\n\t\tNewRef(NewStruct(\"\", StructData{\"field\": Bool(true)})),\n\t}\n\n\ts := NewSet(vs, vals...)\n\tfor i := 1; s.asSequence().isLeaf(); i++ {\n\t\tv := Number(i)\n\t\tvals = append(vals, v)\n\t\ts = s.Edit().Insert(v).Set()\n\t}\n\n\tassert.Equal(len(vals), int(s.Len()))\n\tassert.True(bool(s.First().(Bool)))\n\n\tfor _, v := range vals {\n\t\tassert.True(s.Has(v))\n\t}\n\n\tfor len(vals) > 0 {\n\t\tv := vals[0]\n\t\tvals = vals[1:]\n\t\ts = s.Edit().Remove(v).Set()\n\t\tassert.False(s.Has(v))\n\t\tassert.Equal(len(vals), int(s.Len()))\n\t}\n}\n\nfunc TestSetRemoveLastWhenNotLoaded(t *testing.T) {\n\tassert := assert.New(t)\n\n\tsmallTestChunks()\n\tdefer normalProductionChunks()\n\n\tvs := newTestValueStore()\n\treload := func(s Set) Set {\n\t\treturn vs.ReadValue(vs.WriteValue(s).TargetHash()).(Set)\n\t}\n\n\tts := getTestNativeOrderSet(8, vs)\n\tns := ts.toSet(vs)\n\n\tfor len(ts) > 0 {\n\t\tlast := ts[len(ts)-1]\n\t\tts = ts[:len(ts)-1]\n\t\tns = reload(ns.Edit().Remove(last).Set())\n\t\tassert.True(ts.toSet(vs).Equals(ns))\n\t}\n}\n\nfunc TestSetAt(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tvalues := []Value{Bool(false), Number(42), String(\"a\"), String(\"b\"), String(\"c\")}\n\ts := NewSet(vs, values...)\n\n\tfor i, v := range values {\n\t\tassert.Equal(v, s.At(uint64(i)))\n\t}\n\n\tassert.Panics(func() {\n\t\ts.At(42)\n\t})\n}\n\nfunc TestSetWithStructShouldHaveOptionalFields(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tlist := NewSet(vs,\n\t\tNewStruct(\"Foo\", StructData{\n\t\t\t\"a\": Number(1),\n\t\t}),\n\t\tNewStruct(\"Foo\", StructData{\n\t\t\t\"a\": Number(2),\n\t\t\t\"b\": String(\"bar\"),\n\t\t}),\n\t)\n\tassert.True(\n\t\tMakeSetType(MakeStructType(\"Foo\",\n\t\t\tStructField{\"a\", NumberType, false},\n\t\t\tStructField{\"b\", StringType, true},\n\t\t),\n\t\t).Equals(TypeOf(list)))\n}\n\nfunc TestSetWithNil(t *testing.T) {\n\tvs := newTestValueStore()\n\n\tassert.Panics(t, func() {\n\t\tNewSet(vs, nil)\n\t})\n\tassert.Panics(t, func() {\n\t\tNewSet(vs, Number(42), nil)\n\t})\n}\n" }, { "path": "go/types/simplify.go", "content": "package types\n\nimport (\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// simplifyType returns a type that is a super type of the input type but is\n// much smaller and less complex than a straight union of all those types would\n// be.\n//\n// The resulting type is guaranteed to:\n// a. be a super type of the input type\n// b. have all unions flattened (no union inside a union)\n// c. have all unions folded, which means the union\n// 1. have at most one element each of kind Ref, Set, List, and Map\n// 2. have at most one struct element with a given name\n// e. all named unions are pointing at the same simplified struct, which means\n// that all named unions with the same name form cycles.\n// f. all cycle type that can be resolved have been resolved.\n// g. all types reachable from it also fulfill b-f\n//\n// The union folding is created roughly as follows:\n//\n// - The input types are deduplicated\n// - Any unions in the input set are \"flattened\" into the input set\n// - The inputs are grouped into categories:\n// - ref\n// - list\n// - set\n// - map\n// - struct, by name (each unique struct name will have its own group)\n// - The ref, set, and list groups are collapsed like so:\n// {Ref,Ref,...} -> Ref\n// - The map group is collapsed like so:\n// {Map|Map...} -> Map\n// - Each struct group is collapsed like so:\n// {struct{foo:number,bar:string}, struct{bar:blob, baz:bool}} ->\n// struct{foo?:number,bar:string|blob,baz?:bool}\n//\n// All the above rules are applied recursively.\nfunc simplifyType(t *Type, intersectStructs bool) *Type {\n\tif t.Desc.isSimplifiedForSure() {\n\t\treturn t\n\t}\n\n\t// 1. Clone tree because we are going to mutate it\n\t// 1.1 Replace all named structs and cycle types with a single `struct Name {}`\n\t// 2. When a union type is found change its elemTypes as needed\n\t// 2.1 Merge unnamed structs\n\t// 3. Update the fields of all named structs\n\n\tnamedStructs := map[string]structInfo{}\n\n\tclone := cloneTypeTreeAndReplaceNamedStructs(t, namedStructs)\n\tfolded := foldUnions(clone, typeset{}, intersectStructs)\n\n\tfor name, info := range namedStructs {\n\t\tif len(info.sources) == 0 {\n\t\t\td.PanicIfTrue(name == \"\")\n\t\t\tinfo.instance.Desc = CycleDesc(name)\n\t\t} else {\n\t\t\tfields := foldStructTypesFieldsOnly(name, info.sources, typeset{}, intersectStructs)\n\t\t\tinfo.instance.Desc = StructDesc{name, fields}\n\t\t}\n\t}\n\n\treturn folded\n}\n\n// typeset is a helper that aggregates the unique set of input types for this algorithm, flattening\n// any unions recursively.\ntype typeset map[*Type]struct{}\n\nfunc (ts typeset) add(t *Type) {\n\tswitch t.TargetKind() {\n\tcase UnionKind:\n\t\tfor _, et := range t.Desc.(CompoundDesc).ElemTypes {\n\t\t\tts.add(et)\n\t\t}\n\tdefault:\n\t\tts[t] = struct{}{}\n\t}\n}\n\nfunc (ts typeset) has(t *Type) bool {\n\t_, ok := ts[t]\n\treturn ok\n}\n\ntype structInfo struct {\n\tinstance *Type\n\tsources typeset\n}\n\nfunc cloneTypeTreeAndReplaceNamedStructs(t *Type, namedStructs map[string]structInfo) *Type {\n\tgetNamedStruct := func(name string, t *Type) *Type {\n\t\trecord := namedStructs[name]\n\t\tif t.TargetKind() == StructKind {\n\t\t\trecord.sources.add(t)\n\t\t}\n\t\treturn record.instance\n\t}\n\n\tensureInstance := func(name string) {\n\t\tif _, ok := namedStructs[name]; !ok {\n\t\t\tinstance := newType(StructDesc{Name: name})\n\t\t\tnamedStructs[name] = structInfo{instance, typeset{}}\n\t\t}\n\t}\n\n\tseenStructs := typeset{}\n\tvar rec func(t *Type) *Type\n\trec = func(t *Type) *Type {\n\t\tkind := t.TargetKind()\n\t\tswitch kind {\n\t\tcase BoolKind, NumberKind, StringKind, BlobKind, ValueKind, TypeKind:\n\t\t\treturn t\n\t\tcase ListKind, MapKind, RefKind, SetKind, UnionKind:\n\t\t\telemTypes := make(typeSlice, len(t.Desc.(CompoundDesc).ElemTypes))\n\t\t\tfor i, et := range t.Desc.(CompoundDesc).ElemTypes {\n\t\t\t\telemTypes[i] = rec(et)\n\t\t\t}\n\t\t\treturn newType(CompoundDesc{kind, elemTypes})\n\t\tcase StructKind:\n\t\t\tdesc := t.Desc.(StructDesc)\n\t\t\tname := desc.Name\n\n\t\t\tif name != \"\" {\n\t\t\t\tensureInstance(name)\n\t\t\t\tif seenStructs.has(t) {\n\t\t\t\t\treturn namedStructs[name].instance\n\t\t\t\t}\n\t\t\t} else if seenStructs.has(t) {\n\t\t\t\t// It is OK to use the same unnamed struct type in multiple places.\n\t\t\t\t// Do not clone it again.\n\t\t\t\treturn t\n\t\t\t}\n\t\t\tseenStructs.add(t)\n\n\t\t\tfields := make(structTypeFields, len(desc.fields))\n\t\t\tfor i, f := range desc.fields {\n\t\t\t\tfields[i] = StructField{f.Name, rec(f.Type), f.Optional}\n\t\t\t}\n\t\t\tnewStruct := newType(StructDesc{name, fields})\n\t\t\tif name == \"\" {\n\t\t\t\treturn newStruct\n\t\t\t}\n\n\t\t\treturn getNamedStruct(name, newStruct)\n\n\t\tcase CycleKind:\n\t\t\tname := string(t.Desc.(CycleDesc))\n\t\t\td.PanicIfTrue(name == \"\")\n\t\t\tensureInstance(name)\n\t\t\treturn getNamedStruct(name, t)\n\n\t\tdefault:\n\t\t\tpanic(\"Unknown noms kind\")\n\t\t}\n\t}\n\n\treturn rec(t)\n}\n\nfunc foldUnions(t *Type, seenStructs typeset, intersectStructs bool) *Type {\n\tkind := t.TargetKind()\n\tswitch kind {\n\tcase BoolKind, NumberKind, StringKind, BlobKind, ValueKind, TypeKind, CycleKind:\n\t\tbreak\n\n\tcase ListKind, MapKind, RefKind, SetKind:\n\t\telemTypes := t.Desc.(CompoundDesc).ElemTypes\n\t\tfor i, et := range elemTypes {\n\t\t\telemTypes[i] = foldUnions(et, seenStructs, intersectStructs)\n\t\t}\n\n\tcase StructKind:\n\t\tif seenStructs.has(t) {\n\t\t\treturn t\n\t\t}\n\t\tseenStructs.add(t)\n\t\tfields := t.Desc.(StructDesc).fields\n\t\tfor i, f := range fields {\n\t\t\tfields[i].Type = foldUnions(f.Type, seenStructs, intersectStructs)\n\t\t}\n\n\tcase UnionKind:\n\t\telemTypes := t.Desc.(CompoundDesc).ElemTypes\n\t\tif len(elemTypes) == 0 {\n\t\t\tbreak\n\t\t}\n\t\tts := make(typeset, len(elemTypes))\n\t\tfor _, t := range elemTypes {\n\t\t\tts.add(t)\n\t\t}\n\t\tif len(ts) == 0 {\n\t\t\tt.Desc = CompoundDesc{UnionKind, nil}\n\t\t\treturn t\n\t\t}\n\t\treturn foldUnionImpl(ts, seenStructs, intersectStructs)\n\n\tdefault:\n\t\tpanic(\"Unknown noms kind\")\n\t}\n\treturn t\n}\n\nfunc foldUnionImpl(ts typeset, seenStructs typeset, intersectStructs bool) *Type {\n\ttype how struct {\n\t\tk NomsKind\n\t\tn string\n\t}\n\tout := make(typeSlice, 0, len(ts))\n\tgroups := map[how]typeset{}\n\tfor t := range ts {\n\t\tvar h how\n\t\tswitch t.TargetKind() {\n\t\tcase RefKind, SetKind, ListKind, MapKind:\n\t\t\th = how{k: t.TargetKind()}\n\t\tcase StructKind:\n\t\t\th = how{k: t.TargetKind(), n: t.Desc.(StructDesc).Name}\n\t\tdefault:\n\t\t\tout = append(out, t)\n\t\t\tcontinue\n\t\t}\n\t\tg := groups[h]\n\t\tif g == nil {\n\t\t\tg = typeset{}\n\t\t\tgroups[h] = g\n\t\t}\n\t\tg.add(t)\n\t}\n\n\tfor h, ts := range groups {\n\t\tif len(ts) == 1 {\n\t\t\tfor t := range ts {\n\t\t\t\tout = append(out, t)\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tvar r *Type\n\t\tswitch h.k {\n\t\tcase ListKind, RefKind, SetKind:\n\t\t\tr = foldCompoundTypesForUnion(h.k, ts, seenStructs, intersectStructs)\n\t\tcase MapKind:\n\t\t\tr = foldMapTypesForUnion(ts, seenStructs, intersectStructs)\n\t\tcase StructKind:\n\t\t\tr = foldStructTypes(h.n, ts, seenStructs, intersectStructs)\n\t\t}\n\t\tout = append(out, r)\n\t}\n\n\tfor i, t := range out {\n\t\tout[i] = foldUnions(t, seenStructs, intersectStructs)\n\t}\n\n\tif len(out) == 1 {\n\t\treturn out[0]\n\t}\n\n\tsort.Sort(out)\n\n\treturn newType(CompoundDesc{UnionKind, out})\n}\n\nfunc foldCompoundTypesForUnion(k NomsKind, ts, seenStructs typeset, intersectStructs bool) *Type {\n\telemTypes := make(typeset, len(ts))\n\tfor t := range ts {\n\t\td.PanicIfFalse(t.TargetKind() == k)\n\t\telemTypes.add(t.Desc.(CompoundDesc).ElemTypes[0])\n\t}\n\n\telemType := foldUnionImpl(elemTypes, seenStructs, intersectStructs)\n\treturn makeCompoundType(k, elemType)\n}\n\nfunc foldMapTypesForUnion(ts, seenStructs typeset, intersectStructs bool) *Type {\n\tkeyTypes := make(typeset, len(ts))\n\tvalTypes := make(typeset, len(ts))\n\tfor t := range ts {\n\t\td.PanicIfFalse(t.TargetKind() == MapKind)\n\t\telemTypes := t.Desc.(CompoundDesc).ElemTypes\n\t\tkeyTypes.add(elemTypes[0])\n\t\tvalTypes.add(elemTypes[1])\n\t}\n\n\tkt := foldUnionImpl(keyTypes, seenStructs, intersectStructs)\n\tvt := foldUnionImpl(valTypes, seenStructs, intersectStructs)\n\n\treturn makeCompoundType(MapKind, kt, vt)\n}\n\nfunc foldStructTypesFieldsOnly(name string, ts, seenStructs typeset, intersectStructs bool) structTypeFields {\n\tfieldset := make([]structTypeFields, len(ts))\n\ti := 0\n\tfor t := range ts {\n\t\tdesc := t.Desc.(StructDesc)\n\t\td.PanicIfFalse(desc.Name == name)\n\t\tfieldset[i] = desc.fields\n\t\ti++\n\t}\n\n\treturn simplifyStructFields(fieldset, seenStructs, intersectStructs)\n}\n\nfunc foldStructTypes(name string, ts, seenStructs typeset, intersectStructs bool) *Type {\n\tfields := foldStructTypesFieldsOnly(name, ts, seenStructs, intersectStructs)\n\treturn newType(StructDesc{name, fields})\n}\n\nfunc simplifyStructFields(in []structTypeFields, seenStructs typeset, intersectStructs bool) structTypeFields {\n\t// We gather all the fields/types into allFields. If the number of\n\t// times a field name is present is less that then number of types we\n\t// are simplifying then the field must be optional.\n\t// If we see an optional field we do not increment the count for it and\n\t// it will be treated as optional in the end.\n\n\t// If intersectStructs is true we need to pick the more restrictive version (n: T over n?: T).\n\ttype fieldTypeInfo struct {\n\t\tanyNonOptional bool\n\t\tcount int\n\t\tts typeSlice\n\t}\n\tallFields := map[string]fieldTypeInfo{}\n\n\tfor _, ff := range in {\n\t\tfor _, f := range ff {\n\t\t\tfti, ok := allFields[f.Name]\n\t\t\tif !ok {\n\t\t\t\tfti = fieldTypeInfo{\n\t\t\t\t\tts: make(typeSlice, 0, len(in)),\n\t\t\t\t}\n\t\t\t}\n\t\t\tfti.ts = append(fti.ts, f.Type)\n\t\t\tif !f.Optional {\n\t\t\t\tfti.count++\n\t\t\t\tfti.anyNonOptional = true\n\t\t\t}\n\t\t\tallFields[f.Name] = fti\n\t\t}\n\t}\n\n\tcount := len(in)\n\tfields := make(structTypeFields, len(allFields))\n\ti := 0\n\tfor name, fti := range allFields {\n\t\tnt := makeUnionType(fti.ts...)\n\t\tfields[i] = StructField{\n\t\t\tName: name,\n\t\t\tType: foldUnions(nt, seenStructs, intersectStructs),\n\t\t\tOptional: !(intersectStructs && fti.anyNonOptional) && fti.count < count,\n\t\t}\n\t\ti++\n\t}\n\n\tsort.Sort(fields)\n\n\treturn fields\n}\n" }, { "path": "go/types/simplify_test.go", "content": "package types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestSimplifyStructFields(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttest := func(in []structTypeFields, exp structTypeFields) {\n\t\t// simplifier := newSimplifier(false)\n\t\tact := simplifyStructFields(in, typeset{}, false)\n\t\tassert.Equal(act, exp)\n\t}\n\n\ttest([]structTypeFields{\n\t\t{\n\t\t\tStructField{\"a\", BoolType, false},\n\t\t},\n\t\t{\n\t\t\tStructField{\"a\", BoolType, false},\n\t\t},\n\t},\n\t\tstructTypeFields{\n\t\t\tStructField{\"a\", BoolType, false},\n\t\t},\n\t)\n\n\ttest([]structTypeFields{\n\t\t{\n\t\t\tStructField{\"a\", BoolType, false},\n\t\t},\n\t\t{\n\t\t\tStructField{\"b\", BoolType, false},\n\t\t},\n\t},\n\t\tstructTypeFields{\n\t\t\tStructField{\"a\", BoolType, true},\n\t\t\tStructField{\"b\", BoolType, true},\n\t\t},\n\t)\n\n\ttest([]structTypeFields{\n\t\t{\n\t\t\tStructField{\"a\", BoolType, false},\n\t\t},\n\t\t{\n\t\t\tStructField{\"a\", BoolType, true},\n\t\t},\n\t},\n\t\tstructTypeFields{\n\t\t\tStructField{\"a\", BoolType, true},\n\t\t},\n\t)\n}\n\nfunc TestSimplifyType(t *testing.T) {\n\tassert := assert.New(t)\n\n\trun := func(intersectStructs bool) {\n\t\ttest := func(in, exp *Type) {\n\t\t\tact := simplifyType(in, intersectStructs)\n\t\t\tassert.True(exp.Equals(act), \"Expected: %s\\nActual: %s\", exp.Describe(), act.Describe())\n\t\t}\n\t\ttestSame := func(t *Type) {\n\t\t\ttest(t, t)\n\t\t}\n\n\t\ttestSame(BlobType)\n\t\ttestSame(BoolType)\n\t\ttestSame(NumberType)\n\t\ttestSame(StringType)\n\t\ttestSame(TypeType)\n\t\ttestSame(ValueType)\n\t\ttestSame(makeCompoundType(ListKind, BoolType))\n\t\ttestSame(makeCompoundType(SetKind, BoolType))\n\t\ttestSame(makeCompoundType(RefKind, BoolType))\n\t\ttestSame(makeCompoundType(MapKind, BoolType, NumberType))\n\n\t\t{\n\t\t\t// Cannot do equals on cycle types\n\t\t\tin := MakeCycleType(\"ABC\")\n\t\t\tact := simplifyType(in, intersectStructs)\n\t\t\tassert.Equal(in, act)\n\t\t}\n\n\t\ttest(makeUnionType(BoolType), BoolType)\n\t\ttest(makeUnionType(BoolType, BoolType), BoolType)\n\t\ttestSame(makeUnionType(BoolType, NumberType))\n\t\ttest(makeUnionType(NumberType, BoolType), makeUnionType(BoolType, NumberType))\n\t\ttest(makeUnionType(NumberType, BoolType), makeUnionType(BoolType, NumberType))\n\n\t\ttestSame(makeCompoundType(ListKind, makeUnionType(BoolType, NumberType)))\n\t\ttest(makeCompoundType(ListKind, makeUnionType(BoolType)), makeCompoundType(ListKind, BoolType))\n\t\ttest(makeCompoundType(ListKind, makeUnionType(BoolType, BoolType)), makeCompoundType(ListKind, BoolType))\n\n\t\ttestSame(makeStructType(\"\", nil))\n\t\ttestSame(makeStructType(\"\", structTypeFields{}))\n\t\ttestSame(makeStructType(\"\", structTypeFields{\n\t\t\tStructField{\"b\", BoolType, false},\n\t\t\tStructField{\"s\", StringType, !intersectStructs},\n\t\t}))\n\t\ttest(\n\t\t\tmakeStructType(\"\", structTypeFields{\n\t\t\t\tStructField{\"a\", BoolType, false},\n\t\t\t\tStructField{\"b\", makeUnionType(NumberType, NumberType), false},\n\t\t\t}),\n\t\t\tmakeStructType(\"\", structTypeFields{\n\t\t\t\tStructField{\"a\", BoolType, false},\n\t\t\t\tStructField{\"b\", NumberType, false},\n\t\t\t}),\n\t\t)\n\t\t// non named structs do not create cycles.\n\t\ttestSame(makeStructType(\"\", structTypeFields{\n\t\t\tStructField{\"b\", BoolType, false},\n\t\t\tStructField{\n\t\t\t\t\"s\",\n\t\t\t\tmakeStructType(\"\", structTypeFields{\n\t\t\t\t\tStructField{\"c\", StringType, false},\n\t\t\t\t}),\n\t\t\t\t!intersectStructs,\n\t\t\t},\n\t\t}))\n\n\t\t// merge non named structs in unions\n\t\ttest(\n\t\t\tmakeCompoundType(\n\t\t\t\tUnionKind,\n\t\t\t\tmakeStructType(\"\", structTypeFields{\n\t\t\t\t\tStructField{\"a\", BoolType, false},\n\t\t\t\t}),\n\t\t\t\tmakeStructType(\"\", structTypeFields{\n\t\t\t\t\tStructField{\"b\", BoolType, false},\n\t\t\t\t}),\n\t\t\t),\n\t\t\tmakeStructType(\"\", structTypeFields{\n\t\t\t\tStructField{\"a\", BoolType, !intersectStructs},\n\t\t\t\tStructField{\"b\", BoolType, !intersectStructs},\n\t\t\t}),\n\t\t)\n\n\t\t// List | List -> List\n\t\tfor _, k := range []NomsKind{ListKind, SetKind, RefKind} {\n\t\t\ttest(\n\t\t\t\tmakeCompoundType(\n\t\t\t\t\tUnionKind,\n\t\t\t\t\tmakeCompoundType(k, NumberType),\n\t\t\t\t\tmakeCompoundType(k, BoolType),\n\t\t\t\t),\n\t\t\t\tmakeCompoundType(k,\n\t\t\t\t\tmakeUnionType(BoolType, NumberType),\n\t\t\t\t),\n\t\t\t)\n\t\t}\n\n\t\t// Map | List -> List\n\t\ttest(\n\t\t\tmakeCompoundType(\n\t\t\t\tUnionKind,\n\t\t\t\tmakeCompoundType(MapKind, NumberType, NumberType),\n\t\t\t\tmakeCompoundType(MapKind, BoolType, NumberType),\n\t\t\t),\n\t\t\tmakeCompoundType(MapKind,\n\t\t\t\tmakeUnionType(BoolType, NumberType),\n\t\t\t\tNumberType,\n\t\t\t),\n\t\t)\n\n\t\t// Map | List -> List\n\t\ttest(\n\t\t\tmakeCompoundType(\n\t\t\t\tUnionKind,\n\t\t\t\tmakeCompoundType(MapKind, NumberType, NumberType),\n\t\t\t\tmakeCompoundType(MapKind, NumberType, BoolType),\n\t\t\t),\n\t\t\tmakeCompoundType(MapKind,\n\t\t\t\tNumberType,\n\t\t\t\tmakeUnionType(BoolType, NumberType),\n\t\t\t),\n\t\t)\n\n\t\t// union flattening\n\t\ttest(\n\t\t\tmakeUnionType(NumberType, makeUnionType(NumberType, BoolType)),\n\t\t\tmakeUnionType(BoolType, NumberType),\n\t\t)\n\n\t\t{\n\t\t\t// Cannot do equals on cycle types\n\t\t\tin := makeUnionType(MakeCycleType(\"A\"), MakeCycleType(\"A\"))\n\t\t\texp := MakeCycleType(\"A\")\n\t\t\tact := simplifyType(in, intersectStructs)\n\t\t\tassert.Equal(exp, act)\n\t\t}\n\n\t\t{\n\t\t\t// Cannot do equals on cycle types\n\t\t\tin := makeCompoundType(UnionKind,\n\t\t\t\tmakeCompoundType(ListKind, MakeCycleType(\"A\")),\n\t\t\t\tmakeCompoundType(ListKind, MakeCycleType(\"A\")))\n\t\t\texp := makeCompoundType(ListKind, MakeCycleType(\"A\"))\n\t\t\tact := simplifyType(in, intersectStructs)\n\t\t\tassert.Equal(exp, act, \"Expected: %s\\nActual: %s\", exp.Describe(), act.Describe())\n\t\t}\n\n\t\ttestSame(makeStructType(\"A\", nil))\n\t\ttestSame(makeStructType(\"A\", structTypeFields{}))\n\t\ttestSame(makeStructType(\"A\", structTypeFields{\n\t\t\tStructField{\"a\", BoolType, !intersectStructs},\n\t\t}))\n\t\ttest(\n\t\t\tmakeStructType(\"A\", structTypeFields{\n\t\t\t\tStructField{\"a\", makeUnionType(BoolType, BoolType, NumberType), false},\n\t\t\t}),\n\t\t\tmakeStructType(\"A\", structTypeFields{\n\t\t\t\tStructField{\"a\", makeUnionType(BoolType, NumberType), false},\n\t\t\t}),\n\t\t)\n\n\t\ttestSame(\n\t\t\tmakeStructType(\"A\", structTypeFields{\n\t\t\t\tStructField{\n\t\t\t\t\t\"a\",\n\t\t\t\t\tmakeStructType(\"B\", structTypeFields{\n\t\t\t\t\t\tStructField{\"b\", BoolType, !intersectStructs},\n\t\t\t\t\t}),\n\t\t\t\t\tfalse,\n\t\t\t\t},\n\t\t\t}),\n\t\t)\n\n\t\t{\n\t\t\t// Create pointer cycle manually.\n\t\t\texp := makeStructType(\"A\", structTypeFields{\n\t\t\t\tStructField{\n\t\t\t\t\t\"a\",\n\t\t\t\t\tBoolType, // placeholder\n\t\t\t\t\t!intersectStructs,\n\t\t\t\t},\n\t\t\t})\n\t\t\texp.Desc.(StructDesc).fields[0].Type = exp\n\t\t\ttest(\n\t\t\t\tmakeStructType(\"A\", structTypeFields{\n\t\t\t\t\tStructField{\n\t\t\t\t\t\t\"a\",\n\t\t\t\t\t\tmakeStructType(\"A\", structTypeFields{}),\n\t\t\t\t\t\tfalse,\n\t\t\t\t\t},\n\t\t\t\t}),\n\t\t\t\texp,\n\t\t\t)\n\t\t}\n\n\t\t{\n\t\t\ta := makeStructType(\"S\", structTypeFields{})\n\t\t\texp := makeCompoundType(MapKind, a, a)\n\t\t\ttest(\n\t\t\t\tmakeCompoundType(MapKind,\n\t\t\t\t\tmakeStructType(\"S\", structTypeFields{}),\n\t\t\t\t\tmakeStructType(\"S\", structTypeFields{}),\n\t\t\t\t),\n\t\t\t\texp,\n\t\t\t)\n\t\t}\n\n\t\t{\n\t\t\ta := makeStructType(\"S\", structTypeFields{\n\t\t\t\tStructField{\"a\", BoolType, !intersectStructs},\n\t\t\t\tStructField{\"b\", makeUnionType(BoolType, StringType), false},\n\t\t\t})\n\t\t\texp := makeCompoundType(MapKind, a, a)\n\t\t\ttest(\n\t\t\t\tmakeCompoundType(MapKind,\n\t\t\t\t\tmakeStructType(\"S\", structTypeFields{\n\t\t\t\t\t\tStructField{\"a\", BoolType, false},\n\t\t\t\t\t\tStructField{\"b\", StringType, false},\n\t\t\t\t\t}),\n\t\t\t\t\tmakeStructType(\"S\", structTypeFields{\n\t\t\t\t\t\tStructField{\"b\", BoolType, false},\n\t\t\t\t\t}),\n\t\t\t\t),\n\t\t\t\texp,\n\t\t\t)\n\t\t}\n\n\t\t// Non named do not get merged outside unions\n\t\ttestSame(\n\t\t\tmakeCompoundType(MapKind,\n\t\t\t\tmakeStructType(\"\", structTypeFields{\n\t\t\t\t\tStructField{\"a\", BoolType, false},\n\t\t\t\t\tStructField{\"b\", StringType, false},\n\t\t\t\t}),\n\t\t\t\tmakeStructType(\"\", structTypeFields{\n\t\t\t\t\tStructField{\"b\", BoolType, false},\n\t\t\t\t}),\n\t\t\t),\n\t\t)\n\n\t\t// Cycle in union\n\t\t{\n\t\t\ta := makeStructType(\"A\", structTypeFields{\n\t\t\t\tStructField{\n\t\t\t\t\t\"a\",\n\t\t\t\t\tBoolType, // placeholder\n\t\t\t\t\t!intersectStructs,\n\t\t\t\t},\n\t\t\t})\n\t\t\ta.Desc.(StructDesc).fields[0].Type = a\n\t\t\texp := makeUnionType(NumberType, a, TypeType)\n\t\t\ttest(\n\t\t\t\tmakeCompoundType(UnionKind,\n\t\t\t\t\tmakeStructType(\"A\", structTypeFields{\n\t\t\t\t\t\tStructField{\n\t\t\t\t\t\t\t\"a\",\n\t\t\t\t\t\t\tmakeStructType(\"A\", structTypeFields{}),\n\t\t\t\t\t\t\tfalse,\n\t\t\t\t\t\t},\n\t\t\t\t\t}),\n\t\t\t\t\tNumberType,\n\t\t\t\t\tTypeType,\n\t\t\t\t),\n\t\t\t\texp,\n\t\t\t)\n\t\t}\n\n\t\ttest(\n\t\t\tmakeCompoundType(RefKind,\n\t\t\t\tmakeCompoundType(UnionKind,\n\t\t\t\t\tmakeCompoundType(ListKind,\n\t\t\t\t\t\tBoolType,\n\t\t\t\t\t),\n\t\t\t\t\tmakeCompoundType(SetKind,\n\t\t\t\t\t\tmakeUnionType(StringType, NumberType),\n\t\t\t\t\t),\n\t\t\t\t),\n\t\t\t),\n\t\t\tmakeCompoundType(RefKind,\n\t\t\t\tmakeCompoundType(UnionKind,\n\t\t\t\t\tmakeCompoundType(ListKind,\n\t\t\t\t\t\tBoolType,\n\t\t\t\t\t),\n\t\t\t\t\tmakeCompoundType(SetKind,\n\t\t\t\t\t\tmakeUnionType(NumberType, StringType),\n\t\t\t\t\t),\n\t\t\t\t),\n\t\t\t),\n\t\t)\n\t}\n\n\tt.Run(\"Union\", func(*testing.T) {\n\t\trun(false)\n\t})\n\tt.Run(\"IntersectStructs\", func(*testing.T) {\n\t\trun(true)\n\t})\n}\n" }, { "path": "go/types/string.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"encoding/binary\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n// String is a Noms Value wrapper around the primitive string type.\ntype String string\n\n// Value interface\nfunc (s String) Value() Value {\n\treturn s\n}\n\nfunc (s String) Equals(other Value) bool {\n\treturn s == other\n}\n\nfunc (s String) Less(other Value) bool {\n\tif s2, ok := other.(String); ok {\n\t\treturn s < s2\n\t}\n\treturn StringKind < other.Kind()\n}\n\nfunc (s String) Hash() hash.Hash {\n\treturn getHash(s)\n}\n\nfunc (s String) WalkValues(cb ValueCallback) {\n}\n\nfunc (s String) WalkRefs(cb RefCallback) {\n}\n\nfunc (s String) typeOf() *Type {\n\treturn StringType\n}\n\nfunc (s String) Kind() NomsKind {\n\treturn StringKind\n}\n\nfunc (s String) valueReadWriter() ValueReadWriter {\n\treturn nil\n}\n\nfunc (s String) writeTo(w nomsWriter) {\n\tStringKind.writeTo(w)\n\tw.writeString(string(s))\n}\n\nfunc (s String) valueBytes() []byte {\n\t// We know the size of the buffer here so allocate it once.\n\t// StringKind, Length (UVarint), UTF-8 encoded string\n\tbuff := make([]byte, 1+binary.MaxVarintLen64+len(s))\n\tw := binaryNomsWriter{buff, 0}\n\ts.writeTo(&w)\n\treturn buff[:w.offset]\n}\n" }, { "path": "go/types/string_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestStringEquals(t *testing.T) {\n\tassert := assert.New(t)\n\ts1 := String(\"foo\")\n\ts2 := String(\"foo\")\n\ts3 := s2\n\ts4 := String(\"bar\")\n\tassert.True(s1.Equals(s2))\n\tassert.True(s2.Equals(s1))\n\tassert.True(s1.Equals(s3))\n\tassert.True(s3.Equals(s1))\n\tassert.False(s1.Equals(s4))\n\tassert.False(s4.Equals(s1))\n}\n\nfunc TestStringString(t *testing.T) {\n\tassert := assert.New(t)\n\ts1 := String(\"\")\n\ts2 := String(\"foo\")\n\tassert.Equal(\"\", string(s1))\n\tassert.Equal(\"foo\", string(s2))\n}\n\nfunc TestStringType(t *testing.T) {\n\tassert.True(t, TypeOf(String(\"hi\")).Equals(StringType))\n}\n" }, { "path": "go/types/struct.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"regexp\"\n\t\"sort\"\n\t\"strings\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nvar EmptyStructType = MakeStructType(\"\")\nvar EmptyStruct = newStruct(\"\", nil, nil)\n\ntype StructData map[string]Value\n\ntype Struct struct {\n\tvalueImpl\n}\n\n// readStruct reads the data provided by a decoder and moves the decoder forward.\nfunc readStruct(dec *valueDecoder) Struct {\n\tstart := dec.pos()\n\tskipStruct(dec)\n\tend := dec.pos()\n\treturn Struct{valueImpl{dec.vrw, dec.byteSlice(start, end), nil}}\n}\n\nfunc skipStruct(dec *valueDecoder) {\n\tdec.skipKind()\n\tdec.skipString() // name\n\tcount := dec.readCount()\n\tfor i := uint64(0); i < count; i++ {\n\t\tdec.skipString()\n\t\tdec.skipValue()\n\t}\n}\n\nfunc isStructSameTypeForSure(dec *valueDecoder, t *Type) bool {\n\tdesc := t.Desc.(StructDesc)\n\tdec.skipKind()\n\tif !dec.isStringSame(desc.Name) {\n\t\treturn false\n\t}\n\tcount := dec.readCount()\n\tif count != uint64(len(desc.fields)) {\n\t\treturn false\n\t}\n\tfor i := uint64(0); i < count; i++ {\n\t\tif desc.fields[i].Optional {\n\t\t\treturn false\n\t\t}\n\t\tif !dec.isStringSame(desc.fields[i].Name) {\n\t\t\treturn false\n\t\t}\n\n\t\tif !dec.isValueSameTypeForSure(desc.fields[i].Type) {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\nfunc walkStruct(r *refWalker, cb RefCallback) {\n\tr.skipKind()\n\tr.skipString() // name\n\tcount := r.readCount()\n\tfor i := uint64(0); i < count; i++ {\n\t\tr.skipString()\n\t\tr.walkValue(cb)\n\t}\n}\n\nfunc newStruct(name string, fieldNames []string, values []Value) Struct {\n\tvar vrw ValueReadWriter\n\tw := newBinaryNomsWriter()\n\tStructKind.writeTo(&w)\n\tw.writeString(name)\n\tw.writeCount(uint64(len(fieldNames)))\n\tfor i := 0; i < len(fieldNames); i++ {\n\t\tw.writeString(fieldNames[i])\n\t\tif vrw == nil {\n\t\t\tvrw = values[i].(valueReadWriter).valueReadWriter()\n\t\t}\n\t\tvalues[i].writeTo(&w)\n\t}\n\treturn Struct{valueImpl{vrw, w.data(), nil}}\n}\n\nfunc NewStruct(name string, data StructData) Struct {\n\tverifyStructName(name)\n\tfieldNames := make([]string, len(data))\n\tvalues := make([]Value, len(data))\n\n\ti := 0\n\tfor name := range data {\n\t\tverifyFieldName(name)\n\t\tfieldNames[i] = name\n\t\ti++\n\t}\n\n\tsort.Sort(sort.StringSlice(fieldNames))\n\tfor i = 0; i < len(fieldNames); i++ {\n\t\tvalues[i] = data[fieldNames[i]]\n\t}\n\n\treturn newStruct(name, fieldNames, values)\n}\n\n// StructTemplate allows creating a template for structs with a known shape\n// (name and fields). If a lot of structs of the same shape are being created\n// then using a StructTemplate makes that slightly more efficient.\ntype StructTemplate struct {\n\tname string\n\tfieldNames []string\n}\n\n// MakeStructTemplate creates a new StructTemplate or panics if the name and\n// fields are not valid.\nfunc MakeStructTemplate(name string, fieldNames []string) (t StructTemplate) {\n\tt = StructTemplate{name, fieldNames}\n\n\tverifyStructName(name)\n\tif len(fieldNames) == 0 {\n\t\treturn\n\t}\n\tverifyFieldName(fieldNames[0])\n\tfor i := 1; i < len(fieldNames); i++ {\n\t\tverifyFieldName(fieldNames[i])\n\t\td.PanicIfFalse(fieldNames[i] > fieldNames[i-1])\n\t}\n\treturn\n}\n\n// NewStruct creates a new Struct from the StructTemplate. The order of the\n// values must match the order of the field names of the StructTemplate.\nfunc (st StructTemplate) NewStruct(values []Value) Struct {\n\td.PanicIfFalse(len(st.fieldNames) == len(values))\n\treturn newStruct(st.name, st.fieldNames, values)\n}\n\nfunc (s Struct) Empty() bool {\n\treturn s.Len() == 0\n}\n\n// Value interface\nfunc (s Struct) Value() Value {\n\treturn s\n}\n\nfunc (s Struct) WalkValues(cb ValueCallback) {\n\tdec, count := s.decoderSkipToFields()\n\tfor i := uint64(0); i < count; i++ {\n\t\tdec.skipString()\n\t\tcb(dec.readValue())\n\t}\n}\n\nfunc (s Struct) typeOf() *Type {\n\tdec := s.decoder()\n\treturn readStructTypeOfValue(&dec)\n}\n\nfunc readStructTypeOfValue(dec *valueDecoder) *Type {\n\tdec.skipKind()\n\tname := dec.readString()\n\tcount := dec.readCount()\n\ttypeFields := make(structTypeFields, count)\n\tfor i := uint64(0); i < count; i++ {\n\t\ttypeFields[i] = StructField{\n\t\t\tName: dec.readString(),\n\t\t\tOptional: false,\n\t\t\tType: dec.readTypeOfValue(),\n\t\t}\n\t}\n\treturn makeStructTypeQuickly(name, typeFields)\n}\n\nfunc (s Struct) decoderSkipToFields() (valueDecoder, uint64) {\n\tdec := s.decoder()\n\tdec.skipKind()\n\tdec.skipString()\n\tcount := dec.readCount()\n\treturn dec, count\n}\n\n// Len is the number of fields in the struct.\nfunc (s Struct) Len() int {\n\t_, count := s.decoderSkipToFields()\n\treturn int(count)\n}\n\n// Name is the name of the struct.\nfunc (s Struct) Name() string {\n\tdec := s.decoder()\n\tdec.skipKind()\n\treturn dec.readString()\n}\n\n// IterFields iterates over the fields, calling cb for every field in the\n// struct.\nfunc (s Struct) IterFields(cb func(name string, value Value) (stop bool)) {\n\tdec, count := s.decoderSkipToFields()\n\tfor i := uint64(0); i < count; i++ {\n\t\tif cb(dec.readString(), dec.readValue()) {\n\t\t\tbreak\n\t\t}\n\t}\n}\n\ntype structPartCallbacks interface {\n\tname(n string)\n\tcount(c uint64)\n\tfieldName(n string)\n\tfieldValue(v Value)\n\tend()\n}\n\nfunc (s Struct) iterParts(cbs structPartCallbacks) {\n\tdec := s.decoder()\n\tdec.skipKind()\n\tcbs.name(dec.readString())\n\tcount := dec.readCount()\n\tcbs.count(count)\n\tfor i := uint64(0); i < count; i++ {\n\t\tcbs.fieldName(dec.readString())\n\t\tcbs.fieldValue(dec.readValue())\n\t}\n\tcbs.end()\n}\n\n// MaybeGet returns the value of a field in the struct. If the struct does not a have a field with\n// the name name then this returns (nil, false).\nfunc (s Struct) MaybeGet(n string) (v Value, found bool) {\n\tdec, count := s.decoderSkipToFields()\n\tfor i := uint64(0); i < count; i++ {\n\t\tname := dec.readString()\n\t\tif name == n {\n\t\t\tfound = true\n\t\t\tv = dec.readValue()\n\t\t\treturn\n\t\t}\n\t\tif name > n {\n\t\t\treturn\n\t\t}\n\t\tdec.skipValue()\n\t}\n\n\treturn\n}\n\n// Get returns the value of a field in the struct. If the struct does not a have a field with the\n// name name then this panics.\nfunc (s Struct) Get(n string) Value {\n\tv, ok := s.MaybeGet(n)\n\tif !ok {\n\t\td.Chk.Fail(fmt.Sprintf(`Struct has no field \"%s\"`, n))\n\t}\n\treturn v\n}\n\n// Set returns a new struct where the field name has been set to value. If name is not an\n// existing field in the struct or the type of value is different from the old value of the\n// struct field a new struct type is created.\nfunc (s Struct) Set(n string, v Value) Struct {\n\tverifyFieldName(n)\n\n\tprolog, head, tail, count, found := s.splitFieldsAt(n)\n\n\tw := binaryNomsWriter{make([]byte, len(s.buff)), 0}\n\tw.writeRaw(prolog)\n\n\tif !found {\n\t\tcount++\n\t}\n\tw.writeCount(count)\n\tw.writeRaw(head)\n\tw.writeString(n)\n\tv.writeTo(&w)\n\tw.writeRaw(tail)\n\n\treturn Struct{valueImpl{s.vrw, w.data(), nil}}\n}\n\nfunc (s Struct) SetName(name string) Struct {\n\tverifyStructName(name)\n\n\tw := binaryNomsWriter{make([]byte, len(s.buff)), 0}\n\tStructKind.writeTo(&w)\n\tw.writeString(name)\n\n\tdec := s.decoder()\n\tdec.skipKind()\n\tdec.skipString()\n\n\tw.writeRaw(dec.buff[dec.offset:])\n\treturn Struct{valueImpl{s.vrw, w.data(), nil}}\n}\n\n// splitFieldsAt splits the buffer into two parts. The fields coming before the field we are looking for\n// and the fields coming after it.\nfunc (s Struct) splitFieldsAt(name string) (prolog, head, tail []byte, count uint64, found bool) {\n\tdec := s.decoder()\n\tdec.skipKind()\n\tdec.skipString()\n\tprolog = dec.buff[:dec.offset]\n\tcount = dec.readCount()\n\tfieldsOffset := dec.offset\n\n\tfor i := uint64(0); i < count; i++ {\n\t\tbeforeCurrent := dec.offset\n\t\tfn := dec.readString()\n\t\tdec.skipValue()\n\n\t\tif fn == name {\n\t\t\tfound = true\n\t\t\thead = dec.buff[fieldsOffset:beforeCurrent]\n\t\t\ttail = dec.buff[dec.offset:len(dec.buff)]\n\t\t\tbreak\n\t\t}\n\n\t\tif name < fn {\n\t\t\thead = dec.buff[fieldsOffset:beforeCurrent]\n\t\t\ttail = dec.buff[beforeCurrent:len(dec.buff)]\n\t\t\tbreak\n\t\t}\n\t}\n\n\tif head == nil && tail == nil {\n\t\thead = dec.buff[fieldsOffset:dec.offset]\n\t}\n\n\treturn\n}\n\n// Delete returns a new struct where the field name has been removed.\n// If name is not an existing field in the struct then the current struct is returned.\nfunc (s Struct) Delete(n string) Struct {\n\tprolog, head, tail, count, found := s.splitFieldsAt(n)\n\tif !found {\n\t\treturn s\n\t}\n\n\tw := binaryNomsWriter{make([]byte, len(s.buff)), 0}\n\tw.writeRaw(prolog)\n\tw.writeCount(count - 1)\n\tw.writeRaw(head)\n\tw.writeRaw(tail)\n\n\treturn Struct{valueImpl{s.vrw, w.data(), nil}}\n}\n\nfunc (s Struct) Diff(last Struct, changes chan<- ValueChanged, closeChan <-chan struct{}) {\n\tif s.Equals(last) {\n\t\treturn\n\t}\n\tdec1, dec2 := s.decoder(), last.decoder()\n\tdec1.skipKind()\n\tdec2.skipKind()\n\tdec1.skipString() // Ignore names\n\tdec2.skipString()\n\tcount1, count2 := dec1.readCount(), dec2.readCount()\n\ti1, i2 := uint64(0), uint64(0)\n\tvar fn1, fn2 string\n\n\tfor i1 < count1 && i2 < count2 {\n\t\tif fn1 == \"\" {\n\t\t\tfn1 = dec1.readString()\n\t\t}\n\t\tif fn2 == \"\" {\n\t\t\tfn2 = dec2.readString()\n\t\t}\n\t\tvar change ValueChanged\n\t\tif fn1 == fn2 {\n\t\t\tv1, v2 := dec1.readValue(), dec2.readValue()\n\t\t\tif !v1.Equals(v2) {\n\t\t\t\tchange = ValueChanged{DiffChangeModified, String(fn1), v2, v1}\n\t\t\t}\n\t\t\ti1++\n\t\t\ti2++\n\t\t\tfn1, fn2 = \"\", \"\"\n\t\t} else if fn1 < fn2 {\n\t\t\tv1 := dec1.readValue()\n\t\t\tchange = ValueChanged{DiffChangeAdded, String(fn1), nil, v1}\n\t\t\ti1++\n\t\t\tfn1 = \"\"\n\t\t} else {\n\t\t\tv2 := dec2.readValue()\n\t\t\tchange = ValueChanged{DiffChangeRemoved, String(fn2), v2, nil}\n\t\t\ti2++\n\t\t\tfn2 = \"\"\n\t\t}\n\n\t\tif change != (ValueChanged{}) && !sendChange(changes, closeChan, change) {\n\t\t\treturn\n\t\t}\n\t}\n\n\tfor ; i1 < count1; i1++ {\n\t\tif fn1 == \"\" {\n\t\t\tfn1 = dec1.readString()\n\t\t}\n\t\tv1 := dec1.readValue()\n\t\tif !sendChange(changes, closeChan, ValueChanged{DiffChangeAdded, String(fn1), nil, v1}) {\n\t\t\treturn\n\t\t}\n\t}\n\n\tfor ; i2 < count2; i2++ {\n\t\tif fn2 == \"\" {\n\t\t\tfn2 = dec2.readString()\n\t\t}\n\t\tv2 := dec2.readValue()\n\t\tif !sendChange(changes, closeChan, ValueChanged{DiffChangeRemoved, String(fn2), v2, nil}) {\n\t\t\treturn\n\t\t}\n\t}\n}\n\nvar escapeChar = \"Q\"\nvar headFieldNamePattern = regexp.MustCompile(\"[a-zA-Z]\")\nvar tailFieldNamePattern = regexp.MustCompile(\"[a-zA-Z0-9_]\")\nvar spaceRegex = regexp.MustCompile(\"[ ]\")\nvar escapeRegex = regexp.MustCompile(escapeChar)\n\nvar fieldNameComponentRe = regexp.MustCompile(\"^\" + headFieldNamePattern.String() + tailFieldNamePattern.String() + \"*\")\nvar fieldNameRe = regexp.MustCompile(fieldNameComponentRe.String() + \"$\")\n\ntype encodingFunc func(string, *regexp.Regexp) string\n\nfunc CamelCaseFieldName(input string) string {\n\t//strip invalid struct characters and leave spaces\n\tencode := func(s1 string, p *regexp.Regexp) string {\n\t\tif p.MatchString(s1) || spaceRegex.MatchString(s1) {\n\t\t\treturn s1\n\t\t}\n\t\treturn \"\"\n\t}\n\n\tstrippedField := escapeField(input, encode)\n\tsplitField := strings.Fields(strippedField)\n\n\tif len(splitField) == 0 {\n\t\treturn \"\"\n\t}\n\n\t//Camelcase field\n\toutput := strings.ToLower(splitField[0])\n\tif len(splitField) > 1 {\n\t\tfor _, field := range splitField[1:] {\n\t\t\toutput += strings.Title(strings.ToLower(field))\n\t\t}\n\t}\n\t//Because we are removing characters, we may generate an invalid field name\n\t//i.e. -- 1A B, we will remove the first bad chars and process until 1aB\n\t//1aB is invalid struct field name so we will return \"\"\n\tif !IsValidStructFieldName(output) {\n\t\treturn \"\"\n\t}\n\treturn output\n}\n\nfunc escapeField(input string, encode encodingFunc) string {\n\toutput := \"\"\n\tpattern := headFieldNamePattern\n\tfor _, ch := range input {\n\t\toutput += encode(string([]rune{ch}), pattern)\n\t\tpattern = tailFieldNamePattern\n\t}\n\treturn output\n}\n\n// EscapeStructField escapes names for use as noms structs with regards to non CSV imported data.\n// Disallowed characters are encoded as 'Q'.\n// Note that Q itself is also escaped since it is the escape character.\nfunc EscapeStructField(input string) string {\n\tif !escapeRegex.MatchString(input) && IsValidStructFieldName(input) {\n\t\treturn input\n\t}\n\tencode := func(s1 string, p *regexp.Regexp) string {\n\t\tif p.MatchString(s1) && s1 != escapeChar {\n\t\t\treturn s1\n\t\t}\n\n\t\tvar hs = fmt.Sprintf(\"%X\", s1)\n\t\tvar buf bytes.Buffer\n\t\tbuf.WriteString(escapeChar)\n\t\tif len(hs) == 1 {\n\t\t\tbuf.WriteString(\"0\")\n\t\t}\n\t\tbuf.WriteString(hs)\n\t\treturn buf.String()\n\t}\n\treturn escapeField(input, encode)\n}\n\n// IsValidStructFieldName returns whether the name is valid as a field name in a struct.\n// Valid names must start with `a-zA-Z` and after that `a-zA-Z0-9_`.\nfunc IsValidStructFieldName(name string) bool {\n\tfor i, c := range name {\n\t\tif i == 0 {\n\t\t\tif !isAlpha(c) {\n\t\t\t\treturn false\n\t\t\t}\n\t\t} else if !isAlphaNumOrUnderscore(c) {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn len(name) != 0\n}\n\nfunc isAlpha(c rune) bool {\n\treturn c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z'\n}\n\nfunc isAlphaNumOrUnderscore(c rune) bool {\n\treturn c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || c == '_'\n}\n\nfunc verifyFields(fs structTypeFields) {\n\tfor i, f := range fs {\n\t\tverifyFieldName(f.Name)\n\t\tif i > 0 && strings.Compare(fs[i-1].Name, f.Name) >= 0 {\n\t\t\td.Chk.Fail(\"Field names must be unique and ordered alphabetically\")\n\t\t}\n\t}\n}\n\nfunc verifyName(name, kind string) {\n\tif !IsValidStructFieldName(name) {\n\t\td.Panic(`Invalid struct%s name: \"%s\"`, kind, name)\n\t}\n}\n\nfunc verifyFieldName(name string) {\n\tverifyName(name, \" field\")\n}\n\nfunc verifyStructName(name string) {\n\tif name != \"\" {\n\t\tverifyName(name, \"\")\n\t}\n}\n" }, { "path": "go/types/struct_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc getChunks(v Value) (chunks []Ref) {\n\tv.WalkRefs(func(r Ref) {\n\t\tchunks = append(chunks, r)\n\t})\n\treturn\n}\n\nfunc TestGenericStructEquals(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts1 := NewStruct(\"S1\", StructData{\"s\": String(\"hi\"), \"x\": Bool(true)})\n\ts2 := NewStruct(\"S1\", StructData{\"s\": String(\"hi\"), \"x\": Bool(true)})\n\n\tassert.True(s1.Equals(s2))\n\tassert.True(s2.Equals(s1))\n}\n\nfunc TestGenericStructChunks(t *testing.T) {\n\tassert := assert.New(t)\n\n\tb := Bool(true)\n\ts1 := NewStruct(\"S1\", StructData{\"r\": NewRef(b)})\n\n\tassert.Len(getChunks(s1), 1)\n\tassert.Equal(Bool(true).Hash(), getChunks(s1)[0].TargetHash())\n}\n\nfunc TestGenericStructNew(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts := NewStruct(\"S2\", StructData{\"b\": Bool(true), \"o\": String(\"hi\")})\n\tassert.True(s.Get(\"b\").Equals(Bool(true)))\n\t_, ok := s.MaybeGet(\"missing\")\n\tassert.False(ok)\n\n\ts2 := NewStruct(\"S2\", StructData{\"b\": Bool(false), \"o\": String(\"hi\")})\n\tassert.True(s2.Get(\"b\").Equals(Bool(false)))\n\to, ok := s2.MaybeGet(\"o\")\n\tassert.True(ok)\n\tassert.True(String(\"hi\").Equals(o))\n}\n\nfunc TestGenericStructSet(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\ts := NewStruct(\"S3\", StructData{\"b\": Bool(true), \"o\": String(\"hi\")})\n\ts2 := s.Set(\"b\", Bool(false))\n\n\ts3 := s2.Set(\"b\", Bool(true))\n\tassert.True(s.Equals(s3))\n\n\t// Changes the type\n\ts4 := s.Set(\"b\", Number(42))\n\tassert.True(MakeStructType(\"S3\",\n\t\tStructField{\"b\", NumberType, false},\n\t\tStructField{\"o\", StringType, false},\n\t).Equals(TypeOf(s4)))\n\n\t// Adds a new field\n\ts5 := s.Set(\"x\", Number(42))\n\tassert.True(MakeStructType(\"S3\",\n\t\tStructField{\"b\", BoolType, false},\n\t\tStructField{\"o\", StringType, false},\n\t\tStructField{\"x\", NumberType, false},\n\t).Equals(TypeOf(s5)))\n\n\t// Subtype is not equal.\n\ts6 := NewStruct(\"\", StructData{\"l\": NewList(vs, Number(0), Number(1), Bool(false), Bool(true))})\n\ts7 := s6.Set(\"l\", NewList(vs, Number(2), Number(3)))\n\tt7 := MakeStructTypeFromFields(\"\", FieldMap{\n\t\t\"l\": MakeListType(NumberType),\n\t})\n\tassert.True(t7.Equals(TypeOf(s7)))\n\n\ts8 := NewStruct(\"S\", StructData{\"a\": Bool(true), \"c\": Bool(true)})\n\ts9 := s8.Set(\"b\", Bool(true))\n\tassert.True(s9.Equals(NewStruct(\"S\", StructData{\"a\": Bool(true), \"b\": Bool(true), \"c\": Bool(true)})))\n}\n\nfunc TestGenericStructDelete(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts1 := NewStruct(\"S\", StructData{\"b\": Bool(true), \"o\": String(\"hi\")})\n\n\ts2 := s1.Delete(\"notThere\")\n\tassert.True(s1.Equals(s2))\n\n\ts3 := s1.Delete(\"o\")\n\ts4 := NewStruct(\"S\", StructData{\"b\": Bool(true)})\n\tassert.True(s3.Equals(s4))\n\n\ts5 := s3.Delete(\"b\")\n\ts6 := NewStruct(\"S\", StructData{})\n\tassert.True(s5.Equals(s6))\n}\n\nfunc assertValueChangeEqual(assert *assert.Assertions, c1, c2 ValueChanged) {\n\tassert.Equal(c1.ChangeType, c2.ChangeType)\n\tassert.Equal(EncodedValue(c1.Key), EncodedValue(c2.Key))\n\tif c1.NewValue == nil {\n\t\tassert.Nil(c2.NewValue)\n\t} else {\n\t\tassert.Equal(EncodedValue(c1.NewValue), EncodedValue(c2.NewValue))\n\t}\n\tif c1.OldValue == nil {\n\t\tassert.Nil(c2.OldValue)\n\t} else {\n\t\tassert.Equal(EncodedValue(c1.OldValue), EncodedValue(c2.OldValue))\n\t}\n}\n\nfunc TestStructDiff(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tassertDiff := func(expect []ValueChanged, s1, s2 Struct) {\n\t\tchanges := make(chan ValueChanged)\n\t\tgo func() {\n\t\t\ts1.Diff(s2, changes, nil)\n\t\t\tclose(changes)\n\t\t}()\n\t\ti := 0\n\t\tfor change := range changes {\n\t\t\tassertValueChangeEqual(assert, expect[i], change)\n\t\t\ti++\n\t\t}\n\t\tassert.Equal(len(expect), i, \"Wrong number of changes\")\n\t}\n\n\tvc := func(ct DiffChangeType, fieldName string, oldV, newV Value) ValueChanged {\n\t\treturn ValueChanged{ct, String(fieldName), oldV, newV}\n\t}\n\n\ts1 := NewStruct(\"\", StructData{\"a\": Bool(true), \"b\": String(\"hi\"), \"c\": Number(4)})\n\n\tassertDiff([]ValueChanged{},\n\t\ts1, NewStruct(\"\", StructData{\"a\": Bool(true), \"b\": String(\"hi\"), \"c\": Number(4)}))\n\n\tassertDiff([]ValueChanged{vc(DiffChangeModified, \"a\", Bool(false), Bool(true)), vc(DiffChangeModified, \"b\", String(\"bye\"), String(\"hi\"))},\n\t\ts1, NewStruct(\"\", StructData{\"a\": Bool(false), \"b\": String(\"bye\"), \"c\": Number(4)}))\n\n\tassertDiff([]ValueChanged{vc(DiffChangeModified, \"b\", String(\"bye\"), String(\"hi\")), vc(DiffChangeModified, \"c\", Number(5), Number(4))},\n\t\ts1, NewStruct(\"\", StructData{\"a\": Bool(true), \"b\": String(\"bye\"), \"c\": Number(5)}))\n\n\tassertDiff([]ValueChanged{vc(DiffChangeModified, \"a\", Bool(false), Bool(true)), vc(DiffChangeModified, \"c\", Number(10), Number(4))},\n\t\ts1, NewStruct(\"\", StructData{\"a\": Bool(false), \"b\": String(\"hi\"), \"c\": Number(10)}))\n\n\tassertDiff([]ValueChanged{vc(DiffChangeAdded, \"a\", nil, Bool(true))},\n\t\ts1, NewStruct(\"NewType\", StructData{\"b\": String(\"hi\"), \"c\": Number(4)}))\n\n\tassertDiff([]ValueChanged{vc(DiffChangeAdded, \"b\", nil, String(\"hi\"))},\n\t\ts1, NewStruct(\"NewType\", StructData{\"a\": Bool(true), \"c\": Number(4)}))\n\n\tassertDiff([]ValueChanged{vc(DiffChangeRemoved, \"Z\", Number(17), nil)},\n\t\ts1, NewStruct(\"NewType\", StructData{\"Z\": Number(17), \"a\": Bool(true), \"b\": String(\"hi\"), \"c\": Number(4)}))\n\n\tassertDiff([]ValueChanged{vc(DiffChangeAdded, \"b\", nil, String(\"hi\")), vc(DiffChangeRemoved, \"d\", Number(5), nil)},\n\t\ts1, NewStruct(\"NewType\", StructData{\"a\": Bool(true), \"c\": Number(4), \"d\": Number(5)}))\n\n\ts2 := NewStruct(\"\", StructData{\n\t\t\"a\": NewList(vs, Number(0), Number(1)),\n\t\t\"b\": NewMap(vs, String(\"foo\"), Bool(false), String(\"bar\"), Bool(true)),\n\t\t\"c\": NewSet(vs, Number(0), Number(1), String(\"foo\")),\n\t})\n\n\tassertDiff([]ValueChanged{},\n\t\ts2, NewStruct(\"\", StructData{\n\t\t\t\"a\": NewList(vs, Number(0), Number(1)),\n\t\t\t\"b\": NewMap(vs, String(\"foo\"), Bool(false), String(\"bar\"), Bool(true)),\n\t\t\t\"c\": NewSet(vs, Number(0), Number(1), String(\"foo\")),\n\t\t}))\n\n\tassertDiff([]ValueChanged{\n\t\tvc(DiffChangeModified, \"a\", NewList(vs, Number(1), Number(1)), NewList(vs, Number(0), Number(1))),\n\t\tvc(DiffChangeModified, \"b\", NewMap(vs, String(\"foo\"), Bool(true), String(\"bar\"), Bool(true)), NewMap(vs, String(\"foo\"), Bool(false), String(\"bar\"), Bool(true))),\n\t},\n\t\ts2, NewStruct(\"\", StructData{\n\t\t\t\"a\": NewList(vs, Number(1), Number(1)),\n\t\t\t\"b\": NewMap(vs, String(\"foo\"), Bool(true), String(\"bar\"), Bool(true)),\n\t\t\t\"c\": NewSet(vs, Number(0), Number(1), String(\"foo\")),\n\t\t}))\n\n\tassertDiff([]ValueChanged{\n\t\tvc(DiffChangeModified, \"a\", NewList(vs, Number(0)), NewList(vs, Number(0), Number(1))),\n\t\tvc(DiffChangeModified, \"c\", NewSet(vs, Number(0), Number(2), String(\"foo\")), NewSet(vs, Number(0), Number(1), String(\"foo\"))),\n\t},\n\t\ts2, NewStruct(\"\", StructData{\n\t\t\t\"a\": NewList(vs, Number(0)),\n\t\t\t\"b\": NewMap(vs, String(\"foo\"), Bool(false), String(\"bar\"), Bool(true)),\n\t\t\t\"c\": NewSet(vs, Number(0), Number(2), String(\"foo\")),\n\t\t}))\n\n\tassertDiff([]ValueChanged{\n\t\tvc(DiffChangeModified, \"b\", NewMap(vs, String(\"boo\"), Bool(false), String(\"bar\"), Bool(true)), NewMap(vs, String(\"foo\"), Bool(false), String(\"bar\"), Bool(true))),\n\t\tvc(DiffChangeModified, \"c\", NewSet(vs, Number(0), Number(1), String(\"bar\")), NewSet(vs, Number(0), Number(1), String(\"foo\"))),\n\t},\n\t\ts2, NewStruct(\"\", StructData{\n\t\t\t\"a\": NewList(vs, Number(0), Number(1)),\n\t\t\t\"b\": NewMap(vs, String(\"boo\"), Bool(false), String(\"bar\"), Bool(true)),\n\t\t\t\"c\": NewSet(vs, Number(0), Number(1), String(\"bar\")),\n\t\t}))\n}\n\nfunc TestEscStructField(t *testing.T) {\n\tassert := assert.New(t)\n\tcases := []string{\n\t\t\"a\", \"a\",\n\t\t\"AaZz19_\", \"AaZz19_\",\n\t\t\"Q\", \"Q51\",\n\t\t\"AQ1\", \"AQ511\",\n\t\t\"INSPECTIONQ20STATUS\", \"INSPECTIONQ5120STATUS\",\n\t\t\"$\", \"Q24\",\n\t\t\"_content\", \"Q5Fcontent\",\n\t\t\"Few ¢ents Short\", \"FewQ20QC2A2entsQ20Short\",\n\t\t\"💩\", \"QF09F92A9\",\n\t\t\"https://picasaweb.google.com/data\", \"httpsQ3AQ2FQ2FpicasawebQ2EgoogleQ2EcomQ2Fdata\",\n\t}\n\n\tfor i := 0; i < len(cases); i += 2 {\n\t\torig, expected := cases[i], cases[i+1]\n\t\tassert.Equal(expected, EscapeStructField(orig))\n\t}\n}\n\nfunc TestMakeStructTemplate(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassertInvalidStructName := func(n string) {\n\t\tassert.Panics(func() {\n\t\t\tMakeStructTemplate(n, []string{})\n\t\t})\n\t}\n\n\tassertInvalidStructName(\" \")\n\tassertInvalidStructName(\" a\")\n\tassertInvalidStructName(\"a \")\n\tassertInvalidStructName(\"0\")\n\tassertInvalidStructName(\"_\")\n\tassertInvalidStructName(\"0a\")\n\tassertInvalidStructName(\"_a\")\n\tassertInvalidStructName(\"💩\")\n\n\tassertValidStructName := func(n string) {\n\t\ttemplate := MakeStructTemplate(n, []string{})\n\t\tstr := template.NewStruct(nil)\n\t\tassert.Equal(n, str.Name())\n\t}\n\n\tassertValidStructName(\"\")\n\tassertValidStructName(\"a\")\n\tassertValidStructName(\"A\")\n\tassertValidStructName(\"a0\")\n\tassertValidStructName(\"a_\")\n\tassertValidStructName(\"a0_\")\n\n\tassertInvalidFieldName := func(n string) {\n\t\tassert.Panics(func() {\n\t\t\tMakeStructTemplate(\"\", []string{n})\n\t\t})\n\t}\n\n\tassertInvalidFieldName(\"\")\n\tassertInvalidFieldName(\" \")\n\tassertInvalidFieldName(\" a\")\n\tassertInvalidFieldName(\"a \")\n\tassertInvalidFieldName(\"0\")\n\tassertInvalidFieldName(\"_\")\n\tassertInvalidFieldName(\"0a\")\n\tassertInvalidFieldName(\"_a\")\n\tassertInvalidFieldName(\"💩\")\n\n\tassertValidFieldName := func(n string) {\n\t\tMakeStructTemplate(\"\", []string{n})\n\t}\n\n\tassertValidFieldName(\"a\")\n\tassertValidFieldName(\"A\")\n\tassertValidFieldName(\"a0\")\n\tassertValidFieldName(\"a_\")\n\tassertValidFieldName(\"a0_\")\n\n\tassertInvalidFieldOrder := func(n []string) {\n\t\tassert.Panics(func() {\n\t\t\tMakeStructTemplate(\"\", n)\n\t\t})\n\t}\n\n\tassertInvalidFieldOrder([]string{\"a\", \"a\"})\n\tassertInvalidFieldOrder([]string{\"b\", \"a\"})\n\tassertInvalidFieldOrder([]string{\"a\", \"c\", \"b\"})\n\n\tassertValidFieldOrder := func(n []string) {\n\t\tMakeStructTemplate(\"\", n)\n\t}\n\n\tassertValidFieldOrder([]string{\"a\", \"b\"})\n\tassertValidFieldOrder([]string{\"a\", \"b\", \"c\"})\n\n\ttemplate := MakeStructTemplate(\"A\", []string{\"a\", \"b\"})\n\tstr := template.NewStruct([]Value{Number(42), Bool(true)})\n\tassert.True(NewStruct(\"A\", StructData{\n\t\t\"a\": Number(42),\n\t\t\"b\": Bool(true),\n\t}).Equals(str))\n}\n\nfunc TestStructWithNil(t *testing.T) {\n\tassert.Panics(t, func() {\n\t\tNewStruct(\"A\", StructData{\n\t\t\t\"a\": nil,\n\t\t})\n\t})\n\tassert.Panics(t, func() {\n\t\tNewStruct(\"A\", StructData{\n\t\t\t\"a\": Number(42),\n\t\t\t\"b\": nil,\n\t\t})\n\t})\n}\n\nfunc TestStructIterFields(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttstruct := NewStruct(\"A\", StructData{\n\t\t\"a\": String(\"aaa\"),\n\t\t\"b\": String(\"bbb\"),\n\t\t\"c\": String(\"ccc\"),\n\t})\n\n\t// Iterate over all.\n\ti := 0\n\ttstruct.IterFields(func(k string, v Value) bool {\n\t\tassert.True(tstruct.Get(k).Equals(v))\n\n\t\ti += 1\n\n\t\treturn false\n\t})\n\n\tassert.Equal(3, i)\n\n\t// Iterate and stop.\n\ti = 0\n\ttstruct.IterFields(func(k string, v Value) bool {\n\t\tif k == \"b\" {\n\t\t\treturn true\n\t\t}\n\n\t\ti += 1\n\n\t\treturn false\n\t})\n\n\tassert.Equal(1, i)\n}\n\nfunc TestStructSetName(t *testing.T) {\n\tassert := assert.New(t)\n\ts1 := NewStruct(\"\", StructData{\"foo\": Number(42), \"bar\": String(\"baz\")})\n\ts2 := s1.SetName(\"S\")\n\tassert.Equal(2, s2.Len())\n\tassert.Equal(42.0, float64(s2.Get(\"foo\").(Number)))\n\tassert.Equal(\"baz\", string(s2.Get(\"bar\").(String)))\n\tassert.Equal(\"S\", s2.Name())\n}\n" }, { "path": "go/types/subtype.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nfunc assertSubtype(t *Type, v Value) {\n\tif !IsValueSubtypeOf(v, t) {\n\t\td.Panic(\"Invalid type. %s is not a subtype of %s\", TypeOf(v).Describe(), t.Describe())\n\t}\n}\n\n// IsSubtype determines whether concreteType is a subtype of requiredType. For example, `Number` is a subtype of `Number | String`.\nfunc IsSubtype(requiredType, concreteType *Type) bool {\n\tisSub, _ := isSubtypeTopLevel(requiredType, concreteType)\n\treturn isSub\n}\n\n// IsSubtypeDisallowExtraFields is a slightly weird variant of IsSubtype. It returns true IFF IsSubtype(requiredType, concreteType) AND Structs in concreteType CANNOT have field names absent in requiredType\n// ISSUE: https://github.com/attic-labs/noms/issues/3446\nfunc IsSubtypeDisallowExtraStructFields(requiredType, concreteType *Type) bool {\n\tisSub, hasExtra := isSubtypeDetails(requiredType, concreteType, false, nil)\n\tif hasExtra {\n\t\treturn false\n\t}\n\treturn isSub\n}\n\n// isSubtypeTopLevel returns two values: IsSub and hasExtra. See IsValueSubtypeOf()\n// below for an explanation.\nfunc isSubtypeTopLevel(requiredType, concreteType *Type) (isSub bool, hasExtra bool) {\n\treturn isSubtypeDetails(requiredType, concreteType, false, nil)\n}\n\n// IsSubtypeDetails returns two values:\n// isSub - which indicates whether concreteType is a subtype of requiredType.\n// hasExtra - which indicates whether concreteType has additional fields.\n// See comment below on isValueSubtypeOfDetails\nfunc isSubtypeDetails(requiredType, concreteType *Type, hasExtra bool, parentStructTypes []*Type) (bool, bool) {\n\tif requiredType.Equals(concreteType) {\n\t\treturn true, hasExtra\n\t}\n\n\t// If the concrete type is a union, all component types must be compatible.\n\tif concreteType.TargetKind() == UnionKind {\n\t\tfor _, t := range concreteType.Desc.(CompoundDesc).ElemTypes {\n\t\t\tisSub, hasMore := isSubtypeDetails(requiredType, t, hasExtra, parentStructTypes)\n\t\t\tif !isSub {\n\t\t\t\treturn false, hasExtra\n\t\t\t}\n\t\t\thasExtra = hasExtra || hasMore\n\t\t}\n\t\treturn true, hasExtra\n\t}\n\n\t// If the required type is a union, at least one of the component types must be compatible.\n\tif requiredType.TargetKind() == UnionKind {\n\t\tfor _, t := range requiredType.Desc.(CompoundDesc).ElemTypes {\n\t\t\tisSub, hasMore := isSubtypeDetails(t, concreteType, hasExtra, parentStructTypes)\n\t\t\tif isSub {\n\t\t\t\thasExtra = hasExtra || hasMore\n\t\t\t\treturn true, hasExtra\n\t\t\t}\n\t\t}\n\t\treturn false, hasExtra\n\t}\n\n\tif requiredType.TargetKind() != concreteType.TargetKind() {\n\t\treturn requiredType.TargetKind() == ValueKind, hasExtra\n\t}\n\n\tif desc, ok := requiredType.Desc.(CompoundDesc); ok {\n\t\tconcreteElemTypes := concreteType.Desc.(CompoundDesc).ElemTypes\n\t\tfor i, t := range desc.ElemTypes {\n\t\t\tisSub, hasMore := compoundSubtype(t, concreteElemTypes[i], hasExtra, parentStructTypes)\n\t\t\tif !isSub {\n\t\t\t\treturn false, hasExtra\n\t\t\t}\n\t\t\thasExtra = hasExtra || hasMore\n\t\t}\n\t\treturn true, hasExtra\n\t}\n\n\tif requiredType.TargetKind() == StructKind {\n\t\trequiredDesc := requiredType.Desc.(StructDesc)\n\t\tconcreteDesc := concreteType.Desc.(StructDesc)\n\t\tif requiredDesc.Name != \"\" && requiredDesc.Name != concreteDesc.Name {\n\t\t\treturn false, hasExtra\n\t\t}\n\n\t\t// We may already be computing the subtype for this type if we have a cycle.\n\t\t// In that case we exit the recursive check. We may still find that the type\n\t\t// is not a subtype but that will be handled at a higher level in the callstack.\n\t\t_, found := indexOfType(requiredType, parentStructTypes)\n\t\tif found {\n\t\t\treturn true, hasExtra\n\t\t}\n\n\t\ti, j := 0, 0\n\t\tfor i < requiredDesc.Len() && j < concreteDesc.Len() {\n\t\t\trequiredField := requiredDesc.fields[i]\n\t\t\tconcreteField := concreteDesc.fields[j]\n\t\t\tif requiredField.Name == concreteField.Name {\n\t\t\t\t// Common field name\n\t\t\t\tif !requiredField.Optional && concreteField.Optional {\n\t\t\t\t\treturn false, hasExtra\n\t\t\t\t}\n\n\t\t\t\tisSub, hasMore := isSubtypeDetails(requiredField.Type, concreteField.Type, hasExtra, append(parentStructTypes, requiredType))\n\t\t\t\tif !isSub {\n\t\t\t\t\treturn false, hasExtra\n\t\t\t\t}\n\t\t\t\thasExtra = hasExtra || hasMore\n\n\t\t\t\ti++\n\t\t\t\tj++\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif requiredField.Name < concreteField.Name {\n\t\t\t\t// Concrete lacks field in required\n\t\t\t\tif !requiredField.Optional {\n\t\t\t\t\treturn false, hasExtra\n\t\t\t\t}\n\t\t\t\ti++\n\t\t\t} else {\n\t\t\t\t// Concrete contains extra field\n\t\t\t\thasExtra = true\n\t\t\t\tj++\n\t\t\t}\n\t\t}\n\n\t\tfor i < requiredDesc.Len() {\n\t\t\t// Fields in required not in concrete\n\t\t\tif !requiredDesc.fields[i].Optional {\n\t\t\t\thasExtra = true\n\t\t\t\treturn false, hasExtra\n\t\t\t}\n\t\t\ti++\n\t\t}\n\n\t\thasExtra = hasExtra || j < concreteDesc.Len()\n\t\treturn true, hasExtra\n\t}\n\n\tpanic(\"unreachable\")\n}\n\n// compoundSubtype is called when comparing the element types of two compound types. This is the only case\n// where a concrete type may have be a union type.\nfunc compoundSubtype(requiredType, concreteType *Type, hasExtra bool, parentStructTypes []*Type) (bool, bool) {\n\t// If the concrete type is a union then all the types in the union must be subtypes of the required typ. This also means that a compound type with an empty union is going to be a subtype of all compounds, List<> is a subtype of List for all T.\n\tif concreteType.TargetKind() == UnionKind {\n\t\tfor _, ct := range concreteType.Desc.(CompoundDesc).ElemTypes {\n\t\t\tisSub, hasExtra1 := isSubtypeDetails(requiredType, ct, hasExtra, parentStructTypes)\n\t\t\tif !isSub {\n\t\t\t\treturn false, hasExtra1\n\t\t\t}\n\t\t}\n\t\treturn true, hasExtra\n\t}\n\treturn isSubtypeDetails(requiredType, concreteType, hasExtra, parentStructTypes)\n}\n\nfunc IsValueSubtypeOf(v Value, t *Type) bool {\n\tisSub, _ := isValueSubtypeOfDetails(v, t, false)\n\treturn isSub\n}\n\n// IsValueSubtypeOfDetails returns two values:\n// isSub - which indicates whether v is a subtype of t.\n// hasExtra - which indicates whether v has additional fields. This field has\n// no meaning if IsSub is false.\n//\n// For example, given the following data:\n// type1 := struct S { v := Struct S1 {\n// a Number | string a: \"hello\"\n// b ?int b: 2\n// } }\n// IsValueSubtypeOfDetails(v, type1) would return isSub == true, and hasExtra == false\n//\n// And given these types:\n// type2 := struct S { v := Struct S1 {\n// a Number | string a: \"hello\"\n// b ?int b: 2\n// } c: \"hello again\"\n// }\n// IsValueSubtypeOfDetails(v, type1) would return isSub == true, and hasExtra == true\nfunc IsValueSubtypeOfDetails(v Value, t *Type) (bool, bool) {\n\treturn isValueSubtypeOfDetails(v, t, false)\n}\n\nfunc isValueSubtypeOfDetails(v Value, t *Type, hasExtra bool) (bool, bool) {\n\tswitch t.TargetKind() {\n\tcase BoolKind, NumberKind, StringKind, BlobKind, TypeKind:\n\t\treturn v.Kind() == t.TargetKind(), hasExtra\n\tcase ValueKind:\n\t\treturn true, hasExtra\n\tcase UnionKind:\n\t\tvar anonStruct *Type\n\n\t\tfor _, et := range t.Desc.(CompoundDesc).ElemTypes {\n\t\t\t// Typically if IsSubtype(v.Type(), A|B|C|...) then exactly one of the\n\t\t\t// element types in the union will be a supertype of v.Type() because\n\t\t\t// of type simplification rules (only one of each kind is allowed in\n\t\t\t// the simplified union except for structs, where one of each unique\n\t\t\t// struct name is allowed).\n\t\t\t//\n\t\t\t// However there is one exception which is that type simplification\n\t\t\t// allows the struct with empty name. So if v.Type() is a struct with a\n\t\t\t// name, then it is possible for *two* elements in the union to match\n\t\t\t// it -- a struct with that same name, and a struct with no name.\n\t\t\t//\n\t\t\t// So if we happen across an element type that is an anonymous struct, we\n\t\t\t// save it for later and only try to use it if we can't find anything\n\t\t\t// better.\n\t\t\tif et.TargetKind() == StructKind && et.Desc.(StructDesc).Name == \"\" {\n\t\t\t\tanonStruct = et\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tisSub, hasMore := isValueSubtypeOfDetails(v, et, hasExtra)\n\t\t\tif isSub {\n\t\t\t\thasExtra = hasExtra || hasMore\n\t\t\t\treturn isSub, hasExtra\n\t\t\t}\n\t\t}\n\n\t\tif anonStruct != nil {\n\t\t\tisSub, hasMore := isValueSubtypeOfDetails(v, anonStruct, hasExtra)\n\t\t\tif isSub {\n\t\t\t\thasExtra = hasExtra || hasMore\n\t\t\t\treturn isSub, hasExtra\n\t\t\t}\n\t\t}\n\n\t\treturn false, hasExtra\n\tcase CycleKind:\n\t\tpanic(\"unreachable\") // CycleKind are ephemeral.\n\tdefault:\n\t\tif v.Kind() != t.TargetKind() {\n\t\t\treturn false, hasExtra\n\t\t}\n\t}\n\n\tswitch desc := t.Desc.(type) {\n\tcase StructDesc:\n\t\t// If we provide a named struct type we require that the names match.\n\t\ts := v.(Struct)\n\t\tif desc.Name != \"\" && desc.Name != s.Name() {\n\t\t\treturn false, hasExtra\n\t\t}\n\t\tmissingOptionalFieldCnt := 0\n\t\tfor _, f := range desc.fields {\n\t\t\tfv, ok := s.MaybeGet(f.Name)\n\t\t\tif !ok {\n\t\t\t\tif f.Optional {\n\t\t\t\t\tmissingOptionalFieldCnt += 1\n\t\t\t\t} else {\n\t\t\t\t\treturn false, hasExtra\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\tisSub, hasMore := isValueSubtypeOfDetails(fv, f.Type, hasExtra)\n\t\t\t\tif !isSub {\n\t\t\t\t\treturn false, hasExtra\n\t\t\t\t}\n\t\t\t\thasExtra = hasExtra || hasMore\n\t\t\t}\n\t\t}\n\t\tif s.Len()+missingOptionalFieldCnt > len(desc.fields) {\n\t\t\thasExtra = true\n\t\t}\n\t\treturn true, hasExtra\n\n\tcase CompoundDesc:\n\t\tswitch v := v.(type) {\n\t\tcase Ref:\n\t\t\t// Switching to the type is subtype of type here.\n\t\t\treturn isSubtypeTopLevel(desc.ElemTypes[0], v.TargetType())\n\t\tcase Map:\n\t\t\tkt := desc.ElemTypes[0]\n\t\t\tvt := desc.ElemTypes[1]\n\t\t\tif seq, ok := v.orderedSequence.(mapLeafSequence); ok {\n\t\t\t\tfor _, entry := range seq.entries() {\n\t\t\t\t\tisSub, hasMore := isValueSubtypeOfDetails(entry.key, kt, hasExtra)\n\t\t\t\t\tif !isSub {\n\t\t\t\t\t\treturn false, hasExtra\n\t\t\t\t\t}\n\t\t\t\t\thasExtra = hasExtra || hasMore\n\t\t\t\t\tisSub, hasExtra = isValueSubtypeOfDetails(entry.value, vt, hasExtra)\n\t\t\t\t\tif !isSub {\n\t\t\t\t\t\treturn false, hasExtra\n\t\t\t\t\t}\n\t\t\t\t\thasExtra = hasExtra || hasMore\n\t\t\t\t}\n\t\t\t\treturn true, hasExtra\n\t\t\t}\n\t\t\treturn isMetaSequenceSubtypeOf(v.orderedSequence.(metaSequence), t, hasExtra)\n\t\tcase Set:\n\t\t\tet := desc.ElemTypes[0]\n\t\t\tif seq, ok := v.orderedSequence.(setLeafSequence); ok {\n\t\t\t\tfor _, v := range seq.values() {\n\t\t\t\t\tisSub, hasMore := isValueSubtypeOfDetails(v, et, hasExtra)\n\t\t\t\t\tif !isSub {\n\t\t\t\t\t\treturn false, hasExtra\n\t\t\t\t\t}\n\t\t\t\t\thasExtra = hasExtra || hasMore\n\t\t\t\t}\n\t\t\t\treturn true, hasExtra\n\t\t\t}\n\t\t\treturn isMetaSequenceSubtypeOf(v.orderedSequence.(metaSequence), t, hasExtra)\n\t\tcase List:\n\t\t\tet := desc.ElemTypes[0]\n\t\t\tif seq, ok := v.sequence.(listLeafSequence); ok {\n\t\t\t\tfor _, v := range seq.values() {\n\t\t\t\t\tisSub, hasMore := isValueSubtypeOfDetails(v, et, hasExtra)\n\t\t\t\t\tif !isSub {\n\t\t\t\t\t\treturn false, hasExtra\n\t\t\t\t\t}\n\t\t\t\t\thasExtra = hasExtra || hasMore\n\t\t\t\t}\n\t\t\t\treturn true, hasExtra\n\t\t\t}\n\t\t\treturn isMetaSequenceSubtypeOf(v.sequence.(metaSequence), t, hasExtra)\n\t\t}\n\t}\n\tpanic(\"unreachable\")\n}\n\nfunc isMetaSequenceSubtypeOf(ms metaSequence, t *Type, hasExtra bool) (bool, bool) {\n\t// TODO: iterRefs\n\tfor _, mt := range ms.tuples() {\n\t\t// Each prolly tree is also a List where T needs to be a subtype.\n\t\tisSub, hasMore := isSubtypeTopLevel(t, mt.ref().TargetType())\n\t\tif !isSub {\n\t\t\treturn false, hasExtra\n\t\t}\n\t\thasExtra = hasExtra || hasMore\n\t}\n\treturn true, hasExtra\n}\n" }, { "path": "go/types/subtype_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc assertInvalid(tt *testing.T, t *Type, v Value) {\n\tassert := assert.New(tt)\n\tassert.Panics(func() {\n\t\tassertSubtype(t, v)\n\t})\n}\n\nfunc assertAll(tt *testing.T, t *Type, v Value) {\n\tallTypes := []*Type{\n\t\tBoolType,\n\t\tNumberType,\n\t\tStringType,\n\t\tBlobType,\n\t\tTypeType,\n\t\tValueType,\n\t}\n\n\tfor _, at := range allTypes {\n\t\tif at == ValueType || t.Equals(at) {\n\t\t\tassertSubtype(at, v)\n\t\t} else {\n\t\t\tassertInvalid(tt, at, v)\n\t\t}\n\t}\n}\n\nfunc TestAssertTypePrimitives(t *testing.T) {\n\tassertSubtype(BoolType, Bool(true))\n\tassertSubtype(BoolType, Bool(false))\n\tassertSubtype(NumberType, Number(42))\n\tassertSubtype(StringType, String(\"abc\"))\n\n\tassertInvalid(t, BoolType, Number(1))\n\tassertInvalid(t, BoolType, String(\"abc\"))\n\tassertInvalid(t, NumberType, Bool(true))\n\tassertInvalid(t, StringType, Number(42))\n}\n\nfunc TestAssertTypeValue(t *testing.T) {\n\tvs := newTestValueStore()\n\n\tassertSubtype(ValueType, Bool(true))\n\tassertSubtype(ValueType, Number(1))\n\tassertSubtype(ValueType, String(\"abc\"))\n\tl := NewList(vs, Number(0), Number(1), Number(2), Number(3))\n\tassertSubtype(ValueType, l)\n}\n\nfunc TestAssertTypeBlob(t *testing.T) {\n\tvs := newTestValueStore()\n\n\tblob := NewBlob(vs, bytes.NewBuffer([]byte{0x00, 0x01}))\n\tassertAll(t, BlobType, blob)\n}\n\nfunc TestAssertTypeList(tt *testing.T) {\n\tvs := newTestValueStore()\n\n\tlistOfNumberType := MakeListType(NumberType)\n\tl := NewList(vs, Number(0), Number(1), Number(2), Number(3))\n\tassertSubtype(listOfNumberType, l)\n\tassertAll(tt, listOfNumberType, l)\n\tassertSubtype(MakeListType(ValueType), l)\n}\n\nfunc TestAssertTypeMap(tt *testing.T) {\n\tvs := newTestValueStore()\n\n\tmapOfNumberToStringType := MakeMapType(NumberType, StringType)\n\tm := NewMap(vs, Number(0), String(\"a\"), Number(2), String(\"b\"))\n\tassertSubtype(mapOfNumberToStringType, m)\n\tassertAll(tt, mapOfNumberToStringType, m)\n\tassertSubtype(MakeMapType(ValueType, ValueType), m)\n}\n\nfunc TestAssertTypeSet(tt *testing.T) {\n\tvs := newTestValueStore()\n\n\tsetOfNumberType := MakeSetType(NumberType)\n\ts := NewSet(vs, Number(0), Number(1), Number(2), Number(3))\n\tassertSubtype(setOfNumberType, s)\n\tassertAll(tt, setOfNumberType, s)\n\tassertSubtype(MakeSetType(ValueType), s)\n}\n\nfunc TestAssertTypeType(tt *testing.T) {\n\tt := MakeSetType(NumberType)\n\tassertSubtype(TypeType, t)\n\tassertAll(tt, TypeType, t)\n\tassertSubtype(ValueType, t)\n}\n\nfunc TestAssertTypeStruct(tt *testing.T) {\n\tt := MakeStructType(\"Struct\", StructField{\"x\", BoolType, false})\n\n\tv := NewStruct(\"Struct\", StructData{\"x\": Bool(true)})\n\tassertSubtype(t, v)\n\tassertAll(tt, t, v)\n\tassertSubtype(ValueType, v)\n}\n\nfunc TestAssertTypeUnion(tt *testing.T) {\n\tvs := newTestValueStore()\n\n\tassertSubtype(MakeUnionType(NumberType), Number(42))\n\tassertSubtype(MakeUnionType(NumberType, StringType), Number(42))\n\tassertSubtype(MakeUnionType(NumberType, StringType), String(\"hi\"))\n\tassertSubtype(MakeUnionType(NumberType, StringType, BoolType), Number(555))\n\tassertSubtype(MakeUnionType(NumberType, StringType, BoolType), String(\"hi\"))\n\tassertSubtype(MakeUnionType(NumberType, StringType, BoolType), Bool(true))\n\n\tlt := MakeListType(MakeUnionType(NumberType, StringType))\n\tassertSubtype(lt, NewList(vs, Number(1), String(\"hi\"), Number(2), String(\"bye\")))\n\n\tst := MakeSetType(StringType)\n\tassertSubtype(MakeUnionType(st, NumberType), Number(42))\n\tassertSubtype(MakeUnionType(st, NumberType), NewSet(vs, String(\"a\"), String(\"b\")))\n\n\tassertInvalid(tt, MakeUnionType(), Number(42))\n\tassertInvalid(tt, MakeUnionType(StringType), Number(42))\n\tassertInvalid(tt, MakeUnionType(StringType, BoolType), Number(42))\n\tassertInvalid(tt, MakeUnionType(st, StringType), Number(42))\n\tassertInvalid(tt, MakeUnionType(st, NumberType), NewSet(vs, Number(1), Number(2)))\n}\n\nfunc TestAssertConcreteTypeIsUnion(tt *testing.T) {\n\tassert.True(tt, IsSubtype(\n\t\tMakeStructTypeFromFields(\"\", FieldMap{}),\n\t\tMakeUnionType(\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"bar\": StringType}))))\n\n\tassert.False(tt, IsSubtype(\n\t\tMakeStructTypeFromFields(\"\", FieldMap{}),\n\t\tMakeUnionType(MakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType}),\n\t\t\tNumberType)))\n\n\tassert.True(tt, IsSubtype(\n\t\tMakeUnionType(\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"bar\": StringType})),\n\t\tMakeUnionType(\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType, \"bar\": StringType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"bar\": StringType}))))\n\n\tassert.False(tt, IsSubtype(\n\t\tMakeUnionType(\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType}),\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"bar\": StringType})),\n\t\tMakeUnionType(\n\t\t\tMakeStructTypeFromFields(\"\", FieldMap{\"foo\": StringType, \"bar\": StringType}),\n\t\t\tNumberType)))\n}\n\nfunc TestAssertTypeEmptyListUnion(tt *testing.T) {\n\tvs := newTestValueStore()\n\n\tlt := MakeListType(MakeUnionType())\n\tassertSubtype(lt, NewList(vs))\n}\n\nfunc TestAssertTypeEmptyList(tt *testing.T) {\n\tvs := newTestValueStore()\n\n\tlt := MakeListType(NumberType)\n\tassertSubtype(lt, NewList(vs))\n\n\t// List<> not a subtype of List\n\tassertInvalid(tt, MakeListType(MakeUnionType()), NewList(vs, Number(1)))\n}\n\nfunc TestAssertTypeEmptySet(tt *testing.T) {\n\tvs := newTestValueStore()\n\n\tst := MakeSetType(NumberType)\n\tassertSubtype(st, NewSet(vs))\n\n\t// Set<> not a subtype of Set\n\tassertInvalid(tt, MakeSetType(MakeUnionType()), NewSet(vs, Number(1)))\n}\n\nfunc TestAssertTypeEmptyMap(tt *testing.T) {\n\tvs := newTestValueStore()\n\n\tmt := MakeMapType(NumberType, StringType)\n\tassertSubtype(mt, NewMap(vs))\n\n\t// Map<> not a subtype of Map\n\tassertInvalid(tt, MakeMapType(MakeUnionType(), MakeUnionType()), NewMap(vs, Number(1), Number(2)))\n}\n\nfunc TestAssertTypeStructSubtypeByName(tt *testing.T) {\n\tnamedT := MakeStructType(\"Name\", StructField{\"x\", NumberType, false})\n\tanonT := MakeStructType(\"\", StructField{\"x\", NumberType, false})\n\tnamedV := NewStruct(\"Name\", StructData{\"x\": Number(42)})\n\tname2V := NewStruct(\"foo\", StructData{\"x\": Number(42)})\n\tanonV := NewStruct(\"\", StructData{\"x\": Number(42)})\n\n\tassertSubtype(namedT, namedV)\n\tassertInvalid(tt, namedT, name2V)\n\tassertInvalid(tt, namedT, anonV)\n\n\tassertSubtype(anonT, namedV)\n\tassertSubtype(anonT, name2V)\n\tassertSubtype(anonT, anonV)\n}\n\nfunc TestAssertTypeStructSubtypeExtraFields(tt *testing.T) {\n\tat := MakeStructType(\"\")\n\tbt := MakeStructType(\"\", StructField{\"x\", NumberType, false})\n\tct := MakeStructType(\"\", StructField{\"s\", StringType, false}, StructField{\"x\", NumberType, false})\n\tav := NewStruct(\"\", StructData{})\n\tbv := NewStruct(\"\", StructData{\"x\": Number(1)})\n\tcv := NewStruct(\"\", StructData{\"x\": Number(2), \"s\": String(\"hi\")})\n\n\tassertSubtype(at, av)\n\tassertInvalid(tt, bt, av)\n\tassertInvalid(tt, ct, av)\n\n\tassertSubtype(at, bv)\n\tassertSubtype(bt, bv)\n\tassertInvalid(tt, ct, bv)\n\n\tassertSubtype(at, cv)\n\tassertSubtype(bt, cv)\n\tassertSubtype(ct, cv)\n}\n\nfunc TestAssertTypeStructSubtype(tt *testing.T) {\n\tvs := newTestValueStore()\n\n\tc1 := NewStruct(\"Commit\", StructData{\n\t\t\"value\": Number(1),\n\t\t\"parents\": NewSet(vs),\n\t})\n\tt1 := MakeStructType(\"Commit\",\n\t\tStructField{\"parents\", MakeSetType(MakeUnionType()), false},\n\t\tStructField{\"value\", NumberType, false},\n\t)\n\tassertSubtype(t1, c1)\n\n\tt11 := MakeStructType(\"Commit\",\n\t\tStructField{\"parents\", MakeSetType(MakeRefType(MakeCycleType(\"Commit\"))), false},\n\t\tStructField{\"value\", NumberType, false},\n\t)\n\tassertSubtype(t11, c1)\n\n\tc2 := NewStruct(\"Commit\", StructData{\n\t\t\"value\": Number(2),\n\t\t\"parents\": NewSet(vs, NewRef(c1)),\n\t})\n\tassertSubtype(t11, c2)\n}\n\nfunc TestAssertTypeCycleUnion(tt *testing.T) {\n\t// struct S {\n\t// x: Cycle,\n\t// y: Number,\n\t// }\n\tt1 := MakeStructType(\"S\",\n\t\tStructField{\"x\", MakeCycleType(\"S\"), false},\n\t\tStructField{\"y\", NumberType, false},\n\t)\n\t// struct S {\n\t// x: Cycle,\n\t// y: Number | String,\n\t// }\n\tt2 := MakeStructType(\"S\",\n\t\tStructField{\"x\", MakeCycleType(\"S\"), false},\n\t\tStructField{\"y\", MakeUnionType(NumberType, StringType), false},\n\t)\n\n\tassert.True(tt, IsSubtype(t2, t1))\n\tassert.False(tt, IsSubtype(t1, t2))\n\n\t// struct S {\n\t// x: Cycle | Number,\n\t// y: Number | String,\n\t// }\n\tt3 := MakeStructType(\"S\",\n\t\tStructField{\"x\", MakeUnionType(MakeCycleType(\"S\"), NumberType), false},\n\t\tStructField{\"y\", MakeUnionType(NumberType, StringType), false},\n\t)\n\n\tassert.True(tt, IsSubtype(t3, t1))\n\tassert.False(tt, IsSubtype(t1, t3))\n\n\tassert.True(tt, IsSubtype(t3, t2))\n\tassert.False(tt, IsSubtype(t2, t3))\n\n\t// struct S {\n\t// x: Cycle | Number,\n\t// y: Number,\n\t// }\n\tt4 := MakeStructType(\"S\",\n\t\tStructField{\"x\", MakeUnionType(MakeCycleType(\"S\"), NumberType), false},\n\t\tStructField{\"y\", NumberType, false},\n\t)\n\n\tassert.True(tt, IsSubtype(t4, t1))\n\tassert.False(tt, IsSubtype(t1, t4))\n\n\tassert.False(tt, IsSubtype(t4, t2))\n\tassert.False(tt, IsSubtype(t2, t4))\n\n\tassert.True(tt, IsSubtype(t3, t4))\n\tassert.False(tt, IsSubtype(t4, t3))\n\n\t// struct B {\n\t// b: struct C {\n\t// c: Cycle,\n\t// },\n\t// }\n\n\t// struct C {\n\t// c: struct B {\n\t// b: Cycle,\n\t// },\n\t// }\n\n\ttb := MakeStructType(\"A\",\n\t\tStructField{\n\t\t\t\"b\",\n\t\t\tMakeStructType(\"B\", StructField{\"c\", MakeCycleType(\"A\"), false}),\n\t\t\tfalse,\n\t\t},\n\t)\n\ttc := MakeStructType(\"A\",\n\t\tStructField{\n\t\t\t\"c\",\n\t\t\tMakeStructType(\"B\", StructField{\"b\", MakeCycleType(\"A\"), false}),\n\t\t\tfalse,\n\t\t},\n\t)\n\n\tassert.False(tt, IsSubtype(tb, tc))\n\tassert.False(tt, IsSubtype(tc, tb))\n}\n\nfunc TestIsSubtypeEmptySruct(tt *testing.T) {\n\t// struct {\n\t// a: Number,\n\t// b: struct {},\n\t// }\n\tt1 := MakeStructType(\"X\",\n\t\tStructField{\"a\", NumberType, false},\n\t\tStructField{\"b\", EmptyStructType, false},\n\t)\n\n\t// struct {\n\t// a: Number,\n\t// }\n\tt2 := MakeStructType(\"X\", StructField{\"a\", NumberType, false})\n\n\tassert.False(tt, IsSubtype(t1, t2))\n\tassert.True(tt, IsSubtype(t2, t1))\n}\n\nfunc TestIsSubtypeCompoundUnion(tt *testing.T) {\n\trt := MakeListType(EmptyStructType)\n\n\tst1 := MakeStructType(\"One\", StructField{\"a\", NumberType, false})\n\tst2 := MakeStructType(\"Two\", StructField{\"b\", StringType, false})\n\tct := MakeListType(MakeUnionType(st1, st2))\n\n\tassert.True(tt, IsSubtype(rt, ct))\n\tassert.False(tt, IsSubtype(ct, rt))\n\n\tct2 := MakeListType(MakeUnionType(st1, st2, NumberType))\n\tassert.False(tt, IsSubtype(rt, ct2))\n\tassert.False(tt, IsSubtype(ct2, rt))\n}\n\nfunc TestIsSubtypeOptionalFields(tt *testing.T) {\n\tassert := assert.New(tt)\n\n\ts1 := MakeStructType(\"\", StructField{\"a\", NumberType, true})\n\ts2 := MakeStructType(\"\", StructField{\"a\", NumberType, false})\n\tassert.True(IsSubtype(s1, s2))\n\tassert.False(IsSubtype(s2, s1))\n\n\ts3 := MakeStructType(\"\", StructField{\"a\", StringType, false})\n\tassert.False(IsSubtype(s1, s3))\n\tassert.False(IsSubtype(s3, s1))\n\n\ts4 := MakeStructType(\"\", StructField{\"a\", StringType, true})\n\tassert.False(IsSubtype(s1, s4))\n\tassert.False(IsSubtype(s4, s1))\n\n\ttest := func(t1s, t2s string, exp1, exp2 bool) {\n\t\tt1 := makeTestStructTypeFromFieldNames(t1s)\n\t\tt2 := makeTestStructTypeFromFieldNames(t2s)\n\t\tassert.Equal(exp1, IsSubtype(t1, t2))\n\t\tassert.Equal(exp2, IsSubtype(t2, t1))\n\t\tassert.False(t1.Equals(t2))\n\t}\n\n\ttest(\"n?\", \"n\", true, false)\n\ttest(\"\", \"n\", true, false)\n\ttest(\"\", \"n?\", true, true)\n\n\ttest(\"a b?\", \"a\", true, true)\n\ttest(\"a b?\", \"a b\", true, false)\n\ttest(\"a b? c\", \"a b c\", true, false)\n\ttest(\"b? c\", \"a b c\", true, false)\n\ttest(\"b? c\", \"b c\", true, false)\n\n\ttest(\"a c e\", \"a b c d e\", true, false)\n\ttest(\"a c e?\", \"a b c d e\", true, false)\n\ttest(\"a c? e\", \"a b c d e\", true, false)\n\ttest(\"a c? e?\", \"a b c d e\", true, false)\n\ttest(\"a? c e\", \"a b c d e\", true, false)\n\ttest(\"a? c e?\", \"a b c d e\", true, false)\n\ttest(\"a? c? e\", \"a b c d e\", true, false)\n\ttest(\"a? c? e?\", \"a b c d e\", true, false)\n\n\ttest(\"a c e?\", \"a b c d\", true, false)\n\ttest(\"a c? e\", \"a b d e\", true, false)\n\ttest(\"a c? e?\", \"a b d\", true, false)\n\ttest(\"a? c e\", \"b c d e\", true, false)\n\ttest(\"a? c e?\", \"b c d\", true, false)\n\ttest(\"a? c? e\", \"b d e\", true, false)\n\ttest(\"a? c? e?\", \"b d\", true, false)\n\n\tt1 := MakeStructType(\"\", StructField{\"a\", BoolType, true})\n\tt2 := MakeStructType(\"\", StructField{\"a\", NumberType, true})\n\tassert.False(IsSubtype(t1, t2))\n\tassert.False(IsSubtype(t2, t1))\n}\n\nfunc makeTestStructTypeFromFieldNames(s string) *Type {\n\tif s == \"\" {\n\t\treturn MakeStructType(\"\")\n\t}\n\n\tfs := strings.Split(s, \" \")\n\tfields := make([]StructField, len(fs))\n\tfor i, f := range fs {\n\t\toptional := false\n\t\tif f[len(f)-1:] == \"?\" {\n\t\t\tf = f[:len(f)-1]\n\t\t\toptional = true\n\t\t}\n\t\tfields[i] = StructField{f, BoolType, optional}\n\t}\n\treturn MakeStructType(\"\", fields...)\n}\n\nfunc makeTestStructFromFieldNames(s string) Struct {\n\tt := makeTestStructTypeFromFieldNames(s)\n\tfields := t.Desc.(StructDesc).fields\n\td.Chk.NotEmpty(fields)\n\n\tfieldNames := make([]string, len(fields))\n\tfor i, field := range fields {\n\t\tfieldNames[i] = field.Name\n\t}\n\tvals := make([]Value, len(fields))\n\tfor i := range fields {\n\t\tvals[i] = Bool(true)\n\t}\n\n\treturn newStruct(\"\", fieldNames, vals)\n}\n\nfunc TestIsSubtypeDisallowExtraStructFields(tt *testing.T) {\n\tassert := assert.New(tt)\n\n\ttest := func(t1s, t2s string, exp1, exp2 bool) {\n\t\tt1 := makeTestStructTypeFromFieldNames(t1s)\n\t\tt2 := makeTestStructTypeFromFieldNames(t2s)\n\t\tassert.Equal(exp1, IsSubtypeDisallowExtraStructFields(t1, t2))\n\t\tassert.Equal(exp2, IsSubtypeDisallowExtraStructFields(t2, t1))\n\t\tassert.False(t1.Equals(t2))\n\t}\n\n\ttest(\"n?\", \"n\", true, false)\n\ttest(\"\", \"n\", false, false)\n\ttest(\"\", \"n?\", false, true)\n\n\ttest(\"a b?\", \"a\", true, false)\n\ttest(\"a b?\", \"a b\", true, false)\n\ttest(\"a b? c\", \"a b c\", true, false)\n\ttest(\"b? c\", \"a b c\", false, false)\n\ttest(\"b? c\", \"b c\", true, false)\n\n\ttest(\"a c e\", \"a b c d e\", false, false)\n\ttest(\"a c e?\", \"a b c d e\", false, false)\n\ttest(\"a c? e\", \"a b c d e\", false, false)\n\ttest(\"a c? e?\", \"a b c d e\", false, false)\n\ttest(\"a? c e\", \"a b c d e\", false, false)\n\ttest(\"a? c e?\", \"a b c d e\", false, false)\n\ttest(\"a? c? e\", \"a b c d e\", false, false)\n\ttest(\"a? c? e?\", \"a b c d e\", false, false)\n\n\ttest(\"a c e?\", \"a b c d\", false, false)\n\ttest(\"a c? e\", \"a b d e\", false, false)\n\ttest(\"a c? e?\", \"a b d\", false, false)\n\ttest(\"a? c e\", \"b c d e\", false, false)\n\ttest(\"a? c e?\", \"b c d\", false, false)\n\ttest(\"a? c? e\", \"b d e\", false, false)\n\ttest(\"a? c? e?\", \"b d\", false, false)\n}\n\nfunc TestIsValueSubtypeOf(tt *testing.T) {\n\tassert := assert.New(tt)\n\n\tvs := newTestValueStore()\n\n\tassertTrue := func(v Value, t *Type) {\n\t\tassert.True(IsValueSubtypeOf(v, t))\n\t}\n\n\tassertFalse := func(v Value, t *Type) {\n\t\tassert.False(IsValueSubtypeOf(v, t))\n\t}\n\n\tallTypes := []struct {\n\t\tv Value\n\t\tt *Type\n\t}{\n\t\t{Bool(true), BoolType},\n\t\t{Number(42), NumberType},\n\t\t{String(\"s\"), StringType},\n\t\t{NewEmptyBlob(vs), BlobType},\n\t\t{BoolType, TypeType},\n\t\t{NewList(vs, Number(42)), MakeListType(NumberType)},\n\t\t{NewSet(vs, Number(42)), MakeSetType(NumberType)},\n\t\t{NewRef(Number(42)), MakeRefType(NumberType)},\n\t\t{NewMap(vs, Number(42), String(\"a\")), MakeMapType(NumberType, StringType)},\n\t\t{NewStruct(\"A\", StructData{}), MakeStructType(\"A\")},\n\t\t// Not including CycleType or Union here\n\t}\n\tfor i, rec := range allTypes {\n\t\tfor j, rec2 := range allTypes {\n\t\t\tif i == j {\n\t\t\t\tassertTrue(rec.v, rec.t)\n\t\t\t} else {\n\t\t\t\tassertFalse(rec.v, rec2.t)\n\t\t\t\tassertFalse(rec2.v, rec.t)\n\t\t\t}\n\t\t}\n\t}\n\n\tfor _, rec := range allTypes {\n\t\tassertTrue(rec.v, ValueType)\n\t}\n\n\tassertTrue(Bool(true), MakeUnionType(BoolType, NumberType))\n\tassertTrue(Number(123), MakeUnionType(BoolType, NumberType))\n\tassertFalse(String(\"abc\"), MakeUnionType(BoolType, NumberType))\n\tassertFalse(String(\"abc\"), MakeUnionType())\n\n\tassertTrue(NewList(vs), MakeListType(NumberType))\n\tassertTrue(NewList(vs, Number(0), Number(1), Number(2), Number(3)), MakeListType(NumberType))\n\tassertFalse(NewList(vs, Number(0), Number(1), Number(2), Number(3)), MakeListType(BoolType))\n\tassertTrue(NewList(vs, Number(0), Number(1), Number(2), Number(3)), MakeListType(MakeUnionType(NumberType, BoolType)))\n\tassertTrue(NewList(vs, Number(0), Bool(true)), MakeListType(MakeUnionType(NumberType, BoolType)))\n\tassertFalse(NewList(vs, Number(0)), MakeListType(MakeUnionType()))\n\tassertTrue(NewList(vs), MakeListType(MakeUnionType()))\n\n\t{\n\t\tnewChunkedList := func(vals ...Value) List {\n\t\t\tnewSequenceMetaTuple := func(v Value) metaTuple {\n\t\t\t\tseq := newListLeafSequence(vs, v)\n\t\t\t\tlist := newList(seq)\n\t\t\t\treturn newMetaTuple(vs.WriteValue(list), newOrderedKey(v), 1)\n\t\t\t}\n\n\t\t\ttuples := make([]metaTuple, len(vals))\n\t\t\tfor i, v := range vals {\n\t\t\t\ttuples[i] = newSequenceMetaTuple(v)\n\t\t\t}\n\t\t\treturn newList(newListMetaSequence(1, tuples, vs))\n\t\t}\n\n\t\tassertTrue(newChunkedList(Number(0), Number(1), Number(2), Number(3)), MakeListType(NumberType))\n\t\tassertFalse(newChunkedList(Number(0), Number(1), Number(2), Number(3)), MakeListType(BoolType))\n\t\tassertTrue(newChunkedList(Number(0), Number(1), Number(2), Number(3)), MakeListType(MakeUnionType(NumberType, BoolType)))\n\t\tassertTrue(newChunkedList(Number(0), Bool(true)), MakeListType(MakeUnionType(NumberType, BoolType)))\n\t\tassertFalse(newChunkedList(Number(0)), MakeListType(MakeUnionType()))\n\t}\n\n\tassertTrue(NewSet(vs), MakeSetType(NumberType))\n\tassertTrue(NewSet(vs, Number(0), Number(1), Number(2), Number(3)), MakeSetType(NumberType))\n\tassertFalse(NewSet(vs, Number(0), Number(1), Number(2), Number(3)), MakeSetType(BoolType))\n\tassertTrue(NewSet(vs, Number(0), Number(1), Number(2), Number(3)), MakeSetType(MakeUnionType(NumberType, BoolType)))\n\tassertTrue(NewSet(vs, Number(0), Bool(true)), MakeSetType(MakeUnionType(NumberType, BoolType)))\n\tassertFalse(NewSet(vs, Number(0)), MakeSetType(MakeUnionType()))\n\tassertTrue(NewSet(vs), MakeSetType(MakeUnionType()))\n\n\t{\n\t\tnewChunkedSet := func(vals ...Value) Set {\n\t\t\tnewSequenceMetaTuple := func(v Value) metaTuple {\n\t\t\t\tseq := newSetLeafSequence(vs, v)\n\t\t\t\tset := newSet(seq)\n\t\t\t\treturn newMetaTuple(vs.WriteValue(set), newOrderedKey(v), 1)\n\t\t\t}\n\n\t\t\ttuples := make([]metaTuple, len(vals))\n\t\t\tfor i, v := range vals {\n\t\t\t\ttuples[i] = newSequenceMetaTuple(v)\n\t\t\t}\n\t\t\treturn newSet(newSetMetaSequence(1, tuples, vs))\n\t\t}\n\t\tassertTrue(newChunkedSet(Number(0), Number(1), Number(2), Number(3)), MakeSetType(NumberType))\n\t\tassertFalse(newChunkedSet(Number(0), Number(1), Number(2), Number(3)), MakeSetType(BoolType))\n\t\tassertTrue(newChunkedSet(Number(0), Number(1), Number(2), Number(3)), MakeSetType(MakeUnionType(NumberType, BoolType)))\n\t\tassertTrue(newChunkedSet(Number(0), Bool(true)), MakeSetType(MakeUnionType(NumberType, BoolType)))\n\t\tassertFalse(newChunkedSet(Number(0)), MakeSetType(MakeUnionType()))\n\t}\n\n\tassertTrue(NewMap(vs), MakeMapType(NumberType, StringType))\n\tassertTrue(NewMap(vs, Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(NumberType, StringType))\n\tassertFalse(NewMap(vs, Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(BoolType, StringType))\n\tassertFalse(NewMap(vs, Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(NumberType, BoolType))\n\tassertTrue(NewMap(vs, Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(MakeUnionType(NumberType, BoolType), StringType))\n\tassertTrue(NewMap(vs, Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(NumberType, MakeUnionType(BoolType, StringType)))\n\tassertTrue(NewMap(vs, Number(0), String(\"a\"), Bool(true), String(\"b\")), MakeMapType(MakeUnionType(NumberType, BoolType), StringType))\n\tassertTrue(NewMap(vs, Number(0), String(\"a\"), Number(1), Bool(true)), MakeMapType(NumberType, MakeUnionType(BoolType, StringType)))\n\tassertFalse(NewMap(vs, Number(0), String(\"a\")), MakeMapType(MakeUnionType(), StringType))\n\tassertFalse(NewMap(vs, Number(0), String(\"a\")), MakeMapType(NumberType, MakeUnionType()))\n\tassertTrue(NewMap(vs), MakeMapType(MakeUnionType(), MakeUnionType()))\n\n\t{\n\t\tnewChunkedMap := func(vals ...Value) Map {\n\t\t\tnewSequenceMetaTuple := func(e mapEntry) metaTuple {\n\t\t\t\tseq := newMapLeafSequence(vs, e)\n\t\t\t\tm := newMap(seq)\n\t\t\t\treturn newMetaTuple(vs.WriteValue(m), newOrderedKey(e.key), 1)\n\t\t\t}\n\n\t\t\ttuples := make([]metaTuple, len(vals)/2)\n\t\t\tfor i := 0; i < len(vals); i += 2 {\n\t\t\t\ttuples[i/2] = newSequenceMetaTuple(mapEntry{vals[i], vals[i+1]})\n\t\t\t}\n\t\t\treturn newMap(newMapMetaSequence(1, tuples, vs))\n\t\t}\n\n\t\tassertTrue(newChunkedMap(Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(NumberType, StringType))\n\t\tassertFalse(newChunkedMap(Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(BoolType, StringType))\n\t\tassertFalse(newChunkedMap(Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(NumberType, BoolType))\n\t\tassertTrue(newChunkedMap(Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(MakeUnionType(NumberType, BoolType), StringType))\n\t\tassertTrue(newChunkedMap(Number(0), String(\"a\"), Number(1), String(\"b\")), MakeMapType(NumberType, MakeUnionType(BoolType, StringType)))\n\t\tassertTrue(newChunkedMap(Number(0), String(\"a\"), Bool(true), String(\"b\")), MakeMapType(MakeUnionType(NumberType, BoolType), StringType))\n\t\tassertTrue(newChunkedMap(Number(0), String(\"a\"), Number(1), Bool(true)), MakeMapType(NumberType, MakeUnionType(BoolType, StringType)))\n\t\tassertFalse(newChunkedMap(Number(0), String(\"a\")), MakeMapType(MakeUnionType(), StringType))\n\t\tassertFalse(newChunkedMap(Number(0), String(\"a\")), MakeMapType(NumberType, MakeUnionType()))\n\t}\n\n\tassertTrue(NewRef(Number(1)), MakeRefType(NumberType))\n\tassertFalse(NewRef(Number(1)), MakeRefType(BoolType))\n\tassertTrue(NewRef(Number(1)), MakeRefType(MakeUnionType(NumberType, BoolType)))\n\tassertFalse(NewRef(Number(1)), MakeRefType(MakeUnionType()))\n\n\tassertTrue(\n\t\tNewStruct(\"Struct\", StructData{\"x\": Bool(true)}),\n\t\tMakeStructType(\"Struct\", StructField{\"x\", BoolType, false}),\n\t)\n\tassertTrue(\n\t\tNewStruct(\"Struct\", StructData{\"x\": Bool(true)}),\n\t\tMakeStructType(\"Struct\", StructField{\"x\", BoolType, true}),\n\t)\n\tassertTrue(\n\t\tNewStruct(\"Struct\", StructData{\"x\": Bool(true)}),\n\t\tMakeStructType(\"Struct\"),\n\t)\n\tassertTrue(\n\t\tNewStruct(\"Struct\", StructData{}),\n\t\tMakeStructType(\"Struct\"),\n\t)\n\tassertFalse(\n\t\tNewStruct(\"\", StructData{\"x\": Bool(true)}),\n\t\tMakeStructType(\"Struct\"),\n\t)\n\tassertFalse(\n\t\tNewStruct(\"struct\", StructData{\"x\": Bool(true)}), // lower case name\n\t\tMakeStructType(\"Struct\"),\n\t)\n\tassertTrue(\n\t\tNewStruct(\"Struct\", StructData{\"x\": Bool(true)}),\n\t\tMakeStructType(\"Struct\", StructField{\"x\", MakeUnionType(BoolType, NumberType), true}),\n\t)\n\tassertTrue(\n\t\tNewStruct(\"Struct\", StructData{\"x\": Bool(true)}),\n\t\tMakeStructType(\"Struct\", StructField{\"y\", BoolType, true}),\n\t)\n\tassertFalse(\n\t\tNewStruct(\"Struct\", StructData{\"x\": Bool(true)}),\n\t\tMakeStructType(\"Struct\", StructField{\"x\", StringType, true}),\n\t)\n\n\tassertTrue(\n\t\tNewStruct(\"Node\", StructData{\n\t\t\t\"value\": Number(1),\n\t\t\t\"children\": NewList(vs,\n\t\t\t\tNewStruct(\"Node\", StructData{\n\t\t\t\t\t\"value\": Number(2),\n\t\t\t\t\t\"children\": NewList(vs),\n\t\t\t\t}),\n\t\t\t),\n\t\t}),\n\t\tMakeStructType(\"Node\",\n\t\t\tStructField{\"value\", NumberType, false},\n\t\t\tStructField{\"children\", MakeListType(MakeCycleType(\"Node\")), false},\n\t\t),\n\t)\n\n\tassertFalse( // inner Node has wrong type.\n\t\tNewStruct(\"Node\", StructData{\n\t\t\t\"value\": Number(1),\n\t\t\t\"children\": NewList(vs,\n\t\t\t\tNewStruct(\"Node\", StructData{\n\t\t\t\t\t\"value\": Bool(true),\n\t\t\t\t\t\"children\": NewList(vs),\n\t\t\t\t}),\n\t\t\t),\n\t\t}),\n\t\tMakeStructType(\"Node\",\n\t\t\tStructField{\"value\", NumberType, false},\n\t\t\tStructField{\"children\", MakeListType(MakeCycleType(\"Node\")), false},\n\t\t),\n\t)\n\n\t{\n\t\tnode := func(value Value, children ...Value) Value {\n\t\t\tchildrenAsRefs := make(ValueSlice, len(children))\n\t\t\tfor i, c := range children {\n\t\t\t\tchildrenAsRefs[i] = NewRef(c)\n\t\t\t}\n\t\t\trv := NewStruct(\"Node\", StructData{\n\t\t\t\t\"value\": value,\n\t\t\t\t\"children\": NewList(vs, childrenAsRefs...),\n\t\t\t})\n\t\t\treturn rv\n\t\t}\n\n\t\trequiredType := MakeStructType(\"Node\",\n\t\t\tStructField{\"value\", NumberType, false},\n\t\t\tStructField{\"children\", MakeListType(MakeRefType(MakeCycleType(\"Node\"))), false},\n\t\t)\n\n\t\tassertTrue(\n\t\t\tnode(Number(0), node(Number(1)), node(Number(2), node(Number(3)))),\n\t\t\trequiredType,\n\t\t)\n\t\tassertFalse(\n\t\t\tnode(Number(0),\n\t\t\t\tnode(Number(1)),\n\t\t\t\tnode(Number(2), node(String(\"no\"))),\n\t\t\t),\n\t\t\trequiredType,\n\t\t)\n\t}\n\n\t{\n\t\tt1 := MakeStructType(\"A\",\n\t\t\tStructField{\"a\", NumberType, false},\n\t\t\tStructField{\"b\", MakeCycleType(\"A\"), false},\n\t\t)\n\t\tt2 := MakeStructType(\"A\",\n\t\t\tStructField{\"a\", NumberType, false},\n\t\t\tStructField{\"b\", MakeCycleType(\"A\"), true},\n\t\t)\n\t\tv := NewStruct(\"A\", StructData{\n\t\t\t\"a\": Number(1),\n\t\t\t\"b\": NewStruct(\"A\", StructData{\n\t\t\t\t\"a\": Number(2),\n\t\t\t}),\n\t\t})\n\n\t\tassertFalse(v, t1)\n\t\tassertTrue(v, t2)\n\t}\n\n\t{\n\t\tt := MakeStructType(\"A\",\n\t\t\tStructField{\"aa\", NumberType, true},\n\t\t\tStructField{\"bb\", BoolType, false},\n\t\t)\n\t\tv := NewStruct(\"A\", StructData{\n\t\t\t\"a\": Number(1),\n\t\t\t\"b\": Bool(true),\n\t\t})\n\t\tassertFalse(v, t)\n\t}\n}\n\nfunc TestIsValueSubtypeOfDetails(tt *testing.T) {\n\ta := assert.New(tt)\n\n\ttest := func(vString, tString string, exp1, exp2 bool) {\n\t\tv := makeTestStructFromFieldNames(vString)\n\t\tt := makeTestStructTypeFromFieldNames(tString)\n\t\tisSub, hasExtra := IsValueSubtypeOfDetails(v, t)\n\t\ta.Equal(exp1, isSub, \"expected %t for IsSub, received: %t\", exp1, isSub)\n\t\tif isSub {\n\t\t\ta.Equal(exp2, hasExtra, \"expected %t for hasExtra, received: %t\", exp2, hasExtra)\n\t\t}\n\t}\n\n\ttest(\"x\", \"x\", true, false)\n\ttest(\"x\", \"\", true, true)\n\ttest(\"x\", \"x? y?\", true, false)\n\ttest(\"x z\", \"x? y?\", true, true)\n\ttest(\"x\", \"x y\", false, false)\n}\n" }, { "path": "go/types/type.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package types contains most of the data structures available to/from Noms.\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\n// Type defines and describes Noms types, both built-in and user-defined.\n// Desc provides the composition of the type. It may contain only a types.NomsKind, in the case of\n// primitives, or it may contain additional information -- e.g. element Types for compound type\n// specializations, field descriptions for structs, etc. Either way, checking Kind() allows code\n// to understand how to interpret the rest of the data.\n// If Kind() refers to a primitive, then Desc has no more info.\n// If Kind() refers to List, Map, Ref, Set, or Union, then Desc is a list of Types describing the element type(s).\n// If Kind() refers to Struct, then Desc contains a []field.\n\ntype Type struct {\n\tDesc TypeDesc\n}\n\nfunc newType(desc TypeDesc) *Type {\n\treturn &Type{desc}\n}\n\n// Describe generate text that should parse into the struct being described.\nfunc (t *Type) Describe() (out string) {\n\treturn EncodedValue(t)\n}\n\nfunc (t *Type) TargetKind() NomsKind {\n\treturn t.Desc.Kind()\n}\n\n// Value interface\nfunc (t *Type) Value() Value {\n\treturn t\n}\n\nfunc (t *Type) Equals(other Value) (res bool) {\n\t// This is highly optimized to not having to encode a *Type unless we have too.\n\tif t == other {\n\t\treturn true\n\t}\n\n\tif otherType, ok := other.(*Type); ok {\n\t\treturn t.TargetKind() == otherType.TargetKind() && t.Hash() == other.Hash()\n\t}\n\n\treturn false\n}\n\nfunc (t *Type) Less(other Value) (res bool) {\n\treturn valueLess(t, other)\n}\n\nfunc (t *Type) Hash() hash.Hash {\n\treturn getHash(t)\n}\n\nfunc (t *Type) writeTo(w nomsWriter) {\n\tTypeKind.writeTo(w)\n\tt.writeToAsType(w, map[string]*Type{})\n}\n\nfunc (t *Type) writeToAsType(w nomsWriter, seensStructs map[string]*Type) {\n\tt.Desc.writeTo(w, t, seensStructs)\n}\n\nfunc (t *Type) WalkValues(cb ValueCallback) {\n\tt.Desc.walkValues(cb)\n}\n\nfunc (t *Type) WalkRefs(cb RefCallback) {\n\treturn\n}\n\nfunc (t *Type) typeOf() *Type {\n\treturn TypeType\n}\n\nfunc (t *Type) Kind() NomsKind {\n\treturn TypeKind\n}\n\nfunc (t *Type) valueReadWriter() ValueReadWriter {\n\treturn nil\n}\n\n// TypeOf returns the type describing the value. This is not an exact type but\n// often a simplification of the concrete type.\nfunc TypeOf(v Value) *Type {\n\treturn simplifyType(v.typeOf(), false)\n}\n\n// HasStructCycles determines if the type contains any struct cycles.\nfunc HasStructCycles(t *Type) bool {\n\treturn hasStructCycles(t, nil)\n}\n\nfunc hasStructCycles(t *Type, visited []*Type) bool {\n\tif _, found := indexOfType(t, visited); found {\n\t\treturn true\n\t}\n\n\tswitch desc := t.Desc.(type) {\n\tcase CompoundDesc:\n\t\tfor _, et := range desc.ElemTypes {\n\t\t\tb := hasStructCycles(et, visited)\n\t\t\tif b {\n\t\t\t\treturn true\n\t\t\t}\n\t\t}\n\n\tcase StructDesc:\n\t\tfor _, f := range desc.fields {\n\t\t\tb := hasStructCycles(f.Type, append(visited, t))\n\t\t\tif b {\n\t\t\t\treturn true\n\t\t\t}\n\t\t}\n\n\tcase CycleDesc:\n\t\tpanic(\"unexpected unresolved cycle\")\n\t}\n\n\treturn false\n}\n\nfunc indexOfType(t *Type, tl []*Type) (uint32, bool) {\n\tfor i, tt := range tl {\n\t\tif tt == t {\n\t\t\treturn uint32(i), true\n\t\t}\n\t}\n\treturn 0, false\n}\n" }, { "path": "go/types/type_desc.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sort\"\n)\n\n// TypeDesc describes a type of the kind returned by Kind(), e.g. Map, Number, or a custom type.\ntype TypeDesc interface {\n\tKind() NomsKind\n\twalkValues(cb ValueCallback)\n\twriteTo(w nomsWriter, t *Type, seenStructs map[string]*Type)\n\n\t// isSimplifiedForSure is used to determine if the type should be\n\t// simplified. It may contain false negatives.\n\tisSimplifiedForSure() bool\n\tisSimplifiedInner() bool\n}\n\n// PrimitiveDesc implements TypeDesc for all primitive Noms types:\n// Blob\n// Bool\n// Number\n// String\n// Type\n// Value\ntype PrimitiveDesc NomsKind\n\nfunc (p PrimitiveDesc) Kind() NomsKind {\n\treturn NomsKind(p)\n}\n\nfunc (p PrimitiveDesc) walkValues(cb ValueCallback) {\n}\n\nfunc (p PrimitiveDesc) writeTo(w nomsWriter, t *Type, seenStructs map[string]*Type) {\n\tNomsKind(p).writeTo(w)\n}\n\nfunc (p PrimitiveDesc) isSimplifiedForSure() bool {\n\treturn true\n}\n\nfunc (p PrimitiveDesc) isSimplifiedInner() bool {\n\treturn true\n}\n\n// CompoundDesc describes a List, Map, Set, Ref, or Union type.\n// ElemTypes indicates what type or types are in the container indicated by kind, e.g. Map key and value or Set element.\ntype CompoundDesc struct {\n\tkind NomsKind\n\tElemTypes typeSlice\n}\n\nfunc (c CompoundDesc) Kind() NomsKind {\n\treturn c.kind\n}\n\nfunc (c CompoundDesc) walkValues(cb ValueCallback) {\n\tfor _, t := range c.ElemTypes {\n\t\tcb(t)\n\t}\n}\n\nfunc (c CompoundDesc) writeTo(w nomsWriter, t *Type, seenStructs map[string]*Type) {\n\tc.kind.writeTo(w)\n\tif c.kind == UnionKind {\n\t\tw.writeCount(uint64(len(c.ElemTypes)))\n\t}\n\tfor _, t := range c.ElemTypes {\n\t\tt.writeToAsType(w, seenStructs)\n\t}\n}\n\nfunc (c CompoundDesc) isSimplifiedForSure() bool {\n\tif c.kind == UnionKind {\n\t\treturn len(c.ElemTypes) == 0\n\t}\n\n\tfor _, t := range c.ElemTypes {\n\t\tif !t.Desc.isSimplifiedInner() {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\nfunc (c CompoundDesc) isSimplifiedInner() bool {\n\treturn c.isSimplifiedForSure()\n}\n\n// StructDesc describes a custom Noms Struct.\ntype StructDesc struct {\n\tName string\n\tfields structTypeFields\n}\n\nfunc (s StructDesc) Kind() NomsKind {\n\treturn StructKind\n}\n\nfunc (s StructDesc) walkValues(cb ValueCallback) {\n\tfor _, field := range s.fields {\n\t\tcb(field.Type)\n\t}\n}\n\nfunc (s StructDesc) writeTo(w nomsWriter, t *Type, seenStructs map[string]*Type) {\n\tname := s.Name\n\n\tif name != \"\" {\n\t\tif _, ok := seenStructs[name]; ok {\n\t\t\tCycleKind.writeTo(w)\n\t\t\tw.writeString(name)\n\t\t\treturn\n\t\t}\n\t\tseenStructs[name] = t\n\t}\n\n\tStructKind.writeTo(w)\n\tw.writeString(name)\n\tw.writeCount(uint64(s.Len()))\n\n\t// Write all names, all types and finally all the optional flags.\n\tfor _, field := range s.fields {\n\t\tw.writeString(field.Name)\n\t}\n\tfor _, field := range s.fields {\n\t\tfield.Type.writeToAsType(w, seenStructs)\n\t}\n\tfor _, field := range s.fields {\n\t\tw.writeBool(field.Optional)\n\t}\n}\n\nfunc (s StructDesc) isSimplifiedForSure() bool {\n\tfor _, f := range s.fields {\n\t\tif !f.Type.Desc.isSimplifiedInner() {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\nfunc (s StructDesc) isSimplifiedInner() bool {\n\t// We do not try to to determine if a type is simplified if it contains a struct.\n\treturn false\n}\n\nfunc (s StructDesc) IterFields(cb func(name string, t *Type, optional bool)) {\n\tfor _, field := range s.fields {\n\t\tcb(field.Name, field.Type, field.Optional)\n\t}\n}\n\nfunc (s StructDesc) Field(name string) (typ *Type, optional bool) {\n\tf, i := s.findField(name)\n\tif i == -1 {\n\t\treturn nil, false\n\t}\n\treturn f.Type, f.Optional\n}\n\nfunc (s StructDesc) findField(name string) (*StructField, int) {\n\ti := sort.Search(len(s.fields), func(i int) bool { return s.fields[i].Name >= name })\n\tif i == len(s.fields) || s.fields[i].Name != name {\n\t\treturn nil, -1\n\t}\n\treturn &s.fields[i], i\n}\n\n// Len returns the number of fields in the struct\nfunc (s StructDesc) Len() int {\n\treturn len(s.fields)\n}\n\ntype CycleDesc string\n\nfunc (c CycleDesc) Kind() NomsKind {\n\treturn CycleKind\n}\n\nfunc (c CycleDesc) walkValues(cb ValueCallback) {\n}\n\nfunc (c CycleDesc) writeTo(w nomsWriter, t *Type, seenStruct map[string]*Type) {\n\tpanic(\"Should not write cycle types\")\n}\n\nfunc (c CycleDesc) isSimplifiedForSure() bool {\n\treturn false\n}\n\nfunc (c CycleDesc) isSimplifiedInner() bool {\n\treturn false\n}\n\ntype typeSlice []*Type\n\nfunc (ts typeSlice) Len() int { return len(ts) }\n\nfunc (ts typeSlice) Less(i, j int) bool {\n\treturn unionLess(ts[i], ts[j])\n}\n\nfunc (ts typeSlice) Swap(i, j int) { ts[i], ts[j] = ts[j], ts[i] }\n\n// unionLess is used for sorting union types in a predictable order as well as\n// validating the order when reading union types from a chunk.\nfunc unionLess(ti, tj *Type) bool {\n\tif ti == tj {\n\t\tpanic(\"unreachable\") // unions must not contain the same type twice.\n\t}\n\n\tki, kj := ti.TargetKind(), tj.TargetKind()\n\tif ki == kj {\n\t\tswitch ki {\n\t\tcase StructKind:\n\t\t\t// Due to type simplification, the only thing that matters is the name of the struct.\n\t\t\treturn ti.Desc.(StructDesc).Name < tj.Desc.(StructDesc).Name\n\t\tcase CycleKind:\n\t\t\treturn ti.Desc.(CycleDesc) < tj.Desc.(CycleDesc)\n\t\tdefault:\n\t\t\tpanic(\"unreachable\") // We should have folded all other types into one.\n\t\t}\n\t}\n\treturn ki < kj\n}\n" }, { "path": "go/types/type_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestTypes(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tmapType := MakeMapType(StringType, NumberType)\n\tsetType := MakeSetType(StringType)\n\tmahType := MakeStructType(\"MahStruct\",\n\t\tStructField{\"Field1\", StringType, false},\n\t\tStructField{\"Field2\", BoolType, false},\n\t)\n\trecType := MakeStructType(\"RecursiveStruct\", StructField{\"self\", MakeCycleType(\"RecursiveStruct\"), false})\n\n\tmRef := vs.WriteValue(mapType).TargetHash()\n\tsetRef := vs.WriteValue(setType).TargetHash()\n\tmahRef := vs.WriteValue(mahType).TargetHash()\n\trecRef := vs.WriteValue(recType).TargetHash()\n\n\tassert.True(mapType.Equals(vs.ReadValue(mRef)))\n\tassert.True(setType.Equals(vs.ReadValue(setRef)))\n\tassert.True(mahType.Equals(vs.ReadValue(mahRef)))\n\tassert.True(recType.Equals(vs.ReadValue(recRef)))\n}\n\nfunc TestTypeType(t *testing.T) {\n\tassert.True(t, TypeOf(BoolType).Equals(TypeType))\n}\n\nfunc TestTypeRefDescribe(t *testing.T) {\n\tassert := assert.New(t)\n\tmapType := MakeMapType(StringType, NumberType)\n\tsetType := MakeSetType(StringType)\n\n\tassert.Equal(\"Bool\", BoolType.Describe())\n\tassert.Equal(\"Number\", NumberType.Describe())\n\tassert.Equal(\"String\", StringType.Describe())\n\tassert.Equal(\"Map\", mapType.Describe())\n\tassert.Equal(\"Set\", setType.Describe())\n\n\tmahType := MakeStructType(\"MahStruct\",\n\t\tStructField{\"Field1\", StringType, false},\n\t\tStructField{\"Field2\", BoolType, false},\n\t)\n\tassert.Equal(\"Struct MahStruct {\\n Field1: String,\\n Field2: Bool,\\n}\", mahType.Describe())\n}\n\nfunc TestTypeOrdered(t *testing.T) {\n\tassert := assert.New(t)\n\tassert.True(isKindOrderedByValue(BoolType.TargetKind()))\n\tassert.True(isKindOrderedByValue(NumberType.TargetKind()))\n\tassert.True(isKindOrderedByValue(StringType.TargetKind()))\n\tassert.False(isKindOrderedByValue(BlobType.TargetKind()))\n\tassert.False(isKindOrderedByValue(ValueType.TargetKind()))\n\tassert.False(isKindOrderedByValue(MakeListType(StringType).TargetKind()))\n\tassert.False(isKindOrderedByValue(MakeSetType(StringType).TargetKind()))\n\tassert.False(isKindOrderedByValue(MakeMapType(StringType, ValueType).TargetKind()))\n\tassert.False(isKindOrderedByValue(MakeRefType(StringType).TargetKind()))\n}\n\nfunc TestFlattenUnionTypes(t *testing.T) {\n\tassert := assert.New(t)\n\tassert.Equal(BoolType, MakeUnionType(BoolType))\n\tassert.Equal(MakeUnionType(), MakeUnionType())\n\tassert.Equal(MakeUnionType(BoolType, StringType), MakeUnionType(BoolType, MakeUnionType(StringType)))\n\tassert.Equal(MakeUnionType(BoolType, StringType, NumberType), MakeUnionType(BoolType, MakeUnionType(StringType, NumberType)))\n\tassert.Equal(BoolType, MakeUnionType(BoolType, BoolType))\n\tassert.Equal(BoolType, MakeUnionType(BoolType, MakeUnionType()))\n\tassert.Equal(BoolType, MakeUnionType(MakeUnionType(), BoolType))\n\tassert.True(MakeUnionType(MakeUnionType(), MakeUnionType()).Equals(MakeUnionType()))\n\tassert.Equal(MakeUnionType(BoolType, NumberType), MakeUnionType(BoolType, NumberType))\n\tassert.Equal(MakeUnionType(BoolType, NumberType), MakeUnionType(NumberType, BoolType))\n\tassert.Equal(MakeUnionType(BoolType, NumberType), MakeUnionType(BoolType, NumberType, BoolType))\n\tassert.Equal(MakeUnionType(BoolType, NumberType), MakeUnionType(MakeUnionType(BoolType, NumberType), NumberType, BoolType))\n}\n\nfunc TestVerifyStructFieldName(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassertInvalid := func(n string) {\n\t\tassert.Panics(func() {\n\t\t\tMakeStructType(\"S\", StructField{n, StringType, false})\n\t\t})\n\t}\n\tassertInvalid(\"\")\n\tassertInvalid(\" \")\n\tassertInvalid(\" a\")\n\tassertInvalid(\"a \")\n\tassertInvalid(\"0\")\n\tassertInvalid(\"_\")\n\tassertInvalid(\"0a\")\n\tassertInvalid(\"_a\")\n\tassertInvalid(\"💩\")\n\n\tassertValid := func(n string) {\n\t\tMakeStructType(\"S\", StructField{n, StringType, false})\n\t}\n\tassertValid(\"a\")\n\tassertValid(\"A\")\n\tassertValid(\"a0\")\n\tassertValid(\"a_\")\n\tassertValid(\"a0_\")\n}\n\nfunc TestVerifyStructName(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassertInvalid := func(n string) {\n\t\tassert.Panics(func() {\n\t\t\tMakeStructType(n)\n\t\t})\n\t}\n\n\tassertInvalid(\" \")\n\tassertInvalid(\" a\")\n\tassertInvalid(\"a \")\n\tassertInvalid(\"0\")\n\tassertInvalid(\"_\")\n\tassertInvalid(\"0a\")\n\tassertInvalid(\"_a\")\n\tassertInvalid(\"💩\")\n\n\tassertValid := func(n string) {\n\t\tMakeStructType(n)\n\t}\n\tassertValid(\"\")\n\tassertValid(\"a\")\n\tassertValid(\"A\")\n\tassertValid(\"a0\")\n\tassertValid(\"a_\")\n\tassertValid(\"a0_\")\n}\n\nfunc TestStructUnionWithCycles(tt *testing.T) {\n\tinodeType := MakeStructTypeFromFields(\"Inode\", FieldMap{\n\t\t\"attr\": MakeStructTypeFromFields(\"Attr\", FieldMap{\n\t\t\t\"ctime\": NumberType,\n\t\t\t\"mode\": NumberType,\n\t\t\t\"mtime\": NumberType,\n\t\t}),\n\t\t\"contents\": MakeUnionType(\n\t\t\tMakeStructTypeFromFields(\"Directory\", FieldMap{\n\t\t\t\t\"entries\": MakeMapType(StringType, MakeCycleType(\"Inode\")),\n\t\t\t}),\n\t\t\tMakeStructTypeFromFields(\"File\", FieldMap{\n\t\t\t\t\"data\": BlobType,\n\t\t\t}),\n\t\t\tMakeStructTypeFromFields(\"Symlink\", FieldMap{\n\t\t\t\t\"targetPath\": StringType,\n\t\t\t}),\n\t\t),\n\t})\n\n\tt1, _ := inodeType.Desc.(StructDesc).Field(\"contents\")\n\tt2 := DecodeValue(EncodeValue(t1), nil)\n\n\tassert.True(tt, t1.Equals(t2))\n\t// Note that we cannot ensure pointer equality between t1 and t2 because the\n\t// types used to the construct the Unions, while eventually equivalent, are\n\t// not identical due to the potentially differing placement of the Cycle type.\n\t// We do not remake Union types after putting their component types into\n\t// their canonical ordering.\n}\n\nfunc TestHasStructCycles(tt *testing.T) {\n\tassert := assert.New(tt)\n\n\tassert.False(HasStructCycles(BoolType))\n\tassert.False(HasStructCycles(BlobType))\n\tassert.False(HasStructCycles(NumberType))\n\tassert.False(HasStructCycles(StringType))\n\tassert.False(HasStructCycles(TypeType))\n\tassert.False(HasStructCycles(ValueType))\n\tassert.Panics(func() {\n\t\tHasStructCycles(MakeCycleType(\"Abc\"))\n\t})\n\n\tassert.False(HasStructCycles(MakeStructType(\"\")))\n\tassert.False(HasStructCycles(MakeStructType(\"A\")))\n\n\tassert.True(HasStructCycles(\n\t\tMakeStructType(\"A\", StructField{\"a\", MakeStructType(\"A\"), false})))\n\tassert.True(HasStructCycles(\n\t\tMakeStructType(\"A\", StructField{\"a\", MakeCycleType(\"A\"), false})))\n\tassert.True(HasStructCycles(\n\t\tMakeSetType(MakeStructType(\"A\", StructField{\"a\", MakeCycleType(\"A\"), false}))))\n\tassert.True(HasStructCycles(\n\t\tMakeStructType(\"A\", StructField{\"a\", MakeSetType(MakeCycleType(\"A\")), false})))\n\n\tassert.False(HasStructCycles(\n\t\tMakeMapType(\n\t\t\tMakeStructType(\"A\"),\n\t\t\tMakeStructType(\"A\"),\n\t\t),\n\t))\n\tassert.False(HasStructCycles(\n\t\tMakeMapType(\n\t\t\tMakeStructType(\"A\"),\n\t\t\tMakeCycleType(\"A\"),\n\t\t),\n\t))\n\n\tassert.False(HasStructCycles(\n\t\tMakeStructType(\"\",\n\t\t\tStructField{\"a\", MakeStructType(\"\",\n\t\t\t\tStructField{\"b\", BoolType, false},\n\t\t\t), false},\n\t\t\tStructField{\"b\", MakeStructType(\"\",\n\t\t\t\tStructField{\"b\", BoolType, false},\n\t\t\t), false},\n\t\t)),\n\t)\n}\n" }, { "path": "go/types/util_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype iterator interface {\n\tNext() Value\n}\n\nfunc iterToSlice(iter iterator) ValueSlice {\n\tvs := ValueSlice{}\n\tfor {\n\t\tv := iter.Next()\n\t\tif v == nil {\n\t\t\tbreak\n\t\t}\n\t\tvs = append(vs, v)\n\t}\n\treturn vs\n}\n\nfunc intsToValueSlice(ints ...int) ValueSlice {\n\tvs := ValueSlice{}\n\tfor _, i := range ints {\n\t\tvs = append(vs, Number(i))\n\t}\n\treturn vs\n}\n\nfunc generateNumbersAsValues(n int) []Value {\n\treturn generateNumbersAsValuesFromToBy(0, n, 1)\n}\n\nfunc generateNumbersAsValueSlice(n int) ValueSlice {\n\treturn generateNumbersAsValuesFromToBy(0, n, 1)\n}\n\nfunc generateNumbersAsValuesFromToBy(from, to, by int) ValueSlice {\n\td.Chk.True(to >= from, \"to must be greater than or equal to from\")\n\td.Chk.True(by > 0, \"must be an integer greater than zero\")\n\tnums := []Value{}\n\tfor i := from; i < to; i += by {\n\t\tnums = append(nums, Number(i))\n\t}\n\treturn nums\n}\n\nfunc generateNumbersAsStructs(n int) ValueSlice {\n\treturn generateNumbersAsValuesFromToBy(0, n, 1)\n}\n\nfunc generateNumbersAsStructsFromToBy(from, to, by int) ValueSlice {\n\td.Chk.True(to >= from, \"to must be greater than or equal to from\")\n\td.Chk.True(by > 0, \"must be an integer greater than zero\")\n\tnums := []Value{}\n\tfor i := from; i < to; i += by {\n\t\tnums = append(nums, NewStruct(\"num\", StructData{\"n\": Number(i)}))\n\t}\n\treturn nums\n}\n\nfunc generateNumbersAsRefOfStructs(vrw ValueReadWriter, n int) []Value {\n\tnums := []Value{}\n\tfor i := 0; i < n; i++ {\n\t\tr := vrw.WriteValue(NewStruct(\"num\", StructData{\"n\": Number(i)}))\n\t\tnums = append(nums, r)\n\t}\n\treturn nums\n}\n\nfunc leafCount(c Collection) int {\n\tleaves, _ := LoadLeafNodes([]Collection{c}, 0, c.Len())\n\treturn len(leaves)\n}\n\nfunc leafDiffCount(c1, c2 Collection) int {\n\tcount := 0\n\thashes := make(map[hash.Hash]int)\n\n\tleaves1, _ := LoadLeafNodes([]Collection{c1}, 0, c1.Len())\n\tleaves2, _ := LoadLeafNodes([]Collection{c2}, 0, c2.Len())\n\n\tfor _, l := range leaves1 {\n\t\thashes[l.Hash()]++\n\t}\n\n\tfor _, l := range leaves2 {\n\t\tif c, ok := hashes[l.Hash()]; ok {\n\t\t\tif c == 1 {\n\t\t\t\tdelete(hashes, l.Hash())\n\t\t\t} else {\n\t\t\t\thashes[l.Hash()] = c - 1\n\t\t\t}\n\t\t} else {\n\t\t\tcount++\n\t\t}\n\t}\n\n\tfor _, c := range hashes {\n\t\tcount += c\n\t}\n\n\treturn count\n}\n\nfunc reverseValues(values []Value) []Value {\n\tnewValues := make([]Value, len(values))\n\tfor i := 0; i < len(values); i++ {\n\t\tnewValues[i] = values[len(values)-i-1]\n\t}\n\treturn newValues\n}\n\nfunc spliceValues(values []Value, start int, deleteCount int, newItems ...Value) []Value {\n\tnumCurrentItems := len(values)\n\tnumNewItems := len(newItems)\n\tnewArr := make([]Value, numCurrentItems-deleteCount+numNewItems)\n\tcopy(newArr[0:], values[0:start])\n\tcopy(newArr[start:], newItems[0:])\n\tcopy(newArr[start+numNewItems:], values[start+deleteCount:])\n\treturn newArr\n}\n" }, { "path": "go/types/validate_type.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nfunc validateType(t *Type) {\n\tvalidateTypeImpl(t, map[string]struct{}{})\n}\n\nfunc validateTypeImpl(t *Type, seenStructs map[string]struct{}) {\n\tswitch desc := t.Desc.(type) {\n\tcase CompoundDesc:\n\t\tif desc.Kind() == UnionKind {\n\t\t\tif len(desc.ElemTypes) == 1 {\n\t\t\t\tpanic(\"Invalid union type\")\n\t\t\t}\n\t\t\tfor i := 1; i < len(desc.ElemTypes); i++ {\n\t\t\t\tif !unionLess(desc.ElemTypes[i-1], desc.ElemTypes[i]) {\n\t\t\t\t\tpanic(\"Invalid union order\")\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\tfor _, et := range desc.ElemTypes {\n\t\t\tvalidateTypeImpl(et, seenStructs)\n\t\t}\n\tcase StructDesc:\n\t\tif desc.Name != \"\" {\n\t\t\tif _, ok := seenStructs[desc.Name]; ok {\n\t\t\t\treturn\n\t\t\t}\n\t\t\tseenStructs[desc.Name] = struct{}{}\n\t\t}\n\t\tverifyStructName(desc.Name)\n\t\tverifyFields(desc.fields)\n\t\tfor _, f := range desc.fields {\n\t\t\tvalidateTypeImpl(f.Type, seenStructs)\n\t\t}\n\t}\n}\n" }, { "path": "go/types/validating_decoder.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype ValidatingDecoder struct {\n\tvs *ValueStore\n}\n\nfunc NewValidatingDecoder(cs chunks.ChunkStore) *ValidatingDecoder {\n\treturn &ValidatingDecoder{NewValueStore(cs)}\n}\n\n// DecodedChunk holds a pointer to a Chunk and the Value that results from\n// calling DecodeFromBytes(c.Data()).\ntype DecodedChunk struct {\n\tChunk *chunks.Chunk\n\tValue *Value\n}\n\n// Decode decodes c and checks that the hash of the resulting value\n// matches c.Hash(). It returns a DecodedChunk holding both c and a pointer to\n// the decoded Value.\nfunc (vbs *ValidatingDecoder) Decode(c *chunks.Chunk) DecodedChunk {\n\th := c.Hash()\n\tv := decodeFromBytesWithValidation(c.Data(), vbs.vs)\n\n\tif getHash(v) != h {\n\t\td.Panic(\"Invalid hash found\")\n\t}\n\treturn DecodedChunk{c, &v}\n}\n" }, { "path": "go/types/validating_decoder_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestValidatingBatchingSinkDecode(t *testing.T) {\n\tv := Number(42)\n\tc := EncodeValue(v)\n\tstorage := &chunks.TestStorage{}\n\tvdc := NewValidatingDecoder(storage.NewView())\n\n\tdc := vdc.Decode(&c)\n\tassert.True(t, v.Equals(*dc.Value))\n}\n\nfunc assertPanicsOnInvalidChunk(t *testing.T, data []interface{}) {\n\tstorage := &chunks.TestStorage{}\n\tvs := NewValueStore(storage.NewView())\n\tdataAsByteSlice := toBinaryNomsReaderData(data)\n\tdec := newValueDecoder(dataAsByteSlice, vs)\n\tv := dec.readValue()\n\n\tc := EncodeValue(v)\n\tvdc := NewValidatingDecoder(storage.NewView())\n\n\tassert.Panics(t, func() {\n\t\tvdc.Decode(&c)\n\t})\n}\n\nfunc TestValidatingBatchingSinkDecodeInvalidUnion(t *testing.T) {\n\tdata := []interface{}{\n\t\tuint8(TypeKind),\n\t\tuint8(UnionKind), uint64(2) /* len */, uint8(NumberKind), uint8(BoolKind),\n\t}\n\tassertPanicsOnInvalidChunk(t, data)\n}\n\nfunc TestValidatingBatchingSinkDecodeInvalidStructFieldOrder(t *testing.T) {\n\tdata := []interface{}{\n\t\tuint8(TypeKind),\n\t\tuint8(StructKind), \"S\", uint64(2), /* len */\n\t\t\"b\", \"a\",\n\t\tuint8(NumberKind), uint8(NumberKind),\n\t\tfalse, false,\n\t}\n\tassertPanicsOnInvalidChunk(t, data)\n}\n\nfunc TestValidatingBatchingSinkDecodeInvalidStructName(t *testing.T) {\n\tdata := []interface{}{\n\t\tuint8(TypeKind),\n\t\tuint8(StructKind), \"S \", uint64(0), /* len */\n\t}\n\tassertPanicsOnInvalidChunk(t, data)\n}\n\nfunc TestValidatingBatchingSinkDecodeInvalidStructFieldName(t *testing.T) {\n\tdata := []interface{}{\n\t\tuint8(TypeKind),\n\t\tuint8(StructKind), \"S\", uint64(1), /* len */\n\t\t\"b \", uint8(NumberKind), false,\n\t}\n\tassertPanicsOnInvalidChunk(t, data)\n}\n" }, { "path": "go/types/value.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n)\n\ntype ValueCallback func(v Value)\ntype RefCallback func(ref Ref)\n\n// Valuable is an interface from which a Value can be retrieved.\ntype Valuable interface {\n\t// Kind is the NomsKind describing the kind of value this is.\n\tKind() NomsKind\n\n\tValue() Value\n}\n\n// Value is the interface all Noms values implement.\ntype Value interface {\n\tValuable\n\n\t// Equals determines if two different Noms values represents the same underlying value.\n\tEquals(other Value) bool\n\n\t// Less determines if this Noms value is less than another Noms value.\n\t// When comparing two Noms values and both are comparable and the same type (Bool, Number or\n\t// String) then the natural ordering is used. For other Noms values the Hash of the value is\n\t// used. When comparing Noms values of different type the following ordering is used:\n\t// Bool < Number < String < everything else.\n\tLess(other Value) bool\n\n\t// Hash is the hash of the value. All Noms values have a unique hash and if two values have the\n\t// same hash they must be equal.\n\tHash() hash.Hash\n\n\t// WalkValues iterates over the immediate children of this value in the DAG, if any, not including\n\t// Type()\n\tWalkValues(ValueCallback)\n\n\t// WalkRefs iterates over the refs to the underlying chunks. If this value is a collection that has been\n\t// chunked then this will return the refs of th sub trees of the prolly-tree.\n\tWalkRefs(RefCallback)\n\n\t// typeOf is the internal implementation of types.TypeOf. It is not normalized\n\t// and unions might have a single element, duplicates and be in the wrong\n\t// order.\n\ttypeOf() *Type\n\n\t// writeTo writes the encoded version of the value to a nomsWriter.\n\twriteTo(nomsWriter)\n}\n\ntype ValueSlice []Value\n\nfunc (vs ValueSlice) Len() int { return len(vs) }\nfunc (vs ValueSlice) Swap(i, j int) { vs[i], vs[j] = vs[j], vs[i] }\nfunc (vs ValueSlice) Less(i, j int) bool { return vs[i].Less(vs[j]) }\nfunc (vs ValueSlice) Equals(other ValueSlice) bool {\n\tif vs.Len() != other.Len() {\n\t\treturn false\n\t}\n\n\tfor i, v := range vs {\n\t\tif !v.Equals(other[i]) {\n\t\t\treturn false\n\t\t}\n\t}\n\n\treturn true\n}\n\nfunc (vs ValueSlice) Contains(v Value) bool {\n\tfor _, v := range vs {\n\t\tif v.Equals(v) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\ntype valueReadWriter interface {\n\tvalueReadWriter() ValueReadWriter\n}\n\ntype valueImpl struct {\n\tvrw ValueReadWriter\n\tbuff []byte\n\toffsets []uint32\n}\n\nfunc (v valueImpl) valueReadWriter() ValueReadWriter {\n\treturn v.vrw\n}\n\nfunc (v valueImpl) writeTo(enc nomsWriter) {\n\tenc.writeRaw(v.buff)\n}\n\nfunc (v valueImpl) valueBytes() []byte {\n\treturn v.buff\n}\n\n// IsZeroValue can be used to test if a Value is the same as T{}.\nfunc (v valueImpl) IsZeroValue() bool {\n\treturn v.buff == nil\n}\n\nfunc (v valueImpl) Hash() hash.Hash {\n\treturn hash.Of(v.buff)\n}\n\nfunc (v valueImpl) decoder() valueDecoder {\n\treturn newValueDecoder(v.buff, v.vrw)\n}\n\nfunc (v valueImpl) decoderAtOffset(offset int) valueDecoder {\n\treturn newValueDecoder(v.buff[offset:], v.vrw)\n}\n\nfunc (v valueImpl) asValueImpl() valueImpl {\n\treturn v\n}\n\nfunc (v valueImpl) Equals(other Value) bool {\n\tif otherValueImpl, ok := other.(asValueImpl); ok {\n\t\treturn bytes.Equal(v.buff, otherValueImpl.asValueImpl().buff)\n\t}\n\treturn false\n}\n\nfunc (v valueImpl) Less(other Value) bool {\n\treturn valueLess(v, other)\n}\n\nfunc (v valueImpl) WalkRefs(cb RefCallback) {\n\twalkRefs(v.valueBytes(), cb)\n}\n\ntype asValueImpl interface {\n\tasValueImpl() valueImpl\n}\n\nfunc (v valueImpl) Kind() NomsKind {\n\treturn NomsKind(v.buff[0])\n}\n" }, { "path": "go/types/value_decoder.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport \"github.com/attic-labs/noms/go/d\"\n\ntype valueDecoder struct {\n\ttypedBinaryNomsReader\n\tvrw ValueReadWriter\n}\n\n// typedBinaryNomsReader provides some functionality for reading and skipping types that is shared by both valueDecoder and refWalker.\ntype typedBinaryNomsReader struct {\n\tbinaryNomsReader\n\tvalidating bool\n}\n\nfunc newValueDecoder(buff []byte, vrw ValueReadWriter) valueDecoder {\n\tnr := binaryNomsReader{buff, 0}\n\treturn valueDecoder{typedBinaryNomsReader{nr, false}, vrw}\n}\n\nfunc newValueDecoderWithValidation(nr binaryNomsReader, vrw ValueReadWriter) valueDecoder {\n\treturn valueDecoder{typedBinaryNomsReader{nr, true}, vrw}\n}\n\nfunc (r *valueDecoder) copyString(w nomsWriter) {\n\tstart := r.pos()\n\tr.skipString()\n\tend := r.pos()\n\tw.writeRaw(r.byteSlice(start, end))\n}\n\nfunc (r *valueDecoder) readRef() Ref {\n\treturn readRef(&(r.typedBinaryNomsReader))\n}\n\nfunc (r *valueDecoder) skipRef() {\n\tskipRef(&(r.typedBinaryNomsReader))\n}\n\nfunc (r *valueDecoder) skipBlobLeafSequence() ([]uint32, uint64) {\n\tsize := r.readCount()\n\tvaluesPos := r.pos()\n\tr.offset += uint32(size)\n\treturn []uint32{valuesPos, r.pos()}, size\n}\n\nfunc (r *valueDecoder) skipValueSequence(elementsPerIndex int) ([]uint32, uint64) {\n\tcount := r.readCount()\n\toffsets := make([]uint32, count+1)\n\toffsets[0] = r.pos()\n\tfor i := uint64(0); i < count; i++ {\n\t\tfor j := 0; j < elementsPerIndex; j++ {\n\t\t\tr.skipValue()\n\t\t}\n\t\toffsets[i+1] = r.pos()\n\t}\n\treturn offsets, count\n}\n\nfunc (r *valueDecoder) skipListLeafSequence() ([]uint32, uint64) {\n\treturn r.skipValueSequence(getValuesPerIdx(ListKind))\n}\n\nfunc (r *valueDecoder) skipSetLeafSequence() ([]uint32, uint64) {\n\treturn r.skipValueSequence(getValuesPerIdx(SetKind))\n}\n\nfunc (r *valueDecoder) skipMapLeafSequence() ([]uint32, uint64) {\n\treturn r.skipValueSequence(getValuesPerIdx(MapKind))\n}\n\nfunc (r *valueDecoder) readSequence(kind NomsKind, leafSkipper func() ([]uint32, uint64)) sequence {\n\tstart := r.pos()\n\toffsets := []uint32{start}\n\tr.skipKind()\n\toffsets = append(offsets, r.pos())\n\tlevel := r.readCount()\n\toffsets = append(offsets, r.pos())\n\tvar seqOffsets []uint32\n\tvar length uint64\n\tif level > 0 {\n\t\tseqOffsets, length = r.skipMetaSequence(kind, level)\n\t} else {\n\t\tseqOffsets, length = leafSkipper()\n\t}\n\toffsets = append(offsets, seqOffsets...)\n\tend := r.pos()\n\n\tif level > 0 {\n\t\treturn newMetaSequence(r.vrw, r.byteSlice(start, end), offsets, length)\n\t}\n\n\treturn newLeafSequence(r.vrw, r.byteSlice(start, end), offsets, length)\n}\n\nfunc (r *valueDecoder) readBlobSequence() sequence {\n\tseq := r.readSequence(BlobKind, r.skipBlobLeafSequence)\n\tif seq.isLeaf() {\n\t\treturn blobLeafSequence{seq.(leafSequence)}\n\t}\n\treturn seq\n}\n\nfunc (r *valueDecoder) readListSequence() sequence {\n\tseq := r.readSequence(ListKind, r.skipListLeafSequence)\n\tif seq.isLeaf() {\n\t\treturn listLeafSequence{seq.(leafSequence)}\n\t}\n\treturn seq\n}\n\nfunc (r *valueDecoder) readSetSequence() orderedSequence {\n\tseq := r.readSequence(SetKind, r.skipSetLeafSequence)\n\tif seq.isLeaf() {\n\t\treturn setLeafSequence{seq.(leafSequence)}\n\t}\n\treturn seq.(orderedSequence)\n}\n\nfunc (r *valueDecoder) readMapSequence() orderedSequence {\n\tseq := r.readSequence(MapKind, r.skipMapLeafSequence)\n\tif seq.isLeaf() {\n\t\treturn mapLeafSequence{seq.(leafSequence)}\n\t}\n\treturn seq.(orderedSequence)\n}\n\nfunc (r *valueDecoder) skipList() {\n\tr.skipSequence(ListKind, r.skipListLeafSequence)\n}\n\nfunc (r *valueDecoder) skipSet() {\n\tr.skipSequence(SetKind, r.skipSetLeafSequence)\n}\n\nfunc (r *valueDecoder) skipMap() {\n\tr.skipSequence(MapKind, r.skipMapLeafSequence)\n}\n\nfunc (r *valueDecoder) skipBlob() {\n\tr.skipSequence(BlobKind, r.skipBlobLeafSequence)\n}\n\nfunc (r *valueDecoder) skipSequence(kind NomsKind, leafSkipper func() ([]uint32, uint64)) {\n\tr.skipKind()\n\tlevel := r.readCount()\n\tif level > 0 {\n\t\tr.skipMetaSequence(kind, level)\n\t} else {\n\t\tleafSkipper()\n\t}\n}\n\nfunc (r *valueDecoder) skipOrderedKey() {\n\tswitch r.peekKind() {\n\tcase hashKind:\n\t\tr.skipKind()\n\t\tr.skipHash()\n\tdefault:\n\t\tr.skipValue()\n\t}\n}\n\nfunc (r *valueDecoder) skipMetaSequence(k NomsKind, level uint64) ([]uint32, uint64) {\n\tcount := r.readCount()\n\toffsets := make([]uint32, count+1)\n\toffsets[0] = r.pos()\n\tlength := uint64(0)\n\tfor i := uint64(0); i < count; i++ {\n\t\tr.skipRef()\n\t\tr.skipOrderedKey()\n\t\tlength += r.readCount()\n\t\toffsets[i+1] = r.pos()\n\t}\n\treturn offsets, length\n}\n\nfunc (r *valueDecoder) readValue() Value {\n\tk := r.peekKind()\n\tswitch k {\n\tcase BlobKind:\n\t\treturn newBlob(r.readBlobSequence())\n\tcase BoolKind:\n\t\tr.skipKind()\n\t\treturn Bool(r.readBool())\n\tcase NumberKind:\n\t\tr.skipKind()\n\t\treturn r.readNumber()\n\tcase StringKind:\n\t\tr.skipKind()\n\t\treturn String(r.readString())\n\tcase ListKind:\n\t\treturn newList(r.readListSequence())\n\tcase MapKind:\n\t\treturn newMap(r.readMapSequence())\n\tcase RefKind:\n\t\treturn r.readRef()\n\tcase SetKind:\n\t\treturn newSet(r.readSetSequence())\n\tcase StructKind:\n\t\treturn r.readStruct()\n\tcase TypeKind:\n\t\tr.skipKind()\n\t\treturn r.readType()\n\tcase CycleKind, UnionKind, ValueKind:\n\t\td.Panic(\"A value instance can never have type %s\", k)\n\t}\n\n\tpanic(\"not reachable\")\n}\n\nfunc (r *valueDecoder) skipValue() {\n\tk := r.peekKind()\n\tswitch k {\n\tcase BlobKind:\n\t\tr.skipBlob()\n\tcase BoolKind:\n\t\tr.skipKind()\n\t\tr.skipBool()\n\tcase NumberKind:\n\t\tr.skipKind()\n\t\tr.skipNumber()\n\tcase StringKind:\n\t\tr.skipKind()\n\t\tr.skipString()\n\tcase ListKind:\n\t\tr.skipList()\n\tcase MapKind:\n\t\tr.skipMap()\n\tcase RefKind:\n\t\tr.skipRef()\n\tcase SetKind:\n\t\tr.skipSet()\n\tcase StructKind:\n\t\tr.skipStruct()\n\tcase TypeKind:\n\t\tr.skipKind()\n\t\tr.skipType()\n\tcase CycleKind, UnionKind, ValueKind:\n\t\td.Panic(\"A value instance can never have type %s\", k)\n\tdefault:\n\t\tpanic(\"not reachable\")\n\t}\n}\n\n// readTypeOfValue is basically readValue().typeOf() but it ensures that we do\n// not allocate values where we do not need to.\nfunc (r *valueDecoder) readTypeOfValue() *Type {\n\tk := r.peekKind()\n\tswitch k {\n\tcase BlobKind:\n\t\tr.skipBlob()\n\t\treturn BlobType\n\tcase BoolKind:\n\t\tr.skipKind()\n\t\tr.skipBool()\n\t\treturn BoolType\n\tcase NumberKind:\n\t\tr.skipKind()\n\t\tr.skipNumber()\n\t\treturn NumberType\n\tcase StringKind:\n\t\tr.skipKind()\n\t\tr.skipString()\n\t\treturn StringType\n\tcase ListKind, MapKind, RefKind, SetKind:\n\t\t// These do not decode the actual values anyway.\n\t\treturn r.readValue().typeOf()\n\tcase StructKind:\n\t\treturn readStructTypeOfValue(r)\n\tcase TypeKind:\n\t\tr.skipKind()\n\t\tr.skipType()\n\t\treturn TypeType\n\tcase CycleKind, UnionKind, ValueKind:\n\t\td.Panic(\"A value instance can never have type %s\", k)\n\t}\n\n\tpanic(\"not reachable\")\n}\n\n// isValueSameTypeForSure may return false even though the type of the value is\n// equal. We do that in cases wherer it would be too expensive to compute the\n// type.\n// If this returns false the decoder might not have visited the whole value and\n// its offset is no longer valid.\nfunc (r *valueDecoder) isValueSameTypeForSure(t *Type) bool {\n\tk := r.peekKind()\n\tif k != t.TargetKind() {\n\t\treturn false\n\t}\n\n\tswitch k {\n\tcase BlobKind, BoolKind, NumberKind, StringKind:\n\t\tr.skipValue()\n\t\treturn true\n\tcase ListKind, MapKind, RefKind, SetKind:\n\t\t// TODO: Maybe do some simple cases here too. Performance metrics should determine\n\t\t// what is going to be worth doing.\n\t\t// https://github.com/attic-labs/noms/issues/3776\n\t\treturn false\n\tcase StructKind:\n\t\treturn isStructSameTypeForSure(r, t)\n\tcase TypeKind:\n\t\treturn false\n\tcase CycleKind, UnionKind, ValueKind:\n\t\td.Panic(\"A value instance can never have type %s\", k)\n\t}\n\n\tpanic(\"not reachable\")\n}\n\n// isStringSame checks if the next string in the decoder matches string. It\n// moves the decoder to after the string in all cases.\nfunc (r *valueDecoder) isStringSame(s string) bool {\n\tcount := r.readCount()\n\tstart := uint64(r.offset)\n\tr.offset += uint32(count)\n\tif uint64(len(s)) != count {\n\t\treturn false\n\t}\n\n\tfor i := uint64(0); i < count; i++ {\n\t\tif s[i] != r.buff[start+i] {\n\t\t\treturn false\n\t\t}\n\t}\n\treturn true\n}\n\nfunc (r *valueDecoder) copyValue(w nomsWriter) {\n\tstart := r.pos()\n\tr.skipValue()\n\tend := r.pos()\n\tw.writeRaw(r.byteSlice(start, end))\n}\n\nfunc (r *valueDecoder) readStruct() Value {\n\treturn readStruct(r)\n}\n\nfunc (r *valueDecoder) skipStruct() {\n\tskipStruct(r)\n}\n\nfunc boolToUint32(b bool) uint32 {\n\tif b {\n\t\treturn 1\n\t}\n\treturn 0\n}\n\nfunc (r *valueDecoder) readOrderedKey() orderedKey {\n\tswitch r.peekKind() {\n\tcase hashKind:\n\t\tr.skipKind()\n\t\th := r.readHash()\n\t\treturn orderedKeyFromHash(h)\n\tdefault:\n\t\tv := r.readValue()\n\t\treturn newOrderedKey(v)\n\t}\n}\n\nfunc (r *typedBinaryNomsReader) readType() *Type {\n\tt := r.readTypeInner(map[string]*Type{})\n\tif r.validating {\n\t\tvalidateType(t)\n\t}\n\treturn t\n}\n\nfunc (r *typedBinaryNomsReader) skipType() {\n\tif r.validating {\n\t\tr.readType()\n\t\treturn\n\t}\n\tr.skipTypeInner()\n}\n\nfunc (r *typedBinaryNomsReader) readTypeInner(seenStructs map[string]*Type) *Type {\n\tk := r.readKind()\n\tswitch k {\n\tcase ListKind:\n\t\treturn makeCompoundType(ListKind, r.readTypeInner(seenStructs))\n\tcase MapKind:\n\t\treturn makeCompoundType(MapKind, r.readTypeInner(seenStructs), r.readTypeInner(seenStructs))\n\tcase RefKind:\n\t\treturn makeCompoundType(RefKind, r.readTypeInner(seenStructs))\n\tcase SetKind:\n\t\treturn makeCompoundType(SetKind, r.readTypeInner(seenStructs))\n\tcase StructKind:\n\t\treturn r.readStructType(seenStructs)\n\tcase UnionKind:\n\t\treturn r.readUnionType(seenStructs)\n\tcase CycleKind:\n\t\tname := r.readString()\n\t\td.PanicIfTrue(name == \"\") // cycles to anonymous structs are disallowed\n\t\tt, ok := seenStructs[name]\n\t\td.PanicIfFalse(ok)\n\t\treturn t\n\t}\n\n\td.PanicIfFalse(IsPrimitiveKind(k))\n\treturn MakePrimitiveType(k)\n}\n\nfunc (r *typedBinaryNomsReader) skipTypeInner() {\n\tk := r.readKind()\n\tswitch k {\n\tcase ListKind, RefKind, SetKind:\n\t\tr.skipTypeInner()\n\tcase MapKind:\n\t\tr.skipTypeInner()\n\t\tr.skipTypeInner()\n\tcase StructKind:\n\t\tr.skipStructType()\n\tcase UnionKind:\n\t\tr.skipUnionType()\n\tcase CycleKind:\n\t\tr.skipString()\n\tdefault:\n\t\td.PanicIfFalse(IsPrimitiveKind(k))\n\t}\n}\n\nfunc (r *typedBinaryNomsReader) readStructType(seenStructs map[string]*Type) *Type {\n\tname := r.readString()\n\tcount := r.readCount()\n\tfields := make(structTypeFields, count)\n\n\tt := newType(StructDesc{name, fields})\n\tseenStructs[name] = t\n\n\tfor i := uint64(0); i < count; i++ {\n\t\tt.Desc.(StructDesc).fields[i] = StructField{\n\t\t\tName: r.readString(),\n\t\t}\n\t}\n\tfor i := uint64(0); i < count; i++ {\n\t\tt.Desc.(StructDesc).fields[i].Type = r.readTypeInner(seenStructs)\n\t}\n\tfor i := uint64(0); i < count; i++ {\n\t\tt.Desc.(StructDesc).fields[i].Optional = r.readBool()\n\t}\n\n\treturn t\n}\n\nfunc (r *typedBinaryNomsReader) skipStructType() {\n\tr.skipString() // name\n\tcount := r.readCount()\n\n\tfor i := uint64(0); i < count; i++ {\n\t\tr.skipString() // name\n\t}\n\tfor i := uint64(0); i < count; i++ {\n\t\tr.skipTypeInner()\n\t}\n\tfor i := uint64(0); i < count; i++ {\n\t\tr.skipBool() // optional\n\t}\n}\n\nfunc (r *typedBinaryNomsReader) readUnionType(seenStructs map[string]*Type) *Type {\n\tl := r.readCount()\n\tts := make(typeSlice, l)\n\tfor i := uint64(0); i < l; i++ {\n\t\tts[i] = r.readTypeInner(seenStructs)\n\t}\n\treturn makeUnionType(ts...)\n}\n\nfunc (r *typedBinaryNomsReader) skipUnionType() {\n\tl := r.readCount()\n\tfor i := uint64(0); i < l; i++ {\n\t\tr.skipTypeInner()\n\t}\n}\n" }, { "path": "go/types/value_stats.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n\thumanize \"github.com/dustin/go-humanize\"\n\t\"github.com/golang/snappy\"\n)\n\ntype ValueStats interface {\n\tString() string\n}\n\nfunc WriteValueStats(w io.Writer, v Value, vr ValueReader) {\n\tswitch v.Kind() {\n\tcase BoolKind, NumberKind, StringKind, RefKind, StructKind, TypeKind:\n\t\twriteUnchunkedValueStats(w, v, vr)\n\tcase BlobKind, ListKind, MapKind, SetKind:\n\t\twritePtreeStats(w, v, vr)\n\t}\n}\n\nfunc writeUnchunkedValueStats(w io.Writer, v Value, vr ValueReader) {\n\tfmt.Fprintf(w, \"Kind: %s\\nCompressedSize: %s\\n\", v.Kind().String(), humanize.Bytes(compressedSize(v)))\n}\n\nconst treeRowFormat = \"%5s%20s%20s%20s\\n\"\n\nvar treeLevelHeader = fmt.Sprintf(treeRowFormat, \"Level\", \"Nodes\", \"Values/Node\", \"Size/Node\")\n\nfunc writePtreeStats(w io.Writer, v Value, vr ValueReader) {\n\ttotalCompressedSize := uint64(0)\n\ttotalChunks := uint64(0)\n\n\tfmt.Fprintf(w, \"Kind: %s\\n\", v.Kind().String())\n\tfmt.Fprintf(w, treeLevelHeader)\n\n\tlevel := int64(v.(Collection).asSequence().treeLevel())\n\tnodes := ValueSlice{v}\n\n\t// TODO: For level 0, use NBS to fetch leaf sizes without actually reading leaf data.\n\tfor level >= 0 {\n\t\tchildren := RefSlice{}\n\t\tvisited := hash.HashSet{}\n\t\tchunkCount, valueCount, byteSize := uint64(0), uint64(0), uint64(0)\n\n\t\tfor _, n := range nodes {\n\t\t\tchunkCount++\n\t\t\tif level > 0 {\n\t\t\t\tn.WalkRefs(func(r Ref) {\n\t\t\t\t\tchildren = append(children, r)\n\t\t\t\t})\n\t\t\t}\n\n\t\t\ts := n.(Collection).asSequence()\n\t\t\tvalueCount += uint64(s.seqLen())\n\n\t\t\th := n.Hash()\n\t\t\tif !visited.Has(h) {\n\t\t\t\t// Indexed Ptrees can share nodes within the same tree level. Only count each unique value once\n\t\t\t\tbyteSize += compressedSize(n)\n\t\t\t\tvisited.Insert(h)\n\t\t\t}\n\t\t}\n\n\t\tprintTreeLevel(w, uint64(level), valueCount, chunkCount, byteSize)\n\n\t\tnodes = loadNextLevel(children, vr)\n\t\tlevel--\n\t\ttotalCompressedSize += byteSize\n\t\ttotalChunks += chunkCount\n\t}\n}\n\nfunc printTreeLevel(w io.Writer, level, values, chunks, byteSize uint64) {\n\tavgItems := float64(values) / float64(chunks)\n\tavgSize := byteSize / chunks\n\n\tfmt.Fprintf(w, treeRowFormat,\n\t\tfmt.Sprintf(\"%d\", level),\n\t\thumanize.Comma(int64(chunks)),\n\t\tfmt.Sprintf(\"%.1f\", avgItems),\n\t\thumanize.Bytes(avgSize))\n}\n\nfunc compressedSize(v Value) uint64 {\n\tchunk := EncodeValue(v)\n\tcompressed := snappy.Encode(nil, chunk.Data())\n\treturn uint64(len(compressed))\n}\n\nfunc loadNextLevel(refs RefSlice, vr ValueReader) ValueSlice {\n\ths := make(hash.HashSlice, len(refs))\n\tfor i, r := range refs {\n\t\ths[i] = r.TargetHash()\n\t}\n\n\t// Fetch committed child sequences in a single batch\n\treturn vr.ReadManyValues(hs)\n}\n" }, { "path": "go/types/value_store.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/constants\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/util/sizecache\"\n)\n\n// ValueReader is an interface that knows how to read Noms Values, e.g.\n// datas/Database. Required to avoid import cycle between this package and the\n// package that implements Value reading.\ntype ValueReader interface {\n\tReadValue(h hash.Hash) Value\n\tReadManyValues(hashes hash.HashSlice) ValueSlice\n}\n\n// ValueWriter is an interface that knows how to write Noms Values, e.g.\n// datas/Database. Required to avoid import cycle between this package and the\n// package that implements Value writing.\ntype ValueWriter interface {\n\tWriteValue(v Value) Ref\n}\n\n// ValueReadWriter is an interface that knows how to read and write Noms\n// Values, e.g. datas/Database. Required to avoid import cycle between this\n// package and the package that implements Value read/writing.\ntype ValueReadWriter interface {\n\tValueReader\n\tValueWriter\n}\n\n// ValueStore provides methods to read and write Noms Values to a ChunkStore.\n// It minimally validates Values as they're written, but does not guarantee\n// that these Values are persisted through the ChunkStore until a subsequent\n// Flush.\n// Currently, WriteValue validates the following properties of a Value v:\n// - v can be correctly serialized and its Ref taken\ntype ValueStore struct {\n\tcs chunks.ChunkStore\n\tbufferMu sync.RWMutex\n\tbufferedChunks map[hash.Hash]chunks.Chunk\n\tbufferedChunksMax uint64\n\tbufferedChunkSize uint64\n\twithBufferedChildren map[hash.Hash]uint64 // chunk Hash -> ref height\n\tunresolvedRefs hash.HashSet\n\tenforceCompleteness bool\n\tdecodedChunks *sizecache.SizeCache\n\n\tversOnce sync.Once\n}\n\nfunc PanicIfDangling(unresolved hash.HashSet, cs chunks.ChunkStore) {\n\tabsent := cs.HasMany(unresolved)\n\tif len(absent) != 0 {\n\t\td.Panic(\"Found dangling references to %v\", absent)\n\t}\n}\n\nconst (\n\tdefaultDecodedChunksSize = 1 << 25 // 32MB\n\tdefaultPendingPutMax = 1 << 28 // 256MB\n)\n\n// newTestValueStore creates a simple struct that satisfies ValueReadWriter\n// and is backed by a chunks.TestStore.\nfunc newTestValueStore() *ValueStore {\n\tts := &chunks.TestStorage{}\n\treturn NewValueStore(ts.NewView())\n}\n\n// NewValueStore returns a ValueStore instance that owns the provided\n// ChunkStore and manages its lifetime. Calling Close on the returned\n// ValueStore will Close() cs.\nfunc NewValueStore(cs chunks.ChunkStore) *ValueStore {\n\treturn newValueStoreWithCacheAndPending(cs, defaultDecodedChunksSize, defaultPendingPutMax)\n}\n\nfunc newValueStoreWithCacheAndPending(cs chunks.ChunkStore, cacheSize, pendingMax uint64) *ValueStore {\n\treturn &ValueStore{\n\t\tcs: cs,\n\n\t\tbufferMu: sync.RWMutex{},\n\t\tbufferedChunks: map[hash.Hash]chunks.Chunk{},\n\t\tbufferedChunksMax: pendingMax,\n\t\twithBufferedChildren: map[hash.Hash]uint64{},\n\t\tdecodedChunks: sizecache.New(cacheSize),\n\t\tunresolvedRefs: hash.HashSet{},\n\t\tenforceCompleteness: true,\n\t\tversOnce: sync.Once{},\n\t}\n}\n\nfunc (lvs *ValueStore) expectVersion() {\n\tdataVersion := lvs.cs.Version()\n\tif constants.NomsVersion != dataVersion {\n\t\td.Panic(\"SDK version %s incompatible with data of version %s\", constants.NomsVersion, dataVersion)\n\t}\n}\n\nfunc (lvs *ValueStore) SetEnforceCompleteness(enforce bool) {\n\tlvs.enforceCompleteness = enforce\n}\n\nfunc (lvs *ValueStore) ChunkStore() chunks.ChunkStore {\n\treturn lvs.cs\n}\n\n// ReadValue reads and decodes a value from lvs. It is not considered an error\n// for the requested chunk to be empty; in this case, the function simply\n// returns nil.\nfunc (lvs *ValueStore) ReadValue(h hash.Hash) Value {\n\tlvs.versOnce.Do(lvs.expectVersion)\n\tif v, ok := lvs.decodedChunks.Get(h); ok {\n\t\td.PanicIfTrue(v == nil)\n\t\treturn v.(Value)\n\t}\n\n\tchunk := func() chunks.Chunk {\n\t\tlvs.bufferMu.RLock()\n\t\tdefer lvs.bufferMu.RUnlock()\n\t\tif pending, ok := lvs.bufferedChunks[h]; ok {\n\t\t\treturn pending\n\t\t}\n\t\treturn chunks.EmptyChunk\n\t}()\n\tif chunk.IsEmpty() {\n\t\tchunk = lvs.cs.Get(h)\n\t}\n\tif chunk.IsEmpty() {\n\t\treturn nil\n\t}\n\n\tv := DecodeValue(chunk, lvs)\n\td.PanicIfTrue(v == nil)\n\tlvs.decodedChunks.Add(h, uint64(len(chunk.Data())), v)\n\treturn v\n}\n\n// ReadManyValues reads and decodes Values indicated by |hashes| from lvs and\n// returns the found Values in the same order. Any non-present Values will be\n// represented by nil.\nfunc (lvs *ValueStore) ReadManyValues(hashes hash.HashSlice) ValueSlice {\n\tlvs.versOnce.Do(lvs.expectVersion)\n\tdecode := func(h hash.Hash, chunk *chunks.Chunk) Value {\n\t\tv := DecodeValue(*chunk, lvs)\n\t\td.PanicIfTrue(v == nil)\n\t\tlvs.decodedChunks.Add(h, uint64(len(chunk.Data())), v)\n\t\treturn v\n\t}\n\n\tfoundValues := make(map[hash.Hash]Value, len(hashes))\n\n\t// First, see which hashes can be found in either the Value cache or bufferedChunks.\n\t// Put the rest into a new HashSet to be requested en masse from the ChunkStore.\n\tremaining := hash.HashSet{}\n\tfor _, h := range hashes {\n\t\tif v, ok := lvs.decodedChunks.Get(h); ok {\n\t\t\td.PanicIfTrue(v == nil)\n\t\t\tfoundValues[h] = v.(Value)\n\t\t\tcontinue\n\t\t}\n\n\t\tchunk := func() chunks.Chunk {\n\t\t\tlvs.bufferMu.RLock()\n\t\t\tdefer lvs.bufferMu.RUnlock()\n\t\t\tif pending, ok := lvs.bufferedChunks[h]; ok {\n\t\t\t\treturn pending\n\t\t\t}\n\t\t\treturn chunks.EmptyChunk\n\t\t}()\n\t\tif !chunk.IsEmpty() {\n\t\t\tfoundValues[h] = decode(h, &chunk)\n\t\t\tcontinue\n\t\t}\n\n\t\tremaining.Insert(h)\n\t}\n\n\tif len(remaining) != 0 {\n\t\t// Request remaining hashes from ChunkStore, processing the found chunks as they come in.\n\t\tfoundChunks := make(chan *chunks.Chunk, 16)\n\n\t\tgo func() { lvs.cs.GetMany(remaining, foundChunks); close(foundChunks) }()\n\t\tfor c := range foundChunks {\n\t\t\th := c.Hash()\n\t\t\tfoundValues[h] = decode(h, c)\n\t\t}\n\t}\n\n\trv := make(ValueSlice, len(hashes))\n\tfor i, h := range hashes {\n\t\trv[i] = foundValues[h]\n\t}\n\treturn rv\n}\n\n// WriteValue takes a Value, schedules it to be written it to lvs, and returns\n// an appropriately-typed types.Ref. v is not guaranteed to be actually\n// written until after Flush().\nfunc (lvs *ValueStore) WriteValue(v Value) Ref {\n\tlvs.versOnce.Do(lvs.expectVersion)\n\td.PanicIfFalse(v != nil)\n\n\tc := EncodeValue(v)\n\td.PanicIfTrue(c.IsEmpty())\n\th := c.Hash()\n\theight := maxChunkHeight(v) + 1\n\tr := constructRef(h, TypeOf(v), height)\n\tlvs.bufferChunk(v, c, height)\n\treturn r\n}\n\n// bufferChunk enqueues c (which is the serialization of v) within this\n// ValueStore. Buffered chunks are flushed progressively to the underlying\n// ChunkStore in a way which attempts to locate children and grandchildren\n// sequentially together. The following invariants are retained:\n//\n// 1. For any given chunk currently in the buffer, only direct children of the\n// chunk may also be presently buffered (any grandchildren will have been\n// flushed).\n// 2. The total data occupied by buffered chunks does not exceed\n// lvs.bufferedChunksMax\nfunc (lvs *ValueStore) bufferChunk(v Value, c chunks.Chunk, height uint64) {\n\tlvs.bufferMu.Lock()\n\tdefer lvs.bufferMu.Unlock()\n\n\td.PanicIfTrue(height == 0)\n\th := c.Hash()\n\tif _, present := lvs.bufferedChunks[h]; !present {\n\t\tlvs.bufferedChunks[h] = c\n\t\tlvs.bufferedChunkSize += uint64(len(c.Data()))\n\t}\n\n\tput := func(h hash.Hash, c chunks.Chunk) {\n\t\tlvs.cs.Put(c)\n\t\tlvs.bufferedChunkSize -= uint64(len(c.Data()))\n\t\tdelete(lvs.bufferedChunks, h)\n\t}\n\n\tputChildren := func(parent hash.Hash) {\n\t\tpending, isBuffered := lvs.bufferedChunks[parent]\n\t\tif !isBuffered {\n\t\t\treturn\n\t\t}\n\t\tWalkRefs(pending, func(grandchildRef Ref) {\n\t\t\tgch := grandchildRef.TargetHash()\n\t\t\tif pending, present := lvs.bufferedChunks[gch]; present {\n\t\t\t\tput(gch, pending)\n\t\t\t}\n\t\t})\n\t\tdelete(lvs.withBufferedChildren, parent)\n\t\treturn\n\t}\n\n\t// Enforce invariant (1)\n\tif height > 1 {\n\t\tv.WalkRefs(func(childRef Ref) {\n\t\t\tchildHash := childRef.TargetHash()\n\t\t\tif _, isBuffered := lvs.bufferedChunks[childHash]; isBuffered {\n\t\t\t\tlvs.withBufferedChildren[h] = height\n\t\t\t} else if lvs.enforceCompleteness {\n\t\t\t\t// If the childRef isn't presently buffered, we must consider it an\n\t\t\t\t// unresolved ref.\n\t\t\t\tlvs.unresolvedRefs.Insert(childHash)\n\t\t\t}\n\t\t\tif _, hasBufferedChildren := lvs.withBufferedChildren[childHash]; hasBufferedChildren {\n\t\t\t\tputChildren(childHash)\n\t\t\t}\n\t\t})\n\t}\n\n\t// Enforce invariant (2)\n\tfor lvs.bufferedChunkSize > lvs.bufferedChunksMax {\n\t\tvar tallest hash.Hash\n\t\tvar height uint64 = 0\n\t\tfor parent, ht := range lvs.withBufferedChildren {\n\t\t\tif ht > height {\n\t\t\t\ttallest = parent\n\t\t\t\theight = ht\n\t\t\t}\n\t\t}\n\t\tif height == 0 { // This can happen if there are no pending parents\n\t\t\tvar chunk chunks.Chunk\n\t\t\tfor tallest, chunk = range lvs.bufferedChunks {\n\t\t\t\t// Any pendingPut is as good as another in this case, so take the first one\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tput(tallest, chunk)\n\t\t\tcontinue\n\t\t}\n\n\t\tputChildren(tallest)\n\t}\n}\n\nfunc (lvs *ValueStore) Root() hash.Hash {\n\treturn lvs.cs.Root()\n}\n\nfunc (lvs *ValueStore) Rebase() {\n\tlvs.cs.Rebase()\n}\n\n// Commit() flushes all bufferedChunks into the ChunkStore, with best-effort\n// locality, and attempts to Commit, updating the root to |current| (or keeping\n// it the same as Root()). If the root has moved since this ValueStore was\n// opened, or last Rebased(), it will return false and will have internally\n// rebased. Until Commit() succeeds, no work of the ValueStore will be visible\n// to other readers of the underlying ChunkStore.\nfunc (lvs *ValueStore) Commit(current, last hash.Hash) bool {\n\treturn func() bool {\n\t\tlvs.bufferMu.Lock()\n\t\tdefer lvs.bufferMu.Unlock()\n\n\t\tput := func(h hash.Hash, chunk chunks.Chunk) {\n\t\t\tlvs.cs.Put(chunk)\n\t\t\tdelete(lvs.bufferedChunks, h)\n\t\t\tlvs.bufferedChunkSize -= uint64(len(chunk.Data()))\n\t\t}\n\n\t\tfor parent := range lvs.withBufferedChildren {\n\t\t\tif pending, present := lvs.bufferedChunks[parent]; present {\n\t\t\t\tWalkRefs(pending, func(reachable Ref) {\n\t\t\t\t\tif pending, present := lvs.bufferedChunks[reachable.TargetHash()]; present {\n\t\t\t\t\t\tput(reachable.TargetHash(), pending)\n\t\t\t\t\t}\n\t\t\t\t})\n\t\t\t\tput(parent, pending)\n\t\t\t}\n\t\t}\n\t\tfor _, c := range lvs.bufferedChunks {\n\t\t\t// Can't use put() because it's wrong to delete from a lvs.bufferedChunks while iterating it.\n\t\t\tlvs.cs.Put(c)\n\t\t\tlvs.bufferedChunkSize -= uint64(len(c.Data()))\n\t\t}\n\t\td.PanicIfFalse(lvs.bufferedChunkSize == 0)\n\t\tlvs.withBufferedChildren = map[hash.Hash]uint64{}\n\t\tlvs.bufferedChunks = map[hash.Hash]chunks.Chunk{}\n\n\t\tif lvs.enforceCompleteness {\n\t\t\tif (current != hash.Hash{} && current != lvs.Root()) {\n\t\t\t\tif _, ok := lvs.bufferedChunks[current]; !ok {\n\t\t\t\t\t// If the client is attempting to move the root and the referenced\n\t\t\t\t\t// value isn't still buffered, we need to ensure that it is contained\n\t\t\t\t\t// in the ChunkStore.\n\t\t\t\t\tlvs.unresolvedRefs.Insert(current)\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tPanicIfDangling(lvs.unresolvedRefs, lvs.cs)\n\t\t}\n\n\t\tif !lvs.cs.Commit(current, last) {\n\t\t\treturn false\n\t\t}\n\n\t\tif lvs.enforceCompleteness {\n\t\t\tlvs.unresolvedRefs = hash.HashSet{}\n\t\t}\n\n\t\treturn true\n\t}()\n}\n\n// Close closes the underlying ChunkStore\nfunc (lvs *ValueStore) Close() error {\n\treturn lvs.cs.Close()\n}\n\nfunc getTargetType(refBase Ref) *Type {\n\trefType := TypeOf(refBase)\n\td.PanicIfFalse(RefKind == refType.TargetKind())\n\treturn refType.Desc.(CompoundDesc).ElemTypes[0]\n}\n" }, { "path": "go/types/value_store_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestValueReadWriteRead(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts := String(\"hello\")\n\tvs := newTestValueStore()\n\tassert.Nil(vs.ReadValue(s.Hash())) // nil\n\th := vs.WriteValue(s).TargetHash()\n\tvs.Commit(vs.Root(), vs.Root())\n\tv := vs.ReadValue(h) // non-nil\n\tif assert.NotNil(v) {\n\t\tassert.True(s.Equals(v), \"%s != %s\", EncodedValue(s), EncodedValue(v))\n\t}\n}\n\nfunc TestReadWriteCache(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.TestStorage{}\n\tts := storage.NewView()\n\tvs := NewValueStore(ts)\n\n\tvar v Value = Bool(true)\n\tr := vs.WriteValue(v)\n\tassert.NotEqual(hash.Hash{}, r.TargetHash())\n\tvs.Commit(vs.Root(), vs.Root())\n\tassert.Equal(1, ts.Writes)\n\n\tv = vs.ReadValue(r.TargetHash())\n\tassert.True(v.Equals(Bool(true)))\n\tassert.Equal(1, ts.Reads)\n\n\tv = vs.ReadValue(r.TargetHash())\n\tassert.True(v.Equals(Bool(true)))\n\tassert.Equal(1, ts.Reads)\n}\n\nfunc TestValueReadMany(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvals := ValueSlice{String(\"hello\"), Bool(true), Number(42)}\n\tvs := newTestValueStore()\n\thashes := hash.HashSlice{}\n\tfor _, v := range vals {\n\t\th := vs.WriteValue(v).TargetHash()\n\t\thashes = append(hashes, h)\n\t\tvs.Commit(vs.Root(), vs.Root())\n\t}\n\n\t// Get one Value into vs's Value cache\n\tvs.ReadValue(vals[0].Hash())\n\n\t// Get one Value into vs's pendingPuts\n\tthree := Number(3)\n\tvals = append(vals, three)\n\tvs.WriteValue(three)\n\thashes = append(hashes, three.Hash())\n\n\t// Add one Value to request that's not in vs\n\thashes = append(hashes, Bool(false).Hash())\n\n\tfound := map[hash.Hash]Value{}\n\treadValues := vs.ReadManyValues(hashes)\n\n\tfor i, v := range readValues {\n\t\tif v != nil {\n\t\t\tfound[hashes[i]] = v\n\t\t}\n\t}\n\n\tassert.Len(found, len(vals))\n\tfor _, v := range vals {\n\t\tassert.True(v.Equals(found[v.Hash()]))\n\t}\n}\n\nfunc TestValueWriteFlush(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvals := ValueSlice{String(\"hello\"), Bool(true), Number(42)}\n\tvs := newTestValueStore()\n\thashes := hash.HashSet{}\n\tfor _, v := range vals {\n\t\thashes.Insert(vs.WriteValue(v).TargetHash())\n\t}\n\tassert.NotZero(vs.bufferedChunkSize)\n\n\tvs.Commit(vs.Root(), vs.Root())\n\tassert.Zero(vs.bufferedChunkSize)\n}\n\ntype checkingChunkStore struct {\n\tchunks.ChunkStore\n\ta *assert.Assertions\n\texpectedOrder hash.HashSlice\n}\n\nfunc (cbs *checkingChunkStore) expect(rs ...Ref) {\n\tfor _, r := range rs {\n\t\tcbs.expectedOrder = append(cbs.expectedOrder, r.TargetHash())\n\t}\n}\n\nfunc (cbs *checkingChunkStore) Put(c chunks.Chunk) {\n\tif cbs.a.NotZero(len(cbs.expectedOrder), \"Unexpected Put of %s\", c.Hash()) {\n\t\tcbs.a.Equal(cbs.expectedOrder[0], c.Hash())\n\t\tcbs.expectedOrder = cbs.expectedOrder[1:]\n\t}\n\tcbs.ChunkStore.Put(c)\n}\n\nfunc (cbs *checkingChunkStore) Flush() {\n\tcbs.a.Empty(cbs.expectedOrder)\n}\n\nfunc TestFlushOrder(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.TestStorage{}\n\tccs := &checkingChunkStore{storage.NewView(), assert, nil}\n\tvs := NewValueStore(ccs)\n\t// Graph, which should be flushed grandchildren-first, bottom-up\n\t// l\n\t// / \\\n\t// ml1 ml2\n\t// / \\ \\\n\t// b ml f\n\t// / \\\n\t// s n\n\t//\n\t// Expected order: s, n, b, ml, f, ml1, ml2, l\n\ts := String(\"oy\")\n\tn := Number(42)\n\tsr, nr := vs.WriteValue(s), vs.WriteValue(n)\n\tccs.expect(sr, nr)\n\tml := NewList(vs, sr, nr)\n\n\tb := NewEmptyBlob(vs)\n\tbr, mlr := vs.WriteValue(b), vs.WriteValue(ml)\n\tccs.expect(br, mlr)\n\tml1 := NewList(vs, br, mlr)\n\n\tf := Bool(false)\n\tfr := vs.WriteValue(f)\n\tccs.expect(fr)\n\tml2 := NewList(vs, fr)\n\n\tml1r, ml2r := vs.WriteValue(ml1), vs.WriteValue(ml2)\n\tccs.expect(ml1r, ml2r)\n\tl := NewList(vs, ml1r, ml2r)\n\n\tr := vs.WriteValue(l)\n\tccs.expect(r)\n\tvs.Commit(vs.Root(), vs.Root())\n}\n\nfunc TestFlushOverSize(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.TestStorage{}\n\tccs := &checkingChunkStore{storage.NewView(), assert, nil}\n\tvs := newValueStoreWithCacheAndPending(ccs, 0, 30)\n\n\ts := String(\"oy\")\n\tsr := vs.WriteValue(s)\n\tccs.expect(sr)\n\tNewList(vs, sr) // will write the root chunk\n}\n\nfunc TestTolerateTopDown(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.TestStorage{}\n\tccs := &checkingChunkStore{storage.NewView(), assert, nil}\n\tvs := NewValueStore(ccs)\n\t// Once the L-ML-S portion of this graph is written once, it's legal to make a Struct ST that contains a ref directly to ML and write it. Then you can write S and ML and Flush ST, which contitutes top-down writing.\n\t// L ST\n\t// \\ /\n\t// ML\n\t// /\n\t// S\n\tS := String(\"oy\")\n\tsr := vs.WriteValue(S)\n\tccs.expect(sr)\n\n\tML := NewList(vs, sr)\n\tmlr := vs.WriteValue(ML)\n\tccs.expect(mlr)\n\n\tL := NewList(vs, mlr)\n\tlr := vs.WriteValue(L)\n\tccs.expect(lr)\n\n\tvs.Commit(vs.Root(), vs.Root())\n\n\tassert.Zero(len(vs.bufferedChunks))\n\n\tST := NewStruct(\"\", StructData{\"r\": mlr})\n\tstr := vs.WriteValue(ST) // ST into bufferedChunks\n\tvs.WriteValue(S) // S into bufferedChunks\n\tvs.WriteValue(ML) // ML into bufferedChunks AND withBufferedChunks\n\n\t// At this point, ValueStore believes ST is a standalone chunk, and that ML -> S\n\t// So, it'll look at ML, the one parent it knows about, first and write its child (S). Then, it'll write ML, and then it'll flush the remaining buffered chunks, which is just ST.\n\tccs.expect(sr, mlr, str)\n\tvs.Commit(vs.Root(), vs.Root())\n}\n\nfunc TestPanicOnBadVersion(t *testing.T) {\n\tstorage := &chunks.MemoryStorage{}\n\tt.Run(\"Read\", func(t *testing.T) {\n\t\tcvs := NewValueStore(&badVersionStore{ChunkStore: storage.NewView()})\n\t\tassert.Panics(t, func() { cvs.ReadValue(hash.Hash{}) })\n\t})\n\tt.Run(\"Write\", func(t *testing.T) {\n\t\tcvs := NewValueStore(&badVersionStore{ChunkStore: storage.NewView()})\n\t\tassert.Panics(t, func() {\n\t\t\tcvs.WriteValue(NewEmptyBlob(cvs))\n\t\t\tcvs.Commit(cvs.Root(), cvs.Root())\n\t\t})\n\t})\n}\n\nfunc TestPanicIfDangling(t *testing.T) {\n\tassert := assert.New(t)\n\tvs := newTestValueStore()\n\n\tr := NewRef(Bool(true))\n\tl := NewList(vs, r)\n\tvs.WriteValue(l)\n\n\tassert.Panics(func() {\n\t\tvs.Commit(vs.Root(), vs.Root())\n\t})\n}\n\nfunc TestSkipEnforceCompleteness(t *testing.T) {\n\tvs := newTestValueStore()\n\tvs.SetEnforceCompleteness(false)\n\n\tr := NewRef(Bool(true))\n\tl := NewList(vs, r)\n\tvs.WriteValue(l)\n\n\tvs.Commit(vs.Root(), vs.Root())\n}\n\ntype badVersionStore struct {\n\tchunks.ChunkStore\n}\n\nfunc (b *badVersionStore) Version() string {\n\treturn \"BAD\"\n}\n" }, { "path": "go/types/walk.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport \"github.com/attic-labs/noms/go/hash\"\n\ntype SkipValueCallback func(v Value) bool\n\n// WalkValues loads prolly trees progressively by walking down the tree. We don't wants to invoke\n// the value callback on internal sub-trees (which are valid values) because they are not logical\n// values in the graph\ntype valueRec struct {\n\tv Value\n\tcb bool\n}\n\nconst maxRefCount = 1 << 12 // ~16MB of data\n\n// WalkValues recursively walks over all types.Values reachable from r and calls cb on them.\nfunc WalkValues(target Value, vr ValueReader, cb SkipValueCallback) {\n\tvisited := hash.HashSet{}\n\trefs := map[hash.Hash]bool{}\n\tvalues := []valueRec{{target, true}}\n\n\tfor len(values) > 0 || len(refs) > 0 {\n\t\tfor len(values) > 0 {\n\t\t\trec := values[len(values)-1]\n\t\t\tvalues = values[:len(values)-1]\n\n\t\t\tv := rec.v\n\t\t\tif rec.cb && cb(v) {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif _, ok := v.(Blob); ok {\n\t\t\t\tcontinue // don't traverse into blob ptrees\n\t\t\t}\n\n\t\t\tif r, ok := v.(Ref); ok {\n\t\t\t\trefs[r.TargetHash()] = true\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif col, ok := v.(Collection); ok && !col.asSequence().isLeaf() {\n\t\t\t\tcol.WalkRefs(func(r Ref) {\n\t\t\t\t\trefs[r.TargetHash()] = false\n\t\t\t\t})\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tv.WalkValues(func(sv Value) {\n\t\t\t\tvalues = append(values, valueRec{sv, true})\n\t\t\t})\n\t\t}\n\n\t\tif len(refs) == 0 {\n\t\t\tcontinue\n\t\t}\n\n\t\ths := make(hash.HashSlice, 0, len(refs))\n\t\toldRefs := refs\n\t\trefs = map[hash.Hash]bool{}\n\t\tfor h := range oldRefs {\n\t\t\tif _, ok := visited[h]; ok {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif len(hs) >= maxRefCount {\n\t\t\t\trefs[h] = oldRefs[h]\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\ths = append(hs, h)\n\t\t\tvisited.Insert(h)\n\t\t}\n\n\t\tif len(hs) > 0 {\n\t\t\treadValues := vr.ReadManyValues(hs)\n\t\t\tfor i, sv := range readValues {\n\t\t\t\tvalues = append(values, valueRec{sv, oldRefs[hs[i]]})\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc mightContainStructs(t *Type) (mightHaveStructs bool) {\n\tif t.TargetKind() == StructKind || t.TargetKind() == ValueKind {\n\t\tmightHaveStructs = true\n\t\treturn\n\t}\n\n\tt.WalkValues(func(v Value) {\n\t\tmightHaveStructs = mightHaveStructs || mightContainStructs(v.(*Type))\n\t})\n\n\treturn\n}\n" }, { "path": "go/types/walk_refs.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// WalkRefs calls cb() on each Ref that can be decoded from |c|. The results\n// are precisely equal to DecodeValue(c).WalkRefs(cb), but this should be much\n// faster.\nfunc WalkRefs(c chunks.Chunk, cb RefCallback) {\n\twalkRefs(c.Data(), cb)\n}\n\nfunc walkRefs(data []byte, cb RefCallback) {\n\trw := newRefWalker(data)\n\trw.walkValue(cb)\n}\n\ntype refWalker struct {\n\ttypedBinaryNomsReader\n}\n\nfunc newRefWalker(buff []byte) refWalker {\n\tnr := binaryNomsReader{buff, 0}\n\treturn refWalker{typedBinaryNomsReader{nr, false}}\n}\n\nfunc (r *refWalker) walkRef(cb RefCallback) {\n\tcb(readRef(&(r.typedBinaryNomsReader)))\n}\n\nfunc (r *refWalker) walkBlobLeafSequence() {\n\tsize := r.readCount()\n\tr.offset += uint32(size)\n}\n\nfunc (r *refWalker) walkValueSequence(cb RefCallback) {\n\tcount := int(r.readCount())\n\tfor i := 0; i < count; i++ {\n\t\tr.walkValue(cb)\n\t}\n}\n\nfunc (r *refWalker) walkList(cb RefCallback) {\n\tr.walkListOrSet(ListKind, cb)\n}\n\nfunc (r *refWalker) walkSet(cb RefCallback) {\n\tr.walkListOrSet(SetKind, cb)\n}\n\nfunc (r *refWalker) walkListOrSet(kind NomsKind, cb RefCallback) {\n\tr.skipKind()\n\tlevel := r.readCount()\n\tif level > 0 {\n\t\tr.walkMetaSequence(kind, level, cb)\n\t} else {\n\t\tr.walkValueSequence(cb)\n\t}\n}\n\nfunc (r *refWalker) walkMap(cb RefCallback) {\n\tr.skipKind()\n\tlevel := r.readCount()\n\tif level > 0 {\n\t\tr.walkMetaSequence(MapKind, level, cb)\n\t} else {\n\t\tr.walkMapLeafSequence(cb)\n\t}\n}\n\nfunc (r *refWalker) walkBlob(cb RefCallback) {\n\tr.skipKind()\n\tlevel := r.readCount()\n\tif level > 0 {\n\t\tr.walkMetaSequence(BlobKind, level, cb)\n\t} else {\n\t\tr.walkBlobLeafSequence()\n\t}\n}\n\nfunc (r *refWalker) walkMapLeafSequence(cb RefCallback) {\n\tcount := r.readCount()\n\tfor i := uint64(0); i < count; i++ {\n\t\tr.walkValue(cb) // k\n\t\tr.walkValue(cb) // v\n\t}\n}\n\nfunc (r *refWalker) walkMetaSequence(k NomsKind, level uint64, cb RefCallback) {\n\tcount := r.readCount()\n\tfor i := uint64(0); i < count; i++ {\n\t\tr.walkRef(cb) // ref to child sequence\n\t\tr.skipOrderedKey()\n\t\tr.skipCount() // numLeaves\n\t}\n}\n\nfunc (r *refWalker) skipOrderedKey() {\n\tswitch r.peekKind() {\n\tcase hashKind:\n\t\tr.skipKind()\n\t\tr.skipHash()\n\tdefault:\n\t\tr.walkValue(func(r Ref) {}) // max Value in subtree reachable from here\n\t}\n}\n\nfunc (r *refWalker) walkValue(cb RefCallback) {\n\tk := r.peekKind()\n\tswitch k {\n\tcase BlobKind:\n\t\tr.walkBlob(cb)\n\tcase BoolKind:\n\t\tr.skipKind()\n\t\tr.skipBool()\n\tcase NumberKind:\n\t\tr.skipKind()\n\t\tr.skipNumber()\n\tcase StringKind:\n\t\tr.skipKind()\n\t\tr.skipString()\n\tcase ListKind:\n\t\tr.walkList(cb)\n\tcase MapKind:\n\t\tr.walkMap(cb)\n\tcase RefKind:\n\t\tr.walkRef(cb)\n\tcase SetKind:\n\t\tr.walkSet(cb)\n\tcase StructKind:\n\t\tr.walkStruct(cb)\n\tcase TypeKind:\n\t\tr.skipKind()\n\t\tr.skipType()\n\tcase CycleKind, UnionKind, ValueKind:\n\t\td.Panic(\"A value instance can never have type %s\", k)\n\tdefault:\n\t\tpanic(\"not reachable\")\n\t}\n}\n\nfunc (r *refWalker) walkStruct(cb RefCallback) {\n\twalkStruct(r, cb)\n}\n" }, { "path": "go/types/walk_refs_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"io\"\n\t\"math/rand\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestWalkRefs(t *testing.T) {\n\trunTest := func(v Value, t *testing.T) {\n\t\tassert := assert.New(t)\n\t\texpected := hash.HashSlice{}\n\t\tv.WalkRefs(func(r Ref) {\n\t\t\texpected = append(expected, r.TargetHash())\n\t\t})\n\t\tWalkRefs(EncodeValue(v), func(r Ref) {\n\t\t\tif assert.True(len(expected) > 0) {\n\t\t\t\tassert.Equal(expected[0], r.TargetHash())\n\t\t\t\texpected = expected[1:]\n\t\t\t}\n\t\t})\n\t\tassert.Len(expected, 0)\n\t}\n\n\tt.Run(\"SingleRef\", func(t *testing.T) {\n\t\tt.Parallel()\n\t\tt.Run(\"Typed\", func(t *testing.T) {\n\t\t\tvrw := newTestValueStore()\n\t\t\ts := NewStruct(\"\", StructData{\"n\": Number(1)})\n\t\t\trunTest(NewRef(NewMap(vrw, s, Number(2))), t)\n\t\t})\n\t\tt.Run(\"OfValue\", func(t *testing.T) {\n\t\t\trunTest(ToRefOfValue(NewRef(Bool(false))), t)\n\t\t})\n\t})\n\n\tt.Run(\"Struct\", func(t *testing.T) {\n\t\tt.Parallel()\n\t\tdata := StructData{\n\t\t\t\"ref\": NewRef(Bool(false)),\n\t\t\t\"num\": Number(42),\n\t\t}\n\t\trunTest(NewStruct(\"nom\", data), t)\n\t})\n\n\t// must return a slice with an even number of elements\n\tnewValueSlice := func(r *rand.Rand) ValueSlice {\n\t\tvs := make(ValueSlice, 256)\n\t\tfor i := range vs {\n\t\t\tvs[i] = NewStruct(\"\", StructData{\"n\": Number(r.Uint64())})\n\t\t}\n\t\treturn vs\n\t}\n\n\tt.Run(\"List\", func(t *testing.T) {\n\t\tt.Parallel()\n\t\tvrw := newTestValueStore()\n\t\tr := rand.New(rand.NewSource(0))\n\n\t\tt.Run(\"OfRefs\", func(t *testing.T) {\n\t\t\tl := NewList(vrw, vrw.WriteValue(Number(42)), vrw.WriteValue(Number(0)))\n\t\t\trunTest(l, t)\n\t\t})\n\n\t\tt.Run(\"Chunked\", func(t *testing.T) {\n\t\t\tl := NewList(vrw, newValueSlice(r)...)\n\t\t\tfor l.sequence.isLeaf() {\n\t\t\t\tl = l.Concat(NewList(vrw, newValueSlice(r)...))\n\t\t\t}\n\t\t\trunTest(l, t)\n\t\t})\n\t})\n\n\tt.Run(\"Set\", func(t *testing.T) {\n\t\tt.Parallel()\n\t\tvrw := newTestValueStore()\n\t\tr := rand.New(rand.NewSource(0))\n\n\t\tt.Run(\"OfRefs\", func(t *testing.T) {\n\t\t\ts := NewSet(vrw, vrw.WriteValue(Number(42)), vrw.WriteValue(Number(0)))\n\t\t\trunTest(s, t)\n\t\t})\n\n\t\tt.Run(\"Chunked\", func(t *testing.T) {\n\t\t\ts := NewSet(vrw, newValueSlice(r)...)\n\t\t\tfor s.isLeaf() {\n\t\t\t\te := s.Edit()\n\t\t\t\te = e.Insert(newValueSlice(r)...)\n\t\t\t\ts = e.Set()\n\t\t\t}\n\t\t\trunTest(s, t)\n\t\t})\n\t})\n\n\tt.Run(\"Map\", func(t *testing.T) {\n\t\tt.Parallel()\n\t\tvrw := newTestValueStore()\n\t\tr := rand.New(rand.NewSource(0))\n\n\t\tt.Run(\"OfRefs\", func(t *testing.T) {\n\t\t\tm := NewMap(vrw, vrw.WriteValue(Number(42)), vrw.WriteValue(Number(0)))\n\t\t\trunTest(m, t)\n\t\t})\n\n\t\tt.Run(\"Chunked\", func(t *testing.T) {\n\t\t\tm := NewMap(vrw, newValueSlice(r)...)\n\t\t\tfor m.isLeaf() {\n\t\t\t\te := m.Edit()\n\t\t\t\tvs := newValueSlice(r)\n\t\t\t\tfor i := 0; i < len(vs); i += 2 {\n\t\t\t\t\te = e.Set(vs[i], vs[i+1])\n\t\t\t\t}\n\t\t\t\tm = e.Map()\n\t\t\t}\n\t\t\trunTest(m, t)\n\t\t})\n\t})\n\n\tt.Run(\"Blob\", func(t *testing.T) {\n\t\tt.Parallel()\n\t\tvrw := newTestValueStore()\n\t\tr := rand.New(rand.NewSource(0))\n\n\t\tscratch := make([]byte, 1024)\n\t\tfreshRandomBytes := func() io.Reader {\n\t\t\tr.Read(scratch)\n\t\t\treturn bytes.NewReader(scratch)\n\t\t}\n\t\tb := NewBlob(vrw, freshRandomBytes())\n\t\tfor b.sequence.isLeaf() {\n\t\t\tb = b.Concat(NewBlob(vrw, freshRandomBytes()))\n\t\t}\n\t\trunTest(b, t)\n\t})\n}\n" }, { "path": "go/types/walk_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage types\n\nimport (\n\t\"bytes\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc TestWalkTestSuite(t *testing.T) {\n\tsuite.Run(t, &WalkTestSuite{})\n}\n\nfunc TestWalkAllTestSuite(t *testing.T) {\n\tsuite.Run(t, &WalkAllTestSuite{})\n}\n\ntype WalkAllTestSuite struct {\n\tsuite.Suite\n\tvs *ValueStore\n\tts *chunks.TestStoreView\n}\n\nfunc (suite *WalkAllTestSuite) SetupTest() {\n\tstorage := &chunks.TestStorage{}\n\tsuite.ts = storage.NewView()\n\tsuite.vs = NewValueStore(suite.ts)\n}\n\nfunc (suite *WalkAllTestSuite) assertCallbackCount(v Value, expected int) {\n\tactual := 0\n\tWalkValues(v, suite.vs, func(c Value) (stop bool) {\n\t\tactual++\n\t\treturn\n\t})\n\tsuite.Equal(expected, actual)\n}\n\nfunc (suite *WalkAllTestSuite) assertVisitedOnce(root, v Value) {\n\tactual := 0\n\tWalkValues(v, suite.vs, func(c Value) bool {\n\t\tif c == v {\n\t\t\tactual++\n\t\t}\n\t\treturn false\n\t})\n\tsuite.Equal(1, actual)\n}\n\nfunc (suite *WalkAllTestSuite) TestWalkValuesDuplicates() {\n\tdup := suite.NewList(Number(9), Number(10), Number(11), Number(12), Number(13))\n\tl := suite.NewList(Number(8), dup, dup)\n\n\tsuite.assertCallbackCount(l, 11)\n}\n\nfunc (suite *WalkAllTestSuite) TestWalkAvoidBlobChunks() {\n\tbuff := randomBuff(16)\n\tblob := NewBlob(suite.vs, bytes.NewReader(buff))\n\tr := suite.vs.WriteValue(blob)\n\tsuite.True(r.Height() > 1)\n\toutBlob := suite.vs.ReadValue(r.TargetHash()).(Blob)\n\tsuite.Equal(suite.ts.Reads, 0)\n\tsuite.assertCallbackCount(outBlob, 1)\n\tsuite.Equal(suite.ts.Reads, 0)\n}\n\nfunc (suite *WalkAllTestSuite) TestWalkPrimitives() {\n\tsuite.assertCallbackCount(suite.vs.WriteValue(Number(0.0)), 2)\n\tsuite.assertCallbackCount(suite.vs.WriteValue(String(\"hello\")), 2)\n}\n\nfunc (suite *WalkAllTestSuite) TestWalkComposites() {\n\tsuite.assertCallbackCount(suite.NewList(), 2)\n\tsuite.assertCallbackCount(suite.NewList(Bool(false), Number(8)), 4)\n\tsuite.assertCallbackCount(suite.NewSet(), 2)\n\tsuite.assertCallbackCount(suite.NewSet(Bool(false), Number(8)), 4)\n\tsuite.assertCallbackCount(suite.NewMap(), 2)\n\tsuite.assertCallbackCount(suite.NewMap(Number(8), Bool(true), Number(0), Bool(false)), 6)\n}\n\nfunc (suite *WalkAllTestSuite) TestWalkMultilevelList() {\n\tcount := 1 << 12\n\tnums := make([]Value, count)\n\tfor i := 0; i < count; i++ {\n\t\tnums[i] = Number(i)\n\t}\n\tl := NewList(suite.vs, nums...)\n\tsuite.True(NewRef(l).Height() > 1)\n\tsuite.assertCallbackCount(l, count+1)\n\n\tr := suite.vs.WriteValue(l)\n\toutList := suite.vs.ReadValue(r.TargetHash())\n\tsuite.assertCallbackCount(outList, count+1)\n}\n\nfunc (suite *WalkAllTestSuite) TestWalkType() {\n\tt := MakeStructTypeFromFields(\"TestStruct\", FieldMap{\n\t\t\"s\": StringType,\n\t\t\"b\": BoolType,\n\t\t\"n\": NumberType,\n\t\t\"bl\": BlobType,\n\t\t\"t\": TypeType,\n\t\t\"v\": ValueType,\n\t})\n\tsuite.assertVisitedOnce(t, t)\n\tsuite.assertVisitedOnce(t, BoolType)\n\tsuite.assertVisitedOnce(t, NumberType)\n\tsuite.assertVisitedOnce(t, StringType)\n\tsuite.assertVisitedOnce(t, BlobType)\n\tsuite.assertVisitedOnce(t, TypeType)\n\tsuite.assertVisitedOnce(t, ValueType)\n\n\t{\n\t\tt2 := MakeListType(BoolType)\n\t\tsuite.assertVisitedOnce(t2, t2)\n\t\tsuite.assertVisitedOnce(t2, BoolType)\n\t}\n\n\t{\n\t\tt2 := MakeSetType(BoolType)\n\t\tsuite.assertVisitedOnce(t2, t2)\n\t\tsuite.assertVisitedOnce(t2, BoolType)\n\t}\n\n\t{\n\t\tt2 := MakeRefType(BoolType)\n\t\tsuite.assertVisitedOnce(t2, t2)\n\t\tsuite.assertVisitedOnce(t2, BoolType)\n\t}\n\n\tt2 := MakeMapType(NumberType, StringType)\n\tsuite.assertVisitedOnce(t2, t2)\n\tsuite.assertVisitedOnce(t2, NumberType)\n\tsuite.assertVisitedOnce(t2, StringType)\n\n\tt3 := MakeUnionType(NumberType, StringType, BoolType)\n\tsuite.assertVisitedOnce(t3, t3)\n\tsuite.assertVisitedOnce(t3, BoolType)\n\tsuite.assertVisitedOnce(t3, NumberType)\n\tsuite.assertVisitedOnce(t3, StringType)\n\n\tt4 := MakeCycleType(\"ABC\")\n\tsuite.assertVisitedOnce(t4, t4)\n}\n\nfunc (suite *WalkTestSuite) skipWorker(composite Value) (reached ValueSlice) {\n\tWalkValues(composite, suite.vs, func(v Value) bool {\n\t\tsuite.False(v.Equals(suite.deadValue), \"Should never have reached %+v\", suite.deadValue)\n\t\treached = append(reached, v)\n\t\treturn v.Equals(suite.mustSkip)\n\t})\n\treturn\n}\n\n// Skipping a sub-tree must allow other items in the list to be processed.\nfunc (suite *WalkTestSuite) TestSkipListElement() {\n\twholeList := NewList(suite.vs, suite.mustSkip, suite.shouldSee, suite.shouldSee)\n\treached := suite.skipWorker(wholeList)\n\tfor _, v := range []Value{wholeList, suite.mustSkip, suite.shouldSee, suite.shouldSeeItem} {\n\t\tsuite.True(reached.Contains(v), \"Doesn't contain %+v\", v)\n\t}\n\tsuite.Len(reached, 6)\n}\n\nfunc (suite *WalkTestSuite) TestSkipSetElement() {\n\twholeSet := NewSet(suite.vs, suite.mustSkip, suite.shouldSee).Edit().Insert(suite.shouldSee).Set()\n\treached := suite.skipWorker(wholeSet)\n\tfor _, v := range []Value{wholeSet, suite.mustSkip, suite.shouldSee, suite.shouldSeeItem} {\n\t\tsuite.True(reached.Contains(v), \"Doesn't contain %+v\", v)\n\t}\n\tsuite.Len(reached, 4)\n}\n\nfunc (suite *WalkTestSuite) TestSkipMapValue() {\n\tshouldAlsoSeeItem := String(\"Also good\")\n\tshouldAlsoSee := NewSet(suite.vs, shouldAlsoSeeItem)\n\twholeMap := NewMap(suite.vs, suite.shouldSee, suite.mustSkip, shouldAlsoSee, suite.shouldSee)\n\treached := suite.skipWorker(wholeMap)\n\tfor _, v := range []Value{wholeMap, suite.shouldSee, suite.shouldSeeItem, suite.mustSkip, shouldAlsoSee, shouldAlsoSeeItem} {\n\t\tsuite.True(reached.Contains(v), \"Doesn't contain %+v\", v)\n\t}\n\tsuite.Len(reached, 8)\n}\n\nfunc (suite *WalkTestSuite) TestSkipMapKey() {\n\twholeMap := NewMap(suite.vs, suite.mustSkip, suite.shouldSee, suite.shouldSee, suite.shouldSee)\n\treached := suite.skipWorker(wholeMap)\n\tfor _, v := range []Value{wholeMap, suite.mustSkip, suite.shouldSee, suite.shouldSeeItem} {\n\t\tsuite.True(reached.Contains(v), \"Doesn't contain %+v\", v)\n\t}\n\tsuite.Len(reached, 8)\n}\n\nfunc (suite *WalkAllTestSuite) NewList(vs ...Value) Ref {\n\tv := NewList(suite.vs, vs...)\n\treturn suite.vs.WriteValue(v)\n}\n\nfunc (suite *WalkAllTestSuite) NewMap(vs ...Value) Ref {\n\tv := NewMap(suite.vs, vs...)\n\treturn suite.vs.WriteValue(v)\n}\n\nfunc (suite *WalkAllTestSuite) NewSet(vs ...Value) Ref {\n\tv := NewSet(suite.vs, vs...)\n\treturn suite.vs.WriteValue(v)\n}\n\nfunc (suite *WalkAllTestSuite) TestWalkNestedComposites() {\n\tsuite.assertCallbackCount(suite.NewList(suite.NewSet(), Number(8)), 5)\n\tsuite.assertCallbackCount(suite.NewSet(suite.NewList(), suite.NewSet()), 6)\n\t// {\"string\": \"string\",\n\t// \"list\": [false true],\n\t// \"map\": {\"nested\": \"string\"}\n\t// \"mtlist\": []\n\t// \"set\": [5 7 8]\n\t// []: \"wow\"\n\t// }\n\tnested := suite.NewMap(\n\t\tString(\"string\"), String(\"string\"),\n\t\tString(\"list\"), suite.NewList(Bool(false), Bool(true)),\n\t\tString(\"map\"), suite.NewMap(String(\"nested\"), String(\"string\")),\n\t\tString(\"mtlist\"), suite.NewList(),\n\t\tString(\"set\"), suite.NewSet(Number(5), Number(7), Number(8)),\n\t\tsuite.NewList(), String(\"wow\"), // note that the dupe list chunk is skipped\n\t)\n\tsuite.assertCallbackCount(nested, 25)\n}\n\ntype WalkTestSuite struct {\n\tWalkAllTestSuite\n\tshouldSeeItem Value\n\tshouldSee Value\n\tmustSkip Value\n\tdeadValue Value\n}\n\nfunc (suite *WalkTestSuite) SetupTest() {\n\tstorage := &chunks.TestStorage{}\n\tsuite.ts = storage.NewView()\n\tsuite.vs = NewValueStore(suite.ts)\n\tsuite.shouldSeeItem = String(\"zzz\")\n\tsuite.shouldSee = NewList(suite.vs, suite.shouldSeeItem)\n\tsuite.deadValue = Number(0xDEADBEEF)\n\tsuite.mustSkip = NewList(suite.vs, suite.deadValue)\n}\n" }, { "path": "go/util/clienttest/client_test_suite.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage clienttest\n\nimport (\n\t\"fmt\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/util/exit\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst DefaultMemTableSize = 8 * (1 << 20) // 8MB\n\ntype ClientTestSuite struct {\n\tsuite.Suite\n\tTempDir string\n\tDBDir string\n\tDBDir2 string\n\tExitStatus int\n\tout *os.File\n\terr *os.File\n}\n\ntype ExitError struct {\n\tCode int\n}\n\nfunc (e ExitError) Error() string {\n\treturn fmt.Sprintf(\"main exited with code: %d\", e.Code)\n}\n\nfunc (suite *ClientTestSuite) SetupSuite() {\n\tdir, err := ioutil.TempDir(os.TempDir(), \"nomstest\")\n\td.Chk.NoError(err)\n\tstdOutput, err := ioutil.TempFile(dir, \"out\")\n\td.Chk.NoError(err)\n\terrOutput, err := ioutil.TempFile(dir, \"err\")\n\td.Chk.NoError(err)\n\n\tsuite.TempDir = dir\n\tsuite.DBDir = path.Join(dir, \"db\")\n\tsuite.DBDir2 = path.Join(suite.TempDir, \"db2\")\n\tsuite.out = stdOutput\n\tsuite.err = errOutput\n\texit.Exit = MockExit\n\n\tos.Mkdir(suite.DBDir, 0777)\n\tos.Mkdir(suite.DBDir2, 0777)\n}\n\nfunc (suite *ClientTestSuite) TearDownSuite() {\n\tsuite.out.Close()\n\tsuite.err.Close()\n\tdefer d.Chk.NoError(os.RemoveAll(suite.TempDir))\n}\n\n// MustRun is a wrapper around Run that will panic on Exit or Panic\nfunc (suite *ClientTestSuite) MustRun(m func(), args []string) (stdout string, stderr string) {\n\tvar err interface{}\n\tif stdout, stderr, err = suite.Run(m, args); err != nil {\n\t\tpanic(err)\n\t}\n\treturn\n}\n\n// Run will execute a function passing to it commandline args, and captures stdout,stderr.\n// If m() panics the panic is caught, and returned with recoveredError\n// If m() calls exit.Exit() m() will panic and return ExitError with recoveredError\nfunc (suite *ClientTestSuite) Run(m func(), args []string) (stdout string, stderr string, recoveredErr interface{}) {\n\tfmt.Println(args)\n\torigArgs := os.Args\n\torigOut := os.Stdout\n\torigErr := os.Stderr\n\n\tos.Args = append([]string{\"cmd\"}, args...)\n\tos.Stdout = suite.out\n\tos.Stderr = suite.err\n\n\tdefer func() {\n\t\trecoveredErr = recover()\n\n\t\t// Reset everything right away so that error-checking below goes to terminal.\n\t\tos.Args = origArgs\n\t\tos.Stdout = origOut\n\t\tos.Stderr = origErr\n\n\t\t_, err := suite.out.Seek(0, 0)\n\t\td.Chk.NoError(err)\n\t\tcapturedOut, err := ioutil.ReadAll(suite.out)\n\t\td.Chk.NoError(err)\n\n\t\t_, err = suite.out.Seek(0, 0)\n\t\td.Chk.NoError(err)\n\t\terr = suite.out.Truncate(0)\n\t\td.Chk.NoError(err)\n\n\t\t_, err = suite.err.Seek(0, 0)\n\t\td.Chk.NoError(err)\n\t\tcapturedErr, err := ioutil.ReadAll(suite.err)\n\t\td.Chk.NoError(err)\n\n\t\t_, err = suite.err.Seek(0, 0)\n\t\td.Chk.NoError(err)\n\t\terr = suite.err.Truncate(0)\n\t\td.Chk.NoError(err)\n\t\tstdout, stderr = string(capturedOut), string(capturedErr)\n\t}()\n\n\tsuite.ExitStatus = 0\n\tm()\n\treturn\n}\n\n// Mock exit.Exit() implementation for use during testing.\nfunc MockExit(status int) {\n\tpanic(ExitError{status})\n}\n" }, { "path": "go/util/datetime/date_time.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package datetime implements marshalling of Go DateTime values into Noms structs\n// with type DateTimeType.\npackage datetime\n\nimport (\n\t\"math\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\nconst (\n\tdatetypename = \"DateTime\"\n\thrsEncodingName = \"noms-datetime\"\n)\n\n// DateTime implements marshaling of time.Time to and from Noms.\ntype DateTime struct {\n\ttime.Time\n}\n\n// DateTimeType is the Noms type used to represent date time objects in Noms.\n// The field secSinceEpoch may contain fractions in cases where seconds are\n// not sufficient.\nvar DateTimeType = types.MakeStructTypeFromFields(datetypename, types.FieldMap{\n\t\"secSinceEpoch\": types.NumberType,\n})\n\nvar dateTimeTemplate = types.MakeStructTemplate(datetypename, []string{\"secSinceEpoch\"})\n\n// Epoch is the unix Epoch. This time is very consistent,\n// which makes it useful for testing or checking for uninitialized values\nvar Epoch = DateTime{time.Unix(0, 0)}\n\nfunc init() {\n\tRegisterHRSCommenter(time.Local)\n}\n\n// Now is an alias for a DateTime initialized with time.Now()\nfunc Now() DateTime {\n\treturn DateTime{time.Now()}\n}\n\n// MarshalNoms makes DateTime implement marshal.Marshaler and it makes\n// DateTime marshal into a Noms struct with type DateTimeType.\nfunc (dt DateTime) MarshalNoms(vrw types.ValueReadWriter) (types.Value, error) {\n\treturn dateTimeTemplate.NewStruct([]types.Value{types.Number(float64(dt.Unix()) + float64(dt.Nanosecond())*1e-9)}), nil\n}\n\n// MarshalNomsType makes DateTime implement marshal.TypeMarshaler and it\n// allows marshal.MarshalType to work with DateTime.\nfunc (dt DateTime) MarshalNomsType() (*types.Type, error) {\n\treturn DateTimeType, nil\n}\n\n// UnmarshalNoms makes DateTime implement marshal.Unmarshaler and it allows\n// Noms struct with type DateTimeType able to be unmarshaled onto a DateTime\n// Go struct\nfunc (dt *DateTime) UnmarshalNoms(v types.Value) error {\n\tstrct := struct {\n\t\tSecSinceEpoch float64\n\t}{}\n\terr := marshal.Unmarshal(v, &strct)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\ts, frac := math.Modf(strct.SecSinceEpoch)\n\t*dt = DateTime{time.Unix(int64(s), int64(frac*1e9))}\n\treturn nil\n}\n\ntype DateTimeCommenter struct {\n\ttz *time.Location\n}\n\nfunc (c DateTimeCommenter) Comment(v types.Value) string {\n\tif !types.IsValueSubtypeOf(v, DateTimeType) {\n\t\treturn \"\"\n\t}\n\tvar dt DateTime\n\tmarshal.MustUnmarshal(v, &dt)\n\treturn dt.In(c.tz).Format(time.RFC3339)\n}\n\nfunc RegisterHRSCommenter(tz *time.Location) {\n\thrsCommenter := DateTimeCommenter{tz: tz}\n\ttypes.RegisterHRSCommenter(datetypename, hrsEncodingName, hrsCommenter)\n}\n" }, { "path": "go/util/datetime/date_time_test.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage datetime\n\nimport (\n\t\"strings\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestBasics(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\t// Since we are using float64 in noms we cannot represent all possible times.\n\tdt := DateTime{time.Unix(1234567, 1234567)}\n\n\tnomsValue, err := marshal.Marshal(vs, dt)\n\tassert.NoError(err)\n\n\tvar dt2 DateTime\n\terr = marshal.Unmarshal(nomsValue, &dt2)\n\tassert.NoError(err)\n\n\tassert.True(dt.Equal(dt2.Time))\n}\n\nfunc TestUnmarshal(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttest := func(v types.Struct, t time.Time) {\n\t\tvar dt DateTime\n\t\terr := marshal.Unmarshal(v, &dt)\n\t\tassert.NoError(err)\n\t\tassert.True(dt.Equal(t))\n\t}\n\n\tfor _, name := range []string{\"DateTime\", \"Date\", \"xxx\", \"\"} {\n\t\ttest(types.NewStruct(name, types.StructData{\n\t\t\t\"secSinceEpoch\": types.Number(42),\n\t\t}), time.Unix(42, 0))\n\t}\n\n\ttest(types.NewStruct(\"\", types.StructData{\n\t\t\"secSinceEpoch\": types.Number(42),\n\t\t\"extra\": types.String(\"field\"),\n\t}), time.Unix(42, 0))\n}\n\nfunc TestUnmarshalInvalid(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttest := func(v types.Value) {\n\t\tvar dt DateTime\n\t\terr := marshal.Unmarshal(v, &dt)\n\t\tassert.Error(err)\n\t}\n\n\ttest(types.Number(42))\n\ttest(types.NewStruct(\"DateTime\", types.StructData{}))\n\ttest(types.NewStruct(\"DateTime\", types.StructData{\n\t\t\"secSinceEpoch\": types.String(42),\n\t}))\n\ttest(types.NewStruct(\"DateTime\", types.StructData{\n\t\t\"SecSinceEpoch\": types.Number(42),\n\t}))\n\ttest(types.NewStruct(\"DateTime\", types.StructData{\n\t\t\"msSinceEpoch\": types.Number(42),\n\t}))\n}\n\nfunc TestMarshal(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\ttest := func(dt DateTime, expected float64) {\n\t\tv, err := marshal.Marshal(vs, dt)\n\t\tassert.NoError(err)\n\n\t\tassert.True(types.NewStruct(\"DateTime\", types.StructData{\n\t\t\t\"secSinceEpoch\": types.Number(expected),\n\t\t}).Equals(v))\n\t}\n\n\ttest(DateTime{time.Unix(0, 0)}, 0)\n\ttest(DateTime{time.Unix(42, 0)}, 42)\n\ttest(DateTime{time.Unix(42, 123456789)}, 42.123456789)\n\ttest(DateTime{time.Unix(123456789, 123456789)}, 123456789.123456789)\n\ttest(DateTime{time.Unix(-42, 0)}, -42)\n\ttest(DateTime{time.Unix(-42, -123456789)}, -42.123456789)\n\ttest(DateTime{time.Unix(-123456789, -123456789)}, -123456789.123456789)\n}\n\nfunc TestMarshalType(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tdt := DateTime{time.Unix(0, 0)}\n\ttyp := marshal.MustMarshalType(dt)\n\tassert.Equal(DateTimeType, typ)\n\n\tv := marshal.MustMarshal(vs, dt)\n\tassert.Equal(typ, types.TypeOf(v))\n}\n\nfunc newTestValueStore() *types.ValueStore {\n\tst := &chunks.TestStorage{}\n\treturn types.NewValueStore(st.NewView())\n}\n\nfunc TestZeroValues(t *testing.T) {\n\tassert := assert.New(t)\n\n\tvs := newTestValueStore()\n\tdefer vs.Close()\n\n\tdt1 := DateTime{}\n\tassert.True(dt1.IsZero())\n\n\tnomsDate, _ := dt1.MarshalNoms(vs)\n\n\tdt2 := DateTime{}\n\tmarshal.Unmarshal(nomsDate, &dt2)\n\tassert.True(dt2.IsZero())\n\n\tdt3 := DateTime{}\n\tdt3.UnmarshalNoms(nomsDate)\n\tassert.True(dt3.IsZero())\n}\n\nfunc TestString(t *testing.T) {\n\tassert := assert.New(t)\n\tdt := DateTime{time.Unix(1234567, 1234567)}\n\t// Don't test the actual output since that\n\tassert.IsType(dt.String(), \"s\")\n}\n\nfunc TestEpoch(t *testing.T) {\n\tassert := assert.New(t)\n\tassert.Equal(Epoch, DateTime{time.Unix(0, 0)})\n}\n\nfunc TestHRSComment(t *testing.T) {\n\ta := assert.New(t)\n\tvs := newTestValueStore()\n\n\tdt := Now()\n\tmdt := marshal.MustMarshal(vs, dt)\n\n\texp := dt.Format(time.RFC3339)\n\ts1 := types.EncodedValue(mdt)\n\ta.True(strings.Contains(s1, \"{ // \"+exp))\n\n\tRegisterHRSCommenter(time.UTC)\n\texp = dt.In(time.UTC).Format((time.RFC3339))\n\ts1 = types.EncodedValue(mdt)\n\ta.True(strings.Contains(s1, \"{ // \"+exp))\n\n\ttypes.UnregisterHRSCommenter(datetypename, hrsEncodingName)\n\ts1 = types.EncodedValue(mdt)\n\ta.False(strings.Contains(s1, \"{ // 20\"))\n}\n" }, { "path": "go/util/exit/exit.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package exit provides a mockable implementation of os.Exit.\n// That's all!\npackage exit\n\nimport (\n\t\"os\"\n)\n\nvar def = func(code int) {\n\tos.Exit(code)\n}\n\nvar Exit = def\n\n// Reset sets the implementation of Exit() to the default.\nfunc Reset() {\n\tExit = def\n}\n\n// Fail exits with a failure status.\nfunc Fail() {\n\tExit(1)\n}\n\n// Success exits with a success status.\nfunc Success() {\n\tExit(0)\n}\n" }, { "path": "go/util/functions/all.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage functions\n\nimport \"sync\"\n\n// All runs all functions in |fs| in parallel, and returns when all functions have returned.\nfunc All(fs ...func()) {\n\twg := &sync.WaitGroup{}\n\twg.Add(len(fs))\n\tfor _, f_ := range fs {\n\t\tf := f_\n\t\tgo func() {\n\t\t\tf()\n\t\t\twg.Done()\n\t\t}()\n\t}\n\twg.Wait()\n}\n" }, { "path": "go/util/functions/all_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage functions\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestAll(t *testing.T) {\n\tassert := assert.New(t)\n\n\t// Set |res| via |ch| to test it's running in parallel - if not, they'll deadlock.\n\tvar res int\n\tch := make(chan int)\n\tAll(func() { ch <- 42 }, func() { res = <-ch })\n\n\tassert.Equal(42, res)\n}\n" }, { "path": "go/util/json/from_json.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage json\n\nimport (\n\t\"encoding/json\"\n\t\"io\"\n\t\"reflect\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\nfunc nomsValueFromDecodedJSONBase(vrw types.ValueReadWriter, o interface{}, useStruct bool) types.Value {\n\tswitch o := o.(type) {\n\tcase string:\n\t\treturn types.String(o)\n\tcase bool:\n\t\treturn types.Bool(o)\n\tcase float64:\n\t\treturn types.Number(o)\n\tcase nil:\n\t\treturn nil\n\tcase []interface{}:\n\t\titems := make([]types.Value, 0, len(o))\n\t\tfor _, v := range o {\n\t\t\tnv := nomsValueFromDecodedJSONBase(vrw, v, useStruct)\n\t\t\tif nv != nil {\n\t\t\t\titems = append(items, nv)\n\t\t\t}\n\t\t}\n\t\treturn types.NewList(vrw, items...)\n\tcase map[string]interface{}:\n\t\tvar v types.Value\n\t\tif useStruct {\n\t\t\tstructName := \"\"\n\t\t\tfields := make(types.StructData, len(o))\n\t\t\tfor k, v := range o {\n\t\t\t\tnv := nomsValueFromDecodedJSONBase(vrw, v, useStruct)\n\t\t\t\tif nv != nil {\n\t\t\t\t\tk := types.EscapeStructField(k)\n\t\t\t\t\tfields[k] = nv\n\t\t\t\t}\n\t\t\t}\n\t\t\tv = types.NewStruct(structName, fields)\n\t\t} else {\n\t\t\tkv := make([]types.Value, 0, len(o)*2)\n\t\t\tfor k, v := range o {\n\t\t\t\tnv := nomsValueFromDecodedJSONBase(vrw, v, useStruct)\n\t\t\t\tif nv != nil {\n\t\t\t\t\tkv = append(kv, types.String(k), nv)\n\t\t\t\t}\n\t\t\t}\n\t\t\tv = types.NewMap(vrw, kv...)\n\t\t}\n\t\treturn v\n\n\tdefault:\n\t\td.Chk.Fail(\"Nomsification failed.\", \"I don't understand %+v, which is of type %s!\\n\", o, reflect.TypeOf(o).String())\n\t}\n\treturn nil\n}\n\n// NomsValueFromDecodedJSON takes a generic Go interface{} and recursively\n// tries to resolve the types within so that it can build up and return\n// a Noms Value with the same structure.\n//\n// Currently, the only types supported are the Go versions of legal JSON types:\n// Primitives:\n// - float64\n// - bool\n// - string\n// - nil\n//\n// Composites:\n// - []interface{}\n// - map[string]interface{}\nfunc NomsValueFromDecodedJSON(vrw types.ValueReadWriter, o interface{}, useStruct bool) types.Value {\n\treturn nomsValueFromDecodedJSONBase(vrw, o, useStruct)\n}\n\nfunc FromJSON(r io.Reader, vrw types.ValueReadWriter, opts FromOptions) (types.Value, error) {\n\tdec := json.NewDecoder(r)\n\t// TODO: This is pretty inefficient. It would be better to parse the JSON directly into Noms values,\n\t// rather than going through a pile of Go interfaces.\n\tvar pile interface{}\n\terr := dec.Decode(&pile)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn NomsValueFromDecodedJSON(vrw, pile, opts.Structs), nil\n}\n\n// FromOptions controls how FromJSON works.\ntype FromOptions struct {\n\t// If true, JSON objects are decoded into Noms Structs. Otherwise, they are decoded into Maps.\n\tStructs bool\n}\n" }, { "path": "go/util/json/from_json_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage json\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc TestLibTestSuite(t *testing.T) {\n\tsuite.Run(t, &LibTestSuite{})\n}\n\ntype LibTestSuite struct {\n\tsuite.Suite\n\tvs *types.ValueStore\n}\n\nfunc (suite *LibTestSuite) SetupTest() {\n\tst := &chunks.TestStorage{}\n\tsuite.vs = types.NewValueStore(st.NewView())\n}\n\nfunc (suite *LibTestSuite) TearDownTest() {\n\tsuite.vs.Close()\n}\n\nfunc (suite *LibTestSuite) TestPrimitiveTypes() {\n\tvs := suite.vs\n\tsuite.EqualValues(types.String(\"expected\"), NomsValueFromDecodedJSON(vs, \"expected\", false))\n\tsuite.EqualValues(types.Bool(false), NomsValueFromDecodedJSON(vs, false, false))\n\tsuite.EqualValues(types.Number(1.7), NomsValueFromDecodedJSON(vs, 1.7, false))\n\tsuite.False(NomsValueFromDecodedJSON(vs, 1.7, false).Equals(types.Bool(true)))\n}\n\nfunc (suite *LibTestSuite) TestCompositeTypes() {\n\tvs := suite.vs\n\n\t// [false true]\n\tsuite.EqualValues(\n\t\ttypes.NewList(vs).Edit().Append(types.Bool(false)).Append(types.Bool(true)).List(),\n\t\tNomsValueFromDecodedJSON(vs, []interface{}{false, true}, false))\n\n\t// [[false true]]\n\tsuite.EqualValues(\n\t\ttypes.NewList(vs).Edit().Append(\n\t\t\ttypes.NewList(vs).Edit().Append(types.Bool(false)).Append(types.Bool(true)).List()).List(),\n\t\tNomsValueFromDecodedJSON(vs, []interface{}{[]interface{}{false, true}}, false))\n\n\t// {\"string\": \"string\",\n\t// \"list\": [false true],\n\t// \"map\": {\"nested\": \"string\"}\n\t// }\n\tm := types.NewMap(\n\t\tvs,\n\t\ttypes.String(\"string\"),\n\t\ttypes.String(\"string\"),\n\t\ttypes.String(\"list\"),\n\t\ttypes.NewList(vs).Edit().Append(types.Bool(false)).Append(types.Bool(true)).List(),\n\t\ttypes.String(\"map\"),\n\t\ttypes.NewMap(\n\t\t\tvs,\n\t\t\ttypes.String(\"nested\"),\n\t\t\ttypes.String(\"string\")))\n\to := NomsValueFromDecodedJSON(vs, map[string]interface{}{\n\t\t\"string\": \"string\",\n\t\t\"list\": []interface{}{false, true},\n\t\t\"map\": map[string]interface{}{\"nested\": \"string\"},\n\t}, false)\n\n\tsuite.True(m.Equals(o))\n}\n\nfunc (suite *LibTestSuite) TestCompositeTypeWithStruct() {\n\tvs := suite.vs\n\n\t// {\"string\": \"string\",\n\t// \"list\": [false true],\n\t// \"struct\": {\"nested\": \"string\"}\n\t// }\n\ttstruct := types.NewStruct(\"\", types.StructData{\n\t\t\"string\": types.String(\"string\"),\n\t\t\"list\": types.NewList(vs).Edit().Append(types.Bool(false)).Append(types.Bool(true)).List(),\n\t\t\"struct\": types.NewStruct(\"\", types.StructData{\n\t\t\t\"nested\": types.String(\"string\"),\n\t\t}),\n\t})\n\to := NomsValueFromDecodedJSON(vs, map[string]interface{}{\n\t\t\"string\": \"string\",\n\t\t\"list\": []interface{}{false, true},\n\t\t\"struct\": map[string]interface{}{\"nested\": \"string\"},\n\t}, true)\n\n\tsuite.True(tstruct.Equals(o))\n}\n\nfunc (suite *LibTestSuite) TestPanicOnUnsupportedType() {\n\tvs := suite.vs\n\tsuite.Panics(func() { NomsValueFromDecodedJSON(vs, map[int]string{1: \"one\"}, false) }, \"Should panic on map[int]string!\")\n}\n" }, { "path": "go/util/json/to_json.go", "content": "// Copyright 2019 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage json\n\nimport (\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// ToJSON encodes a Noms value as JSON.\nfunc ToJSON(v types.Value, w io.Writer, opts ToOptions) error {\n\t// TODO: This is a quick hack that is expedient. We should marshal directly to the writer without\n\t// allocating a bunch of Go values.\n\tp, err := toPile(v, opts)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tenc := json.NewEncoder(w)\n\tenc.SetIndent(\"\", opts.Indent)\n\treturn enc.Encode(p)\n}\n\n// ToOptions controls how ToJSON works.\ntype ToOptions struct {\n\t// Enable support for encoding Noms Lists. Lists are encoded as JSON arrays.\n\tLists bool\n\t// Enable support for encoding Noms Maps. Maps are encoded as JSON objects.\n\tMaps bool\n\t// Enable support for encoding Noms Sets. Sets are encoded as JSON arrays.\n\tSets bool\n\t// Enable support for encoding Noms Structs. Structs are encoded as JSON objects.\n\tStructs bool\n\t// String to use for indent when pretty-printing\n\tIndent string\n}\n\nfunc toPile(v types.Value, opts ToOptions) (ret interface{}, err error) {\n\tswitch v := v.(type) {\n\tcase types.Bool:\n\t\treturn bool(v), nil\n\tcase types.Number:\n\t\treturn float64(v), nil\n\tcase types.String:\n\t\treturn string(v), nil\n\tcase types.Struct:\n\t\tif !opts.Structs {\n\t\t\treturn nil, errors.New(\"Struct marshaling not enabled\")\n\t\t}\n\t\tr := map[string]interface{}{}\n\t\tif v.Name() != \"\" {\n\t\t\treturn nil, errors.New(\"Named struct marshaling not supported\")\n\t\t}\n\t\tv.IterFields(func(k string, cv types.Value) (stop bool) {\n\t\t\tvar cp interface{}\n\t\t\tcp, err = toPile(cv, opts)\n\t\t\tif err != nil {\n\t\t\t\treturn true\n\t\t\t}\n\t\t\tr[k] = cp\n\t\t\treturn false\n\t\t})\n\t\treturn r, err\n\tcase types.Map:\n\t\tif !opts.Maps {\n\t\t\treturn nil, errors.New(\"Map marshaling not enabled\")\n\t\t}\n\t\tr := make(map[string]interface{}, v.Len())\n\t\tv.Iter(func(k, cv types.Value) (stop bool) {\n\t\t\tsk, ok := k.(types.String)\n\t\t\tif !ok {\n\t\t\t\terr = fmt.Errorf(\"Map key kind %s not supported\", types.KindToString[k.Kind()])\n\t\t\t\treturn true\n\t\t\t}\n\t\t\tvar cp interface{}\n\t\t\tcp, err = toPile(cv, opts)\n\t\t\tif err != nil {\n\t\t\t\treturn true\n\t\t\t}\n\t\t\tr[string(sk)] = cp\n\t\t\treturn false\n\t\t})\n\t\treturn r, err\n\tcase types.List:\n\t\tif !opts.Lists {\n\t\t\treturn nil, errors.New(\"List marshaling not enabled\")\n\t\t}\n\t\tr := make([]interface{}, v.Len())\n\t\tv.Iter(func(cv types.Value, i uint64) (stop bool) {\n\t\t\tvar cp interface{}\n\t\t\tcp, err = toPile(cv, opts)\n\t\t\tif err != nil {\n\t\t\t\treturn true\n\t\t\t}\n\t\t\tr[i] = cp\n\t\t\treturn false\n\t\t})\n\t\treturn r, err\n\tcase types.Set:\n\t\tif !opts.Sets {\n\t\t\treturn nil, errors.New(\"Set marshaling not enabled\")\n\t\t}\n\t\tr := make([]interface{}, 0, v.Len())\n\t\tv.Iter(func(cv types.Value) (stop bool) {\n\t\t\tvar cp interface{}\n\t\t\tcp, err = toPile(cv, opts)\n\t\t\tif err != nil {\n\t\t\t\treturn true\n\t\t\t}\n\t\t\tr = append(r, cp)\n\t\t\treturn false\n\t\t})\n\t\treturn r, err\n\t}\n\treturn nil, fmt.Errorf(\"Unsupported kind: %s\", types.KindToString[v.Kind()])\n}\n" }, { "path": "go/util/json/to_json_test.go", "content": "// Copyright 2019 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage json\n\nimport (\n\t\"bytes\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nfunc TestToJSONSuite(t *testing.T) {\n\tsuite.Run(t, &ToJSONSuite{})\n}\n\ntype ToJSONSuite struct {\n\tsuite.Suite\n\tvs *types.ValueStore\n}\n\nfunc (suite *ToJSONSuite) SetupTest() {\n\tst := &chunks.TestStorage{}\n\tsuite.vs = types.NewValueStore(st.NewView())\n}\n\nfunc (suite *ToJSONSuite) TearDownTest() {\n\tsuite.vs.Close()\n}\n\nfunc (suite *ToJSONSuite) TestToJSON() {\n\ttc := []struct {\n\t\tdesc string\n\t\tin types.Value\n\t\topts ToOptions\n\t\texp string\n\t\texpError string\n\t}{\n\t\t{\"true\", types.Bool(true), ToOptions{}, \"true\", \"\"},\n\t\t{\"false\", types.Bool(false), ToOptions{}, \"false\", \"\"},\n\t\t{\"42\", types.Number(42), ToOptions{}, \"42\", \"\"},\n\t\t{\"88.8\", types.Number(88.8), ToOptions{}, \"88.8\", \"\"},\n\t\t{\"empty string\", types.String(\"\"), ToOptions{}, `\"\"`, \"\"},\n\t\t{\"foobar\", types.String(\"foobar\"), ToOptions{}, `\"foobar\"`, \"\"},\n\t\t{\"strings with newlines\", types.String(`\"\\nmonkey`), ToOptions{}, `\"\\\"\\\\nmonkey\"`, \"\"},\n\t\t{\"structs when not enabled\", types.NewStruct(\"\", types.StructData{}), ToOptions{}, \"\", \"Struct marshaling not enabled\"},\n\t\t{\"named struct\", types.NewStruct(\"Person\", types.StructData{}), ToOptions{Structs: true}, \"\", \"Named struct marshaling not supported\"},\n\t\t{\"struct nested errors\", types.NewStruct(\"\", types.StructData{\"foo\": types.NewList(suite.vs)}), ToOptions{Structs: true}, \"\", \"List marshaling not enabled\"},\n\t\t{\"empty struct\", types.NewStruct(\"\", types.StructData{}), ToOptions{Structs: true}, \"{}\", \"\"},\n\t\t{\"non-empty struct\", types.NewStruct(\"\", types.StructData{\"str\": types.String(\"bar\"), \"num\": types.Number(42)}), ToOptions{Structs: true}, `{\"num\":42,\"str\":\"bar\"}`, \"\"},\n\t\t{\"list when not enabled\", types.NewList(suite.vs), ToOptions{}, \"\", \"List marshaling not enabled\"},\n\t\t{\"list nested errors\", types.NewList(suite.vs, types.NewSet(suite.vs)), ToOptions{Lists: true}, \"\", \"Set marshaling not enabled\"},\n\t\t{\"empty list\", types.NewList(suite.vs), ToOptions{Lists: true}, \"[]\", \"\"},\n\t\t{\"non-empty list\", types.NewList(suite.vs, types.Number(42), types.String(\"foo\")), ToOptions{Lists: true}, `[42,\"foo\"]`, \"\"},\n\t\t{\"sets when not enabled\", types.NewSet(suite.vs), ToOptions{}, \"\", \"Set marshaling not enabled\"},\n\t\t{\"set nested errors\", types.NewSet(suite.vs, types.NewList(suite.vs)), ToOptions{Sets: true}, \"\", \"List marshaling not enabled\"},\n\t\t{\"empty set\", types.NewSet(suite.vs), ToOptions{Sets: true}, \"[]\", \"\"},\n\t\t{\"non-empty set\", types.NewSet(suite.vs, types.Number(42), types.String(\"foo\")), ToOptions{Sets: true}, `[42,\"foo\"]`, \"\"},\n\t\t{\"maps when not enabled\", types.NewMap(suite.vs), ToOptions{}, \"\", \"Map marshaling not enabled\"},\n\t\t{\"map nested errors\", types.NewMap(suite.vs, types.String(\"foo\"), types.NewSet(suite.vs)), ToOptions{Maps: true}, \"\", \"Set marshaling not enabled\"},\n\t\t{\"map non-string key\", types.NewMap(suite.vs, types.Number(42), types.Number(42)), ToOptions{Maps: true}, \"\", \"Map key kind Number not supported\"},\n\t\t{\"empty map\", types.NewMap(suite.vs), ToOptions{Maps: true}, \"{}\", \"\"},\n\t\t{\"non-empty map\", types.NewMap(suite.vs, types.String(\"foo\"), types.String(\"bar\"), types.String(\"baz\"), types.Number(42)), ToOptions{Maps: true}, `{\"baz\":42,\"foo\":\"bar\"}`, \"\"},\n\t\t{\"complex value\", types.NewStruct(\"\", types.StructData{\n\t\t\t\"list\": types.NewList(suite.vs,\n\t\t\t\ttypes.NewSet(suite.vs,\n\t\t\t\t\ttypes.NewMap(suite.vs, types.String(\"foo\"), types.String(\"bar\"), types.String(\"hot\"), types.Number(42))))}), ToOptions{Structs: true, Lists: true, Sets: true, Maps: true}, `{\"list\":[[{\"foo\":\"bar\",\"hot\":42}]]}`, \"\"},\n\t}\n\n\tfor _, t := range tc {\n\t\tbuf := &bytes.Buffer{}\n\t\terr := ToJSON(t.in, buf, t.opts)\n\t\tif t.expError != \"\" {\n\t\t\tsuite.EqualError(err, t.expError, t.desc)\n\t\t\tsuite.Equal(\"\", string(buf.Bytes()), t.desc)\n\t\t} else {\n\t\t\tsuite.NoError(err)\n\t\t\tsuite.Equal(t.exp+\"\\n\", string(buf.Bytes()), t.desc)\n\t\t}\n\t}\n}\n" }, { "path": "go/util/math/minmax.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage math\n\n// MaxInt returns the larger of x or y.\nfunc MaxInt(x, y int) int {\n\tif x > y {\n\t\treturn x\n\t}\n\treturn y\n}\n\n// MinInt returns the smaller of x or y.\nfunc MinInt(x, y int) int {\n\tif x < y {\n\t\treturn x\n\t}\n\treturn y\n}\n" }, { "path": "go/util/outputpager/page_output.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage outputpager\n\nimport (\n\t\"io\"\n\t\"os\"\n\t\"os/exec\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/kingpin\"\n\tgoisatty \"github.com/mattn/go-isatty\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nvar (\n\tnoPager bool\n)\n\ntype Pager struct {\n\tWriter io.Writer\n\tstdin, stdout *os.File\n\tmtx *sync.Mutex\n\tdoneCh chan struct{}\n}\n\nfunc Start() *Pager {\n\tif noPager || !IsStdoutTty() {\n\t\treturn &Pager{os.Stdout, nil, nil, nil, nil}\n\t}\n\n\tlessPath, err := exec.LookPath(\"less\")\n\td.Chk.NoError(err)\n\n\t// -F ... Quit if entire file fits on first screen.\n\t// -S ... Chop (truncate) long lines rather than wrapping.\n\t// -R ... Output \"raw\" control characters.\n\t// -X ... Don't use termcap init/deinit strings.\n\tcmd := exec.Command(lessPath, \"-FSRX\")\n\n\tstdin, stdout, err := os.Pipe()\n\td.Chk.NoError(err)\n\tcmd.Stdout = os.Stdout\n\tcmd.Stderr = os.Stderr\n\tcmd.Stdin = stdin\n\tcmd.Start()\n\n\tp := &Pager{stdout, stdin, stdout, &sync.Mutex{}, make(chan struct{})}\n\tgo func() {\n\t\terr := cmd.Wait()\n\t\td.Chk.NoError(err)\n\t\tp.closePipe()\n\t\tp.doneCh <- struct{}{}\n\t}()\n\treturn p\n}\n\nfunc (p *Pager) Stop() {\n\tif p.Writer != os.Stdout {\n\t\tp.closePipe()\n\t\t// Wait until less has fully exited, otherwise it might not have printed the terminal restore characters.\n\t\t<-p.doneCh\n\t}\n}\n\nfunc (p *Pager) closePipe() {\n\tp.mtx.Lock()\n\tdefer p.mtx.Unlock()\n\tif p.stdin != nil {\n\t\t// Closing the pipe will cause any outstanding writes to stdout fail, and fail from now on.\n\t\tp.stdin.Close()\n\t\tp.stdout.Close()\n\t\tp.stdin, p.stdout = nil, nil\n\t}\n}\n\nfunc RegisterOutputpagerFlags(cmd *kingpin.CmdClause) {\n\tcmd.Flag(\"no-pager\", \"suppress paging functionality\").BoolVar(&noPager)\n}\n\nfunc IsStdoutTty() bool {\n\treturn goisatty.IsTerminal(os.Stdout.Fd())\n}\n" }, { "path": "go/util/profile/profile.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage profile\n\nimport (\n\t\"io\"\n\t\"os\"\n\t\"runtime\"\n\t\"runtime/pprof\"\n\n\t\"github.com/attic-labs/kingpin\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nvar (\n\tcpuProfile string\n\tmemProfile string\n\tblockProfile string\n)\n\nfunc RegisterProfileFlags(app *kingpin.Application) {\n\t// Must reset globals because under test this can get called multiple times.\n\tcpuProfile = \"\"\n\tmemProfile = \"\"\n\tblockProfile = \"\"\n\tapp.Flag(\"cpuprofile\", \"write cpu profile to file\").StringVar(&cpuProfile)\n\tapp.Flag(\"memprofile\", \"write memory profile to file\").StringVar(&memProfile)\n\tapp.Flag(\"blockprofile\", \"write block profile to file\").StringVar(&blockProfile)\n}\n\n// MaybeStartProfile checks the -blockProfile, -cpuProfile, and -memProfile flag and, for each that is set, attempts to start gathering profiling data into the appropriate files. It returns an object with one method, Stop(), that must be called in order to flush profile data to disk before the process terminates.\nfunc MaybeStartProfile() interface {\n\tStop()\n} {\n\tp := &prof{}\n\tif blockProfile != \"\" {\n\t\tf, err := os.Create(blockProfile)\n\t\td.PanicIfError(err)\n\t\truntime.SetBlockProfileRate(1)\n\t\tp.bp = f\n\t}\n\tif cpuProfile != \"\" {\n\t\tf, err := os.Create(cpuProfile)\n\t\td.PanicIfError(err)\n\t\tpprof.StartCPUProfile(f)\n\t\tp.cpu = f\n\t}\n\tif memProfile != \"\" {\n\t\tf, err := os.Create(memProfile)\n\t\td.PanicIfError(err)\n\t\tp.mem = f\n\t}\n\treturn p\n}\n\ntype prof struct {\n\tbp io.WriteCloser\n\tcpu io.Closer\n\tmem io.WriteCloser\n}\n\nfunc (p *prof) Stop() {\n\tif p.bp != nil {\n\t\tpprof.Lookup(\"block\").WriteTo(p.bp, 0)\n\t\tp.bp.Close()\n\t\truntime.SetBlockProfileRate(0)\n\t}\n\tif p.cpu != nil {\n\t\tpprof.StopCPUProfile()\n\t\tp.cpu.Close()\n\t}\n\tif p.mem != nil {\n\t\tpprof.WriteHeapProfile(p.mem)\n\t\tp.mem.Close()\n\t}\n}\n" }, { "path": "go/util/progressreader/reader.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package progressreader provides an io.Reader that reports progress to a callback\npackage progressreader\n\nimport (\n\t\"io\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/util/status\"\n)\n\ntype Callback func(seen uint64)\n\nfunc New(inner io.Reader, cb Callback) io.Reader {\n\treturn &reader{inner, uint64(0), time.Time{}, cb}\n}\n\ntype reader struct {\n\tinner io.Reader\n\tseen uint64\n\tlastTime time.Time\n\tcb Callback\n}\n\nfunc (r *reader) Read(p []byte) (n int, err error) {\n\tn, err = r.inner.Read(p)\n\tr.seen += uint64(n)\n\n\tif now := time.Now(); now.Sub(r.lastTime) >= status.Rate || err == io.EOF {\n\t\tr.cb(r.seen)\n\t\tr.lastTime = now\n\t}\n\treturn\n}\n" }, { "path": "go/util/random/id.go", "content": "package random\n\nimport (\n\t\"crypto/rand\"\n\t\"encoding/hex\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\nvar (\n\treader = rand.Reader\n)\n\n// Id creates a unique ID which is a random 16 byte hex string\nfunc Id() string {\n\tdata := make([]byte, 16)\n\t_, err := reader.Read(data)\n\td.Chk.NoError(err)\n\treturn hex.EncodeToString(data)\n}\n" }, { "path": "go/util/random/id_test.go", "content": "package random\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype testReader byte\n\nfunc (r *testReader) Read(dest []byte) (int, error) {\n\tfor i := 0; i < len(dest); i++ {\n\t\tdest[i] = byte(*r)\n\t}\n\treturn len(dest), nil\n}\n\nfunc TestBasic(t *testing.T) {\n\tassert := assert.New(t)\n\n\tfunc() {\n\t\tvar r testReader\n\t\toldReader := reader\n\t\treader = &r\n\t\tdefer func() {\n\t\t\treader = oldReader\n\t\t}()\n\n\t\tr = testReader(byte(0x00))\n\t\tassert.Equal(\"00000000000000000000000000000000\", Id())\n\t\tr = testReader(byte(0x01))\n\t\tassert.Equal(\"01010101010101010101010101010101\", Id())\n\t\tr = testReader(byte(0xFF))\n\t\tassert.Equal(\"ffffffffffffffffffffffffffffffff\", Id())\n\t}()\n\n\tone := Id()\n\ttwo := Id()\n\tassert.NotEqual(one, two)\n}\n" }, { "path": "go/util/sizecache/size_cache.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage sizecache\n\n// SizeCache implements a simple LRU cache of interface{}-typed key-value pairs.\n// When items are added, the \"size\" of the item must be provided. LRU items will\n// be expired until the total of all items is below the specified size for the\n// SizeCache\nimport (\n\t\"container/list\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\ntype sizeCacheEntry struct {\n\tsize uint64\n\tlruEntry *list.Element\n\tvalue interface{}\n}\n\ntype SizeCache struct {\n\ttotalSize uint64\n\tmaxSize uint64\n\tmu sync.Mutex\n\tlru list.List\n\tcache map[interface{}]sizeCacheEntry\n\texpireCb func(elm interface{})\n}\n\ntype ExpireCallback func(key interface{})\n\n// New creates a SizeCache that will hold up to |maxSize| item data.\nfunc New(maxSize uint64) *SizeCache {\n\treturn NewWithExpireCallback(maxSize, nil)\n}\n\n// NewWithExpireCallback creates a SizeCache that will hold up to |maxSize|\n// item data, and will call cb(key) when the item corresponding with that key\n// expires.\nfunc NewWithExpireCallback(maxSize uint64, cb ExpireCallback) *SizeCache {\n\treturn &SizeCache{\n\t\tmaxSize: maxSize,\n\t\tcache: map[interface{}]sizeCacheEntry{},\n\t\texpireCb: cb,\n\t}\n}\n\n// entry() checks if the value is in the cache. If not in the cache, it returns an\n// empty sizeCacheEntry and false. It it is in the cache, it moves it to\n// to the back of lru and returns the entry and true.\n// Callers should have locked down the |c| with a call to c.mu.Lock() before\n// calling this entry().\nfunc (c *SizeCache) entry(key interface{}) (sizeCacheEntry, bool) {\n\tentry, ok := c.cache[key]\n\tif !ok {\n\t\treturn sizeCacheEntry{}, false\n\t}\n\tc.lru.MoveToBack(entry.lruEntry)\n\treturn entry, true\n}\n\n// Get checks the searches the cache for an entry. If it exists, it moves it's\n// lru entry to the back of the queue and returns (value, true). Otherwise, it\n// returns (nil, false).\nfunc (c *SizeCache) Get(key interface{}) (interface{}, bool) {\n\tc.mu.Lock()\n\tdefer c.mu.Unlock()\n\n\tif entry, ok := c.entry(key); ok {\n\t\treturn entry.value, true\n\t}\n\treturn nil, false\n}\n\n// Add will add this element to the cache at the back of the queue as long it's\n// size does not exceed maxSize. If the addition of this entry causes the size of\n// the cache to exceed maxSize, the necessary entries at the front of the queue\n// will be deleted in order to keep the total cache size below maxSize.\nfunc (c *SizeCache) Add(key interface{}, size uint64, value interface{}) {\n\tif size <= c.maxSize {\n\t\tc.mu.Lock()\n\t\tdefer c.mu.Unlock()\n\n\t\tif _, ok := c.entry(key); ok {\n\t\t\t// this value is already in the cache; just return\n\t\t\treturn\n\t\t}\n\n\t\tnewEl := c.lru.PushBack(key)\n\t\tce := sizeCacheEntry{size: size, lruEntry: newEl, value: value}\n\t\tc.cache[key] = ce\n\t\tc.totalSize += ce.size\n\t\tfor el := c.lru.Front(); el != nil && c.totalSize > c.maxSize; {\n\t\t\tkey1 := el.Value\n\t\t\tce, ok := c.cache[key1]\n\t\t\tif !ok {\n\t\t\t\td.Panic(\"SizeCache is missing expected value\")\n\t\t\t}\n\t\t\tnext := el.Next()\n\t\t\tdelete(c.cache, key1)\n\t\t\tc.totalSize -= ce.size\n\t\t\tc.lru.Remove(el)\n\t\t\tif c.expireCb != nil {\n\t\t\t\tc.expireCb(key1)\n\t\t\t}\n\t\t\tel = next\n\t\t}\n\t}\n}\n\n// Drop will remove the element associated with the given key from the cache.\nfunc (c *SizeCache) Drop(key interface{}) {\n\tc.mu.Lock()\n\tdefer c.mu.Unlock()\n\n\tif entry, ok := c.entry(key); ok {\n\t\tc.totalSize -= entry.size\n\t\tc.lru.Remove(entry.lruEntry)\n\t\tdelete(c.cache, key)\n\t}\n}\n" }, { "path": "go/util/sizecache/size_cache_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage sizecache\n\nimport (\n\t\"fmt\"\n\t\"sort\"\n\t\"sync\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc hashFromString(s string) hash.Hash {\n\treturn hash.Of([]byte(s))\n}\n\nfunc TestSizeCache(t *testing.T) {\n\tassert := assert.New(t)\n\tdefSize := uint64(200)\n\n\tc := New(1024)\n\tfor i, v := range []string{\"data-1\", \"data-2\", \"data-3\", \"data-4\", \"data-5\", \"data-6\", \"data-7\", \"data-8\", \"data-9\"} {\n\t\tc.Add(hashFromString(v), defSize, v)\n\t\tmaxElements := uint64(i + 1)\n\t\tif maxElements >= uint64(5) {\n\t\t\tmaxElements = uint64(5)\n\t\t}\n\t\tassert.Equal(maxElements*defSize, c.totalSize)\n\t}\n\n\t_, ok := c.Get(hashFromString(\"data-1\"))\n\tassert.False(ok)\n\tassert.Equal(hashFromString(\"data-5\"), c.lru.Front().Value)\n\n\tv, ok := c.Get(hashFromString(\"data-5\"))\n\tassert.True(ok)\n\tassert.Equal(\"data-5\", v.(string))\n\tassert.Equal(hashFromString(\"data-5\"), c.lru.Back().Value)\n\tassert.Equal(hashFromString(\"data-6\"), c.lru.Front().Value)\n\n\tc.Add(hashFromString(\"data-7\"), defSize, \"data-7\")\n\tassert.Equal(hashFromString(\"data-7\"), c.lru.Back().Value)\n\tassert.Equal(uint64(1000), c.totalSize)\n\n\tc.Add(hashFromString(\"no-data\"), 0, nil)\n\tv, ok = c.Get(hashFromString(\"no-data\"))\n\tassert.True(ok)\n\tassert.Nil(v)\n\tassert.Equal(hashFromString(\"no-data\"), c.lru.Back().Value)\n\tassert.Equal(uint64(1000), c.totalSize)\n\tassert.Equal(6, c.lru.Len())\n\tassert.Equal(6, len(c.cache))\n\n\tfor _, v := range []string{\"data-5\", \"data-6\", \"data-7\", \"data-8\", \"data-9\"} {\n\t\tc.Get(hashFromString(v))\n\t\tassert.Equal(hashFromString(v), c.lru.Back().Value)\n\t}\n\tassert.Equal(hashFromString(\"no-data\"), c.lru.Front().Value)\n\n\tc.Add(hashFromString(\"data-10\"), 200, \"data-10\")\n\tassert.Equal(uint64(1000), c.totalSize)\n\tassert.Equal(5, c.lru.Len())\n\tassert.Equal(5, len(c.cache))\n\n\t_, ok = c.Get(hashFromString(\"no-data\"))\n\tassert.False(ok)\n\t_, ok = c.Get(hashFromString(\"data-5\"))\n\tassert.False(ok)\n\n\tc.Drop(hashFromString(\"data-10\"))\n\tassert.Equal(uint64(800), c.totalSize)\n\tassert.Equal(4, c.lru.Len())\n\tassert.Equal(4, len(c.cache))\n}\n\nfunc TestSizeCacheWithExpiry(t *testing.T) {\n\texpired := []string{}\n\texpire := func(key interface{}) {\n\t\texpired = append(expired, key.(string))\n\t}\n\n\tc := NewWithExpireCallback(5, expire)\n\tdata := []string{\"a\", \"b\", \"c\", \"d\", \"e\"}\n\tfor i, k := range data {\n\t\tc.Add(k, 1, i)\n\t}\n\n\tc.Add(\"big\", 5, \"thing\")\n\tsort.Sort(sort.StringSlice(expired))\n\tassert.Equal(t, data, expired)\n}\n\nfunc concurrencySizeCacheTest(data []string) {\n\tdchan := make(chan string, 128)\n\tgo func() {\n\t\tfor _, d := range data {\n\t\t\tdchan <- d\n\t\t}\n\t\tclose(dchan)\n\t}()\n\n\tcache := New(25)\n\twg := sync.WaitGroup{}\n\n\tfor i := 0; i < 3; i++ {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tfor d := range dchan {\n\t\t\t\tcache.Add(d, uint64(len(d)), d)\n\t\t\t}\n\t\t\twg.Done()\n\t\t}()\n\t}\n\twg.Wait()\n}\n\n// I can't guarantee this will fail if the code isn't correct, but in the\n// previous version of SizeCache, this was able to reliably repro bug #2663.\nfunc TestConcurrency(t *testing.T) {\n\tassert := assert.New(t)\n\tgenerateDataStrings := func(numStrings, numValues int) []string {\n\t\tl := []string{}\n\t\tfor i := 0; len(l) < numStrings; i++ {\n\t\t\tfor j := 0; j < numValues && len(l) < numStrings; j++ {\n\t\t\t\tl = append(l, fmt.Sprintf(\"data-%d\", i))\n\t\t\t}\n\t\t}\n\t\treturn l\n\t}\n\n\tdata := generateDataStrings(50, 3)\n\tfor i := 0; i < 100; i++ {\n\t\tassert.NotPanics(func() { concurrencySizeCacheTest(data) })\n\t}\n}\n\nfunc TestTooLargeValue(t *testing.T) {\n\tassert := assert.New(t)\n\n\tc := New(1024)\n\tc.Add(hashFromString(\"big-data\"), 2048, \"big-data\")\n\t_, ok := c.Get(hashFromString(\"big-data\"))\n\tassert.False(ok)\n}\n\nfunc TestZeroSizeCache(t *testing.T) {\n\tassert := assert.New(t)\n\n\tc := New(0)\n\tc.Add(hashFromString(\"data1\"), 200, \"data1\")\n\t_, ok := c.Get(hashFromString(\"data1\"))\n\tassert.False(ok)\n}\n" }, { "path": "go/util/status/status.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\n// Package status prints status messages to a console, overwriting previous values.\npackage status\n\nimport (\n\t\"fmt\"\n\t\"time\"\n)\n\nconst (\n\tclearLine = \"\\x1b[2K\\r\"\n\tRate = 100 * time.Millisecond\n)\n\nvar (\n\tlastTime time.Time\n\tlastFormat string\n\tlastArgs []interface{}\n)\n\nfunc Clear() {\n\tfmt.Print(clearLine)\n\treset(time.Time{})\n}\n\nfunc WillPrint() bool {\n\treturn time.Now().Sub(lastTime) >= Rate\n}\n\nfunc Printf(format string, args ...interface{}) {\n\tnow := time.Now()\n\tif now.Sub(lastTime) < Rate {\n\t\tlastFormat, lastArgs = format, args\n\t} else {\n\t\tfmt.Printf(clearLine+format, args...)\n\t\treset(now)\n\t}\n}\n\nfunc Done() {\n\tif lastArgs != nil {\n\t\tfmt.Printf(clearLine+lastFormat, lastArgs...)\n\t}\n\tfmt.Println()\n\treset(time.Time{})\n}\n\nfunc reset(time time.Time) {\n\tlastTime = time\n\tlastFormat, lastArgs = \"\", nil\n}\n" }, { "path": "go/util/test/equals_ignore_hashes.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage test\n\nimport (\n\t\"regexp\"\n\t\"strconv\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nvar pattern = regexp.MustCompile(\"([0-9a-v]{\" + strconv.Itoa(hash.StringLen) + \"})\")\n\n// EqualsIgnoreHashes compares two strings, ignoring hashes in them.\nfunc EqualsIgnoreHashes(tt *testing.T, expected, actual string) {\n\tif RemoveHashes(expected) != RemoveHashes(actual) {\n\t\tassert.Equal(tt, expected, actual)\n\t}\n}\n\nfunc RemoveHashes(str string) string {\n\treturn pattern.ReplaceAllString(str, strings.Repeat(\"*\", hash.StringLen))\n}\n" }, { "path": "go/util/verbose/verbose.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage verbose\n\nimport (\n\t\"log\"\n\n\t\"github.com/attic-labs/kingpin\"\n)\n\nvar (\n\tverbose bool\n\tquiet bool\n)\n\n// RegisterVerboseFlags registers -v|--verbose flags for general usage\nfunc RegisterVerboseFlags(app *kingpin.Application) {\n\t// Must reset globals because under test this can get called multiple times.\n\tverbose = false\n\tquiet = false\n\tapp.Flag(\"verbose\", \"show more\").Short('v').BoolVar(&verbose)\n\tapp.Flag(\"quite\", \"show less\").Short('q').BoolVar(&quiet)\n}\n\n// Verbose returns True if the verbose flag was set\nfunc Verbose() bool {\n\treturn verbose\n}\n\nfunc SetVerbose(v bool) {\n\tverbose = v\n}\n\n// Quiet returns True if the verbose flag was set\nfunc Quiet() bool {\n\treturn quiet\n}\n\nfunc SetQuiet(q bool) {\n\tquiet = q\n}\n\n// Log calls Printf(format, args...) iff Verbose() returns true.\nfunc Log(format string, args ...interface{}) {\n\tif Verbose() {\n\t\tif len(args) > 0 {\n\t\t\tlog.Printf(format+\"\\n\", args...)\n\t\t} else {\n\t\t\tlog.Println(format)\n\t\t}\n\t}\n}\n" }, { "path": "go/util/writers/max_line_writer.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage writers\n\nimport \"io\"\n\nvar (\n\t// MaxLinesErr is an instance of MaxLinesError that gets returned by\n\t// Write() whenever the number of lines written has exceeded the number\n\t// in |MaxLineWriter.MaxLines|.\n\tMaxLinesErr = MaxLinesError{\"Maximum number of lines written\"}\n)\n\n// MaxLinesError is the type of error returned by Write() whenever the number\n// of lines written has exceeded the number in |MaxLineWriter.MaxLines|.\ntype MaxLinesError struct {\n\tmsg string\n}\n\nfunc (e MaxLinesError) Error() string { return e.msg }\n\n// MaxLineWriter provides an io.Writer interface that counts the number of lines\n// that have been written. It will stop writing and returns an error if the\n// number of lines written exceeds the number specified in MaxLineWriter.NumLines.\ntype MaxLineWriter struct {\n\tDest io.Writer\n\tMaxLines uint32\n\tNumLines uint32\n}\n\n// Write() stops writing and returns an error if an attempt is made to write\n// any byte after |MaxLines| newLines have been written. For example, if MaxLines\n// is 1, all bytes will be written up to and including the 1st newline. If there\n// are any bytes in |data| after the 1st newline, an error will be returned.\n//\n// Callers can change the value of |w.MaxLines| before any call to Write().\n// Setting MaxLines to 0 will allow any number of newLines.\nfunc (w *MaxLineWriter) Write(data []byte) (int, error) {\n\tif len(data) == 0 {\n\t\treturn 0, nil\n\t}\n\n\tcheckMax := w.MaxLines > 0\n\n\tif checkMax && w.NumLines >= w.MaxLines {\n\t\treturn 0, MaxLinesErr\n\t}\n\n\tvar err error\n\tbyteCnt := 0\n\n\tfor i, b := range data {\n\t\tif b == byte('\\n') {\n\t\t\tw.NumLines++\n\t\t\tif checkMax && w.NumLines > w.MaxLines {\n\t\t\t\terr = MaxLinesErr\n\t\t\t\tbreak\n\t\t\t}\n\t\t} else if checkMax && w.NumLines >= w.MaxLines {\n\t\t\terr = MaxLinesErr\n\t\t\tbreak\n\t\t}\n\t\tbyteCnt = i\n\t}\n\n\tcnt, err1 := w.Dest.Write(data[:byteCnt+1])\n\tif err1 != nil {\n\t\treturn cnt, err1\n\t}\n\treturn cnt, err\n}\n" }, { "path": "go/util/writers/prefix_writer.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage writers\n\nimport \"io\"\n\n// PrefixWriter makes it easy to prefix lines with a custom prefix. Each time\n// it writes a byte after a newline('\\n') character it calls PrefixFunc() to get\n// the byte slice that should be written. |NeedsPrefix| can be set to true to\n// cause a prefix to be written immediately. This is useful for causing a prefix\n// to get written on the first line.\ntype PrefixWriter struct {\n\tDest io.Writer\n\tPrefixFunc func(w *PrefixWriter) []byte\n\tNeedsPrefix bool\n\tNumLines uint32\n}\n\n// Write() will add a prefix to the beginning of each line. It obtains the\n// prefix by call |PrefixFunc(w *PrefixWriter)| before printing out any character\n// following a newLine. Callers can force a prefix to be printed out before the\n// first character in |data| by setting NeedsPrefix to true. Conversely, callers\n// can suppress prefixes from being printed by setting NeedsPrefix to false.\n\nfunc (w *PrefixWriter) Write(data []byte) (int, error) {\n\twrittenCnt := 0\n\tfor i, b := range data {\n\t\tif w.NeedsPrefix {\n\t\t\tw.NeedsPrefix = false\n\t\t\td1 := w.PrefixFunc(w)\n\t\t\tcnt, err := w.Dest.Write(d1)\n\t\t\twrittenCnt += cnt\n\t\t\tif err != nil {\n\t\t\t\treturn writtenCnt, err\n\t\t\t}\n\t\t}\n\t\tif b == byte('\\n') {\n\t\t\tw.NumLines++\n\t\t\tw.NeedsPrefix = true\n\t\t}\n\t\tcnt, err := w.Dest.Write(data[i : i+1])\n\t\twrittenCnt += cnt\n\t\tif err != nil {\n\t\t\treturn writtenCnt, err\n\t\t}\n\t}\n\treturn writtenCnt, nil\n}\n" }, { "path": "go/util/writers/writers_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage writers\n\nimport (\n\t\"bytes\"\n\t\"io\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype maxLineTestCase struct {\n\tdata string\n\tmaxLines uint32\n\texpected string\n\terrorExpected bool\n}\n\nfunc TestMaxLineWriter(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttcs := []maxLineTestCase{\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", 1, \"hey there\\n\", true},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", 2, \"hey there\\nthis text contains\\n\", true},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", 3, \"hey there\\nthis text contains\\n3 lines\\n\", false},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\nand more\\n\", 3, \"hey there\\nthis text contains\\n3 lines\\n\", true},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", 4, \"hey there\\nthis text contains\\n3 lines\\n\", false},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", 0, \"hey there\\nthis text contains\\n3 lines\\n\", false},\n\t\t{\"\\n\\n\\n\\n\", 2, \"\\n\\n\", true},\n\t}\n\n\tfor i, tc := range tcs {\n\t\tbuf := bytes.NewBuffer(nil)\n\t\tmlw := MaxLineWriter{Dest: buf, MaxLines: tc.maxLines}\n\t\tl, err := mlw.Write([]byte(tc.data))\n\t\tassert.Equal(len(tc.expected), l, \"test #%d case failed\", i)\n\t\tif tc.errorExpected {\n\t\t\tassert.Error(err, \"test #%d case failed\", i)\n\t\t\tassert.IsType(MaxLinesError{}, err, \"test #%d case failed\", i)\n\t\t} else {\n\t\t\tassert.NoError(err, \"test #%d case failed\", i)\n\t\t}\n\t\tassert.Equal(tc.expected, buf.String(), \"test #%d case failed\", i)\n\t}\n}\n\ntype prefixTestCase struct {\n\tdata string\n\tprefix string\n\texpected string\n\tneedsPrefix bool\n}\n\nfunc TestPrefixWriter(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttcs := []prefixTestCase{\n\t\t{\"\\n\", \"yo:\", \"yo:\\n\", true},\n\t\t{\"\\n\", \"yo:\", \"\\n\", false},\n\t\t{\"\\n\\n\", \"yo:\", \"yo:\\nyo:\\n\", true},\n\t\t{\"\\n\\n\", \"yo:\", \"\\nyo:\\n\", false},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"yo:\", \"yo:hey there\\nyo:this text contains\\nyo:3 lines\\n\", true},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"yo:\", \"hey there\\nyo:this text contains\\nyo:3 lines\\n\", false},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"\", \"hey there\\nthis text contains\\n3 lines\\n\", true},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"\", \"hey there\\nthis text contains\\n3 lines\\n\", false},\n\t}\n\n\tfor _, tc := range tcs {\n\t\tgetPrefix := func(w *PrefixWriter) []byte {\n\t\t\treturn []byte(tc.prefix)\n\t\t}\n\t\tbuf := bytes.NewBuffer(nil)\n\t\tpw := PrefixWriter{Dest: buf, PrefixFunc: getPrefix, NeedsPrefix: tc.needsPrefix}\n\t\tl, err := pw.Write([]byte(tc.data))\n\t\tassert.NoError(err)\n\t\tassert.Equal(len(tc.expected), l)\n\t\tassert.Equal(tc.expected, buf.String())\n\t}\n}\n\ntype prefixMaxLineTestCase struct {\n\tdata string\n\tprefix string\n\texpected string\n\tneedsPrefix bool\n\tmaxLines uint32\n\terrorExpected bool\n}\n\nfunc TestPrefixMaxLineWriter(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttcs := []prefixMaxLineTestCase{\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"yo:\", \"yo:hey there\\nyo:this text contains\\nyo:3 lines\\n\", true, 0, false},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"yo:\", \"yo:hey there\\n\", true, 1, true},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"yo:\", \"hey there\\nyo:this text contains\\nyo:3 lines\\n\", false, 0, false},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"yo:\", \"hey there\\nyo:this text contains\\n\", false, 2, true},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"\", \"hey there\\nthis text contains\\n3 lines\\n\", true, 0, false},\n\t\t{\"hey there\\nthis text contains\\n3 lines\\n\", \"\", \"hey there\\nthis text contains\\n\", false, 2, true},\n\t}\n\n\tdoTest := func(tc prefixMaxLineTestCase, tcNum int, buf *bytes.Buffer, tw io.Writer) {\n\t\tl, err := tw.Write([]byte(tc.data))\n\t\tif tc.errorExpected {\n\t\t\tassert.Error(err, \"test #%d case failed\", tcNum)\n\t\t\tassert.IsType(MaxLinesError{}, err, \"test #%d case failed\", tcNum)\n\t\t} else {\n\t\t\tassert.NoError(err, \"test #%d case failed\", tcNum)\n\t\t}\n\t\tassert.Equal(len(tc.expected), l, \"test #%d case failed\", tcNum)\n\t\tassert.Equal(tc.expected, buf.String(), \"test #%d case failed\", tcNum)\n\t}\n\n\tfor i, tc := range tcs {\n\t\tgetPrefix := func(w *PrefixWriter) []byte {\n\t\t\treturn []byte(tc.prefix)\n\t\t}\n\t\tbuf := &bytes.Buffer{}\n\t\tmlw := &MaxLineWriter{Dest: buf, MaxLines: tc.maxLines}\n\t\tpw := &PrefixWriter{Dest: mlw, PrefixFunc: getPrefix, NeedsPrefix: tc.needsPrefix}\n\t\tdoTest(tc, i, buf, pw)\n\n\t\tbuf = &bytes.Buffer{}\n\t\tpw = &PrefixWriter{Dest: buf, PrefixFunc: getPrefix, NeedsPrefix: tc.needsPrefix}\n\t\tmlw = &MaxLineWriter{Dest: pw, MaxLines: tc.maxLines}\n\t\tdoTest(tc, i, buf, mlw)\n\t}\n}\n" }, { "path": "go.mod", "content": "module github.com/attic-labs/noms\n\ngo 1.12\n\nrequire (\n\tgithub.com/BurntSushi/toml v0.3.1\n\tgithub.com/aboodman/noms-gx v0.0.0-20180714061401-d6cb97cb040b\n\tgithub.com/alecthomas/kingpin v2.2.6+incompatible // indirect\n\tgithub.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc // indirect\n\tgithub.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf // indirect\n\tgithub.com/attic-labs/graphql v0.0.0-20190507195614-b6552d20145f\n\tgithub.com/attic-labs/kingpin v2.2.7-0.20180312050558-442efcfac769+incompatible\n\tgithub.com/aws/aws-sdk-go v1.19.26\n\tgithub.com/clbanning/mxj v1.8.4\n\tgithub.com/codahale/blake2 v0.0.0-20150924215134-8d10d0420cbf\n\tgithub.com/dustin/go-humanize v1.0.0\n\tgithub.com/golang/snappy v0.0.1\n\tgithub.com/hanwen/go-fuse v1.0.0\n\tgithub.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7\n\tgithub.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d\n\tgithub.com/julienschmidt/httprouter v1.2.0\n\tgithub.com/kch42/buzhash v0.0.0-20160816060738-9bdec3dec7c6\n\tgithub.com/mattn/go-colorable v0.1.1 // indirect\n\tgithub.com/mattn/go-isatty v0.0.7\n\tgithub.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b\n\tgithub.com/shirou/gopsutil v2.18.12+incompatible\n\tgithub.com/skratchdot/open-golang v0.0.0-20190402232053-79abb63cd66e\n\tgithub.com/stretchr/testify v1.3.0\n\tgithub.com/syndtr/goleveldb v1.0.0\n\tgolang.org/x/net v0.0.0-20190503192946-f4e77d36d62c\n\tgolang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a\n\tgolang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223\n\tgopkg.in/alecthomas/kingpin.v2 v2.2.6\n)\n" }, { "path": "go.sum", "content": "cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=\ngithub.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=\ngithub.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=\ngithub.com/aboodman/noms-gx v0.0.0-20180714061401-d6cb97cb040b h1:6pe29GhapzKB4egv5gr8JHHFiExf2m81JJw4BKDBC6g=\ngithub.com/aboodman/noms-gx v0.0.0-20180714061401-d6cb97cb040b/go.mod h1:ni7quUEZfdz5Q36a9VJgeUlTaYfwY3fS3j/v5WIz8zs=\ngithub.com/alecthomas/kingpin v2.2.6+incompatible h1:5svnBTFgJjZvGKyYBtMB0+m5wvrbUHiqye8wRJMlnYI=\ngithub.com/alecthomas/kingpin v2.2.6+incompatible/go.mod h1:59OFYbFVLKQKq+mqrL6Rw5bR0c3ACQaawgXx0QYndlE=\ngithub.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc h1:cAKDfWh5VpdgMhJosfJnn5/FoN2SRZ4p7fJNX58YPaU=\ngithub.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=\ngithub.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf h1:qet1QNfXsQxTZqLG4oE62mJzwPIB8+Tee4RNCL9ulrY=\ngithub.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=\ngithub.com/attic-labs/graphql v0.0.0-20190507195614-b6552d20145f h1:WMEteRGdJItAZfxaCPyL6SEfyh4+bE+LsN50UKz46EA=\ngithub.com/attic-labs/graphql v0.0.0-20190507195614-b6552d20145f/go.mod h1:1U3eDKPYQXn3o4jpC2rAlH9THIo+ZOKWSI0FyeG1SEI=\ngithub.com/attic-labs/kingpin v2.2.6+incompatible h1:gzq18qMaCcbpq2ysBO51P6D8bficBtmYk6ZjiSOK/OQ=\ngithub.com/attic-labs/kingpin v2.2.6+incompatible/go.mod h1:Cp18FeDCvsK+cD2QAGkqerGjrgSXLiJWnjHeY2mneBc=\ngithub.com/attic-labs/kingpin v2.2.7-0.20180312050558-442efcfac769+incompatible h1:wd5mq8xSfwCYd1JpQ309s+3tTlP/gifcG2awOA3x5Vk=\ngithub.com/attic-labs/kingpin v2.2.7-0.20180312050558-442efcfac769+incompatible/go.mod h1:Cp18FeDCvsK+cD2QAGkqerGjrgSXLiJWnjHeY2mneBc=\ngithub.com/aws/aws-sdk-go v1.19.26 h1:GavKlzJDfYQGoS4jn2F+KYYZlR8QEhrLPfpf8+oJhS4=\ngithub.com/aws/aws-sdk-go v1.19.26/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=\ngithub.com/clbanning/mxj v1.8.4 h1:HuhwZtbyvyOw+3Z1AowPkU87JkJUSv751ELWaiTpj8I=\ngithub.com/clbanning/mxj v1.8.4/go.mod h1:BVjHeAH+rl9rs6f+QIpeRl0tfu10SXn1pUSa5PVGJng=\ngithub.com/codahale/blake2 v0.0.0-20150924215134-8d10d0420cbf h1:5ZeQB3mThuz5C2MSER6T5GdtXTF9CMMk42F9BOyRsEQ=\ngithub.com/codahale/blake2 v0.0.0-20150924215134-8d10d0420cbf/go.mod h1:BO2rLUAZMrpgh6GBVKi0Gjdqw2MgCtJrtmUdDeZRKjY=\ngithub.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=\ngithub.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=\ngithub.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=\ngithub.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=\ngithub.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=\ngithub.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=\ngithub.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4=\ngithub.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=\ngithub.com/hanwen/go-fuse v1.0.0 h1:GxS9Zrn6c35/BnfiVsZVWmsG803xwE7eVRDvcf/BEVc=\ngithub.com/hanwen/go-fuse v1.0.0/go.mod h1:unqXarDXqzAk0rt98O2tVndEPIpUgLD9+rwFisZH3Ok=\ngithub.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=\ngithub.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af h1:pmfjZENx5imkbgOkpRUYLnmbU7UEFbjtDA2hxJ1ichM=\ngithub.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=\ngithub.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7 h1:K//n/AqR5HjG3qxbrBCL4vJPW0MVFSs9CPK1OOJdRME=\ngithub.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7/go.mod h1:2iMrUgbbvHEiQClaW2NsSzMyGHqN+rDFqY705q49KG0=\ngithub.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d h1:c93kUJDtVAXFEhsCh5jSxyOJmFHuzcihnslQiX8Urwo=\ngithub.com/juju/gnuflag v0.0.0-20171113085948-2ce1bb71843d/go.mod h1:2PavIy+JPciBPrBUjwbNvtwB6RQlve+hkpll6QSNmOE=\ngithub.com/julienschmidt/httprouter v1.2.0 h1:TDTW5Yz1mjftljbcKqRcrYhd4XeOoI98t+9HbQbYf7g=\ngithub.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=\ngithub.com/kch42/buzhash v0.0.0-20160816060738-9bdec3dec7c6 h1:l6Y3mFnF46A+CeZsTrT8kVIuhayq1266oxWpDKE7hnQ=\ngithub.com/kch42/buzhash v0.0.0-20160816060738-9bdec3dec7c6/go.mod h1:UtDV9qK925GVmbdjR+e1unqoo+wGWNHHC6XB1Eu6wpE=\ngithub.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=\ngithub.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=\ngithub.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=\ngithub.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg=\ngithub.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=\ngithub.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=\ngithub.com/mattn/go-isatty v0.0.7 h1:UvyT9uN+3r7yLEYSlJsbQGdsaB/a0DlgWP3pql6iwOc=\ngithub.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=\ngithub.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b h1:j7+1HpAFS1zy5+Q4qx1fWh90gTKwiN4QCGoY9TWyyO4=\ngithub.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=\ngithub.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=\ngithub.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=\ngithub.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=\ngithub.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=\ngithub.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=\ngithub.com/shirou/gopsutil v2.18.12+incompatible h1:1eaJvGomDnH74/5cF4CTmTbLHAriGFsTZppLXDX93OM=\ngithub.com/shirou/gopsutil v2.18.12+incompatible/go.mod h1:5b4v6he4MtMOwMlS0TUMTu2PcXUg8+E1lC7eC3UO/RA=\ngithub.com/skratchdot/open-golang v0.0.0-20190402232053-79abb63cd66e h1:VAzdS5Nw68fbf5RZ8RDVlUvPXNU6Z3jtPCK/qvm4FoQ=\ngithub.com/skratchdot/open-golang v0.0.0-20190402232053-79abb63cd66e/go.mod h1:sUM3LWHvSMaG192sy56D9F7CNvL7jUJVXoqM1QKLnog=\ngithub.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=\ngithub.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=\ngithub.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=\ngithub.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE=\ngithub.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ=\ngolang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=\ngolang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20190503192946-f4e77d36d62c h1:uOCk1iQW6Vc18bnC13MfzScl+wdKBmM9Y9kU7Z83/lw=\ngolang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=\ngolang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a h1:tImsplftrFpALCYumobsd0K86vlAs/eXGFms2txfJfA=\ngolang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=\ngolang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU=\ngolang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223 h1:DH4skfRX4EBpamg7iV4ZlCpblAHI6s6TDM39bFZumv8=\ngolang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=\ngoogle.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=\ngopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc=\ngopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=\ngopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=\ngopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=\ngopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=\n" }, { "path": "release.sh", "content": "#!/bin/sh\ngit tag -f latest HEAD\ngit push -f origin latest\n" }, { "path": "samples/cli/nomsconfig/README.md", "content": "# nomsconfig\n\nThe noms cli now provides experimental support for configuring a convenient default database and database aliases.\n\nYou can enable this support by placing a *.nomsconfig* config file (like the [one](.nomsconfig) in this sample) in the directory where you'd like to use the configuration. Like git, any noms command issued from that directory or below will use it.\n\n# Features\n\n- *Database Aliases* - Define simple names to be used in place of database URLs\n- *Default Database* - Define one database to be used by default when no database in mentioned\n- *Dot (`.`) Shorthand* - Use `.` instead of repeating dataset/object name in destination\n\n# Example\n\nThis example defines a simple [.nomsconfig](.nomsconfig) to try:\n\n```shell\n# Default database URL to be used whenever a database is not explictly provided\n[db.default]\nurl = \"ldb:.noms/tour\"\n\n# DB alias named `origin` that refers to the remote cli-tour db \n[db.origin]\nurl = \"http://demo.noms.io/cli-tour\"\n\n# DB alias named `temp` that refers to a noms db stored under /tmp\n[db.temp]\nurl = \"ldb:/tmp/noms/shared\n\n```\n\nThe *[db.default]* section:\n\n - Defines a default database\n - It will be used implicitly whenever a database url is omitted in a command\n\nThe *[db.origin]* and *[db.shared]* sections:\n\n - Define aliases that can be used wherever a db url is required\n - You can define additional aliases by adding *[db.**alias**]* sections using any **alias** you prefer\n\nDot (`.`) shorthand:\n\n - When issuing a command that requires a source and destination (like `noms sync`), \n you can use `.` in place of the dataset/object in the destination. This is shorthand \n that repeats whatever was used in the source (see below).\n\n\nYou can kick the tires by running noms commands from this directory. Here are some examples and what to expect:\n\n```shell\nnoms ds # -> noms ds ldb:.noms/tour\nnoms ds default # -> noms ds ldb:.noms/tour\nnoms ds origin # -> noms ds http://demo.noms.io/cli-tour\n\nnoms sync origin::sf-film-locations sf-films # sync ds from origin to default\n\nnoms log sf-films # -> noms log ldb:.noms/tour::sf-films\nnoms log origin::sf-film-locations # -> noms log http://demo.noms.io/cli-tour::sf-film-locations\n\nnoms show '#1a2aj8svslsu7g8hplsva6oq6iq3ib6c' # -> noms show ldb:.noms/tour::'#1a2a...'\nnoms show origin::'#1a2aj8svslsu7g8hplsva6oq6iq3ib6c' # -> noms show http://demo.noms.io/cli-tour::'#1a2a...'\n\nnoms diff '#1a2aj8svslsu7g8hplsva6oq6iq3ib6c' origin::. # diff default::object with origin::object\n\nnoms sync origin::sf-bike-parking . # sync origin::sf-bike-parking to default::sf-bike-parking\n\n``` \n\nA few more things to note:\n\n - Relative paths will be expanded relative to the directory where the *.nomsconfg* is defined\n - Use `noms config` to see the current alias definitions with expanded paths\n - Use `-v` or `--verbose` on any command to see how the command arguments are being resolved\n - Explicit DB urls are still fully supported" }, { "path": "samples/go/csv/README.md", "content": "# CSV Importer\n\nImports a CSV file as `List` where `T` is a struct with fields corresponding to the CSV's column headers. The struct spec can also be set manually with the `-header` flag.\n\n## Usage\n\n```shell\n$ cd csv-import\n$ go build\n$ ./csv-import http://localhost:8000::foo\n```\n\n## Some places for CSV files\n\n- https://data.cityofnewyork.us/api/views/kku6-nxdu/rows.csv?accessType=DOWNLOAD\n- http://www.opendatacache.com/\n\n# CSV Exporter\n\nExport a dataset in CSV format to stdout with column headers.\n\n## Usage\n\n```shell\n$ cd csv-export\n$ go build\n$ ./csv-export http://localhost:8000:foo\n```\n" }, { "path": "samples/go/csv/common.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"fmt\"\n\t\"unicode/utf8\"\n)\n\n// StringToRune returns the rune contained in delimiter or an error.\nfunc StringToRune(delimiter string) (rune, error) {\n\tdlimLen := len(delimiter)\n\tif dlimLen == 0 {\n\t\treturn 0, fmt.Errorf(\"delimiter flag must contain exactly one character (rune), not an empty string\")\n\t}\n\n\td, runeSize := utf8.DecodeRuneInString(delimiter)\n\tif d == utf8.RuneError {\n\t\treturn 0, fmt.Errorf(\"Invalid utf8 string in delimiter flag: %s\", delimiter)\n\t}\n\tif dlimLen != runeSize {\n\t\treturn 0, fmt.Errorf(\"delimiter flag is too long. It must contain exactly one character (rune), but instead it is: %s\", delimiter)\n\t}\n\treturn d, nil\n}\n" }, { "path": "samples/go/csv/csv_reader.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"bufio\"\n\t\"encoding/csv\"\n\t\"io\"\n)\n\nvar (\n\trByte byte = 13 // the byte that corresponds to the '\\r' rune.\n\tnByte byte = 10 // the byte that corresponds to the '\\n' rune.\n)\n\ntype reader struct {\n\tr *bufio.Reader\n}\n\n// Read replaces CR line endings in the source reader with LF line endings if the CR is not followed by a LF.\nfunc (r reader) Read(p []byte) (n int, err error) {\n\tn, err = r.r.Read(p)\n\tbn, err := r.r.Peek(1)\n\tfor i, b := range p {\n\t\t// if the current byte is a CR and the next byte is NOT a LF then replace the current byte with a LF\n\t\tif j := i + 1; b == rByte && ((j < len(p) && p[j] != nByte) || (len(bn) > 0 && bn[0] != nByte)) {\n\t\t\tp[i] = nByte\n\t\t}\n\t}\n\treturn\n}\n\nfunc SkipRecords(r *csv.Reader, n uint) error {\n\tvar err error\n\tfor i := uint(0); i < n; i++ {\n\t\t_, err = r.Read()\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn err\n}\n\n// NewCSVReader returns a new csv.Reader that splits on comma\nfunc NewCSVReader(res io.Reader, comma rune) *csv.Reader {\n\tbufRes := bufio.NewReader(res)\n\tr := csv.NewReader(reader{r: bufRes})\n\tr.Comma = comma\n\tr.FieldsPerRecord = -1 // Don't enforce number of fields.\n\treturn r\n}\n" }, { "path": "samples/go/csv/csv_reader_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"bytes\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestCR(t *testing.T) {\n\ttestFile := []byte(\"a,b,c\\r1,2,3\\r\")\n\tdelimiter, err := StringToRune(\",\")\n\n\tr := NewCSVReader(bytes.NewReader(testFile), delimiter)\n\tlines, err := r.ReadAll()\n\n\tassert.NoError(t, err, \"An error occurred while reading the data: %v\", err)\n\tif len(lines) != 2 {\n\t\tt.Errorf(\"Wrong number of lines. Expected 2, got %d\", len(lines))\n\t}\n}\n\nfunc TestLF(t *testing.T) {\n\ttestFile := []byte(\"a,b,c\\n1,2,3\\n\")\n\tdelimiter, err := StringToRune(\",\")\n\n\tr := NewCSVReader(bytes.NewReader(testFile), delimiter)\n\tlines, err := r.ReadAll()\n\n\tassert.NoError(t, err, \"An error occurred while reading the data: %v\", err)\n\tif len(lines) != 2 {\n\t\tt.Errorf(\"Wrong number of lines. Expected 2, got %d\", len(lines))\n\t}\n}\n\nfunc TestCRLF(t *testing.T) {\n\ttestFile := []byte(\"a,b,c\\r\\n1,2,3\\r\\n\")\n\tdelimiter, err := StringToRune(\",\")\n\n\tr := NewCSVReader(bytes.NewReader(testFile), delimiter)\n\tlines, err := r.ReadAll()\n\n\tassert.NoError(t, err, \"An error occurred while reading the data: %v\", err)\n\tif len(lines) != 2 {\n\t\tt.Errorf(\"Wrong number of lines. Expected 2, got %d\", len(lines))\n\t}\n}\n\nfunc TestCRInQuote(t *testing.T) {\n\ttestFile := []byte(\"a,\\\"foo,\\rbar\\\",c\\r1,\\\"2\\r\\n2\\\",3\\r\")\n\tdelimiter, err := StringToRune(\",\")\n\n\tr := NewCSVReader(bytes.NewReader(testFile), delimiter)\n\tlines, err := r.ReadAll()\n\n\tassert.NoError(t, err, \"An error occurred while reading the data: %v\", err)\n\tif len(lines) != 2 {\n\t\tt.Errorf(\"Wrong number of lines. Expected 2, got %d\", len(lines))\n\t}\n\tif strings.Contains(lines[1][1], \"\\n\\n\") {\n\t\tt.Error(\"The CRLF was converted to a LFLF\")\n\t}\n}\n\nfunc TestCRLFEndOfBufferLength(t *testing.T) {\n\ttestFile := make([]byte, 4096*2, 4096*2)\n\ttestFile[4095] = 13 // \\r byte\n\ttestFile[4096] = 10 // \\n byte\n\tdelimiter, err := StringToRune(\",\")\n\n\tr := NewCSVReader(bytes.NewReader(testFile), delimiter)\n\tlines, err := r.ReadAll()\n\n\tassert.NoError(t, err, \"An error occurred while reading the data: %v\", err)\n\tif len(lines) != 2 {\n\t\tt.Errorf(\"Wrong number of lines. Expected 2, got %d\", len(lines))\n\t}\n}\n" }, { "path": "samples/go/csv/kind_slice.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"fmt\"\n\t\"strconv\"\n\t\"strings\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// KindSlice is an alias for []types.NomsKind. It's needed because types.NomsKind are really just 8 bit unsigned ints, which are what Go uses to represent 'byte', and this confuses the Go JSON marshal/unmarshal code -- it treats them as byte arrays and base64 encodes them!\ntype KindSlice []types.NomsKind\n\nfunc (ks KindSlice) MarshalJSON() ([]byte, error) {\n\telems := make([]string, len(ks))\n\tfor i, k := range ks {\n\t\telems[i] = fmt.Sprintf(\"%d\", k)\n\t}\n\treturn []byte(\"[\" + strings.Join(elems, \",\") + \"]\"), nil\n}\n\nfunc (ks *KindSlice) UnmarshalJSON(value []byte) error {\n\telems := strings.Split(string(value[1:len(value)-1]), \",\")\n\t*ks = make(KindSlice, len(elems))\n\tfor i, e := range elems {\n\t\tival, err := strconv.ParseUint(e, 10, 8)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\t(*ks)[i] = types.NomsKind(ival)\n\t}\n\treturn nil\n}\n" }, { "path": "samples/go/csv/kind_slice_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestKindSliceJSON(t *testing.T) {\n\tassert := assert.New(t)\n\n\tks := KindSlice{types.NumberKind, types.StringKind, types.BoolKind}\n\tb, err := json.Marshal(&ks)\n\tassert.NoError(err)\n\n\tassert.Equal(fmt.Sprintf(\"[%d,%d,%d]\", ks[0], ks[1], ks[2]), string(b))\n\n\tvar uks KindSlice\n\terr = json.Unmarshal(b, &uks)\n\tassert.NoError(err, \"error with json.Unmarshal\")\n\tassert.Equal(ks, uks)\n}\n" }, { "path": "samples/go/csv/read.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"encoding/csv\"\n\t\"fmt\"\n\t\"io\"\n\t\"sort\"\n\t\"strconv\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n// StringToKind maps names of valid NomsKinds (e.g. Bool, Number, etc) to their associated types.NomsKind\nvar StringToKind = func(kindMap map[types.NomsKind]string) map[string]types.NomsKind {\n\tm := map[string]types.NomsKind{}\n\tfor k, v := range kindMap {\n\t\tm[v] = k\n\t}\n\treturn m\n}(types.KindToString)\n\n// StringsToKinds looks up each element of strs in the StringToKind map and returns a slice of answers\nfunc StringsToKinds(strs []string) KindSlice {\n\tkinds := make(KindSlice, len(strs))\n\tfor i, str := range strs {\n\t\tk, ok := StringToKind[str]\n\t\tif !ok {\n\t\t\td.Panic(\"StringToKind[%s] failed\", str)\n\t\t}\n\t\tkinds[i] = k\n\t}\n\treturn kinds\n}\n\n// KindsToStrings looks up each element of kinds in the types.KindToString map and returns a slice of answers\nfunc KindsToStrings(kinds KindSlice) []string {\n\tstrs := make([]string, len(kinds))\n\tfor i, k := range kinds {\n\t\tstrs[i] = k.String()\n\t}\n\treturn strs\n}\n\n//EscapeStructFieldFromCSV removes special characters and replaces spaces with camelCasing (camel case turns to camelCase)\nfunc EscapeStructFieldFromCSV(input string) string {\n\tif types.IsValidStructFieldName(input) {\n\t\treturn input\n\t}\n\treturn types.CamelCaseFieldName(input)\n}\n\n// MakeStructTemplateFromHeaders creates a struct type from the headers using |kinds| as the type of each field. If |kinds| is empty, default to strings.\nfunc MakeStructTemplateFromHeaders(headers []string, structName string, kinds KindSlice) (temp types.StructTemplate, fieldOrder []int, kindMap []types.NomsKind) {\n\tuseStringType := len(kinds) == 0\n\td.PanicIfFalse(useStringType || len(headers) == len(kinds))\n\n\tfieldMap := make(map[string]types.NomsKind, len(headers))\n\torigOrder := make(map[string]int, len(headers))\n\tfieldNames := make(sort.StringSlice, len(headers))\n\n\tfor i, key := range headers {\n\t\tfn := EscapeStructFieldFromCSV(key)\n\t\torigOrder[fn] = i\n\t\tkind := types.StringKind\n\t\tif !useStringType {\n\t\t\tkind = kinds[i]\n\t\t}\n\t\t_, ok := fieldMap[fn]\n\t\tif ok {\n\t\t\td.Panic(`Duplicate field name \"%s\"`, key)\n\t\t}\n\t\tfieldMap[fn] = kind\n\t\tfieldNames[i] = fn\n\t}\n\n\tsort.Sort(fieldNames)\n\n\tkindMap = make([]types.NomsKind, len(fieldMap))\n\tfieldOrder = make([]int, len(fieldMap))\n\n\tfor i, fn := range fieldNames {\n\t\tkindMap[i] = fieldMap[fn]\n\t\tfieldOrder[origOrder[fn]] = i\n\t}\n\n\ttemp = types.MakeStructTemplate(structName, fieldNames)\n\treturn\n}\n\n// ReadToList takes a CSV reader and reads data into a typed List of structs.\n// Each row gets read into a struct named structName, described by headers. If\n// the original data contained headers it is expected that the input reader has\n// already read those and are pointing at the first data row.\n// If kinds is non-empty, it will be used to type the fields in the generated\n// structs; otherwise, they will be left as string-fields.\n// In addition to the list, ReadToList returns the typeDef of the structs in the\n// list.\nfunc ReadToList(r *csv.Reader, structName string, headers []string, kinds KindSlice, vrw types.ValueReadWriter, limit uint64) (l types.List) {\n\ttemp, fieldOrder, kindMap := MakeStructTemplateFromHeaders(headers, structName, kinds)\n\tvalueChan := make(chan types.Value, 128) // TODO: Make this a function param?\n\tlistChan := types.NewStreamingList(vrw, valueChan)\n\n\tcnt := uint64(0)\n\tfor {\n\t\trow, err := r.Read()\n\t\tif cnt >= limit || err == io.EOF {\n\t\t\tclose(valueChan)\n\t\t\tbreak\n\t\t} else if err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t\tcnt++\n\n\t\tfields := readFieldsFromRow(row, headers, fieldOrder, kindMap)\n\t\tvalueChan <- temp.NewStruct(fields)\n\t}\n\n\treturn <-listChan\n}\n\ntype column struct {\n\tch chan types.Value\n\tlist <-chan types.List\n\tzeroValue types.Value\n\thdr string\n}\n\n// ReadToColumnar takes a CSV reader and reads data from each column into a\n// separate list. Values from columns in each successive row are appended to the\n// column-specific lists whose type is described by headers. Finally, a new\n// \"Columnar\" struct is created that consists of one field for each column and\n// each field contains a list of values.\n// If the original data contained headers it is expected that the input reader\n// has already read those and are pointing at the first data row.\n// If kinds is non-empty, it will be used to type the fields in the generated\n// structs; otherwise, they will be left as string-fields.\n// In addition to the list, ReadToList returns the typeDef of the structs in the\n// list.\nfunc ReadToColumnar(r *csv.Reader, structName string, headers []string, kinds KindSlice, vrw types.ValueReadWriter, limit uint64) (s types.Struct) {\n\tvalueChan := make(chan types.Value, 128) // TODO: Make this a function param?\n\tcols := []column{}\n\tfieldOrder := []int{}\n\tfor i, hdr := range headers {\n\t\tch := make(chan types.Value, 1024)\n\t\tcols = append(cols, column{\n\t\t\tch: ch,\n\t\t\tlist: types.NewStreamingList(vrw, ch),\n\t\t\thdr: hdr,\n\t\t})\n\t\tfieldOrder = append(fieldOrder, i)\n\t}\n\n\tcnt := uint64(0)\n\tfor {\n\t\trow, err := r.Read()\n\t\tif cnt >= limit || err == io.EOF {\n\t\t\tclose(valueChan)\n\t\t\tbreak\n\t\t} else if err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t\tcnt++\n\n\t\tfields := readFieldsFromRow(row, headers, fieldOrder, kinds)\n\t\tfor i, v := range fields {\n\t\t\tcols[i].ch <- v\n\t\t}\n\t}\n\n\tsd := types.StructData{}\n\tfor _, col := range cols {\n\t\tclose(col.ch)\n\t\tr := vrw.WriteValue(<-col.list)\n\t\tsd[col.hdr] = r\n\t}\n\treturn types.NewStruct(\"Columnar\", sd)\n}\n\n// getFieldIndexByHeaderName takes the collection of headers and the name to search for and returns the index of name within the headers or -1 if not found\nfunc getFieldIndexByHeaderName(headers []string, name string) int {\n\tfor i, header := range headers {\n\t\tif header == name {\n\t\t\treturn i\n\t\t}\n\t}\n\treturn -1\n}\n\n// getPkIndices takes collection of primary keys as strings and determines if they are integers, if so then use those ints as the indices, otherwise it looks up the strings in the headers to find the indices; returning the collection of int indices representing the primary keys maintaining the order of strPks to the return collection\nfunc getPkIndices(strPks []string, headers []string) []int {\n\tresult := make([]int, len(strPks))\n\tfor i, pk := range strPks {\n\t\tpkIdx, ok := strconv.Atoi(pk)\n\t\tif ok == nil {\n\t\t\tresult[i] = pkIdx\n\t\t} else {\n\t\t\tresult[i] = getFieldIndexByHeaderName(headers, pk)\n\t\t}\n\t\tif result[i] < 0 {\n\t\t\td.Chk.Fail(fmt.Sprintf(\"Invalid pk: %v\", pk))\n\t\t}\n\t}\n\treturn result\n}\n\nfunc readFieldsFromRow(row []string, headers []string, fieldOrder []int, kindMap []types.NomsKind) types.ValueSlice {\n\tfields := make(types.ValueSlice, len(headers))\n\tfor i, v := range row {\n\t\tif i < len(headers) {\n\t\t\tfieldOrigIndex := fieldOrder[i]\n\t\t\tval, err := StringToValue(v, kindMap[fieldOrigIndex])\n\t\t\tif err != nil {\n\t\t\t\td.Chk.Fail(fmt.Sprintf(\"Error parsing value for column '%s': %s\", headers[i], err))\n\t\t\t}\n\t\t\tfields[fieldOrigIndex] = val\n\t\t}\n\t}\n\treturn fields\n}\n\n// primaryKeyValuesFromFields extracts the values of the primaryKey fields into\n// array. The values are in the user-specified order. This function returns 2\n// objects:\n// 1) a ValueSlice containing the first n-1 keys.\n// 2) a single Value which will be used as the key in the leaf map created by\n// GraphBuilder\nfunc primaryKeyValuesFromFields(fields types.ValueSlice, fieldOrder, pkIndices []int) (types.ValueSlice, types.Value) {\n\tnumPrimaryKeys := len(pkIndices)\n\n\tif numPrimaryKeys == 1 {\n\t\treturn nil, fields[fieldOrder[pkIndices[0]]]\n\t}\n\n\tkeys := make(types.ValueSlice, numPrimaryKeys-1)\n\tvar value types.Value\n\tfor i, idx := range pkIndices {\n\t\tk := fields[fieldOrder[idx]]\n\t\tif i < numPrimaryKeys-1 {\n\t\t\tkeys[i] = k\n\t\t} else {\n\t\t\tvalue = k\n\t\t}\n\t}\n\treturn keys, value\n}\n\n// ReadToMap takes a CSV reader and reads data into a typed Map of structs. Each\n// row gets read into a struct named structName, described by headers. If the\n// original data contained headers it is expected that the input reader has\n// already read those and are pointing at the first data row.\n// If kinds is non-empty, it will be used to type the fields in the generated\n// structs; otherwise, they will be left as string-fields.\nfunc ReadToMap(r *csv.Reader, structName string, headersRaw []string, primaryKeys []string, kinds KindSlice, vrw types.ValueReadWriter, limit uint64) types.Map {\n\ttemp, fieldOrder, kindMap := MakeStructTemplateFromHeaders(headersRaw, structName, kinds)\n\tpkIndices := getPkIndices(primaryKeys, headersRaw)\n\td.Chk.True(len(pkIndices) >= 1, \"No primary key defined when reading into map\")\n\tgb := types.NewGraphBuilder(vrw, types.MapKind)\n\n\tcnt := uint64(0)\n\tfor {\n\t\trow, err := r.Read()\n\t\tif cnt >= limit || err == io.EOF {\n\t\t\tbreak\n\t\t} else if err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t\tcnt++\n\n\t\tfields := readFieldsFromRow(row, headersRaw, fieldOrder, kindMap)\n\t\tgraphKeys, mapKey := primaryKeyValuesFromFields(fields, fieldOrder, pkIndices)\n\t\tst := temp.NewStruct(fields)\n\t\tgb.MapSet(graphKeys, mapKey, st)\n\t}\n\treturn gb.Build().(types.Map)\n}\n" }, { "path": "samples/go/csv/read_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"bytes\"\n\t\"encoding/csv\"\n\t\"math\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nvar LIMIT = uint64(math.MaxUint64)\n\nfunc TestReadToList(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\n\tdataString := `a,1,true\nb,2,false\n`\n\tr := NewCSVReader(bytes.NewBufferString(dataString), ',')\n\n\theaders := []string{\"A\", \"B\", \"C\"}\n\tkinds := KindSlice{types.StringKind, types.NumberKind, types.BoolKind}\n\tl := ReadToList(r, \"test\", headers, kinds, db, LIMIT)\n\n\tassert.Equal(uint64(2), l.Len())\n\n\tassert.True(l.Get(0).(types.Struct).Get(\"A\").Equals(types.String(\"a\")))\n\tassert.True(l.Get(1).(types.Struct).Get(\"A\").Equals(types.String(\"b\")))\n\n\tassert.True(l.Get(0).(types.Struct).Get(\"B\").Equals(types.Number(1)))\n\tassert.True(l.Get(1).(types.Struct).Get(\"B\").Equals(types.Number(2)))\n\n\tassert.True(l.Get(0).(types.Struct).Get(\"C\").Equals(types.Bool(true)))\n\tassert.True(l.Get(1).(types.Struct).Get(\"C\").Equals(types.Bool(false)))\n}\n\nfunc TestReadToMap(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\n\tdataString := `a,1,true\nb,2,false\n`\n\tr := NewCSVReader(bytes.NewBufferString(dataString), ',')\n\n\theaders := []string{\"A\", \"B\", \"C\"}\n\tkinds := KindSlice{types.StringKind, types.NumberKind, types.BoolKind}\n\tm := ReadToMap(r, \"test\", headers, []string{\"0\"}, kinds, db, LIMIT)\n\n\tassert.Equal(uint64(2), m.Len())\n\tassert.True(types.TypeOf(m).Equals(\n\t\ttypes.MakeMapType(types.StringType, types.MakeStructType(\"test\",\n\t\t\ttypes.StructField{Name: \"A\", Type: types.StringType, Optional: false},\n\t\t\ttypes.StructField{Name: \"B\", Type: types.NumberType, Optional: false},\n\t\t\ttypes.StructField{Name: \"C\", Type: types.BoolType, Optional: false},\n\t\t))))\n\n\tassert.True(m.Get(types.String(\"a\")).Equals(types.NewStruct(\"test\", types.StructData{\n\t\t\"A\": types.String(\"a\"),\n\t\t\"B\": types.Number(1),\n\t\t\"C\": types.Bool(true),\n\t})))\n\tassert.True(m.Get(types.String(\"b\")).Equals(types.NewStruct(\"test\", types.StructData{\n\t\t\"A\": types.String(\"b\"),\n\t\t\"B\": types.Number(2),\n\t\t\"C\": types.Bool(false),\n\t})))\n}\n\nfunc testTrailingHelper(t *testing.T, dataString string) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb1 := datas.NewDatabase(storage.NewView())\n\tdefer db1.Close()\n\n\tr := NewCSVReader(bytes.NewBufferString(dataString), ',')\n\n\theaders := []string{\"A\", \"B\"}\n\tkinds := KindSlice{types.StringKind, types.StringKind}\n\tl := ReadToList(r, \"test\", headers, kinds, db1, LIMIT)\n\tassert.Equal(uint64(3), l.Len())\n\n\tstorage = &chunks.MemoryStorage{}\n\tdb2 := datas.NewDatabase(storage.NewView())\n\tdefer db2.Close()\n\tr = NewCSVReader(bytes.NewBufferString(dataString), ',')\n\tm := ReadToMap(r, \"test\", headers, []string{\"0\"}, kinds, db2, LIMIT)\n\tassert.Equal(uint64(3), m.Len())\n}\n\nfunc TestReadTrailingHole(t *testing.T) {\n\tdataString := `a,b,\nd,e,\ng,h,\n`\n\ttestTrailingHelper(t, dataString)\n}\n\nfunc TestReadTrailingHoles(t *testing.T) {\n\tdataString := `a,b,,\nd,e\ng,h\n`\n\ttestTrailingHelper(t, dataString)\n}\n\nfunc TestReadTrailingValues(t *testing.T) {\n\tdataString := `a,b\nd,e,f\ng,h,i,j\n`\n\ttestTrailingHelper(t, dataString)\n}\n\nfunc TestEscapeStructFieldFromCSV(t *testing.T) {\n\tassert := assert.New(t)\n\tcases := []string{\n\t\t\"a\", \"a\",\n\t\t\"1a\", \"a\",\n\t\t\"AaZz19_\", \"AaZz19_\",\n\t\t\"Q\", \"Q\",\n\t\t\"AQ\", \"AQ\",\n\t\t\"_content\", \"content\",\n\t\t\"Few ¢ents Short\", \"fewEntsShort\",\n\t\t\"CAMEL💩case letTerS\", \"camelcaseLetters\",\n\t\t\"https://picasaweb.google.com/data\", \"httpspicasawebgooglecomdata\",\n\t\t\"💩\", \"\",\n\t\t\"11 1💩\", \"\",\n\t\t\"-- A B\", \"aB\",\n\t\t\"-- A --\", \"a\",\n\t\t\"-- A -- B\", \"aB\",\n\t}\n\n\tfor i := 0; i < len(cases); i += 2 {\n\t\torig, expected := cases[i], cases[i+1]\n\t\tassert.Equal(expected, EscapeStructFieldFromCSV(orig))\n\t}\n}\n\nfunc TestReadParseError(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\n\tdataString := `a,\"b`\n\tr := NewCSVReader(bytes.NewBufferString(dataString), ',')\n\n\theaders := []string{\"A\", \"B\"}\n\tkinds := KindSlice{types.StringKind, types.StringKind}\n\tfunc() {\n\t\tdefer func() {\n\t\t\tr := recover()\n\t\t\tassert.NotNil(r)\n\t\t\t_, ok := r.(*csv.ParseError)\n\t\t\tassert.True(ok, \"Should be a ParseError\")\n\t\t}()\n\t\tReadToList(r, \"test\", headers, kinds, db, LIMIT)\n\t}()\n}\n\nfunc TestDuplicateHeaderName(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\tdataString := \"1,2\\n3,4\\n\"\n\tr := NewCSVReader(bytes.NewBufferString(dataString), ',')\n\theaders := []string{\"A\", \"A\"}\n\tkinds := KindSlice{types.StringKind, types.StringKind}\n\tassert.Panics(func() { ReadToList(r, \"test\", headers, kinds, db, LIMIT) })\n}\n\nfunc TestEscapeFieldNames(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\tdataString := \"1,2\\n\"\n\tr := NewCSVReader(bytes.NewBufferString(dataString), ',')\n\theaders := []string{\"A A\", \"B\"}\n\tkinds := KindSlice{types.NumberKind, types.NumberKind}\n\n\tl := ReadToList(r, \"test\", headers, kinds, db, LIMIT)\n\tassert.Equal(uint64(1), l.Len())\n\tassert.Equal(types.Number(1), l.Get(0).(types.Struct).Get(EscapeStructFieldFromCSV(\"A A\")))\n\n\tr = NewCSVReader(bytes.NewBufferString(dataString), ',')\n\tm := ReadToMap(r, \"test\", headers, []string{\"1\"}, kinds, db, LIMIT)\n\tassert.Equal(uint64(1), l.Len())\n\tassert.Equal(types.Number(1), m.Get(types.Number(2)).(types.Struct).Get(EscapeStructFieldFromCSV(\"A A\")))\n}\n\nfunc TestDefaults(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\tdataString := \"42,,,\\n\"\n\tr := NewCSVReader(bytes.NewBufferString(dataString), ',')\n\theaders := []string{\"A\", \"B\", \"C\", \"D\"}\n\tkinds := KindSlice{types.NumberKind, types.NumberKind, types.BoolKind, types.StringKind}\n\n\tl := ReadToList(r, \"test\", headers, kinds, db, LIMIT)\n\tassert.Equal(uint64(1), l.Len())\n\trow := l.Get(0).(types.Struct)\n\tassert.Equal(types.Number(42), row.Get(\"A\"))\n\tassert.Equal(types.Number(0), row.Get(\"B\"))\n\tassert.Equal(types.Bool(false), row.Get(\"C\"))\n\tassert.Equal(types.String(\"\"), row.Get(\"D\"))\n}\n\nfunc TestBooleanStrings(t *testing.T) {\n\tassert := assert.New(t)\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\tdataString := \"true,false\\n1,0\\ny,n\\nY,N\\nY,\\n\"\n\tr := NewCSVReader(bytes.NewBufferString(dataString), ',')\n\theaders := []string{\"T\", \"F\"}\n\tkinds := KindSlice{types.BoolKind, types.BoolKind}\n\n\tl := ReadToList(r, \"test\", headers, kinds, db, LIMIT)\n\tassert.Equal(uint64(5), l.Len())\n\tfor i := uint64(0); i < l.Len(); i++ {\n\t\trow := l.Get(i).(types.Struct)\n\t\tassert.True(types.Bool(true).Equals(row.Get(\"T\")))\n\t\tassert.True(types.Bool(false).Equals(row.Get(\"F\")))\n\t}\n}\n" }, { "path": "samples/go/csv/schema.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"encoding/csv\"\n\t\"fmt\"\n\t\"io\"\n\t\"math\"\n\t\"strconv\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\ntype schemaOptions []*typeCanFit\n\nfunc newSchemaOptions(fieldCount int) schemaOptions {\n\toptions := make([]*typeCanFit, fieldCount, fieldCount)\n\tfor i := 0; i < fieldCount; i++ {\n\t\toptions[i] = &typeCanFit{true, true, true}\n\t}\n\treturn options\n}\n\nfunc (so schemaOptions) Test(fields []string) {\n\tfor i, t := range so {\n\t\tif i < len(fields) {\n\t\t\tt.Test(fields[i])\n\t\t}\n\t}\n}\n\nfunc (so schemaOptions) MostSpecificKinds() KindSlice {\n\tkinds := make(KindSlice, len(so))\n\tfor i, t := range so {\n\t\tkinds[i] = t.MostSpecificKind()\n\t}\n\treturn kinds\n}\n\nfunc (so schemaOptions) ValidKinds() []KindSlice {\n\tkinds := make([]KindSlice, len(so))\n\tfor i, t := range so {\n\t\tkinds[i] = t.ValidKinds()\n\t}\n\treturn kinds\n}\n\ntype typeCanFit struct {\n\tboolType bool\n\tnumberType bool\n\tstringType bool\n}\n\nfunc (tc *typeCanFit) MostSpecificKind() types.NomsKind {\n\tif tc.boolType {\n\t\treturn types.BoolKind\n\t} else if tc.numberType {\n\t\treturn types.NumberKind\n\t} else {\n\t\treturn types.StringKind\n\t}\n}\n\nfunc (tc *typeCanFit) ValidKinds() (kinds KindSlice) {\n\tif tc.numberType {\n\t\tkinds = append(kinds, types.NumberKind)\n\t}\n\tif tc.boolType {\n\t\tkinds = append(kinds, types.BoolKind)\n\t}\n\tkinds = append(kinds, types.StringKind)\n\treturn kinds\n}\n\nfunc (tc *typeCanFit) Test(value string) {\n\ttc.testNumbers(value)\n\ttc.testBool(value)\n}\n\nfunc (tc *typeCanFit) testNumbers(value string) {\n\tif !tc.numberType {\n\t\treturn\n\t}\n\n\tfval, err := strconv.ParseFloat(value, 64)\n\tif err != nil {\n\t\ttc.numberType = false\n\t\treturn\n\t}\n\n\tif fval > math.MaxFloat64 {\n\t\ttc.numberType = false\n\t}\n}\n\nfunc (tc *typeCanFit) testBool(value string) {\n\tif !tc.boolType {\n\t\treturn\n\t}\n\t_, err := strconv.ParseBool(value)\n\ttc.boolType = err == nil\n}\n\nfunc GetSchema(r *csv.Reader, numSamples int, numFields int) KindSlice {\n\tso := newSchemaOptions(numFields)\n\tfor i := 0; i < numSamples; i++ {\n\t\trow, err := r.Read()\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\t\tso.Test(row)\n\t}\n\treturn so.MostSpecificKinds()\n}\n\nfunc GetFieldNamesFromIndices(headers []string, indices []int) []string {\n\tresult := make([]string, len(indices))\n\tfor i, idx := range indices {\n\t\tresult[i] = headers[idx]\n\t}\n\treturn result\n}\n\n// combinations - n choose m combination without repeat - emit all possible `length` combinations from values\nfunc combinationsWithLength(values []int, length int, emit func([]int)) {\n\tn := len(values)\n\n\tif length > n {\n\t\treturn\n\t}\n\n\tindices := make([]int, length)\n\tfor i := range indices {\n\t\tindices[i] = i\n\t}\n\n\tresult := make([]int, length)\n\tfor i, l := range indices {\n\t\tresult[i] = values[l]\n\t}\n\temit(result)\n\n\tfor {\n\t\ti := length - 1\n\t\tfor ; i >= 0 && indices[i] == i+n-length; i -= 1 {\n\t\t}\n\n\t\tif i < 0 {\n\t\t\treturn\n\t\t}\n\n\t\tindices[i] += 1\n\t\tfor j := i + 1; j < length; j += 1 {\n\t\t\tindices[j] = indices[j-1] + 1\n\t\t}\n\n\t\tfor ; i < len(indices); i += 1 {\n\t\t\tresult[i] = values[indices[i]]\n\t\t}\n\t\temit(result)\n\t}\n}\n\n// combinationsLengthsFromTo - n choose m combination without repeat - emit all possible combinations of all lengths from smallestLength to largestLength (inclusive)\nfunc combinationsLengthsFromTo(values []int, smallestLength, largestLength int, emit func([]int)) {\n\tfor i := smallestLength; i <= largestLength; i++ {\n\t\tcombinationsWithLength(values, i, emit)\n\t}\n}\n\nfunc makeKeyString(row []string, indices []int, separator string) string {\n\tvar result string\n\tfor _, i := range indices {\n\t\tresult += separator\n\t\tresult += row[i]\n\t}\n\treturn result\n}\n\n// FindPrimaryKeys reads numSamples from r, using the first numFields and returns slices of []int indices that are primary keys for those samples\nfunc FindPrimaryKeys(r *csv.Reader, numSamples, maxLenPrimaryKeyList, numFields int) [][]int {\n\tdataToTest := make([][]string, 0, numSamples)\n\tfor i := int(0); i < numSamples; i++ {\n\t\trow, err := r.Read()\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\t\tdataToTest = append(dataToTest, row)\n\t}\n\n\tindices := make([]int, numFields)\n\tfor i := int(0); i < numFields; i++ {\n\t\tindices[i] = i\n\t}\n\n\tpksFound := make([][]int, 0)\n\tcombinationsLengthsFromTo(indices, 1, maxLenPrimaryKeyList, func(combination []int) {\n\t\tkeys := make(map[string]bool, numSamples)\n\t\tfor _, row := range dataToTest {\n\t\t\tkey := makeKeyString(row, combination, \"$&$\")\n\t\t\tif _, ok := keys[key]; ok {\n\t\t\t\treturn\n\t\t\t}\n\t\t\tkeys[key] = true\n\t\t}\n\t\t// need to copy the combination because it will be changed by caller\n\t\tpksFound = append(pksFound, append([]int{}, combination...))\n\t})\n\treturn pksFound\n}\n\n// StringToValue takes a piece of data as a string and attempts to convert it to a types.Value of the appropriate types.NomsKind.\nfunc StringToValue(s string, k types.NomsKind) (types.Value, error) {\n\tswitch k {\n\tcase types.NumberKind:\n\t\tif s == \"\" {\n\t\t\treturn types.Number(float64(0)), nil\n\t\t}\n\t\tfval, err := strconv.ParseFloat(s, 64)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"Could not parse '%s' into number (%s)\", s, err)\n\t\t}\n\t\treturn types.Number(fval), nil\n\tcase types.BoolKind:\n\t\t// TODO: This should probably be configurable.\n\t\tswitch s {\n\t\tcase \"true\", \"1\", \"y\", \"yes\", \"Y\", \"YES\":\n\t\t\treturn types.Bool(true), nil\n\t\tcase \"false\", \"0\", \"n\", \"no\", \"N\", \"NO\", \"\":\n\t\t\treturn types.Bool(false), nil\n\t\tdefault:\n\t\t\treturn nil, fmt.Errorf(\"Could not parse '%s' into bool\", s)\n\t\t}\n\tcase types.StringKind:\n\t\treturn types.String(s), nil\n\tdefault:\n\t\td.Panic(\"Invalid column type kind:\" + types.KindToString[k])\n\t}\n\tpanic(\"not reached\")\n}\n" }, { "path": "samples/go/csv/schema_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"fmt\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestSchemaDetection(t *testing.T) {\n\tassert := assert.New(t)\n\ttest := func(input [][]string, expect []KindSlice) {\n\t\toptions := newSchemaOptions(len(input[0]))\n\t\tfor _, values := range input {\n\t\t\toptions.Test(values)\n\t\t}\n\n\t\tassert.Equal(expect, options.ValidKinds())\n\t}\n\ttest(\n\t\t[][]string{\n\t\t\t{\"foo\", \"1\", \"5\"},\n\t\t\t{\"bar\", \"0\", \"10\"},\n\t\t\t{\"true\", \"1\", \"23\"},\n\t\t\t{\"1\", \"1\", \"60\"},\n\t\t\t{\"1.1\", \"false\", \"75\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{types.StringKind},\n\t\t\t{types.BoolKind, types.StringKind},\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind,\n\t\t\t},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"foo\"},\n\t\t\t{\"bar\"},\n\t\t\t{\"true\"},\n\t\t\t{\"1\"},\n\t\t\t{\"1.1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{types.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"true\"},\n\t\t\t{\"1\"},\n\t\t\t{\"1.1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{types.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"true\"},\n\t\t\t{\"false\"},\n\t\t\t{\"True\"},\n\t\t\t{\"False\"},\n\t\t\t{\"TRUE\"},\n\t\t\t{\"FALSE\"},\n\t\t\t{\"1\"},\n\t\t\t{\"0\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{types.BoolKind, types.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"1.1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"1.1\"},\n\t\t\t{\"4.940656458412465441765687928682213723651e-50\"},\n\t\t\t{\"-4.940656458412465441765687928682213723651e-50\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"1.1\"},\n\t\t\t{\"1.797693134862315708145274237317043567981e+102\"},\n\t\t\t{\"-1.797693134862315708145274237317043567981e+102\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"1.1\"},\n\t\t\t{\"1.797693134862315708145274237317043567981e+309\"},\n\t\t\t{\"-1.797693134862315708145274237317043567981e+309\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"0\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.BoolKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"0\"},\n\t\t\t{\"-1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"0\"},\n\t\t\t{\"-0\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"280\"},\n\t\t\t{\"0\"},\n\t\t\t{\"-1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"-180\"},\n\t\t\t{\"0\"},\n\t\t\t{\"-1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"33000\"},\n\t\t\t{\"0\"},\n\t\t\t{\"-1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"-44000\"},\n\t\t\t{\"0\"},\n\t\t\t{\"-1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"2547483648\"},\n\t\t\t{\"0\"},\n\t\t\t{\"-1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{\"1\"},\n\t\t\t{\"-4347483648\"},\n\t\t\t{\"0\"},\n\t\t\t{\"-1\"},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{fmt.Sprintf(\"%d\", uint64(1<<63))},\n\t\t\t{fmt.Sprintf(\"%d\", uint64(1<<63)+1)},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n\ttest(\n\t\t[][]string{\n\t\t\t{fmt.Sprintf(\"%d\", uint64(1<<32))},\n\t\t\t{fmt.Sprintf(\"%d\", uint64(1<<32)+1)},\n\t\t},\n\t\t[]KindSlice{\n\t\t\t{\n\t\t\t\ttypes.NumberKind,\n\t\t\t\ttypes.StringKind},\n\t\t},\n\t)\n}\n\nfunc TestCombinationsWithLength(t *testing.T) {\n\tassert := assert.New(t)\n\ttest := func(input []int, length int, expect [][]int) {\n\t\tcombinations := make([][]int, 0)\n\t\tcombinationsWithLength(input, length, func(combination []int) {\n\t\t\tcombinations = append(combinations, append([]int{}, combination...))\n\t\t})\n\n\t\tassert.Equal(expect, combinations)\n\t}\n\ttest([]int{0}, 1, [][]int{\n\t\t{0},\n\t})\n\ttest([]int{1}, 1, [][]int{\n\t\t{1},\n\t})\n\ttest([]int{0, 1}, 1, [][]int{\n\t\t{0},\n\t\t{1},\n\t})\n\ttest([]int{0, 1}, 2, [][]int{\n\t\t{0, 1},\n\t})\n\ttest([]int{70, 80, 90, 100}, 1, [][]int{\n\t\t{70},\n\t\t{80},\n\t\t{90},\n\t\t{100},\n\t})\n\ttest([]int{70, 80, 90, 100}, 2, [][]int{\n\t\t{70, 80},\n\t\t{70, 90},\n\t\t{70, 100},\n\t\t{80, 90},\n\t\t{80, 100},\n\t\t{90, 100},\n\t})\n\ttest([]int{70, 80, 90, 100}, 3, [][]int{\n\t\t{70, 80, 90},\n\t\t{70, 80, 100},\n\t\t{70, 90, 100},\n\t\t{80, 90, 100},\n\t})\n}\n\nfunc TestCombinationsWithLengthFromTo(t *testing.T) {\n\tassert := assert.New(t)\n\ttest := func(input []int, smallestLength, largestLength int, expect [][]int) {\n\t\tcombinations := make([][]int, 0)\n\t\tcombinationsLengthsFromTo(input, smallestLength, largestLength, func(combination []int) {\n\t\t\tcombinations = append(combinations, append([]int{}, combination...))\n\t\t})\n\n\t\tassert.Equal(expect, combinations)\n\t}\n\ttest([]int{0}, 1, 1, [][]int{\n\t\t{0},\n\t})\n\ttest([]int{1}, 1, 1, [][]int{\n\t\t{1},\n\t})\n\ttest([]int{0, 1}, 1, 2, [][]int{\n\t\t{0},\n\t\t{1},\n\t\t{0, 1},\n\t})\n\ttest([]int{0, 1}, 2, 2, [][]int{\n\t\t{0, 1},\n\t})\n\ttest([]int{70, 80, 90, 100}, 1, 3, [][]int{\n\t\t{70},\n\t\t{80},\n\t\t{90},\n\t\t{100},\n\t\t{70, 80},\n\t\t{70, 90},\n\t\t{70, 100},\n\t\t{80, 90},\n\t\t{80, 100},\n\t\t{90, 100},\n\t\t{70, 80, 90},\n\t\t{70, 80, 100},\n\t\t{70, 90, 100},\n\t\t{80, 90, 100},\n\t})\n}\n" }, { "path": "samples/go/csv/write.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"encoding/csv\"\n\t\"fmt\"\n\t\"io\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\nfunc getElemDesc(s types.Collection, index int) types.StructDesc {\n\tt := types.TypeOf(s).Desc.(types.CompoundDesc).ElemTypes[index]\n\tif types.StructKind != t.TargetKind() {\n\t\td.Panic(\"Expected StructKind, found %s\", t.Kind())\n\t}\n\treturn t.Desc.(types.StructDesc)\n}\n\n// GetListElemDesc ensures that l is a types.List of structs, pulls the types.StructDesc that describes the elements of l out of vr, and returns the StructDesc.\nfunc GetListElemDesc(l types.List, vr types.ValueReader) types.StructDesc {\n\treturn getElemDesc(l, 0)\n}\n\n// GetMapElemDesc ensures that m is a types.Map of structs, pulls the types.StructDesc that describes the elements of m out of vr, and returns the StructDesc.\n// If m is a nested types.Map of types.Map, then GetMapElemDesc will descend the levels of the enclosed types.Maps to get to a types.Struct\nfunc GetMapElemDesc(m types.Map, vr types.ValueReader) types.StructDesc {\n\tt := types.TypeOf(m).Desc.(types.CompoundDesc).ElemTypes[1]\n\tif types.StructKind == t.TargetKind() {\n\t\treturn t.Desc.(types.StructDesc)\n\t} else if types.MapKind == t.TargetKind() {\n\t\t_, v := m.First()\n\t\treturn GetMapElemDesc(v.(types.Map), vr)\n\t}\n\tpanic(fmt.Sprintf(\"Expected StructKind or MapKind, found %s\", t.Kind().String()))\n}\n\nfunc writeValuesFromChan(structChan chan types.Struct, sd types.StructDesc, comma rune, output io.Writer) {\n\tfieldNames := getFieldNamesFromStruct(sd)\n\tcsvWriter := csv.NewWriter(output)\n\tcsvWriter.Comma = comma\n\tif csvWriter.Write(fieldNames) != nil {\n\t\td.Panic(\"Failed to write header %v\", fieldNames)\n\t}\n\trecord := make([]string, len(fieldNames))\n\tfor s := range structChan {\n\t\ti := 0\n\t\ts.WalkValues(func(v types.Value) {\n\t\t\trecord[i] = fmt.Sprintf(\"%v\", v)\n\t\t\ti++\n\t\t})\n\t\tif csvWriter.Write(record) != nil {\n\t\t\td.Panic(\"Failed to write record %v\", record)\n\t\t}\n\t}\n\n\tcsvWriter.Flush()\n\tif csvWriter.Error() != nil {\n\t\td.Panic(\"error flushing csv\")\n\t}\n}\n\n// WriteList takes a types.List l of structs (described by sd) and writes it to output as comma-delineated values.\nfunc WriteList(l types.List, sd types.StructDesc, comma rune, output io.Writer) {\n\tstructChan := make(chan types.Struct, 1024)\n\tgo func() {\n\t\tl.IterAll(func(v types.Value, index uint64) {\n\t\t\tstructChan <- v.(types.Struct)\n\t\t})\n\t\tclose(structChan)\n\t}()\n\twriteValuesFromChan(structChan, sd, comma, output)\n}\n\nfunc sendMapValuesToChan(m types.Map, structChan chan<- types.Struct) {\n\tm.IterAll(func(k, v types.Value) {\n\t\tif subMap, ok := v.(types.Map); ok {\n\t\t\tsendMapValuesToChan(subMap, structChan)\n\t\t} else {\n\t\t\tstructChan <- v.(types.Struct)\n\t\t}\n\t})\n}\n\n// WriteMap takes a types.Map m of structs (described by sd) and writes it to output as comma-delineated values.\nfunc WriteMap(m types.Map, sd types.StructDesc, comma rune, output io.Writer) {\n\tstructChan := make(chan types.Struct, 1024)\n\tgo func() {\n\t\tsendMapValuesToChan(m, structChan)\n\t\tclose(structChan)\n\t}()\n\twriteValuesFromChan(structChan, sd, comma, output)\n}\n\nfunc getFieldNamesFromStruct(structDesc types.StructDesc) (fieldNames []string) {\n\tstructDesc.IterFields(func(name string, t *types.Type, optional bool) {\n\t\tif !types.IsPrimitiveKind(t.TargetKind()) {\n\t\t\td.Panic(\"Expected primitive kind, found %s\", t.TargetKind().String())\n\t\t}\n\t\tfieldNames = append(fieldNames, name)\n\t})\n\treturn\n}\n" }, { "path": "samples/go/csv/write_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage csv\n\nimport (\n\t\"bytes\"\n\t\"encoding/csv\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/clienttest\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst (\n\tTEST_ROW_STRUCT_NAME = \"row\"\n\tTEST_ROW_FIELDS = \"anid,month,rainfall,year\"\n\tTEST_DATA_SIZE = 200\n\tTEST_YEAR = 2012\n)\n\nfunc TestCSVWrite(t *testing.T) {\n\tsuite.Run(t, &csvWriteTestSuite{})\n}\n\ntype csvWriteTestSuite struct {\n\tclienttest.ClientTestSuite\n\tfieldTypes []*types.Type\n\trowStructDesc types.StructDesc\n\tcomma rune\n\ttmpFileName string\n}\n\nfunc typesToKinds(ts []*types.Type) KindSlice {\n\tkinds := make(KindSlice, len(ts))\n\tfor i, t := range ts {\n\t\tkinds[i] = t.TargetKind()\n\t}\n\treturn kinds\n}\n\nfunc (s *csvWriteTestSuite) SetupTest() {\n\tinput, err := ioutil.TempFile(s.TempDir, \"\")\n\td.Chk.NoError(err)\n\ts.tmpFileName = input.Name()\n\tdefer input.Close()\n\n\tfieldNames := strings.Split(TEST_ROW_FIELDS, \",\")\n\ts.fieldTypes = []*types.Type{types.StringType, types.NumberType, types.NumberType, types.NumberType}\n\tfields := make([]types.StructField, len(fieldNames))\n\tfor i, name := range fieldNames {\n\t\tfields[i] = types.StructField{\n\t\t\tName: name,\n\t\t\tType: s.fieldTypes[i],\n\t\t}\n\t}\n\trowStructType := types.MakeStructType(TEST_ROW_STRUCT_NAME, fields...)\n\ts.rowStructDesc = rowStructType.Desc.(types.StructDesc)\n\ts.comma, _ = StringToRune(\",\")\n\tcreateCsvTestExpectationFile(input)\n}\n\nfunc (s *csvWriteTestSuite) TearDownTest() {\n\tos.Remove(s.tmpFileName)\n}\n\nfunc createCsvTestExpectationFile(w io.Writer) {\n\t_, err := io.WriteString(w, TEST_ROW_FIELDS)\n\td.Chk.NoError(err)\n\t_, err = io.WriteString(w, \"\\n\")\n\td.Chk.NoError(err)\n\tfor i := 0; i < TEST_DATA_SIZE; i++ {\n\t\t_, err = io.WriteString(w, fmt.Sprintf(\"a - %3d,%d,%d,%d\\n\", i, i%12, i%32, TEST_YEAR+i%4))\n\t\td.Chk.NoError(err)\n\t}\n}\n\nfunc startReadingCsvTestExpectationFile(s *csvWriteTestSuite) (cr *csv.Reader, headers []string) {\n\tres, err := os.Open(s.tmpFileName)\n\td.PanicIfError(err)\n\tcr = NewCSVReader(res, s.comma)\n\theaders, err = cr.Read()\n\td.PanicIfError(err)\n\treturn\n}\n\nfunc createTestList(s *csvWriteTestSuite) types.List {\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\tcr, headers := startReadingCsvTestExpectationFile(s)\n\tl := ReadToList(cr, TEST_ROW_STRUCT_NAME, headers, typesToKinds(s.fieldTypes), db, LIMIT)\n\treturn l\n}\n\nfunc createTestMap(s *csvWriteTestSuite) types.Map {\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\tcr, headers := startReadingCsvTestExpectationFile(s)\n\treturn ReadToMap(cr, TEST_ROW_STRUCT_NAME, headers, []string{\"anid\"}, typesToKinds(s.fieldTypes), db, LIMIT)\n}\n\nfunc createTestNestedMap(s *csvWriteTestSuite) types.Map {\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\tcr, headers := startReadingCsvTestExpectationFile(s)\n\treturn ReadToMap(cr, TEST_ROW_STRUCT_NAME, headers, []string{\"anid\", \"year\"}, typesToKinds(s.fieldTypes), db, LIMIT)\n}\n\nfunc verifyOutput(s *csvWriteTestSuite, r io.Reader) {\n\tres, err := os.Open(s.tmpFileName)\n\td.PanicIfError(err)\n\tactual, err := ioutil.ReadAll(r)\n\td.Chk.NoError(err)\n\texpected, err := ioutil.ReadAll(res)\n\td.Chk.NoError(err)\n\ts.True(string(expected) == string(actual), \"csv files are different\")\n}\n\nfunc (s *csvWriteTestSuite) TestCSVWriteList() {\n\tl := createTestList(s)\n\tw := new(bytes.Buffer)\n\ts.True(TEST_DATA_SIZE == l.Len(), \"list length\")\n\tWriteList(l, s.rowStructDesc, s.comma, w)\n\tverifyOutput(s, w)\n}\n\nfunc (s *csvWriteTestSuite) TestCSVWriteMap() {\n\tm := createTestMap(s)\n\tw := new(bytes.Buffer)\n\ts.True(TEST_DATA_SIZE == m.Len(), \"map length\")\n\tWriteMap(m, s.rowStructDesc, s.comma, w)\n\tverifyOutput(s, w)\n}\n\nfunc (s *csvWriteTestSuite) TestCSVWriteNestedMap() {\n\tm := createTestNestedMap(s)\n\tw := new(bytes.Buffer)\n\ts.True(TEST_DATA_SIZE == m.Len(), \"nested map length\")\n\tWriteMap(m, s.rowStructDesc, s.comma, w)\n\tverifyOutput(s, w)\n}\n" }, { "path": "samples/go/decent/README.md", "content": "# About\n\nThis directory contains two sample applications that demonstrate using Noms in a decentralized environment.\n\nBoth applications implement multiuser chat, using different strategies.\n\n`p2p-chat` is the simplest possible example: a fully local noms replica is run on each node, and all nodes synchronize continuously with each other over HTTP.\n\n`ipfs-chat` backs Noms with the [IPFS](https://ipfs.io/) network, so that nodes don't have to keep a full local replica of all data. However, because [Filecoin](http://filecoin.io/) doesn't yet exist, *some node* does have to keep a full replica, so ipfs-chat has a `daemon` mode so that you can run a persistent node somewhere to be the replica of last resort.\n" }, { "path": "samples/go/decent/data/godfather.html", "content": "\n\n
\n
\n\n\tTHE GODFATHER\n\t_____________\n\n\tScreenplay\n\n\tby\n\n\tMARIO PUZO\n\n\tand\n\n\tFRANCIS FORD COPPOLA\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nTHIRD DRAFT\t\t\t\tPARAMOUNT PICTURES\n\t\t\t\t\t1 Gulf and Western Plaza\nMarch 29, 1971\t\t\t\tNew York, New York 10019\n\n\n\n\n\n\tINT DAY: DON'S OFFICE (SUMMER 1945)\n\n\tThe PARAMOUNT Logo is presented austerely over a black\n\tbackground.  There is a moment's hesitation, and then the\n\tsimple words in white lettering:\n\n\t\t\t\t  THE GODFATHER\n\n\tWhile this remains, we hear: \"I believe in America.\"\n\tSuddenly we are watching in CLOSE VIEW, AMERIGO BONASERA, a\n\tman of sixty, dressed in a black suit, on the verge of great\n\temotion.\n\n\t\t\t\tBONASERA\n\t\tAmerica has made my fortune.\n\n\tAs he speaks, THE VIEW imperceptibly begins to loosen.\n\n\t\t\t\tBONASERA\n\t\tI raised my daughter in the American\n\t\tfashion; I gave her freedom, but\n\t\ttaught her never to dishonor her\n\t\tfamily.  She found a boy friend,\n\t\tnot an Italian.  She went to the\n\t\tmovies with him, stayed out late.\n\t\tTwo months ago he took her for a\n\t\tdrive, with another boy friend.\n\t\tThey made her drink whiskey and\n\t\tthen they tried to take advantage\n\t\tof her.  She resisted; she kept her\n\t\thonor.  So they beat her like an\n\t\tanimal.  When I went to the hospital\n\t\ther nose was broken, her jaw was\n\t\tshattered and held together by\n\t\twire, and she could not even weep\n\t\tbecause of the pain.\n\n\tHe can barely speak; he is weeping now.\n\n\t\t\t\tBONASERA\n\t\tI went to the Police like a good\n\t\tAmerican.  These two boys were\n\t\tarrested and brought to trial.  The\n\t\tjudge sentenced them to three years\n\t\tin prison, and suspended the\n\t\tsentence.  Suspended sentence!\n\t\tThey went free that very day.  I\n\t\tstood in the courtroom like a fool,\n\t\tand those bastards, they smiled at\n\t\tme.  Then I said to my wife, for\n\t\tJustice, we must go to The Godfather.\n\n\tBy now, THE VIEW is full, and we see Don Corleone's office\n\tin his home.\n\n\tThe blinds are closed, and so the room is dark, and with\n\tpatterned shadows.  We are watching BONASERA over the\n\tshoulder of DON CORLEONE.  TOM HAGEN sits near a small\n\ttable, examining some paperwork, and SONNY CORLEONE stands\n\timpatiently by the window nearest his father, sipping from a\n\tglass of wine.  We can HEAR music, and the laughter and\n\tvoices of many people outside.\n\n\t\t\t\tDON CORLEONE\n\t\tBonasera, we know each other for\n\t\tyears, but this is the first time\n\t\tyou come to me for help.  I don't\n\t\tremember the last time you invited\n\t\tme to your house for coffee...even\n\t\tthough our wives are friends.\n\n\t\t\t\tBONASERA\n\t\tWhat do you want of me?  I'll give\n\t\tyou anything you want, but do what\n\t\tI ask!\n\n\t\t\t\tDON CORLEONE\n\t\tAnd what is that Bonasera?\n\n\tBONASERA whispers into the DON's ear.\n\n\t\t\t\tDON CORLEONE\n\t\tNo.  You ask for too much.\n\n\t\t\t\tBONASERA\n\t\tI ask for Justice.\n\n\t\t\t\tDON CORLEONE\n\t\tThe Court gave you justice.\n\n\t\t\t\tBONASERA\n\t\tAn eye for an eye!\n\n\t\t\t\tDON CORLEONE\n\t\tBut your daughter is still alive.\n\n\t\t\t\tBONASERA\n\t\tThen make them suffer as she\n\t\tsuffers.  How much shall I pay you.\n\n\tBoth HAGEN and SONNY react.\n\n\t\t\t\tDON CORLEONE\n\t\tYou never think to protect yourself\n\t\twith real friends.  You think it's\n\t\tenough to be an American.  All\n\t\tright, the Police protects you,\n\t\tthere are Courts of Law, so you\n\t\tdon't need a friend like me.\n\t\tBut now you come to me and say Don\n\t\tCorleone, you must give me justice.\n\t\tAnd you don't ask in respect or\n\t\tfriendship.  And you don't think to\n\t\tcall me Godfather; instead you come\n\t\tto my house on the day my daughter\n\t\tis to be married and you ask me to\n\t\tdo murder...for money.\n\n\t\t\t\tBONASERA\n\t\tAmerica has been good to me...\n\n\t\t\t\tDON CORLEONE\n\t\tThen take the justice from the\n\t\tjudge, the bitter with the sweet,\n\t\tBonasera.  But if you come to me\n\t\twith your friendship, your loyalty,\n\t\tthen your enemies become my enemies,\n\t\tand then, believe me, they would\n\t\tfear you...\n\n\tSlowly, Bonasera bows his head and murmurs.\n\n\t\t\t\tBONASERA\n\t\tBe my friend.\n\n\t\t\t\tDON CORLEONE\n\t\tGood.  From me you'll get Justice.\n\n\t\t\t\tBONASERA\n\t\tGodfather.\n\n\t\t\t\tDON CORLEONE\n\t\tSome day, and that day may never\n\t\tcome, I would like to call upon you\n\t\tto do me a service in return.\n\n\tEXT DAY: MALL (SUMMER 1945)\n\n\tA HIGH ANGLE of the CORLEONE MALL in bright daylight.  There\n\tare at least five hundred guests filling the main courtyard\n\tand gardens.  There is music and laughing and dancing and\n\tcountless tables covered with food and wine.\n\n\tDON CORLEONE stands at the Gate, flanked on either side by a\n\tson: FREDO and SONNY, all dressed in the formal attire of\n\tthe wedding party.  He warmly shakes the hands, squeezes the\n\thands of the friends and guests, pinches the cheeks of the\n\tchildren, and makes them all welcome.  They in turn carry\n\twith them gallons of homemade wine, cartons of freshly baked\n\tbread and pastries, and enormous trays of Italian delicacies.\n\n\tThe entire family poses for a family portrait: DON CORLEONE,\n\tMAMA, SONNY, his wife, SANDRA, and their children, TOM HAGEN\n\tand his wife, THERESA, and their BABY; CONSTANZIA, the\n\tbride, and her bridegroom, CARLO RIZZI.  As they move into\n\tthe pose, THE DON seems preoccupied.\n\n\t\t\t\tDON CORLEONE\n\t\tWhere's Michael?\n\n\t\t\t\tSONNY\n\t\tHe'll be here Pop, it's still early.\n\n\t\t\t\tDON CORLEONE\n\t\tThen the picture will wait for him.\n\n\tEveryone in the group feels the uneasiness as the DON moves\n\tback to the house.  SONNY gives a delicious smile in the\n\tdirection of the Maid-of-Honor, LUCY MANCINI.  She returns\n\tit.  Then he moves to his wife.\n\n\t\t\t\tSONNY\n\t\tSandra, watch the kids.  They're\n\t\trunning wild.\n\n\t\t\t\tSANDRA\n\t\tYou watch yourself.\n\n\tHAGEN kisses his WIFE, and follows THE DON, passing the wine\n\tbarrels, where a group of FOUR MEN nervously wait.  TOM\n\tcrooks a finger at NAZORINE, who doublechecks that he is\n\tnext, straightens, and follows HAGEN.\n\n\tEXT DAY: MALL ENTRANCE (SUMMER 1945)\n\n\tOutside the main gate of the Mall, SEVERAL MEN in suits,\n\tworking together with a MAN in a dark sedan, walk in and out\n\tof the rows of parked cars, writing license plate numbers\n\tdown in their notebooks.  We HEAR the music and laughter\n\tcoming from the party in the distance.\n\n\tA MAN stops at a limousine and copies down the number.\n\n\tBARZINI, dignified in a black homburg, is always under the\n\twatchful eyes of TWO BODYGUARDS as he makes his way to\n\tembrace DON CORLEONE in the courtyard.\n\n\tThe MEN walk down another row of parked cars.  Put another\n\tnumber in the notebook.  A shiney new Cadillac with wooden\n\tbumpers.\n\n\tPETER CLEMENZA, dancing the Tarantella joyously, bumping\n\tbellies with the ladies.\n\n\t\t\t\tCLEMENZA\n\t\tPaulie...wine...WINE.\n\n\tHe mops his sweating forehead with a big handkerchief.\n\tPAULIE hustles, gets a glass of icy black wine, and brings\n\tit to him.\n\n\t\t\t\tPAULIE\n\t\tYou look terrif on the floor!\n\n\t\t\t\tCLEMENZA\n\t\tWhat are you, a dance judge?  Go do\n\t\tyour job; take a walk around the\n\t\tneighborhood... see everything is\n\t\tokay.\n\n\tPAULIE nods and leaves; CLEMENZA takes a breath, and leaps\n\tback into the dance.\n\n\tThe MEN walk down another row of parked cars.  Put another\n\tnumber in the notebook.\n\n\tTESSIO, a tall, gentle-looking man, dances with a NINE-YEAR-\n\tOLD GIRL, her little black party shoes planted on his\n\tenormous brown shoes.\n\n\tThe MEN move on to other parked cars, when SONNY storms out\n\tof the gate, his face flushed with anger, followed by\n\tCLEMENZA and PAULIE.\n\n\t\t\t\tSONNY\n\t\tBuddy, this is a private party.\n\n\tThe MAN doesn't answer, but points to the DRIVER of the\n\tsedan.  SONNY menacingly thrusts his reddened face at him.\n\tThe DRIVER merely flips open his wallet to a greed card,\n\twithout saying a word.  SONNY steps back, spits on the\n\tground, turns, and walks away, followed by CLEMENZA, PAULIE,\n\tand another TWO MEN.  He doesn't say a thing for most of the\n\twalk back into the courtyard, and then, muttered to PAULIE.\n\n\t\t\t\tSONNY\n\t\tGoddamn FBI...don't respect nothing.\n\n\tINT DAY: DON'S OFFICE (SUMMER 1945)\n\n\tDON CORLEONE sits quietly behind his massive desk in the\n\tdark study.\n\n\t\t\t\tNAZORINE\n\t\t...a fine boy from Sicily, captured\n\t\tby the American Army, and sent to\n\t\tNew Jersey as a prisoner of war...\n\n\t\t\t\tDON CORLEONE\n\t\tNazorine, my friend, tell me what I\n\t\tcan do.\n\n\t\t\t\tNAZORINE\n\t\tNow that the war is over, Enzo,\n\t\tthis boy is being repatriated to\n\t\tItaly.  And you see, Godfather...\n\t\t\t  (he wrings his hands,\n\t\t\t  unable to express himself)\n\t\tHe...my daughter...they...\n\n\t\t\t\tDON CORLEONE\n\t\tYou want him to stay in this country.\n\n\t\t\t\tNAZORINE\n\t\tGodfather, you understand everything.\n\n\t\t\t\tDON CORLEONE\n\t\tTom, what we need is an Act of\n\t\tCongress to allow Enzo to become a\n\t\tcitizen.\n\n\t\t\t\tNAZORINE\n\t\t\t  (impressed)\n\t\tAn Act of Congress!\n\n\t\t\t\tHAGEN\n\t\t\t  (nodding)\n\t\tIt will cost.\n\n\tThe DON shrugs; such are the way with those things; NAZORINE\n\tnods.\n\n\t\t\t\tNAZORINE\n\t\tIs that all?  Godfather, thank\n\t\tyou...\n\t\t\t  (backing out, enthusiastically)\n\t\tOh, wait till you see the cake I\n\t\tmade for your beautiful daughter!\n\n\tNAZORINE backs out, all smiles, and nods to the GODFATHER.\n\tDON CORLEONE rises and moves to the Venetian blinds.\n\n\t\t\t\tHAGEN\n\t\tWho do I give this job to?\n\n\tThe DON moves to the windows, peeking out through the blinds.\n\n\t\t\t\tDON CORLEONE\n\t\tNot to one of our paisans...give it\n\t\tto a Jew Congressman in another\n\t\tdistrict.  Who else is on the list\n\t\tfor today?\n\n\tThe DON is peeking out to the MEN around the barrel, waiting\n\tto see him.\n\n\t\t\t\tHAGEN\n\t\tFrancesco Nippi.  His nephew has\n\t\tbeen refused parole.  A bad case.\n\n\tEXT DAY: MALL (SUMMER 1945)\n\n\tWHAT HE SEES:\n\n\tNIPPI waits nervously by the barrel.\n\n\t\t\t\tHAGEN (O.S.)\n\t\tHis father worked with you in the\n\t\tfreight yards when you were young.\n\n\tLUCA BRASI sitting alone, grotesque and quiet.\n\n\t\t\t\tHAGEN (O.S.)\n\t\tHe's not on the list, but Luca\n\t\tBrasi wants to see you.\n\n\tINT DAY: DON'S OFFICE (SUMMER 1945)\n\n\tThe DON turns to HAGEN.\n\n\t\t\t\tDON CORLEONE\n\t\tIs it necessary?\n\n\t\t\t\tHAGEN\n\t\tYou understand him better than\n\t\tanyone.\n\n\tThe DON nods to this.  Turns back to the blinds and peeks out.\n\n\tEXT DAY: MALL (SUMMER 1945)\n\n\tWHAT HE SEES:\n\n\tMICHAEL CORLEONE, dressed in the uniform of a Marine Captain,\n\tleads KAY ADAMS through the wedding crowd, occasionally\n\tstopped and greeted by FRIENDS of the family.\n\n\tINT DAY: DON'S OFFICE (SUMMER 1945)\n\n\tThe DON, inside the office, peering through the blinds,\n\tfollowing them.\n\n\tEXT DAY: MALL (SUMMER 1945)\n\n\tMICHAEL moves through the crowd, embraces MAMA and introduces\n\ther to his GIRL.\n\n\tEXT DAY: OFFICE WINDOW (SUMMER 1945)\n\n\tThe DON's eyes peering through the blinds.\n\n\tEXT DAY: MALL TABLES (SUMMER 1945)\n\n\tKAY and MICHAEL settle by a table on the edge of the wedding,\n\tburdened down with plates of food and glasses and wine.  She\n\tis exhilarated by the enormity of the affair, the music and\n\tthe vitality.\n\n\t\t\t\tKAY\n\t\tI've never seen anything like it.\n\n\t\t\t\tMICHAEL\n\t\tI told you I had a lot of relatives.\n\n\tKAY looking about, a young and lively thing in a gift shop.\n\tWe see what she sees:\n\n\tHer interest is caught by THREE MEN standing by the wine\n\tbarrels.\n\n\t\t\t\tKAY\n\t\t\t  (amused)\n\t\tMichael, what are those men doing?\n\n\t\t\t\tMICHAEL\n\t\tThey're waiting to see my father.\n\n\t\t\t\tKAY\n\t\tThey're talking to themselves.\n\n\t\t\t\tMICHAEL\n\t\tThey're going to talk to my father,\n\t\twhich means they're going to ask\n\t\thim for something, which means they\n\t\tbetter get it right.\n\n\t\t\t\tKAY\n\t\tWhy do they bother him on a day\n\t\tlike this?\n\n\t\t\t\tMICHAEL\n\t\tBecause they know that no Sicilian\n\t\twill refuse a request on his\n\t\tdaughter's wedding day.\n\n\tEXT DAY: WEDDING PARTY (SUMMER 1945)\n\n\tCONNIE CORLEONE, the Bride, is pressing the bodice of her\n\toverly-fluffy white gown against the groom, CARLO RIZZI.  He\n\tis bronzed, with curly blondish hair and lovely dimples.\n\tShe absolutely adores him and can barely take her eyes from\n\thim long enough to thank the various GUESTS for the white\n\tenvelopes they are putting into the large white purse she\n\tholds.  In fact, if we watch carefully, we can see that one\n\tof her hands is slid under his jacket, and into his shirt,\n\twhere she is provocatively rubbing the hair on his chest.\n\tCARLO, on the other hand, has his blue eyes trained on the\n\tbulging envelopes, and is trying to guess how much cash the\n\tthings hold.\n\n\tDiscreetly, he moves her hand off of his skin.\n\n\t\t\t\tCARLO\n\t\t\t  (whispered)\n\t\tCut it out, Connie.\n\n\tThe purse, looped by a ribbon of silk around CONNIE's arm,\n\tis fat with money.\n\n\t\t\t\tPAULIE (O.S.)\n\t\tWhat do you think?  Twenty grand?\n\n\tA little distance away, a young man, PAULIE GATTO, catches a\n\tprosciutto sandwich thrown by a friend, without once taking\n\teyes from the purse.\n\n\t\t\t\tPAULIE\n\t\tWho knows?  Maybe more.  Twenty,\n\t\tthirty grand in small bills cash in\n\t\tthat silk purse.  Holy Toledo, if\n\t\tthis was somebody else's wedding!\n\n\tSONNY is sitting at the Wedding Dias, talking to LUCY\n\tMANCINI, the Maid of Honor.  Every once in a while he\n\tglances across the courtyard, where his WIFE is talking with\n\tsome WOMEN.\n\n\tHe bends over and whispers something into LUCY's ear.\n\n\tSANDRA and the WOMEN are in the middle of a big, ribald laugh.\n\n\t\t\t\tWOMAN\n\t\tIs it true what they say about your\n\t\thusband, Sandra?\n\n\tSANDRA's hands separate with expanding width further and\n\tfurther apart until she bursts into a peal of laughter.\n\tThrough her separated hands she sees the Wedding Dais.\n\tSONNY and LUCY are gone.\n\n\tINT DAY: DON'S HALL & STAIRS (SUMMER 1945)\n\n\tThe empty hallway.  The bathroom door opens and LUCY\n\tsurreptitiously steps out.\n\n\tShe looks up where SONNY is standing on the second landing,\n\tmotioning for her to come up.\n\n\tShe lifts her petticoats off the ground and hurries upstairs.\n\n\tEXT DAY: MALL TABLES (SUMMER 1945)\n\n\tKAY and MICHAEL.\n\n\t\t\t\tKAY\n\t\t\t  (in a spooky low tone)\n\t\tMichael, that scarey guy...Is he a\n\t\trelative?\n\n\tShe has picked out LUCA BRASI.\n\n\t\t\t\tMICHAEL\n\t\tNo.  His name is Luca Brasi.  You\n\t\twouldn't like him.\n\n\t\t\t\tKAY\n\t\t\t  (Excited)\n\t\tWho is he?\n\n\t\t\t\tMICHAEL\n\t\t\t  (Sizing her up)\n\t\tYou really want to know?\n\n\t\t\t\tKAY\n\t\tYes.  Tell me.\n\n\t\t\t\tMICHAEL\n\t\tYou like spaghetti?\n\n\t\t\t\tKAY\n\t\tYou know I love spaghetti.\n\n\t\t\t\tMICHAEL\n\t\tThen eat your spaghetti and I'll\n\t\ttell you a Luca Brasi story.\n\n\tShe starts to eat her spaghetti.\n\n\tShe begins eating, looking at him eagerly.\n\n\t\t\t\tMICHAEL\n\t\tOnce upon a time, about fifteen\n\t\tyears ago some people wanted to\n\t\ttake over my father's olive oil\n\t\tbusiness.  They had Al Capone send\n\t\tsome men in from Chicago to kill my\n\t\tfather, and they almost did.\n\n\t\t\t\tKAY\n\t\tAl Capone!\n\n\t\t\t\tMICHAEL\n\t\tMy Father sent Luca Brasi after\n\t\tthem.  He tied the two Capone men\n\t\thand and foot, and stuffed small\n\t\tbath towels into their mouths.\n\t\tThen he took an ax, and chopped one\n\t\tman's feet off...\n\n\t\t\t\tKAY\n\t\tMichael...\n\n\t\t\t\tMICHAEL\n\t\tThen the legs at the knees...\n\n\t\t\t\tKAY\n\t\tMichael you're trying to scare me...\n\n\t\t\t\tMICHAEL\n\t\tThen the thighs where they joined\n\t\tthe torso.\n\n\t\t\t\tKAY\n\t\tMichael, I don't want to hear\n\t\tanymore...\n\n\t\t\t\tMICHAEL\n\t\tThen Luca turned to the other man...\n\n\t\t\t\tKAY\n\t\tMichael, I love you.\n\n\t\t\t\tMICHAEL\n\t\t...who out of sheer terror had\n\t\tswallowed the bath towel in his\n\t\tmouth and suffocated.\n\n\tThe smile on his face seems to indicate that he is telling a\n\ttall story.\n\n\t\t\t\tKAY\n\t\tI never know when you're telling me\n\t\tthe truth.\n\n\t\t\t\tMICHAEL\n\t\tI told you you wouldn't like him.\n\n\t\t\t\tKAY\n\t\tHe's coming over here!\n\n\tLUCA comes toward them to meet TOM HAGEN halfway, just near\n\ttheir table.\n\n\t\t\t\tMICHAEL\n\t\tTom...Tom, I'd like you to meet Kay\n\t\tAdams.\n\n\t\t\t\tKAY\n\t\t\t  (having survived LUCA)\n\t\tHow do you do.\n\n\t\t\t\tMICHAEL\n\t\tMy brother, Tom Hagen.\n\n\t\t\t\tHAGEN\n\t\tHello Kay.  Your father's inside,\n\t\tdoing some business.\n\t\t\t  (privately)\n\t\tHe's been asking for you.\n\n\t\t\t\tMICHAEL\n\t\tThanks Tom.\n\n\tHAGEN smiles and moves back to the house, LUCA ominously\n\tfollowing.\n\n\t\t\t\tKAY\n\t\tIf he's your brother, why does he\n\t\thave a different name?\n\n\t\t\t\tMICHAEL\n\t\tMy brother Sonny found him living\n\t\tin the streets when he was a kid,\n\t\tso my father took him in.  He's a\n\t\tgood lawyer.\n\n\tINT DAY: DON'S OFFICE (SUMMER 1945)\n\n\tDON CORLEONE at the window.  He has seen the intimacy of the\n\tYOUNG COUPLE.\n\n\t\t\t\tLUCA (O.S.)\n\t\tDon Corleone...\n\n\tTHE DON turns to the stiffly formal LUCA, and he moves\n\tforward to kiss his hand.  He takes the envelope from his\n\tjacket, holds it out, but does not release it until he makes\n\ta formal speech.\n\n\t\t\t\tLUCA\n\t\t\t  (with difficulty)\n\t\tDon Corleone...I am honored, and\n\t\tgrateful...that you invited me to\n\t\tyour home...on the wedding day of\n\t\tyour...daughter.\n\t\tMay their first child...be a\n\t\tmasculine child.  I pledge my never\n\t\tending loyalty.\n\t\t\t  (he offers the envelope)\n\t\tFor your daughter's bridal purse.\n\n\t\t\t\tDON CORLEONE\n\t\tThank you, Luca, my most valued\n\t\tfriend.\n\n\tTHE DON takes it, and then LUCA's hand, which he squeezes so\n\ttightly we might imagine it to be painful.\n\n\t\t\t\tLUCA\n\t\tLet me leave you, Don Corleone.  I\n\t\tknow you are busy.\n\n\tHe turns, almost an about-face, and leaves the study with\n\tthe same formality he entered with.  DON CORLEONE breathes\n\tmore easily, and gives the thick envelope to HAGEN.\n\n\t\t\t\tDON CORLEONE\n\t\tI'm sure it's the most generous\n\t\tgift today.\n\n\t\t\t\tHAGEN\n\t\tThe Senator called--apologized for\n\t\tnot coming personally, but said\n\t\tyou'd understand.  Also, some of\n\t\tthe Judges...they've all sent gifts.\n\t\tAnd another call from Virgil\n\t\tSollozzo.\n\n\tDON CORLEONE is not pleased.\n\n\t\t\t\tHAGEN\n\t\tThe action is narcotics.  Sollozzo\n\t\thas contacts in Turkey for the\n\t\tpoppy, in Sicily for the plants to\n\t\tprocess down to morphine or up to\n\t\theroin.  Also he has access to this\n\t\tcountry.  He's coming to us for\n\t\tfinancial help, and some sort of\n\t\timmunity from the law.  For that we\n\t\tget a piece of the action, I\n\t\tcouldn't find out how much.\n\t\tSollozzo is vouched for by the\n\t\tTattaglia family, and they may have\n\t\ta piece of the action.  They call\n\t\tSollozzo the Turk.\n\t\tHe's spent a lot of time in Turkey\n\t\tand is suppose to have a Turkish\n\t\twife and kids.  He's suppose to be\n\t\tvery quick with the knife, or was,\n\t\twhen he was younger.  Only in\n\t\tmatters of business and with some\n\t\treasonable complaint.  Also he has\n\t\tan American wife and three children\n\t\tand he is a good family man.\n\n\tTHE DON nods.\n\n\t\t\t\tHAGEN\n\t\tHe's his own boss, and very\n\t\tcompetent.\n\n\t\t\t\tDON CORLEONE\n\t\tAnd with prison record.\n\n\t\t\t\tHAGEN\n\t\tTwo terms; one in Italy, one in the\n\t\tUnited States.  He's known to the\n\t\tGovernment as a top narcotics man.\n\t\tThat could be a plus for us; he\n\t\tcould never get immunity to testify.\n\n\t\t\t\tDON CORLEONE\n\t\tWhen did he call?\n\n\t\t\t\tHAGEN\n\t\tThis morning.\n\n\t\t\t\tDON CORLEONE\n\t\tOn a day like this.  Consiglero, do\n\t\tyou also have in your notes the the\n\t\tTurk made his living from\n\t\tProstitution before the war, like\n\t\tthe Tattaglias do now.  Write that\n\t\tdown before you forget it.  The\n\t\tTurk will wait.\n\n\tWe now begin to hear a song coming over the loud-speakers\n\tfrom outside.  In Italian, with unmistakable style.\n\n\t\t\t\tDON CORLEONE\n\t\tWhat that?  It sounds like Johnny.\n\n\tHe moves to the window, pulls the blinds up, flooding the\n\troom with light.\n\n\t\t\t\tDON CORLEONE\n\t\tIt is Johnny.  He came all the way\n\t\tfrom California to be at the wedding.\n\n\t\t\t\tHAGEN\n\t\tShould I bring him in.\n\n\t\t\t\tDON CORLEONE\n\t\tNo.  Let the people enjoy him.  You\n\t\tsee?  He is a good godson.\n\n\t\t\t\tHAGEN\n\t\tIt's been two years.  He's probably\n\t\tin trouble again.\n\n\tEXT DAY: MALL (SUMMER 1945)\n\n\tJOHNNY FONTANE on the bandstand, singing to the delight and\n\texcitement of the wedding GUESTS.\n\n\t\t\t\tKAY\n\t\tI didn't know your family knew\n\t\tJohnny Fontane.\n\n\t\t\t\tMICHAEL\n\t\tSure.\n\n\t\t\t\tKAY\n\t\tI used to come down to New York\n\t\twhenever he sang at the Capitol and\n\t\tscream my head off.\n\n\t\t\t\tMICHAEL\n\t\tHe's my father's godson; he owes\n\t\thim his whole career.\n\n\tJOHNNY finishes the song and the CROWD screams with delight.\n\tThey call out for another when DON CORLEONE appears.\n\n\t\t\t\tDON CORLEONE\n\t\tMy Godson has come three thousand\n\t\tmiles to do us honor and no one\n\t\tthinks to wet his throat.\n\n\tAt once a dozen wine glasses are offered to JOHNNY, who\n\ttakes a sip from each as he moves to embrace his GODFATHER.\n\n\t\t\t\tJOHNNY\n\t\tI kept trying to call you after my\n\t\tdivorce and Tom always said you\n\t\twere busy.  When I got the Wedding\n\t\tinvitation I knew you weren't sore\n\t\tat me anymore, Godfather.\n\n\t\t\t\tDON CORLEONE\n\t\tCan I do something for you still?\n\t\tYou're not too rich, or too famous\n\t\tthat I can't help you?\n\n\t\t\t\tJOHNNY\n\t\tI'm not rich anymore, Godfather,\n\t\tand...my career, I'm almost washed\n\t\tup...\n\n\tHe's very disturbed.  The GODFATHER indicates that he come\n\twith him to the office so no one will notice.  He turns to\n\tHAGEN.\n\n\t\t\t\tDON CORLEONE\n\t\tTell Santino to come in with us.\n\t\tHe should hear some things.\n\n\tThey go, leaving HAGEN scanning the party looking for SONNY.\n\n\tINT DAY: DON'S OFFICE (SUMMER 1945)\n\n\tHAGEN glances up the staircase.\n\n\t\t\t\tHAGEN\n\t\tSonny?\n\n\tThen he goes up.\n\n\tINT DAY: DON'S UPSTAIRS ROOM (SUMMER 1945)\n\n\tSONNY and LUCY are in a room upstairs; he has lifted her\n\tgown's skirts almost over her head, and has her standing\n\tagainst the door.  Her face peeks out from the layers of\n\tpetticoats around it like a flower in ecstasy.\n\n\t\t\t\tLUCY\n\t\tSonnyeeeeeeee.\n\n\tHer head bouncing against the door with the rhythm of his\n\tbody.  But there is a knocking as well.  They stop, freeze\n\tin that position.\n\n\t\t\t\tHAGEN (O.S.)\n\t\tSonny?  Sonny, you in there?\n\n\tINT DAY: DON'S UPSTAIRS HALLWAY (SUMMER 1945)\n\n\tOutside, HAGEN by the door.\n\n\t\t\t\tHAGEN\n\t\tThe old man wants you; Johnny's\n\t\there...he's got a problem.\n\n\t\t\t\tSONNY (O.S.)\n\t\tOkay.  One minute.\n\n\tHAGEN hesitates.  We HEAR LUCY's head bouncing against the\n\tdoor again.  TOM leaves.\n\n\tINT DAY: DON'S OFFICE (SUMMER 1945)\n\n\t\t\t\tDON CORLEONE\n\t\tACT LIKE A MAN!  By Christ in\n\t\tHeaven, is it possible you turned\n\t\tout no better than a Hollywood\n\t\tfinocchio.\n\n\tBoth HAGEN and JOHNNY cannot refrain from laughing.  The DON\n\tsmiles.  SONNY enters as noiselessly as possible, still\n\tadjusting his clothes.\n\n\t\t\t\tDON CORLEONE\n\t\tAll right, Hollywood...Now tell me\n\t\tabout this Hollywood Pezzonovanta\n\t\twho won't let you work.\n\n\t\t\t\tJOHNNY\n\t\tHe owns the studio.  Just a month\n\t\tago he bought the movie rights to\n\t\tthis book, a best seller.  And the\n\t\tmain character is a guy just like\n\t\tme.  I wouldn't even have to act,\n\t\tjust be myself.\n\n\tThe DON is silent, stern.\n\n\t\t\t\tDON CORLEONE\n\t\tYou take care of your family?\n\n\t\t\t\tJOHNNY\n\t\tSure.\n\n\tHe glances at SONNY, who makes himself as inconspicuous as\n\the can.\n\n\t\t\t\tDON CORLEONE\n\t\tYou look terrible.  I want you to\n\t\teat well, to rest.  And spend time\n\t\twith your family.  And then, at the\n\t\tend of the month, this big shot\n\t\twill give you the part you want.\n\n\t\t\t\tJOHNNY\n\t\tIt's too late.  All the contracts\n\t\thave been signed, they're almost\n\t\tready to shoot.\n\n\t\t\t\tDON CORLEONE\n\t\tI'll make him an offer he can't\n\t\trefuse.\n\n\tHe takes JOHNNY to the door, pinching his cheek hard enough\n\tto hurt.\n\n\t\t\t\tDON CORLEONE\n\t\tNow go back to the party and leave\n\t\tit to me.\n\n\tHe closes the door, smiling to himself.  Turns to HAGEN.\n\n\t\t\t\tDON CORLEONE\n\t\tWhen does my daughter leave with\n\t\ther bridegroom?\n\n\t\t\t\tHAGEN\n\t\tThey'll cut the cake in a few\n\t\tminutes...leave right after that.\n\t\tYour new son-in-law, do we give him\n\t\tsomething important?\n\n\t\t\t\tDON CORLEONE\n\t\tNo, give him a living.  But never\n\t\tlet him know the family's business.\n\t\tWhat else, Tom?\n\n\t\t\t\tHAGEN\n\t\tI've called the hospital; they've\n\t\tnotified Consiglere Genco's family\n\t\tto come and wait.  He won't last\n\t\tout the night.\n\n\tThis saddens the DON.  He sighs.\n\n\t\t\t\tDON CORLEONE\n\t\tGenco will wait for me.  Santino,\n\t\ttell your brothers they will come\n\t\twith me to the hospital to see\n\t\tGenco.  Tell Fredo to drive the big\n\t\tcar, and ask Johnny to come with us.\n\n\t\t\t\tSONNY\n\t\tAnd Michael?\n\n\t\t\t\tDON CORLEONE\n\t\tAll my sons.\n\t\t\t  (to HAGEN)\n\t\tTom, I want you to go to California\n\t\ttonight.  Make the arrangements.\n\t\tBut don't leave until I come back\n\t\tfrom the hospital and speak to you.\n\t\tUnderstood?\n\n\t\t\t\tHAGEN\n\t\tUnderstood.\n\n\tEXT DAY: MALL (SUMMER 1945)\n\n\tNow all the wedding GUESTS excitedly clap their hands over\n\tthe entrance of the cake: NAZORINE is beaming as he wheels\n\tin a serving table containing the biggest, gaudiest, most\n\textravagant wedding cake ever baked, an incredible monument\n\tof his gratitude.  The CROWD is favorably impressed: they\n\tbegin to clink their knives or forks against their glasses,\n\tin the traditional request for the Bride to cut the cake and\n\tkiss the Groom.  Louder and louder, five hundred forks\n\thitting five hundred glasses.\n\n\tEXT DAY: MALL (SUMMER 1945)\n\n\tSilence.\n\n\tHIGH ANGLE ON THE MALL, late day.  The GUESTS are gone.  A\n\tsingle black car is in the courtyard.  FREDDIE is behind the\n\tdriver's seat: the DON enters the car, looks at MICHAEL, who\n\tsits between SONNY and JOHNNY in the rear seat.\n\n\t\t\t\tDON CORLEONE\n\t\tWill your girl friend get back to\n\t\tthe city all right?\n\n\t\t\t\tMICHAEL\n\t\tTom said he'd take care of it.\n\n\tThe DON pulls the door shut; and the car pulls out, through\n\tthe gate of the great Corleone Mall.\n\n\tINT DAY: HOSPITAL CORRIDOR (SUMMER 1945)\n\n\tA long white hospital corridor, at the end of which we can\n\tsee a grouping of FIVE WOMEN, some old and some young, but\n\tall plump and dressed in black.\n\n\tDON CORLEONE and his SONS move toward the end.  But then the\n\tDON slows, putting his hand on MICHAEL's shoulder.  MICHAEL\n\tstops and turns toward his FATHER.  The two looks at one\n\tanother for some time.  SILENCE.  DON CORLEONE then lifts\n\this hand, and slowly touches a particular medal on MICHAEL's\n\tuniform.\n\n\t\t\t\tDON CORLEONE\n\t\tWhat was this for?\n\n\t\t\t\tMICHAEL\n\t\tFor bravery.\n\n\t\t\t\tDON CORLEONE\n\t\tAnd this?\n\n\t\t\t\tMICHAEL\n\t\tFor killing a man.\n\n\t\t\t\tDON CORLEONE\n\t\tWhat miracles you do for strangers.\n\n\t\t\t\tMICHAEL\n\t\tI fought for my country.  It was my\n\t\tchoice.\n\n\t\t\t\tDON CORLEONE\n\t\tAnd now, what do you choose to do?\n\n\t\t\t\tMICHAEL\n\t\tI'm going to finish school.\n\n\t\t\t\tDON CORLEONE\n\t\tGood.  When you are finished, come\n\t\tand talk to me.  I have hopes for\n\t\tyou.\n\n\tAgain they regard each other without a word.  MICHAEL turns,\n\tand continues on.  DON CORLEONE watches a moment, and then\n\tfollows.\n\n\tINT DAY: HOSPITAL ROOM (SUMMER 1945)\n\n\tDON CORLEONE enters the hospital room, moving closest to OUR\n\tVIEW.  He is followed by his SONS, JOHNNY and the WOMEN.\n\n\t\t\t\tDON CORLEONE\n\t\t\t  (whispered)\n\t\tGenco, I've brought my sons to pay\n\t\ttheir respects.  And look, even\n\t\tJohnny Fontane, all the way from\n\t\tHollywood.\n\n\tGENCO is a tiny, wasted skeleton of a man.  DON CORLEONE\n\ttakes his bony hand, as the others arrange themselves around\n\this bed, each clasping the other hand in turn.\n\n\t\t\t\tGENCO\n\t\tGodfather, Godfather, it's your\n\t\tdaughter's wedding day, you cannot\n\t\trefuse me.  Cure me, you have the\n\t\tpower.\n\n\t\t\t\tDON CORLEONE\n\t\tI have no such power...but Genco,\n\t\tdon't fear death.\n\n\t\t\t\tGENCO\n\t\t\t  (with a sly wink)\n\t\tIt's been arranged, then?\n\n\t\t\t\tDON CORLEONE\n\t\tYou blaspheme.  Resign yourself.\n\n\t\t\t\tGENCO\n\t\tYou need your old Consigliere.  Who\n\t\twill replace me?\n\t\t\t  (suddenly)\n\t\tStay with me Godfather.  Help me\n\t\tmeet death.  If he sees you, he\n\t\twill be frightened and leave me in\n\t\tpeace.  You can say a word, pull a\n\t\tfew strings, eh?  We'll outwit that\n\t\tbastard as we outwitted all those\n\t\tothers.\n\t\t\t  (clutching his hand)\n\t\tGodfather, don't betray me.\n\n\tThe DON motions all the others to leave the room.  They do.\n\tHe returns his attention to GENCO, holding his hand and\n\twhispering things we cannot hear, as they wait for death.\n\n\tINT NIGHT: AIRPLANE (SUMMER 1945)\n\n\tFADE IN:\n\n\tThe interior of a non-stop Constellation.  HAGEN is one of\n\tthe very few passengers on this late flight.  He looks like\n\tany young lawyer on a business trip.  He is tired from the\n\tdifficult preparation and duties that he has just executed\n\tduring the wedding.  On the seat next to him is an enormous,\n\tbulging briefcase.  He closes his eyes.\n\n\tINT NIGHT: HONEYMOON HOTEL (SUMMER 1945)\n\n\tThe honeymoon hotel: CARLO and CONNIE.  CARLO is in his\n\tundershorts, sitting up on the bed, anxiously taking the\n\tenvelopes out of the silk bridal purse and counting the\n\tcontents.  CONNIE prepares herself in the large marble\n\tbathroom.  She rubs her hands over his bronze shoulders, and\n\ttries to get his interest.\n\n\tINT NIGHT: DON'S OFFICE (SUMMER 1945)\n\n\tDON CORLEONE in his office.  LUCA BRASI sitting near to him.\n\n\t\t\t\tDON CORLEONE\n\t\tLuca, I am worried about this man\n\t\tSollozzo.  Find out what you can,\n\t\tthrough the Tattaglias.  Let them\n\t\tbelieve you could be tempted away\n\t\tfrom the Corleone Family, if the\n\t\tright offer was made.  Learn what\n\t\the has under his fingernails...\n\n\tINT NIGHT: MANCINI APT. HALL (SUMMER 1945)\n\n\tThe hallway of an apartment building.  SONNY enters, climbs\n\ttwo steps at a time.  He knocks, and then whispers.\n\n\t\t\t\tSONNY\n\t\tIt's me, Sonny.\n\n\tThe door opens, and two lovely arms are around him, pulling\n\thim into the apartment.\n\n\tINT NIGHT: LUCA'S ROOM (WINTER 1945)\n\n\tLUCA BRASI's tiny room.  He is partly dressed.  He kneels\n\tand reaches under his bed and pulls out a small, locked\n\ttrunk.  He opens it, and takes out a heavy, bullet-proof\n\tvest.  He puts it on, over his wool undershirt, and then\n\tputs on his shirt and jacket.  He takes his gun, quickly\n\tdisassembles, checks, and reassembles it.  And leaves.\n\n\tINT NIGHT: DON'S OFFICE (SUMMER 1945)\n\n\tA CLOSE VIEW of DON CORLEONE thinking quietly.\n\n\tINT NIGHT: MOVING TRAIN (SUMMER 1945)\n\n\tMICHAEL and KAY on a train, speeding on their way to New\n\tHampshire.\n\n\tINT NIGHT: SUBWAY (WINTER 1945)\n\n\tLUCA, in his bulky jacket, sitting quietly on an empty\n\tsubway train.\n\n\tINT NIGHT: AIRPLANE (SUMMER 1945)\n\n\tHAGEN on the Constellation.  He reaches into his briefcase,\n\tand takes out several pictures and papers.\n\n\tOne photograph is of a smiling man, JACK WOLTZ, linked arm\n\tin arm with fifteen movie stars on either side, including a\n\tlovely young child star to his immediate right.\n\n\tHAGEN considers other papers.\n\n\tINT NIGHT: DON'S OFFICE (SUMMER 1945)\n\n\tDON CORLEONE looks, and then moves HAGEN into an embrace.\n\tHe straightens his arms and looks at TOM deeply.\n\n\t\t\t\tDON CORLEONE\n\t\tRemember my new Consigliere, a\n\t\tlawyer with his briefcase can steal\n\t\tmore than a hundred men with guns.\n\n\tEXT DAY: WOLTZ ESTATE GATE (SUMMER 1945)\n\n\tJACK WOLTZ ESTATE.  HAGEN stands before the impressive gate,\n\tarmed only with his briefcase.  A GATEMAN opens the gate,\n\tand TOM enters.\n\n\tEXT DAY: WOLTZ GARDENS (SUMMER 1945)\n\n\tHAGEN and WOLTZ comfortably stroll along beautiful formal\n\tgardens, martinis in hand.\n\n\t\t\t\tWOLTZ\n\t\tYou should have told me your boss\n\t\twas Corleone, Tom, I had to check\n\t\tyou out.  I thought you were just\n\t\tsome third rate hustler Johnny was\n\t\trunning in to bluff me.\n\t\t\t  (a piece of statuary)\n\t\tFlorence, thirteenth century.\n\t\tDecorated the garden of a king.\n\n\tThey cross the garden and head toward the stables.\n\n\t\t\t\tWOLTZ\n\t\tI'm going to show you something\n\t\tbeautiful.\n\n\tThey pass the stables, and come to rest by a stall with a\n\thuge bronze plaque attached to the outside wall: \"KHARTOUM.\"\n\tTWO SECURITY GUARDS are positioned in chairs nearby; they\n\trise as WOLTZ approaches.\n\n\t\t\t\tWOLTZ\n\t\tYou like horses?  I like horses, I\n\t\tlove 'em.  Beautiful, expensive\n\t\tRacehorses.\n\n\tThe animal inside is truly beautiful.  WOLTZ whispers to him\n\twith true love in his voice.\n\n\t\t\t\tWOLTZ\n\t\tKhartoum...Kartoum...You are\n\t\tlooking at six hundred thousand\n\t\tdollars on four hoofs.  I bet even\n\t\tRussian Czars never paid that kind\n\t\tof dough for a single horse.  But\n\t\tI'm not going to race him I'm going\n\t\tto put him out to Stud.\n\n\tINT NIGHT: WOLTZ DINING ROOM (SUMMER 1945)\n\n\tHAGEN and WOLTZ sit at an enormous dining room table,\n\tattended by SEVERAL SERVANTS.  Great paintings hang on the\n\twalls.  The meal is elaborate and sumptuous.\n\n\t\t\t\tHAGEN\n\t\tMr. Corleone is Johnny's Godfather.\n\t\tThat is very close, a very sacred\n\t\treligious relationship.\n\n\t\t\t\tWOLTZ\n\t\tOkay, but just tell him this is one\n\t\tfavor I can't give.  But he should\n\t\ttry me again on anything else.\n\n\t\t\t\tHAGEN\n\t\tHe never asks a second favor when\n\t\the has been refused the first.\n\t\tUnderstood?\n\n\t\t\t\tWOLTZ\n\t\tYou smooth son of a bitch, let me\n\t\tlay it on the line for you, and\n\t\tyour boss.  Johnny Fontane never\n\t\tgets that movie.  I don't care how\n\t\tmany Dago, Guinea, wop Greaseball\n\t\tGoombahs come out of the woodwork!\n\n\t\t\t\tHAGEN\n\t\tI'm German-Irish.\n\n\t\t\t\tWOLTZ\n\t\tOkay my Kraut-Mick friend, Johnny\n\t\twill never get that part because I\n\t\thate that pinko punk and I'm going\n\t\tto run him out of the Movies.  And\n\t\tI'll tell you why.  He ruined one\n\t\tof Woltz Brothers' most valuable\n\t\tproteges.  For five years I had\n\t\tthis girl under training; singing\n\t\tlessons!  Acting lessons!  Dancing\n\t\tlessons!  We spent hundreds of\n\t\tthousands of dollars--I was going\n\t\tto make her a star.  I'll be even\n\t\tmore frank, just to show you that\n\t\tI'm not a hard-hearted man, that it\n\t\twasn't all dollars and cents.  That\n\t\tgirl was beautiful and young and\n\t\tinnocent and she was the greatest\n\t\tpiece of ass I've ever ad and I've\n\t\thad them all over the world.  Then\n\t\tJohnny comes along with that olive\n\t\toil voice and guinea charm and she\n\t\truns off.  She threw it all away to\n\t\tmake me look ridiculous.  A MAN IN\n\t\tMY POSITION CANNOT AFFORD TO BE\n\t\tMADE TO LOOK RIDICULOUS!\n\n\tEXT DAY: GENCO OLIVE OIL CO. (SUMMER 1945)\n\n\tAn unimposing little building in New York City on Mott\n\tStreet with a large old sign: \"GENCO OLIVE OIL IMPORTS,\n\tINC.\" next to an open-faced fruit market.\n\n\tA dark Buick pulls up, and a single small man, whom we\n\tcannot see well because of the distance, gets out and enters\n\tthe building.  This is VIRGIL SOLLOZZO.\n\n\tINT DAY: OLIVE OIL OFFICES (SUMMER 1945)\n\n\tLooking toward the staircase we can hear SOLLOZZO's footsteps\n\tbefore he actually rises into view.  He is a small man, very\n\tdark, with curly black hair.  But wiry, and tight and hard,\n\tand obviously very dangerous.  He is greeted at the head of\n\tthe stairs by SONNY, who takes his hand and shakes it,\n\tintroducing himself.  For a moment, there is a complex of\n\thandshaking quite formal, and whispered respectful\n\tintroductions.  Finally, SOLLOZZO is taken into the DON's\n\tglass paneled office; the two principals are introduced.\n\tThey are very respectful of one another.  Folding chairs are\n\tbrought in by FREDDIE, and soon they are all sitting around\n\tin a circle; the DON, SOLLOZZO, SONNY, HAGEN, FREDDIE,\n\tCLEMENZA and TESSIO.  The DON is the slightest bit foolish\n\twith all his compatriots, whereas SOLLOZZO has brought no\n\tone.  Throughout all that transpires, however, it is clear\n\tthat this scene is between two men: SOLLOZZO and DON CORLEONE.\n\n\t\t\t\tSOLLOZZO\n\t\tMy business is heroin, I have poppy\n\t\tfields, laboratories in Narseilles\n\t\tand Sicily, ready to go into\n\t\tproduction.  My importing methods\n\t\tare as safe as these things can be,\n\t\tabout five per cent loss.  The risk\n\t\tis nothing, the profits enormous.\n\n\t\t\t\tDON CORLEONE\n\t\tWhy do you come to me?  Why do I\n\t\tdeserve your generosity?\n\n\t\t\t\tSOLLOZZO\n\t\tI need two million dollars in\n\t\tcash...more important, I need a\n\t\tfriend who has people in high\n\t\tplaces; a friend who can guarantee\n\t\tthat if one of my employees be\n\t\tarrested, they would get only light\n\t\tsentences.  Be my friend.\n\n\t\t\t\tDON CORLEONE\n\t\tWhat percentages for my family?\n\n\t\t\t\tSOLLOZZO\n\t\tThirty per cent.  In the first year\n\t\tyour share would be four million\n\t\tdollars; then it would go up.\n\n\t\t\t\tDON CORLEONE\n\t\tAnd what is the percentage of the\n\t\tTattaglia family?\n\n\tSOLLOZZO nods toward HAGEN.\n\n\t\t\t\tSOLLOZZO\n\t\tMy compliments.  I'll take care of\n\t\tthem from my share.\n\n\t\t\t\tDON CORLEONE\n\t\tSo.  I receive 30 per cent just for\n\t\tfinance and legal protection.  No\n\t\tworries about operations, is that\n\t\twhat you tell me?\n\n\t\t\t\tSOLLOZZO\n\t\tIf you think two million dollars in\n\t\tcash is just finance, I congratulate\n\t\tyou Don Corleone.\n\n\tThere is a long silence; in which each person present feels\n\tthe tension.  The DON is about to give his answer.\n\n\t\t\t\tDON CORLEONE\n\t\tI said I would see you because I've\n\t\theard you're a serious man, to be\n\t\ttreated with respect...\n\t\t\t  (pause)\n\t\tBut I'll say no to you.\n\n\tWe feel this around the room.\n\n\t\t\t\tDON CORLEONE\n\t\tI'll give you my reasons.  I have\n\t\tmany, many friends in Politics.\n\t\tBut they wouldn't be so friendly if\n\t\tmy business was narcotics instead\n\t\tof gambling.  They think gambling\n\t\tis something like liquor, a harmless\n\t\tvice...and they think narcotics is\n\t\tdirty business.\n\n\tSOLLOZZO takes a breath.\n\n\t\t\t\tDON CORLEONE\n\t\tNo...how a man makes his living is\n\t\tnone of my business.  But this\n\t\tproposition of yours is too risky.\n\t\tAll the people in my family lived\n\t\twell the last ten years, I won't\n\t\trisk that out of greed.\n\n\t\t\t\tSOLLOZZO\n\t\tAre you worried about security for\n\t\tyour million?\n\n\t\t\t\tDON CORLEONE\n\t\tNo.\n\n\t\t\t\tSOLLOZZO\n\t\tThe Tattaglias will guarantee your\n\t\tinvestment also.\n\n\tThis startles SONNY; he blurts out.\n\n\t\t\t\tSONNY\n\t\tThe Tattaglia family guarantees our\n\t\tinvestment?\n\n\tSOLLOZZO hears him first, and then very slowly turns to face\n\thim.  Everyone is the room knows that SONNY has stepped out\n\tof line.\n\n\t\t\t\tDON CORLEONE\n\t\tYoung people are greedy, and they\n\t\thave no manners.  They speak when\n\t\tthey should listen.  But I have a\n\t\tsentimental weakness for my\n\t\tchildren, and I've spoiled them, as\n\t\tyou see.  But Signor Sollozzo, my\n\t\tno is final.\n\n\tSOLLOZZO nods, understands that this is the dismissal.  He\n\tglances one last time at SONNY.  He rises; all the others do\n\tas well.  He bows to the DON, shakes his hand, and formally\n\ttakes his leave.  When the footsteps can no longer be heard:\n\n\tThe DON turns to SONNY.\n\n\t\t\t\tDON CORLEONE\n\t\tSantino, never let anyone outside\n\t\tthe family know what you are\n\t\tthinking.  I think your brain is\n\t\tgoing soft from all that comedy you\n\t\tplay with that young girl.\n\n\tTWO OFFICE WORKERS are carrying an enormous floral display\n\twith the word \"THANK YOU\" spelled out in flowers.\n\n\t\t\t\tDON CORLEONE\n\t\tWhat is this nonsense?\n\n\t\t\t\tHAGEN\n\t\tIt's from Johnny.  It was announced\n\t\tthis morning.  He's going to play\n\t\tthe lead in the new Woltz Brothers\n\t\tfilm.\n\n\tINT DAY: WOLTZ'S BEDROOM (SUMMER 1945)\n\n\tIt is large, dominated by a huge bed, in which a man,\n\tpresumably WOLTZ, is sleeping.  Soft light bathes the room\n\tfrom the large windows.  We move closer to him until we see\n\this face, and recognize JACK WOLTZ.  He turns uncomfortably;\n\tmutters, feels something strange in his bedsheets.  Something\n\twet.\n\n\tHe wakens, feels the sheets with displeasure; they are wet.\n\tHe looks at his hand; the wetness is blood.  He is\n\tfrightened, pulls aside the covers, and sees fresh blood on\n\this sheets and pajamas.  He grunts, pulls the puddle of\n\tblood in his bed.  He feels his own body frantically,\n\tmoving, down, following the blood, until he is face to face\n\twith the great severed head of Khartoum lying at the foot of\n\this bed.  Just blood from the hacked neck.  White reedy\n\ttendons show.  He struggles up to his elbows in the puddle\n\tof blood to see more clearly.  Froth covers the muzzle, and\n\tthe enormous eyes of the animal are yellowed and covered\n\twith blood.\n\n\tWOLTZ tries to scream; but cannot.  No sound comes out.\n\tThen, finally and suddenly an ear-splitting scream of pure\n\tterror escapes from WOLTZ, who is rocking on his hands and\n\tknees in an uncontrolled fit, blood all over him.\n\n\tINT DAY: OLIVE OIL OFFICES (SUMMER 1945)\n\n\tCLOSE VIEW on the GODFATHER.  Nodding.\n\n\t\t\t\tDON CORLEONE\n\t\tSend Johnny my congratulations.\n\n\t----------------------------------------FADE OUT--------\n\n\t(SCENES 12 & 12 OMITTED)\n\n\tFADE IN:\n\n\tEXT DAY: FIFTH AVENUE (WINTER 1945)\n\n\tFifth Avenue in the snow.  Christmas week.  People are\n\tbundled up with rosy faces, rushing to buy presents.\n\n\tKAY and MICHAEL exit a Fifth Avenue department store,\n\tcarrying a stack of gaily wrapped gifts, arm in arm.\n\n\t\t\t\tKAY\n\t\tWe have something for your mother,\n\t\tfor Sonny, we have the tie for\n\t\tFredo and Tom Hagen gets the\n\t\tReynolds pen...\n\n\t\t\t\tMICHAEL\n\t\tAnd what do you want for Christmas?\n\n\t\t\t\tKAY\n\t\tJust you.\n\n\tThey kiss.\n\n\tINT DAY: HOTEL ROOM (WINTER 1945)\n\n\tCLOSE ON a wooden radio, playing quiet Music.  THE VIEW PANS\n\tAROUND the dark hotel room, curtained against the daylight.\n\n\t\t\t\tMICHAEL (O.S.)\n\t\tWe'll have a quiet, civil ceremony\n\t\tat the City Hall, no big fuss, no\n\t\tfamily, just a couple of friends as\n\t\twitnesses.\n\n\tThe two are in each other's arms in a mess of bedsheets on\n\tthe two single beds that they have pushed together.\n\n\t\t\t\tKAY\n\t\tWhat will your father say?\n\n\t\t\t\tMICHAEL\n\t\tAs long as I tell him beforehand he\n\t\twon't object.  He'll be hurt, but\n\t\the won't object.\n\n\t\t\t\tKAY\n\t\tWhat time do they expect us?\n\n\t\t\t\tMICHAEL\n\t\tFor dinner.  Unless I call and tell\n\t\tthem we're still in New Hampshire.\n\n\t\t\t\tKAY\n\t\tMichael.\n\n\t\t\t\tMICHAEL\n\t\tThen we can have dinner, see a\n\t\tshow, and spend one more night.\n\n\tHe moves to the telephone.\n\n\t\t\t\tMICHAEL (CONT'D.)\n\t\tOperator.  Get me\n\t\t\t  (fill in number)\n\n\n\t\t\t\tKAY\n\t\tMichael, what are you doing?\n\n\t\t\t\tMICHAEL\n\t\tShhh, you be the long distance\n\t\toperator.  Here.\n\n\t\t\t\tKAY\n\t\tHello...this is Long Distance.  I\n\t\thave a call from New Hampshire.  Mr.\n\t\tMichael Corleone.  One moment please.\n\n\tShe hands the phone to MICHAEL who continues the deception.\n\n\t\t\t\tMICHAEL\n\t\tHello, Tom?  Michael.  Yeah...\n\t\tlisten, we haven't left yet.  I'm\n\t\tdriving down to the city with Kay\n\t\ttomorrow morning.  There's something\n\t\timportant I want to tell the old\n\t\tman before Christmas.  Will he be\n\t\thome tomorrow night?\n\n\tINT DAY: OLIVE OIL OFFICE (WINTER 1945)\n\n\tHAGEN in the Olive Oil Company office.  In the background,\n\tthrough the glass partitions, we can see the DON, at work in\n\this office.  TOM is tired, and steeped in paperwork.\n\n\t\t\t\tHAGEN (O.S.)\n\t\tSure.  Anything I can do for you.\n\n\t\t\t\tMICHAEL (O.S.)\n\t\tNo.  I guess I'll see you Christmas.\n\t\tEveryone's going to be out at Long\n\t\tBeach, right?\n\n\t\t\t\tHAGEN\n\t\tRight.\n\n\tHe smiles.  MICHAEL has hung up.  He looks at the piles of\n\twork, and can't face it.  He rises, puts on his coat and\n\that, and continues out.\n\n\tHe peeks into the DON's office.\n\n\t\t\t\tHAGEN\n\t\tMichael called; he's not leaving\n\t\tNew Hampshire until tomorrow\n\t\tmorning.  I've got to go, I promised\n\t\tTheresa I'd pick up some toys for\n\t\tthe kids.\n\n\tThe DON smiles and nods.\n\n\tTOM smiles, and leaves; OUR VIEW remaining with DON CORLEONE.\n\tFREDDIE is sitting on a bench in the corner, reading the\n\tafternoon paper.  He puts aside the papers the office\n\tmanager has prepared for him, and then moves to FREDDIE,\n\traps his knuckles on his head to take his nose out of the\n\tpaper.\n\n\t\t\t\tDON CORLEONE\n\t\tTell Paulie to get the car from the\n\t\tlot; I'll be ready to go home in a\n\t\tfew minutes.\n\n\t\t\t\tFREDO\n\t\tI'll have to get it myself; Paulie\n\t\tcalled in sick this morning.\n\n\t\t\t\tDON CORLEONE\n\t\tThat's the third time this month.\n\t\tI think maybe you'd better get a\n\t\thealthier bodyguard for me.  Tell\n\t\tTom.\n\n\t\t\t\tFREDO\n\t\t\t  (going)\n\t\tPaulie's a good kid.  If he's sick,\n\t\the's sick.  I don't mind getting\n\t\tthe car.\n\n\tFREDDIE leaves.  He slowly puts on his jacket.  Looks out\n\this window.\n\n\tEXT DUSK: OLIVE OIL CO. (WINTER 1945)\n\n\tFREDDIE crosses the street.\n\n\tINT DUSK: OLIVE OIL OFFICE (WINTER 1945)\n\n\t\t\t\tOFFICE MANAGER\n\t\tBuon Watale, Don Corleone.\n\n\tThe MANAGER helps him on with his overcoat.  Once again, the\n\tDON glances out his window.\n\n\tThe black car pulls up; FREDDIE driving.\n\n\t\t\t\tDON CORLEONE\n\t\tMerry Christmas.\n\t\t\t  (handing the MANAGER\n\t\t\t  an envelope)\n\n\n\tAnd he starts down the stairs.\n\n\tEXT DUSK: OLIVE OIL CO. (WINTER 1945)\n\n\tThe light outside is very cold, and beginning to fail.  When\n\tFREDDIE sees his FATHER coming, he moves back into the\n\tdriver's seat.  The DON moves to the car, and is about to\n\tget in when he hesitates, and turns back to the long, open\n\tfruit stand near the corner.\n\n\tThe PROPRIETOR springs to serve him.  The DON walks among\n\tthe trays and baskets, and merely points to a particular\n\tpiece of fruit.  As he selects, the MAN gingerly picks the\n\tpieces of fruit up and puts them into a paper bag.  The DON\n\tpays with a five dollar bill, waits for his change, and then\n\tturns back to the car.\n\n\tEXT DUSK: POLKS TOY STORE (WINTER 1945)\n\n\tTOM HAGEN exits carrying a stack of presents, all gift\n\twrapped.  He continues past the windows.  As he walks,\n\tsomeone walks right in his way.  He looks up.  It is SOLLOZZO.\n\n\tHe takes TOM by the arm and walks along with him.\n\n\t\t\t\tSOLLOZZO\n\t\t\t  (quietly)\n\t\tDon't be frightened.  I just want\n\t\tto talk to you.\n\n\tA car parked at the curb suddenly flings its rear door open.\n\n\t\t\t\tSOLLOZZO\n\t\t\t  (urgently)\n\t\tGet in; I want to talk to you.\n\n\tHAGEN pulls his arm free.  He is frightened.\n\n\t\t\t\tHAGEN\n\t\tI haven't got time.\n\n\tTWO MEN suddenly appear on either side of him.\n\n\t\t\t\tSOLLOZZO\n\t\tGet in the car.  If I wanted to\n\t\tkill you you'd be dead already.\n\t\tTrust me.\n\n\tHAGEN, sick to his stomach, moves with his ESCORTS, leaving\n\tour VIEW on the Mechanical windows gaily bobbing the story\n\tof Hansel and Gretel.  We HEAR the car doors shut, and the\n\tcar drive off.\n\n\tEXT NIGHT: RADIO CITY - PHONE BOOTH (WINTER 1945)\n\n\tRADIO CITY MUSIC HALL during the Christmas show.  KAY and\n\tMICHAEL exit; tears are still streaming down her cheeks, and\n\tshe sniffles, and dries her tears with Kleenex.  KAY\n\tnostalgically hums \"The Bells of Saint Mary's,\" as they walk\n\tarm in arm.\n\n\t\t\t\tKAY\n\t\tWould you like me better if I were\n\t\ta nun?\n\n\t\t\t\tMICHAEL\n\t\tNo.\n\n\t\t\t\tKAY\n\t\tWould you like me better if I were\n\t\tIngrid Bergman?\n\n\tThey have passed a little enclosed newsstand.  KAY sees\n\tsomething that terrifies her.  She doesn't know what to do.\n\tMICHAEL still walks, thinking about her question.\n\n\t\t\t\tKAY\n\t\t\t  (a little voice)\n\t\tMichael?\n\n\t\t\t\tMICHAEL\n\t\tI'm thinking about it.\n\n\t\t\t\tKAY\n\t\tMichael...\n\n\t\t\t\tMICHAEL\n\t\tNo, I would not like you better if\n\t\tyou were Ingrid Bergman.\n\n\tShe cannot answer him.  Rather she pulls him by the arm,\n\tback to the newsstand, and points.  His face goes grave.\n\n\tThe headlines read: \"VITO CORLEONE SHOT, CHIEFTAN GUNNED\n\tDOWN.\"\n\n\tMICHAEL is petrified; quickly he takes each edition, drops a\n\tdollar in the tray, and hungrily reads through them.  KAY\n\tknows to remain silent.\n\n\t\t\t\tMICHAEL\n\t\t\t  (desperately)\n\t\tThey don't say if he's dead or alive.\n\n\tEXT DUSK: OLIVE OIL CO. (WINTER 1945)\n\n\tDON CORLEONE by the fruit stand; he is about to move to the\n\tcar, when TWO MEN step from the corner.  Suddenly, the DON\n\tdrops the bag of fruit and darts with startling quickness\n\ttoward the parked car.\n\n\t\t\t\tDON CORLEONE\n\t\tFredo, Fredo!\n\n\tThe paper bag has hit the ground, and the fruit begins\n\trolling along the sidewalk, as we HEAR gunshots.\n\n\tFive bullets catch the DON in the back; he arches in pain,\n\tand continues toward the car.\n\n\tThe PROPRIETOR of the fruit stand rushes for cover, knocking\n\tover an entire case of fruit.\n\n\tThe TWO GUNMEN move in quickly, anxious to finish him off.\n\n\tTheir feet careful to avoid the rolling fruit.  There are\n\tmore GUNSHOTS.\n\n\tFREDDIE is hysterical; he tries to get out of the car;\n\thaving difficulty opening the door.  He rushes out, a gun\n\ttrembling in his hand; his mouth open.  He actually drops\n\tthe gun.\n\n\tThe gun falls amid the rolling fruit.\n\n\tThe GUNMEN are panicked.  They fire once more at the downed\n\tDON CORLEONE.  His leg and arm twitch where they are hit;\n\tand pools of blood are beginning to form.\n\n\tThe GUNMEN are obviously in a state of panic and confusion;\n\tthey disappear around the corner as quickly as they came.\n\n\tThe PEOPLE about the avenue have all but disappeared:\n\trather, we catch glimpses of them, poking their heads safely\n\tfrom around corners, inside doorways and arches, and from\n\twindows.  But the street itself is now empty.\n\n\tFREDDIE is in shock; he looks at his FATHER; now great\n\tpuddles of blood have formed, and the DON is lifeless and\n\tface down in them.\n\n\tFREDDIE falls back on to the curb and sits there, saying\n\tsomething we cannot understand.  He begins to weep profusely.\n\n\tINT NIGHT: SUBWAY (WINTER 1945)\n\n\tLUCA BRASI riding alone on a subway car, late at night.  He\n\tgets off.\n\n\tHe emerges at a subway terminal, proceeds out.\n\n\tEXT NITE: NIGHT CLUB STREET (WINTER 1945)\n\n\tLUCA walks down the late night street.  He approaches an\n\telegant New York Nightclub, whose gaudy neon sign is still\n\twinking this late at night.  He waits and watches.  Then the\n\tsign goes out; and he proceeds into the club.\n\n\tINT NITE: NIGHTCLUB (WINTER 1945)\n\n\tThe main floor of the Nightclub is very large, with endless\n\tglistening wooden floors.  Now, at this late time, the\n\tchairs have been stacked on the tables and a NEGRO JANITOR\n\tis waxing them.  A single HAT-CHECK GIRL is counting her\n\treceipts.  LUCA moves past the empty bandstand, and sits at\n\tthe bar.  ANOTHER MAN, dark and very well-built, moves\n\tbehind the bar.\n\n\t\t\t\tMAN\n\t\tLuca...I'm Bruno Tattaglia.\n\n\t\t\t\tLUCA\n\t\tI know.\n\n\tLUCA looks up; and out of the shadows emerges SOLLOZZO.\n\n\t\t\t\tSOLLOZZO\n\t\tDo you know who I am?\n\n\tLUCA Nods.\n\n\t\t\t\tSOLLOZZO\n\t\tYou've been talking to the\n\t\tTattaglias.  They thought we could\n\t\tdo business.\n\n\tLUCA listens.\n\n\t\t\t\tSOLLOZZO\n\t\tI need somebody strong to protect\n\t\tmy operation, physically.  I've\n\t\theard you're not happy with your\n\t\tfamily, you might make a switch.\n\n\t\t\t\tLUCA\n\t\tIf the money is good enough.\n\n\t\t\t\tSOLLOZZO\n\t\tOn the first shipment, I can\n\t\tguarantee you fifty thousand dollars.\n\n\tLUCA looks at him; he had no idea the offer would be so good.\n\n\tSOLLOZZO extends his hand, but LUCA pretends not to see it,\n\trather, he busies himself putting a cigarette in his mouth.\n\tBRUNO TATTAGLIA, behind the bar, makes a cigarette lighter\n\tmagically appear, and holds it to LUCA's cigarette.  Then,\n\the does an odd thing; he drops the lighter on the bar, and\n\tputs his hand lightly on LUCA's, almost patting it.\n\n\tINT NITE: SONNY'S LIVING ROOM (WINTER 1945)\n\n\tThe telephone in SONNY's house is ringing.  He approaches\n\tit, obviously fresh from a nap.\n\n\t\t\t\tSONNY\n\t\tYeah.\n\n\t\t\t\tVOICE (O.S.)\n\t\tDo you recognize my voice?\n\n\t\t\t\tSONNY\n\t\tI think so.  Detective squad?\n\n\t\t\t\tVOICE (O.S.)\n\t\tRight.  Don't say my name, just\n\t\tlisten.  Somebody shot your father\n\t\toutside his place fifteen minutes\n\t\tago.\n\n\t\t\t\tSONNY\n\t\tIs he alive?\n\n\t\t\t\tVOICE (O.S.)\n\t\tI think so, but I can't get close\n\t\tenough.  There's a lot of blood.\n\t\tI'll try to find out more.\n\n\t\t\t\tSONNY\n\t\tFind out anything you can...you got\n\t\ta Grand coming.\n\t\t\t  (click)\n\n\n\tSONNY cradles the phone.  An incredible rage builds up in\n\thim, his face actually turning red.  He would like to rip\n\tthe phone to pieces in his bare hands.  Then he controls it.\n\tQuickly, he dials another number.\n\n\t\t\t\tSONNY\n\t\tTheresa, let me talk to Tom.  Not\n\t\tyet?  Have him call me as soon as\n\t\the gets home.\n\n\tHe hangs up.\n\n\t\t\t\tSANDRA (O.S.)\n\t\tSonny?  Sonny, who is it?\n\t\t\t  (she enters the room)\n\t\tWhat is it?\n\n\t\t\t\tSONNY\n\t\t\t  (calmly)\n\t\tThey shot the old man.\n\n\t\t\t\tSANDRA\n\t\tOh God...\n\n\t\t\t\tSONNY\n\t\tHoney...don't worry.  Nothing else\n\t\tis going to happen.\n\n\tThere is a POUNDING on the door.  A BABY starts crying.\n\n\t\t\t\tSANDRA\n\t\t\t  (really frightened)\n\t\tSONNY?\n\n\tSONNY reaches into a cabinet drawer, takes out a gun, and\n\tmoves quickly.  He opens the front door quickly.  It is\n\tCLEMENZA.  He enters, SONNY closes the door.  SANDRA goes to\n\tlook after the baby.\n\n\t\t\t\tCLEMENZA\n\t\t\t  (excited)\n\t\tYou heard about your father?\n\n\t\t\t\tSONNY\n\t\tYeah.\n\n\t\t\t\tCLEMENZA\n\t\tThe word is out in the streets that\n\t\the's dead.\n\n\t\t\t\tSONNY\n\t\tWhere the hell was Paulie, why\n\t\twasn't he with the Don?\n\n\t\t\t\tCLEMENZA\n\t\tPaulie's been a little sick all\n\t\twinter...he was home.\n\n\t\t\t\tSONNY\n\t\tHow many times did he stay home the\n\t\tlast couple of months?\n\n\t\t\t\tCLEMENZA\n\t\tMaybe three, four times.  I always\n\t\tasked Freddie if he wanted another\n\t\tbodyguard, but he said no.  Things\n\t\thave been so smooth the last ten\n\t\tyears...\n\n\t\t\t\tSONNY\n\t\tGo get Paulie, I don't care how\n\t\tsick he is.  Pick him up yourself,\n\t\tand bring him to my father's house.\n\n\t\t\t\tCLEMENZA\n\t\tThat's all?  Don't you want me to\n\t\tsend some people over here?\n\n\t\t\t\tSONNY\n\t\tNo, just you and Paulie.\n\n\tCLEMENZA leaves; SONNY moves to SANDRA, who sits on the\n\tcouch weeping quietly, comforting her BABY.\n\n\t\t\t\tSONNY\n\t\tA couple of our people will come to\n\t\tstay here.  Do whatever they say;\n\t\tI'm going over to the main house.\n\t\tIf you want me, use Pop's special\n\t\tphone.\n\n\tThe telephone rings again.  SONNY answers it.\n\n\t\t\t\tSONNY\n\t\tHello.\n\n\t\t\t\tSOLLOZZO (O.S.)\n\t\tSantino Corleone?\n\n\tSANDRA moves behind him, anxious to know who it is.  SONNY\n\tindicates that she be quiet.\n\n\t\t\t\tSONNY\n\t\tYeah.\n\n\t\t\t\tSOLLOZZO (O.S.)\n\t\tWe have Tom Hagen.  In about three\n\t\thours he'll be released with our\n\t\tproposition.  Don't do anything\n\t\tuntil you've heard what he has to\n\t\tsay.  You can only cause a lot of\n\t\ttrouble.  What's done is done.\n\t\t\t  (a pause)\n\t\tDon't lose that famous temper of\n\t\tyours.\n\n\t\t\t\tSONNY\n\t\t\t  (quietly)\n\t\tI'll wait.\n\n\tEXT NITE: MALL (WINTER 1945)\n\n\tFULL VIEW OF THE CORLEONE MALL.  It is night, but the\n\tcourtyard is bathed with white light from floodlights on the\n\ttops of all the houses.  It is very cold.  We see the figure\n\tof SONNY cross the Mall, and let himself into the main house.\n\n\tINT NITE: DON'S KITCHEN (WINTER 1945)\n\n\tSONNY walks into the empty, darkened house.  Then he calls\n\tout.\n\n\t\t\t\tSONNY\n\t\tMa?  Ma, where are you.\n\n\tThe kitchen door swings open.  He moves quickly and takes\n\ther by the arm.  He is deliberately calm.\n\n\t\t\t\tSONNY\n\t\tMa, I just got a call.  Pop's\n\t\thurt...I don't know how bad.\n\n\t\t\t\tMAMA\n\t\t\t  (quietly)\n\t\tSantino?  Have they killed him?\n\n\t\t\t\tSONNY\n\t\t\t  (almost in tears)\n\t\tWe don't know yet, Ma.\n\n\t\t\t\tMAMA\n\t\tI'll get dressed.  In case we can\n\t\tsee him...\n\n\tShe moves out of the kitchen, and continues upstairs.  SONNY\n\tturns the gas from the pan of peppers she was frying.  He\n\ttakes some bread without thinking, and dips it in the oil,\n\tand sloppily eats some of the peppers, as he moves into his\n\tfather's office.\n\n\tINT NITE: DON'S OFFICE (WINTER 1945)\n\n\tHe switches the lights on in the DON's office.  The massive\n\tdesk dominates the room.  SONNY moves quickly to the\n\ttelephone, pulling a small chair to the side of the desk,\n\tand dials a number.\n\n\t\t\t\tSONNY\n\t\tTessio...This is Santino Corleone.\n\t\tI want fifty reliable men out here.\n\n\t\t\t\tTESSIO (O.S.)\n\t\tI heard, Sonny...but what about\n\t\tClemenza's regime?\n\n\t\t\t\tSONNY\n\t\tI don't want to use Clemenza's\n\t\tpeople right now.  Understood?\n\n\tHe hangs up.  He moves quickly to a wall safe; operates the\n\tdial, and removes a small notebook.  He takes it back to the\n\tdesk, and runs over the list of numbers with his forefinger.\n\tWe follow the names, until the finger stops at one: LUCA\n\tBRASI.  SONNY dials the number.  There is no answer.\n\n\t\t\t\tSONNY\n\t\tLuca.\n\n\tINT NITE: BUILDING (WINTER 1945)\n\n\tThe interior of an abandoned building.  SEVERAL MEN in suits\n\tand ties sit around in the booths.\n\n\tHAGEN sits in one: SOLLOZZO sits across from him.\n\n\t\t\t\tSOLLOZZO\n\t\tI know you're not in the muscle end\n\t\tof the family--so I don't want you\n\t\tto be afraid.  I want you to help\n\t\tthe Corleones and I want you to\n\t\thelp me.\n\n\tHAGEN's hands are trembling as he tries to put a cigarette\n\tin his mouth.  ONE of the BUTTON MEN brings a bottle of rye\n\tto the table, and pours a little into a delicate, flowered\n\tchina cup.  HAGEN sips gratefully.\n\n\t\t\t\tSOLLOZZO\n\t\tYour boss is dead...\n\n\tHAGEN is overwhelmed: actual tears spring to his eyes.\n\tSOLLOZZO pauses respectfully.\n\n\t\t\t\tSOLLOZZO\n\t\t\t  (pushing the bottle)\n\t\tHave some more.  We got him outside\n\t\this office, just before I picked\n\t\tyou up.  You have to make the peace\n\t\tbetween me and Santino.\n\n\tHAGEN still is focused on the grief of losing the old man.\n\n\t\t\t\tSOLLOZZO\n\t\tSonny was hot for my deal, right?\n\t\tYou know it's the smart thing to\n\t\tdo, too.  I want you to talk Sonny\n\t\tinto it.\n\n\t\t\t\tHAGEN\n\t\t\t  (pulling himself together)\n\t\tSonny will come after you with\n\t\teverything he's got.\n\n\tSOLLOZZO rises, impatiently.\n\n\t\t\t\tSOLLOZZO\n\t\tThat's going to be his first\n\t\treaction.  You have to talk some\n\t\tsense into him.  The Tattaglia\n\t\tfamily stands behind me with all\n\t\ttheir people.  The other New York\n\t\tFamilies will go along with anything\n\t\tthat prevents a full scale war.\n\n\tHe leans close to HAGEN.\n\n\t\t\t\tSOLLOZZO\n\t\tThe Don was slipping; in the old\n\t\tdays I could never have gotten to\n\t\thim.  Now he's dead, nothing can\n\t\tbring him back.  Talk to Sonny,\n\t\ttalk to the Caporegimes, Clemenza\n\t\tand Tessio...it's good business.\n\n\t\t\t\tHAGEN\n\t\tEven Sonny won't be able to call\n\t\toff Luca Brasi.\n\n\t\t\t\tSOLLOZZO\n\t\tI'll worry about Luca.  You take\n\t\tcare of Sonny and the other two kids.\n\n\t\t\t\tHAGEN\n\t\tI'll try...It's what the Don would\n\t\twant us to do.\n\n\t\t\t\tSOLLOZZO\n\t\t\t  (lifting his hands in\n\t\t\t  an expression of harmlessness)\n\t\tGood...then you can go...\n\t\t\t  (he escorts him to\n\t\t\t  the door)\n\t\tI don't like violence.  I'm a\n\t\tbusinessman, and blood is a big\n\t\texpense.\n\n\tHe opens the door; they step out together.\n\n\tEXT NITE: BUILDING\n\n\tHAGEN, SOLLOZZO exit.\n\n\tBut a car pulls up, and ONE of SOLLOZZO'S MEN rushes out.\n\tHe indicates with some urgency that he wants to talk to\n\tSOLLOZZO in private.\n\n\tThen SOLLOZZO moves with a grave expression.  He opens the\n\tdoor, indicating that HAGEN should be led back in.\n\n\t\t\t\tSOLLOZZO\n\t\tThe old man is still alive.  Five\n\t\tbullets in his Sicilian hide and\n\t\the's still alive.\n\t\t\t  (he gives a fatalistic\n\t\t\t  shrug)\n\t\tBad luck for me, bad luck for you.\n\n\tEXT NITE: MALL (WINTER 1945)\n\n\tMICHAEL driving during the night.  There is a little fog in\n\tthe air, and moisture has formed on the windshield, making\n\tit difficult to see well.  The wipers move across the view,\n\tas the gate of the Corleone Mall appears before us, still\n\tdecorated for Christmas.  The courtyard is bathed with white\n\tfloodlight, giving this place a cold and isolated look.  The\n\tnarrow entrance mouth of the Mall is sealed off with a link\n\tchain.  There are strange cars parked along the curving\n\tcement walk.  SEVERAL MEN are congregated about the gate and\n\tchain; ONE of them approaches MICHAEL's car.\n\n\t\t\t\tMAN\n\t\tWho're you?\n\n\tANOTHER peeks his ugly face almost right up to MICHAEL, and\n\tthen turns.\n\n\t\t\t\tMAN 2\n\t\tIt's the Don's kid; take the car,\n\t\tI'll bring him inside.\n\n\tThe FIRST MAN opens the car door, and MICHAEL steps out.\n\n\tINT NITE: HALL (WINTER 1945)\n\n\tThe Hallway of the main house is filled with MEN MICHAEL\n\tdoesn't recognize.  They pay little attention to him.  Most\n\tof them are waiting; sitting uncomfortably; no one is talking.\n\n\tINT NITE: DON'S LIVING ROOM (WINTER 1945)\n\n\tMICHAEL moves into the living room; there is a Christmas\n\ttree, and countless greeting cards taped to the walls.\n\n\tTHERESA HAGEN is sitting stiffly on the sofa, smoking a\n\tcigarette; on the coffee table in front of her is a water\n\tglass half filled with whiskey.  On the other side of the\n\tsofa sits CLEMENZA; his face is impassive, but he is\n\tsweating, and the cigar in his hand glistens slickly black\n\twith his saliva.  PAULIE GATTO sits tensely and alone on the\n\tother side of the room.  CLEMENZA sees MICHAEL, looks up at\n\thim.\n\n\t\t\t\tCLEMENZA\n\t\tYour mother's at the hospital with\n\t\tthe old man: He's gonna pull through.\n\n\tMICHAEL nods his relief.\n\n\t\t\t\tMICHAEL\n\t\tThanks.\n\n\tHe moves to THERESA.\n\n\t\t\t\tMICHAEL\n\t\t\t  (gently)\n\t\tYou heard from Tom yet?\n\n\tWithout looking up, she clings to him for a moment, and\n\ttrembles.  Occasionally, STRANGE MEN will cross through the\n\troom; everyone speaks in a whisper.\n\n\t\t\t\tMICHAEL\n\t\t\t  (taking her hand)\n\t\tC'mon.\n\n\tHe leads her into his father's office without knocking.\n\n\tINT NITE: DON'S OFFICE (WINTER 1945)\n\n\tSONNY and TESSIO are huddled around a yellow pad.  They look\n\tup, startled.\n\n\t\t\t\tSONNY\n\t\tDon't worry, Theresa; they just\n\t\twant to give Tom the proposition,\n\t\tthen they're going to turn him loose.\n\n\tHe reassuringly hugs THERESA, and then to MICHAEL's surprise,\n\the kisses him on the cheek.\n\n\t\t\t\tSONNY\n\t\tI was worried when we couldn't get\n\t\tin touch with you in that hick town.\n\n\t\t\t\tMICHAEL\n\t\tHow's Mom?\n\n\t\t\t\tSONNY\n\t\tGood.  She's been through it before.\n\t\tMe too.  You were too young to know\n\t\tabout it.  You better wait outside;\n\t\tthere're some things you shouldn't\n\t\thear.\n\n\t\t\t\tMICHAEL\n\t\tI can help you out...\n\n\t\t\t\tSONNY\n\t\tOh no you can't, the old man'd be\n\t\tsore as hell if I let you get mixed\n\t\tup in this.\n\n\t\t\t\tMICHAEL\n\t\tJesus Christ, he's my father, Sonny.\n\n\t\t\t\tSONNY\n\t\tTheresa.\n\n\tShe understands, and leaves them alone.\n\n\t\t\t\tSONNY\n\t\tAll right, Mikey...who do we have\n\t\tto hit, Clemenza or Paulie?\n\n\t\t\t\tMICHAEL\n\t\tWhat?\n\n\t\t\t\tSONNY\n\t\tOne of them fingered the old man.\n\n\tMICHAEL didn't realize that the men waiting outside were on\n\ttrial for their lives.\n\n\t\t\t\tMICHAEL\n\t\tClemenza?  No, I don't believe it.\n\n\t\t\t\tSONNY\n\t\tYou're right, kid, Clemenza is okay.\n\t\tIt was Paulie.\n\n\t\t\t\tMICHAEL\n\t\tHow can you be sure?\n\n\t\t\t\tSONNY\n\t\tOn the three days Paulie was sick\n\t\tthis month, he got calls from a\n\t\tpayphone across from the old man's\n\t\tbuilding.  We got people in the\n\t\tphone company.\n\t\t\t  (he shrugs)\n\t\tThank God it was Paulie...we'll\n\t\tneed Clemenza bad.\n\n\tMICHAEL is just realizing the gravity and extent of the\n\tsituation.\n\n\t\t\t\tMICHAEL\n\t\tIs it going to be all-out war, like\n\t\tlast time?\n\n\t\t\t\tSONNY\n\t\tUntil the old man tells me different.\n\n\t\t\t\tMICHAEL\n\t\tThen wait, Sonny.  Talk to Pop.\n\n\t\t\t\tSONNY\n\t\tSollozzo is a dead man, I don't\n\t\tcare what it costs.  I don't care\n\t\tif we have to fight all the five\n\t\tfamilies in New York.  The Tattaglia\n\t\tfamily's going to eat dirt.  I\n\t\tdon't care if we all go down\n\t\ttogether.\n\n\t\t\t\tMICHAEL\n\t\t\t  (softly)\n\t\tThat's not how Pop would have\n\t\tplayed it.\n\n\t\t\t\tSONNY\n\t\tI know I'm not the man he was.  But\n\t\tI'll tell you this and he'll tell\n\t\tyou too.  When it comes to real\n\t\taction, I can operate as good as\n\t\tanybody short range.\n\n\t\t\t\tMICHAEL\n\t\t\t  (calmly)\n\t\tAll right, Sonny.  All right.\n\n\t\t\t\tSONNY\n\t\tChrist, if I could only contact Luca.\n\n\t\t\t\tMICHAEL\n\t\tIs it like they say?  Is he that\n\t\tgood?\n\n\tOutside, we HEAR THERESA cry out, almost a scream of relief.\n\tThen open the door and rush out.\n\n\tEveryone is standing: in the doorway, TOM HAGEN is wrapped\n\tin a tight embrace with his WIFE.\n\n\t\t\t\tHAGEN\n\t\tIf I plead before the Supreme\n\t\tCourt, I'll never do better than I\n\t\tdid tonight with that Turk.\n\n\tEXT NITE: MALL, FEATURING DON'S HOUSE (WINTER 1945)\n\n\tThe windows of the main house are dark except for the DON's\n\tstudy.  It stands out against the cold, dark night.\n\n\tINT NITE: DON'S LIVING ROOM (WINTER 1945)\n\n\tThe living room is empty, save for PAULIE GATTO sitting on\n\tthe edge of the sofa.  The clock reads: 4:00 a.m.\n\n\tINT NITE: DON'S OFFICE (WINTER 1945)\n\n\tSONNY, MICHAEL, HAGEN, CLEMENZA and TESSIO; all exhausted,\n\tin shirtsleeves, about to fall asleep.  It is four in the\n\tmorning; there is evidence of many cups of coffee and many\n\tsnacks.  They can barely talk anymore.\n\n\t\t\t\tHAGEN\n\t\tIs the hospital covered?\n\n\t\t\t\tSONNY\n\t\tThe cops have it locked in and I\n\t\tgot my people there visiting Pop\n\t\tall the time.  What about the hit\n\t\tlist.\n\n\tHAGEN widens his sleepy eyes, and looks at the yellow pad.\n\n\t\t\t\tHAGEN\n\t\tToo much, too far, too personal.\n\t\tThe Don would consider this all\n\t\tpurely a business dispute: Get rid\n\t\tof Sollozzo, and everything falls\n\t\tin line.  YOU don't have to go\n\t\tafter the Tattaglias.\n\n\tCLEMENZA nods.\n\n\t\t\t\tHAGEN\n\t\tWhat about Luca?  Sollozzo didn't\n\t\tseem worried about Luca.  That\n\t\tworries me.\n\n\t\t\t\tSONNY\n\t\tIf Luca sold out we're in real\n\t\ttrouble.\n\n\t\t\t\tHAGEN\n\t\tHas anyone been able to get in\n\t\ttouch with him?\n\n\t\t\t\tSONNY\n\t\tNo, and I've been calling all night.\n\t\tMaybe he's shacked up.\n\n\t\t\t\tHAGEN\n\t\tLuca never sleeps over with a broad.\n\t\tHe always goes home when he's\n\t\tthrough.  Mike, keep ringing Luca's\n\t\tnumber.\n\n\tMICHAEL, very tired, picks up the phone, and dials the\n\tnumber once again.  He can hear the phone ringing on the\n\tother end but no one answers.  Then hangs up.\n\n\t\t\t\tHAGEN\n\t\tKeep trying every fifteen minutes.\n\t\t\t  (exhausted)\n\n\n\t\t\t\tSONNY\n\t\tTom, you're the Consigliere, what\n\t\tdo we do if the old man dies?\n\n\t\t\t\tHAGEN\n\t\tWithout your father's political\n\t\tcontacts and personal influence,\n\t\tthe Corleone family loses half its\n\t\tstrength.  Without your father, the\n\t\tother New York families might wind\n\t\tup supporting Sollozzo, and the\n\t\tTattaglias just to make sure there\n\t\tisn't a long destructive war.  The\n\t\told days are over, this is 1946;\n\t\tnobody wants bloodshed anymore.  If\n\t\tyour father dies...make the deal,\n\t\tSonny.\n\n\t\t\t\tSONNY\n\t\t\t  (angry)\n\t\tThat's easy to say; it's not your\n\t\tfather.\n\n\t\t\t\tHAGEN\n\t\t\t  (quietly)\n\t\tI was as good a son to him as you\n\t\tor Mike.\n\n\t\t\t\tSONNY\n\t\tOh Christ Tom, I didn't mean it\n\t\tthat way.\n\n\t\t\t\tHAGEN\n\t\tWe're all tired...\n\n\t\t\t\tSONNY\n\t\tOK, we sit tight until the old man\n\t\tcan give us the lead.  But Tom, I\n\t\twant you to stay inside the Mall.\n\t\tYou too, Mike, no chances.  Tessio,\n\t\tyou hold your people in reserve,\n\t\tbut have them nosing around the\n\t\tcity.  The hospital is yours; I\n\t\twant it tight, fool-proof, 24 hours\n\t\ta day.\n\n\tThere is a timid knock on the door.\n\n\t\t\t\tSONNY\n\t\tWhat is it?\n\n\tPAULIE GATTO looks in.\n\n\t\t\t\tCLEMENZA\n\t\tI tol' you to stay put, Paulie...\n\n\t\t\t\tPAULIE\n\t\tThe guy at the gate's outside...says\n\t\tthere's a package...\n\n\t\t\t\tSONNY\n\t\tTessio, see what it is.\n\n\tTESSIO gets up, leaves.\n\n\t\t\t\tPAULIE\n\t\tYou want me to hang around?\n\n\t\t\t\tSONNY\n\t\tYeah.  Hang around.\n\n\t\t\t\tPAULIE\n\t\tOutside?\n\n\t\t\t\tCLEMENZA\n\t\tOutside.\n\n\t\t\t\tPAULIE\n\t\tSure.\n\n\tHe closes the door.\n\n\t\t\t\tSONNY\n\t\tClemenza.  You take care of Paulie.\n\t\tI don't ever want to see him again.\n\t\tUnderstood?\n\n\t\t\t\tCLEMENZA\n\t\tUnderstood.\n\n\t\t\t\tSONNY\n\t\tOkay, now you can move your men\n\t\tinto the Mall, replace Tessio's\n\t\tpeople.  Mike, tomorrow you take a\n\t\tcouple of Clemenza's people and go\n\t\tto Luca's apartment and wait for\n\t\thim to show.  That crazy bastard\n\t\tmight be going after Sollozzo right\n\t\tnow if he's heard the news.\n\n\t\t\t\tHAGEN\n\t\tMaybe Mike shouldn't get mixed up\n\t\tin this so directly.  You know the\n\t\told man doesn't want that.\n\n\t\t\t\tSONNY\n\t\tOK forget it, just stay on the phone.\n\n\tMICHAEL is embarrassed to be so protected.  He dials Luca\n\tBrasi's number once again.  The ring repeats, but no one\n\tanswers.\n\n\tTESSIO comes back, carrying Luca Brasi's bullet-proof vest\n\tin his hand.  He unwraps it; there is a large fish wrapped\n\tinside.\n\n\t\t\t\tCLEMENZA\n\t\tA Sicilian message: Luca Brasi\n\t\tsleeps with the fishes.\n\n\tINT. NITE: NIGHTCLUB (WINTER 1945)\n\n\tLUCA sits at the Bar of the Tattaglia Nightclub, as we\n\tremember him.  BRUNO TATTAGLIA had just patted his hand.\n\tLUCA looks up at him.\n\n\tThen SOLLOZZO pats the other hand, almost affectionately.\n\tLUCA is just about to twist his hands away, when they both\n\tclamp down as hard as they can.  Suddenly, a garrote is\n\tthrown around his neck, and pulled violently tight.  His\n\tface begins to turn to purple blotches, and then totally\n\tpurple, right before our eyes; his tongue hangs out, in a\n\tfar more extreme way than a normal tongue could.  His eyes\n\tbulge.\n\n\tONE of the MEN looks down at him in disgust as LUCA's\n\tstrength leaves him.\n\n\t\t\t\tBRUNO\n\t\t\t  (making an ugly face)\n\t\tOh Christ...all over the floor.\n\n\tSOLLOZZO lets LUCA's hand go with a victorious smile on his\n\tface.\n\n\tLUCA falls to the floor.\n\n\t\t\t\tSOLLOZZO\n\t\tThe Godfather is next.\n\n\t----------------------------------------FADE OUT--------\n\n\tFADE IN:\n\n\tEXT DAY: CLEMENZA'S HOUSE (WINTER 1945)\n\n\tMorning in a simple Brooklyn suburb.  There are rows of\n\tpleasant houses; driveway after driveway, down the block.  A\n\tdark, somber young man of thirty-one or two walks with a\n\tnoticeable limp down the sidewalk, and rings the bell.  This\n\tis ROCCO LAMPONE.  The woman of the house, MRS. CLEMENZA,\n\ttalks to him through the screen door, and then points to the\n\tside of the house.  ROCCO moves to the garage, which is\n\tspecially heated, and in which CLEMENZA is busy at work\n\twashing a shiny brand new Lincoln.  LAMPONE admires the car.\n\n\t\t\t\tLAMPONE\n\t\tNice.\n\n\t\t\t\tCLEMENZA\n\t\tCrazy Detroit delivered it with a\n\t\twooden bumper.  They're going to\n\t\tsend me the chrome bumpers in a\n\t\tcouple months.  I waited two years\n\t\tfor this car to come with wooden\n\t\tbumpers!\n\n\tHe scrubs and polishes with great affection.\n\n\t\t\t\tCLEMENZA\n\t\tToday you make your bones on Paulie.\n\t\tYou understand everything?\n\n\t\t\t\tLAMPONE\n\t\tSure.\n\n\tAs he scrubs around the glove compartment, he opens it,\n\tunwraps a gun and gives it to LAMPONE.\n\n\t\t\t\tCLEMENZA\n\t\t.22 soft-nosed load.  Accurate up\n\t\tto five feet.\n\n\tLAMPONE expertly puts the gun away.  GATTO's car pulls into\n\tthe driveway, and he sounds the horn.\n\n\tThe two men walk to the car.  GATTO is driving, a bit\n\tnervous, like he doesn't know what is up.  LAMPONE gets in\n\tthe rear seat; CLEMENZA in the front, making a grunt of\n\trecognition.  He looks at his wristwatch, as though wanting\n\tto chide PAULIE for being late.  PAULIE flinches a little\n\twhen he sees LAMPONE will ride behind him; he half turns:\n\n\t\t\t\tPAULIE\n\t\tRocco, sit on the other side.  A\n\t\tbig guy like you blocks my rearview\n\t\tmirror.\n\n\tCLEMENZA turns sourly to PAULIE.\n\n\t\t\t\tCLEMENZA\n\t\tGoddamn Sonny.  He's running scared.\n\t\tHe's already thinking of going to\n\t\tthe mattresses.  We have to find a\n\t\tplace on the West Side.  Paulie,\n\t\tyou know a good location?\n\n\tPAULIE relaxes a bit; he thinks he's off any possible hook\n\the was on.  Also there's the money he can make by selling\n\tSollozzo any secret location.\n\n\t\t\t\tPAULIE\n\t\tI'll think about it.\n\n\t\t\t\tCLEMENZA\n\t\t\t  (grunting)\n\t\tDrive while you thinking; I wanna\n\t\tget to the City this month!\n\n\tThe car pulls out.\n\n\tEXT DAY: PAULIE'S CAR - ON ROAD (WINTER 1945)\n\n\tInside PAULIE drives; and CLEMENZA sits in a grump.  OUR\n\tVIEW does not show LAMPONE in the rear seat.\n\n\tEXT DAY: PAULIE'S CAR AT TUNNEL (WINTER 1945)\n\n\tThe Car crosses to the Midtown Tunnel in the late Winter\n\tlight.\n\n\tINT DAY: PAULIE'S CAR IN TUNNEL (WINTER 1945)\n\n\tInside the tunnel; GATTO doesn't like not seeing LAMPONE.\n\tHe tries to adjust his rearview mirror to catch a glimpse of\n\thim.\n\n\t\t\t\tCLEMENZA\n\t\tPay attention!\n\n\tEXT DAY: PAULIE'S CAR AT MATTRESS (WINTER 1945)\n\n\tThe car is parked in the City.  PAULIE comes down from an\n\tavailable apartment and gets back into the car.\n\n\t\t\t\tPAULIE\n\t\tGood for ten men...\n\n\t\t\t\tCLEMENZA\n\t\tOK, go to Arthur Avenue; I'm\n\t\tsuppose to call when I found\n\t\tsomethin'.\n\n\tThe car pulls off.\n\n\tEXT DAY: RESTAURANT (WINTER 1945)\n\n\tNew part of the city; the car pulls up in a parking lot.\n\tCLEMENZA get outs, glances at LAMPONE, then to PAULIE.\n\n\t\t\t\tCLEMENZA\n\t\tYou wait; I'll call.\n\n\tHe walks, tucking his shirt into his pants, around the\n\tcorner and enters the Luna Restaurant.\n\n\tINT DAY: RESTAURANT (WINTER 1945)\n\n\tCLEMENZA enters the little restaurant, sits down at a table.\n\tThe WAITERS know him; immediately put a bottle of wine, some\n\tbread--and then a plate of veal on his table.  He eats.\n\n\tEXT DAY: RESTAURANT (WINTER 1945)\n\n\tCLEMENZA exits the restaurant, belches, adjusts his pants;\n\the is well fed.\n\n\tWe move with him around the corner, not knowing what to\n\texpect has happened to Paulie.\n\n\tThere is the car; PAULIE is still sitting behind the wheel,\n\tLAMPONE in the rear seat.  CLEMENZA steps in.\n\n\t\t\t\tCLEMENZA\n\t\tHe talked my ear off.  Want us to\n\t\tgo back to Long Beach; have another\n\t\tjob for us.  Rocco, you live in the\n\t\tCity, can we drop you off?\n\n\t\t\t\tLAMPONE (O.S.)\n\t\tAh, I left my car at your place.\n\n\t\t\t\tCLEMENZA\n\t\tOK, then you gotta come back.\n\n\tThe car pulls out.  By now, PAULIE is completely relaxed and\n\tsecure.\n\n\t\t\t\tPAULIE\n\t\tYou think we'll go for that last\n\t\tplace?\n\n\t\t\t\tCLEMENZA\n\t\tMaybe, or you gotta know now.\n\n\t\t\t\tPAULIE\n\t\tHoly cow, I don't gotta know nothing.\n\n\tEXT DAY: PAULIE'S CAR ON CAUSEWAY (WINTER 1945)\n\n\tThe car moves along the ready beach area of the causeway.\n\tInside, CLEMENZA turns to PAULIE.\n\n\t\t\t\tCLEMENZA\n\t\tPaulie, pull over.  I gotta take a\n\t\tleak.\n\n\tThe car pulls off the Causeway, into the reeds.  CLEMENZA\n\tsteps out of the car, OUR VIEW MOVING with him.\n\n\tHe turns his back three quarters from us (we can no longer\n\tsee the car), unzips, and we hear the sound of urine hitting\n\tthe ground.  We wait on this for a moment; and then there\n\tare two GUNSHOTS.  CLEMENZA finishes his leak, zips up and\n\tturns, moving back to the car.\n\n\tPAULIE is dead, bleeding from the mouth; the windows behind\n\thim are shattered.\n\n\t\t\t\tCLEMENZA\n\t\tLeave the gun.\n\n\tLAMPONE gets out, the two men walk through the reeds a few\n\tfeet where there is another car.  They get in, and drive off.\n\n\t---------------------------------------FADE OUT---------\n\n\tEXT DAY: MALL (WINTER 1945)\n\n\tHIGH ANGLE OF THE MALL.  It is late afternoon.  Many strange\n\tcars are parked on the nearby streets.  We can see the group\n\tof BUTTON MEN, stationed here and there, obviously sentries\n\twith concealed weapons.\n\n\tMICHAEL walks along in the rear yard.\n\n\tHe is bundled in a warm marine coat.  He looks at the\n\tstrange men, regarding them with an uncertain awe.  They\n\tlook back at him, at first suspiciously and then with the\n\trespect of his position.  He is like an exile Prince.  He\n\twanders past them, and hesitates and looks at the yard.\n\n\tA rusted set of garden swings; and other home playground\n\tequipment.  The basketball ring now half coming off.  This\n\tis where he was a child.  Then a shout.\n\n\t\t\t\tCLEMENZA (O.S.)\n\t\tMike.  Hey Mikey; telephone.\n\n\tCLEMENZA had shouted from the kitchen window.  MICHAEL\n\thurries into the house.\n\n\tINT DAY: DON'S KITCHEN (WINTER 1945)\n\n\tCLEMENZA is in the kitchen, cooking over an enormous pot.\n\tHe points to the kitchen wall phone which is hanging off the\n\thook.\n\n\t\t\t\tCLEMENZA\n\t\tSome dame.\n\n\tMICHAEL picks it up.\n\n\t\t\t\tMICHAEL\n\t\tHello.  Kay?\n\n\t\t\t\tKAY (O.S.)\n\t\tHow is your father?\n\n\t\t\t\tMICHAEL\n\t\tHe'll be OK.\n\n\t\t\t\tKAY (O.S.)\n\t\t\t  (pause)\n\t\tI love you.\n\n\tHe glances at the THUGS in the kitchen.  Tries to shield the\n\tphone.\n\n\t\t\t\tKAY (O.S.)\n\t\tI LOVE YOU.\n\n\t\t\t\tMICHAEL\n\t\tYeah Kay, I'm here.\n\n\t\t\t\tKAY (O.S.)\n\t\tCan you say it?\n\n\t\t\t\tMICHAEL\n\t\tHuh?\n\n\t\t\t\tKAY (O.S.)\n\t\tTell me you love me.\n\n\tMICHAEL glances at the HOODS at the kitchen table.  He curls\n\tup in a corner, and in a quarter voice:\n\n\t\t\t\tMICHAEL\n\t\tI can't...\n\n\t\t\t\tKAY (O.S.)\n\t\tPlease say it.\n\n\t\t\t\tMICHAEL\n\t\tLook.  I'll see you tonight, OK?\n\n\t\t\t\tKAY (O.S.)\n\t\tOK.\n\t\t\t  (click)\n\n\n\tCLEMENZA is getting ready to build a tomato sauce for all\n\tthe button men stationed around the house.\n\n\t\t\t\tCLEMENZA\n\t\tHow come you don't tell that nice\n\t\tgirl you love her...here, learn\n\t\tsomething... you may have to feed\n\t\tfifty guys some day.  You start\n\t\twith olive oil...fry some garlic,\n\t\tsee.  And then fry some sausage...or meat\n\t\tballs if you like...then you throw\n\t\tin the tomatoes, the tomato\n\t\tpaste...some basil; and a little\n\t\tred wine...that's my trick.\n\n\tSONNY peeks into the kitchen; sees CLEMENZA.\n\n\t\t\t\tSONNY\n\t\tYou take care of Paulie?\n\n\t\t\t\tCLEMENZA\n\t\tYou won't see Paulie anymore.  He's\n\t\tsick for good this winter.\n\n\tMICHAEL starts to leave.\n\n\t\t\t\tSONNY\n\t\tWhere are you going?\n\n\t\t\t\tMICHAEL\n\t\tTo the city.\n\n\t\t\t\tSONNY\n\t\t\t  (to Clemenza; dipping\n\t\t\t  bread into the sauce)\n\t\tSend some bodyguards.\n\n\t\t\t\tMICHAEL\n\t\tI don't need them, Sonny.  I'm just\n\t\tgoing to see Pop in the hospital.\n\t\tAlso, I got other things.\n\n\t\t\t\tCLEMENZA\n\t\tSollozzo knows Mike's a civilian.\n\n\t\t\t\tSONNY\n\t\tOK, but be careful.\n\n\tEXT NITE: CAR\n\n\tMICHAEL sits in the rear seat, calmly, as he is being driven\n\tinto the city.  THREE BUTTONMEN are crowded into the front\n\tseat.\n\n\tINT NITE: HOTEL LOBBY\n\n\tMICHAEL crosses the lobby, past lines of servicemen trying\n\tto book rooms.\n\n\tINT NITE: HOTEL\n\n\tMICHAEL and KAY eating a quiet dinner at the hotel.  He is\n\tpreoccupied, she's concerned.\n\n\t\t\t\tMICHAEL\n\t\tVisiting hour ends at eight thirty.\n\t\tI'll just sit with him; I want to\n\t\tshow respect.\n\n\t\t\t\tKAY\n\t\tCan I go to the hospital with you?\n\n\t\t\t\tMICHAEL\n\t\tI don't think so.  You don't want\n\t\tto end up on page 3 of the Daily\n\t\tNews.\n\n\t\t\t\tKAY\n\t\tMy parents don't read the Daily\n\t\tNews.  All right, if you think I\n\t\tshouldn't.  I can't believe the\n\t\tthings the papers are printing.\n\t\tI'm sure most of it's not true.\n\n\t\t\t\tMICHAEL\n\t\tI don't think so either.\n\t\t\t  (silence)\n\t\tI better go.\n\n\t\t\t\tKAY\n\t\tWhen will I see you again?\n\n\t\t\t\tMICHAEL\n\t\tI want you to go back to New\n\t\tHampshire...think things over.\n\n\tHe leans over her; kisses her.\n\n\t\t\t\tKAY\n\t\tWhen will I see you again?\n\n\t\t\t\tMICHAEL\n\t\tGoodbye.\n\n\tQuietly, he moves out the door.\n\n\tKAY lies on the bed a while, and then, to herself:\n\n\t\t\t\tKAY\n\t\tGoodbye.\n\n\tEXT NITE: DON'S HOSPITAL (WINTER 1945)\n\n\tA taxi pulls up in front of a hospital, marked clearly with\n\ta neon sign \"HOSPITAL--EMERGENCY.\"  MICHAEL steps out, pays\n\tthe fare...and then stops dead in his tracks.\n\n\tMICHAEL looks.\n\n\tHe sees the hospital in the night; but it is deserted.  He\n\tis the only one on the street.  There are gay, twinkling\n\tChristmas decorations all over the building.  He walks,\n\tslowly at first, and then ever so quickly, up the steps.  He\n\thesitates, looks around.  This area is empty.  He checks the\n\taddress on a scrap of paper.  It is correct.  He tries the\n\tdoor, it is empty.\n\n\tHe walks in.\n\n\tINT NITE: HOSPITAL LOBBY (WINTER 1945)\n\n\tMICHAEL stands in the center of an absolutely empty hospital\n\tlobby.  He looks to the right; there is a long, empty\n\tcorridor.  To the left: the same.\n\n\tHIGH FULL ANGLE, as MICHAEL walks through the desolated\n\tbuilding lit by eerie green neon lighting.  All we hear are\n\this sole footsteps.\n\n\tHe walks up to a desk marked \"INFORMATION\".  No one is there.\n\tHe moves quickly to a door marked \"OFFICE\"; swings into it;\n\tno one is there.  He looks onto the desk:  There is half a\n\tsandwich, and a half-filled bottle of coke.\n\n\t\t\t\tMICHAEL\n\t\tHello?  Hello?\n\n\tNow he knows something is happening, he moves quickly,\n\talertly.  MICHAEL walking down the hospital corridors; all\n\talone.  The floors have just been mopped.  They are still wet.\n\n\tINT NITE: HOSPITAL STAIRS\n\n\tNow he turns onto a staircase; ever quickening; up several\n\tflights.\n\n\tINT NITE: 4TH FLOOR CORRIDOR\n\n\tHe steps out onto the fourth floor.  He looks.  There are\n\tmerely empty corridors.  He takes out his scrap of paper;\n\tchecks it.  \"Room 4A.\"  Now he hurries, trying to follow the\n\tcode of hospital rooms; following the right arrows, quicker\n\tand quicker they flash by him.  Now he stops, looks up \"4A--\n\tCorleone\".\n\n\tThere is a special card table set up there with some\n\tmagazines...and some smoking cigarettes still in the\n\tashtray--but no detectives, no police, no bodyguards.\n\n\tINT NITE: DON'S ROOM 4A\n\n\tSlowly he pushes the door open, almost afraid at what he\n\twill find.  He looks.  Lit by the moonlight through the\n\twindow, he can see a FIGURE in the hospital bed alone in the\n\troom, and under a transparent oxygen tent.  All that can be\n\theard is the steady though strained breathing.  Slowly\n\tMICHAEL walks up to it, and is relieved to see his FATHER,\n\tsecurely asleep.  Tubes hang from a steel gallows beside the\n\tbed, and run to his nose and mouth.\n\n\t\t\t\tVOICE (O.S.)\n\t\tWhat are you doing here?\n\n\tThis startles MICHAEL; who almost jumps around.  It is a\n\tNURSE lit from the light behind her in the hallway.\n\n\t\t\t\tNURSE\n\t\tYou're not supposed to be here now.\n\n\tMICHAEL calms himself, and moves to her.\n\n\t\t\t\tMICHAEL\n\t\tI'm Michael Corleone--this is my\n\t\tfather.  What happened to the\n\t\tdetectives who were guarding him?\n\n\t\t\t\tNURSE\n\t\tOh your father just had too many\n\t\tvisitors.  It interfered with the\n\t\thospital service.  The police came\n\t\tand made them all leave just ten\n\t\tminutes ago.\n\t\t\t  (comfortingly)\n\t\tBut don't worry.  I look in on him.\n\n\t\t\t\tMICHAEL\n\t\tYou just stand here one minute...\n\n\tQuickly he moves to the telephone, dials a number.\n\n\t\t\t\tMICHAEL\n\t\tSonny...Sonny--Jesus Christ, I'm\n\t\tdown at the hospital.  I came down\n\t\tlate.  There's no one here.  None\n\t\tof Tessio's people--no detectives,\n\t\tno one.  The old man is completely\n\t\tunprotected.\n\n\t\t\t\tSONNY (O.S.)\n\t\tAll right, get him in a different\n\t\troom; lock the door from the inside.\n\t\tI'll have some men there inside of\n\t\tfifteen minutes.  Sit tight, and\n\t\tdon't panic.\n\n\t\t\t\tMICHAEL\n\t\t\t  (furiously, but kept inside)\n\t\tI won't panic.\n\n\tHe hangs up; returns to the NURSE...\n\n\t\t\t\tNURSE\n\t\tYou cannot stay here...I'm sorry.\n\n\t\t\t\tMICHAEL\n\t\t\t  (coldly)\n\t\tYou and I are going to move my\n\t\tfather right now...to another room\n\t\ton another floor...Can you\n\t\tdisconnect those tubes so we can\n\t\twheel the bed out?\n\n\t\t\t\tNURSE\n\t\tAbsolutely not!  We have to get\n\t\tpermission from the Doctor.\n\n\t\t\t\tMICHAEL\n\t\tYou've read about my father in the\n\t\tpapers.  You've seen that no one's\n\t\there to guard him.  Now I've just\n\t\tgotten word that men are coming to\n\t\tthis hospital to kill him.  Believe\n\t\tme and help me.\n\n\t\t\t\tNURSE\n\t\t\t  (frightened)\n\t\tWe don't have to disconnect them,\n\t\twe can wheel the stand with the bed.\n\n\tShe does so...and they perform the very difficult task of\n\tmoving the bed and the apparatus, out of the room.\n\n\tINT NITE: 4TH FLOOR HOSPITAL (WINTER 1945)\n\n\tThey roll the bed, the stand, and all the tubes silently\n\tdown the corridor.  We hear FOOTSTEPS coming up the stairs.\n\tMICHAEL hears them, stops.\n\n\t\t\t\tMICHAEL\n\t\tHurry, into there.\n\n\tThey push it into the first available room.  MICHAEL peeks\n\tout from the door.  The footsteps are louder; then they\n\temerge.  It is ENZO, NAZORINE's helper, carrying a bouquet\n\tof flowers.\n\n\t\t\t\tMICHAEL\n\t\t\t  (stepping out)\n\t\tWho is it?\n\n\t\t\t\tENZO\n\t\tMichael...do you remember me, Enzo,\n\t\tthe baker's helper to Nazorine, now\n\t\this son-in-law.\n\n\t\t\t\tMICHAEL\n\t\tEnzo, get out of here.  There's\n\t\tgoing to be trouble.\n\n\tA look of fear sweeps through ENZO's face.\n\n\t\t\t\tENZO\n\t\tIf there...will be trouble...I stay\n\t\twith you, to help.  I owe it to the\n\t\tGodfather.\n\n\tMICHAEL thinks, realizes he needs all the help he can get.\n\n\t\t\t\tMICHAEL\n\t\tGo outside; stand in front...I'll\n\t\tbe out in a minute.\n\n\tINT NITE: DON'S SECOND HOSPITAL ROOM (WINTER 1945)\n\n\tThey part.  MICHAEL moves into the hospital room where they\n\tput his FATHER.\n\n\t\t\t\tNURSE\n\t\t\t  (frightened)\n\t\tHe's awake.\n\n\tMICHAEL looks at the OLD MAN, his eyes are open, though he\n\tcannot speak.  MICHAEL touches his face tenderly.\n\n\t\t\t\tMICHAEL\n\t\tPop...Pop, it's me Michael.  Shhhh,\n\t\tdon't try to speak.  There are men\n\t\twho are coming to try to kill you.\n\t\tBut I'm with you...I'm with you\n\t\tnow...\n\n\tThe OLD MAN tries to speak...but cannot.  MICHAEL tenderly\n\tputs his finger to his FATHER's lips.\n\n\tEXT NITE: DON'S HOSPITAL STREET (WINTER 1945)\n\n\tOutside the hospital is empty save for a nervous ENZO,\n\tpacing back and forth brandishly the flowers as his only\n\tweapon.  MICHAEL exits the hospital and moves to him.  They\n\tboth stand under a lamppost in the cold December night.\n\tThey are both frightened; MICHAEL gives ENZO a cigarette,\n\tlights it.  ENZO's hands are trembling, MICHAEL's are not.\n\n\t\t\t\tMICHAEL\n\t\tGet rid of those and look like\n\t\tyou've got a gun in your pocket.\n\n\tThe windows of the hospital twinkle with Christmas\n\tdecorations.\n\n\t\t\t\tMICHAEL\n\t\tListen...\n\n\tWe HEAR the sound of a single automobile coming.  MICHAEL\n\tand ENZO look with fear in their eyes.  Then MICHAEL takes\n\tthe bouquet of flowers and stuffs them under his jacket.\n\tThey stand, hands in their pockets.\n\n\tA long low black car turns the corner and cruises by them.\n\tMICHAEL's and ENZO's faces are tough, impassive.  The car\n\tseems as though it will stop; and then quickly accelerates.\n\tMICHAEL and ENZO are relieved.  MICHAEL looks down; the\n\tBAKER's hands are shaking.  He looks at his own, and they\n\tare not.\n\n\tAnother moment goes by and we can hear the distant sound of\n\tpolice sirens.  They are clearly coming toward the hospital,\n\tgetting louder and louder.  MICHAEL heaves a sigh of relief.\n\n\tIn a second, a patrol car makes a screaming turn in front of\n\tthe hospital; then two more squad cars follow with uniformed\n\tPOLICE and DETECTIVES.  He smiles his relief and starts\n\ttoward them.  TWO huge, burly POLICEMEN suddenly grab his\n\tarms while ANOTHER frisks him.  A massive POLICE CAPTAIN,\n\tspattered with gold braid and scrambled eggs on his hat,\n\twith beefy red face and white hair seems furious.  This is\n\tMcCLUSKEY.\n\n\t\t\t\tMCCLUSKEY\n\t\tI thought I got all you guinea\n\t\thoods locked up.  Who the hell are\n\t\tyou and what are you doing here?\n\n\tANOTHER COP standing nearby:\n\n\t\t\t\tCOP\n\t\tHe's clean, Captain.\n\n\tMICHAEL studies McCLUSKEY closely.\n\n\t\t\t\tMICHAEL\n\t\t\t  (quietly)\n\t\tWhat happened to the detectives who\n\t\twere supposed to be guarding my\n\t\tfather?\n\n\t\t\t\tMCCLUSKEY\n\t\t\t  (furious)\n\t\tYou punk-hood.  Who the hell are\n\t\tyou to tell me my business.  I\n\t\tpulled them off.  I don't care how\n\t\tmany Dago gangsters kill each other.\n\t\tI wouldn't lift a finger to keep\n\t\tyour old man from getting knocked\n\t\toff.  Now get the hell out of here;\n\t\tget off this street you punk, and\n\t\tstay away from this hospital.\n\n\tMICHAEL stands quiet.\n\n\t\t\t\tMICHAEL\n\t\tI'll stay until you put guards\n\t\taround my father's room.\n\n\t\t\t\tMCCLUSKEY\n\t\tPhil, lock this punk up.\n\n\t\t\t\tA DETECTIVE\n\t\tThe Kid's clean, Captain...He's a\n\t\twar hero, and he's never been mixed\n\t\tup in the rackets...\n\n\t\t\t\tMCCLUSKEY\n\t\t\t  (furious)\n\t\tGoddam it, I said lock him up.  Put\n\t\tthe cuffs on him.\n\n\t\t\t\tMICHAEL\n\t\t\t  (deliberately, right\n\t\t\t  to McCLUSKEY's face,\n\t\t\t  as he's being handcuffed)\n\t\tHow much is the Turk paying you to\n\t\tset my father up, Captain?\n\n\tWithout any warning, McCLUSKEY leans back and hits MICHAEL\n\tsquarely on the jaw with all his weight and strength.\n\tMICHAEL groans, and lifts his hand to his jaw.  He looks at\n\tMcCLUSKEY; we are his VIEW and everything goes spinning, and\n\the falls to the ground, just as we see HAGEN and CLEMENZA'S\n\tMEN arrive.\n\n\t---------------------------------------FADE OUT---------\n\n\tEXT DAY: MALL (WINTER 1945)\n\n\tHIGH ANGLE VIEW of THE CORLEONE MALL.  The gateway now has a\n\tlong black car blocking it.  There are more BUTTON MEN\n\tstationed more formally; and some of them visibly carrying\n\trifles; those of the houses close to the courtyard have MEN\n\tstanding by open windows.  It is clear that the war is\n\tescalating.  A car pulls up and out get CLEMENZA, LAMPONE,\n\tMICHAEL and HAGEN.  MICHAEL's jaw is wired and bandaged.  He\n\tstops and looks up at the open window.  We can see MEN\n\tholding rifles.\n\n\t\t\t\tMICHAEL\n\t\tChrist, Sonny really means business.\n\n\tThey continue walking.  TESSIO joins them.  The various\n\tBODYGUARDS make no acknowledgment.\n\n\t\t\t\tCLEMENZA\n\t\tHow come all the new men?\n\n\t\t\t\tTESSIO\n\t\tWe'll need them now.  After the\n\t\thospital incident, Sonny got mad.\n\t\tWe hit Bruno Tattaglia four o'clock\n\t\tthis morning.\n\n\tINT DAY: DON'S HALLWAY\n\n\tThey enter the house past the scores of new and strange faces.\n\n\tINT DAY: DON'S OFFICE (WINTER 1945)\n\n\tSONNY is in the DON's office; he is excited and exuberant.\n\n\t\t\t\tSONNY\n\t\tI've got a hundred button men on\n\t\tthe streets twenty-four hours a day.\n\t\tIf Sollozzo shows one hair on his\n\t\tass he's dead.\n\n\tHe sees MICHAEL, and holds his bandaged face in his hand,\n\tkiddingly.\n\n\t\t\t\tSONNY\n\t\tMikey, you look beautiful!\n\n\t\t\t\tMICHAEL\n\t\tCut it out.\n\n\t\t\t\tSONNY\n\t\tThe Turk wants to talk!  The nerve\n\t\tof that son of a bitch!  After he\n\t\tcraps out last night he wants a meet.\n\n\t\t\t\tHAGEN\n\t\tWas there a definite proposal?\n\n\t\t\t\tSONNY\n\t\tSure, he wants us to send Mike to\n\t\tmeet him to hear his proposition.\n\t\tThe promise is the deal will be so\n\t\tgood we can't refuse.\n\n\t\t\t\tHAGEN\n\t\tWhat about that Tattaglias?  What\n\t\twill they do about Bruno?\n\n\t\t\t\tSONNY\n\t\tPart of the deal: Bruno cancels out\n\t\twhat they did to my father.\n\n\t\t\t\tHAGEN\n\t\tWe should hear what they have to say.\n\n\t\t\t\tSONNY\n\t\tNo, no Consiglere.  Not this time.\n\t\tNo more meetings, no more\n\t\tdiscussions, no more Sollozzo\n\t\ttricks.  Give them one message: I\n\t\tWANT SOLLOZZO.  If not, it's all\n\t\tout war.  We go to the mattresses\n\t\tand we put all the button men out\n\t\ton the street.\n\n\t\t\t\tHAGEN\n\t\tThe other families won't sit still\n\t\tfor all out war.\n\n\t\t\t\tSONNY\n\t\tThen THEY hand me Sollozzo.\n\n\t\t\t\tHAGEN\n\t\tCome ON Sonny, your father wouldn't\n\t\twant to hear this.  This is not a\n\t\tpersonal thing, this is Business.\n\n\t\t\t\tSONNY\n\t\tAnd when they shot me father...\n\n\t\t\t\tHAGEN\n\t\tYes, even the shooting of your\n\t\tfather was business, not personal...\n\n\t\t\t\tSONNY\n\t\tNo no, no more advice on how to\n\t\tpatch it up Tom.  You just help me\n\t\twin.  Understood?\n\n\tHAGEN bows his head; he is deeply concerned.\n\n\t\t\t\tHAGEN\n\t\tI found out about this Captain\n\t\tMcCluskey who broke Mike's jaw.\n\t\tHe's definitely on Sollozzo's\n\t\tpayroll, and for big money.\n\t\tMcCluskey's agreed to be the Turk's\n\t\tbodyguard.  What you have to\n\t\tunderstand is that while Sollozzo\n\t\tis guarded like this, he's\n\t\tinvulnerable.  Nobody has ever\n\t\tgunned down a New York Police\n\t\tCaptain.  Never.  It would be\n\t\tdisastrous.  All the five families\n\t\twould come after you Sonny; the\n\t\tCorleone family would be outcasts;\n\t\teven the old man's political\n\t\tprotection would run for cover.  So\n\t\tjust...take that into consideration.\n\n\t\t\t\tSONNY\n\t\t\t  (still fuming)\n\t\tMcCluskey can't stay with the Turk\n\t\tforever.  We'll wait.\n\n\t\t\t\tMICHAEL\n\t\tWe can't wait.  No matter what\n\t\tSollozzo say about a deal, he's\n\t\tfiguring out how to kill Pop.  You\n\t\thave to get Sollozzo now.\n\n\t\t\t\tSONNY\n\t\tThe kid's right.\n\n\t\t\t\tHAGEN\n\t\tWhat about McCluskey?\n\n\t\t\t\tMICHAEL\n\t\tLet's say now that we have to kill\n\t\tMcCluskey.  We'll clear that up\n\t\tthrough our Newspaper contacts later.\n\n\t\t\t\tSONNY\n\t\tGo on Mike.\n\n\t\t\t\tMICHAEL\n\t\tThey want me to go to the conference\n\t\twith Sollozzo.  Set up the meeting\n\t\tfor two days from now.  Sonny, get\n\t\tour informers to find out where the\n\t\tmeeting will be held.\n\t\tInsist it has to be a public place:\n\t\ta bar or restaurant at the height\n\t\tof the dinner hour.  So I'll feel\n\t\tsafe.  They'll check me when I meet\n\t\tthem so I won't be able to carry a\n\t\tweapon; but Clemenza, figure out a\n\t\tway to have one planted there for\n\t\tme.\n\t\t\t  (pause)\n\t\tThen I'll kill them both.\n\n\tEveryone in the room is astonished; they all look at MICHAEL.\n\tSilence.  SONNY suddenly breaks out in laughter.  He points\n\ta finger at MICHAEL, trying to speak.\n\n\t\t\t\tSONNY\n\t\tYou?  You, the high-class college\n\t\tkid.  You never wanted to get mixed\n\t\tup in the family business.  Now you\n\t\twanta gun down a police Captain and\n\t\tthe Turk just because you got\n\t\tslapped in the face.  You're taking\n\t\tit personal, it's just business and\n\t\the's taking it personal.\n\n\tNow CLEMENZA and TESSIO are also smiling; only HAGEN keeps\n\this face serious.\n\n\t\t\t\tMICHAEL\n\t\t\t  (angrily, but cold)\n\t\tSonny, it's all personal, and I\n\t\tlearned it from him, the old man,\n\t\tthe Godfather.  He took my joining\n\t\tthe Marines personal.  I take\n\t\tSollozzo trying to kill my father\n\t\tpersonal, and you know I'll kill\n\t\tthem Sonny.\n\n\tMICHAEL radiates danger...SONNY stops laughing.\n\n\tINT DAY: CLEMENZA'S CELLAR (WINTER 1945)\n\n\tCLOSE on a revolver.\n\n\t\t\t\tCLEMENZA (O.S.)\n\t\tIt's as cold as they come,\n\t\timpossible to trace.\n\t\t\t  (he turns it upside down)\n\t\tDon't worry about prints Mike, I\n\t\tput a special tape on the trigger\n\t\tand butt.  Here.\n\t\t\t  (he hands the gun to\n\t\t\t  another pair of hands)\n\t\tWhatsamatter?  Trigger too tight.\n\t\t\t  (it fires: very LOUD)\n\t\tI left it noisy, so it'll scare any\n\t\tpain-in-the-neck innocent bystander\n\t\taway.\n\n\tMICHAEL is alone with CLEMENZA in a cellar workshop.\n\n\t\t\t\tCLEMENZA\n\t\tJust let your hand drop to your\n\t\tside, and let the gun slip out.\n\t\tEverybody will still think you got\n\t\tit.  They'll be starin' at your\n\t\tface, see?  Then walk out of the\n\t\tplace real fast, but don't run.\n\t\tDon't look anybody directly in the\n\t\teye, but don't look away from them\n\t\tneither.  Hey, they'll be scared\n\t\tstiff o you, believe me.  Nobody's\n\t\tgonna bother with you.  Don't worry\n\t\tabout nothing; you'd be surprised\n\t\thow good these things go.  O.K.,\n\t\tput your hat on, let's see how you\n\t\tlook.  Helps with identification.\n\n\tThey put the hat on; CLEMENZA adjusts it.\n\n\t\t\t\tCLEMENZA\n\t\tMostly it gives witnesses an excuse\n\t\tto change their identification when\n\t\twe make them see the light.  Then\n\t\tyou take a long vacation and we\n\t\tcatch the hell.\n\n\t\t\t\tMICHAEL\n\t\tHow bad will it be?\n\n\t\t\t\tCLEMENZA\n\t\tProbably all the other families\n\t\twill line up against us.  But, it's\n\t\talright.  These things have to\n\t\thappen once every ten years or\n\t\tso...gets rid of the bad blood.\n\t\tYou gotta stop 'em at the beginning.\n\t\tLike they shoulda stopped Hitler at\n\t\tMunich, they shoulda never let him\n\t\tget away with that, they were just\n\t\tasking for big trouble...\n\n\tINT DAY: DON'S HALL & LIVING ROOM (WINTER 1945)\n\n\tMICHAEL steps into the foyer of the main house.  A card\n\ttable is set up with a man playing cards with three of the\n\tCorleone buttonmen.\n\n\tHe continues into the living room.  It's a mess.  SONNY\n\tasleep on the sofa.  On the coffee table are the remains of\n\ta take-out Chinese food dinner, and a half-empty bottle of\n\twhisky.  The radio is playing.\n\n\t\t\t\tMICHAEL\n\t\tWhy don't you stop living like a\n\t\tbum and get this place cleaned up.\n\n\t\t\t\tSONNY\n\t\tWhat are you, inspecting the\n\t\tbarracks?\n\t\t\t  (SONNY sits up with\n\t\t\t  his head in his hands)\n\t\tYou ready?  Did Clemenza tell you\n\t\tbe sure to drop the gun right away?\n\n\t\t\t\tMICHAEL\n\t\tA million times.\n\n\t\t\t\tSONNY\n\t\tSollozzo and McCluskey are going to\n\t\tpick you up in an hour and a half\n\t\ton Times Square, under the big\n\t\tCamels sign.\n\n\t\t\t\tHAGEN\n\t\tWe don't let Mike go until we have\n\t\tthe hostage, Sonny.\n\n\t\t\t\tCLEMENZA\n\t\tIt's okay...the hostage is outside\n\t\tplaying pinochle with three of my\n\t\tmen.\n\n\tThe phone rings in the DON's office.\n\n\t\t\t\tSONNY\n\t\tThat could be a Tattaglia informer\n\t\twith the meeting place.\n\n\tINT DAY: DON'S OFFICE (WINTER 1945)\n\n\tHAGEN has hurried into the Den to get the phone; the OTHERS\n\tmove in.\n\n\tHAGEN's on the phone; he writes something down.\n\n\t\t\t\tSONNY\n\t\tOne of Tattaglia's people?\n\n\t\t\t\tHAGEN\n\t\tNo.  Our informer in McCluskey's\n\t\tprecinct.  Tonight at 8:00 he\n\t\tsigned out for Louis' Restaurant in\n\t\tthe Bronx.  Anyone know it.\n\n\t\t\t\tTESSIO\n\t\tSure, I do.  It's perfect for us.\n\t\tA small family place with big\n\t\tbooths where people can talk in\n\t\tprivate.  Good food.  Everybody\n\t\tminds their business.  Perfect.\n\t\t\t  (he moves to the desk\n\t\t\t  and makes a crude drawing)\n\t\tThis is the entrance, Mike.  When\n\t\tyou finish just walk out and turn\n\t\tleft, then turn the corner.\n\t\tClemenza, you gotta work fast to\n\t\tplant the gun.  They got an old-\n\t\tfashioned toilet with a space\n\t\tbetween the water container and the\n\t\twall.  We can tape the gun behind\n\t\tthere.\n\n\t\t\t\tCLEMENZA\n\t\tMike, they're gonna frisk you in\n\t\tthe car.  You'll be clean so they\n\t\twon't worry 'bout nothing.  In the\n\t\trestaurant, wait and talk a while,\n\t\tand then ask permission to go.  See?\n\t\tThen when you come out, don't waste\n\t\ttime; don't sit down...you come out\n\t\tblasting.  And don't take chances.\n\t\tIn the head, two shots apiece.  And\n\t\tout as fast as your legs can move.\n\n\t\t\t\tSONNY\n\t\tI want somebody very good, very\n\t\tsafe to plant that gun.  I don't\n\t\twant my brother coming out of that\n\t\ttoilet with just his dick in his\n\t\thand.\n\n\t\t\t\tCLEMENZA\n\t\tThe gun will be there.\n\n\t\t\t\tSONNY\n\t\t\t  (to MICHAEL, warmly)\n\t\tYou're on, kid...I'll square it\n\t\twith Mom your not seeing her before\n\t\tyou left.  And I'll get a message\n\t\tto your girl friend when I think\n\t\tthe time is right.\n\n\t\t\t\tCLEMENZA\n\t\tWe gotta move...\n\n\t\t\t\tMICHAEL\n\t\tO.K.  How long do you think before\n\t\tI can come back?\n\n\t\t\t\tSONNY\n\t\tProbably a year...\n\n\t\t\t\tHAGEN\n\t\t\t  (starting to crack)\n\t\tJesus, I don't know...\n\n\t\t\t\tSONNY\n\t\tCan you do it Mike?\n\n\tMICHAEL moves out.\n\n\tEXT NITE: CAMELS SIGN (WINTER 1945)\n\n\tThe enormous \"CAMELS\" sign, puffing smoke, below it stands\n\tMICHAEL, dressed in a warm overcoat, and wearing the hat\n\tCLEMENZA had given him.  A long black car pulls around the\n\tcorner and slows before him.  The DRIVER, leaning over, open\n\tthe front door.\n\n\t\t\t\tDRIVER\n\t\tGet in, Mike.\n\n\tHe does, the car drives off.\n\n\tEXT NITE: SOLLOZZO'S CAR (WINTER 1945)\n\n\tInside the car, SOLLOZZO reaches his hand over the back seat\n\tand shakes MIKE's hand.\n\n\t\t\t\tSOLLOZZO\n\t\tI'm glad you came, Mike.  I hope we\n\t\tcan straighten everything out.  All\n\t\tthis is terrible, it's not the way\n\t\tI wanted things to happen at all.\n\t\tIt should never have happened.\n\n\t\t\t\tMICHAEL\n\t\tI want to settle things tonight.  I\n\t\twant my father left alone.\n\n\t\t\t\tSOLLOZZO\n\t\tHe won't be; I swear to you be my\n\t\tchildren he won't be.  Just keep an\n\t\topen mind when we talk.  I hope\n\t\tyou're not a hothead like your\n\t\tbrother, Sonny.  It's impossible to\n\t\ttalk business with him.\n\n\tMcCLUSKEY grunts.\n\n\t\t\t\tMCCLUSKEY\n\t\tHe's a good kid.  He's all right.\n\t\tTurn around, up on your knees,\n\t\tfacing me.\n\n\tHe gives MICHAEL a thorough frisk.\n\n\t\t\t\tMCCLUSKEY\n\t\tI'm sorry about the other night\n\t\tMike.  I'm getting too old for my\n\t\tjob, too grouchy.  Can't stand the\n\t\taggravation.  You know how it is.\n\t\tHe's clean.\n\n\tEXT NITE: SOLLOZZO'S CAR - WEST SIDE HIGHWAY (WINTER 1945)\n\n\tMICHAEL looks at the DRIVER and then ahead to see where\n\tthey're heading.\n\n\tThe car takes the George Washington Bridge.  MICHAEL is\n\tconcerned.\n\n\t\t\t\tMICHAEL\n\t\tWe're going to New Jersey?\n\n\t\t\t\tSOLLOZZO\n\t\t\t  (sly)\n\t\tMaybe.\n\n\tMICHAEL closes his eyes.\n\n\tEXT NITE: SOLLOZZO'S CAR ON G.W. BRIDGE (WINTER 1945)\n\n\tThe car speeds along the George Washington Bridge on its way\n\tto New Jersey.  Then suddenly it hits the divider,\n\ttemporarily lifts into the air, and bounces over into the\n\tlanes going back to New York.  It then hits it very fast, on\n\tthe way back to the city.\n\n\tEXT NITE: SOLLOZZO'S CAR (WINTER 1945)\n\n\tSOLLOZZO checks to see the cars that had been following, and\n\tthen leans to the DRIVER.\n\n\t\t\t\tSOLLOZZO\n\t\tNice work; I'll remember it.\n\n\tMICHAEL is relieved.\n\n\tEXT NITE: LUNA AZURA RESTAURANT (WINTER 1945)\n\n\tThe car pulls up in front of a little family restaurant in\n\tthe Bronx:  The \"LUNA AZURA\".  There is no one on the street.\n\tMICHAEL looks to see if the DRIVER is going to get out with\n\tthem.  He gets out, and opens the door.  SOLLOZZO, McCLUSKEY\n\tand MICHAEL get out; the DRIVER remains leaning against the\n\tcar.  They enter the restaurant.\n\n\tINT NITE: LUNA AZURA (WINTER 1945)\n\n\tA very small family restaurant with a mosaic tile floor.\n\tSOLLOZZO, MICHAEL and McCLUSKEY sit around a rather small\n\tround table near the center of the room.  There are empty\n\tbooths along the side walls; with a handful of CUSTOMERS,\n\tand ONE or TWO WAITERS.  It is very quiet.\n\n\t\t\t\tMCCLUSKEY\n\t\tIs the Italian food good here?\n\n\t\t\t\tSOLLOZZO\n\t\tTry the veal; it's the finest in\n\t\tNew York.\n\n\tThe solitary WAITER brings a bottle of wine to the table.\n\tThey watch him silently as he uncorks it and pours three\n\tglasses.  Then, when he leaves, SOLLOZZO turns to McCLUSKEY:\n\n\t\t\t\tSOLLOZZO\n\t\tI am going to talk Italian to Mike.\n\n\t\t\t\tMCCLUSKEY\n\t\tSure, you two go right ahead; I'll\n\t\tconcentrate on my veal and my\n\t\tspaghetti.\n\n\tSOLLOZZO now begins in rapid Sicilian.  MICHAEL listening\n\tcarefully and nodding every so often.  Then MICHAEL answers\n\tin Sicilian, and SOLLOZZO goes on.  The WAITER occasionally\n\tbrings food; and they hesitate while he is there; then go on.\n\tThen MICHAEL, having difficulty expressing himself in\n\tItalian, accidentally lapses into English.\n\n\t\t\t\tMICHAEL\n\t\t\t  (using English for emphasis)\n\t\tMost important...I want a sure\n\t\tguarantee that no more attempts\n\t\twill be made on my father's life.\n\n\t\t\t\tSOLLOZZO\n\t\tWhat guarantees can I give you?  I\n\t\tam the hunted one.  I've missed my\n\t\tchance.  You think too highly of\n\t\tme, my friend...I am not so\n\t\tclever...all I want if a truce...\n\n\tMICHAEL looks long and hard at SOLLOZZO, who is smiling\n\tholding his open hands up as if to say: \"I have no tricks up\n\tmy sleeve\".  Then he looks away and makes a distressed look\n\ton his face.\n\n\t\t\t\tSOLLOZZO\n\t\tWhat is it?\n\n\t\t\t\tMICHAEL\n\t\tIs it all right if I go to the\n\t\tbathroom?\n\n\tSOLLOZZO is intuitively suspicious.  He studies MICHAEL with\n\this dark eyes.  Then he thrusts his hand onto MICHAEL's\n\tthigh feeling in and around, searching for a weapon.\n\n\t\t\t\tMCCLUSKEY\n\t\tI frisked him; I've frisked\n\t\tthousands of young punks; he's clean.\n\n\tHe looks at a MAN sitting at a table opposite them;\n\tindicating the bathroom with his eyes.  The MAN nods,\n\tindicating no one is there.\n\n\t\t\t\tSOLLOZZO\n\t\tDon't take too long.\n\n\tMICHAEL gets up and calmly walks to the bathroom, and\n\tdisappears inside.\n\n\tINT NITE: LUNA AZURA TOILET (WINTER 1945)\n\n\tMICHAEL steps into the small bathroom; he is breathing very\n\thard.  He actually uses the urinal.  Then he washes his\n\thands with the bar of pink soap; and dries them thoroughly.\n\tThen he moves to the booth, up to the old-fashioned toilet.\n\tSlowly he reaches behind the water tank; he panics when he\n\tcannot feel the gun.  We see behind the tank his hand is\n\tjust a few inches from the gun...he gropes\n\tsearchingly...finally coming to rest on the gun.\n\n\tCLOSE ON MICHAEL; the feel of it reassures him.  Then he\n\tbreaks it loose from the tape holding it; he takes a deep\n\tbreath and shoves it under his waistband.  For some\n\tunexplainable reason he hesitates once again, deliberately\n\twashes his hands and dries them.  Then he goes out.\n\n\tINT NITE: LUNA AZURA (WINTER 1945)\n\n\tHe hesitates by the bathroom door; and looks at his table.\n\tMcCLUSKEY is eating a plate of spaghetti and veal.  SOLLOZZO\n\tturns around upon hearing the door, and looks directly at\n\tMICHAEL.  MICHAEL looks back.  Then he smiles and continues\n\tback to the table.  He sits down.\n\n\t\t\t\tMICHAEL\n\t\tNow I can talk.  I feel much better.\n\n\tThe MAN by the far wall had been stiff with attention; now\n\the too relaxes.  SOLLOZZO leans toward MICHAEL who sits down\n\tcomfortably and his hands move under the table and unbutton\n\this jacket.  SOLLOZZO begins to speak in Sicilian once again\n\tbut MICHAEL's heart is pounding so hard he can barely hear\n\thim.\n\n\tThe WAITER comes to ask about the order, SOLLOZZO turns to\n\tspeak, and without warning, MICHAEL shoves the table away\n\tfrom him with his left hand, and with his right hand puts\n\tthe gun right against SOLLOZZO's head, just touching his\n\ttemple.  He pulls the trigger, and we see part of SOLLOZZO's\n\thead blown away, and a spray of fine mist of blood cover the\n\tentire area.\n\n\tThe WAITER looks in amazement; suddenly his white jacket is\n\tsprayed and stained with blood.\n\n\tSOLLOZZO seems in a perpetual fall to the floor; through he\n\tseems to hang in space suspended.\n\n\tMICHAEL pivots, and looks:\n\n\tThere is McCLUSKEY, frozen, the fork with a piece of veal\n\tsuspended in air before his gaping mouth.\n\n\tMICHAEL fires; catching McCLUSKEY in his thick bulging\n\tthroat.  He makes a horrible, gagging, choking sound.  Then\n\tcoolly, and deliberately, MICHAEL fires again, fires right\n\tthrough McCLUSKEY's white-topped skull.\n\n\tThe air is filled with pink mist.\n\tMICHAEL swings toward the MAN standing by the bathroom wall.\n\tHe does not make a move, seemingly paralyzed.\n\tNow he carefully shows his hands to be empty.\n\tThe WAITER steps backward through the mist of blood, an\n\texpression of horror on his face.\n\tMICHAEL looks at his two victims:\n\tSOLLOZZO still in his chair, side of his body propped up by\n\tthe table.\n\tMcCLUSKEY finally falls from the chair to the table.\n\tMICHAEL is wildly at a peak.  He starts to move out.  His\n\thand: is frozen by his side, STILL GRIPPING THE GUN.\n\tHe moves, not letting the gun go.\n\n\tMICHAEL's face; frozen in its expression.\n\n\tHis hand: still holding the gun.\n\n\tHis face: finally he closes his eyes.\n\n\tHis hand relaxes, the gun falls to the floor with a dull thud.\n\n\tHe walks quickly out of the restaurant, looks back.\n\n\tHe sees a frozen tableau of the murder; as though it had\n\tbeen recreated in wax.\n\n\tThen he leaves.\n\n\t---------------------------------------FADE OUT---------\n\n\tFADE IN:\n\n\tINT DAY: MATTRESS (WINTER 1945)\n\n\tA MAN is his shirtsleeves plays a sentimental tune on an old\n\tupright piano, while his cigarette burns on the edge.\n\tANOTHER stands nearby, listening quietly.\n\n\tA little distance away, TEN MEN sit around a crude table,\n\tquietly eating.  They talk in low, relaxed voices, and there\n\tis an occasional laugh.\n\n\tROCCO LAMPONE stands by a window, which has been covered\n\twith a heavy-mesh wire grating, gazing out.\n\n\tA large bowl of pasta is passed, and the MEN eat heartily.\n\n\tThe sentimental tune is continued over the following:\n\n\tINT DAY: BODIES IN CAR (WINTER 1945)\n\n\tA MAN and a WOMAN, blood coming out of their noses, lie\n\tstill together in a bullet-riddled automobile.\n\n\tINT DAY: BODY IN BARBER SHOP (WINTER 1945)\n\n\tA MAN is covered by a sheet on the floor of a barber shop.\n\n\tINT DAY: MATTRESS\n\n\tTen mattresses are spread out around the otherwise empty\n\tliving room of an apartment.  THREE or FOUR MEN including\n\tCLEMENZA, are taking naps.\n\n\tAn arsenal of hand guns are spread out on a card table.\n\n\tThe MEN at the table continue their dinner; passing and\n\tpouring the wine.\n\n\tTrash is thrown in 2 or 3 garbage cans kept in the apartment.\n\n\tINT DAY: BODY IN OFFICE (WINTER 1945)\n\n\tA MAN, his clothes soaked in blood, lies on the floor of an\n\toffice building, dead, under an enormous portrait of Harry S.\n\tTruman.\n\n\tEXT DAY: BODY ON STOOP (WINTER 1945)\n\n\tANOTHER MAN, his trousers soaked in blood, lies spanning\n\tthree steps of a front stoop.\n\n\tINT NITE: MATTRESS (WINTER 1945)\n\n\tTESSIO, sits in a simple straight-backed chair, doing a\n\tcrossword puzzle.\n\n\tA thin, boyish BUTTON MAN, writes a letter.\n\n\tSix or seven empty mattresses, with tossed unmade blankets.\n\tCoffee cans beside them serve as ash trays.\n\n\tA MAN by the table pulls the cork on another bottle of\n\tRuffino, and wine is poured as the MEN eat.\n\n\tEXT DAY: BODY IN ALLEY (WINTER 1945)\n\n\tA CORPSE is half out of an overturned garbage can in a quiet\n\talley.\n\n\tINT DAY: BODY AT TABLE (WINTER 1945)\n\n\tA MAN in a formal jacket and tie is slumped over a table, in\n\ta pool of blood on the tablecloth.\n\n\tINT DAY: MATTRESS (WINTER 1945)\n\n\tA neatly stacked pile of newspapers in the corner of an\n\tapartment.  We catch a glimpse of one headline: \"Five Family\n\tWar...\"\n\n\tThe table.  The MEN are sitting around cracking nuts.  ONE\n\thas fallen asleep on his arms at the table.\n\n\tSEVERAL MEN are taking naps on the Mattresses.\n\n\tThe PIANO PLAYER finishes the tune with finesse.  Picks up\n\tand takes a drag from his cigarette.  The OTHER MAN nods\n\tappreciatively.\n\n\t\t\t\tMAN\n\t\tNice Augie...nice.\n\n\tEXT DAY: MANCINI BLDG. (SPRING '46)\n\n\tSeveral cars are parked in front of a pleasant New York\n\tapartment building.  We recognize a couple of SONNY's\n\tbodyguards loafing by the cars, pitching playing cards\n\tagainst the curb.\n\n\tInside the building, two others wait quietly by the rows of\n\tbrass mailboxes: they have been there quite awhile.\n\n\tUp one flight of stairs, a single man sits on the step,\n\tsmoking a cigarette.\n\n\tOne of the men by the mailboxes checks his pocketwatch,\n\twhich is attached to a key chain.  We HEAR the sound of a\n\tdoor opening; they look up.\n\n\tThe man sitting on the stop stands; and looks.\n\n\tSONNY backs out of an apartment, the arms of LUCY MANCINI\n\twrapped around him.  She doesn't want to let go of him; she\n\tdraws him back into the apartment for a moment, and then he\n\tcomes out alone, adjusting his clothes.\n\n\tHe jauntily skips down the steps, trailed by the bodyguard\n\ton the first floor, and moves outside toward his car.  The\n\tmen quickly take up their positions.  As he gets in his car:\n\n\t\t\t\tDRIVER\n\t\tPick up your sister?\n\n\t\t\t\tSONNY\n\t\tYeah.\n\n\tThe car drives off; accompanied and escorted by the\n\tbodyguards in their cars.\n\n\tINT DAY: CONNIE'S HALL (SPRING '46)\n\n\tHe knocks on the door.  No answer.  Then again.\n\n\t\t\t\tCONNIE'S VOICE\n\t\tWho is it?\n\n\t\t\t\tSONNY\n\t\tIt's me, Sonny.\n\n\tWe hear the bolt slide back, and see the door open.  SONNY\n\tenters, but CONNIE has quickly moved into the hallway, her\n\tback to him.\n\n\t\t\t\tSONNY\n\t\t\t  (tenderly)\n\t\tConnie, what is it?\n\n\tHe turns her around in his arms.\n\n\tHer face is swollen and bruised; and we can tell from her\n\trough, red eyes that she has been crying for a long time.\n\tAs soon as he realizes what's happened, his face goes red\n\twith rage.  She sees it coming, and clings to him, preventing\n\thim from running out of the apartment.\n\n\t\t\t\tCONNIE\n\t\t\t  (desperately)\n\t\tIt was my fault!  I started a fight\n\t\twith him and I tried to hit him so\n\t\the hit me.  He didn't even try to\n\t\thit me hard Sonny, I walked into it.\n\n\tSonny listens, and calms himself.  He touches her shoulder,\n\tthe thin silk robe.\n\n\t\t\t\tSONNY\n\t\tI'm goin' to have the doctor come\n\t\tover and take a look at you.\n\n\tHe starts to leave.\n\n\t\t\t\tCONNIE\n\t\tOh Sonny, please don't do anything.\n\t\tPlease don't.\n\n\tHe stops, and then laughs good naturedly.\n\n\t\t\t\tSONNY\n\t\tHey.  Con.  What'm I goin' to do?\n\t\tMake your kid a orphan before he's\n\t\tborn.\n\n\tShe laughs with him.  He kisses her reassuringly, and leaves.\n\n\tEXT DAY: CONNIE'S STREET\n\n\tCARLO settles down on the front steps of the 112th St.\n\t\"Book\" with SALLY RAGS and COACH, who have been drinking\n\tbeer out of glasses and a pitcher of beer from around the\n\tcorner.  The ball game is blaring from the radio; and the\n\tkids on the street are still playing stickball.\n\n\tCARLO has barely settled down, when the kids in the street\n\tsuddenly scatter, and a car comes screeching up the block\n\tand to a halt in front of the candy store.  The tires\n\tscream, and before it seems as though it has even stopped, a\n\tMAN comes hurtling out of the driver's seat, moving so fast\n\tthe everyone is paralyzed.  It is a moment before we\n\trecognize that it is SONNY.\n\n\tHis face is contorted with anger; in a split second he is on\n\tthe stoop and has CARLO by the throat.\n\n\tHe pulls CARLO away from the others, trying to get him down\n\tinto the street.  But CARLO reaches out for the iron railing,\n\tand hangs on, his hand in a lock, cringing away, trying to\n\thide his head and face in the hollow of the his shoulders.\n\tHis shirt is ripped away in SONNY's hand.\n\n\tSALLY RAGS and COACH, merely sit, watching, stunned.\n\n\tSONNY is pounding the cowered CARLO with all his strength,\n\tin a continuous monologue of indistinguishable cursing.  His\n\tblows are powerful; and begin to draw blood.\n\n\tThe kids who have been playing stickball, move up, watching\n\tin fascination.\n\n\tCARLO's hands are clenched tight around the railing.\n\n\tSONNY beats him mercilessly.\n\n\tNow SONNY's bodyguards' car pulls up, and they too become\n\tspectators.\n\n\tSONNY's tight fists are going down like hammers, into\n\tCARLO's face and body.\n\n\tCARLO's nose is bleeding profusely; but still he does\n\tnothing, other than hang onto the railing.\n\n\tSONNY grabs hold of CARLO's massive body, and tries to drag\n\thim off of the hold on the railing, his teeth clenched in\n\tthe effort.  Then he tries loosening CARLO's locked hands;\n\teven biting them.  CARLO screams but he does not let go.\n\n\tIt's clear that CARLO is much stronger than he is, and will\n\tnot be moved.  SONNY knees him in the mouth, and beats him\n\tmore; but he is exhausted.  Totally out of breath, he\n\tstammers haltingly to the bleeding CARLO.\n\n\t\t\t\tSONNY\n\t\tYou...bastard...You...hurt my\n\t\tsister... again...and I'll\n\t\tkill...you.\n\n\tHe wipes the sweat from his face, and then turns suddenly.\n\tand hurries back to the car, in a moment his car is gone,\n\tleaving even his bodyguards in confusion.  We notice ONE MAN\n\twith a sports jacket in the group of spectators especially\n\tinterested.\n\n\tCARLO finally relaxes the clenched, locked hands.  He slumps\n\tonto the stoop.\n\n\t---------------------------------------FADE OUT---------\n\n\tFADE IN:\n\n\tEXT DAY: MALL (SPRING 1946)\n\n\tHIGH ANGLE on the Corleone Mall.  It is a gray, rainy day.\n\tYoung BUTTON MEN in raincoats stand in quiet groups of\n\tvarious points around the main house and compound.  Things\n\thave changed; one house has been extensively enlarged; a new\n\tand secure gate house has been built.  Security measures\n\tthat had been make-shift and temporarily have now been made\n\ta permanent part of the Mall, evolving it into a Medieval\n\tFortress.  We notice a huge crater in the courtyard; the\n\tresult of a recent bomb attempt.  The house nearest the\n\tcrater is damaged by fire.\n\n\tA taxi arrives; KAY ADAMS steps out, huddled in a bright\n\tyellow raincoat; she lets the cab go, and hurries to the\n\tshelter of the gate house.\n\n\tThey are not expecting her, and ask her to wait while they\n\tcall the main house.\n\n\tKAY looks at the imposing, depressing Mall, while rain still\n\truns down onto her face.\n\n\tShe notices the bomb crater, and the fire damage; and the\n\tsullen faces of the BUTTON MEN.\n\n\tTOM HAGEN exits the Main House, and hurries toward her.\n\n\t\t\t\tHAGEN\n\t\tKay, we weren't expecting you.  You\n\t\tshould call...\n\n\t\t\t\tKAY\n\t\tI've tried calling and writing.  I\n\t\twant to reach Michael.\n\n\t\t\t\tHAGEN\n\t\tNobody knows where he is.  We know\n\t\the's all right, but that's all.\n\n\tKAY looks in the direction of the crater, filling with\n\trainwater.\n\n\t\t\t\tKAY\n\t\tWhat was that?\n\n\t\t\t\tHAGEN\n\t\tAn accident.  No one was hurt.\n\n\t\t\t\tKAY\n\t\tListen Tom, I let my cab go; can I\n\t\tcome in to call another one?\n\n\tTOM is clearly reluctant to involve her any more than he has\n\tto.\n\n\t\t\t\tHAGEN\n\t\tSure...I'm sorry.\n\n\tThey hurry through the rain and into the Main House.\n\n\tINT DAY: DON'S LIVING ROOM (SPRING 1946)\n\n\tIn the living room, KAY shakes the water from her coat and\n\ttakes her rainhat off.\n\n\t\t\t\tKAY\n\t\tWill you give this to him.\n\n\t\t\t\tHAGEN\n\t\tIf I accept that letter and you\n\t\ttold a Court of Law I accepted it,\n\t\tthey would interpret it as my\n\t\thaving knowledge of his whereabouts.\n\t\tJust wait Kay, he'll contact you.\n\n\tWe hear footsteps descending the staircase; MAMA CORLEONE\n\tenters the room; the OLD WOMAN squints at KAY, evaluating her.\n\n\t\t\t\tMAMA\n\t\tYou're Mikey's little girl.\n\n\tKAY nods yes; there are still tears in her eyes.\n\n\t\t\t\tMAMA\n\t\tYou eat anything?\n\n\tKAY shakes her head.\n\n\t\t\t\tMAMA\n\t\t\t  (to HAGEN)\n\t\tDisgrazia, you don't even give the\n\t\tpoor girl a cup of coffee?\n\n\tHAGEN shrugs helplessly; on an impulse, KAY quickly moves\n\ttoward MAMA, the letter extended.\n\n\t\t\t\tKAY\n\t\tWill you give this letter to Michael.\n\n\t\t\t\tHAGEN\n\t\tMama, no.\n\n\t\t\t\tMAMA\n\t\tYou tell me what to do?  Even he\n\t\tdon't tell me what to do.\n\n\tShe takes the letter from KAY, who is grateful and relieved.\n\n\t\t\t\tKAY\n\t\tWhy did they blame Michael?\n\n\t\t\t\tMAMA\n\t\tYou listen to me, you go home to\n\t\tyour family, and you find a good\n\t\tyoung man and get married.  Forget\n\t\tabout Mikey; he's no good for you,\n\t\tanymore.\n\n\tShe looks directly into KAY's eyes; and KAY understands what\n\tthat means.\n\n\tEXT DAY: DON'S HOSPITAL (SPRING 1946)\n\n\tA hospital in New York City.  POLICE and teams of PRIVATE\n\tDETECTIVES are stationed guarding the area.  An ambulance\n\twith a team of DETECTIVES and BUTTON-MEN GUARDS exit the\n\thospital with rifles in hand; followed by SEVERAL HOSPITAL\n\tASSISTANTS wheeling a hospital stretcher, presumably carrying\n\tthe DON.\n\n\tTESSIO and CLEMENZA emerge, with OTHER BUTTON MEN bringing\n\tup the rear.  HAGEN walks with the stretcher, and for a\n\tmoment they disappear behind the ambulance.  Then suddenly,\n\tsiren blasting, it speeds off, accompanied by dark low-slung\n\tcars.\n\n\tEXT DAY: MALL (SPRING 1946)\n\n\tThe Corleone Mall.\n\n\tEqually impressive security stands ready at the Corleone\n\tMall.  EXTRA BUTTON MEN, as well as SOME POLICE, and PRIVATE\n\tDETECTIVES.\n\n\tIt all seems to be under the supervision of ROCCO LAMPONE.\n\tAll is silent.  The WOMEN and CHILDREN, dressed in Sunday\n\tclothes, wait.\n\n\tEXT DAY: AMBULANCE (SPRING 1946)\n\n\tOne ambulance, speeding along the Grand Central Parkway,\n\tpreceded and followed by a dark car, each one carrying a\n\tteam of BUTTON MEN.\n\n\tSitting next to the DRIVER of the ambulance is a GUARD with\n\ta rifle on his lap.\n\n\tINT DAY: DON'S HALL (SPRING 1946)\n\n\tInside the Main CORLEONE House:\n\n\tHospital ORDERLIES carry the DON on his stretcher carefully\n\tunder the watchful eyes of CLEMENZA, TESSIO, LAMPONE and\n\tvarious GUARDS and BUTTON MEN.\n\n\tAll the CORLEONE family is here today: MAMA, FREDO, SANDRA,\n\tTHERESA, CONNIE, CARLO; the various CORLEONE CHILDREN.\n\n\tINT DAY: DON'S BEDROOM (SPRING 1946)\n\n\tThe DON is made comfortable in his room, which has all but\n\tbeen converted into a hospital room, with complete and\n\textensive equipment.  The various CHILDREN get a turn to\n\tkiss the OLD MAN, as he is made comfortable... and then\n\tSONNY indicates that all the CHILDREN, WOMEN, and CARLO\n\tshould leave.\n\n\tThey do, the door is closed.\n\n\tINT DAY: DON'S DINING ROOM (SPRING 1946)\n\n\tThe mood is quite happy downstairs, as the WOMEN prepare the\n\tSunday dinner, and set the table.\n\n\tCARLO sits alone among them, a frown on his face.\n\n\t\t\t\tCONNIE\n\t\tWhat's the matter, Carlo?\n\n\t\t\t\tCARLO\n\t\tShut up.\n\n\tINT DAY: DON'S BEDROOM (SPRING 1946)\n\n\tAll the MEN of the family stand around the hospital bed with\n\tgrim faces, SONNY and HAGEN closest to the OLD MAN.  The DON\n\tdoes not speak, yet he asks questions with his looks and\n\tglances, as clearly as if they were verbalized.  HAGEN is\n\tthe spokesman for the family.\n\n\t\t\t\tHAGEN\n\t\t...since McCluskey's killing, the\n\t\tpolice have cracked down on most of\n\t\tour operations...on the other\n\t\tfamilies too.  There's been a lot\n\t\tof bad blood.\n\n\tThe OLD MAN glances at SONNY.\n\n\t\t\t\tSONNY\n\t\tPop, they hit us and we hit them\n\t\tback.\n\n\t\t\t\tHAGEN\n\t\tWe put out a lot of material\n\t\tthrough our contacts in the\n\t\tNewspapers...about McCluskey's\n\t\tbeing tied up with Sollozzo in the\n\t\tDrug Rackets...things are starting\n\t\tto loosen up.\n\n\tThe OLD MAN nods.\n\n\t\t\t\tSONNY\n\t\tFreddie's gonna go to Las\n\t\tVegas...under the protection of Don\n\t\tFrancesco of L.A.  I want him to\n\t\trest...\n\n\t\t\t\tFREDO\n\t\tI'm goin' to learn the casino\n\t\tbusiness.\n\n\tThe DON nods approvingly.  Then he searches around the room\n\tfor a face he does not see.  HAGEN knows who he's looking for.\n\n\t\t\t\tHAGEN\n\t\tMichael...\n\t\t\t  (he takes a breath)\n\t\tIt was Michael who killed Sollozzo.\n\n\tThe DON closes his eyes, and then reopens them in anger and\n\trage.\n\n\t\t\t\tHAGEN\n\t\tHe's safe now...we're already\n\t\tworking on ways to bring him back.\n\n\tThe DON is very angry, he motions with a weak hand that they\n\tleave him alone.\n\n\tINT. DAY: DON'S STAIRS AND HALL (SPRING 1946)\n\n\tHAGEN is very upset as he comes down the Stairs; SONNY is\n\texpansive and optimistic.\n\n\t\t\t\tSONNY\n\t\tWe'll let the old man take it easy\n\t\tfor a couple of weeks.  I want to\n\t\tget things going good before he\n\t\tgets better.  What's the matter\n\t\twith you?\n\n\t\t\t\tHAGEN\n\t\tYou start operating, the five\n\t\tfamilies will start their raids\n\t\tagain.  We're at a stalemate Sonny,\n\t\tyour war is costing us a lot of\n\t\tmoney.\n\n\t\t\t\tSONNY\n\t\tNo more stalemate Tom, we got the\n\t\tsoldiers, we'll match them gun for\n\t\tgun if that's how they want it.\n\t\tThey know me for what I am, Tom--\n\t\tand they're scared of me.\n\n\t\t\t\tHAGEN\n\t\tYes.  That's true, you're getting a\n\t\thell of a reputation.\n\n\t\t\t\tSONNY\n\t\tWell it's war!  We might not be in\n\t\tthis shape if we had a real war-\n\t\ttime Consiglere, a Sicilian.  Pop\n\t\thad Genco, who do I have?\n\t\t\t  (TOM starts to leave)\n\t\tHey Tom, hey...hey.  It's Sunday,\n\t\twe're gonna have dinner.  Don't be\n\t\tsore.\n\n\tINT DAY: DON'S DINING ROOM (SPRING 1946)\n\n\tThe FAMILY, WIVES, CHILDREN and all sit around the table\n\tover Sunday dinner.  SONNY is at the head of the table.\n\n\tEXT DAY: MALL (SPRING 1946)\n\n\tSOME of the CORLEONE GRANDCHILDREN play in the enclosed\n\tMall, in the proximity of the BUTTON MEN stationed liberally\n\tby the gate.\n\n\tONE CHILD misses a ball, it rolls by the gate house.  A\n\tyoung BUTTON MAN scoops it up and throws it back, smiling.\n\n\t-----------------------------------------FADE OUT-------\n\n\tINT DAY: CONNIE'S APT. (SPRING 1946)\n\n\tCONNIE and CARLO's apartment.  She's in a slip, on the phone.\n\tWe HEAR the shower going in the bathroom.\n\n\t\t\t\tCONNIE\n\t\tWho is this?\n\n\t\t\t\tGIRL (O.S.)\n\t\t\t  (giggle)\n\t\tI'm a friend of Carlo's.  I just\n\t\twanted to tell him I can't see him\n\t\ttonight; I have to go out of town.\n\n\tCONNIE's face turns red.\n\n\t\t\t\tCONNIE\n\t\tYou lousy tramp bitch.\n\t\t\t  (click)\n\n\n\tShe slams the phone down; just as CARLO is coming out of the\n\tbathroom drying his golden body.\n\n\t\t\t\tCARLO\n\t\tWhat was that?\n\n\t\t\t\tCONNIE\n\t\tYour girl friend.  She says she\n\t\tcan't make it tonight.  You lousy\n\t\tbastard you have the nerve to give\n\t\tyour whores my telephone number.\n\t\tI'll kill you, you bastard!\n\n\tShe hauls off and punches him knowingly; he laughs, so then\n\tshe flings herself at him, kicking and scratching; her heavy\n\tbelly heaving under the thin slip.\n\n\t\t\t\tCARLO\n\t\t\t  (defending himself)\n\t\tYou're crazy.  She was kidding\n\t\taround; I don't know, some nut.\n\n\tHe pushes her aside, and moves into the bedroom to continue\n\tdressing.\n\n\t\t\t\tCONNIE\n\t\tYou're staying home.  You're not\n\t\tgoing out.\n\n\t\t\t\tCARLO\n\t\tOK, OK.  You gonna make me something\n\t\tto eat at least?\n\n\tThat calms her down; she stands there a moment, breathing\n\theavily; and then she nods, and goes into the kitchen, and\n\tstarts her wifely duties.\n\n\tCARLO is dressed; puts on some cologne; CONNIE appears in\n\tthe doorway.\n\n\t\t\t\tCONNIE\n\t\tThe food is on the table.\n\n\t\t\t\tCARLO\n\t\tI'm not hungry yet.\n\n\t\t\t\tCONNIE\n\t\tEat it, it's on the table.\n\n\t\t\t\tCARLO\n\t\tBa Fa Goulle.\n\n\t\t\t\tCONNIE\n\t\tBA FA GOULE YOU!\n\n\tShe turns deliberately, goes out into the kitchen.  A moment\n\tlater we begin to hear the sound of dishes breaking.  CARLO\n\tslowly walks out, where we can see CONNIE systematically\n\tsmashing all the dishes against the sink, sending the greasy\n\tveal and peppers all over the apartment floor.\n\n\t\t\t\tCARLO\n\t\tYou filthy guinea spoiled brat.\n\t\tClean it up or I'll kick your head\n\t\tin.\n\n\t\t\t\tCONNIE\n\t\tLike hell I will.\n\n\tShe stands there, solid, ready to punch him again.  Slowly,\n\the slides his belt out of his trousers, and doubles it in\n\this hand.\n\n\t\t\t\tCARLO\n\t\tClean it up!\n\n\tHe swings the belt against her heavy hips.  She moves back\n\tinto the kitchen, and gets a kitchen knife, and holds it\n\tready.\n\n\t\t\t\tCARLO\n\t\tEven the female Corleones are\n\t\tmurderers.\n\n\tHe puts the strap down on a table, and moves after her.  She\n\tmakes a sudden thrust at his groin, which he avoids.  He\n\tpulls the knife away, cutting his hand in the process.  She\n\tgets away momentarily, but he pursues her around the table,\n\tgets her; and starts to slap her in the face.\n\n\tShe breaks away from him, and rushes into the bedroom.\n\n\t\t\t\tCONNIE\n\t\tThe baby!  The baby!\n\n\tINT DAY: CONNIE'S BEDROOM  (SPRING 1946)\n\n\tShe runs into the bedroom; he follows.  She moves into a\n\tcorner, and then like a desperate animal, tries to hide\n\tunder the bed.\n\n\tHe reaches under, and pulls her out by the hair.\n\n\tHe slaps her in the face until she begins to weep; then he\n\tthrows her on the bed, contemptuously.  He grabs part of her\n\tthigh, pinching it very hard.\n\n\t\t\t\tCARLO\n\t\tYou're fat as a pig.\n\n\tThen he pushes her away, and walks out of the room, leaving\n\ther in tears.  She is crying; she pulls herself to the\n\tbedroom phone, and in a whisper:\n\n\t\t\t\tCONNIE\n\t\tMama...mama, it's Connie.  Mama, I\n\t\tcan't talk any louder.  No, I don't\n\t\twant to talk to Sonny.\n\n\tWe can tell that the phone has been passed to SONNY.\n\n\tINT DAY: DON'S KITCHEN (SPRING 1946)\n\n\tIn the kitchen at the Mall, MAMA cannot understand the\n\twhispering and she has given the phone to SONNY.\n\n\t\t\t\tSONNY\n\t\tYeah Connie.\n\n\t\t\t\tCONNIE (O.S.)\n\t\tSonny, just send a car to bring me\n\t\thome.  I'll tell you then, it's\n\t\tnothing Sonny, don't you come.\n\t\tSend TOM, please Sonny, it's\n\t\tnothing; I just want to come home.\n\n\tSONNY's face is turning red.\n\n\t\t\t\tSONNY\n\t\t\t  (in a controlled voice)\n\t\tYou wait there.  You just wait there.\n\n\tHe hangs up the phone; and just stands there for a moment.\n\n\t\t\t\tSONNY\n\t\t\t  (quietly)\n\t\tThat sonofabitch; that sonofabitch...\n\n\tHAGEN enters the room; he knows what is happening, knows he\n\tcannot interfere.\n\n\tEXT DAY: MALL\n\n\tSONNY leaves the house.  HAGEN moves to the outside mall\n\tjust as SONNY's car is driving off.  He moves to a group of\n\tBUTTON MEN.\n\n\t\t\t\tHAGEN\n\t\tGo after him.\n\n\tEXT DAY: CAUSEWAY (SPRING 1946)\n\n\tSONNY's car on the Jones Beach Causeway, speeds quickly by.\n\tAfter a pause, another car, with the CORLEONE BODYGUARDS, is\n\ttrailing.\n\n\tSONNY is driving; he is very angry.\n\n\tEXT NITE: TOLL BOOTHS (SPRING 1946)\n\n\tSONNY in his car; driving back.  Still breathing hard and\n\tstill furious.  Then he thinks it's funny; he enjoyed it.\n\tHe starts laughing, louder and louder, as he pulls up to a\n\ttoll booth, stops, and extends his hand with a coin to the\n\tCOLLECTOR.\n\n\t---------------------------------------FADE OUT---------\n\n\tFADE IN:\n\n\tINT NITE: AMERIGO BONASERA'S APARTMENT\n\n\tThe serious-faced UNDERTAKER is on the telephone.\n\n\t\t\t\tHAGEN (O.S.)\n\t\tThis is Tom Hagen.  I'm calling for\n\t\tDon Corleone, at his request.\n\n\tBONASERA looks at his WIFE, with deep anxiety in his eyes.\n\tBONASERA's lips are suddenly dry.\n\n\t\t\t\tBONASERA\n\t\tYes, I understand.  I'm listening.\n\n\t\t\t\tHAGEN (O.S.)\n\t\tYou owe the Don a service.  In one\n\t\thour, not before, perhaps later, he\n\t\twill be at your funeral parlor to\n\t\task for your help.  Be there to\n\t\tgreet him.  If you have any\n\t\tobjections speak now, and I'll\n\t\tinform him.\n\n\tSilence.  BONASERA stutters, then speaks in fright.\n\n\t\t\t\tBONASERA\n\t\tAnything...Anything the Godfather\n\t\twishes.\n\n\t\t\t\tHAGEN (O.S.)\n\t\tGood.  He never doubted you.\n\n\t\t\t\tBONASERA\n\t\tThe Don himself is coming to me\n\t\ttonight?\n\n\t\t\t\tHAGEN (O.S.)\n\t\tYes.\n\t\t\t  (click)\n\n\n\tBONASERA is sweating; slowly he lowers the phone; his WIFE\n\tsees his pale expression, and follows him into the room.\n\n\tSilently, he begins the ritual of dressing.  His WIFE knows\n\tsomething serious is happening, and never takes her eyes\n\tfrom him.  He lights a cigarette.\n\n\t\t\t\tBONASERA\n\t\tFor the last year, they have been\n\t\tkilling one another.  So now, what?\n\t\tYour Godfather comes to me...Why?\n\t\t\t  (whispering, slyly)\n\t\tThey've killed someone so important\n\t\tthat they wish to make his body\n\t\tdisappear.\n\n\t\t\t\tMRS. BONASERA\n\t\t\t  (frightened)\n\t\tAmerigo!\n\n\t\t\t\tBONASERA\n\t\tThey could make me an accomplice to\n\t\ttheir murder.  They could send me\n\t\tto jail!\n\n\tHe slips into his trousers.  Then he moves to his WIFE to\n\ttie his tie, as she has done for years.\n\n\t\t\t\tBONASERA\n\t\tAnd if the other families find\n\t\tout...they will make me their enemy.\n\t\tThey could come here to our house.\n\t\tI curse the day I ever went to the\n\t\tGodfather.\n\n\tEXT NITE: FUNERAL PARLOR (SPRING 1946)\n\n\tWith his ring of keys, he opens the funeral parlor, enters.\n\n\tINT NITE: FUNERAL PARLOR (SPRING 1946)\n\n\tBONASERA walks through the darkened funeral parlor, without\n\tturning on the lights; then into the rear, preparation room,\n\tpast the tables, and equipment.  He operates the chain that\n\tlifts a large overhead garage type door.  And looks out into\n\tthe alley.\n\n\tHe sits on a bench, and waits.\n\n\tEXT NITE: FUNERAL PARLOR ALLEY (SPRING 1946)\n\n\tThe tires of a car roll very quietly along the small alley;\n\twe notice a dark car approach the rear of BONASERA's funeral\n\tparlor.\n\n\tCLEMENZA gets out, and moves to the open, rear door.\n\tBONASERA greets him, too petrified to speak.  He notices TWO\n\tOTHER MEN get out of the car, and carry a stretcher with a\n\tCORPSE swaddled in a gray blanket, with yellowed feet\n\tprotruding.\n\n\tBONASERA closes his eyes in fear, but indicates which way\n\tthe MEN should carry their sinister burden.\n\n\tINT NITE: FUNERAL PARLOR EMBALMING ROOM (SPRING 1946)\n\n\tThey carry the CORPSE to one of the tables in the embalming\n\troom.\n\n\tThen BONASERA turns to see ANOTHER MAN step out of the\n\tdarkness somewhat uncertainly.  It is DON CORLEONE.\n\n\tHe walks up to BONASERA, very close, without speaking.  His\n\tcold eyes looking directly at the frightened UNDERTAKER.\n\tThen, after a long gaze:\n\n\t\t\t\tDON CORLEONE\n\t\tWell my friend, are you ready to do\n\t\tme this service?\n\n\tBONASERA nods.  The DON moves to the CORPSE on the embalming\n\ttable; he makes a gesture, and the OTHER MEN leave them alone.\n\n\t\t\t\tBONASERA\n\t\tWhat do you wish me to do?\n\n\t\t\t\tDON CORLEONE\n\t\t\t  (staring at the table)\n\t\tI want you to use all your powers,\n\t\tall your skill, as you love me.  I\n\t\tdo not want his mother to see him\n\t\tas he is.\n\n\tHe draws down the gray blanket.\n\n\tBONASERA lets out a gasp of horror at what he sees:\n\n\tThe bullet-smashed face of SONNY CORLEONE.\n\n\tEXT NITE: TOLL BOOTHS (SPRING 1946)\n\n\tSONNY extends his hand with a coin at the toll booth.\n\n\tA car suddenly swerves in front of him, trapping him in the\n\tbooth, and in incredible rally of machine gun fire greets\n\thim, coming through and smashing the windows of the toll\n\tbooths on both side of him, and from the front window of the\n\tcar blocking him.\n\n\tThe windows of his car are shot out.\n\n\tBullet holes puncture the doors of his car.\n\n\tHis hand, with the coin in it, falls inside the car.\n\n\tHis arms, shoulders are riddled by the fire, and still it\n\tcontinues, as though the ASSASSINS cannot take a chance that\n\the will survive it.\n\n\tSuddenly, he lets out an enormous ROAR, like a bull, and\n\tactually, opens the door, and steps out of the car, UNDER\n\tfire.\n\n\tHis face is hit; and finally he falls to the ground.\n\n\tA FULL SHOT...as the ASSASSINS scramble for their cars and\n\tmake off in the distance.\n\n\tSONNY's BODYGUARDS stop a safe distance away, realizing they\n\tare too late.\n\n\tINT NITE: DON'S LIVING ROOM (SPRING 1946)\n\n\tView on HAGEN's ashen face in the living room.  He is silent\n\ta moment, and then:\n\n\t\t\t\tHAGEN\n\t\t\t  (quietly)\n\t\tOK.  Go to Clemenza's house and\n\t\ttell him to come here right away.\n\t\tHe'll tell you what to do.\n\n\tThe MEN leave him alone.  He is quiet, standing in the\n\tmiddle of the living room a moment.  He looks in the\n\tdirection of the kitchen, where he can see fragments of MAMA\n\tmoving around.\n\n\tINT NITE: UPSTAIRS (SPRING 1946)\n\n\tTOM proceeds up stairs, and quietly in the direction of the\n\tDON's room.  He opens the DON's door.  Looks in.\n\n\tINT NITE: DON'S BEDROOM (SPRING 1946)\n\n\tThe DON in his hospital bed.  Asleep under sedation.  HAGEN\n\thesitates.  He cannot go in; he cannot tell the OLD MAN.  He\n\tcloses the door.\n\n\tINT NITE: DON'S OFFICE (SPRING 1946)\n\n\tHAGEN alone in the office.  He is drinking.  He looks up at\n\tthe sound of cars; the CAPOREGIMES are arriving.  Then he\n\thears footsteps.\n\n\tThe door opens; and in a robe, with slippers, DON CORLEONE\n\tslowly enters the room.  He walks directly to his stuffed\n\tarmchair, sits down.  His face is stern, as he looks into\n\tHAGEN's eyes.\n\n\t\t\t\tDON CORLEONE\n\t\tGive me a drop of anisette.\n\n\tHAGEN rises, and pours a glass for the OLD MAN.\n\n\t\t\t\tDON CORLEONE\n\t\tMy wife was weeping before she fell\n\t\tasleep, outside my window I saw my\n\t\tcaporegimes to the house, and it is\n\t\tmidnight.  So, Consigliere of mine,\n\t\tI think you should tell your Don\n\t\twhat everyone knows.\n\n\t\t\t\tHAGEN\n\t\t\t  (quietly)\n\t\tI didn't tell Mama anything.  I was\n\t\tabout to come up and wake you and\n\t\ttell you.  Just now.\n\n\t\t\t\tDON CORLEONE\n\t\tBut you needed a drink first.\n\n\t\t\t\tHAGEN\n\t\tYes.\n\n\t\t\t\tDON CORLEONE\n\t\tNow you've had your drink.\n\n\tPause.\n\n\t\t\t\tHAGEN\n\t\tThey shot Sonny on the Causeway.\n\t\t\t  (pause)\n\t\tHe's dead.\n\n\tDON CORLEONE blinks.  One feels that just for a second he\n\tloses all physical strength; he clasps his hands in front of\n\thim on the top of the desk and looks into HAGEN's eyes.\n\n\t\t\t\tDON CORLEONE\n\t\tI want no inquiries made.  No acts\n\t\tof vengeance.\n\t\t\t  (pause)\n\t\tConsigliere, arrange a meeting with\n\t\tthe heads of the five\n\t\tfamilies...this war stops now.\n\n\tHe rises and unsteadily leaves the room, turns...\n\n\t\t\t\tDON CORLEONE\n\t\tCall Bonasera...he will do me a\n\t\tservice.\n\n\tAnd leaves.  HAGEN moves to the phone; dials...\n\n\t\t\t\tHAGEN\n\t\tThis is Tom Hagen; I'm calling for\n\t\tDon Corleone, at his request.\n\n\t\t\t\tBONASERA (O.S.)\n\t\tYes, I understand I'm listening.\n\n\t\t\t\tHAGEN\n\t\tYou owe the Don a service.  He has\n\t\tno doubt that you will repay it.\n\n\tEXT DAY: BANK BUILDING (SPRING 1946)\n\n\tDay in Manhattan.  An impressive Bank Building in the\n\tfinancial center of New York.  Many limousines are parked,\n\tuniforms and plain-clothed CHAUFFEURS waiting quietly.\n\n\tINT DAY: BOARD ROOM (SPRING 1946)\n\n\tThe Board Room of a bank, daylight shines in the windows.\n\n\tCARLO TRAMONTI, an impressive, handsome middle-aged man,\n\tsits quietly, smoking a Di Napoli cigar, OUR VIEW moves to a\n\tMAN sitting to his left, and a little to the rear, and\n\tsettles on JOSEPH ZALUCHI, a moon-faced amiable-looking man;\n\tas the view continues, around the table, we HEAR:\n\n\t\t\t\tDON CORLEONE (O.S.)\n\t\tI want to thank you all for coming.\n\t\tI consider it a service done to me\n\t\tpersonally and I am in the debt of\n\t\teach and every one of you.\n\t\tEspecially those of you who have\n\t\ttraveled from such distances as\n\t\tCalifornia, St. Louis, Kansas City;\n\t\tand New Orleans...\n\n\tThe VIEW PASSES to FRANK FALCONE and ANTHONY MOLINARI, both\n\tyounger than any of the others; then on to DOMENICK PANZA,\n\tshort and squat sitting in a wheelchair; then around the\n\ttable to DON VINCENENZO FORLENZA, who is whispering to his\n\tJEWISH ASSISTANT; the VIEW PASSES on to ANTHONY STRACCI, an\n\tolder man, sipping from a drink and smoking a cigar; OTTILIO\n\tCUNEO, in his middle sixties with a jolly round face; then\n\tDON PHILLIP TATTAGLIA, a delicate older man with dyed hair\n\tand a pencil mustache; and finally, EMILIO BARZINI, in his\n\tearly sixties, a man to 'respect'; whom we had seen at\n\tCONNIE's Wedding.\n\n\t\t\t\tDON CORLEONE\n\t\tAh well, let's get down to business.\n\t\tWe are all honorable men here, we\n\t\tdon't have to give assurances as if\n\t\twe were lawyers.\n\t\t\t  (he sits, gazes out\n\t\t\t  at them, and sighs)\n\t\tHow did things ever go so far?\n\t\tWell, no matter.  A lot of\n\t\tfoolishness has come to pass.  It\n\t\twas so unfortunate, so unnecessary.\n\n\tThe VIEW examines the room once again, as the DON speaks.  A\n\tlarge, clicking board is changing numbers at various times,\n\tand two tapes, showing the fluctuations of the Market during\n\tthe day's trading, and projected above.\n\n\tDON CORLEONE pauses; and TOM HAGEN hands him a cold drink.\n\n\t\t\t\tDON CORLEONE\n\t\tTattaglia has lost a son; I have\n\t\tlost a son.  We are quits.  Let\n\t\tthere be a peace...\n\t\t\t  (he gestures\n\t\t\t  expressively,\n\t\t\t  submissively, with\n\t\t\t  his hands)\n\t\tThat is all I want...\n\n\t\t\t\tBARZINI\n\t\tDon Corleone is too modest.  He had\n\t\tthe judges and politicians in his\n\t\tpocket and he refused to share them.\n\t\tHis refusal is not the act of a\n\t\tfriend.  He takes the bread out of\n\t\tthe mouths of our families.  Times\n\t\thave changed, it's not like the old\n\t\tdays where everyone can go his own\n\t\tway.  If Don Corleone had all the\n\t\tjudges and politicians in New York,\n\t\tthen he must share them or let\n\t\tothers use them.  Certainly he can\n\t\tpresent a bill for such services,\n\t\twe're not Communists, after all.\n\t\tBut he has to let us draw water\n\t\tfrom the well.  It's that simple.\n\n\t\t\t\tDON CORLEONE\n\t\tMy friends, I didn't refuse out of\n\t\tmalice.  You all know me.  When\n\t\thave I ever refused an accommodation?\n\t\tBut why, this time?  Because I\n\t\tthink this drug business will\n\t\tdestroy us in the years to come.\n\t\tIt's not like whiskey or gambling\n\t\tor even women which most people\n\t\twant and is forbidden them by the\n\t\tpezzonovante of the Church and the\n\t\tGovernment.  But drugs?  No.  Even\n\t\tpolicemen, who help us in gambling\n\t\tand other things would refuse to\n\t\thelp us in drugs.  But...I am\n\t\twilling to do whatever all of you\n\t\tthink is necessary.\n\n\t\t\t\tDON ZALUCHI\n\t\tI don't believe in drugs.  For\n\t\tyears I paid my people extra so\n\t\tthey wouldn't do that kind of\n\t\tbusiness...$200 a week.  But it\n\t\tdidn't matter.  Somebody comes to\n\t\tthem and says, \"I have powders, if\n\t\tyou put up three, four thousand\n\t\tdollar investment, we can make\n\t\tfifty thousand distributing.\"  Who\n\t\tcan resist such a profit?\n\t\tThere's no way to control it, as a\n\t\tbusiness...to keep it respectable.\n\t\t\t  (rapping the table)\n\t\tI don't want it near schools!  I\n\t\tdon't want it sold to children.\n\t\tThat is an infamita.\n\t\t\t  (thinking)\n\t\tIn my city I would try to keep the\n\t\ttraffic in the dark people, the\n\t\tcolored.  They are the best\n\t\tcustomers, the least troublesome,\n\t\tand they are animals anyway.  They\n\t\thave no respect for their wives or\n\t\ttheir families or themselves.  Let\n\t\tthem lose their souls with drugs.\n\t\tBut something has to be done, we\n\t\tcan't have everybody running around\n\t\tdoing just what they please, like a\n\t\tbunch of anarchists.\n\n\t\t\t\tBARZINI\n\t\tThen, are we agreed; the traffic in\n\t\tdrugs will be permitted, but\n\t\tcontrolled; and Don Corleone agrees\n\t\tto give it protection in the East.\n\n\tDON CORLEONE nods.\n\n\t\t\t\tBARZINI\n\t\tThat's the whole matter then, we\n\t\thave the peace, and let me pay my\n\t\trespects to Don Corleone, whom we\n\t\thave all known over the years as a\n\t\tman of his word.\n\t\t\t  (noticing TATTAGLIA\n\t\t\t  is uneasy)\n\t\tDon Philip?\n\n\t\t\t\tTATTAGLIA\n\t\tI agree to everything here, I'm\n\t\twilling to forget my own misfortune.\n\t\tBut I must hear strict assurance\n\t\tfrom Corleone.  When time goes by\n\t\tand his position becomes stronger,\n\t\twill he attempt any individual\n\t\tvengeance?\n\n\tThey all look at the DON; especially HAGEN, who feels that\n\tDON CORLEONE has given a great deal, and must have something\n\telse in mind.  Slowly the DON rises.\n\n\t\t\t\tDON CORLEONE\n\t\tI forego my vengeance for my dead\n\t\tson, for the common good.  But I\n\t\thave selfish reasons.  My youngest\n\t\tson had to flee, accused of\n\t\tSollozzo's murder, and I must now\n\t\tmake arrangements so that he can\n\t\tcome home with safety, cleared of\n\t\tall those false charges.  That is\n\t\tmy affair, and I will make those\n\t\tarrangements.\n\t\t\t  (with strength)\n\t\tBut I am a superstitious man...and\n\t\tso if some unlucky accident should\n\t\tbefall my youngest son, if some\n\t\tpolice officer should accidentally\n\t\tshoot him, or if he should hang\n\t\thimself in his cell, or if my son\n\t\tis struck by a bolt of lightning,\n\t\tthen I will blame some of the\n\t\tpeople here.  That, I could never\n\t\tforgive, but...aside from that, let\n\t\tme swear by the souls of my\n\t\tGrandchildren that I will never be\n\t\tthe one to break the peace we have\n\t\tmade.\n\n\tEXT NITE: DON'S LIMO (SPRING 1946)\n\n\tThe DON's black limousine.  He sits quietly in the padded\n\trear seat; TOM HAGEN next to him.\n\n\tIt is night.  Lights flash by them every so often.\n\n\t\t\t\tHAGEN\n\t\tWhen I meet with Tattaglia's\n\t\tpeople; should I insist that all\n\t\this drug middle-men be clean?\n\n\t\t\t\tDON CORLEONE\n\t\tMention it, don't insist.  Barzini\n\t\tis a man who will know that without\n\t\tbeing told.\n\n\t\t\t\tHAGEN\n\t\tYou mean Tattaglia.\n\n\t\t\t\tDON CORLEONE\n\t\t\t  (shaking his head)\n\t\tBarzini.\n\n\t\t\t\tHAGEN\n\t\t\t  (a revelation)\n\t\tHe was the one behind Sollozzo?\n\n\t\t\t\tDON CORLEONE\n\t\tTattaglia is a pimp.  He could\n\t\tnever have outfought Santino.  But\n\t\tI wasn't sure until this day.  No,\n\t\tit was Barzini all along.\n\n\tThe black limousine speeds away from us in the night.\n\n\t------------------------------------------FADE OUT------\n\n\tFADE IN:\n\n\tEXT DAY: ESTABLISHING SICILY SHOT\n\n\tA CLOSE VIEW OF MICHAEL, moving as he walks, sullen and\n\tdowncast, the left side of his face healed, but left\n\tgrotesque and misshapen.\n\n\tGRADUALLY, THE VIEW LOOSENS, he wears a warm navy Pea\n\tjacket, and walks with his hands in his pockets.\n\n\tTHE VIEW LOOSENS FURTHER, revealing a Sicilian SHEPHERD on\n\teither side of him, each carrying a shotgun slung over his\n\tshoulder, CALO, a squat and husky young man with a simple\n\thonest quality, and FABRIZZIO, slender and handsome, likable,\n\tand with a pleasing build.  Each of the SHEPHERDS  carry\n\tknapsacks.\n\n\tThe THREE YOUNG MEN continue over the Sicilian landscape,\n\toverlooking an impressive view of land and sea.\n\n\tEXT DAY: SICILY ROAD\n\n\tThe THREE move through a flock of wind-blown sheep, and make\n\ttheir way to a dusty rural road.  We HEAR a rinky horn\n\tsound, as a pre-war Italian automobile makes its way to them.\n\tAn OLD MAN peeks from the window, waving to MICHAEL.  The\n\tcar pulls in front of them and stops.  MICHAEL nods\n\trespectfully.\n\n\t\t\t\tMICHAEL\n\t\tDon Tommassino.\n\n\t\t\t\tDON TOMMASSINO\n\t\tMichael, why must you do this.  We\n\t\thave been lucky so far, all these\n\t\tmonths you've been here we've kept\n\t\tyour name a secret.  It is from\n\t\tlove for your father that I've\n\t\tasked you never to more than an\n\t\thour from the Villa.\n\n\t\t\t\tMICHAEL\n\t\tCalo and Fabrizzio are with me;\n\t\tnothing will happen.\n\n\t\t\t\tDON TOMMASSINO\n\t\tYou must understand that your\n\t\tFather's enemies have friends in\n\t\tPalermo.\n\n\t\t\t\tMICHAEL\n\t\tI know.\n\n\t\t\t\tDON TOMMASSINO\n\t\tWhere are you going?\n\n\t\t\t\tMICHAEL\n\t\tCorleone.\n\n\t\t\t\tDON TOMMASSINO\n\t\tThere is nothing there.  Not anymore.\n\n\t\t\t\tMICHAEL\n\t\tI was told that my Grandfather was\n\t\tmurdered on its main street; and\n\t\this murderers came to kill my\n\t\tfather there when he was twelve\n\t\tyears old.\n\n\t\t\t\tDON TOMMASSINO\n\t\tLong ago.  Now there is nothing:\n\t\tthe men killed each other in family\n\t\tvendettas...the others escaped to\n\t\tAmerica.\n\n\t\t\t\tMICHAEL\n\t\tDon Tommassino...I should see this\n\t\tplace.\n\n\tDON TOMMASSINO thinks a moment, then concedes.\n\n\t\t\t\tDON TOMMASSINO\n\t\tThat is your birthright...but\n\t\tMichael, use this car.\n\n\t\t\t\tMICHAEL\n\t\tNo...I would like to walk to\n\t\tCorleone.\n\n\tThe OLD MAN sighs, and then returns to his car.\n\n\t\t\t\tDON TOMMASSINO\n\t\tBe careful Michael, don't let them\n\t\tknow your name.\n\n\tThe old car sputters off; MICHAEL watches, and then continues\n\ton his journey.\n\n\tEXT DAY: COUNTRYSIDE\n\n\tThe THREE pass through abundant areas of flowers and fruit\n\ttrees, in bloom and bursting with life.\n\n\tEXT DAY: VILLAGE\n\n\tThey continue in the empty streets of a little town; the\n\tpost-war poverty is evident in the skinny dogs; and the\n\tempty streets.  Occasionally, a military vehicle, the only\n\tgasoline-powered vehicles on the road, will pass.  And there\n\tare many POLICE evident, most of them carrying machine guns.\n\n\tThe THREE pass under an enormous banner slung over the main\n\troad \"VOTA COMMUNISTA\".\n\n\tEXT DAY: COUNTRY ROAD\n\n\tThey continue through dusty country roads, where occasionally\n\ta donkey pulling a cart, or a lone horseman will pass them.\n\n\tEXT DAY: FIELD\n\n\tOut in a field, in the distance, they come upon a procession\n\tof peasants and activists, perhaps two hundred strong,\n\tmarching, and singing, and in the lead, are five or six men\n\tcarrying billowing red banners.\n\n\tEXT DAY: GROVE\n\n\tThey are in an orange grove; on the other side of the trees\n\tis a deep, tall field of wild flowers.\n\n\tThe Shepherds unsling their guns and knapsacks, and take out\n\tloaves of bread, some wine, sausage and cheese.\n\n\tMICHAEL rests against a tree, and uses his handkerchief.\n\n\t\t\t\tFABRIZZIO\n\t\tYou tell us about America.\n\n\t\t\t\tMICHAEL\n\t\tHow do you know I come from America?\n\n\t\t\t\tFABRIZZIO\n\t\tWe hear.  We were told you were a\n\t\tPezzonovanta...big shot.\n\n\t\t\t\tMICHAEL\n\t\tOnly the son of a Pezzonovanta.\n\n\t\t\t\tFABRIZZIO\n\t\tHey America!  Is she as rich as\n\t\tthey say?\n\n\t\t\t\tMICHAEL\n\t\tYes.\n\n\t\t\t\tFABRIZZIO\n\t\tTake me to America!  You need a\n\t\tgood lupara in America?\n\t\t\t  (pats his shotgun)\n\t\tYou take me, I'll be the best man\n\t\tyou got.  \"Oh say, can you\n\t\tseeee...By da star early light...\"\n\n\tMICHAEL laughs.\n\n\tEXT DAY: ANOTHER ROAD\n\n\tThe TRIO continues down a dirt road, as an American Military\n\tconvoy speeds by; FABRIZZIO waves, and calls out to each of\n\tthe U.S. drivers, as they move by.\n\n\t\t\t\tFABRIZZIO\n\t\tAmerica.\n\t\tHey America!\n\t\tTake me with you!\n\t\tHey, take me to America G.I.!\n\n\tEXT DAY: CORLEONE HILL\n\n\tThey continue their long hike, high on a promentory; until\n\tthey hesitate, and look down.\n\n\t\t\t\tCALO\n\t\tCorleone.\n\n\tThey can see a grim Sicilian village, almost devoid of people.\n\n\tEXT DAY: CORLEONE STREET\n\n\tMICHAEL and his bodyguards move through the empty streets of\n\tthe village.  They walk behind him, and spread to either\n\tside about fifteen feet away from him.\n\n\tThey move down ancient steps, past an old stone fountain.\n\tMICHAEL hesitates, cups his hands and drinks some water.\n\tThey go on.\n\n\tThey move up a very narrow old street.  MICHAEL looks at the\n\tdoorways that they pass.\n\n\tMOVING VIEW: Each door has a plaque, with a ribbon or flower.\n\n\tCALO sees MICHAEL looking.\n\n\t\t\t\tCALO\n\t\tThe names of the dead.\n\n\tMICHAEL hesitates in the center of the main street.  He looks.\n\n\tThe street is empty, barren.  Occasionally, an old woman\n\twill pass.\n\n\tMICHAEL turns his head.\n\n\tThe other side of the street: empty and deathly.\n\n\tA HIGH VIEW of MICHAEL standing in the center of the old\n\tstreet, the shepherds a respectful distance away.\n\n\t-------------------------------------FADE OUT-----------\n\n\tEXT DAY: BARONIAL ESTATE\n\n\tA green ribboned field of a baronial Estate.  Further ahead\n\tis a villa so Roman it looks as though it had just been\n\tdiscovered in the ruins of Pompeii.  There is a group of\n\tyoung village GIRLS accompanied by two stocky MATRONS,\n\tdressed in black.  They have been gathering the pink sulla,\n\tpurple wisteria, and mixing them with orange and lemon\n\tblossoms.  They are singing, off in the distance as they work.\n\n\tMICHAEL, CALO and FABRIZZIO are silent as they watch this\n\tFantasy-like scene.\n\n\t\t\t\tFABRIZZIO\n\t\t\t  (calling out to them)\n\t\tHey, beautiful girls!\n\n\t\t\t\tMICHAEL\n\t\t\t  (sternly)\n\t\tShhhhh.\n\n\tHe settles down to watch.\n\n\tThe GIRLS are dressed in cheap gaily painted frocks that\n\tcling to their bodies.  They are still in their teens, but\n\tdeveloped and womanly.\n\n\tThey are moving along the fields, picking blossoms, not\n\taware of the three men watching them from the orange grove.\n\tThree or four of the girls begin chasing one of them\n\tplayfully, in the direction of the grove.\n\n\tThe GIRL being chased holds a bunch of purple grapes in her\n\tleft hand and with the right, picks more grapes, and throws\n\tthem back at her pursuers laughing.\n\n\tThey come closer and closer.  Just short of the grove, she\n\tposes, startled, her large, oval shaped eyes catching the\n\tview of the THREE MEN.  She stands there on her toes about\n\tto run.\n\n\tMICHAEL sees her; now face to face.  He looks.\n\n\tHer face.  Incredibly beautiful with olive skin, black hair\n\tand a rich mouth.\n\n\t\t\t\tFABRIZZIO\n\t\t\t  (murmuring)\n\t\tJesus Christ, take my soul.  I'm\n\t\tdying.\n\n\tQuickly, she turns, and runs away.\n\n\tMICHAEL stands up never taking his eyes from her.  We hold\n\ton him for a long while; and eventually hear the SHEPHERDS\n\tlaughing.  Then he turns to them.\n\n\t\t\t\tFABRIZZIO\n\t\tYou got hit by the thunderbolt, eh?\n\n\tCALO pats him on the shoulder.\n\n\t\t\t\tCALO\n\t\tEasy man.\n\n\t\t\t\tMICHAEL\n\t\tWhat are you talking about?\n\n\t\t\t\tFABRIZZIO\n\t\tYou can't hide it when you're hit\n\t\tby the thunderbolt.\n\n\tEXT DAY: BARONIAL VILLAGE\n\n\tThe little village built attendant to the Baronial Estate,\n\tis decked with the flowers the girls had been picking.\n\n\tMICHAEL, followed by the bodyguards, moves into the central\n\tsquare, and onto the balcony of a little cafe.\n\n\tThe proprietor of the cafe, VITELLI, is a short burly man;\n\the greets them cheerfully, and sets a dish of chickpeas at\n\ttheir table.\n\n\t\t\t\tFABRIZZIO\n\t\tYou know all the girls in this\n\t\ttown, eh?  We saw some beauties\n\t\tcoming down the road.  One in\n\t\tparticular got our friend hit with\n\t\tthe Thunderbolt...\n\t\t\t  (he indicates MICHAEL)\n\n\tVITELLI gives a big knowing laugh, and looks at MICHAEL with\n\tnew interest.\n\n\t\t\t\tVITELLI\n\t\tYou had better bring a few bottles\n\t\thome with you, my friend; you'll\n\t\tneed help sleeping tonight.\n\t\t\t  (he laughs)\n\n\t\t\t\tFABRIZZIO\n\t\tThis one could seduce the devil.  A\n\t\tbody! and eyes as big and black as\n\t\tolives.\n\n\t\t\t\tVITELLI\n\t\t\t  (laughing with\n\t\t\t  them...pouring more wine)\n\t\tI know about what you mean!\n\n\t\t\t\tFABRIZZIO\n\t\tThis was a beauty.  Right, Calo?\n\n\t\t\t\tVITELLI\n\t\t\t  (laughing)\n\t\tBeautiful all over, eh?\n\n\t\t\t\tFABRIZZIO\n\t\tAnd hair.  Black and curly, like a\n\t\tdoll.  And such a mouth.\n\n\tVITELLI does not laugh quite so much.\n\n\t\t\t\tVITELLI\n\t\tYes, we have beautiful girls here...\n\t\tbut virtuous.\n\n\tVITELLI is no longer drinking with them.\n\n\t\t\t\tMICHAEL\n\t\tShe wore a red dress, and a red\n\t\tribbon in her hair.  She looks more\n\t\tGreek than Italian.  Do you know a\n\t\tbeauty like that?\n\n\tAs MICHAEL describes her, VITELLI laughed less and less,\n\tuntil he wears a scowl.\n\n\t\t\t\tVITELLI\n\t\tNo.\n\n\tThen he curtly leaves him, and walks into the back room.\n\n\t\t\t\tFABRIZZIO\n\t\tGod in Heaven, I think I\n\t\tunderstand...\n\n\tHe goes into the back room after the innkeeper.  Then he\n\treturns.\n\n\t\t\t\tFABRIZZIO\n\t\tLet's get out of here; he's boiling\n\t\tup his blood to do us mischief.\n\t\tIt's his daughter.\n\n\tThey start to leave; but MICHAEL doesn't move.\n\n\t\t\t\tCALO\n\t\tCome quickly.\n\n\t\t\t\tMICHAEL\n\t\tInnkeeper.  More wine!\n\n\t\t\t\tFABRIZZIO\n\t\t\t  (whispered)\n\t\tThe old bastard mentioned two sons\n\t\the only has to whistle up.\n\n\tMICHAEL turns to FABRIZZIO with his cold authority.\n\n\t\t\t\tMICHAEL\n\t\tTell him to come to me.\n\n\tThe two BODYGUARDS shoulder their luparas, and disappear in\n\ta moment they return with the red-faced angry VITELLI\n\tbetween them.\n\n\t\t\t\tMICHAEL\n\t\t\t  (quietly)\n\t\tI understand I've offended you by\n\t\ttalking about your daughter.  I\n\t\toffer you my apologies, I'm a\n\t\tstranger in this country, I don't\n\t\tknow the customs very well.  Let me\n\t\tsay this, I meant no disrespect to\n\t\tyou or her...\n\n\tCALO and FABRIZZIO are impressed.\n\n\t\t\t\tVITELLI\n\t\t\t  (shrugs)\n\t\tWho are you and what do you want\n\t\tfrom my daughter?\n\n\t\t\t\tMICHAEL\n\t\tI am an American hiding in Sicily\n\t\tfrom the police of my country.  My\n\t\tname is Michael.  You can inform\n\t\tthe police and make your fortune\n\t\tbut then your daughter would lose a\n\t\tfather rather than gain a husband.\n\t\tIn any case, I want to meet your\n\t\tdaughter.  With your permission and\n\t\tunder the supervision of your\n\t\tfamily.  With all decorum.  With\n\t\tall respect.  I am an honorable man.\n\n\tCALO and FABRIZZIO are stupefied; VITELLI pauses, and then\n\tasks:\n\n\t\t\t\tVITELLI\n\t\tAre you a friend of the friends?\n\n\t\t\t\tMICHAEL\n\t\tWhen the proper time comes, I'll\n\t\ttell you everything that a wife's\n\t\tfather should know.\n\n\t\t\t\tFABRIZZIO\n\t\tIt's the real Thunderbolt, then.\n\n\t\t\t\tVITELLI\n\t\t\t  (formally)\n\t\tCome Sunday morning:  My name is\n\t\tVitelli and my house is up there on\n\t\tthe hill, above the village.\n\n\t\t\t\tMICHAEL\n\t\tYour daughter's name?\n\n\t\t\t\tVITELLI\n\t\tAppolonia.\n\n\t-------------------------------------FADE OUT-----------\n\n\tEXT DAY: TOMMASSINO COURTYARD\n\n\tMUSIC comes up; as MICHAEL, dressed in new clothes from\n\tPalermo, and carrying a stack of wrapped gifts, gets into an\n\tAlfa Romeo.  CALO and FABRIZZIO each dressed in their Sunday\n\tbest, are in the rear seat, huddled together, with their\n\tluparas on their shoulders.\n\n\tDON TOMMASSINO waves them off, as the little car drives off,\n\trocky and bouncing on the dirt road.\n\n\tThe Sunday churchbells ring.\n\n\t--------------------------------------DISSOLVE----------\n\n\tEXT DAY: VITELLI HOUSE\n\n\tMICHAEL is presented to each of the Vitelli relatives, by\n\tthe yard of their little hilltop house; the BROTHERS; the\n\tMOTHER, who is given a gift; several UNCLES and AUNTS.\n\tFinally APPOLONIA enters, dressed beautifully in appropriate\n\tSunday clothing.  Now he presents the wrapped gift to\n\tAPPOLONIA.  She looks at her MOTHER, who with a nod gives\n\ther permission to open it.  She unwraps it.  Her eyes light\n\tat the sight of a heavy gold chain; to be worn as a necklace.\n\n\tShe looks at him.\n\n\t\t\t\tAPPOLONIA\n\t\tGrazia.\n\n\t--------------------------------------DISSOLVE----------\n\n\tEXT DAY: VITELLI CAFE\n\n\tNow the little Alpha drives into the village near VITELLI's\n\tcafe.\n\n\tMICHAEL is, as ever, accompanied with his two BODYGUARDS,\n\tthough they are all dressed differently.\n\n\tThey go up to the cafe...and sit with VITELLI, who is\n\ttalking and talking.\n\n\tMICHAEL looks at APPOLONIA; who sits, respectfully quiet.\n\tShe wears the gold necklace around her neck.\n\n\t--------------------------------------DISSOLVE----------\n\n\tEXT DAY: HILLTOP NEAR VITELLI HOME\n\n\tMICHAEL and APPOLONIA are walking through a hilltop path,\n\tseemingly alone, although a respectful distance apart.\n\n\tAs the VIEW PANS with them, we notice that her MOTHER and a\n\thalf dozen AUNTS are twenty paces behind them, and ten paces\n\tfurther behind are CALO and FABRIZZIO, their luparas on\n\ttheir shoulders.\n\n\tFurther up the hill, APPOLONIA stumbles on a loose stone,\n\tand falls briefly onto MICHAEL's arm.  She modestly regains\n\ther balance, and they continue walking.\n\n\tBehind them, her MOTHER giggles to herself.\n\n\t--------------------------------------DISSOLVE----------\n\n\tEXT DAY: VITELLI VILLAGE CHURCH\n\n\tChurch bells in an ancient belfry ring out.  Music, old and\n\tdissonant, plays.\n\n\tThere is a bridal procession in the street of the village;\n\tthe same in feeling and texture as it might have been five\n\thundred years ago.\n\n\tDonkeys and other animals have been decorated with abundant\n\tflowers; children carrying candles and wearing white\n\tconfirmation gowns walk in the procession, followed by\n\tcountless townspeople, members of the clergy, even the police.\n\n\tWe present the entire bridal procession and ceremony with\n\tall the ritual and pageantry, as it has always been, in\n\tSicily.\n\n\tAPPOLONIA is radiant as the Bride; MICHAEL is handsome\n\tdespite the grotesque jaw and occasional white handkerchief.\n\n\t--------------------------------------DISSOLVE----------\n\n\tEXT NITE: VITELLI VILLAGE SQUARE\n\n\tCALO and FABRIZZIO dance wildly through the night of the\n\tgreat wedding celebration.  It is held in the Village\n\tSquare; under the watchful eyes of SHEPHERDS above on the\n\ttops of buildings, carrying luparas.\n\n\t--------------------------------------DISSOLVE----------\n\n\tINT NITE: MICHAEL'S ROOM IN VILLA\n\n\tMICHAEL opens the shutters in his darkened room; moonlight\n\tfills the room.\n\n\tHe turns, and there, in her wedding slip, is APPOLONIA.  A\n\tlittle frightened; but lovely.\n\n\tHe moves to her; and for a moment just stands before her,\n\tlooking at her incredible face; her lovely hair and body.\n\n\tSlowly and tenderly he kisses her.  Her tiny hands come up\n\tto his face; touch his cheek and embrace him.\n\n\tShe lets her bridal slip fall to the floor.\n\n\t--------------------------------------FADE OUT----------\n\n\tINT DAY: MICHAEL'S ROOM AT VILLA\n\n\tMorning.  MICHAEL sits on the window ledge, gazing into the\n\troom.\n\n\tAPPOLONIA is asleep; she is naked, and only partially\n\tcovered by the bedsheets.\n\n\tHe looks at her for a long time in the early morning light.\n\n\tEXT DAY: TOMMASSINO COURTYARD\n\n\tHIGH ANGLE ON DON TOMMASSINO'S VILLA\n\n\tWe HEAR girlish laughter; the little Alpha is driving\n\terratically, knocking down an occasional wall, and almost\n\thitting th inner court wall.\n\n\tAPPOLONIA is laughing, driving.  MICHAEL pretends to be\n\tfrightened, as he teaches her to drive.\n\n\tOutside the walls, we notice SHEPHERDS with luparas, walking\n\tguard duty.\n\n\tThe car stops and a laughing MICHAEL gets out.\n\n\t\t\t\tMICHAEL\n\t\tIt's safer to teach you English.\n\n\t\t\t\tAPPOLONIA\n\t\tMonday, Tuesday, Wednesday,\n\t\tThursday, Friday...See, I learned\n\t\tit.  Now teach me to drive!\n\n\tDON TOMMASSINO enters the Courtyard.  He seems tired and\n\tconcerned.\n\n\t\t\t\tMICHAEL\n\t\tCiao, Don Tommassino.\n\n\tAPPOLONIA kisses him.\n\n\t\t\t\tMICHAEL\n\t\tThings went badly in Palermo?\n\n\t\t\t\tDON TOMMASSINO\n\t\tThe younger men have no respect.\n\t\tThings are changing; I don't know\n\t\twhat will happen.  Michael, because\n\t\tof the wedding, people now know\n\t\tyour name.\n\n\t\t\t\tMICHAEL\n\t\tIs that why there are more men on\n\t\tthe walls?\n\n\t\t\t\tDON TOMMASSINO\n\t\tEven so, I don't think it is safe\n\t\there anymore.  I've made plans to\n\t\tmove you to a villa near Siracuse.\n\t\tYou must go right away.\n\n\t\t\t\tMICHAEL\n\t\tWhat is it?\n\n\t\t\t\tDON TOMMASSINO\n\t\tBad news from America.  Your\n\t\tbrother, Santino.  He has been\n\t\tkilled.\n\n\tFor a moment, the whole world of New York, Sollozzo, the\n\tFive Family War, all comes back to MICHAEL.\n\n\tEXT DAY: VILLA COURTYARD\n\n\tMorning.  MICHAEL leans out of the bedroom window.\n\n\tBelow, FABRIZZIO is sitting in one of the garden chairs,\n\tcombing his thick hair.\n\n\tMICHAEL whistles and FABRIZZIO looks up to his window.\n\n\t\t\t\tMICHAEL\n\t\tGet the car.  I'll be leaving in\n\t\tten minutes.  Where's Calo?\n\n\t\t\t\tFABRIZZIO\n\t\tCalo is having a cup of coffee in\n\t\tthe kitchen.  Is your wife coming\n\t\twith you?\n\n\t\t\t\tMICHAEL\n\t\tNo, she's going home to her family.\n\t\tShe'll join me in a few weeks...\n\n\tINT DAY: VILLA KITCHEN\n\n\tMICHAEL, dressed, crosses from the hallway, and into the\n\tkitchen.  CALO is just finishing a bite.  He rises when he\n\tsees MICHAEL.\n\n\t\t\t\tCALO\n\t\tShould I get your bag?\n\n\t\t\t\tMICHAEL\n\t\tNo, I'll get it.  Where's Appolonia?\n\n\t\t\t\tCALO\n\t\t\t  (smiling)\n\t\tShe is sitting in the driver's seat\n\t\tof the car, dying to step on the\n\t\tgas.  She'll be a real American\n\t\twoman before she gets to America.\n\n\tMICHAEL smiles.\n\n\t\t\t\tMICHAEL\n\t\tTell Fabrizzio and wait for me in\n\t\tthe car.\n\n\tHe leaves the kitchen, after a quick sip of coffee.\n\n\tHe looks out from the opening in the doorway.\n\n\tEXT DAY: VILLA COURTYARD\n\n\tThere is the car, with APPOLONIA sitting in the driver's\n\tseat, playing with the wheel like a child.\n\n\tCALO moves to the car, and puts a lunch basket in the rear\n\tseat.\n\n\tThen MICHAEL seems disturbed.\n\n\tOver, on the other side of the courtyard, he sees FABRIZZIO\n\tdisappear through the gate.\n\n\t\t\t\tMICHAEL\n\t\t\t  (muttering to himself)\n\t\tWhere the hell is he going?\n\n\tMICHAEL goes down the hallway, and outside.\n\n\tMICHAEL steps out into the bright sunlight of the outer\n\tcourtyard, causing him to shade his eyes.\n\n\tAPPOLONIA sees him, and waves, motioning that he should stay\n\twhere he is.\n\n\t\t\t\tAPPOLONIA\n\t\t\t  (calling out)\n\t\tI'll drive to you.\n\n\tHe smiles affectionately.\n\n\tCALO stands beside the car, smiling, with his lupara dangling\n\tby his side.  There is no sight of FABRIZZIO.  Suddenly the\n\tsmile fades from MICHAEL's face.  He steps forward and holds\n\tout his hand.\n\n\t\t\t\tMICHAEL\n\t\tNo.  No!\n\n\tHis shout is drowned in the roar of a tremendous EXPLOSION,\n\tas she switched on the ignition.\n\n\tPart of the wall is caved in, the kitchen door is blown off;\n\tand there is nothing left of the Alpha, or of Appolonia.\n\n\tMICHAEL is thrown against the wall, and knocked unconscious.\n\n\tINT DAY: VILLA BEDROOM\n\n\tMICHAEL is unconscious in a darkened room.  We hear\n\twhispering around him, but can't make any of it out.  A soft\n\tcloth is applied to his face; gradually his eyes open.  DON\n\tTOMMASSINO is there, close to him.  He looks at them and\n\tfrom their grave expressions, he knows his wife is dead.\n\n\t\t\t\tMICHAEL\n\t\tFabrizzio.  Let your shepherds know\n\t\tthat the one who gives me Fabrizzio\n\t\twill own the finest pastures in\n\t\tSicily.\n\n\t--------------------------------------FADE OUT----------\n\n\tFADE IN:\n\n\tEXT DAY: MALL (SPRING 1951)\n\n\tEaster.\n\n\tA HIGH VIEW ON THE CORLEONE MALL in the springtime.  Hordes\n\tof little CHILDREN including many of the Corleone Children\n\tand Grandchilren, rush about carrying little Easter baskets,\n\tsearching here and there for candy treasures and hidden\n\tEaster eggs.\n\n\tThe DON himself, much older, much smaller in size, wearing\n\tbaggy pants and a plaid shirt and an old hat, moves around\n\this garden, tending rows and rows of rich tomato plants.\n\n\tSuddenly, he stops and looks.\n\n\tMICHAEL stands there, still holding his suitcase.\n\n\tGreat emotion comes over the DON, who takes a few steps in\n\tMICHAEL's direction.\n\n\tMICHAEL leaves his suitcase and walks to his favorite son\n\tand embraces him.\n\n\t\t\t\tDON CORLEONE\n\t\tBe my son...\n\n\tINT DAY: THE OLIVE OIL FACTORY\n\n\tDON CORLEONE leads MICHAEL through the corridors of the\n\tbuilding.\n\n\t\t\t\tDON CORLEONE\n\t\tThis old building has seen its day.\n\t\tNo way to do business...too small,\n\t\ttoo old.\n\n\tThey enter the DON's glass-panelled office.\n\n\t\t\t\tDON CORLEONE\n\t\tHave you thought about a wife?  A\n\t\tfamily?\n\n\t\t\t\tMICHAEL\n\t\t\t  (pained)\n\t\tNo.\n\n\t\t\t\tDON CORLEONE\n\t\tI understand, Michael.  But you\n\t\tmust make a family, you know.\n\n\t\t\t\tMICHAEL\n\t\tI want children, I want a family.\n\t\tBut I don't know when.\n\n\t\t\t\tDON CORLEONE\n\t\tAccept what's happened, Michael.\n\n\t\t\t\tMICHAEL\n\t\tI could accept everything that's\n\t\thappened; I could accept it, but\n\t\tthat I never had a choice.  From\n\t\tthe time I was born, you had laid\n\t\tthis all out for me.\n\n\t\t\t\tDON CORLEONE\n\t\tNo, I wanted other things for you.\n\n\t\t\t\tMICHAEL\n\t\tYou wanted me to be your son.\n\n\t\t\t\tDON CORLEONE\n\t\tYes, but sons who would be\n\t\tprofessors, scientists,\n\t\tmusicians...and grandchildren who\n\t\tcould be, who knows, a Governor, a\n\t\tPresident even, nothing's impossible\n\t\there in America.\n\n\t\t\t\tMICHAEL\n\t\tThen why have I become a man like\n\t\tyou?\n\n\t\t\t\tDON CORLEONE\n\t\tYou are like me, we refuse to be\n\t\tfools, to be puppets dancing on a\n\t\tstring pulled by other men.  I\n\t\thoped the time for guns and killing\n\t\tand massacres was over.  That was\n\t\tmy misfortune.  That was your\n\t\tmisfortune.  I was hunted on the\n\t\tstreets of Corleone when I was\n\t\ttwelve years old because of who my\n\t\tfather was.  I had no choice.\n\n\t\t\t\tMICHAEL\n\t\tA man has to choose what he will be.\n\t\tI believe that.\n\n\t\t\t\tDON CORLEONE\n\t\tWhat else do you believe in?\n\n\tMICHAEL doesn't answer.\n\n\t\t\t\tDON CORLEONE\n\t\tBelieve in a family.  Can you\n\t\tbelieve in your country?  Those\n\t\tPezzonovante of the State who\n\t\tdecide what we shall do with our\n\t\tlives?  Who declare wars they wish\n\t\tus to fight in to protect what they\n\t\town.  Do you put your fate in the\n\t\thands of men whose only talent is\n\t\tthat they tricked a bloc of people\n\t\tto vote for them?  Michael, in five\n\t\tyears the Corleone family can be\n\t\tcompletely legitimate.  Very\n\t\tdifficult things have to happen to\n\t\tmake that possible.  I can't do\n\t\tthem anymore, but you can, if you\n\t\tchoose to.\n\n\tMICHAEL listens.\n\n\t\t\t\tDON CORLEONE\n\t\tBelieve in a family; believe in a\n\t\tCode of Honor, older and higher,\n\t\tbelieve in Roots that go back\n\t\tthousands of years into your Race.\n\t\tMake a family, Michael, and protect\n\t\tit.  These are our affairs, sono cosa\n\t\tnostra, Governments only protect\n\t\tmen who have their own individual\n\t\tpower.  Be one of those men...you\n\t\thave the choice.\n\n\t--------------------------------------FADE OUT----------\n\n\tEXT DAY: STOCK FOOTAGE LAS VEGAS (1955)\n\n\tA MOVING VIEW, driving up the Las Vegas Strip of 1955.\n\n\t\t\t\tFREDO (O.S.)\n\t\tThere's a new one.  Construction\n\t\tgoing on everywhere.\n\n\tMORE VIEWS, showing new hotels and casinos being built; the\n\tbill marquees read: \"MARTIN AND LEWIS\", \"PATTI PAGE\", etc.\n\n\t\t\t\tFREDO (O.S.)\n\t\tThat's one of the family's new ones.\n\t\tNot bad, eh?\n\n\tEXT DAY: FLAMINGO (1955)\n\n\tThe car pulls up at the Flamingo Hotel.\n\n\tInside the car: MICHAEL, FREDO, TOM HAGEN and a new man,\n\tNERI, quiet and sinister.\n\n\t\t\t\tMICHAEL\n\t\tWhy didn't Moe Green meet us at the\n\t\tairport?\n\n\t\t\t\tFREDO\n\t\tHe had business at the hotel, but\n\t\the'll drop in for dinner.\n\n\tFrom the expression on MICHAEL's face we know this is a\n\tdiscourtesy.\n\n\tINT DAY: FLAMINGO HOTEL SUITE (1955)\n\n\tA whole entourage precedes FREDO and his V.I.P. party of\n\tMICHAEL, HAGEN and NERI.  Great fuss is made.  They are\n\tbeing shown into the hotel's 'special' suite.\n\n\t\t\t\tFREDO\n\t\tYou look wonderful, kid; really\n\t\twonderful.  That doctor did some\n\t\tjob on your face.\n\n\t\t\t\tMICHAEL\n\t\tYou look good, too.\n\n\tThey enter the suite.\n\n\t\t\t\tFREDO\n\t\tNice, eh?\n\n\tFREDO is as excited as a kid, snapping orders at the\n\tbellboys, waiters and maids.\n\n\t\t\t\tFREDO\n\t\t\t  (hurrying into the bedroom)\n\t\tKid, take a look-see.\n\n\tMICHAEL gives a look to HAGEN, and continues into the bedroom.\n\n\tThere is an enormous circular bed on a huge platform,\n\tmirrors to each side.  FREDO points upward.\n\n\tA VIEW into a large CEILING mirror.\n\n\t\t\t\tFREDO\n\t\tEver seen anything like that before?\n\n\t\t\t\tMICHAEL\n\t\t\t  (dryly)\n\t\tNo.\n\n\tINT NITE: FLAMINGO SUITE BEDROOM (1955)\n\n\tMICHAEL is alone in the bedroom.  He is just finishing\n\tdressing; he puts on his jacket.  From the window, with the\n\tlights blinking, we can tell it's late at night.  MICHAEL\n\tpasses into the other room.\n\n\tHe stops, looks.  He is disturbed.\n\n\tINT NITE: FLAMINGO SUITE (1955)\n\n\tA magnificent, circular table has been set up in his suite;\n\ta lavish table setting for eight.  Standing by the table are\n\tHAGEN, JOHNNY FONTANE, looking wonderful, a little heavier,\n\tbeautifully dressed; FREDO, a dandy, and TWO LAS VEGAS GIRLS.\n\tNERI stands quietly by the door.\n\n\t\t\t\tFREDO\n\t\tMike!  The party starting!\n\n\t\t\t\tMICHAEL\n\t\tCome here a minute, Fredo.\n\n\tFREDO goes to him, a big smile all over his face.\n\n\t\t\t\tMICHAEL\n\t\tWho are those girls?\n\n\t\t\t\tFREDO\n\t\t\t  (jokingly)\n\t\tThat's for you to find out.\n\n\t\t\t\tMICHAEL\n\t\tGive them some money and send them\n\t\thome.\n\n\t\t\t\tFREDO\n\t\tMike!\n\n\t\t\t\tMICHAEL\n\t\tGet rid of them...\n\n\tINT NITE: FLAMINGO SUITE (1955)\n\n\tThey are seated around the lavish table in Michael's suite.\n\tMICHAEL is speaking to JOHNNY.\n\n\t\t\t\tMICHAEL\n\t\tJohnny, the Corleone family is\n\t\tthinking of selling out all our\n\t\tinterests in the Olive Oil business\n\t\tand settling here.  Moe Greene will\n\t\tsell us his interest so it can be\n\t\twholly owned by friends of the\n\t\tfamily.\n\n\tFREDDIE seems anxious.\n\n\t\t\t\tFREDO\n\t\tMike, you sure about Moe selling.\n\t\tHe never mentioned it to me and he\n\t\tloves the business.\n\n\t\t\t\tMICHAEL\n\t\tI'll make him an offer he can't\n\t\trefuse.\n\n\tMICHAEL turns to JOHNNY.\n\n\t\t\t\tMICHAEL\n\t\tJohnny, the Don wants you to help\n\t\tus get started.  We figure\n\t\tentertainment will be the big\n\t\tfactor in drawing gamblers.  We\n\t\thope you'll sign a contract to\n\t\tappear five times a year for maybe\n\t\ta week long engagement.\n\t\tWe hope your friends in the movies\n\t\twill do the same.  We count on you\n\t\tto convince them.\n\n\t\t\t\tJOHNNY\n\t\tSure, I'll do anything for my\n\t\tGodfather.  You know that, Mike.\n\n\tThere is knock on the door.  NERI rises, looks at MICHAEL,\n\twho nods.  NERI opens the door, and MOE GREENE enters,\n\tfollowed by TWO BODYGUARDS.  He is a handsome hood, dressed\n\tin the Hollywood style.  His BODYGUARDS are more West Coast\n\tstyle.\n\n\t\t\t\tMOE\n\t\tMike, good to see you.  Got\n\t\teverything you want?\n\n\t\t\t\tMICHAEL\n\t\tThanks.\n\n\t\t\t\tMOE\n\t\tThe chef cooked for you special;\n\t\tthe dancers will kick your tongue\n\t\tout and you credit is good!\n\t\t\t  (to his BODYGUARDS)\n\t\tDraw chips for all these people so\n\t\tthey can play on the house.\n\n\t\t\t\tMICHAEL\n\t\tIs my credit good enough to buy you\n\t\tout?\n\n\tMOE laughs.\n\n\t\t\t\tMOE\n\t\tBuy me out?...\n\n\t\t\t\tMICHAEL\n\t\tThe hotel, the casino.  The Corleone\n\t\tfamily wants to buy you out.\n\n\tGREENE stops laughing; the room becomes tense.  NERI eyes\n\tthe BODYGUARDS.\n\n\t\t\t\tMOE\n\t\t\t  (furious)\n\t\tThe Corleone family wants to buy me\n\t\tout.  I buy you out.  You don't buy\n\t\tme out.\n\n\t\t\t\tMICHAEL\n\t\tYour casino loses money.  Maybe we\n\t\tcan do better.\n\n\t\t\t\tMOE\n\t\tYou think I scam?\n\n\t\t\t\tMICHAEL\n\t\t\t  (the worst insult)\n\t\tYou're unlucky.\n\n\t\t\t\tMOE\n\t\tYou goddamn dagos.  I do you a\n\t\tfavor and take Freddie in when\n\t\tyou're having a bad time, and then\n\t\tyou try to push me out.\n\n\t\t\t\tMICHAEL\n\t\tYou took Freddie in because the\n\t\tCorleone family bankrolled your\n\t\tcasino.  You and the Corleone\n\t\tfamily are evened out.  This is for\n\t\tbusiness; name your price.\n\n\t\t\t\tMOE\n\t\tThe Corleone family don't have that\n\t\tkind of muscle anymore.  The\n\t\tGodfather is sick.  You're getting\n\t\tchased out of New York by Barzini\n\t\tand the other families, and you\n\t\tthink you can find easier pickings\n\t\there.  I've talked to Barzini; I\n\t\tcan make a deal with him and keep\n\t\tmy hotel!\n\n\t\t\t\tMICHAEL\n\t\t\t  (quietly, deadly)\n\t\tIs that why you thought you could\n\t\tslap Freddie around in public?\n\n\t\t\t\tFREDO\n\t\t\t  (his face turns red)\n\t\tAh Mike, that was nothing.  Moe\n\t\tdidn't mean anything.  He flies off\n\t\tthe handle sometimes; but me and\n\t\thim are good friends.  Right, Moe?\n\n\t\t\t\tMOE\n\t\tYeah sure.  Sometimes I gotta kick\n\t\tasses to make this place run right.\n\t\tFreddie and I had a little argument\n\t\tand I had to straighten him out.\n\n\t\t\t\tMICHAEL\n\t\tYou straightened my brother out?\n\n\t\t\t\tMOE\n\t\tHell, he was banging cocktail\n\t\twaitresses two at a time.  Players\n\t\tcouldn't get a drink.\n\n\tMICHAEL rises from his chair, and says in a tone of dismissal:\n\n\t\t\t\tMICHAEL\n\t\tI have to go back to New York\n\t\ttomorrow.  Think of your price.\n\n\t\t\t\tMOE\n\t\tYou son of a bitch, you think you\n\t\tcan brush me off like that?  I made\n\t\tmy bones when you were going out\n\t\twith cheerleaders.\n\n\t\t\t\tFREDO\n\t\t\t  (frightened)\n\t\tTom, you're the Consigliere; you\n\t\tcan talk to the Don and advise him.\n\n\t\t\t\tMICHAEL\n\t\tThe Don has semi-retired.  I'm\n\t\trunning the Family business now.\n\t\tSo anything you have to say, say it\n\t\tto me.\n\n\tNobody answers.  MICHAEL nods to NERI, who opens the door.\n\tMOE exits angrily.\n\n\t\t\t\tMICHAEL\n\t\tFreddie, you're my older brother.\n\t\tI love you.  But don't ever take\n\t\tsides with anybody against the\n\t\tFamily again.\n\n\tEXT DAY: N.Y. AIRPORT (1955)\n\n\tKAY sits in the back of a limousine parked by the Newark\n\tAIRPORT.  ROCCO LAMPONE is leaning against it.\n\n\tShe has a little three year old boy; MICHAEL's son, who\n\tplays with a cardboard bird on a string.\n\n\tTwo other cars are stationed discreetly, with men we have\n\tlearned to tell are bodyguards.\n\n\tMICHAEL, HAGEN and NERI exit the airport with TWO NEGRO\n\tPORTERS carrying luggage.\n\n\tNERI sees something, and taps MICHAEL on the shoulder.\n\n\tMICHAEL turns, and sees KAY.\n\n\tLAMPONE opens the car door; KAY steps out with the BOY, and\n\tMICHAEL embraces her, and kisses his son.  Automatically,\n\tthe luggage is put in.  NERI replaces LAMPONE as the driver;\n\tand LAMPONE joins the other men.  HAGEN gets into one of the\n\tother cars.\n\n\tAnd the limo drives off, preceded and followed by the other\n\tsedans.\n\n\tINT DAY: LIMO (1955)\n\n\tThe little BOY looks out the window as they drive.\n\n\t\t\t\tMICHAEL\n\t\tI have to see my father and his\n\t\tpeople when we get back to the Mall.\n\n\t\t\t\tKAY\n\t\tOh Michael.\n\n\t\t\t\tMICHAEL\n\t\tWe'll go to the show tomorrow\n\t\tnight--we can change the tickets.\n\n\t\t\t\tKAY\n\t\tDon't you want dinner first?\n\n\t\t\t\tMICHAEL\n\t\tNo, you eat...don't wait up for me.\n\n\t\t\t\tKAY\n\t\tWake me up when you come to bed?\n\n\tThe little BOY flies his cardboard bird out of the speeding\n\tlimousine window.\n\n\tEXT DAY: MALL (1955)\n\n\tThe limousine arrives at the Mall.  We are inside.\n\n\t\t\t\tKAY\n\t\tYour sister wants to ask you\n\t\tsomething.\n\n\t\t\t\tMICHAEL\n\t\tLet HER ask.\n\n\tNERI opens the door.  KAY wants to talk just a little more.\n\n\t\t\t\tKAY\n\t\tShe's afraid to.  Michael...\n\n\tMICHAEL nods to NERI; who gives them their privacy a moment\n\tlonger.\n\n\t\t\t\tKAY\n\t\tWhy are you so cold to her and\n\t\tCarlo?  They live with us on the\n\t\tMall now, but you never get close\n\t\tto them.\n\n\t\t\t\tMICHAEL\n\t\tI'm busy.\n\n\t\t\t\tKAY\n\t\tConnie and Carlo want you to be\n\t\tgodfather to their little boy.\n\n\tNERI opens the door; MICHAEL starts to get out; KAY too.\n\n\tHe smiles at her, tired, and a little sad.\n\n\t\t\t\tKAY\n\t\tWill you?\n\n\t\t\t\tMICHAEL\n\t\tLet me think about it, O.K.?\n\n\tShe smiles; MICHAEL goes with NERI to the Main House; KAY\n\tand the little BOY move to the house that was Sonny's.\n\n\tINT DAY: DON'S OFFICE (1955)\n\n\tVIEW ON DON CORLEONE, much older, much smaller in size.  He\n\twears baggy pants, and a warm plaid shirt.  He sits in a\n\tchair, gazing out through the window, into the garden.\n\n\t\t\t\tTESSIO (O.S.)\n\t\tBarzini's people chisel my territory\n\t\tand we do nothing about it.  Pretty\n\t\tsoon there won't be one place in\n\t\tBrooklyn I can hang my hat.\n\n\t\t\t\tMICHAEL (O.S.)\n\t\tJust be patient.\n\n\t\t\t\tTESSIO\n\t\tI'm not asking you for help, Mike.\n\t\tJust take off the handcuffs.\n\n\t\t\t\tMICHAEL (O.S.)\n\t\tBe patient.\n\n\t\t\t\tCLEMENZA (O.S.)\n\t\tWe gotta fight sometime.  Let us at\n\t\tleast recruit our regimes to full\n\t\tstrength.\n\n\t\t\t\tMICHAEL (O.S.)\n\t\tNo, I don't want to give Barzini an\n\t\texcuse to start fighting.\n\n\t\t\t\tTESSIO (O.S.)\n\t\tMike, you're wrong.\n\n\t\t\t\tCLEMENZA (O.S.)\n\t\tDon Corleone...Don Corleone.\n\n\tThe OLD MAN looks up.  CLEMENZA stand before him in the Den.\n\tBeside him is an anxious TESSIO.  NERI stands by the door;\n\tHAGEN is seated; MICHAEL sits behind the big desk.\n\n\t\t\t\tCLEMENZA\n\t\tYou said there would come a day\n\t\twhen Tessio and me could form our\n\t\town Families.  Only with your\n\t\tbenediction, of course.  I ask\n\t\tpermission...\n\n\t\t\t\tDON CORLEONE\n\t\tMy son is head of the Family now.\n\t\tIf you have his permission, you\n\t\thave my good will.\n\n\t\t\t\tMICHAEL\n\t\tIn six months you can break off\n\t\tfrom the Corleone Family and go on\n\t\tyour own.  Carlo, I'm counting on\n\t\tyou to make the move to Nevada;\n\t\tyou'll be my right-hand man out\n\t\tthere.  Tom Hagen is no longer the\n\t\tConsigliere.\n\n\tEveryone is a bit surprised; look to see HAGEN's reaction.\n\tHe remains inexpressive.\n\n\t\t\t\tMICHAEL\n\t\tHe's going to be our lawyer in\n\t\tVegas.  Nobody goes to him with any\n\t\tother business as of now, this\n\t\tminute.  No reflection on Tom;\n\t\tthat's the way I want it.  Besides,\n\t\tif I ever need any advice, who's a\n\t\tbetter Consigliere than my father.\n\n\t\t\t\tCLEMENZA\n\t\tThen in a six month time we're on\n\t\tour own; is that it?\n\n\t\t\t\tMICHAEL\n\t\tMaybe less...\n\n\t\t\t\tTESSIO\n\t\tLet us fill up our Regimes.\n\n\t\t\t\tMICHAEL\n\t\tNo.  I want things very calm for\n\t\tanother six months.\n\n\t\t\t\tTESSIO\n\t\tForgive me, Godfather, let our\n\t\tyears of friendship be my excuse.\n\t\tHow can you hope for success there\n\t\twithout your strength here to back\n\t\tyou up?  The two  go hand in hand.\n\t\tAnd with you gone from here the\n\t\tBarzini and the Tattaglias will be\n\t\ttoo strong for us.\n\n\t\t\t\tCLEMENZA\n\t\tAnd I don't like Barzini.  I say\n\t\tthe Corleone Family has to move\n\t\tfrom strength, not weakness.  We\n\t\tshould build our Regimes and take\n\t\tback our lost territories in Staten\n\t\tIsland, at least.\n\n\t\t\t\tDON CORLEONE\n\t\tDo you have faith in my judgement?\n\n\t\t\t\tCLEMENZA\n\t\tYes, Godfather...\n\n\t\t\t\tDON CORLEONE\n\t\tThen do what Michael says...\n\n\t\t\t\tMICHAEL\n\t\tAll I can say is that things are\n\t\tbeing resolved that are more\n\t\teffective than a thousand buttonmen\n\t\ton the streets.  Understood?\n\n\tThere are uneasy looks all around.\n\n\t\t\t\tCARLO\n\t\tUnderstood.  I just wish I was\n\t\tdoing more to help out.\n\n\t\t\t\tMICHAEL\n\t\tI'll come to you when I need you.\n\n\tHe looks at CLEMENZA, TESSIO and HAGEN.  They all nod,\n\treluctantly.\n\n\t\t\t\tMICHAEL\n\t\tAll right, then it's resolved.\n\n\tNERI knows the meeting is over, he opens the Den's door.\n\n\tCLEMENZA and TESSIO pay their respects to the DON and leave,\n\tthen CARLO.  NERI watches CARLO as he walks down the\n\tcorridor, casting a nervous look back at the sinister man.\n\n\tThen NERI closes the door.\n\n\tMICHAEL relaxes.\n\n\t\t\t\tHAGEN\n\t\tMike, why are you cutting me out of\n\t\tthe action?\n\n\t\t\t\tMICHAEL\n\t\tTom, we're going to be legitimate\n\t\tall the way, and you're the legal\n\t\tman.  What could be more important\n\t\tthan that.\n\n\t\t\t\tHAGEN\n\t\tI'm not talking about that.  I'm\n\t\ttalking about Rocco Lampone building\n\t\ta secret regime.  Why does Neri\n\t\treport directly to you, rather than\n\t\tthrough me or a caporegime?\n\n\t\t\t\tDON CORLEONE\n\t\tI told you that it wouldn't escape\n\t\this eye.\n\n\t\t\t\tMICHAEL\n\t\tHow did you find out?\n\n\t\t\t\tHAGEN\n\t\tBookkeepers know everything.\n\t\tRocco's men are all a little too\n\t\tgood for the jobs they're supposed\n\t\tto be doing.  They get a little\n\t\tmore money than the job's worth.\n\t\t\t  (pause)\n\t\tLampone's a good man; he's operating\n\t\tperfectly.\n\n\t\t\t\tMICHAEL\n\t\tNot so perfectly if you noticed.\n\n\t\t\t\tHAGEN\n\t\tMike, why am I out?\n\n\t\t\t\tMICHAEL\n\t\tYou're not a wartime Consigliere.\n\t\tThings may get tough with the move\n\t\twe're trying.\n\n\t\t\t\tHAGEN\n\t\tOK, but then I agree with Tessio.\n\t\tYou're going about it all wrong;\n\t\tyou're making the move out of\n\t\tweakness... Barzini's a wolf, and\n\t\tif he tears you apart, the other\n\t\tfamilies won't come running to help\n\t\tthe Corleones...\n\n\t\t\t\tDON CORLEONE\n\t\tTom, I never thought you were a bad\n\t\tConsigliere, I thought Santino a\n\t\tbad Don, rest in peace.  He had a\n\t\tgood heart but he wasn't the right\n\t\tman to head the family when I had\n\t\tmy misfortune.  Michael has all my\n\t\tconfidence, as you do.  For reasons\n\t\twhich you can't know, you must have\n\t\tno part in what will happen.\n\n\t\t\t\tHAGEN\n\t\tMaybe I can help.\n\n\t\t\t\tMICHAEL\n\t\t\t  (coldly)\n\t\tYou're out, Tom.\n\n\tTOM pauses, thinks...and then he nods in acquiescence.  TOM\n\tleaves.\n\n\tMICHAEL looks at NERI.\n\n\t\t\t\tMICHAEL\n\t\tI'm going to talk to my father.\n\n\tNERI nods, and then leaves.  The DON opens the doors,\n\tbreathes in the air, and steps outside.\n\n\tEXT DAY: THE GARDEN (1955)\n\n\t\t\t\tDON CORLEONE\n\t\tI see you have your Luca Brasi.\n\n\t\t\t\tMICHAEL\n\t\tI'll need him.\n\n\t\t\t\tDON CORLEONE\n\t\tThere are men in this world who\n\t\tdemand to be killed.  They argue in\n\t\tgambling games; they jump out of\n\t\ttheir cars in a rage if someone so\n\t\tmuch as scratches their fender.\n\t\tThese people wander through the\n\t\tstreets calling out \"Kill me, kill\n\t\tme.\"  Luca Brasi was like that.\n\t\tAnd since he wasn't scared of\n\t\tdeath, and in fact, looked for\n\t\tit...I made him my weapon.  Because\n\t\tI was the only person in the world\n\t\tthat he truly hoped would not kill\n\t\thim.  I think you have done the\n\t\tsame with this man.\n\n\tThey walk through the DON's vegetable garden.  Tomatoes,\n\tpeppers, carefully tended, and covered with a silky netting.\n\tMICHAEL follows; the DON turns and looks at him.  Then\n\tstoops over to right a tomato plant that had been pushed over.\n\n\t\t\t\tDON CORLEONE\n\t\tBarzini will move against you first.\n\n\t\t\t\tMICHAEL\n\t\tHow?\n\n\t\t\t\tDON CORLEONE\n\t\tHe will get in touch with you\n\t\tthrough someone you absolutely\n\t\ttrust.  That person will arrange a\n\t\tmeeting, guarantee your safety...\n\n\tHe rises, and looks at Michael...\n\n\t\t\t\tDON CORLEONE\n\t\t...and at that meeting you will be\n\t\tassassinated.\n\n\tThe DON walks on further.\n\n\t\t\t\tDON CORLEONE\n\t\tYour wife and children...you're\n\t\thappy with them?\n\n\t\t\t\tMICHAEL\n\t\tYes.\n\n\t\t\t\tDON CORLEONE\n\t\tGood.\n\n\tMICHAEL wants to express something...hesitates, then:\n\n\t\t\t\tMICHAEL\n\t\tI've always respected you...\n\n\tA long silence.  The DON smiles at MICHAEL.\n\n\t\t\t\tDON CORLEONE\n\t\tAnd I...you.\n\n\tEXT DAY: CHURCH (1955)\n\n\tKAY and MAMA walking from the black car that has just left\n\tthem off.\n\n\t\t\t\tKAY\n\t\tHow is your husband feeling?\n\n\t\t\t\tMAMA\n\t\tHe's not the same since they shot\n\t\thim.  He lets Michael do all the\n\t\twork.  He just plays the fool with\n\t\this garden, his peppers, his\n\t\ttomatoes, as if he was some peasant\n\t\tstill.  But men are like that...\n\n\tShe stops toward the Church.\n\n\t\t\t\tMAMA\n\t\tYou come in, too.\n\n\tKAY shakes her head.\n\n\t\t\t\tMAMA\n\t\tThe Priest ain't gonna bite you\n\t\tcause you're not Catholic.\n\t\t\t  (whispered)\n\t\tHe's in the back drinkin' his wine.\n\n\tKAY laughs and follows MAMA up the steps of the Church.\n\tThey enter.\n\n\tINT DAY: CHURCH (1955)\n\n\tInside the Church, KAY watches as MAMA blesses herself from\n\tthe holy water.\n\n\t\t\t\tMAMA\n\t\tYou can.\n\n\tTentatively, KAY dips her fingers into the water, and\n\tblesses herself.  Then SHE follows MAMA down the aisle, in\n\tawe at the high ceiling, the art, the windows, and finally\n\tthe Altar.\n\n\tMAMA stops by the impressive tiers of candles.  There is a\n\tlarge coin box for those who wish to pay for lighting\n\tcandles.  MAMA fumbles in her purse for change; KAY gives\n\ther some.\n\n\tMAMA drops the coins in the box, one by one; then takes the\n\ttaper, and in a pattern known only to her, and with great\n\tdignity, she closes her eyes, says a prayer, and then lights\n\ttwenty candles.\n\n\tShe finishes, and bows her head.\n\n\tEXT DAY: BONASERA'S FUNERAL HOME\n\n\tVery few people in the streets.  TOTAL SILENCE.  But black\n\tflower cars as far as the eye can see, for blocks and blocks.\n\tAn expression of respect, of honor and fear that is enormous.\n\tCertainly no more could be done for a President or a King.\n\n\tEach car carries an elaborate floral decoration.  We show\n\tthese in detail; and the flowered messages: \"A Benefactor to\n\tMankind\", \"He Knew and Pitied\"...\"Our Don Our Leader\"...\"The\n\tSacred Heart\"...\n\n\tEXT DAY: MALL (1955)\n\n\tHIGH ANGLE ON THE CORLEONE MALL\n\n\tSilence.\n\n\tThe flower cars, funeral limousines, and private cars fill\n\tall the areas attendant to the Corleone residence.\n\n\tHundreds of people fill the Mall, reminiscent in size of the\n\twedding of Connie and Carlo; of course, now the mood is\n\tsomber and respectful.\n\n\tMICHAEL, MAMA, FREDO and HAGEN stand by the flowered platform\n\twhich holds the ornate coffin.  We cannot see the remains of\n\tDon Corleone.\n\n\tBONASERA is nearby, ready to do service to the bereaved\n\tfamily.  One by one the mourners come by, weeping, or merely\n\twith grave expressions; pay their respects and continue on.\n\n\tThe VIEW ALTERS,\n\n\tand we see that the line is endless.  JOHNNY FONTANE, tears\n\topenly falling, takes his turn.\n\n\tChildren are taken by the hand, and lifted for their last\n\tlook at the great man.\n\n\tCLEMENZA whispers into the ear of LAMPONE.  LAMPONE\n\timmediately arranges for the members of the Five New York\n\tFamilies to pay their respects.\n\n\tFirst CUNEO, then STRACHI and then ZALUCHI.  Then PHILIP\n\tTATTAGLIA, who merely passes by the Coffin.\n\n\tThen BARZINI in a black homburg, standing a long time.\n\n\tMICHAEL watches the scene.\n\n\tBARZINI crosses himself and passes on, immediately rejoined\n\tby his men.\n\n\tAs BARZINI leaves, it seems as though everyone is fawning on\n\thim; perhaps asking for favors: But at any rate, it is clear\n\tfrom the doors opened for him, the cigars lit for him, that\n\the is the new Capo di Capi--the place formerly held by Don\n\tCorleone.\n\n\tMICHAEL watches silently.\n\n\tBARZINI is searching for somebody with his eyes.  First\n\tCLEMENZA.  Then TESSIO.\n\n\tCONNIE rushes into MICHAEL's arms, tears in her eyes.  He\n\tembraces and comforts her.\n\n\tEverywhere MICHAEL goes, NERI is a few feet away--watching\n\tall who come close to him.\n\n\tEXT DAY: MALL (LATER)\n\n\tLater on the Mall; some people have left, although there are\n\tstill hundreds of mourners.\n\n\tA young GIRL approaches TESSIO.  She's about 18.\n\n\t\t\t\tGIRL\n\t\tDo you remember me?\n\n\t\t\t\tTESSIO\n\t\tNo...\n\n\t\t\t\tGIRL\n\t\tWe danced together at Connie's\n\t\twedding.\n\n\tTESSIO makes a gesture, which is to say 'How you've grown',\n\tand they move though the crowd, looking for Michael.  He\n\tfinds him.\n\n\t\t\t\tTESSIO\n\t\tMike, could I have a minute?\n\n\tMIKE; nods; and they move to a private place.  NERI is close\n\tby.\n\n\t\t\t\tTESSIO\n\t\tBarzini wants to arrange a meeting.\n\t\tSays we can straighten any of our\n\t\tproblems out.\n\n\t\t\t\tMICHAEL\n\t\tHe talked to you?\n\n\t\t\t\tTESSIO\n\t\t\t  (nods)\n\t\tI can arrange security.\n\n\tMICHAEL looks at him.\n\n\tEXT DAY: CEMETERY (1955)\n\n\tThe Cemetery.  Late day.\n\n\tThe hundreds of cars, limousines and flower cars line the\n\tstone wall that surrounds this Italian-Catholic cemetary in\n\tQueens Village.\n\n\tHundreds of people stand in a cluster; others watch; take\n\tpictures, etc.\n\n\tMICHAEL stands with his family, his MOTHER...and TOM HAGEN.\n\n\t\t\t\tMICHAEL\n\t\t\t  (softly)\n\t\tChrist, Tom; I needed more time\n\t\twith him.  I really needed him.\n\n\t\t\t\tHAGEN\n\t\tDid he give you his politicians?\n\n\t\t\t\tMICHAEL\n\t\tNot all...I needed another four\n\t\tmonths and I would have had them\n\t\tall.\n\t\t\t  (he looks at TOM)\n\t\tI guess you've figured it all out?\n\n\t\t\t\tHAGEN\n\t\tHow will they come at you?\n\n\t\t\t\tMICHAEL\n\t\tI know now.\n\t\t\t  (a passion wells up\n\t\t\t  inside of MICHAEL)\n\t\tI'll make them call me Don.\n\n\t\t\t\tHAGEN\n\t\tHave you agreed on a meeting?\n\n\t\t\t\tMICHAEL\n\t\t\t  (nods)\n\t\tA week from tonight.  In Brooklyn\n\t\ton Tessio's ground, where I'll be\n\t\tsafe.\n\n\tHAGEN looks at him; understands.\n\n\t\t\t\tMICHAEL\n\t\tBut after the Baptism.  I've\n\t\tdecided to stand as godfather to\n\t\tConnie's baby.\n\n\tThey look up.\n\n\tThe coffin is lowered into an excavation, behind which\n\tstands an enormous stone monument; it is of a weeping angel,\n\twith the bold inscription: CORLEONE.\n\n\t---------------------------------------FADE OUT---------\n\n\tFADE IN:\n\n\tINT DAY: NERI'S APT. (1955)\n\n\tALBERT NERI moves around in his small Corona Apartment; he\n\tpulls a small trunk from under his bed.  He opens it, and we\n\tsee in it, nearly folded, a New York City Policeman's\n\tuniform.  He takes it out piece by piece, almost reverently.\n\tThen the badge, and the identification card; with his\n\tpicture on it.  Slowly, in the solitude of his room, he\n\tbegins to dress.\n\n\tINT DAY: MICHAEL'S BEDROOM (1955)\n\n\tMICHAEL and KAY are getting dressed for the christening in\n\ttheir room.  MICHAEL looks very well; very calm; KAY is\n\tbeginning to take on a matronly look.\n\n\tINT DAY: MOTEL ROOM (1955)\n\n\tIn a Long Island motel.\n\n\tROCCO LAMPONE carefully disassembles a revolver; oils it,\n\tchecks it, and puts it back together.\n\n\tEXT DAY: CLEMENZA'S HOUSE (1955)\n\n\tPETER CLEMENZA about to get in his Lincoln.  He hesitates,\n\ttakes a rag and cleans some dirt off of the fender, and then\n\tgets in, drives off.\n\n\tEXT DAY: CHURCH (1955)\n\n\tThe Church.\n\n\tVarious relatives and friends are beginning to gather at the\n\tChurch.  They laugh and talk.  A MONSIGNOR is officiating.\n\tNot all of the participants have arrived yet.\n\n\tCONNIE is there, with a beaming CARLO.  She holds the\n\tinfant; showing him off to interested people.\n\n\tEXT DAY: U.N. PLAZA (1955)\n\n\tNERI walks down the sidewalk in the neighborhood of the UN\n\tBuilding.  He is dressed as, and has the bearing of, a\n\tpoliceman.  He carries a huge flashlight.\n\n\tEXT DAY: MOTEL BALCONY (1955)\n\n\tLAMPONE steps out onto the little balcony of a Sea-Resort\n\tMotel; We can see the bright, neon lit sign advertising\n\t\"ROOMS FACING THE SEA--VACANY\".\n\n\tINT DAY: CHURCH\n\n\tThe Church.\n\n\tCONNIE holds the baby; the MONSIGNOR is speaking; KAY and\n\tMICHAEL stand side by side around the urn.\n\n\t\t\t\tPRIEST\n\t\t\t  (to MICHAEL)\n\t\tDo you pledge to guide and protect\n\t\tthis child if he is left fatherless?\n\t\tDo you promise to shield him\n\t\tagainst the wickedness of the world?\n\n\t\t\t\tMICHAEL\n\t\tYes, I promise.\n\n\tEXT DAY: FIFTH AVE.\n\n\tNERI continues up the 55th St. and Fifth Avenue area.  He\n\tcontinues until he is in front of Rockefeller Center.  On\n\this side of the street, he spots a limousine waiting directly\n\tacross from the main entrance of the building.  Slowly he\n\tapproaches the limo, and taps on its fender with his\n\tnightstick.\n\n\tThe DRIVER looks up in surprise.\n\n\tNERI points to the \"No Parking\" sign.\n\n\tThe DRIVER turns his head away.\n\n\t\t\t\tNERI\n\t\tOK, wise guy, you wanna summons, or\n\t\tyou wanna move?\n\n\t\t\t\tDRIVER\n\t\t\t  (obviously a hood)\n\t\tYou better check with your precinct.\n\n\t\t\t\tNERI\n\t\tMove it!\n\n\tThe DRIVER takes a ten dollar bill, folds it deliberately,\n\tand hands it out the window, trying to put it under NERI's\n\tjacket.\n\n\tNERI backs up, letting the bill fall onto the street.  Then\n\the crooks a finger at the DRIVER.\n\n\t\t\t\tNERI\n\t\tLet me see you license and\n\t\tregistration.\n\n\tEXT DAY: MOTEL BALCONY\n\n\tLAMPONE on the motel balcony spots a Cadillac pulling up.\n\tIt parks.  A young, pretty GIRL gets out.  Quickly, he\n\treturns into the room.\n\n\tINT DAY: HOTEL STAIRS (1955)\n\n\tCLEMENZA is climbing the back stairs of a large hotel.  He\n\trounds the corner, puffs a little, and then continues upward.\n\n\tINT DAY: CHURCH\n\n\tThe Church.  Close on the PRIEST's fingers as he gently\n\tapplies oil to the infant's ears and nostrils.\n\n\t\t\t\tPRIEST\n\t\tEphetha...be opened...So you may\n\t\tperceive the fragrance of God's\n\t\tsweetness.\n\n\tEXT DAY: ROCKEFELLER CENTER (1955)\n\n\tThe DRIVER of the limousine in front of Rockefeller Center\n\tis arguing with NERI.\n\n\tNow the DRIVER looks up.\n\n\tWHAT HE SEES:\n\n\tTWO MEN in topcoats exit the building, through the revolving\n\tglass doors.\n\n\tNERI opens up fire, trapping BARZINI in the shattering glass\n\tdoors.  The doors still rotate, moving the dead body of\n\tBARZINI within them.\n\n\tINT DAY: CHURCH\n\n\tIn the Church--the VIEW on MICHAEL.  The PRIEST hands him\n\tthe infant.\n\n\t\t\t\tPRIEST\n\t\tDo you renounce Satan.\n\n\t\t\t\tMICHAEL\n\t\tI do renounce him.\n\n\t\t\t\tPRIEST\n\t\tAnd all his works?\n\n\t\t\t\tMICHAEL\n\t\tI do renounce them.\n\n\tINT DAY: MOTEL MURDER (1955)\n\n\tLAMPONE, backed up by two other MEN in his regime, runs down\n\tthe iron-rail steps, and kicks in the door on Room 7F.\n\tPHILIP TATTAGLIA, old and wizened and naked, leaps up; a\n\tsemi-nude young GIRL leans up.\n\n\tThey are riddled with gunfire.\n\n\tINT DAY: HOTEL STAIRS (1955)\n\n\tCLEMENZA, huffing and puffing, climbs the back stairs, with\n\this package.\n\n\tINT DAY: CHURCH\n\n\tThe PRIEST pours water over the forehead of the infant\n\tMICHAEL holds.\n\n\t\t\t\tPRIEST\n\t\tDo you wish to be baptized?\n\n\t\t\t\tMICHAEL\n\t\tI do wish to be baptized.\n\n\tINT DAY: HOTEL ELEVATOR MURDER (1955)\n\n\tCLEMENZA, out of breath, climbs the final few steps.\n\n\tHe walks through some glass doors, and moves to an ornate\n\televator waiting shaft.\n\n\tThe lights indicate the elevator has arrived.\n\n\tThe doors open, and we see a surprised CUNEO standing with\n\tthe dapper MOE GREENE.\n\n\tCLEMENZA fires into the small elevator with a shotgun.\n\n\tThe PRIEST hands a lighted candle to MICHAEL.\n\n\t\t\t\tPRIEST\n\t\tI christen you Michael Francis Rizzi.\n\n\tFlash bulbs go off.  Everyone is smiles, and crowds around\n\tMICHAEL, KAY, CONNIE...and CARLO.\n\n\t--------------------------------------FADE OUT----------\n\n\tEXT DAY: CHURCH (1955)\n\n\tThe christening party outside the Church.\n\n\tFour or five limousines have been waiting; now pull up to\n\treceive MAMA, CONNIE and the baby; and the others.\n\n\tEveryone is very happy; only MICHAEL seems aloof and grave.\n\n\tAs the fuss is going on, a car pulls up.  LAMPONE gets out\n\tand works his way to MICHAEL.  He whispers in his ear.  This\n\tis the news MICHAEL has been waiting for.\n\n\tCONNIE holds the baby up to MICHAEL.\n\n\t\t\t\tCONNIE\n\t\tKiss your Godfather.\n\n\tThe infant turns its head, and MICHAEL uses that as an\n\texcuse to back away.\n\n\t\t\t\tMICHAEL\n\t\tCarlo...we've had a change in the\n\t\tplans.  Mama, Connie, Kay and the\n\t\tkids will have to take the trip out\n\t\tto Vegas without us.\n\n\t\t\t\tCONNIE\n\t\tOh Mike, it's our first vacation\n\t\ttogether.\n\n\t\t\t\tCARLO\n\t\t\t  (anxious to please)\n\t\tJesus, Connie...Sure, Mike...\n\n\t\t\t\tMICHAEL\n\t\tGo back to your house and wait for\n\t\tme...\n\n\tHe kisses KAY.\n\n\t\t\t\tMICHAEL\n\t\t\t  (to KAY)\n\t\tI'll just be a couple of days...\n\n\tPeople are guided to the correct limousines; they start to\n\tdrive off.\n\n\tINT DAY: DON'S KITCHEN\n\n\tTESSIO sits in the Kitchen of the Main House on the Mall.\n\n\tHAGEN enters.\n\n\t\t\t\tHAGEN\n\t\tYou'd better make your call to\n\t\tBarzini; Michael's ready.\n\n\tTESSIO nods; moves to the telephone and dials a number.\n\n\t\t\t\tTESSIO\n\t\tWe're on our way to Brooklyn.\n\n\tHe hangs up and smiles.\n\n\t\t\t\tTESSIO\n\t\tI hope Mike can get us a good deal\n\t\ttonight.\n\n\t\t\t\tHAGEN\n\t\t\t  (gravely)\n\t\tI'm sure he will.\n\n\tEXT DAY: MALL (1955)\n\n\tThe TWO MEN walk out onto the Mall, toward a car.  On their\n\tway they are stopped by TWO BODYGUARDS.\n\n\t\t\t\tBUTTON MAN\n\t\tThe boss says he'll come in a\n\t\tseparate car.  He says for you two\n\t\tto go on ahead.\n\n\t\t\t\tTESSIO\n\t\t\t  (frowning)\n\t\tHell, he can't do that.  It screws\n\t\tup all my arrangements.\n\n\tTHREE MORE BODYGUARDS appear around him.\n\n\t\t\t\tHAGEN\n\t\t\t  (gently)\n\t\tI can't go with you either, Tessio.\n\n\tHe flashes at the men surrounding him; for a moment he\n\tpanics, and then he accepts it.\n\n\t\t\t\tTESSIO\n\t\t\t  (after the pause)\n\t\tTell Mike it was business...I\n\t\talways liked him.\n\n\t\t\t\tHAGEN\n\t\tHe understands that.\n\n\tTESSIO looks at the men, and then pauses.\n\n\t\t\t\tTESSIO\n\t\t\t  (softly)\n\t\tTom, can you get me off the hook?\n\t\tFor old times' sake?\n\n\t\t\t\tHAGEN\n\t\tI can't.\n\n\tHAGEN turns, and walks away from the group.  Then about\n\ttwenty paces away, he stops, and looks back.\n\n\tTESSIO is led into a waiting car.\n\n\tHAGEN looks away, and walks off.\n\n\tINT DAY: CARLO'S LIVING ROOM (1955)\n\n\tCARLO RIZZI is alone in his house, smoking, waiting rather\n\tnervously.  He moves to the window and looks out.\n\n\tWHAT HE SEES:\n\n\tEXT DAY: MALL (1955)\n\n\tMICHAEL, still dressed in a dark suit; followed by NERI,\n\tLAMPONE and CLEMENZA, then HAGEN.\n\n\tThey move toward us.\n\n\tExcitedly, CARLO moves to the front door; opens it.\n\n\tHe wears a broad smile.\n\n\t\t\t\tCARLO\n\t\tGodfather!\n\n\t\t\t\tMICHAEL\n\t\tYou have to answer for Santino.\n\n\tThe smile on CARLO's face slowly fades, then, in a foolish\n\tattempt for safety, he slams the door in their faces and\n\tbacks into the living room.\n\n\tINT DAY: CARLO'S LIVING ROOM (1955)\n\n\tThe door opens, and the grim party enters.\n\n\t\t\t\tMICHAEL\n\t\tYou fingered Sonny for the Barzini\n\t\tpeople.  That little farce you\n\t\tplayed out with my sister.  Did\n\t\tBarzini kid you that would fool a\n\t\tCorleone?\n\n\t\t\t\tCARLO\n\t\t\t  (dignity)\n\t\tI swear I'm innocent.  I swear on\n\t\tthe head of my children, I'm\n\t\tinnocent.  Mike, don't do this to\n\t\tme, please Mike, don't do this to me!\n\n\t\t\t\tMICHAEL\n\t\t\t  (quietly)\n\t\tBarzini is dead.  So is Philip\n\t\tTattaglia, so are Strachi, Cuneo\n\t\tand Moe Greene...I want to square\n\t\tall the family accounts tonight.\n\t\tSo don't tell me you're innocent;\n\t\tadmit what you did.\n\n\tCARLO is silent; he wants to talk but is terrified.\n\n\t\t\t\tMICHAEL\n\t\t\t  (almost kindly)\n\t\tDon't be frightened.  Do you think\n\t\tI'd make my sister a widow?  Do you\n\t\tthink I'd make your children\n\t\tfatherless?  After all, I'm\n\t\tGodfather to your son.  No, your\n\t\tpunishment is that you're out of\n\t\tthe family business.  I'm putting\n\t\tyou on a plane to Vegas--and I want\n\t\tyou to stay there.  I'll send\n\t\tConnie an allowance, that's all.\n\t\tBut don't keep saying you're\n\t\tinnocent; it insults my intelligence\n\t\tand makes me angry.  Who approached\n\t\tyou, Tattaglia or Barzini?\n\n\t\t\t\tCARLO\n\t\t\t  (sees his way out)\n\t\tBarzini.\n\n\t\t\t\tMICHAEL\n\t\t\t  (softly)\n\t\tGood, good.  Leave now; there's a\n\t\tcar waiting to take you to the\n\t\tairport.\n\n\tCARLO moves to the door; opens it.  There is a car waiting;\n\twith a group of MEN around it.\n\n\tHe looks back at MICHAEL, who reassures him.\n\n\t\t\t\tMICHAEL\n\t\tI'll call your wife and tell her\n\t\twhat flight you're on.\n\n\tEXT DAY: MALL\n\n\tCARLO moves out to the Mall; the BUTTONMEN are putting his\n\tthings in the trunk.\n\n\tONE opens the front door for him.\n\n\tSOMEONE is sitting in the rear seat, though we cannot see who.\n\n\tCARLO gets into the car; out of nervousness, he looks back\n\tto see the other man.\n\n\tIt is CLEMENZA, who nods cordially.\n\n\tThe motor starts, and as the car pulls away, CLEMENZA\n\tsuddenly throws the garrote around CARLO's neck.  He chokes\n\tand leaps up like a fish on a line, kicking his feet.\n\n\tThe garrote is pulled tighter; CARLO's face turns color.\n\n\tHis thrashing feet kick right through the front windshield.\n\n\tThen the body goes slack.\n\n\tCLEMENZA makes a foul face, and opens the window as the car\n\tdrives off.\n\n\tEXT DAY: CARLO'S STEPS (1955)\n\n\tMICHAEL and his party.  They watch.\n\n\tThen he turns and walks off, and they follow.\n\n\t---------------------------------------FADE OUT---------\n\n\tFADE IN:\n\n\tINT NITE: MICHAEL'S LIMO EN ROUTE (1955)\n\n\tMICHAEL sits alone in the back of his car; NERI is driving.\n\n\tThey do not speak for a long time; it is night--car lights\n\tflash by.\n\n\tNERI turns back.\n\n\t\t\t\tNERI\n\t\tYou know I would never question\n\t\tanything you say.\n\n\t\t\t\tMICHAEL\n\t\t\t  (smiles)\n\t\tSpeak your mind.\n\n\t\t\t\tNERI\n\t\tI'll do this for you; you know I\n\t\tshould.\n\n\t\t\t\tMICHAEL\n\t\tNo.  This I have to do.\n\n\tEXT NITE: PIZZA STREET (1955)\n\n\tMICHAEL's car pulls up in a quiet neighborhood, near an\n\tItalian Pizzeria.  NERI opens the door.\n\n\t\t\t\tMICHAEL\n\t\tSit in the car.\n\n\tINT NITE: PIZZA PLACE (1955)\n\n\tHe walks alone into the restaurant.  A MAN is tossing pizza\n\tdough in the air.\n\n\t\t\t\tMICHAEL\n\t\tWhere's the boss?\n\n\t\t\t\tMAN\n\t\tIn the back.  Hey Frank, someone\n\t\twants you.\n\n\tA MAN comes out of the shadows, with a strong Italian accent.\n\n\t\t\t\tMAN\n\t\tWhat is it?\n\n\tHe stops, frozen in fear.  It is FABRIZZIO.\n\n\tVIEW ON MICHAEL.  Gunfire from under his coat.  FABRIZZIO is\n\tcut down.  MICHAEL throws the gun down; turns and exits.\n\n\tEXT DAY: MALL (1955)\n\n\tHIGH ANGLE ON THE CORLEONE MALL\n\n\tSeveral moving vans are parked in the Mall; one feels that\n\tthese are the final days; the families are moving out; signs\n\tindicating that the property is for sale are evident.\n\n\tA black limousine pulls up, and before it has even stopped,\n\tthe rear door flies open, and CONNIE attempts to run out,\n\trestrained by MAMA.  She manages to break free and runs\n\tacross the Mall into Michael's house.\n\n\tINT DAY: DON'S LIVING ROOM (1955)\n\n\tInside the Corleone house.  Big boxes have been packed;\n\tfurniture prepared for shipping.\n\n\t\t\t\tCONNIE\n\t\tMichael!\n\n\tShe hurries into the living room, where she comes upon\n\tMICHAEL and KAY.\n\n\t\t\t\tKAY\n\t\t\t  (comforting)\n\t\tConnie...\n\n\tBut CONNIE avoids her, and moves directly to MICHAEL.  NERI\n\tis watchful.\n\n\t\t\t\tCONNIE\n\t\tYou lousy bastard; you killed my\n\t\thusband...\n\n\t\t\t\tKAY\n\t\tConnie...\n\n\t\t\t\tCONNIE\n\t\tYou waited until our father died\n\t\tand nobody could stop you and you\n\t\tkilled him, you killed him!  You\n\t\tblamed him about Sonny, you always\n\t\tdid, everybody did.  But you never\n\t\tthought about me, never gave a damn\n\t\tabout me.\n\t\t\t  (crying)\n\t\tWhat am I going to do now, what am\n\t\tI going to do.\n\n\tTWO of Michael's BODYGUARDS move closer, ready for orders\n\tfrom him.  But he stands there, waiting for his sister to\n\tfinish.\n\n\t\t\t\tKAY\n\t\tConnie, how could you say such\n\t\tthings?\n\n\t\t\t\tCONNIE\n\t\tWhy do you think he kept Carlo on\n\t\tthe Mall?  All the time he knew he\n\t\twas going to kill my husband.  But\n\t\the didn't dare while my father was\n\t\talive.  And then he stood Godfather\n\t\tto our child.  That coldhearted\n\t\tbastard.\n\t\t\t  (to KAY)\n\t\tAnd do you know how many men he had\n\t\tkilled with Carlo?  Just read the\n\t\tpapers.  That's your husband.\n\n\tShe tries to spit into MICHAEL's face; but in her hysteria\n\tshe has no saliva.\n\n\t\t\t\tMICHAEL\n\t\tGet her home and get a doctor.\n\n\tThe TWO BODYGUARDS immediately take her arms and move her,\n\tgently but firmly.\n\n\tKAY is shocked; never taking her look of amazement from\n\tMICHAEL.  He feels her look.\n\n\t\t\t\tMICHAEL\n\t\tShe's hysterical.\n\n\tBut KAY won't let him avoid her eyes.\n\n\t\t\t\tKAY\n\t\tMichael, it's not true.  Please\n\t\ttell me.\n\n\t\t\t\tMICHAEL\n\t\tDon't ask me.\n\n\t\t\t\tKAY\n\t\tTell me!\n\n\t\t\t\tMICHAEL\n\t\tAll right, this one time I'll let\n\t\tyou ask about my affairs, one last\n\t\ttime.\n\n\t\t\t\tKAY\n\t\tIs it true?\n\n\tShe looks directly into his eyes, he returns the look, so\n\tdirectly that we know he will tell the truth.\n\n\t\t\t\tMICHAEL\n\t\t\t  (after a very long pause)\n\t\tNo.\n\n\tKAY is relieved; she throws her arms around him, and hugs\n\thim.  Then she kisses him.\n\n\t\t\t\tKAY\n\t\t\t  (through her tears)\n\t\tWe both need a drink.\n\n\tINT DAY: DON'S KITCHEN (1955)\n\n\tShe moves back into the kitchen and begins to prepare the\n\tdrinks.  From her vantage point, as she smilingly makes the\n\tdrinks, she sees CLEMENZA, NERI and ROCCO LAMPONE enter the\n\thouse with their BODYGUARDS.\n\n\tShe watches with curiosity, as MICHAEL stands to receive\n\tthem.  He stands arrogantly at ease, weight resting on one\n\tfoot slightly behind the other.  One hand on his hip, like a\n\tRoman Emperor.  The CAPOREGIMES stand before him.\n\n\tCLEMENZA takes MICHAEL's hand, kissing it.\n\n\t\t\t\tCLEMENZA\n\t\tDon Corleone...\n\n\tThe smile fades from KAY's face, as she looks at what her\n\thusband has become.\n\n\tINT DAY: CHURCH (1955)\n\n\tKAY wears a shawl over her hand.  She drops many coins in\n\tthe coin box, and lifts a burning taper, and one by one, in\n\ta pattern known only to herself, lights thirty candles.\n\n\t\t\t\t\tTHE END\n\n\n\n\n
\n
\n\n\n" }, { "path": "samples/go/decent/data/godfather2.html", "content": "\n\n
\n
\n\nThe Godfather: Part two\n\n\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\t\t\t\tTHE GODFATHER\n\n\t\t\t\t Part Two\n\n\t\t\t\tScreenplay by\n\n\t\t\t\tMario Puzo\n\n\t\t\t\t    and\n\n\t\t\t Francis Ford Coppola\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nSECOND DRAFT\n\nSeptember 24, 1973\n\n\n\n\n\nFADE IN:\n\nThe Paramount Pictures logo is presented over a simple black\nbackground, as a single trumpet plays the familiar theme of\na waltz.  White lettering fades in:\n\n\t\t     Mario Puzo's THE GODFATHER\n\nThere is a pause, as the trumpet concludes, and there is the\nadditional title: - Part Two -\n\nINT. DON CORLEONE'S OLD OFFICE - CLOSE VIEW ON MICHAEL\nCORLEONE - DAY\n\nstanding impassively, like a young Prince, recently crowned\nKing.\n\nCLOSE VIEW ON Michael's hand.  ROCCO LAMPONE kisses his hand.\nThen it is taken away.  We can SEE only the empty desk and\nchair of Michael's father, Vito Corleone.  We HEAR, over\nthis, very faintly a funeral dirge played in the distance,\nas THE VIEW MOVES SLOWLY CLOSER to the empty desk and chair.\n\n\t\t\t\t\t\t\t\tDISSOLVE TO:\n\nEXT. A SICILIAN LANDSCAPE - FULL VIEW - DAY\n\nWe can barely make out the funeral procession passing over\nthe burnt-brown of a dry river bed.  The figures move\nslowly, seemingly from out of hundreds of years of the past.\n\nThe MUSICIANS walking unsteadily on the rocky bed, their\ninstruments harsh and blaring.\n\nThey are followed by six young peasant men, carrying the\ncrude wooden coffin on their shoulders.  Then the widow, a\nstrong large woman, dressed in black, and not accepting the\narms of those walking with her.\n\nBehind her, not more than twenty relatives, few children and\npaisani continue alone behind the coffin.\n\nSuddenly, we HEAR the shots of the lupara, and the musicians\nstop their playing.  The entire procession scatters in odd\ndirections along the rocky river bed.\n\nThe young men struggle with the burden of the heavy coffin,\nthrowing it out of balance and nearly crashing to the ground.\nWe hear a woman SCREAMING:\n\n\t\t\t\tWOMAN\n\t\t\t(Sicilian)\n\t\tThey've killed young Paolo!  They've\n\t\tkilled the boy Paolo!\n\nEXT. SICILIAN LANDSCAPE - MED. VIEW - DAY\n\nacross the slain body of a fourteen year old boy, lying on\nthe parched ground.  In the distance we see four or five of\nthe mourning women, the wind blowing their black dresses and\nveils, running up to the body of the boy.  They begin to\nwail, and cry out in anguished Sicilian, as the widow, the\nmother of the murdered boy, holds her child in her arms, his\nfresh blood wetting her strong hands.\n\nEXT. BARONIAL ESTATE - TIGHT MOVING VIEW - DAY\n\nA boy, eight or nine, with wide, frightened eyes, being\npulled quickly by the hand.  This is VITO ANDOLINI, who is\nto become The Godfather.\n\nThe VIEW ALTERS revealing that he is being pulled along by\nhis Mother, the Widow, across a field leading to the\nornamental gates of a Baronial Estate of some forgotten Noble.\n\nAt various positions near the gates are men with shotguns,\nor lupara.  The gates are opened; and the Widow and her boy\nare shown before DON FRANCESCO, a man in his sixties.  He\nwears his trousers with suspenders, and an open white shirt\nsloppily tucked in over his enormous belly.  He wears a hat\nto protect him from the white-hot sun, and proudly displays\na gold watch and chain over his vest.\n\nHe sits in a chair, near a group of his men in the garden,\nlistening to the Widow, who stands before him with her only\nson.\n\n\t\t\t\tWIDOW\n\t\t\t(Sicilian)\n\t\tDon Francesco.  You murdered my\n\t\thusband, because he would not bend.\n\t\tAnd his oldest son Paolo, because\n\t\the swore revenge.  But Vitone is\n\t\tonly nine, and dumb-witted.  He\n\t\tnever speaks.\n\n\t\t\t\tDON FRANCESCO\n\t\t\t(Sicilian)\n\t\tI'm not afraid of his words.\n\n\t\t\t\tWIDOW\n\t\t\t(Sicilian)\n\t\tHe is weak.\n\n\t\t\t\tDON FRANCESCO\n\t\t\t(Sicilian)\n\t\tHe will grow strong.\n\n\t\t\t\tWIDOW\n\t\t\t(Sicilian)\n\t\tThe child cannot harm you.\n\n\t\t\t\tDON FRANCESCO\n\t\t\t(Sicilian)\n\t\tHe will be a man, and then he will\n\t\tcome for revenge.\n\nAs she pleads, the Widow moves closer to the Don, until she\nhas practically thrown herself to her knees before him.\n\n\t\t\t\tWIDOW\n\t\t\t(Sicilian)\n\t\tI beg you, Don Francesco, spare my\n\t\tonly son.  He is all I have.  In\n\t\tthe name of the Holy Spirit, I\n\t\tswear he will never be a danger to\n\t\tyou...\n\nSuddenly, she reaches under her skirt, where she has hidden\na kitchen knife.\n\n\t\t\t\tWIDOW\n\t\t\t(continuing)\n\t\tBut I will kill you myself!\n\t\t\t(she lunges at the\n\t\t\tMafia chieftain)\n\t\tVitone, go!\n\nThe boy runs as fast as he can out through the gates.  Then\nthere is a lupara blast.  He turns, and sees his Mother\nflung a distance of five feet from the short range of the\nterrible blast of the shotgun.  Then he sees the men turn\ntheir attention to him.  One fires at him; but the boy is\nquick, and disappears into a grove of olive trees.\n\nEXT. STREETS OF CORLEONE - NIGHT\n\nTwo men roam the deserted streets of Corleone, carrying\nlupare.  Every so often, they stop, and one shouts in a\nloud, almost singsong voice, like a fish peddler.  Their\nnames are MOSCA and STROLLO.\n\n\t\t\t\tMOSCA\n\t\t\t(Sicilian)\n\t\tOur Friend promises misery to\n\t\tanyone who harbors the boy Vito\n\t\tAndolini.\n\t\t\t(he turns and shouts\n\t\t\tin the other direction)\n\t\tOur Friend promises misery to\n\t\tanyone who harbors the boy Vito\n\t\tAndolini.\n\nINT. A HOUSE - NIGHT\n\nA family quietly eats their dinner.  The father is the local\npoliceman, as indicated by his uniform jacket and gun,\nhanging nearby.\n\n\t\t\t\tSTROLLO\n\t\t\t(Sicilian, O.S.)\n\t\tOur Friend will be hard with any\n\t\tfamily who gives help to Vito\n\t\tAndolini.\n\nOne of the children looks up, about to speak.  But the\nfather sternly indicates that nothing must be said.  They go\non with their dinner.\n\nEXT. THE STREETS OF CORLEONE - FULL VIEW - NIGHT\n\nThe men continue walking up and throughout the streets, far\nin the distance.\n\n\t\t\t\tMOSCA\n\t\t\t(Sicilian O.S.)\n\t\t...misery to any family who harbors\n\t\tthe boy, Vito...\n\nINT. A BARN - NIGHT\n\nFour little girls watch with wide eyes as their mother and\nfather bind Vito tightly in swaddled cloth, and then lift\nhim up to the side of a mule; counter-balancing a heavy load\nof firewood.  The father looks at the boy's almost stoically\ncalm little face.\n\n\t\t\t\tFATHER\n\t\t\t(Sicilian)\n\t\tVito...We pray for you.\n\nHe pulls the fabric over the boy's face.\n\n\t\t\t\tMOSCA\n\t\t\t(Sicilian O.S.)\n\t\t...Andolini...\n\n\t\t\t\tSTROLLO\n\t\t\t(Sicilian O.S.)\n\t\tOur Friend promises misery to any\n\t\tfamily...\n\nEXT. THE CHURCH PLAZA - NIGHT\n\nThe men continue on their night-walk, up to the plaza of the\nchurch.\n\n\t\t\t\tSTROLLO\n\t\t\t(Sicilian)\n\t\t...who harbors the boy Vitone\n\t\tAndolini.\n\nThe figure of a single man on a mule passes them.\n\n\t\t\t\tMOSCA\n\t\t\t(Sicilian)\n\t\tLet no one give help to the boy\n\t\tVito Andolini...\n\nThe man on the mule makes his way out of the village and\ndisappears into the distance.\n\nWe begin to hear, very quietly, the Waltz repeated once again.\n\nEXT. STEAMSHIP - CLOSE VIEW ON VITO - DAY\n\nhuddled in blankets, on the deck of the ship in Steerage.\nHe does not say a word.  The Waltz grows louder as the VIEW\nALTERS, revealing the hundreds of immigrant families huddled\ntogether with all their earthly possessions on their way to\nAmerica.\n\nThen, suddenly, the Waltz stops.\n\nTHE NEW YORK HARBOR - DAY\n\nSILENCE.  We glide past the Statue of Liberty.\n\nVIEW on the IMMIGRANTS standing on shipboard silently;\nlooking.  Vito is standing with them, his eyes wide.\n\nCAMERA MOVES IN on the statue, then MOVING PAST, on to the\nbeautiful buildings of Ellis Island.\n\nEXT. ELLIS ISLAND - DAY\n\nA tugboat pulls a barge brimming with immigrants into the\nEllis Island harbor.  Uniformed officials of the Immigration\nService load them up toward the main building.\n\nINT. ELLIS PROCESSING HALL - DAY\n\nThe hundreds of immigrant families sit on rows of benches in\nthe great hall.  Various painted lines lead to the steps and\nprocessing rooms above.\n\nThere is the babble of many interviews going on\nsimultaneously, uncertainly, in different languages.\n\nVito is bundled in an old coat, with a large tag pinned on\nit: \"Vitone Andolini -- Corleone, Sicilia.\"\n\nHe stands, moves up in the line, when several other immigrant\nboys, older than he, rush up an push him back in the line.\nWeak from the trip, he falls to the floor.  The boys laugh,\nderisive in a language he cannot understand.  He struggles\nto his feet, lifting his makeshift bags; staring at them in\nan icy hatred.\n\nINT. PROCESSING ROOM - DAY\n\nThree or four interviews are crowded into the small room;\nthey are conducted in English.  From the expression on\nVito's face, and from the fragmented of the English, we\nrealize that he doesn't understand a word of it.\n\n\t\t\t\tOFFICIAL\n\t\t\t(English)\n\t\tWhat is your name?\n\nThe man waits, impatiently.\n\n\t\t\t\tOFFICIAL\n\t\tYour name?\n\nVito doesn't answer.  The Official pulls the tag pinned onto\nhis coat and copies to down on his form, using a typewriter.\n\n\t\t\t\tOFFICIAL\n\t\t\t(speaking as he types)\n\t\tVito...Corleone.  Step up, over\n\t\tthere.\n\nHe hands the form to another official.\n\nCLOSE VIEW on the form.  The name has been entered as Vito\nCorleone.\n\nINT. MEDICAL EXAM - DAY\n\nVito is stripped to the waist, as other immigrants wait.\n\nThe DOCTOR is just finishing his examination.  He shakes his\nhead, and then writes on the medical form.\n\n\t\t\t\tDOCTOR\n\t\tCan you understand me?\n\nVito stares blankly.\n\n\t\t\t\tDOCTOR\n\t\tYou understand?  Smallpox.  Smallpox.\n\nHe doesn't understand.  The doctor turns to the Immigration\nOfficial.\n\n\t\t\t\tDOCTOR\n\t\tQuarantine...six months.\n\nUNDERGROUND PASSAGEWAY - MOVING VIEW - DAY\n\nOfficials move a group of immigrant men, including Vito, to\nthe quarantine section of the Island.\n\nINT. QUARANTINE HALLWAY - DAY\n\nThe official stops at each doorway, and reads off a name.\n\n\t\t\t\tOFFICIAL\n\t\tSalvatore Ormenta.\n\nThe man moves into the room, and the group proceeds.\n\n\t\t\t\tOFFICIAL\n\t\tVito Corleone.\n\nNo one responds.  The guard moves to the boy, reads his new\nname tag.  And then, not unkindly:\n\n\t\t\t\tGUARD\n\t\tThat's you.\n\nHe opens the door, and Vito enters the room.\n\nEXT. THE STATUE OF LIBERTY - DAY\n\nThe VIEW slowly begins to pull back, revealing this to be\nthe view from inside the quarantine cell, where Vito stands\non his bench, looking out to the statue through the barred\nwindow.\n\nThen he turns, and sits in the corner.  He is silent for a\nlong time.\n\nThen, in a sweet, pure voice, he sings to himself in Sicilian.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nINT. CATHOLIC CHURCH - MOVING CLOSE SHOT - DAY\n\nA nine year old boy, dressed immaculately in white, with a\nlarge white silk bow tied to his shoulder, moving slowly\ndown the aisle of the church with a group of other children\ndressed in white.  He has dark black hair, and his face is\nunmistakably similar to young Vito's.  He moves slowly, his\nhands clasped around a golden missal.  We HEAR only the pure\nvoice of Vito in Sicilian, his sad song reaching out from\nthe past, as ANTHONY CORLEONE, his Grandson, moves on the\nway to his First Holy Communion more than fifty years later.\n\nFULL VIEW\n\nThe little children move in procession down to the Altar,\nwhere the PRIEST raises the Host, and performs the Communion\nMass in Latin.\n\n\t\t\t\tPRIEST\n\t\tEcce Agnus Dei, ecce qui tollit\n\t\tpeccata mundi.\n\nMOVING VIEW ON THE PRIEST\n\nand Altar boys, as he moves along the row of kneeling\nchildren, blessing them, and administering their first\nCommunion.\n\nCLOSE MOVING VIEW\n\nas the innocent faces receive the Host; finally, the Priest\ncomes to Anthony.\n\n\t\t\t\tPRIEST\n\t\tCorpus Christi.\n\n\t\t\t\tANTHONY\n\t\tAmen.\n\nEXT. LAKE TAHOE ESTATE - DAY\n\nThe lawns of this great estate on the shore of Lake Tahoe\nare covered with guests of a wonderful party to honor the\nFirst Holy Communion of Anthony Corleone, the son of Mr. and\nMrs. Michael Corleone.  A full dance orchestra plays music\nof the times on a pavilion bandstand built especially for\nthe occasion.  Speedboats roar through the water, pulling\nyouthful waterskiers; and the pool and private harbor are\nfilled with laughing, swimming guests.  It is Fall of 1958.\n\nMED. VIEW\n\nAnthony, in his Communion suit sits alone at the table,\nlooking like a lonely young Prince.\n\n\t\t\t\tKAY (O.S.)\n\t\tSmile, Anthony.  Smile.\n\nHe does, and a flash goes off.\n\n\t\t\t\tPHOTOGRAPHER (O.S.)\n\t\tNow, one with the whole family.\n\n\t\t\t\tKAY (O.S.)\n\t\tMr. Corleone can't right now...\n\nKAY CORLEONE enters from the side, leading her four year old\ndaughter, MARY, and MAMA CORLEONE to pose with Anthony.\n\n\t\t\t\tKAY (O.S.)\n\t\t...but we'll get one with the ladies.\n\n\t\t\t\tPHOTOGRAPHER\n\t\tAll together now, c'mon, Anthony...\n\t\tCHEESE and\n\t\t\t(flash)\n\n\t\t\t\tKAY\n\t\tThank you.\n\nShe smiles as she leaves the photographer, and then lets out\na weary sigh to Mama, as she touches the slightly protruding\nbelly.\n\n\t\t\t\tKAY\n\t\tDo you think it'll show in the\n\t\tpicture?\n\n\t\t\t\tMAMA\n\t\tTwo months never shows.  Two months\n\t\tlook like you had a big lunch.\n\n\t\t\t\tVOICE (O.S.)\n\t\tOh, Mrs. Corleone.\n\nA slender, aristocratic WOMAN in her late forties is waving\nto KAY.\n\n\t\t\t\tMRS. BARRETT\n\t\tHello, Mrs. Corleone.  I'm Fran\n\t\tBarrett, our place is just down the\n\t\tlake.  This is my husband, Marshall.\n\n\t\t\t\tKAY\n\t\tI'm so happy you could come.\n\n\t\t\t\tMR. BARRETT\n\t\tThe place is transformed.  We've\n\t\tbeen watching workmen come and go\n\t\tall summer.\n\n\t\t\t\tMRS. BARRETT\n\t\tWhere is Mr. Corleone?\n\n\t\t\t\tKAY\n\t\tA business meeting ran late...but\n\t\the promised he wouldn't be long.\n\nKay puts her arm around little Anthony's shoulder.\n\n\t\t\t\tKAY\n\t\tThis is our son Anthony Vito\n\t\tCorleone.  Today he made his First\n\t\tHoly Communion.\n\nEXT. TAHOE GATE AND KENNELS - DAY\n\nA confusion of cars; arriving and parking.  The squad of\nparking attendants are supplemented by a whole team of the\nlocal Police, working as high-class parking valets.\n\nA very beautiful, statuesque woman, though slightly drunk,\nDEANNA DUNN, slams the door of a powder blue Mercedes and\nhurries barefoot through the great stone gate.\n\n\t\t\t\tDEANNA\n\t\tI will not shut my mouth, and keep\n\t\tyour Goddamn hands off of me!\n\nShe is followed by a harried, FREDDIE CORLEONE, dressed with\nflash in the Hollywood style, and carrying her shoes in his\nhands.\n\n\t\t\t\tFREDO\n\t\tHoney!  Wait a minute; let's go for\n\t\ta drive.\n\n\t\t\t\tDEANNA\n\t\tI just had a drive; besides, I want\n\t\tto see my brother-in-law Michael.\n\n\t\t\t\tFREDO\n\t\t\t(trying to get her to\n\t\t\tput her shoes on)\n\t\tYeah, but I don't want him to see\n\t\tyou.\n\nDeanna pauses reflectively a moment, allowing Fredo to get\nher shoes on.\n\n\t\t\t\tDEANNA\n\t\tWhat beats me, is how you guys\n\t\tcould be brothers.  You musta been\n\t\tyour Mother's rotten egg.\n\nShe kicks off the shoes, giggling, and runs toward a waiter.\n\n\t\t\t\tDEANNA\n\t\t\t(lifting a glass of champagne)\n\t\tYoung man, young man...thank you,\n\t\tyoung man.\n\n\t\t\t\tWAITER\n\t\t\t(impressed)\n\t\tExcuse me, but aren't you...\n\n\t\t\t\tDEANNA\n\t\tYes, you saw me in the movies, Good\n\t\tHumor man, and yes, I had more off\n\t\tthan my shoes!\n\n\t\t\t\tFREDO\n\t\tGoddamn bitch.\n\n\t\t\t\tDEANNA\n\t\tRelax, Freddie honey.  Come dance\n\t\twith me.\n\nShe extends her hand to him.\n\n\t\t\t\tFREDO\n\t\tListen, Michael's got a lot of nice\n\t\tpeople here.  Friends of Kay's.\n\t\tHe'll never forgive me if you ruin\n\t\this party.\n\n\t\t\t\tDEANNA\n\t\tI hate to see you cringe in front\n\t\tof him.  How come you're so scared\n\t\tof your own kid brother?\n\n\t\t\t\tFREDO\n\t\tHe's the head of the family.\n\nDisgusted, she turns around, and heads toward the music.\n\n\t\t\t\tDEANNA\n\t\tDon't follow me!\n\nEXT. TAHOE LAWN AND TABLES - MED. SHOT - DAY\n\nRushing through the tables, waving an arm jangling with gold\njewelry, and carrying several gift-wrapped packages, is a\nhardened and aging CONNIE CORLEONE.  She is followed by a\nblond, and wrinkled-handsome escort named MERLE.\n\n\t\t\t\tCONNIE\n\t\tMama...Mama!  Here I am!\n\nShe throws her arms around her Mother, who returns the\naffection somewhat reproachfully.\n\n\t\t\t\tMAMA\n\t\tConstanzia.  We expected you last\n\t\tweek; we sent the car to pick you\n\t\tup at the airport last week.\n\n\t\t\t\tCONNIE\n\t\tI know, it was chaos; but anyway,\n\t\there I am one week late.\n\t\t\t(lifting a shiny\n\t\t\tgreen package out of\n\t\t\tMerle's arms)\n\t\tThis is for my Mama.  You remember\n\t\tMerle?\n\n\t\t\t\tMAMA\n\t\t\t(not giving him a\n\t\t\tchance to greet her)\n\t\tYes, thank you.\n\n\t\t\t\tCONNIE\n\t\tHow are the kids?\n\n\t\t\t\tMAMA\n\t\tWell, thank you, they asked for you\n\t\tall week.\n\n\t\t\t\tCONNIE\n\t\tI got surprises for everybody!\n\n\t\t\t\tMAMA\n\t\t\t(glancing at the wrapping)\n\t\tBought at the airport.\n\n\t\t\t\tCONNIE\n\t\t\t(gazing about)\n\t\tThis is swell.  Where's Michael?\n\t\tI've got things to get straight\n\t\twith him and I can't wait on line.\n\n\t\t\t\tMAMA\n\t\tYou go see your children first, and\n\t\tthen you wait to see your brother\n\t\tlike everybody else.\n\nEXT. THE BOATHOUSE - DAY\n\nA porch-like foyer of the boathouse, where a group of five\nor six men wait, some nervously.  Some sit, and some pace.\n\nMED. CLOSE VIEW\n\non one of these men, FRANKIE PENTANGELI, approaching his\nsixties, with gray hair (the little of it left).  He's a bit\nscruffy, this morning's shave of his white beard is not\nperfect, and he seems tired.  He is accompanied by an\nassociate-bodyguard, WILLY CICCI; thin and dark, and also\ndressed up for the occasion.  Frankie tries to get the\nattention of one of the waiters; a college-groomed young man\nin white sports jacket and black bow-tie.\n\n\t\t\t\tPENTANGELI\n\t\tHey, kid!  You got any red wine?\n\n\t\t\t\tWAITER\n\t\t\t(offering the tray)\n\t\tOnly champagne and cocktails.\n\n\t\t\t\tPENTANGELI\n\t\tForget it...\n\nFinally, he sees someone he recognizes, Fredo, and shouts\nout in a husky voice:\n\n\t\t\t\tPENTANGELI\n\t\tFredo!  Sonuvabitch.  You look great.\n\nFredo squints in his direction; finally recognizes him.\n\n\t\t\t\tFREDO\n\t\tWho's that?  Pentangeli?  Frankie\n\t\t\"Five-Angels\"...thought you were\n\t\tnever coming West.\n\n\t\t\t\tPENTANGELI\n\t\t\t(affectionately)\n\t\tGotta check up on my boys.  Hey,\n\t\twhat's with the food?  Some kid in\n\t\ta white jacket brings me a ritz\n\t\tcracker with some chopped liver.\n\t\t'Canapes,' he says.  I say, 'Can a\n\t\tpeas, my ass, that's a ritz cracker\n\t\twith chopped liver.' Go get me a\n\t\tsalami sandwich and a glass of wine\n\t\tor I'll send you and your white\n\t\tjacket to the dry cleaners!\n\nThey get a good laugh at this fresh breath of New York.\n\n\t\t\t\tFREDO\n\t\tGee, Frankie, it's good to see you.\n\t\tReminds me of old times.\n\n\t\t\t\tPENTANGELI\n\t\tYou remember Willy Cicci, don't\n\t\tyou, Freddie?  We was all together\n\t\twith the old man Clemenza in\n\t\tBrooklyn... before...uh...\n\n\t\t\t\tFREDO\n\t\tWe were all upset about that.\n\n\t\t\t\tPENTANGELI\n\t\tThat's what I'm here to talk to\n\t\tyour brother about.  What's with\n\t\thim, I got to get a letter of\n\t\tintroduction to have a 'sitdown'?\n\n\t\t\t\tFREDO\n\t\t\t(throwing his arm\n\t\t\taround him)\n\t\tC'mon, I see what I can do.\n\nEXT. TAHOE PAVILION - MED. VIEW - DAY\n\nThe orchestra wears white summer sportcoats and black tuxedo\nslacks as they play a tango behind monogrammed music stands.\nA professional dance team, probably imported from Vegas,\ndance the tango for the excited guests.\n\nINT. TAHOE BOATHOUSE - DAY\n\nA large and very beautiful room overlooking the lake.  It is\ndominated by an enormous bar, behind which stands ALBERT\nNERI, discreetly in the background.\n\nMICHAEL CORLEONE sits on a large sofa, his back to us.\nStanding to one side is a tired and somewhat uneasy TOM\nHAGEN.  Standing before Michael is SANDRA CORLEONE, Sonny's\nwidow; her daughter, one of the twins, FRANCESCA CORLEONE,\nand a handsome young man of twenty, GARDNER SHAW.\n\n\t\t\t\tSANDRA\n\t\tMichael, this is Gardner Shaw.\n\t\tFrancesca and he have been seeing\n\t\teach other for six months now.\n\t\tGardner, this is Francie's Uncle\n\t\tMichael.\n\n\t\t\t\tGARDNER\n\t\t\t(a little nervous)\n\t\tI've heard a lot about you, Mr.\n\t\tCorleone.\n\n\t\t\t\tMICHAEL (O.S.)\n\t\tSit down.  Francie.\n\nThe couple sit themselves on the sofa opposite Michael.\n\n\t\t\t\tSANDRA\n\t\tThey would like to set an engagement\n\t\tdate, and...\n\n\t\t\t\tMICHAEL\n\t\tLet them speak for themselves.\n\nVIEW ON MICHAEL, calm, thoughtful.  One can tell that he has\nspecial affection for his niece.\n\n\t\t\t\tFRANCESCA\n\t\tWe love each other, Uncle Michael.\n\t\tAnd, we want to be married.  I came\n\t\tto ask for your blessing.\n\nThere is a loud KNOCKING on the door; then Fredo's voice.\n\n\t\t\t\tFREDO (O.S.)\n\t\tHey, Mike...guess who's here?\n\nNeri goes to answer it, cracks the door open.\n\n\t\t\t\tNERI\n\t\tNot now, Freddie...\n\n\t\t\t\tFREDO\n\t\tTell Mike Frankie 'Five-Angels' is\n\t\there.\n\n\t\t\t\tNERI\n\t\tNot now...\n\nNeri closes the door, and Michael looks at the nervous young\nman.\n\n\t\t\t\tMICHAEL\n\t\tFrancesca is my oldest brother's\n\t\tdaughter.  He died many years ago,\n\t\tand ever since I've felt much more\n\t\tof a father than an uncle.  I love\n\t\ther very much.  I'm pleased and\n\t\timpressed that you had the thought\n\t\tto come to me before going on with\n\t\tyour plans.  It shows me that\n\t\tyou're a considerate man, and will\n\t\tbe good to her.  What are you\n\t\tstudying in college?\n\n\t\t\t\tGARDNER\n\t\tMy major is Fine Arts, sir.\n\n\t\t\t\tMICHAEL\n\t\tHow will Fine Arts support your new\n\t\twife?\n\n\t\t\t\tGARDNER\n\t\tIt's embarrassing to say, sir, but\n\t\tI'm a major stockholder in the\n\t\tfamily corporation.\n\n\t\t\t\tMICHAEL\n\t\t\t(smiling)\n\t\tNever be embarrassed by your wealth.\n\t\tThis recent contempt for money is\n\t\tstill another trick of the rich to\n\t\tkeep the poor without it.\n\t\t\t(warmly)\n\t\tOf course I give you my blessing.\n\t\tLet's set the wedding soon...it\n\t\twill be my pleasure to give the\n\t\tbride away.\n\nThey all smile, and rise.\n\n\t\t\t\tMICHAEL\n\t\t\t(continuing)\n\t\t...and take a few courses in\n\t\tBusiness Administration just to be\n\t\ton the safe side!\n\nThey laugh; Michael moves toward them.  Francesca throws her\narms around him, and kisses her favorite uncle.  The flushed\nyoung man shakes his hand heartily.\n\n\t\t\t\tFRANCESCA\n\t\tThank you, Uncle Michael.\n\nThey all take their leave; Michael turns to Hagen.\n\n\t\t\t\tMICHAEL\n\t\tMake her dowry impressive.  He\n\t\tcomes from a family who still\n\t\tthinks an Italian bride goes\n\t\tbarefoot.\n\nEXT. TAHOE SWIMMING POOLS AND HARBOR - DAY\n\nFrancesca and Gardner are greeted by her twin sister and\ntheir young friends, who squeal and embrace at the good news.\nSomeone throws someone in the pool, and life is good.\n\nMED. CLOSE\n\nFrancesca kisses her Aunt Kay.\n\n\t\t\t\tFRANCESCA\n\t\tUncle Michael is the greatest man\n\t\tever!\n\nVIEW on Kay - happy for her niece.\n\nINT. TAHOE BOATHOUSE - DAY\n\nMichael sits in the darkened boathouse.  Tom Hagen paces.\nMichael is looking at photographs.  Neri stands over him.\n\nCLOSE ON MICHAEL\n\nstudying the pictures.\n\n\t\t\t\tNERI (O.S.)\n\t\tHis name is Fred Vincent.  He owns\n\t\ta small pizza parlor in Buffalo...\n\nCLOSE ON THE PICTURES\n\nSnapshots of a middle-aged man, handsome, Italian.  There is\nsomething familiar about him.\n\n\t\t\t\tNERI (O.S.)\n\t\t\t(continuing)\n\t\t...American wife and two small kids.\n\t\tWe traced him and found that he's\n\t\tin the country illegally, from\n\t\tSicily...\n\nMichael looks at another picture.  The same man.  Only\nyounger, and dressed in Sicilian shepherd's clothing.  We\nremember him as FABRIZZIO...Michael's traitorous bodyguard\nin Sicily.\n\n\t\t\t\tNERI (O.S.)\n\t\t...came over around 1956.  Sponsored\n\t\tby the Barzini Family.\n\nMichael puts the pictures down.\n\n\t\t\t\tMICHAEL\n\t\tIt's him.  Fabrizzio.\n\t\t\t(almost to himself)\n\t\tRevenge is a dish that tastes best\n\t\twhen it's cold.\n\n\t\t\t\tNERI\n\t\tHow do you want me to handle it?\n\nMichael glances at Hagen, who has been waiting in the room.\n\n\t\t\t\tMICHAEL\n\t\tLater.  Tom?\n\nHagen brings him a folder; then, as Michael glances through\nit:\n\n\t\t\t\tHAGEN\n\t\tI've cleared it through the\n\t\tSenator's chief aide, a man named\n\t\tTurnbull.  Turnbull's a heavy\n\t\tgambler, and into us for over a\n\t\thundred grand, so I figure his\n\t\tinformation is reliable.\n\nNeri moves to the bar, to prepare Michael a drink.\n\n\t\t\t\tHAGEN\n\t\tThe Senator can be set up; but he\n\t\tthinks of himself as a clean\n\t\tpolitician.  So it's got to be on\n\t\tterms he can live with: campaign\n\t\tcontribution, donation to a\n\t\tcharitable cause that he controls,\n\t\tthings like that.  If he gets even\n\t\tthe inkling that you think you're\n\t\tbuying him, he'll freeze up.\n\t\tNevada's a funny state, they like\n\t\tthings both ways here... All right.\n\t\tTurnbull says the Senator will be\n\t\there at two-thirty, and he's been\n\t\tprimed.  He knows you'll want to\n\t\tmeet with him alone, and he knows\n\t\tit's about the Tropicana's license.\n\t\tAt any rate, he expects to be\n\t\tintroduced around to some of the\n\t\tinfluential people here today, and\n\t\tgenerally treated as an ordinary\n\t\tguest.  Just go light on him,\n\t\tMikey, sometimes the biggest crooks\n\t\tdon't like to think of themselves\n\t\tas crooks...\n\nMichael glances at Hagen, as though that last remark was\nunnecessary.\n\n\t\t\t\tHAGEN\n\t\tI'm sorry; of course, you know that.\n\n\t\t\t\tMICHAEL\n\t\tTwo-thirty.  That gives me time to\n\t\tsee my boy.\n\n\t\t\t\tHAGEN\n\t\tConnie's outside.\n\nMichael doesn't want to see her.\n\n\t\t\t\tHAGEN\n\t\tI promised; she said it was urgent.\n\nMichael nods.\n\n\t\t\t\tMICHAEL\n\t\tAll right.  Apologize to Pentangeli.\n\nNeri opens the door; Hagen exits, and Connie steps in\nimpatiently, followed by Merle.\n\n\t\t\t\tMICHAEL\n\t\tI said I would see my sister, alone.\n\n\t\t\t\tMERLE\n\t\tI think this concerns me too.\n\t\t\t(taking a cigarette\n\t\t\tfrom the dispenser)\n\t\tYou don't, do you?\n\nConnie steps forward, kisses Michael on the cheek.\n\n\t\t\t\tCONNIE\n\t\tHow are you, honey?  You've met\n\t\tMerle, haven't you.  He was with me\n\t\tin Vegas.\n\n\t\t\t\tMICHAEL\n\t\tI saw him with you.\n\n\t\t\t\tCONNIE\n\t\tWe're going to Europe next week.  I\n\t\twant to get passage booked on the\n\t\tQueen.\n\n\t\t\t\tMICHAEL\n\t\tWhy do you come to me?  Why don't\n\t\tyou go to a travel agent?\n\n\t\t\t\tMERLE\n\t\tWe're going to get married first.\n\nMichael is silent.  Then he rises, and moves to the window\noverlooking the lake.\n\n\t\t\t\tMICHAEL\n\t\tThe ink on your divorce isn't dry.\n\t\tYour children see you on weekends;\n\t\tyour oldest boy, Michael Francis...\n\t\twas in some trouble with the Reno\n\t\tpolice over some petty theft that\n\t\tyou don't even know about.\n\n\t\t\t\tCONNIE\n\t\tMichael...\n\n\t\t\t\tMICHAEL\n\t\tYou fly around the world with lazy\n\t\tyoung men who don't have any love\n\t\tfor you, and use you like a whore.\n\n\t\t\t\tCONNIE\n\t\tYou're not my father!\n\n\t\t\t\tMICHAEL\n\t\tThen why do you come to me?\n\n\t\t\t\tCONNIE\n\t\tBecause I need MONEY!\n\n\t\t\t\tMICHAEL\n\t\t\t(softly)\n\t\tConnie, I want to be reasonable\n\t\twith you.  You have a house here,\n\t\twith us.  You can live here with\n\t\tyour kids...and you won't be\n\t\tdeprived of anything.  I don't know\n\t\tmuch about Merle; I don't know what\n\t\the does for a living; what he lives\n\t\ton.  Why don't you tell him marriage\n\t\tis really out of the question; and\n\t\tthat you can't see him any more.\n\t\tHe'll understand.  But if you\n\t\tdisobey me, and marry this pimp...it\n\t\twould disappoint me.\n\n\t\t\t\tCONNIE\n\t\tIt was my father's money; and I'm\n\t\tentitled to what I need.  Where is\n\t\tTom Hagen?\n\nShe turns angrily, leaving Michael standing face to face\nwith Merle.\n\n\t\t\t\tMICHAEL\n\t\tAre you finished?\n\n\t\t\t\tMERLE\n\t\tI think so.\n\n\t\t\t\tMICHAEL\n\t\tThen out.\n\nMerle puts out his cigarette and leaves, quickly.\n\nEXT. TAHOE PAVILION - FULL VIEW - DAY\n\nThe orchestra has struck up a \"Paul Jones,\" where two\nconcentric circles of young people march in opposite\ndirections, until the music stops.  Then they take whomever\nis opposite them as their new dance partner.\n\nVIEW ON THE HARBOR AREA\n\nFrancesca and her twin, Gardner and their elite young\nfriends roar out of the private harbor, to get up on the\nwater skis.  We notice ROCCO LAMPONE, move along a path\nleading to a separate and more private boathouse.  A small\ncovered craft approaches, ties off, and a group of three men\nstep on to the pathway, shake hands with Lampone - and\nfollow him to the large boathouse where Michael conducts his\nbusiness.\n\nCLOSE VIEW\n\nPentangeli has led Mama up to the dance floor, and is having\nsome difficulty with the orchestra.\n\n\t\t\t\tPENTANGELI\n\t\tI can't believe that out of thirty\n\t\tprofessional musicians, not one of\n\t\tyou is Italian!\n\t\t\t(as the musicians laugh)\n\t\tC'mon, give us a tarantella.\n\nHe waves his hands, conducting, and singing.  The piano\nstarts a vamp, the drums uncertainly join in.  A clarinet\nstarts to play \"Pop Goes the Weasel,\" and soon the rest of\nthe orchestra is playing that.  They look to Pentangeli for\napproval.  Disgusted, he goes back to his table, eating a\nhandful of canapes.\n\nINT. THE BOATHOUSE - DAY\n\nRocco ushers an older Italian, bundled up against the cold\nand wet of his boatride, to Michael.\n\nThe man shows respect to Michael, who quickly indicates that\nNeri should get him a drink.\n\n\t\t\t\tMICHAEL\n\t\tRocco, his friends must be hungry.\n\t\tSee what you can do, but I'd like\n\t\tto keep them away from the guests.\n\nThe older man, JOHNNY 'BLUE BOY' OLA, gestures to his\nbodyguards, and they follow Lampone.\n\n\t\t\t\tMICHAEL\n\t\tYou know my lawyer, Tom Hagen.\n\t\tJohnny Ola.\n\n\t\t\t\tOLA\n\t\tSure, I remember Tom from the old\n\t\tdays.\n\nTom shakes hands with Ola, remembering him, and his\nimportance.\n\n\t\t\t\tMICHAEL\n\t\tTom isn't going to sit in with us,\n\t\tJohnny.  He only handles specific\n\t\tareas of the family business.  Tom?\n\n\t\t\t\tHAGEN\n\t\tSure, Mikey.\n\nHe gathers up some of his papers, as the three men remain\nsilent, waiting for him to go before they talk.  It's clear\nTom doesn't want to be excluded.\n\n\t\t\t\tHAGEN\n\t\tIf you need anything, just...\n\n\t\t\t\tMICHAEL\n\t\tJust tell Rocco I'm waiting.\n\nHagen nods and leaves.  As soon as the door closes:\n\n\t\t\t\tOLA\n\t\tI just left our friend in Miami.\n\n\t\t\t\tMICHAEL\n\t\tHow is his health?\n\n\t\t\t\tOLA\n\t\tNot good.\n\n\t\t\t\tMICHAEL\n\t\tIs there anything I can do; anything\n\t\tI can send?\n\n\t\t\t\tOLA\n\t\tHe appreciates your concern,\n\t\tMichael, and your respect.\n\nThere's a KNOCK on the door; a moment, and then Rocco\nquietly enters and takes his place without disturbing the\nconversation.\n\n\t\t\t\tOLA\n\t\tThe hotel's registered owners are\n\t\tone Jacob Lawrence, and Sidney\n\t\tBarclay, both Beverly Hills\n\t\tattorneys.  In reality it's split\n\t\tbetween the Old Lakeville Road\n\t\tGroup from Cleveland, and our\n\t\tfriend in Miami.  He takes care of\n\t\tothers outside the country, you\n\t\tknow who I mean.  Meyer Klingman\n\t\truns the store, and does all right,\n\t\tbut I've been instructed to tell\n\t\tyou, that if you move him out, our\n\t\tfriend in Miami will go along with\n\t\tyou.\n\n\t\t\t\tMICHAEL\n\t\tHe's very kind, tell him it's\n\t\tappreciated.  I'm sure it will be\n\t\tprofitable all the way around.\n\n\t\t\t\tOLA\n\t\tHe always makes money for his\n\t\tpartners.  One by one, our old\n\t\tfriends are gone.  Death, natural\n\t\tor not, prison, deported.  Our\n\t\tfriend in Miami is the only one\n\t\tleft, because he always made money\n\t\tfor his partners.\n\nThe door opens suddenly, and standing there in his white\nCommunion suit, is Michael's boy Anthony.  A moment later,\nKay appears, and takes the boy's hand.\n\n\t\t\t\tKAY\n\t\tAnthony, Daddy's busy.\n\n\t\t\t\tMICHAEL\n\t\t\t(rising)\n\t\tThis is my boy, and my wife.  Mr.\n\t\tJohn Ola of Miami.\n\n\t\t\t\tKAY\n\t\tI'm sorry, Michael.  Senator\n\t\tGeary's here, and Mr. and Mrs.\n\t\tBarrett wanted to thank you before\n\t\tthey left.  Won't you join us, Mr.\n\t\tOla?\n\n\t\t\t\tMICHAEL\n\t\tMr. Ola's just leaving, Kay.\n\t\tPlease tell the Senator I won't be\n\t\ta minute.\n\nPause; she stands there a moment.\n\n\t\t\t\tMICHAEL\n\t\t\t(continuing)\n\t\tKay.\n\n\t\t\t\tKAY\n\t\tYes, Michael.\n\nEXT. THE BOATHOUSE - DAY\n\nKay closes the door.  It seems as though Michael has violated\nsome sort of promise to her by having this man here today.\nShe looks up toward the first boathouse.\n\nWHAT SHE SEES:\n\nThe covered launch, and Ola's three bodyguards, eating while\nthey wait.\n\nMED. VIEW\n\nAnthony runs away from her, heading toward the house.\n\n\t\t\t\tKAY\n\t\tAnthony!\n\t\t\t(she runs after him)\n\t\tAnthony, where are you going?\n\nMoodily, the boy stops, turns, and walks back to his table\nof honor without answering her.\n\nEXT. TAHOE TABLES AND PAVILION - VIEW ON THE PAVILION - DAY\n\nThe orchestra has taken its break; now two couples in formal\ndress are performing the Quartet from Rigoletto.\n\nVIEW ON HAGEN\n\nsitting by himself, a little down, having a drink.  He's\nwaiting for Michael to re-summon him.  SANDRA, Sonny's\nwidow, sits opposite him.\n\n\t\t\t\tHAGEN\n\t\tWhere's my wife?\n\n\t\t\t\tSANDRA\n\t\tWith Mama, putting the baby to\n\t\tsleep.  Francesca's very happy.\n\t\tMichael was kind to her.  She\n\t\tidolizes him.\n\t\t\t(pause; she looks at\n\t\t\ta despondent Hagen)\n\t\tThe children are all out in the\n\t\tspeedboat.  I'm going to my house.\n\nSandra gets up, still an attractive woman, and walks alone\nto the back path that leads to her home on the estate.\n\nVIEW ON THE PAVILION\n\nThe returned orchestra strikes a big, show-biz chord,\nintended to command the guests' attention.\n\nThe orchestra LEADER raises his hands for silence, and makes\nan announcement over the P.A. system.\n\n\t\t\t\tMAESTRO\n\t\tLadies and gentlemen, a most\n\t\tdistinguished guest would like to\n\t\tsay a few words: Senator and Mrs.\n\t\tPat Geary of the state of Nevada!\n\nA big hand, as the smiling SENATOR introduces his WIFE by\nholding her arm up to the crowd, and then proceeds alone to\nthe bandstand.\n\nMED. VIEW\n\nMichael stands with Kay and Mrs. Geary.  The Senator's\npresence seems to be a statement of political and social\nstatus.\n\nA little distance away, his beautiful son Anthony sits\nquietly, in an unmistakably morose mood.\n\nINT. TAHOE - SANDRA'S HOUSE - DAY\n\nWe HEAR the applause and whistles echoing in the distance.\nSandra stands in her bedroom, looking at the door.  We SEE a\nphotograph of SONNY, and also one of their wedding.\n\nA moment goes by, and then Tom Hagen enters, closing the\ndoor behind him.\n\nWe begin to HEAR Senator Geary's amplified voice resounding\nover the lake.  Hagen moves to Sandra.  She takes him in her\narms, comforting, holding his head against her full breast.\n\n\t\t\t\tHAGEN\n\t\t\t(quietly)\n\t\tHe doesn't want my help any more.\n\t\tHe doesn't need it.\n\n\t\t\t\tSANDRA\n\t\tWe don't know that's true, he never\n\t\tsaid that.\n\n\t\t\t\tHAGEN\n\t\tI can feel it in the way he talks\n\t\tto me.\n\nHe moves to the dresser; pours himself a drink.\n\n\t\t\t\tHAGEN\n\t\tJust now when Johnny Ola showed up,\n\t\the asked me to leave them alone.\n\t\tOla is Hyman Roth's Sicilian\n\t\tcontact.  I was on the inside of\n\t\tten, twenty meetings with him.  But\n\t\ttoday Mike asked me to leave, like\n\t\tan outsider.\n\n\t\t\t\tSANDRA\n\t\tTalk to him.  Tell him how you feel.\n\n\t\t\t\tHAGEN\n\t\tIt's as though he blames me for the\n\t\tground the family lost when I was\n\t\tConsigliere to Sonny.\n\nSandra pulls Hagen to her, and kisses him passionately on\nthe mouth.\n\n\t\t\t\tHAGEN\n\t\tI love Michael, I want to help him,\n\t\tbe close to him.  I don't want to\n\t\tend up a third string lawyer making\n\t\tproperty settlements for the hotels.\n\nSandra knows he needs her.  Slowly she begins to undress.\n\n\t\t\t\tSANDRA\n\t\tWe have a little time now.\n\nEXT. THE PAVILION - VIEW ON SENATOR GEARY - DAY\n\n\t\t\t\tSENATOR GEARY\n\t\t...my thanks, and the thanks of the\n\t\tyoung people of the State of\n\t\tNevada, for this most impressive\n\t\tendowment...\n\t\t\t(he holds a check in\n\t\t\this hand)\n\t\t...made to the University in the\n\t\tname of Anthony Vito Corleone.\n\t\tThank you, Mr. and Mrs. Michael\n\t\tCorleone.\n\nApplause.  Senator Geary returns the microphone to the\nMaestro who adds:\n\n\t\t\t\tMAESTRO\n\t\tAnd now, the Nevada Boys' Choir\n\t\thave prepared a special thank you\n\t\tfor Mr. Michael Corleone.\n\nHe turns to a small Choir Master, who leads the Boys' Choir\nin a choral arrangement of \"MR. WONDERFUL.\"\n\nVIEW ON SENATOR GEARY\n\nshaking hands with Michael, as Press Photographers snap\npictures, showing the check; showing a special award of\nGratitude from the State; Mrs. Corleone and Mrs. Geary; all\ntogether; Michael and his son; Senator Geary and Michael's\nson; and on and on.  In the midst of this:\n\n\t\t\t\tSENATOR GEARY\n\t\tWhere can we meet alone?\n\nMichael indicates the boathouse a distance away, where Neri\nseems to be waiting for them.  Then Michael leans to Rocco:\n\n\t\t\t\tMICHAEL\n\t\tFind Hagen.\n\nRocco sets off; as more pictures are taken, and the:\n\nBOYS' CHOIR\n\nsings its lovely arrangement of \"Mr. Wonderful.\"\n\nINT. TAHOE BOATHOUSE - MED. VIEW - DAY\n\nMichael, the Senator, Neri and Rocco make a group in the\ndark, large room.\n\n\t\t\t\tMICHAEL\n\t\tIt was very kind of you to come to\n\t\tmy home on this occasion, Senator.\n\t\tMy wife has been very concerned\n\t\twith making a good impression on\n\t\tthe people who are our neighbors,\n\t\tand your appearance here has made\n\t\ther very happy.  If I can ever\n\t\tperform a service for you, you only\n\t\thave to ask.\n\nThe door opens, and Hagen sheepishly makes his way in.\n\n\t\t\t\tMICHAEL\n\t\tMy lawyer, Tom Hagen.  He arranged\n\t\tthis all through your man Turnbull.\n\n\t\t\t\tSENATOR GEARY\n\t\tI thought we would meet alone.\n\n\t\t\t\tMICHAEL\n\t\tI trust these men with my life.\n\t\tThey are my right arms; I cannot\n\t\tinsult them by sending them away.\n\n\t\t\t\tSENATOR GEARY\n\t\t\t(taking out some medication)\n\t\tSome water.\n\nHe addresses that to Neri, who resentfully goes to fetch the\nSenator a glass of water.\n\n\t\t\t\tSENATOR GEARY\n\t\tAlright, Corleone.  I'm going to be\n\t\tvery frank with you.  Maybe more\n\t\tfrank than any man in my position\n\t\thas ever spoken to you before.\n\nMichael nods, indicating that he should do so.\n\n\t\t\t\tSENATOR GEARY\n\t\tThe Corleone family controls two\n\t\tmajor hotels in Vegas; one in Reno.\n\t\tThe licenses were grandfathered in,\n\t\tso you had no difficulties with the\n\t\tGaming Commission.  But I have the\n\t\tidea from sources...\n\t\t\t(takes the water from\n\t\t\tNeri and swallows his pills)\n\t\t...that you're planning to move in\n\t\ton the Tropicana.  In another week\n\t\tor so you'll move Klingman out,\n\t\twhich leaves you with only one\n\t\ttechnicality.  The license, which\n\t\tis now in Klingman's name.\n\n\t\t\t\tMICHAEL\n\t\tTurnbull is a good man.\n\n\t\t\t\tSENATOR GEARY\n\t\tLet's forget the bullshit, I don't\n\t\twant to stay here any longer than I\n\t\thave to.  You can have the license\n\t\tfor two hundred and fifty thousand\n\t\tin cash, plus a monthly fee equal\n\t\tto five percent of the gross...\n\nMichael is taken aback; he looks at Hagen.\n\n\t\t\t\tSENATOR GEARY\n\t\t...of all three Corleone hotels.\n\nHagen is frustrated; all his information was wrong.\n\n\t\t\t\tMICHAEL\n\t\tSenator Geary, I speak to you as a\n\t\tbusinessman who has made a large\n\t\tinvestment in your state.  I have\n\t\tmade that state my home; plan to\n\t\traise my children here.  The\n\t\tlicense fee from the Gambling\n\t\tCommission costs one thousand\n\t\tdollars; why would I ever consider\n\t\tpaying more?\n\n\t\t\t\tSENATOR GEARY\n\t\tI'm going to squeeze you, Corleone,\n\t\tbecause I don't like you; I don't\n\t\tlike the kind of man you are.  I\n\t\tdespise your masquerade, and the\n\t\tdishonest way you pose yourself and\n\t\tyour fucking family.\n\nVIEW ON HAGEN\n\nglances at Michael.\n\nVIEW ON MICHAEL\n\nmakes no outward reaction.\n\n\t\t\t\tMICHAEL\n\t\t\t(quietly)\n\t\tWe're all part of the same\n\t\thypocrisy, Senator.  But never\n\t\tthink it applies to my family.\n\n\t\t\t\tSENATOR GEARY\n\t\tAll right, then let me say you'll\n\t\tpay me because it's in your\n\t\tinterests to pay me.\n\nVIEW ON GEARY\n\nrising.\n\n\t\t\t\tSENATOR GEARY\n\t\tI'll expect your answer, with\n\t\tpayment, by tomorrow morning.  Only\n\t\tdon't contact me...from now on,\n\t\tdeal only through Turnbull.\n\nHe is almost out the door.\n\n\t\t\t\tMICHAEL\n\t\tSenator...\n\t\t\t(cold and calm)\n\t\t...you can have my answer now if\n\t\tyou'd like.\n\nGeary turns back.\n\n\t\t\t\tMICHAEL\n\t\tMy offer is this.  Nothing...not\n\t\teven the thousand dollars for the\n\t\tGaming Commission, which I'd\n\t\tappreciate if you would put up\n\t\tpersonally.\n\nGeary returns Michael's hard look; then laughs and leaves.\nSlowly Michael turns to Hagen.\n\nVIEW ON HAGEN\n\nembarrassed at being so off the mark.\n\n\t\t\t\tMICHAEL\n\t\tIt's all right, Tom, we'll talk\n\t\tlater.  Tell Frankie Pentangeli I'd\n\t\tlike him to have dinner at my\n\t\tfamily table before we do business.\n\nEXT. THE PAVILION - NIGHT\n\nNow the light has faltered, and the young waiters have put\nup the night lights.  The tables are all properly set for\ndinner, with candles on each one.\n\nThe orchestra is playing quiet, unobtrusive dinner music,\nand many of the guests have begun to help themselves to the\nimpressive buffet, under a party tent.\n\nMED. VIEW\n\nMichael sits at a large table with Kay, his son Anthony,\nMama, Hagen and TERESA, Connie and Merle' Fredo and Deanna,\nand Frankie Pentangeli.\n\n\t\t\t\tMAMA\n\t\tCent' Anne.\n\nThis, the table of honor, all raise their glasses and repeat\nthe toast.\n\n\t\t\t\tDEANNA\n\t\tWhat's 'cent' Anne?'\n\n\t\t\t\tFREDO\n\t\tA hundred years...it's a toast.\n\n\t\t\t\tCONNIE\n\t\tIt means we should all live happily\n\t\tfor one hundred years.  The family.\n\t\tIf my Father were alive, it'd be\n\t\ttrue.\n\n\t\t\t\tMAMA\n\t\tConnie.\n\n\t\t\t\tCONNIE\n\t\tMerle, have you met my sister-in-\n\t\tlaw Deanna?\n\n\t\t\t\tDEANNA\n\t\tWhat a pleasure, Merle.\n\t\t\t(shaking hands)\n\n\n\t\t\t\tMAMA\n\t\t\t(Sicilian)\n\t\tThose two are perfect for each other.\n\n\t\t\t\tMERLE\n\t\tWhat's that mean?\n\n\t\t\t\tCONNIE\n\t\tMama!\n\n\t\t\t\tPENTANGELI\n\t\t\t(Sicilian)\n\t\tMichael, in all respect, I didn't\n\t\tcome three thousand miles for dinner.\n\n\t\t\t\tMICHAEL\n\t\t\t(Sicilian)\n\t\tI know.\n\n\t\t\t\tPENTANGELI\n\t\t\t(Sicilian)\n\t\tWhen do we talk?\n\n\t\t\t\tMICHAEL\n\t\t\t(Sicilian)\n\t\tAfter dinner.\n\nBy now, the conversation has become exclusively Sicilian,\nwith Merle and Deanna, looking from side to side like in a\ntennis match.  Finally, Kay, to be polite:\n\n\t\t\t\tKAY\n\t\tAnthony, you were talking to Mr.\n\t\tPentangeli?\n\n\t\t\t\tANTHONY\n\t\tHis name is \"Five-Angels.\"\n\n\t\t\t\tPENTANGELI\n\t\tYeah, the kid and me talked Sicilian.\n\t\tA one-way conversation!\n\nINT. TAHOE BOATHOUSE - NIGHT\n\nPentangeli is angry; but because it is Michael he is talking\nto, he keeps his voice low and represses his desire to shout.\n\n\t\t\t\tPENTANGELI\n\t\tSure, Pete Clemenza died of a heart\n\t\tattack, but the Rosato Brothers\n\t\tgave it to him.\n\n\t\t\t\tMICHAEL\n\t\tWe were all heartbroken at the\n\t\tnews; but that wasn't cause to\n\t\tstart a war.\n\n\t\t\t\tPENTANGELI\n\t\tOkay, now it's my family in\n\t\tBrooklyn; and I wanna keep up\n\t\tClemenza's loyalty to you.  But how\n\t\tcan I run my family with you\n\t\tchallenging my every move?  You're\n\t\ttoo far from the street, Mike, the\n\t\tonly way to reason with the Rosato\n\t\tBrothers is to whack 'em and whack\n\t\t'em fast.\n\n\t\t\t\tMICHAEL\n\t\tYou were unfair with them.\n\n\t\t\t\tPENTANGELI\n\t\tSays who?\n\n\t\t\t\tMICHAEL\n\t\tClemenza promised Rosato three\n\t\tterritories in the Bronx after he\n\t\tdied, and then you took over and\n\t\twelched.\n\n\t\t\t\tPENTANGELI\n\t\tClemenza promised them nothing, he\n\t\thated the sonsuvbitches.\n\n\t\t\t\tMICHAEL\n\t\tThey feel cheated.\n\n\t\t\t\tPENTANGELI\n\t\tMichael, you're sitting up here in\n\t\tthe Sierra Mountains with champagne\n\t\tcocktails making judgment on the\n\t\tway I run my family.\n\n\t\t\t\tMICHAEL\n\t\t\t(suddenly in Sicilian)\n\t\tYour family still carries the name\n\t\tCorleone, and you will run it like\n\t\ta Corleone!\n\n\t\t\t\tPENTANGELI\n\t\t\t(Sicilian)\n\t\tAnd while I feed my family in New\n\t\tYork, you put the knife in my back\n\t\tin Miami.\n\n\t\t\t\tMICHAEL\n\t\t\t(firm)\n\t\tFrankie, you're a good old man, and\n\t\tyou've been loyal to my Father for\n\t\tyears...so I hope you can explain\n\t\twhat you mean.\n\n\t\t\t\tPENTANGELI\n\t\tThe Rosatos are running crazy;\n\t\ttaking hostages, spitting in my\n\t\tface, because they're backed by the\n\t\tJew in Miami.\n\n\t\t\t\tMICHAEL\n\t\tI know.  That's why I want you to\n\t\tbe fair with them.\n\n\t\t\t\tPENTANGELI\n\t\tHow can you be fair with animals?\n\t\tThey recruit niggers and spicks;\n\t\tthey do violence in their own\n\t\tGrandmother's neighborhoods.  And\n\t\teverything is dope and whores; the\n\t\tgambling is left to last.  Let me\n\t\trun my family without you on my\n\t\tback.  I want them taken care of.\n\n\t\t\t\tMICHAEL\n\t\tNo.  There are things that I have\n\t\tplanned with Hyman Roth.  I don't\n\t\twant them disturbed.\n\n\t\t\t\tPENTANGELI\n\t\tYou give your loyalty to a Jew over\n\t\tyour own blood.\n\n\t\t\t\tMICHAEL\n\t\tFrankie, you know my father\n\t\trespected Roth, did business with\n\t\thim.\n\n\t\t\t\tPENTANGELI\n\t\tHe did business...but he never\n\t\ttrusted him.\n\nPentangeli takes his hat, and leaves.\n\n\t\t\t\tNERI\n\t\tShould he go?\n\n\t\t\t\tMICHAEL\n\t\tThe old man had too much vino\n\t\trosso, or he'd never talk openly\n\t\tthat way.  Let him go back to New\n\t\tYork; I've already made my plans.\n\t\t\t(he checks his watch)\n\t\tIt's late; I've spent so little\n\t\ttime at the party.\n\nEXT. THE LAWNS AND TABLES - FULL VIEW - NIGHT\n\nBy now the sun has fallen and the lawns of the Corleone\nestate are lit by moonlight.  Beautifully dressed couples\ndance as the orchestra plays a foxtrot of the late fifties.\n\nVIEW ON THE DANCE FLOOR\n\nDeanna has been dancing with Fredo; she has gotten drunk and\nit teasing her husband by flirting with other men on the\nfloor.\n\n\t\t\t\tDEANNA\n\t\tI wanta dance...whatsa matter with\n\t\tthat?\n\n\t\t\t\tFREDO\n\t\tDancing is alright; you're falling\n\t\ton the floor.\n\n\t\t\t\tDEANNA\n\t\tWhatsamatter, you don't want me to\n\t\tdance with him 'cause he's a man!\n\n\t\t\t\tFREDO\n\t\tDeanna, I'm going to belt you right\n\t\tin the mouth!\n\n\t\t\t\tDEANNA\n\t\tThese Eye-ties are really crazy\n\t\twhen it comes to their wives.\n\nBy now guests are starting to notice the disturbance;\nMichael is with Kay, and some friends; Rocco catches his eye.\n\n\t\t\t\tDEANNA (O.S.)\n\t\tJesus, never marry a WOP, they\n\t\ttreat their wives like shit.\n\nVIEW on Kay, listening, embarrassed by her flashy sister-in-\nlaw.\n\nVIEW ON FREDO AND DEANNA\n\nRocco passes by Fredo and whispers:\n\n\t\t\t\tROCCO\n\t\tFreddie, Mike says take care of it,\n\t\tor I have to.\n\n\t\t\t\tDEANNA\n\t\tHe's a friend of your brother!\n\nWithout another word, Rocco grabs firm hold of her and\nwhisks her out of the crowd.\n\n\t\t\t\tDEANNA\n\t\t\"Shuffle off to Buffa...\n\t\tShuffle off to Buffa...\n\t\tShuffle off to Buffalooooo...\"\n\nFreddie mops his forehead, and moves to Michael.\n\n\t\t\t\tFREDO\n\t\tHey Mike, what can I say?\n\n\t\t\t\tMICHAEL\n\t\tForget it, just go take care of her.\n\nEXT. THE HARBOR DECK - NIGHT\n\nA large group of Tahoe teenagers join the Corleone youngsters\nsitting around a large fire out by the harbor.  Gardner and\nFrancie, sitting arm in arm.\n\nEXT. TABLE OF HONOR - MED. VIEW - NIGHT\n\nLittle Anthony, in his white suit, sitting alone.\n\nEXT. MAIN GATE - NIGHT\n\nA taxi pulls up, and is signaled over to the gate by a\npoliceman carrying a torch flashlight.\n\nConnie and Merle enter; Merle tips the cop, and the cab\ndrives off.\n\nEXT. DANCE FLOOR AND PAVILION - MOVING TWO SHOT - NIGHT\n\nKay and Michael dancing in the moonlight.\n\n\t\t\t\tMICHAEL\n\t\tHow's the baby?\n\n\t\t\t\tKAY\n\t\tSleeping inside me.\n\n\t\t\t\tMICHAEL\n\t\tDoes it feel like a boy?\n\n\t\t\t\tKAY\n\t\tYes, Michael, it does.\n\n\t\t\t\tMICHAEL\n\t\tI'm sorry about some of the people\n\t\tI had to see today.  It was bad\n\t\ttiming... but it couldn't be helped.\n\n\t\t\t\tKAY\n\t\tIt made me think of what you told\n\t\tme once.  In five years, the\n\t\tCorleone family will be completely\n\t\tlegitimate.  That was seven years\n\t\tago.\n\nHe has no answer for her; except that he loves and values\nher, and holds her tight, as they dance amid their guests,\nall dressed elegantly for the social event of the season.\n\nThe VIEW LOOSENS to show the entire, night-lit party.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nEXT. LAS VEGAS CHAPEL - MED. CLOSE VIEW - NIGHT\n\nA Cadillac limousine waits for some people inside the tacky,\nLas Vegas marriage mill.\n\nINT. THE CHAPEL - NIGHT\n\nSome quiet, informally dressed couples wait in the rear of\nthe chapel; some talking, others sitting nervously.\n\nA single organ plays some standard wedding music.\n\nThe VIEW PANS up to the altar, where Connie and Merle, in\nthe same clothing they wore to the Tahoe party, are being\nmarried by a Justice of the Peace.\n\nThe Justice goes through the bored, simple ceremony, and we\nbegin to HEAR an echo of the waltz Connie danced with her\nfather, when she was married all those years ago in Long\nIsland.\n\nEXT. THE TROPICANA - LAS VEGAS - MED. VIEW - DAY\n\nA dark car pulls up to the glitter of the neon facade.\nAlbert Neri, alone, leaves it to the parking valets, and\nmoves quickly through the automatic doors, into the main\ncasino.  We still hear the CORLEONE WALTZ.\n\nINT. THE TROPICANA - DAY\n\nAlbert Neri enters the room; glances around a moment, and\nthen heads toward the crap table, where a short, middle-aged\nman, KLINGMAN, stands by the pit boss.  Several security\nguards of the casino, are at their posts.\n\n\t\t\t\tNERI\n\t\tAre you Klingman?\n\n\t\t\t\tKLINGMAN\n\t\tWho's asking?\n\n\t\t\t\tNERI\n\t\tWhere can we talk?\n\n\t\t\t\tKLINGMAN\n\t\tRight here.\n\n\t\t\t\tNERI\n\t\tI represent the interests of the\n\t\tCorleone family.  We make the\n\t\tinvitation to you to tie up your\n\t\taffairs and be out of the hotel by\n\t\tMonday morning.\n\n\t\t\t\tKLINGMAN\n\t\tWho do you think you're talking to?\n\n\t\t\t\tNERI\n\t\tYou said you were Klingman.\n\n\t\t\t\tKLINGMAN\n\t\tYou don't come in here, talk to an\n\t\towner in Las Vegas like that.\n\n\t\t\t\tNERI\n\t\tYou missed my point; you are no\n\t\tlonger an owner.\n\n\t\t\t\tKLINGMAN\n\t\tGet out of my hotel.\n\nNeri stands in front of him, smiling.\n\n\t\t\t\tKLINGMAN\n\t\tBoys, get him out of here.\n\nQuickly, Neri moves toward Klingman, and slaps him hard\nseveral times in the face, knocking off his glasses... Red-\nfaced, Klingman gets down on his knees to pick them up once,\nagain.  Glasses on, he looks to his guards.\n\nWHAT HE SEES\n\nThey stand, motionless.\n\nVIEW ON KLINGMAN\n\nHumiliated, Klingman moves across the casino floor, past a\nfew interested gamblers, and his own people.  Neri slowly\nfollows.\n\nINT. SHOWROOM - MED. VIEW - DAY\n\nA typical, Lido de Paris type of show is in rehearsal, as\nKlingman backs into the showroom.\n\nHIS VIEW\n\nNeri keeps coming.\n\nVIEW ON KLINGMAN\n\nrealizes that no one will help him.  He finally capitulates.\n\n\t\t\t\tKLINGMAN\n\t\tAll right!  All right, I'll be out.\n\nNeri keeps moving, then heads past the terrified man, sits\ndown at a table, and looks up at the stage.\n\n\t\t\t\tNERI\n\t\t\t(to the staring performers)\n\t\tKeep it going.\n\nEXT. A STREET IN BUFFALO, NEW YORK - NIGHT\n\nThe neon lights that spell out \"FRED'S PIZZERIA\" go out;\nafter a moment a man in an overcoat steps out, and turns to\nlock the door of his restaurant.  The Corleone Waltz\ncontinues over this.  He turns.\n\nMED. CLOSE VIEW\n\nFRED VINCENT, whom we remember as the Sicilian Fabrizzio.\nHe moves toward his parked car.  Gets in.\n\nMED. LONG VIEW\n\nThe starter turns, and the automobile blows in a great\nexplosion.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nEXT. THE TAHOE ESTATE - NIGHT\n\nThe waltz continues over the VIEW of the empty, but still\nilluminated pavilion.  There is the debris of the great\nparty spread over the grounds, which a silent crew of\nworkmen are at work cleaning up,\n\nMED. VIEW\n\nMichael walks alone, followed by two of the family dogs,\nIrish Setters.\n\nHe walks to the water line, and looks out across the lake.\nHe picks up a stick, and throws it for the dogs; who go\nscampering after it.\n\nWe notice that a respectful distance away, there are\nbodyguards watching every move he makes.\n\nCLOSE VIEW ON MICHAEL\n\nlooking across the lake.  There is much on his mind.  The\nSOUND of the waltz, begins to segue into the echoed music\nand laughter of an old Italian Music Hall from the past.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nINT. NEW YORK THEATRE - 1915 - NIGHT\n\nVITO CORLEONE is a shy young man of 23, holding his hat in\nhis hand, being led down the crowded aisle of this Italian\nVaudeville theatre by an energetic and fulfilled GENCO\nABBANDANDO, his friend in America.  This entire sequence is\nplayed in Sicilian.\n\n\t\t\t\tGENCO\n\t\tCome on, you've got to see her!\n\nVIEW ON THE STAGE\n\nA tattered melodrama is in progress in Neapolitan.  The sets\nare two-dimensional, and flap whenever there's an entrance\nor exit.\n\nThe hero, PEPPINO, is weeping as he sings about how he's\nleft his Mother in Italy, while he is in this new country,\nAmerica.\n\nVIEW ON THE AUDIENCE\n\nAll poor, Italo-Americans.  Genco finds a few seats, and\nleads Vito to them, stepping on a few shoes in the process.\nThey have barely come to their seats, when an excited Genco\nnudges Vito, and points to the stage.  People shout that\nthey should sit down.\n\nVIEW ON THE STAGE\n\nThere is a knock on the door, and a young girl enters,\ndelivering a letter to Peppino in his tenement in America.\nThe girl is pretty; and obviously the object of Genco's\naffection.  The letter brings bad news.  Peppino's Mother is\ndead.  He weeps, and sings the final song, which most of the\naudience knows: SENZA MAMMA.\n\nVIEW ON THE AUDIENCE\n\nGenco is enthralled with the young actress.  The people in\nthe audience are singing along with Peppino.\n\nVIEW ON THE STAGE\n\nThe actress, object of Genco's affection, makes a dramatic\nexit.\n\nVIEW ON THE AUDIENCE\n\nGenco pulls on Vito's jacket, indicating that now that his\nlove is offstage, they should leave.  Vito rises with him,\nand they make their way all the way down the aisle.\n\nINT. BACKSTAGE THEATRE - MOVING VIEW - NIGHT\n\nGenco hurrying down the backstage corridor, hoping to catch\na glimpse of the actress.  He is followed by Vito.  Suddenly,\nGenco stops short.\n\nGENCO'S VIEW\n\nA heavy-set, fierce looking Italian wearing an expensive\nlight-colored suit and a cream colored fedora.  This is\nFANUCCI.  He is discussing a business matter with the\ntheatre IMPRESARIO; a large, strong looking man, who is\nsweating nonetheless.  He doesn't seem to be giving in to\nFanucci.  He holds a locked strongbox.\n\nVIEW ON VITO\n\nwatching.  The two men argue in Italian.\n\nMED. VIEW\n\nThe young ACTRESS crosses into the area, unaware of the\ndifficulties.  The impresario sees her, and frightened,\nmotions that she should keep away.\n\n\t\t\t\tIMPRESARIO\n\t\tCarla!\n\nBut Fanucci grabs her easily by her slender wrist, and with\nlightning speed, produces a knife which he holds against her\ncheek.  The impresario wrings his hands in agony.\n\n\t\t\t\tIMPRESARIO\n\t\t\t(Sicilian)\n\t\tNo...please, not my daughter.\n\nWhereupon he begins to unlock the box which holds the\nreceipts for the night's box-office.\n\nVIEW ON GENCO AND VITO\n\nhiding, watching.  At first, Genco is enraged, as though he\nwould rush up to help his enamorata.\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tThe Black Hand.\n\nThen he backs away.  Vito looks at him shocked and\ndisappointed in this cowardly behavior.  Genco shakes his\nhead, and points, as though to say that where Fanucci is\nconcerned, there is nothing to be done.\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\t\t (whispered)\n\t\tLet's get out of here.\n\nVIEW ON FANUCCI\n\nhas released the girl.  Her father pulls her away from him,\nand slaps her for no reason; then he pays Fanucci.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tBecause you protested, it will cost\n\t\ta hundred more.\n\nEXT. NEW YORK ALLEY - NIGHT\n\nGenco and Vito; Genco leans against the wall, breathlessly,\nas though he's had a near escape.\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tI know what you are thinking,\n\t\tVitone, but you don't understand\n\t\tyet how things are.  Fanucci is of\n\t\tthe Black Hand.  Everyone in the\n\t\tneighborhood pays him, even my\n\t\tfather.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tHe's an Italian?\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tA pig of a Neaponitan.\n\t\t\t(spits)\n\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tWhy?  Why does he bother other\n\t\tItalians?\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tBecause he knows them; he knows\n\t\tthey have no one to protect them.\n\t\tVitone?  What do you think of my\n\t\tangel?\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tBeautiful.\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tBeautiful.\n\n\t\t\t\tVITO\n\t\tFor you, she is beautiful.  For me,\n\t\tthere is only my wife!\n\n\t\t\t\tGENCO\n\t\tI know.  That's why I brought you\n\t\twith me!\n\nGenco embraces his good friend, and they laughingly walk\ndown the alley.\n\nThe stage door opens, and Fanucci exits, a sinister figure\nin white, moving down the alley just in front of them, into\nthe night.\n\nThe two friends hold their breath, until he disappears.\n\nEXT. NEW YORK STREETS - MOVING VIEW - DAY\n\nVito moves through the street, carrying groceries that he is\nto deliver.\n\nIt is cold, and so vendors are huddled around fires they\nhave lit in old cans and drums.\n\nHe turns up an alleyway, and then stops.\n\nVIEW UP THE ALLEY\n\nWith great strength, Fanucci lifts one of them up into the\nair and throws him down hard to the concrete; but another,\nholding onto his back, manages to produce a switchblade\nknife and awkwardly reaching around from behind the moving\nman, slits Fanucci's throat from one side to the other.\n\nFanucci groans like some great hurt animal.  Blood pours\nfrom the deep, smile-like slit in his throat.\n\nHe throws the young man off his back.\n\nVIEW ON VITO\n\nstepping back in the alley.\n\nVIEW ON FANUCCI\n\nHe takes off his white fedora, and runs down the alley\ntoward Vito, catching the flowing blood in his hat.\n\nThe young attackers scurry off in various directions.\n\nINT. ABBANDANDO GROCERY STORE - DAY\n\nA tiny shop featuring imported food: trays of cured meats,\nprosciutto, copagole, mortadella lies on the counter covered\nwith netting to keep away the thousands of flies.\n\nOlive oil is sold in bulk, as well as wine, cheese and bacala.\n\nGenco works here for his father, and is busy slicing paper\nthin prosciutto for a customer, by hand.  Vito works in the\nback as a stock clerk.\n\nFinished with his customer, Genco moves to his friend.\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tI bet you can't guess what happened?\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tWhat?\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tSome guys from Ninth Avenue jumped\n\t\tFanucci today; slit his throat from\n\t\tear to ear.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tNo, I didn't know.  Is he dead?\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tNah.  Those guys aren't murderers.\n\t\tThey wanted to scare him, that's\n\t\tall.  Make him look bad.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tIn Sicily, when you attack a man,\n\t\tyou had better finish him.\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tI wish they had.  He takes fifty\n\t\tdollars a week from my father's\n\t\tcash drawer.  But you can't kill a\n\t\tman like Fanucci.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tWhy?\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tBecause he's what we say...\n\t\t\"connected\"... You wait, see what\n\t\thappens to those guys from Ninth\n\t\tAvenue.\n\nA customer enters; and Genco moves away to serve him.\n\nCLOSE VIEW ON VITO\n\nrecalling what he had seen and thought.\n\nEXT. NEW YORK ALLEYWAY - MED. VIEW - NIGHT\n\nA young man, one of those who had tried to kill Fanucci,\nruns down an alleyway, breathlessly.  Then he stops, and\nlooks behind himself.  Whoever was following him is gone.\nHe turns and walks ahead.  Then the mammoth, white-suited\nfigure of Fanucci leaps down before him from the fire-escape.\nHe grins at the young man, and then raises his neck, showing\nthe gruesome wound that marks his throat.\n\nHe takes out his pistol and fires point-blank at his attacker.\n\nINT. TINY TENEMENT - FULL VIEW - NIGHT\n\nThe very small, railroad type flat where Vito lives with his\nnew family.\n\nIt is late at night, and he is exhausted.\n\nHe returns home; where his young wife, CARMELLA, goes\nthrough the silent ritual of preparing a simple meal for him.\nHe sits and eats quietly.\n\nINT. TENEMENT ROOM - NIGHT\n\nVito and Carmella enter the darkened bedroom, and approach a\nmetal crib.  Vito reaches down and takes the small hand of\nthe baby between his thick peasant fingers.  Carmella waits\na respectful distance behind him.\n\nINT. ABBANDANDO GROCERY - DAY\n\nThe shop bell RINGS; SINGER ABBANDANDO turns to see a\nsmiling Fanucci tipping his hat, like an old customer.\n\n\t\t\t\tFANUCCI\n\t\tBuon giorno.\n\nImmediately, Vito turns back to his work, and Signor\nAbbandando moves to Fanucci with a sigh.\n\nVito notices the two men talking quietly at one side of the\nstore, while he goes about his work.  Genco works his way\ncloser to his friend.\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tWhat did I tell you.  The one who\n\t\tcut him was found in an alley.  And\n\t\tthe family of the others paid\n\t\tFanucci all their savings to make\n\t\thim forswear his vengeance.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\t\t (surprised)\n\t\tAnd he agreed?\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tHe took the money.  Now he wants\n\t\tdouble from everybody in the\n\t\tneighborhood, including Papa.\n\nVito watches the heated, but inevitable transaction.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\t\t (almost to himself)\n\t\tA real mafioso doesn't sell his\n\t\tvengeance.\n\nMED. VIEW\n\nSignor Abbandando seems to be arguing with Fanucci, and\nevery so often they turn and relate to where Vito is working.\nThen Fanucci leaves, the little bell RINGING; and Signor\nAbbandando reluctantly moves to Vito.\n\n\t\t\t\tSIG. ABBANDANDO\n\t\t\t(Sicilian)\n\t\tVitone.  How is your son?\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tWe are all well.\n\nIt is clear that he has something difficult to tell the\nyoung man.\n\n\t\t\t\tSIG. ABBANDANDO\n\t\tVitone...I...Fanucci has a nephew.\n\nVito looks at him a while, as the old man struggles to tell\nhim.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tAnd you must give him my job.\n\nThe old man nods, regretfully.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tYou have been kind to me since I\n\t\twas a boy; taken care of me, and\n\t\tbeen as a father.  I will always be\n\t\tgrateful to you.  Thank you.\n\nVito takes off his apron, and leaves, passing the youth who\nloiters by the counter.\n\nEXT. THE STREET - DAY\n\nmaking his way from the store.\n\n\t\t\t\tSIG. ABBANDANDO\n\t\t\t(Sicilian o.s.)\n\t\tVitone!\n\nHe turns, and Abbandando has followed him out of the shop,\nholding a basket of some groceries.\n\n\t\t\t\tSIG. ABBANDANDO\n\t\tHere...for your family.\n\n\t\t\t\tVITO\n\t\tNo...please understand...I cannot\n\t\taccept.\n\nINT. VITO'S TENEMENT - MED. VIEW - NIGHT\n\nVito and his wife sit quietly at the table; the two are\nquiet and sad.\n\nSuddenly, we HEAR a noise, and Vito is astonished to see a\nyoung man, PETER CLEMENZA, leaning out of the window on the\nother side of the air shaft which separates their apartments.\n\n\t\t\t\tCLEMENZA\n\t\tHey Paisan!  Hold this for me until\n\t\tI ask for it.  Hurry up!\n\nAutomatically Vito reaches over to the empty space at the\nair shaft, and takes the bundle of rags.  Clemenza's round\nface is strained and urgent, obviously in some kind of\ntrouble.  Suddenly, he closes the window and there is\nactivity that we cannot see in the other apartment.\n\nVito looks to his wife, and then closes the window and\nwindow dressing and takes the bundle into a private part of\nhis kitchen and begins to unwrap it.\n\nWHAT HE SEES:\n\nFive oily guns.  He immediately wraps them again, and\ncarries them to a private closet, and hides it, and returns\nto his wife.  He sits down back at the table; and she knows\nnot to ask him what has happened.\n\nEXT. NEW YORK STREETS - DAY\n\nVito is walking through the crowded streets with a group of\nworkmen; they all wear work clothes, and paper hats on their\nheads.\n\nVito looks to his left, and realizes that Clemenza is\nwalking silently with him; by contrast, Clemenza dresses well.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\t\t (casually)\n\t\tDo you have my goods still?\n\nVito nods.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tDid you look inside?\n\nVito, his face impassive, shakes his head 'no.'\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tI'm not interested in things that\n\t\tdon't concern me.\n\nINT. DOWNTOWN ITALIAN SOCIAL CLUB - DAY\n\nVito and Clemenza drinking wine; they've become friends.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tI have a friend who has a fine rug.\n\t\tMaybe your wife would like it.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tWe have no money for a rug.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tNo.  He would give it away.  I know\n\t\thow to repay a consideration.\n\nVito thinks, then nods.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tShe would like it.\n\nINT. HALLWAY WEALTHY APARTMENT BUILDING - DAY\n\nThe two men proceed up the hallway; Vito is impressed with\nthe opulence.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tYour friend lives in a fine building.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tOh yes, the very best.\n\nClemenza knocks on the door as though he is well known here;\nthen rings.  No answer.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tAh, he's not at home.  Oh, well, he\n\t\twouldn't mind.\n\nQuickly and expertly he takes out a tool and pries open the\ndoor.\n\nINT. WEALTHY APARTMENT - FULL VIEW - DAY\n\nVito looks in awe at the luxurious apartment, which features\na fabulous rich red wool rug.\n\nClemenza immediately moves some of the furniture away, and\ndrops to the floor.\n\n\t\t\t\tCLEMENZA\n\t\tA little help.\n\nVito joins him, and the two begin rolling the rug.  We HEAR\na BUZZER RING.  Clemenza immediately drops his side of the\nroll, and moves to the window.  He pulls a gun from his\njacket.\n\nVIEW ON Vito watching.  He moves so he can see out the window.\n\nTHEIR VIEW\n\nA Policeman stands at the exterior door, waiting.  He rings\nthe buzzer again.\n\nVIEW ON CLEMENZA\n\ncocking his gun.  Vito realizes that if the Policeman should\npursue it any further he is a dead man.  The Policeman gives\nup and leaves.\n\nClemenza puts away his gun.\n\nINT. VITO'S TENEMENT HALLWAY - DAY\n\nThe two men run up the steps, laughing, carrying the fine rug.\n\nINT. VITO'S TENEMENT - DAY\n\nThey are on their knees cutting the rug to fit the small\nroom.  Carmella watches, holding the baby SANTINO.\n\nMED. CLOSE ON CLEMENZA\n\nLike a professional, cutting quickly, with the proper tools.\nHe sings as he works.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nEXT. DOWNTOWN WAREHOUSE - NIGHT\n\nClemenza knocks on the steel door of this downtown building.\nVito waits with him, holding some packages; and another\nyouth, TESSIO, tall and thin and deadly waits with them.\n\nThe door is lifted, and they are greeted by a bright,\nmiddle-aged Italian named AUGUSTINO who leads them into a\nmachine shop.\n\nINT. MACHINE SHOP - NIGHT\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tGood, you waited for us.\n\nClemenza looks up on a higher level.\n\nHIS VIEW\n\nThere is a nine year old boy, operating a drill press.\n\nMED. VIEW\n\n\t\t\t\tTESSIO\n\t\t\t(Sicilian)\n\t\tWho is he?\n\n\t\t\t\tAUGUSTINO\n\t\t\t(Sicilian)\n\t\tMy son, Carmine...it's all right.\n\nThe men then quickly open the packages they've brought;\nrevealing gun, including a more sophisticated machine weapon.\n\nAugustino takes them and expertly begins to clean and\nprepare them.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\t\t (to Vito)\n\t\tPaisan Augustino was a gunsmith in\n\t\tthe Italian army.  We do each other\n\t\tfavors.\n\n\t\t\t\tAUGUSTINO\n\t\t\t(Sicilian)\n\t\t\t (while he works)\n\t\tMy boy is studying the flute.  He\n\t\tplays very well.  He helps me at\n\t\tnight so we can buy him a silver\n\t\tflute someday.  Now he has one made\n\t\tof wood.  Carmine...play...play for\n\t\tmy friends.\n\nVIEW ON THE BOY\n\nwide-eyed... he shuts off the press; and takes out a shabby\nwooden flute.  And begins to play a simple and pure melody.\n\nCLOSE ON VITO\n\nlistening.\n\nCLOSE ON AUGUSTINO\n\nproudly smiling, as he prepares the machine gun.\n\nCLOSE ON TESSIO\n\nlistening, smiling.\n\nFULL VIEW\n\nThe men listening, as the boy's father prepares their guns.\n\nEXT. WAREHOUSE AREA - NIGHT\n\nTessio and Clemenza quickly load racks of cheap dresses.\n\nVito sits behind the wheel of the truck.  He seems reserved,\nand we get the impression that he is studying every move his\ntwo friends are making.\n\nINT. TENEMENT STAIRS - DAY\n\nClemenza runs up a flight of stairs with an armful of\ndresses.  He knocks on a door, and a pretty HOUSEWIFE answers.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tLady, I got a bargain on these\n\t\tdresses.  Five dollars each.  You\n\t\tgotta pay at least fifteen, maybe\n\t\ttwenty in a store.  Look at them,\n\t\tfirst class.\n\nHe holds the dresses up and the woman seems interested.  She\nhandles a couple of them and stands aside so Clemenza can\nenter her apartment.\n\n\t\t\t\tWOMAN\n\t\t\t(Sicilian)\n\t\tI don't know which one I like best.\n\nShe holds the dresses against her body, Clemenza approving\nof each one; and then she goes to her purse and takes out\nfive singles and gives them to him.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tYou'd look beautiful in all of\n\t\tthese.  You should buy at least two.\n\n\t\t\t\tWOMAN\n\t\t\t(Sicilian)\n\t\tAre you kidding?  My husband will\n\t\tkill me if he knows I paid five\n\t\tdollars for one dress.\n\nShe holds one up, then another.  She is torn.  Clemenza\nshakes his head and straightens the dress on her body.  His\nhand brushes her arm; she looks at him smiling.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tYou can have two for five.\n\nShe smiles back.\n\nEXT. TENEMENT BUILDING - DAY\n\nClemenza jumps down the stairs, and out to the middle of the\nstreet, where Vito and Tessio are waiting in the car with\nsome of the stock.\n\n\t\t\t\tTESSIO\n\t\t\t(Sicilian)\n\t\tWhat took so long?\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tShe couldn't decide.\n\nTessio and Clemenza each take more armsful of dresses and\ndivide the neighborhood.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tVito, take the rest of the stock\n\t\tover to Dandine's warehouse; he'll\n\t\tmove it to a wholesaler.\n\nThe three part.  Vito drives the truck off.\n\nMOVING VIEW\n\nVito drives the truck through the downtown streets; he turns\na corner and stops for a light.\n\nSuddenly, to his left, he sees the formidable figure of\nFanucci.\n\nHe grabs young Corleone by the shoulder.\n\nCLOSE VIEW ON FANUCCI\n\nfrightening, revealing the large circular scar, now healed.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tAhhh, young fellow.  People tell me\n\t\tyou're rich, you and your two\n\t\tfriends.  Yet, you don't show\n\t\tenough respect to send a few\n\t\tdresses to my home.  You know I\n\t\thave three daughters.\n\nVito says nothing.  Fanucci thumbs through the stock.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tThis is my neighborhood.  You and\n\t\tyour friends have to show me a\n\t\tlittle respect, ah?  This truck you\n\t\thijacked was in my neighborhood.\n\t\tYou should let me wet my beak a\n\t\tlittle.\n\nFanucci takes a few of the dresses.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tI understand each of you cleared\n\t\taround six hundred dollars.  I\n\t\texpect two hundred dollars for my\n\t\tprotection and I'll forget the\n\t\tinsult.  After all, young people\n\t\tdon't know the courtesies due a man\n\t\tlike myself.\n\nVito smiles at him and nods.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tOtherwise the police will come to\n\t\tsee you and your wife and children\n\t\twill be dishonored and destitute.\n\t\tOf course, if my information as to\n\t\tyour gains is incorrect, I'll dip\n\t\tmy beak just a little.  Just a\n\t\tlittle, but no less than one\n\t\thundred dollars, and don't try to\n\t\tdeceive me, eh paisan?\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\t\t (quietly)\n\t\tMy two friends have my share of the\n\t\tmoney.  I'll have to speak to them\n\t\tafter we deliver these to the\n\t\twholesaler.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tYou tell your friends I expect them\n\t\tto let me wet my beak in the same\n\t\tmanner.  Don't be afraid to tell\n\t\tthem.  Clemenza and I know each\n\t\tother well, he understands these\n\t\tthings.  Let yourself be guided by\n\t\thim.  He has more experience in\n\t\tthese matters.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\t\t (shrugging innocently)\n\t\tYou must understand, this is all\n\t\tnew to me...\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tI understand...\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tBut thank you for speaking to me as\n\t\ta Godfather.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\t\t (impressed)\n\t\tYou're a good fellow.\n\nHe takes Vito's hands and clasps them in his own.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tYou have respect.  A fine thing in\n\t\tthe young.  Next time, speak to me\n\t\tfirst, eh?  Perhaps I can help you\n\t\tmake your plans.\n\nFanucci turns with the dresses draped over his arms, waving\nto Vito.\n\nVito throws the truck in gear, and drives off.\n\nCLOSE VIEW ON VITO\n\nWe know that throughout this encounter he has seethed with\nan icy rage.\n\nINT. VITO'S APARTMENT - MED. VIEW - NIGHT\n\nHe wife serves a dinner for her husband and his two friends.\nThey discuss Fanucci as they eat.\n\n\t\t\t\tTESSIO\n\t\t\t(Sicilian)\n\t\tDo you think he'd be satisfied with\n\t\tthe two hundred dollars?  I think\n\t\the would.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tThat scar-faced bastard will find\n\t\tout what we got from the wholesaler.\n\t\tHe won't take a dime less than\n\t\tthree hundred dollars.\n\n\t\t\t\tTESSIO\n\t\t\t(Sicilian)\n\t\tWhat if we don't pay?\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\t\t (gestures, it's hopeless)\n\t\tYou know his friends...real animals.\n\t\tAnd his connections with the police.\n\t\tSure he'd like us to tell him our\n\t\tplans so he can set us up for the\n\t\tcops and earn their gratitude.\n\t\tThen they would owe him a favor;\n\t\tthat's how he operates.  We'll have\n\t\tto pay.  Three hundred, are we\n\t\tagreed?\n\n\t\t\t\tTESSIO\n\t\t\t(Sicilian)\n\t\tWhat can we do?\n\nClemenza doesn't even bother checking for Vito's opinion.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tThey say Fanucci has a license from\n\t\tMaranzalla himself to work this\n\t\tneighborhood.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tIf you like, why not give me fifty\n\t\tdollars each to pay Fanucci.  I\n\t\tguarantee he will accept that\n\t\tamount from me.\n\n\t\t\t\tTESSIO\n\t\t\t(Sicilian)\n\t\tWhen Fanucci says two hundred he\n\t\tmeans two hundred.  You can't talk\n\t\twith him.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tI'll reason with him.  Leave\n\t\teverything in my hands.  I'll\n\t\tsettle this problem to your\n\t\tsatisfaction.\n\nTessio and Clemenza regard him suspiciously.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tI never lie to people I've accepted\n\t\tas my friends.  Speak to Fanucci\n\t\tyourself tomorrow.  Let him ask you\n\t\tfor the money, but don't pay it,\n\t\tand don't in any way quarrel with\n\t\thim.  Tell him you have to get the\n\t\tmoney and will send me as your\n\t\tmessenger.  Let him understand that\n\t\tyou're willing to pay what he asks,\n\t\tdon't bargain.  I'll go to his\n\t\thouse, and quarrel with him.  He\n\t\tlikes me; enjoys explaining how\n\t\tthings are here.  He uses ten\n\t\tsentences when he only needs one,\n\t\tso while he talks, I'll kill him.\n\nClemenza, irritated, makes a large belch, and washes his\nfood down with wine.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\tVitone!\n\t\t\t(to Tessio)\n\t\tOur driver has drunk too much wine.\n\n\t\t\t\tTESSIO\n\t\t\t(Sicilian)\n\t\t\t (laughs at himself)\n\t\tHe's going to kill Fanucci.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\t\t (stern)\n\t\tThen, after that, what?  Joe\n\t\t'Little Knife' Pisani; Willie\n\t\tBufalino, maybe, Mr. Maranzalla\n\t\thimself, c'mon!\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tFanucci is not connected; he is\n\t\talone.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\t\t (sarcastically)\n\t\tWhat?  You read it in the papers?\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tThis man informs to the police;\n\t\tthis man allows his vengeance to be\n\t\tbought off... No, he is alone.\n\n\t\t\t\tTESSIO\n\t\t\t(Sicilian)\n\t\tIf you're wrong...\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tIf I'm wrong, they will kill me.\n\nBoth Clemenza and Tessio are impressed with their young\nfriend; his willingness to risk his life on his perception\nof the situation.\n\nEXT. NEW YORK STREET - MED. VIEW - NIGHT\n\nA ten piece Italian street band plays in front of the church\nto commemorate the first night of the Festa di San Gennaro.\nPeople swarm in crowds, delighted by the music, as out of\nthe church four men carry the statue of San Gennaro down to\nthe street.\n\nMOVING VIEW\n\nClemenza moves along the booths that have been set up along\nthe street: sausage cooking on an open fire; pork livers and\nsweetbeards.  He stops for a sandwich, and makes an irritated\ngesture when the vendor expects to be paid.  He crosses to a\nchurch-sponsored booth with a great Wheel of Fortune, and\nslaps a dollar on a number.  Standing next to him is Vito;\nthey embrace.\n\n\t\t\t\tCLEMENZA\n\t\t\t(Sicilian)\n\t\t\t (quietly)\n\t\tAll three daughters are at church;\n\t\the is alone.  It's important that\n\t\tyou let his neighbors see you leave.\n\t\tTessio has broken the latch on the\n\t\tskylight of his building.\n\nThe wheel stops; they both lose.\n\n\t\t\t\tCLEMENZA\n\t\t\t(English)\n\t\tSee, Brother Carmello, even the\n\t\tchurch makes numbers.\n\n\t\t\t\tPRIEST\n\t\t\t(English)\n\t\tIt's only the way we collect that's\n\t\tdifferent.\n\nVito has left while Clemenza jokes with the Priest.\n\nEXT. FESTA STREET - NIGHT\n\nVito passes the booths of food, crossing toward a small and\ndark club.\n\nINT. SOCIAL CLUB - NIGHT\n\nWe can still HEAR the crowds and music of the festa.  Vito\nenters; the club is empty, except for the large white figure\nsitting alone at a small table.  Fanucci barely acknowledges\nVito as he joins him.\n\nWithout a word, Vito counts out two hundred dollars on the\ntable.  Fanucci looks, then takes off his fedora and puts it\non the table over the money.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tI think there's only two hundred\n\t\tdollars under my hat.\n\t\t\t(he peeks)\n\t\tI'm right.  Only two hundred dollars.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tI'm a little short.  I've been out\n\t\tof work.  Let me owe you the money\n\t\tfor a few weeks.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tAh, you're a sharp young fellow.\n\t\tHow is it I've never noticed you\n\t\tbefore\n\t\t\t(he takes the two\n\t\t\thundred and pours\n\t\t\tsome wine for Vito)\n\t\tYou're too quiet for your own\n\t\tinterest.  I could find some work\n\t\tfor you to do that would be very\n\t\tprofitable.\n\t\t\t(he rises)\n\t\tNo hard feelings, eh?  If I can\n\t\tever do you a service let me know.\n\t\tYou've done a good job for yourself\n\t\ttonight.\n\nEXT. FESTA STREET - MED. VIEW - NIGHT\n\nBy now the musicians have left, but still families are\nwalking the street, and stopping at the booths.\n\nFanucci stands there a moment; he is known by everyone, and\nconsiders himself highly loved.\n\nThen Fanucci begins the walk through the festa, on his way\nhome.\n\nEXT. ROOFTOPS - MOVING VIEW - NIGHT\n\nVito silently moves along the rooftop; paralleling Fanucci's\nwalk.\n\nWe HEAR the sounds of the festa, and every so often catch a\nglimpse of the patterned lights, and the crowds below.\n\nEXT. FESTA STREETS - MOVING VIEW ON FANUCCI - NIGHT\n\nwalking through the crowded streets.  The statue of San\nGennaro is arranged in some midnight religious ceremony.\n\nThe VIEW LIFTS UP, to the rooftops.\n\nEXT. THE ROOFTOPS - NIGHT\n\nVito makes the leap that separates two buildings; then\ncrosses toward the large skylight in the center of the\nbuilding.\n\nEXT. THE STREETS - NIGHT\n\nThe procession in the streets is preceded by ten altar boys;\nand the glittering Monstrance, something of an altar carried\nout into the streets.\n\nThe priest begins this nocturnal service, as the crowds in\nthe street kneel down in prayer.\n\nINT. FANUCCI'S BUILDING - NIGHT\n\nFanucci unlocks the door to his building; we can HEAR the\nservices in the background.\n\nEXT. THE ROOFTOP - NIGHT\n\nVito tries the trap door on the roof; it is stuck firmly\nshut; despite Clemenza's instructions.  He struggles with\nit, but no luck.\n\nFrom the distance, the Choir begins to Latin.  Vito moves\naround the skylight, to an identical trap, tries this one;\nit opens.\n\nEXT. THE MONSTRANCE - MED. VIEW ON THE PRIEST - NIGHT\n\nperforming the services in Latin.  The ten altar boys are in\nattendance.\n\nEXT. THE ROOFTOP - NIGHT\n\nVito reaches down into the trap, and pulls out the newly\noiled gun that has been left for him.  He slides down into\nthe building.\n\nINT. FANUCCI'S HALLWAY - DOWN ANGLE - NIGHT\n\nFanucci proceeds up the staircase with loud, heavy steps.\nAn OLD WOMAN on one of the flights sees him, and immediately\nmoves to her apartment.\n\n\t\t\t\tFANUCCI\n\t\t\t(Sicilian)\n\t\tWhat's the matter, Signora?  You\n\t\tdon't say 'good evening'?\n\n\t\t\t\tWOMAN\n\t\t\t(Sicilian)\n\t\t'Good evening,' Signor Fanucci.\n\nShe quickly disappears behind her door.  Fanucci laughs,\ncontinues up, singing to himself.  The MASS outside is\nalways in evidence.\n\nINT. HALLWAY - NIGHT\n\nVito climbs down from the attic, and finds Fanucci's rear\ndoor open.  He slips in, and makes his way past the open\nwindows, out of which pour the music and chanting of the\nMass.  Slowly and quietly he pulls them down, shut.\n\nHe moves quietly to a glass door, and peeks out.\n\nWHAT HE SEES:\n\nThree young women, Fanucci's DAUGHTERS, laughing and talking.\n\nVIEW ON VITO\n\nA slip up.  Tessio had said they were out.  He steps outside\nto the alley where he can look into the apartment.\n\nANOTHER VIEW\n\nFanucci opens the door of his apartment, and enters.\n\nCLOSE VIEW ON VITO\n\nHe begins to wrap the gun.\n\nVIEW ON THE DAUGHTERS\n\nTheir father greets them with a kiss; and a little religious\ngift he has bought for each.\n\nCLOSE ON THE GUN\n\nwrapped in this primitive method of a silencer.  The VIEW\nTILTS to Vito, caught in the dilemma of having to kill all\nor none of them.  Then something catches his eye.\n\nWHAT HE SEES:\n\nA small gray alley cat is attracted to the young man, comes\nup to him and rubs itself against him.  Vito rubs the\nanimal, speaking softly in Sicilian, then, gaining its\nconfidence, lifts it up and carefully lets it into Fanucci's\napartment.\n\nHe steps back, holding the gun.  We HEAR some Italian\nshouted in the house; a loud sound from the cat, and some of\nthe thumping footsteps of Fanucci.\n\nVIEW ON VITO\n\nholding the wrapped gun, waiting.\n\nWHAT HE SEES:\n\nThe white blob of Fanucci opening the door and cursing in\nItalian as he throws the cat out.\n\nVIEW ON VITO\n\nsqueezing the trigger; the muffled, but still LOUD BLAST\nresounding in the building.\n\nVIEW ON FANUCCI\n\nHe holds onto the door frame, trying to stand erect, trying\nto reach for his gun.  The force of his struggle has torn\nthe buttons off his jacket and made it swing loose.  His gun\nis exposed but so is a spidery vein on the white shirtfront\nof his stomach.  Carefully, as if plunging a needle into\nthis vein, Vito Corleone fires a second bullet.\n\nFanucci falls to his knees, propping the door open, giving a\nterrible groan.  We begin to hear the VOICES of girls inside\nthe apartment.\n\nVito quickly opens his wallet, removes the two hundred,\nquickly fires one last bullet into Fanucci's sweaty cheek.\nNow the towel the gun was wrapped with catches fire,\nliterally on Vito's hand; quickly he throws it to the\nground, stamps it out...and disappears upward.\n\nEXT. THE ROOFTOPS - MED. VIEW - NIGHT\n\nVito moves like a cat along the rooftops; we HEAR the\nconclusion of the Mass down below.\n\nCLOSE ON VITO\n\nPausing; we can SEE the great spectacle of lights and\ncandles on the streets below.\n\nHe empties the gun of bullets and smashes the barrel against\nthe side of the roof ledge.  He reverses it in his hand, and\nbreaks the butt into two separate halves against the chimney.\nHe smashes it again, and the pistol breaks into barrel and\nhandle, two separate pieces.\n\nHe then moves along the rooftop, dropping the two separate\nfragments into various air shafts.\n\nEXT. THE STREET PROCESSION - MOVING VIEW - NIGHT\n\nThe Priest, having completed the ceremony, follows as the\nMonstrance is carried off through the streets, as the Choir\nsings.\n\nEXT. THE ROOFTOPS - MOVING VIEW - NIGHT\n\nVito is a dark figure, moving with agility across the\nrooftops.\n\nINT. FANUCCI'S VESTIBULE - MED. VIEW - NIGHT\n\nThe corpse that was Fanucci, stained with blood.\n\nEXT. PROCESSION - CLOSE VIEW - NIGHT\n\nThe statue of San Gennaro, followed by the altar boys.\n\nEXT. CORLEONE TENEMENT - MED. VIEW - NIGHT\n\nVito's wife; her baby and several friends and neighbors sit\nhappily on the front stoop of their tenement.  Some of the\nmen drink wine poured out of a pitcher; we can still HEAR\nthe music and night sounds of the Festa.\n\nA neighbor is singing a Neapolitan song.\n\nQuietly, without a word, and with only a momentary glance\nfrom his wife, Vito joins the little group; takes a glass of\nwine, and listens to the song.\n\nCLOSE VIEW ON VITO\n\nlistening to the song.  He reaches out and takes the small\nhand of his son.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tSantino, your papa loves you.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nINT. ANTHONY'S TAHOE ROOM - NIGHT\n\nThe room is large, lit from the outside by a bright evening.\nWe can see the outline of many toys on the shelves built\nalong the wall.  We see the dark figure of Michael Corleone\nenter the room and approach the bed where his son Anthony\nlies curled in messy blankets.  Michael quietly arranges his\nsmall hands and feet and covers the little boy.  Suddenly,\nAnthony turns, his eyes open.  He is staring, perfectly\nawake, at his father.\n\n\t\t\t\tMICHAEL\n\t\tCan't you sleep?\n\nNo answer.\n\n\t\t\t\tMICHAEL\n\t\tAre you alright?\n\n\t\t\t\tANTHONY\n\t\tYes.\n\n\t\t\t\tMICHAEL\n\t\tDid you like your party?\n\n\t\t\t\tANTHONY\n\t\tI got lots of presents.\n\n\t\t\t\tMICHAEL\n\t\tDo you like them?\n\n\t\t\t\tANTHONY\n\t\tI didn't know the people who gave\n\t\tthem to me.\n\n\t\t\t\tMICHAEL\n\t\tThey were friends.\n\nHe kisses his boy, and then turns.\n\n\t\t\t\tANTHONY\n\t\tDid you see my present for you?\n\n\t\t\t\tMICHAEL\n\t\tNo, where is it?\n\n\t\t\t\tANTHONY\n\t\tOn your pillow.\n\n\t\t\t\tMICHAEL\n\t\tI'm leaving very early tomorrow,\n\t\tbefore you wake up.\n\n\t\t\t\tANTHONY\n\t\tI know.  How long will you be gone?\n\n\t\t\t\tMICHAEL\n\t\tJust a few days.\n\n\t\t\t\tANTHONY\n\t\tWill you take me?\n\n\t\t\t\tMICHAEL\n\t\tI can't.\n\n\t\t\t\tANTHONY\n\t\tWhy do you have to go?\n\n\t\t\t\tMICHAEL\n\t\tTo do business.\n\n\t\t\t\tANTHONY\n\t\tI can help you.\n\n\t\t\t\tMICHAEL\n\t\tSome day you will.\n\nMichael kisses him again.\n\nINT. MICHAEL-KAY'S BEDROOM - NIGHT\n\nThe room is lit from a small night lamp on Michael's side of\nthe large bed.\n\nKay is huddled in blankets, asleep.  Michael closes the door\nto his room, moves to his side of the bed, and glances down\nto the pillow.\n\nVIEW ON THE PILLOW\n\nis a child's drawing of a long limousine, with a man in a\nhat sitting in the back seat.\n\nAn arrow pointing to him is marked \"DAD.\" Under it, a nine\nyear old's handwriting says: \"Do you like it?  Check YES __\nI liked it or NO __ I didn't like it.\" Michael turns,\nlooking for a pencil, and moves to the dresser, where he\nplaces a check next to \"YES.\"\n\nHe starts to cross back toward his side of the bed, when Kay\nturns, almost in her sleep:\n\n\t\t\t\tKAY\n\t\tMichael?  Why are the drapes open?\n\nHis eyes dart back to the curved, beautifully leaded windows\nof the room.  The DRAPES are opened.  Then, without a\nsecond's hesitation, he leaps to the floor, still holding\nhis son's drawing, as a spray of machine gun bullets sweep\nacross the windows; glass shattering all over the room.\n\nKay screams out; rising, still half-asleep.  Michael crawls\ntoward her, and pulls her down to the floor to him.\n\nThen, for a moment, there is silence, soon filled by the\nshouts of men; as flashes of light sweep by the window, as\nguards with flashlights come running.\n\nMichael holds Kay to him, knowing they have both survived,\nand then gently:\n\n\t\t\t\tMICHAEL\n\t\tGo with the kids.\n\nEXT. THE TAHOE ESTATE - NIGHT\n\nSuddenly, the great floodlights are turned on, bathing lawns\nin an intense blue light.\n\nGroups of ordinarily dressed security men drawn in from all\ndirections; a state of confusion prevails.  There is no sign\nof the attackers.\n\nVIEW BY MICHAEL'S HOUSE\n\nMichael is joined by Rocco Lampone, his gun drawn.\n\n\t\t\t\tROCCO\n\t\tThey're still on the property.\n\t\tMaybe you better stay inside.\n\n\t\t\t\tMICHAEL\n\t\tKeep them alive.\n\nSix men take up posts by Michael's house.\n\n\t\t\t\tROCCO\n\t\tWe'll try.\n\n\t\t\t\tMICHAEL\n\t\tIt's important.\n\nHe returns inside.\n\nEXT. MAIN GATE AND KENNELS - NIGHT\n\nThe character of the summer estate has changed: bright\nfloodlights illuminate the main points of entry: the main\ngate; the waterway; the stone wall that encompasses the\nestate on all sides.\n\nSeveral men with flashlights reinforce the guard at the main\ngate.\n\nFULL VIEW\n\nOff in the distance, we see another group of men with\nflashlights combing the waterline.  We hear indistinguishable\nshouts.\n\nVIEW ON THE KENNELS\n\nThe wire gates are opened, and the trained dogs go out\nyelping into the outer edge of the estate.\n\nROOFTOP\n\nOne of Rocco's men turns the large floodlight scanning\ndarkened forest areas, where men could hide.\n\nMOVING VIEW\n\nMen with flashlights and dogs.  Moving through the dark areas.\n\nLOOSE VIEW\n\nA small Corleone launch, with a bright spotlight slowly\ncruises the boundaries of the estate.  We SEE the silhouette\nof men with guns, quietly waiting and watching.\n\nEXT. MICHAEL'S HOUSE - NIGHT\n\nSome of the bodyguards by the shattered windows of Michael's\nbedroom.\n\nThe curtains are drawn from inside.\n\nINT. MICHAEL'S HOUSE - NIGHT\n\nKay, the children, and some women servants have come down\nfrom the various rooms into the central living area, that\ncan be most easily secured.  The little girl is still\nasleep; they make you think of an immigrant family, with\ntheir blankets and frightened faces, all waiting in a\ncentral room.\n\nMichael goes up to Kay, squeezes her hand, and whispers:\n\n\t\t\t\tMICHAEL\n\t\tIt will be all right.  We were lucky.\n\nShe says nothing; but her face expresses the anger she feels\nover the jeopardy Michael has placed his children in.  She\nholds her young daughter in her arms.\n\nThe door opens, and Rocco enters.  He quickly realizes he is\nholding his gun in plain view in front of the family, and\nputs it away.  Michael moves to him, and they talk a distance\naway from Kay.\n\n\t\t\t\tROCCO\n\t\tYour family all seem to be okay in\n\t\tthe other houses; your Mother's\n\t\tstill sleeping.\n\n\t\t\t\tMICHAEL\n\t\tAnd?\n\n\t\t\t\tROCCO\n\t\tNo sign of them yet; but they're\n\t\tstill on the Estate.\n\nWe HEAR loud shouting from outside.\n\n\t\t\t\tDEANNA (O.S.)\n\t\tGoddamn you!  You're all nuts here,\n\t\tI'm not goin' to calm down...\n\nMICHAEL'S VIEW\n\nThrough the door, that Rocco opens.\n\nDeanna, in her nightgown, has been frightened by the\ngunshots; while Fredo in his bathrobe, tries to get her back\ninto the house.\n\n\t\t\t\tFREDO\n\t\tDeanna, will you get back into the\n\t\thouse!\n\n\t\t\t\tDEANNA\n\t\tI'm getting out of here I said;\n\t\tthese guys all have guns!\n\n\t\t\t\tMICHAEL\n\t\tFredo, can't you shut that woman up!\n\t\t\t(to Rocco's men)\n\t\tGet her in here!\n\nThe bodyguards, gracefully help Fredo bring the hysterical\nDeanna into the safety of the house.\n\n\t\t\t\tDEANNA\n\t\t\t(whimpering)\n\t\tI don't want to stay here...\n\n\t\t\t\tFREDO\n\t\tMike, what can I do, she's a\n\t\thysterical woman...\n\n\t\t\t\tKAY\n\t\tLeave her alone!  You're talking as\n\t\tthough she has no right to be\n\t\tfrightened when there are machine\n\t\tguns going off in her backyard.\n\n\t\t\t\tMICHAEL\n\t\t\t(to Rocco)\n\t\tHave Tom Hagen meet me in the\n\t\tHarbor House.\n\nEXT. TAHOE ESTATE - HIGH ANGLE - NIGHT\n\nMichael walks the short distance from his house, to the\nboathouse where he conducts his business away from his family.\n\nA small group of bodyguards, carrying machine guns, make the\nwalk with him from all sides, a respectful distance away.\nIt gives the appearance of a lonely President moving in his\ncompound, followed by teams of Secret Service men.\n\nThe boathouse is already secured by teams of men, hastily\nwakened from their lodge house; a barracks-like structure\nwhere reinforcements are lodged just for this kind of\nemergency.\n\nFULL VIEW\n\nIn the distance, we can see the teams of men and dogs, with\ntheir lights, guns and shouts, combing every inch of the\nestate.\n\nINT. THE BOATHOUSE - EMPTY VIEW - NIGHT\n\nMichael alone in the great room.  He moves to a walk-in\nsafe, quickly runs through the combination, and opens it.\nHe takes out an envelope, and puts it into his pocket;\nthere's a KNOCK on the door, and Hagen enters.  He had been\nasleep, and has quickly thrown on a robe.\n\n\t\t\t\tMICHAEL\n\t\tSit down, Tom.\n\nEXT. TAHOE BOATHOUSE - NIGHT\n\nFrom outside the leaded windows, a disoriented Hagen sits\ndown; Michael starts to talk to him; obviously about\nsomething very serious.\n\nThe patrol securing the boathouse, walk past the window.\nMichael says something to Tom, who rises, and pulls the\ndrapes, obscuring OUR VIEW.\n\nINT. TAHOE BOATHOUSE - NIGHT\n\nMichael talks intimately to Tom.\n\n\t\t\t\tMICHAEL\n\t\tThere's a lot I can't tell you, Tom.\n\t\tI know that's upset you in the\n\t\tpast; and you've felt that it was\n\t\tbecause of some lack of trust or\n\t\tconfidence.  But it is because I do\n\t\ttrust you that I've kept so much\n\t\tsecret from you.  It's precisely\n\t\tthat at this moment, you are the\n\t\tonly one that I can completely\n\t\ttrust.  In time, you'll understand\n\t\teverything.\n\n\t\t\t\tHAGEN\n\t\t\t(nods with this statement)\n\t\tBut your people... Neri... Rocco;\n\t\tyou don't think...\n\n\t\t\t\tMICHAEL\n\t\tNo, I have confidence in their\n\t\tloyalty... but this is life and\n\t\tdeath, and Tom, you are my brother.\n\nHagen in very moved.\n\n\t\t\t\tHAGEN\n\t\tMikey, I hoped...\n\n\t\t\t\tMICHAEL\n\t\tNo Tom, just listen.  All my people\n\t\tare businessmen; their loyalty is\n\t\tbased on that.  One thing I learned\n\t\tfrom my father is to try to think\n\t\tas the people around you think...and\n\t\ton that basis, anything is possible.\n\t\tFredo has a good heart, but he is\n\t\tweak...and stupid, and stupid\n\t\tpeople are the most dangerous of\n\t\tall.  I've kept you out of things,\n\t\tTom, because I've always known that\n\t\tyour instincts were legitimate, and\n\t\tI wanted you to know very little of\n\t\tthings that would make you an\n\t\taccomplice, for your own protection.\n\t\tI never blamed you for the setbacks\n\t\tthe family took under Sonny; I know\n\t\tyou were in a position of limited\n\t\tpower, and you did your best to\n\t\tadvise and caution him.  What I am\n\t\tsaying is that now, for how long I\n\t\tdo not know, you will be the Don.\n\t\tIf what I think has happened is\n\t\ttrue; I will leave tonight, and\n\t\tabsolutely no one will know how to\n\t\tcontact me.  And even you are not\n\t\tto try to reach me unless it is\n\t\tabsolutely necessary.  I give you\n\t\tcomplete power: over Neri... Fredo,\n\t\teveryone.  I am trusting you with\n\t\tthe lives of my wife and children,\n\t\tand the future of this family,\n\t\tsolely resting on your judgment and\n\t\ttalent.\n\nVIEW ON HAGEN\n\nA man who has steadily declined over the last five years,\nrealizing that total power and responsibility is being\nplaced on him.\n\n\t\t\t\tMICHAEL\n\t\t\t(continuing)\n\t\t...But Tom, you must know that I do\n\t\tthis only because I believe you are\n\t\tthe only one who is capable of\n\t\ttaking over for me.\n\nVIEW ON MICHAEL\n\ntaking out the envelope.\n\n\t\t\t\tMICHAEL\n\t\tI've prepared this; have had it for\n\t\tover a month.  It won't explain\n\t\teverything; but indicates where I\n\t\twill be, so in a sense, it is my\n\t\tlife.\n\t\t\t(he hands the envelope\n\t\t\tto Hagen)\n\t\tAlso, there are three tasks that\n\t\tmust be executed immediately.  Pop\n\t\twould have given those to Luca --\n\t\tYou knew Pop as well as anyone, act\n\t\tas though you were him.  It\n\t\tdiscusses Kay as well; that will be\n\t\tthe most difficult.  The men who\n\t\ttried to kill me tonight, will\n\t\tnever leave the estate.\n\n\t\t\t\tHAGEN\n\t\tWill we...be able to get who\n\t\tordered it out of them?\n\n\t\t\t\tMICHAEL\n\t\tI don't think so.  Unless I'm very\n\t\twrong...they're already dead.\n\t\tKilled by someone inside...very\n\t\tfrightened that they botched it.\n\t\tThat's why I am going to disappear\n\t\tin a few minutes, and leave\n\t\teverything to you.\n\n\t\t\t\tHAGEN\n\t\tBut if you're wrong...\n\n\t\t\t\tMICHAEL\n\t\tIf I'm wrong...\n\nThere is a KNOCK on the door.\n\n\t\t\t\tMICHAEL\n\t\t...I don't think I'm wrong.\n\t\t\t(he indicates the knock)\n\t\tYes.\n\nThe door opens; it is Rocco; Michael rises, after making a\nknowing glance toward Tom, and moves to talk quietly to a\nfrightened and agitated Rocco.\n\nEXT. STONE WALL AND STREAM - MOVING VIEW - NIGHT\n\nA group of men with flashlights and guns lead Michael, Tom\nand Rocco to the stone bridge spanning the stream which runs\nthrough the estate.\n\nLOW CLOSE VIEW\n\nMichael's dispassionate face, looking down.  THE VIEW MOVES\nto Hagen's, and then down to the murky water under the\nbridge, where we see the bodies of three strangers, lying in\nthe moving water; machine-type guns nearby, with their\nthroats cut.  Light from the many flashlights illuminates\nthe grotesque scene.\n\n\t\t\t\tMICHAEL (O.S.)\n\t\tFish them out.\n\nSeveral of the men wade down into the stream; Rocco helps,\nand even Tom steps down to get a better look at who they\nwere.  They are total strangers; Rocco examines the type of\nguns they used.\n\nWhen they climb back onto the ground, Michael is gone.\nEveryone notices it, but no one says anything.\n\nHagen stands there, holding the envelope Michael had given\nhim in his hand.\n\nHe realizes that now, he is the DON.\n\n\t\t\t\tHAGEN\n\t\tGet rid of the bodies.  Tomorrow\n\t\tmorning I want a report made to the\n\t\tlocal police, and paper, that some\n\t\texplosives we keep on the property\n\t\twere accidentally ignited.\n\nThe men respond; Hagen makes the lonely walk back to the\nlighted section of the compound, which now resembles a\nprison camp.\n\n\t\t\t\t\t\t\tFADE OUT.\n\nFADE IN:\n\nINT. STATE SENATE FLOOR - DAY\n\nThe Senate is in session; Senator Geary is on the floor\nduring a vote.  An aide approaches him, with a slip of paper.\n\nINT. GEARY'S OFFICE - DAY\n\nThe Senator steps behind his desk.\n\n\t\t\t\tSENATOR GEARY\n\t\tAll right, Mr. Hagen, you've got\n\t\tten minutes.\n\nHe flicks the switch of a small tape recorder.\n\n\t\t\t\tSENATOR GEARY\n\t\t...and the tape will be running.\n\n\t\t\t\tHAGEN\n\t\tActually, I've come with good news;\n\t\tthe Corleone family has done you a\n\t\tfavor.\n\nThe Senator immediately shuts the tape recorder off.\n\n\t\t\t\tSENATOR GEARY\n\t\tWhat the hell are you talking about?\n\n\t\t\t\tHAGEN\n\t\tWe know you're a busy man, with\n\t\tplenty of enemies -- we saw the\n\t\topportunity to do you a favor, and\n\t\twe did.  No strings.\n\n\t\t\t\tSENATOR GEARY\n\t\tNo strings.\n\n\t\t\t\tHAGEN\n\t\tYou know there's a Senate\n\t\tInvestigating Committee recently\n\t\tset up; we thought it would be\n\t\tunfortunate if they were to trace\n\t\tanything though-provoking to your\n\t\tname.\n\n\t\t\t\tSENATOR GEARY\n\t\tNo one can trace anything to me; I\n\t\tpride myself on that.\n\n\t\t\t\tHAGEN\n\t\tDo you gamble?\n\n\t\t\t\tSENATOR GEARY\n\t\tA little; what's so thought-\n\t\tprovoking about that?\n\n\t\t\t\tHAGEN\n\t\tDo you owe markers?\n\n\t\t\t\tSENATOR GEARY\n\t\tMaybe two, three thousand dollars.\n\nHagen leans forward, and deposits a handful of paper on the\nSenator's desk.\n\n\t\t\t\tHAGEN\n\t\tThe Corleone family has paid them\n\t\toff for you...as an expression of\n\t\tour esteem.\n\nGeary quickly looks through the paid markers.\n\n\t\t\t\tSENATOR GEARY\n\t\tThere's thirty grand worth of paid\n\t\toff markers -- I never owed that\n\t\tmuch.\n\n\t\t\t\tHAGEN\n\t\tOur mistake.  But what does it\n\t\tmatter; it was our money.\n\t\t\t(rising)\n\t\tWe don't even expect thanks.\n\n\t\t\t\tSENATOR GEARY\n\t\tYou paid off thirty grand I never\n\t\towed.\n\n\t\t\t\tHAGEN\n\t\tWe'll keep it quiet; the people who\n\t\tknow are trustworthy...the Committee\n\t\tneedn't find out.\n\n\t\t\t\tSENATOR GEARY\n\t\tAnd what's the price of their not\n\t\tfinding out.\n\n\t\t\t\tHAGEN\n\t\tSimple.  Be friendly like us.  Not\n\t\thostile.\n\n\t\t\t\tSENATOR GEARY\n\t\t\t(he despises Hagen)\n\t\tThanks...friend.\n\nEXT. TAHOE ESTATE - FULL VIEW - DAY\n\nThere are more men on duty than usual; not that there are\nguns apparent, but it's clear that the boundaries are being\npatrolled.\n\nVIEW BY MICHAEL'S HOUSE\n\nKay exits her house, followed by her children; she helps\nthem into her station wagon like any housewife, and drives\nalong the path leading to the main gate.\n\nShe's about to drive through, when one of the men steps in\nfront of her, raising his hand.\n\n\t\t\t\tKAY\n\t\t\t(graciously)\n\t\tYes.\n\n\t\t\t\tMAN\n\t\tI'm sorry, Mrs. Corleone.  We're\n\t\tnot to let you through.\n\n\t\t\t\tKAY\n\t\t\t(disbelieving)\n\t\tI'm going to the market.\n\n\t\t\t\tMAN\n\t\tIf you could just give us a list,\n\t\twe'll pick up anything you want.\n\n\t\t\t\tKAY\n\t\tWhose orders are these?\n\n\t\t\t\tMAN\n\t\tMr. Hagen's, ma'am.\n\nWe notice Hagen walking to them in the background.\n\n\t\t\t\tHAGEN\n\t\tKay.\n\nVIEW THROUGH THE GATE\n\nHagen approaches the car; Kay gets out so they can talk away\nfrom the children.\n\n\t\t\t\tHAGEN\n\t\tI wanted to explain this myself...\n\t\tI had business in Carson City.\n\nHe walks with her a little way from the others; the children\nrun out of the station wagon, and start to play.\n\n\t\t\t\tHAGEN\n\t\tIt's Michael's request...for your\n\t\tsafety.  We can send out for\n\t\tanything you need.\n\n\t\t\t\tKAY\n\t\tI'm supposed to stay in my house.\n\n\t\t\t\tHAGEN\n\t\tWithin the compound will be fine.\n\n\t\t\t\tKAY\n\t\tI was supposed to take the children\n\t\tto New England next week.\n\n\t\t\t\tHAGEN\n\t\tThat's off now.\n\n\t\t\t\tKAY\n\t\tI'm going to see my parents.\n\n\t\t\t\tHAGEN\n\t\tKay, Michael didn't tell me a lot;\n\t\tand what he did tell me, I can't\n\t\trepeat.  But the responsibility for\n\t\tyou and the kids was the most\n\t\timportant thing he left me with.\n\n\t\t\t\tKAY\n\t\tHow long does this go on?\n\n\t\t\t\tHAGEN\n\t\tI don't know.\n\t\t\t(pause)\n\t\tI'm sorry, Kay...\n\n\t\t\t\tKAY\n\t\tAm I a prisoner?\n\n\t\t\t\tHAGEN\n\t\tThat's not the way we look at it.\n\nAngrily, without another word, Kay turns away from him, and\nwalks to her children, ignoring the running station wagon.\n\nEXT. ITALIAN LUXURY LINER - DAY\n\nThe luxury liner making its way across the Atlantic.\n\nINT. ITALIAN LUXURY LINER - MED. VIEW - DAY\n\nThe PURSER followed by several white uniformed associates\nknocks on the door of something designated the \"Leonardo\nSuite.\" He is holding a telegram.\n\nThe door opens, and a tanned Merle peeks out of the door.\n\n\t\t\t\tPURSER\n\t\t\t(holding up the telegram)\n\t\tI'm terribly sorry to disturb you\n\t\tbut we have received two telegrams.\n\n\t\t\t\tMERLE\n\t\t\t(reluctantly)\n\t\tWell...come in.\n\nThis entourage enters the suite, an impressive and\nbeautifully spacious luxury suite.  Connie is relaxing.\n\n\t\t\t\tCONNIE\n\t\tWhat is it?\n\n\t\t\t\tPURSER\n\t\tYes.  One is from our office in New\n\t\tYork.  The check that you wrote for\n\t\tyour passage has been returned.\n\n\t\t\t\tCONNIE\n\t\tCan't be...\n\n\t\t\t\tMERLE\n\t\tWhy don't you wire your bank?\n\n\t\t\t\tPURSER\n\t\tThe other telegram is from your\n\t\tbank.  Your account has been closed\n\t\tand the company is warned not to\n\t\textend any credit.\n\n\t\t\t\tCONNIE\n\t\tI'll take care of it in Naples.\n\n\t\t\t\tPURSER\n\t\tThe company hopes so.  But for now,\n\t\twe have orders to change your\n\t\taccommodations.\n\nAnd with that, the men in white begin to pack Connie and\nMerle's luggage.\n\n\t\t\t\tCONNIE\n\t\tThat son of a bitch!\n\nINT. ITALIAN LUXURY LINER - TINY THIRD CLASS CABIN - NIGHT\n\nConnie and Merle are attempting to sleep in the miniature\ncabin in bunk beds.  The little space is crowded with their\ntrunks and luggage.  Merle can barely hang onto the bunk,\nthe boat pitches so violently below.\n\nEXT. TRAIN TRACKS - MED. VIEW - DAY\n\nA train speeds across the countryside.\n\nINT. THE TRAIN - DAY\n\nInside the corridor, a porter advances, and knocks on the\ndoor of a stateroom.  A voice tells him to enter.  OUR VIEW\nenters with him as he carries a tray of lunch.  From this\nPOV we see Michael Corleone sitting in the compartment.\n\n\t\t\t\tPORTER\n\t\tMr. Paul?\n\n\t\t\t\tMICHAEL\n\t\tYes.\n\n\t\t\t\tPORTER\n\t\tYou ordered lunch?\n\n\t\t\t\tMICHAEL\n\t\tPut it right there.\n\nThe porter does so; as he places the tray down, he catches a\nglimpse of a second person in the compartment with Michael.\n\nHIS VIEW\n\nA very fierce, almost maniacal looking man, BUSSETTA.  He\nnods that the porter should leave.\n\n\t\t\t\tMICHAEL\n\t\tThank you.\n\nThe porter takes his advice and leaves quickly, closing the\ndoor behind him.\n\nVIEW THROUGH THE WINDOW\n\nMichael and his mysterious companion have lunch together on\nthe moving train.\n\nEXT. GULFSTREAM RACE TRACK IN MIAMI - HIGH FULL VIEW - DAY\n\nThe empty parking lot of the Gulfstream track, on an off-\nrace day.\n\nCLOSER VIEW\n\nMichael sits behind the wheel of a nondescript late model\ncar.  Bussetta sits in the rear.\n\nAnother car swings into the lot.  Michael starts his car,\nand pulls out of the lot; the second car following.\n\nNEW VIEW\n\nThis car pulls out and begins to follow them.  Michael\nglances back by adjusting the rear view mirror, and nods to\nBussetta.\n\nMichael's car begins to slow down, allowing the other car to\novertake them.\n\nThe overtaking car hesitates a moment, moving side by side\nwith them.\n\nMichael glances toward the driver.\n\nMICHAEL'S VIEW\n\nWe recognize Johnny Ola, who waves a greeting to Michael,\nand then continues on to lead him.\n\nEXT. SUBURBAN MIAMI NEIGHBORHOOD - DAY\n\nOla's car leads Michael's through a middle-class suburban\narea of $30,000 to $40,000 homes.  There are small channels\nwith sporting and fishing boats parked near the houses.\nOla's car pulls up in front of a very simple, tract-type\nhome.  Michael's car parks nearby.\n\n\t\t\t\tMICHAEL\n\t\t\t(Sicilian)\n\t\t\t (to Bussetta)\n\t\tYou'll wait in the car.\n\nOla has gotten out of his car and walks up the little path\nto the front door.  Michael waits.\n\nOla rings the bell, and after a moment, a rather pretty,\nmiddle-aged WOMAN answers, remaining behind the screen door.\nOla says a few things to her and she disappears, leaving the\ndoor open.\n\nOla comes down the steps, looks at Michael, nodding to him.\nOla then gets into his car and drives off.  Michael walks up\nthe walkway and enters the little house, closing the door\nbehind him.\n\nThis woman, TERRI ROTH, is in the kitchen, looking out at\nMichael.\n\n\t\t\t\tTERRI\n\t\tI'm just going to make lunch.  How\n\t\tabout a tuna fish sandwich?\n\n\t\t\t\tMICHAEL\n\t\tThank you, Mrs. Roth.\n\nShe hurries halfway up the staircase.\n\n\t\t\t\tTERRI\n\t\tHyman...HYMAN, your friend is here.\n\t\t\t(turning to Michael)\n\t\tWhy don't you go right upstairs, Mr.\n\t\tPaul?\n\n\t\t\t\tMICHAEL\n\t\tFine.\n\nHe continues upstairs; she goes into the kitchen.\n\n\t\t\t\tTERRI\n\t\tI'll give a yell when lunch is ready.\n\nMichael continues up to a small den on the second floor; we\ncan HEAR the sound of a baseball game coming over the\ntelevision.\n\nINT. HYMAN ROTH'S DEN - DAY\n\nMichael enters the den: it's very comfortable, but somewhat\nlike a senior citizen's retirement home in Florida.\n\nMICHAEL'S VIEW\n\nThere, sitting before the television is a small man in his\nmiddle sixties, thin, with a wizened face, looking like a\nsmall-time retired Jewish businessman.  This is HYMAN ROTH.\n\n\t\t\t\tROTH\n\t\tSit down, this is almost over.  You\n\t\tfollow the baseball games?\n\n\t\t\t\tMICHAEL\n\t\tNot for a few years.\n\n\t\t\t\tROTH\n\t\tI like sporting events -- I really\n\t\tenjoy watching them in the afternoon.\n\t\tOne of the things I love about this\n\t\tcountry.  I loved baseball ever\n\t\tsince Arnold Rothstein fixed the\n\t\tWorld Series of 1919...I heard you\n\t\thad some trouble.\n\n\t\t\t\tMICHAEL\n\t\tYes.\n\n\t\t\t\tROTH\n\t\tWhat a mistake; people behaving\n\t\tlike that, with guns.\n\t\t\t(he shakes his head)\n\t\tIt was my understanding we left all\n\t\tthat behind.  But, let me tell you,\n\t\tthe important thing is that you're\n\t\tall right.  Good health is the most\n\t\timportant thing; more than success;\n\t\tmore than power; more than money.\n\n\t\t\t\tMICHAEL\n\t\tThe incident of the other night is\n\t\ta nuisance that I can take care of.\n\t\tI came to you because I want\n\t\tnothing to affect our agreement; I\n\t\twanted to clear everything I'm\n\t\tgoing to do with you, just in case.\n\n\t\t\t\tROTH\n\t\tYou're a considerate young man.\n\n\t\t\t\tMICHAEL\n\t\tYou're a great man, Mr. Roth, I\n\t\thave much to learn from you.\n\n\t\t\t\tROTH\n\t\t\t(warmly)\n\t\tHowever I can help you...\n\n\t\t\t\tMICHAEL\n\t\tThe Rosato Brothers have performed\n\t\tservices for you in the past; I\n\t\tunderstand that they are under your\n\t\tprotection.\n\n\t\t\t\tROTH\n\t\t\t(simply)\n\t\tWe do favors for each other...\n\n\t\t\t\tMICHAEL\n\t\tTechnically, they are still under\n\t\tthe Clemenza wing of the Corleone\n\t\tFamily, now run by Frankie\n\t\tPentangeli.  After Clemenza died,\n\t\tthe Rosatos wanted territory of\n\t\ttheir own.  Pentangeli refused, and\n\t\tcame to me, asking for permission\n\t\tto eliminate them.  I, of course,\n\t\tknew of their relationship with\n\t\tyou, and in gratitude for your help\n\t\twith the Tropicana matter, turned\n\t\thim down.  Pentangeli was furious,\n\t\tand paid one hundred and fifty\n\t\tthousand dollars to have me killed.\n\t\tI was lucky and he was stupid.\n\t\tI'll visit him soon.\n\t\t\t(leaning toward the\n\t\t\told man, sincerely)\n\t\tThe important thing is that nothing\n\t\tjeopardize our plans, yours and\n\t\tmine.  This thing of ours, that we\n\t\twill build.\n\nThe old man touches Michael's hand, warmly.\n\n\t\t\t\tROTH\n\t\tNothing is more important.\n\n\t\t\t\tMICHAEL\n\t\t\t(quietly)\n\t\tPentangeli is a dead man; do you\n\t\tobject?\n\n\t\t\t\tROTH\n\t\tIt's always bad for business; but\n\t\tyou have no choice.\n\n\t\t\t\tMICHAEL\n\t\tThen it's done.  I must choose his\n\t\treplacement: it cannot be Rosato.\n\n\t\t\t\tROTH\n\t\tOf course you must keep control of\n\t\tyour family.\n\nHe turns to Michael, turning the volume higher on the\ntelevision, and moving closer to his young partner.\n\n\t\t\t\tROTH\n\t\tMichael, these things are\n\t\tunimportant.  Who should be the\n\t\tmanager of a dime store, Joe or\n\t\tJack?  Unimportant.  You do what\n\t\tyou think is right.  You're a young\n\t\tman, and I'm old and sick.  What we\n\t\tdo together in the next few months\n\t\twill be history, Michael; it has\n\t\tnever been done before.  We will do\n\t\tthis historical thing together, and\n\t\teven your Father could never dream\n\t\tit would be possible.  We are\n\t\tbigger than U.S. Steel, you and\n\t\tme... because in America, anything\n\t\tis possible!\n\t\t\t(pause)\n\t\tBut soon I will be dead, and it\n\t\twill all belong to you.\n\nThere is a KNOCK on the door, and Terri Roth pushes the door\nopen with her hip.\n\n\t\t\t\tTERRI\n\t\tMy goodness, you'll rupture your\n\t\teardrums, Hyman.\n\nShe puts the tray down, and turns down the television.\n\nEXT. ROTH'S HOUSE - MED. VIEW - DAY\n\nThe sinister Bussetta waits patiently in the rear seat of\nthe car, outside Roth's modest house.\n\nEXT. DOWNTOWN NEW YORK - MOVING VIEW - DAY\n\nA black Cadillac moves down the street, slowed by the\nFestivities of the Festa that is in progress: people milling\naround, buying souvenirs at the many stands set up.\n\nSausage and grilled meats are prepared, just as they were\nyears ago.  Electric lights are strung from the street\nlamps, and brightly colored banners pronounce the \"Festa of\nthe Madonna.\"\n\nMOVING CLOSE VIEW\n\nWilly Cicci drives, frustrated that he cannot go any faster.\nNext to him, Frankie Pentangeli sits, catching a few seconds'\nsnooze.\n\nMED. VIEW\n\nThe black car pulls up; another car that had been following\nit parks nearby.\n\nOne of Pentangeli's button men gets out of the car, and\nsteps into a small Italian restaurant; he exits quickly, and\nnods affirmatively toward Pentangeli's Cadillac.\n\nThe group of them step out quickly, men huddled around\nPentangeli, and enter the restaurant.\n\nINT. THE RESTAURANT - DAY\n\nThe restaurant is quite empty, despite the excitement out on\nthe street.\n\nPentangeli immediately sits at a table with a tall, dark,\nsnappily dressed young man, CARMINE ROSATO.\n\nNearby, on the other side of the room is Rosato's brother,\nTONY, seated with a group of their men.\n\nAt another table in the restaurant is a table of Pentangeli's\npeople: they are joined by bodyguards.\n\n\t\t\t\tPENTANGELI\n\t\tRosato, where's your brother?\n\n\t\t\t\tROSATO\n\t\tSitting right behind you.\n\nPentangeli glances behind himself.\n\n\t\t\t\tPENTANGELI\n\t\tHe don't want to talk?\n\n\t\t\t\tROSATO\n\t\tWe worked it all out beforehand.\n\n\t\t\t\tPENTANGELI\n\t\tAre we going to eat or what?\n\n\t\t\t\tROSATO\n\t\tSure, on me.  I got Diner's Club.\n\n\t\t\t\tPENTANGELI\n\t\t\t(sarcastically)\n\t\tForget it; I'm suddenly without an\n\t\tappetite.  You're making big\n\t\ttrouble, Carmine.\n\n\t\t\t\tROSATO\n\t\tYou weren't straight with us,\n\t\tFrankie, what else could we do?\n\n\t\t\t\tPENTANGELI\n\t\tWe could have talked first, saved a\n\t\tlot of running around.\n\n\t\t\t\tROSATO\n\t\tYou wasn't listening, you didn't\n\t\twant to talk.\n\n\t\t\t\tPENTANGELI\n\t\tDon't I look like I'm listening?\n\n\t\t\t\tROSATO\n\t\tWe want Brooklyn one hundred\n\t\tpercent.  No more taxes to you.  We\n\t\twant to be only loosely connected\n\t\twith your family -- sort of a\n\t\tunder-family all of our own.  Then\n\t\twe can act on all internal matters\n\t\twithout talking.  Also we want you\n\t\tto inform Michael Corleone that we\n\t\tcan deal directly with him.\n\n\t\t\t\tPENTANGELI\n\t\tI'm a little hungry, maybe I'll\n\t\torder something.  Joe.\n\t\t\t(one of his men)\n\t\tGet me some bracciole or something.\n\t\tAnd pay cash.\n\t\t\t(to Rosato)\n\t\tAnd in return for these concessions,\n\t\twhat do you do for me?\n\n\t\t\t\tROSATO\n\t\tWe will release the hostages,\n\t\tnumber one.  Number two, we're here\n\t\tfor you to count on when you need\n\t\tus.  We're independent, but we're\n\t\there if you need us.  In general,\n\t\twe'll cooperate with you and your\n\t\tbusinesses, and you in turn will\n\t\tcooperate with us.  Pari persu.\n\n\t\t\t\tPENTANGELI\n\t\tPari Persu; what the fuck is Pari\n\t\tpersu...?\n\n\t\t\t\tROSATO\n\t\tMy lawyer went over this beforehand.\n\n\t\t\t\tPENTANGELI\n\t\tWhat assurances do I have that\n\t\tthere will be no more kidnapping,\n\t\tno more hits?\n\n\t\t\t\tROSATO\n\t\tThe same assurance we got from you.\n\n\t\t\t\tPENTANGELI\n\t\tWhat if I say shove it?\n\n\t\t\t\tROSATO\n\t\tThen Carmine Fucillo and Tony Blue\n\t\tDeRosa will need to be fitted for\n\t\tslabs.\n\n\t\t\t\tPENTANGELI\n\t\tYou want a war?\n\n\t\t\t\tROSATO\n\t\tWe got no choice.\n\n\t\t\t\tPENTANGELI\n\t\tYou know if there's a way I'll go\n\t\tto the commission and the commission\n\t\twill side with me.  That puts me\n\t\tand the other New York families\n\t\tagainst you.\n\n\t\t\t\tROSATO\n\t\tWe got friends in the commission.\n\n\t\t\t\tPENTANGELI\n\t\t\t(getting angry)\n\t\tI'm talking about Italians!\n\n\t\t\t\tROSATO\n\t\tWhat about Michael Corleone?\n\n\t\t\t\tPENTANGELI\n\t\tHe supports me.\n\n\t\t\t\tROSATO\n\t\tMaybe, yes... maybe no.\n\nOne of Pentangeli's men approaches with a plate of Italian\nfood.\n\nPentangeli stands up, angered by this remark of Rosato's; he\npushes the dish of food out of the surprised Bodyguard's\nhands.\n\n\t\t\t\tPENTANGELI\n\t\tYou drove old Pete Clemenza to his\n\t\tgrave, Carmine; you and your\n\t\tbrother.  Turning on him; trouble\n\t\tin his territories -- you and your\n\t\tdemands.  I hold you responsible,\n\t\tjust as though you shot him in the\n\t\thead.  And I ain't gonna let that\n\t\tgo for long!\n\nPentangeli walks out of the restaurant; there's a little\ntension between the bodyguards of the two factions.\n\n\t\t\t\tROSATO (O.S.)\n\t\tHey, Five-Angels...\n\nHe gives him the arm.\n\nFrankie's face turns red, like he wants to have it out here\nand now; but Willy Cicci calms his down, and they all make\ntheir move out.\n\nEXT. THE RESTAURANT - DAY\n\nPentangeli gets into the car.\n\n\t\t\t\tPENTANGELI\n\t\tNobody I hate calls me Five-Angels\n\t\tto my face!\n\nHe slams the door.\n\nEXT. PENTANGELI'S LONG BEACH ESTATE - DAY\n\nPart of the old estate of Don Corleone.  By now, the wall\nhas been torn down, and the other houses sold off.\n\nHis car is parked; Pentangeli steps out, still angry over\nthe confrontation.  As he approaches the house, he notices\nsomething strained about the bodyguards who discreetly guard\nhis house.  No one seems to want to tell him.\n\n\t\t\t\tPENTANGELI\n\t\tWhat's up?\n\nPentangeli glances over to the front door foyer.\n\nPENTANGELI'S VIEW\n\nThe strange and silent Bussetta, the man who now always\ntravels with Michael.\n\nINT. PENTANGELI'S HOUSE - DAY\n\nPentangeli enters; he sees his WIFE, standing oddly in the\nhallway.\n\n\t\t\t\tPENTANGELI\n\t\t\t(Sicilian)\n\t\tWhat's this?\n\n\t\t\t\tWIFE\n\t\tMichael Corleone.\n\n\t\t\t\tPENTANGELI\n\t\tOne Michael Corleone...Dove?\n\n\t\t\t\tWIFE\n\t\t\t(Sicilian)\n\t\tHe's in your study.\n\nHe knows it is very very serious for Michael to be here in\nhis home.\n\nHe automatically moves into his study.\n\nINT. PENTANGELI'S STUDY - DAY\n\nMichael stands quietly in the room.  This was once his\nfather's study, although it is totally redecorated.\nPentangeli starts sweating, and moves toward the young Don,\nand kisses his hand.\n\n\t\t\t\tPENTANGELI\n\t\tDon Corleone, I wish you let me\n\t\tknow you was coming.  We could have\n\t\tprepared something for you.\n\n\t\t\t\tMICHAEL\n\t\tI didn't want you to know I was\n\t\tcoming.  You heard what happened in\n\t\tmy home?\n\n\t\t\t\tPENTANGELI\n\t\tMichael, yes, we was all relieved...\n\n\t\t\t\tMICHAEL\n\t\t\t(furious)\n\t\tIn my home!  In the same room where\n\t\tmy wife was sleeping; where my\n\t\tchildren come in their pajamas, and\n\t\tplay with their toys.\n\nHe's terrified Pentangeli with his anger; then, just as\nsuddenly, he talks quietly, calmly.\n\n\t\t\t\tMICHAEL\n\t\tI want you to help me take my\n\t\trevenge.\n\n\t\t\t\tPENTANGELI\n\t\tMichael, anything.  What is it I\n\t\tcan do for you?\n\n\t\t\t\tMICHAEL\n\t\tI want you to settle these troubles\n\t\twith the Rosato Brothers.\n\n\t\t\t\tPENTANGELI\n\t\tI was just going to contact you,\n\t\tMichael; we just had a 'sit-down' -\n\t\tin fact, I just come from there.\n\n\t\t\t\tMICHAEL\n\t\tI want you to settle on their terms.\n\n\t\t\t\tPENTANGELI\n\t\tMike, I don't understand.  Don't\n\t\task me to do that.\n\n\t\t\t\tMICHAEL\n\t\tTrust me; do as I ask.\n\n\t\t\t\tPENTANGELI\n\t\tIt would be the beginning of the\n\t\tend for my family.  How can I keep\n\t\tall my other territories in like if\n\t\tI let two wise-guys stand up and\n\t\tdemand this and that, and then give\n\t\tit to them?\n\n\t\t\t\tMICHAEL\n\t\tFrankie...do you respect me?  Do I\n\t\thave your loyalty?\n\n\t\t\t\tPENTANGELI\n\t\tAlways... But sometimes I don't\n\t\tunderstand.  I know I'll never have\n\t\tyour kind of brains, in big deals.\n\t\tBut Mike, this is a street thing.\n\t\tAnd Hyman Roth in Miami is behind\n\t\tthe Rosato Brothers.\n\n\t\t\t\tMICHAEL\n\t\tI know.\n\n\t\t\t\tPENTANGELI\n\t\tThen why do you want me to lay down\n\t\tto them?\n\n\t\t\t\tMICHAEL\n\t\t\t(coldly, but convincing)\n\t\tFrankie, Roth tried to have me\n\t\tkilled.  I'm sure it was him, but I\n\t\tdon't know yet why.\n\n\t\t\t\tPENTANGELI\n\t\tJesus Christ, Michael, then let's\n\t\thit 'em now, while we still got the\n\t\tmuscle.\n\n\t\t\t\tMICHAEL\n\t\tThis was my father's old study.\n\t\tWhen I was a kid, we had to be\n\t\tquiet when we played near here.\n\t\tWhen I was older, I learned many\n\t\tthings from him here.  I was happy\n\t\tthat this house never went to\n\t\tstrangers; first Clemenza took it\n\t\tover, and then you.  My father\n\t\ttaught me, in this room, never to\n\t\tact until you know everything\n\t\tthat's behind things.  Never.  If\n\t\tHyman Roth sees that I interceded\n\t\twith you in the Rosato Brothers'\n\t\tfavor, he'll think his relationship\n\t\twith me is still sound.  I'm going\n\t\tsomewhere to meet him tomorrow.  We\n\t\thave friends in some very important\n\t\tbusiness that we're making.  Do\n\t\tthis for me; you make the peace\n\t\twith the Rosato Brothers on their\n\t\tterms.  Let the word out that I\n\t\tforced you; you're not happy wit\n\t\thit, but acquiesced, just because\n\t\tof me.  It will get back to Hyman\n\t\tRoth.  Do this, Frankie.  You can\n\t\ttrust me.\n\n\t\t\t\tPENTANGELI\n\t\tSure, Mike.  I'll go along.\n\n\t\t\t\tMICHAEL\n\t\tGood.\n\nThey embrace; Michael kisses him.  He looks at the young\nDon, thoughtfully.\n\nINT. TROPICANA HOTEL - CLOSE VIEW - DAY\n\nThe money trays are carefully unloaded from the gaming\ntables, and put on a cart with others.\n\nThe cart, preceded and followed by security guards, is then\nwheeled through the casino, into a private, counting room.\n\nINT. COUNTING ROOM - MED. VIEW - DAY\n\nThe guards leave the room; the door is locked after them,\nleaving only Hagen.  Neri and an ACCOUNTANT, a very fat man.\nThe numbered boxes are opened, and cash and checks are\nspread out on the counting table.\n\nThe accountant begins with amazing speed and skill, to count\nand divide the money.\n\n\t\t\t\tNERI\n\t\tFifteen percent skim?\n\n\t\t\t\tHAGEN\n\t\tTwenty-five this time.\n\nThe accountant stops, and looks up to Neri.\n\n\t\t\t\tNERI\n\t\tIt might show.\n\n\t\t\t\tHAGEN\n\t\tMike wants it.\n\nNeri nods, and the accountant continues.  Neri opens a door,\nallowing a sandy-haired man, a COURIER, into the room.  The\ncream is placed into his pouch personally by Neri.\n\n\t\t\t\tNERI\n\t\tWe've never sent this much with one\n\t\tcourier.\n\n\t\t\t\tHAGEN\n\t\t\t(to the courier)\n\t\tYour plans are a little different\n\t\tthis time.  You skip Miami, and go\n\t\tstraight to Geneva.  It's to be\n\t\tdeposited to this number.\n\t\t\t(handing him a small envelope)\n\t\tAnd it's got to be there by Monday\n\t\tmorning, no slip-up.\n\n\t\t\t\tCOURIER\n\t\tI think I was 'picked-up' last trip.\n\t\tThat hour layover I had at Kennedy.\n\t\tI went over and bought a paper...\n\nNeri has finished putting the 'creamed' money into the pouch.\n\n\t\t\t\tNERI\n\t\tThose were our people.\n\n\t\t\t\tCOURIER\n\t\tOkay, just thought you should know.\n\nHe is just about to close and lock the pouch, when Hagen\ngestures that he should wait, and adds more stacks of\ncarefully packaged bills into the pouch.  Then Neri locks\nit, and handcuffs it to the courier's arm, looking\ninquiringly at Hagen.\n\n\t\t\t\tHAGEN\n\t\tLet them count.\n\nThe courier is shown out through a private door, and then\nthe first door is opened.  Two accountants come in with the\nguards, and the trays are opened, and the counting process\nis begun all over again, this time with the State Tally\nsheets.\n\nINT. TROPICANA CORRIDOR - MOVING VIEW - DAY\n\nThe courier continues on his way; followed by Hagen and Neri.\n\n\t\t\t\tNERI\n\t\tWhat's up?\n\n\t\t\t\tHAGEN\n\t\tNo questions.\n\n\t\t\t\tNERI\n\t\tI got to ask questions, Tom,\n\t\tthere's three million dollars cash\n\t\tin that pouch; Mike is gone and I\n\t\thave no word from him.\n\n\t\t\t\tHAGEN\n\t\tAl, as far as you're concerned, I'm\n\t\tthe Don.\n\n\t\t\t\tNERI\n\t\tHow do I know you haven't gone into\n\t\tbusiness for yourself?\n\nThis hurts Tom; but he is a reasonable man, and he knows he\nowes Neri some explanation.\n\n\t\t\t\tHAGEN\n\t\tYou've been through a lot with us\n\t\tso I'm going to give you the truth.\n\t\tMike knows it was someone within\n\t\tthe compound that set him up for\n\t\tthat hit.  So nobody is to know\n\t\twhere he is, not you, not Rocco,\n\t\tnot even his brother Fredo.  Sorry,\n\t\tAl, I know how you feel about\n\t\tMike...but he still remembers Tessio.\n\nEXT. KEY WEST - NIGHT\n\nMichael is led to a desolate, night-lit private dock.  He is\nfollowed by the ever-present Bussetta, and they are helped\nonto a light-weight, luxury cabin cruiser.  The crew cast\noff various ropes, and the boat sets out into the night.\n\n\t\t\t\t\t\t\tFADE OUT.\n\nFADE IN:\n\nEXT. TAHOE ESTATE - DAY\n\nA seaplane lands nicely by the private Corleone harbor;\nHagen disembarks with his inevitable overloaded briefcase.\nHe continues down the ramp, past several Buttonmen, dressed\nin summer casual attire, and who resemble secret service men\nrather than thugs.\n\nHis wife THERESA lies on a blanket on the great lawn, with\nher youngest children, who run to their father for a kiss.\n\n\t\t\t\tTHERESA\n\t\tHungry?\n\n\t\t\t\tHAGEN\n\t\tJust a little.\n\n\t\t\t\tTHERESA\n\t\tI've invited Mama, Sandra and the\n\t\tkids for barbecue.\n\n\t\t\t\tHAGEN\n\t\tWhat about Kay?\n\n\t\t\t\tTHERESA\n\t\tI couldn't find her.  She's been so\n\t\tbroody, sticks to herself.\n\nEXT. TAHOE LAWN BARBECUE - MED. VIEW - DAY\n\nHagen and Sonny's boys are throwing a football around on the\nlawn; the littler kids running after them.\n\nCoals are burning in the old style stone barbecue, and\nseveral tables are set for the family.\n\nIn the distance, there is always evidence of the bodyguards.\n\nTheresa, Mama and Sandra prepare the steaks.\n\nHagen relaxes in a sports shirt.\n\n\t\t\t\tHAGEN\n\t\tLet me try Kay.\n\nHe crosses the lawn, to the house on the beach where Michael\nand his family live.  Is about to knock on the door:\n\n\t\t\t\tHAGEN'S SON\n\t\tHey, Pop, heads up!\n\nThe football is flying in his direction; he catches it and\nthrows it back.  Then he cracks the door open, and peeks in.\n\nINT. MICHAEL'S HOUSE - DAY\n\n\t\t\t\tHAGEN\n\t\tKay?\n\nHe steps in, the beautiful summer living room is neat, but\nempty.\n\n\t\t\t\tHAGEN\n\t\tAnyone hungry?\n\nHe moves through the house more quickly; into the dining and\nrecreation room areas.  A cat jumps off a pile of cushions\nand runs across the room.\n\n\t\t\t\tHAGEN\n\t\tHello?\n\n\t\t\t\tSANDRA (O.S.)\n\t\tShe's gone, Tom.\n\nSandra has followed him into Michael's house.\n\n\t\t\t\tHAGEN\n\t\tWhat do you mean gone?\n\n\t\t\t\tSANDRA\n\t\tThe Barretts from Rubicon Bay came\n\t\tby in a new speedboat.  Rocco tried\n\t\tto say she wasn't in, but Kay\n\t\tspotted them and asked if they\n\t\twould take her and the kids for a\n\t\tride.  That was three hours ago.\n\n\t\t\t\tHAGEN\n\t\t\t(furious)\n\t\tWhy didn't someone tell me!\n\n\t\t\t\tSANDRA\n\t\tI wanted to tell you alone; your\n\t\twife doesn't know what's going on.\n\nHagen rushes out of the house.\n\nEXT. TAHOE LAWN - DAY\n\nHagen moves quickly out of Michael's house; moving across\nthe lawn to the boathouse.\n\n\t\t\t\tHAGEN'S SON\n\t\tHey, Dad!\n\nThis time he ignores the thrown ball, and moves directly to\nRocco, who is by some men near the boathouse.\n\n\t\t\t\tHAGEN\n\t\tRocco!\n\n\t\t\t\tROCCO\n\t\tI know.  I went down to the Barrett\n\t\thouse.  But she's gone.  They drove\n\t\ther and the kids to North Tahoe\n\t\tairport.\n\n\t\t\t\tHAGEN\n\t\tGoddamn it, where were you?\n\n\t\t\t\tROCCO\n\t\tI was in my house.  Willy tried,\n\t\tbut it would have taken some\n\t\tstrong-arm to stop her, and he\n\t\tfigured you wouldn't want that.\n\nINT. THE BOATHOUSE - DAY\n\nThey enter the boathouse.\n\n\t\t\t\tHAGEN\n\t\t\t(to one of the men)\n\t\tGet me a Scotch and water.\n\nThe man hurries behind the bar.\n\n\t\t\t\tROCCO\n\t\tShe took a flight to San Francisco.\n\t\tWe figure she's going to connect to\n\t\tNew Hampshire; her parents' place.\n\n\t\t\t\tHAGEN\n\t\t\t(almost to himself)\n\t\tI can't let him down.\n\nHe swallows the drink down in several gulps.  And then looks\nup to his men watching him.  He's embarrassed to have shown\nsuch weakness.\n\n\t\t\t\tHAGEN\n\t\tAll right, let me think a minute.\n\nRocco clears the men out.\n\n\t\t\t\tROCCO\n\t\tMe too, Tom?\n\n\t\t\t\tHAGEN\n\t\tYeah, give me a minute.\n\nRocco gone, Hagen moves behind the enormous bar, and pours\nhimself a giant drink.  He drinks that, and calms himself.\n\n\t\t\t\tHAGEN\n\t\tOh Christ, Pop.  It was so good\n\t\twhen you were alive.  I felt I\n\t\tcould handle anything...\n\nEXT. VIEW FROM BOAT - FULL VIEW - DAY\n\nA beautiful coastal view of a tropical Caribbean city.  An\nextraordinary view, high buildings, palm trees, all set\nright on the bay.\n\nMED. CLOSE VIEW ON MICHAEL\n\non the cruiser, Bussetta a little distance away, watching,\nbut never speaking.  The dark-skinned CAPTAIN of the cruiser\nkeeps pointing repeatedly.\n\n\t\t\t\tCAPTAIN\n\t\tHabana, Habana.\n\nEXT. HAVANA STREET - MOVING VIEW - DAY\n\nMichael and Bussetta are driven in a Mercury sedan, making\nits way through the streets of Havana.\n\nCLOSE VIEW ON MICHAEL\n\nlooking out the window.\n\nMICHAEL'S POV\n\nCrowded streets, occasional roving bands playing for the\ntourists; there is much evidence of tourism: Americans\nwalking through the streets with cameras.  Occasionally, we\nsee a Cuban with a row of numbers attached to his hat,\ncarrying a big sheet of the daily lottery numbers.  From all\nof these street impressions, the city is booming with\nactivity, but there is also much evidence of whores and\npimps and little children begging in the streets.\n\nMED. VIEW\n\nThe big American car stops at an intersection.  Bussetta is\nsitting in the forward passenger side; while Michael is in\nthe back.  He hears tapping on the window; he turns and sees\nfour Cuban boys tapping on his window and extending their\nhands, and rubbing their stomachs as though they were hungry.\nThe Cuban driver rolls down his window and shouts them away\nin Spanish.\n\nINT. HAVANA CASINO LOBBY - MOVING VIEW - NIGHT\n\nMichael is led through a beautiful wooden lobby of the\nhotel, done in Spanish style, apparently just recently\ncompleted.  He is approached by a thin, mousy man, SAM ROTH,\nwho ushers him toward the casino entrance.\n\n\t\t\t\tSAM ROTH\n\t\tHiya, Mr. Corleone, I'm Sam Roth.\n\t\tWelcome to the Capri; my brother's\n\t\tupstairs.  You wanta take a rest\n\t\tbefore you see him, or can I get\n\t\tyou something, anything at all?\n\n\t\t\t\tMICHAEL\n\t\tNo, I'm fine.\n\nHe leads Michael into the main casino.\n\n\t\t\t\tSAM ROTH\n\t\tThis is it!  We think it makes\n\t\tVegas look like the corner crap game.\n\n\t\t\t\tMICHAEL\n\t\tVery impressive.\n\n\t\t\t\tSAM ROTH\n\t\tJake, Jake, come over here.  Mike,\n\t\tI want you to meet Jake Cohen; he\n\t\tmanages the casino for us.\n\n\t\t\t\tCOHEN\n\t\t\t(appreciating\n\t\t\tMichael's status)\n\t\tMr. Corleone.\n\nSam turns to Bussetta and extends his glad-hand.\n\n\t\t\t\tSAM ROTH\n\t\tPleasure to meet you, I'm sure...\n\nHe gets no response whatsoever from Bussetta.\n\nINT. PRESIDENTIAL PALACE - NIGHT\n\nAn extremely tall, well-built Cuban, tanned and wearing an\nattractive mustache, LEON, in his middle forties, reads from\na prepared paper.  His sentences are translated by a smaller\nman, standing to his rear.\n\n\t\t\t\tLEON\n\t\t\t(Spanish)\n\t\tMost respected gentlemen, allow me\n\t\tto welcome you to the City of\n\t\tHavana, the Republic of Cuba on\n\t\tbehalf of His Excellency, Fulgencio\n\t\tBatista.\n\nTHE VIEW BEGINS TO MOVE along the various men gathered for\nthis meeting.\n\n\t\t\t\tLEON (O.S.)\n\t\tI'd like to thank this distinguished\n\t\tgroup of American Industrialists,\n\t\tfor continuing to work with Cuba,\n\t\tfor the greatest period of\n\t\tprosperity in her entire history.\n\t\tMr. William Proxmiro, representing\n\t\tthe General Fruit Company... Messrs.\n\t\tCorngold and Dant, of the United\n\t\tTelephone and Telegraph Company; Mr.\n\t\tPetty, regional Vice-President of\n\t\tthe Pan American Mining Corporation;\n\t\tand, of course, our friend Mr.\n\t\tRobert Allen, of South American\n\t\tSugar.  Mr. Nash of the American\n\t\tState Department.  And Mr. Hyman\n\t\tRoth of Miami, and Michael Corleone\n\t\tof Nevada representing our\n\t\tAssociates in Tourism and Leisure\n\t\tActivities.\n\nVIEW ON THE ENTIRE GROUP\n\nLeon pauses to take a drink of water.  Then proudly, he\nlifts a shiny yellow telephone for all to see.\n\n\t\t\t\tLEON\n\t\tThe President would like to take\n\t\tthis opportunity to thank U T&T for\n\t\ttheir lovely gift: a solid gold\n\t\ttelephone!  He thought all you\n\t\tgentlemen would care to take a look\n\t\tat it.\n\nHe hands the heavy phone set to one of his aides, and it is\npassed in turn to each of the men in attendance.\n\n\t\t\t\tCORNGOLD\n\t\tYour Excellency, perhaps you could\n\t\tdiscuss the status of rebel activity\n\t\tand how this may affect our\n\t\tbusinesses.\n\nMED. CLOSE VIEW ON MICHAEL\n\nHe receives the telephone, and glances at it before passing\nit on to Hyman Roth.\n\n\t\t\t\tLEON (O.S.)\n\t\tOf course.  The rebel movement is\n\t\tbasically unpopular, and since July\n\t\tof 1958 has been contained in the\n\t\tOriente Province, in the mountains\n\t\tof the Sierra Muestre.\n\nMichael passes the phone on to Roth.\n\n\t\t\t\tLEON\n\t\t\t(continuing)\n\t\tWe began a highly successful\n\t\toffensive against them in March,\n\t\tand activities within the city\n\t\titself are at a minimum.  I can\n\t\tassure you we'll tolerate no\n\t\tguerrillas in the casinos or\n\t\tswimming pools!\n\nGeneral subdued laughter.\n\nA CUBAN STREET - LATE DAY\n\nPolice are stopping traffic.  Michael's Mercury is among the\ncars; a police officer, seeing that some important person is\nbeing driven, walks up to the driver.  He leans forward, and\nsays something in Spanish to the driver.\n\nThe driver, in turn, leans over to Michael.\n\n\t\t\t\tDRIVER\n\t\tHe says it will just be a short\n\t\ttime and they'll let us through.\n\nMichael looks out the window.\n\nMICHAEL'S VIEW\n\nThe old building has been totally surrounded by police and\nmilitary vehicles.  Right at this moment, they are waiting\nlazily, but soldiers are there with automatic weapons ready.\nThere is a momentary commotion inside the building, and the\nmen brace up.  A Captain of the Army detachment says\nsomething in Spanish over a megaphone; and his men put their\nweapons at the ready, as other policemen lead a group of\ncivilians out of the building with their hands up.\n\nThey are moved over to some military truck, where they are\nfrisked before being loaded.\n\nAll of a sudden, one of the civilian rebels breaks loose,\nand rushes toward the command vehicle.  He hurls himself\ninto the vehicle, as two police try to pull him out.  A\nsecond later, and there is an explosion; the man obviously\nhaving hidden a grenade on his body, sacrificing his own\nlife to take the life of the Captain.\n\nThere is a commotion, but the military quickly quell it.\n\nCLOSE VIEW ON MICHAEL\n\nwatching.  The police rush to Michael's car and guide it\noutside of the trouble area.\n\nMED. VIEW\n\nas they lead and escort the Mercury out of the area.\n\nEXT. HAVANA COUNTRY CLUB - CLOSE VIEW - DAY\n\nSome glasses; rum is poured into them; then Coca Cola.\nQuarter limes are squeezed.\n\n\t\t\t\tSAM ROTH (O.S.)\n\t\tRum... Coca Cola...a squeeze of\n\t\tfresh lime...\n\nSam prepares the drinks for his brother, Hyman, and a group\nof men, including Michael.\n\n\t\t\t\tMAN\n\t\tCuba Libres.\n\n\t\t\t\tMICHAEL\n\t\tI was told the Cubans now call this\n\t\tdrink: \"La Mentira.\"\n\n\t\t\t\tROTH\n\t\tI still don't speak Spanish, Michael.\n\n\t\t\t\tMICHAEL\n\t\tIt means... \"The Lie.\"\n\nA moment's hesitation, then a few of the men laugh.  Now two\nCubans in white carry a table which has a lovely small cake\non it.\n\n\t\t\t\tSAM ROTH\n\t\tThe cake is here.\n\nThey all raise their glasses to the old man.\n\n\t\t\t\tEVERYONE\n\t\t\t(ad lib)\n\t\tHappy Birthday!\n\nRoth glances at the cake and its inscription, is pleased.\n\n\t\t\t\tROTH\n\t\tI hope my age is correct: I am\n\t\talways accurate about my age.\n\nSome laugh.  He nods, and they begin to cut it, put a piece\non plates, and carry them to the different men.\n\n\t\t\t\tROTH\n\t\tEverything we've learned in Vegas\n\t\tis true here; but we can go further.\n\t\tThe bigger, the swankier, the\n\t\tplusher the store, the more a sense\n\t\tof legitimacy, and the bigger\n\t\tbusiness we do.\n\t\t\t(looking at the plate\n\t\t\tbrought to him)\n\t\tA smaller piece.  What we've\n\t\tproposed to the Cuban Government is\n\t\tthat it put up half the cash on a\n\t\tdollar for dollar basis.\n\t\t\t(accepting a smaller piece)\n\t\tThank you.  We can find people in\n\t\tthe United States who will put up\n\t\tour share for a small piece of the\n\t\taction, yet we will retain control.\n\n\t\t\t\tONE OF THE MEN\n\t\tHow much?\n\n\t\t\t\tROTH\n\t\tA hundred million dollars.  But\n\t\tonly if this Government relaxes its\n\t\trestrictions on importing building\n\t\tmaterials; we'll need some new\n\t\tlaws, too, but that will be no\n\t\tdifficulty.\n\n\t\t\t\tANOTHER MAN\n\t\tWhat are import duties now?\n\n\t\t\t\tROTH\n\t\tAs much as seventy percent.  Also,\n\t\tI'm working out an arrangement with\n\t\tthe Minister of Labor so that all\n\t\tour pit bosses, stick-men and\n\t\tDealers, can be considered\n\t\tspecialized technicians eligible\n\t\tfor two year visas.  As of now\n\t\tthey're only allowed in Cuba for\n\t\tsix months at a time.  In short,\n\t\twe're in a full partnership with\n\t\tthe Cuban Government.\n\nVIEW ON MICHAEL\n\nis handed a piece of cake.  Roth moves over to a folder of\ndocuments.\n\n\t\t\t\tROTH\n\t\t\t(continuing)\n\t\tHere are applications from Friends\n\t\tall over the States.  I understand\n\t\tSanto Virgilio in Tampa is trying\n\t\tto make his own deal.  Well, the\n\t\tCuban Government will brush him off.\n\t\tThe Lakeville Road Boys are going\n\t\tto take over the Nacionale here.\n\t\tI'm planning a new hotel casino to\n\t\tbe known as Riviera.  The new Capri\n\t\twill go to the Corleone Family.\n\nMED. VIEW\n\nThe cake is sliced and carried to each of the men.\n\n\t\t\t\tROTH\n\t\tThen there's the Sevilla Biltmore;\n\t\tthe Havana Hilton, which is going\n\t\tto cost twenty-four million --\n\t\tCuban banks will put up half, the\n\t\tTeamsters will bankroll the rest.\n\t\tGenerally, there will be friends\n\t\tfor all our friends including the\n\t\tLieutenant Governor of Nevada;\n\t\tEddie Levine of Newport will bring\n\t\tin the Pennino Brothers, Dino and\n\t\tEddie; they'll handle actual casino\n\t\toperations.\n\nAnd seeing that all of his friends have been served, Roth\nraises his fork.\n\n\t\t\t\tROTH\n\t\tEnjoy.\n\n\t\t\t\tMICHAEL\n\t\tI saw an interesting thing today.\n\t\tA man was being arrested by the\n\t\tMilitary Police; probably an urban\n\t\tguerrilla.  Rather than be taken\n\t\talive, he exploded a grenade hidden\n\t\tin his jacket, taking the command\n\t\tvehicle with him.\n\nThe various men look up as Michael eats his cake, wondering\nwhat the point of it is.\n\n\t\t\t\tMICHAEL\n\t\tIt occurred to me: the police are\n\t\tpaid to fight, and the Rebels are\n\t\tnot.\n\n\t\t\t\tSAM ROTH\n\t\tSo?\n\n\t\t\t\tMICHAEL\n\t\tSo, that occurred to me.\n\nVIEW ON ROTH\n\nHe understands Michael's point, if the others do not.\n\n\t\t\t\tROTH\n\t\tThis country has had rebels for the\n\t\tlast fifty years; it's part of\n\t\ttheir blood.  Believe me, I know...\n\t\tI've been coming here since the\n\t\ttwenties; we were running molasses\n\t\tout of Havana when you were a baby.\n\t\tTo trucks owned by your father.\n\t\t\t(he chuckles warmly\n\t\t\tover the memory)\n\t\tWe'll talk when we're alone.\n\nAnd he returns his attention to the men who are gathered\nwith him on his birthday.\n\nEXT. ROTH'S PRIVATE TERRACE - DAY\n\nMichael sits alone with the old man, on a terrace that\noverlooks the city.\n\n\t\t\t\tROTH\n\t\tYou have to be careful what you say\n\t\tin front of the others... they\n\t\tfrighten easy.  It's always been\n\t\tthat way, most men frighten easy.\n\n\t\t\t\tMICHAEL\n\t\tWe're making a big investment in\n\t\tCuba.  That's my only concern.\n\n\t\t\t\tROTH\n\t\tMy concern is that the three\n\t\tmillion never arrived at Batista's\n\t\tnumbered account in Switzerland.\n\t\tHe thinks it's because you have\n\t\tsecond thoughts about his ability\n\t\tto stop the rebels.\n\n\t\t\t\tMICHAEL\n\t\tThe money was sent.\n\n\t\t\t\tROTH\n\t\tThen you have to trace it.  Michael,\n\t\tpeople here look at me as a reliable\n\t\tman.  I can't afford not to be\n\t\tlooked on as a reliable man.  But\n\t\tyou know all that; there's nothing\n\t\tyou can learn from me.  You\n\t\tshouldn't have to put up with a\n\t\tsick old man as a partner.\n\n\t\t\t\tMICHAEL\n\t\tI wouldn't consider anyone else.\n\n\t\t\t\tROTH\n\t\tExcept the President of the United\n\t\tStates.\n\nHe laughs slyly, as though this is some private joke between\nthem.  Then his laughter becomes a cough, which he painfully\nstifles with a handkerchief.\n\n\t\t\t\tROTH\n\t\tIf only I could live to see it,\n\t\tkid; to be there with you.  How\n\t\tbeautifully we've done it, step by\n\t\tstep.  Here, protected, free to\n\t\tmake our profits without the\n\t\tJustice Department, the FBI; ninety\n\t\tmiles away in partnership with a\n\t\tfriendly government.  Ninety miles,\n\t\tjust a small step, looking for a\n\t\tman who desperately wants to be\n\t\tPresident of the United States, and\n\t\thaving the cash to make it possible.\n\n\t\t\t\tMICHAEL\n\t\tYou'll be there to see it; you'll\n\t\tbe there.\n\nINT. MICHAEL'S SUITE - NIGHT\n\nThe telephone has just rung; Michael listens.\n\n\t\t\t\tOPERATOR\n\t\tWe have your call to Tahoe, Nevada,\n\t\tsir.\n\n\t\t\t\tMICHAEL\n\t\tThank you.\n\t\t\t(click, click)\n\t\tTom?  Tom, is that you?\n\n\t\t\t\tROCCO (O.S.)\n\t\tNo, Tom's out of town.  This is\n\t\tRocco.  Who is this?\n\nMichael is openly disturbed that Hagen is not there.  He\nhangs up without answering.\n\nEXT. NEW ENGLAND HOUSE - DAY\n\nTom Hagen steps out of a taxicab a bit tentatively, and then\nsteps toward the door of a pleasant New England house.  He\nrings the bell and waits, hat in hand.  A moment later, the\ndoor opens, and Kay is standing there.\n\n\t\t\t\tKAY\n\t\tI'm not surprised to see you, Tom.\n\nINT. SMALL ROOM - NEW ENGLAND HOUSE - MED. VIEW - DAY\n\nOut to the yard, where we can see glimpses of little Anthony\nplaying by himself.\n\n\t\t\t\tKAY (O.S.)\n\t\tI can't love a man like that; I\n\t\tcan't live with him, I can't let\n\t\thim be father to my children.  Look.\n\nThe little boy, moodily by himself.\n\nVIEW ON KAY\n\nobviously moved.\n\n\t\t\t\tKAY\n\t\tHe's not like a little boy... he\n\t\tdoesn't talk to me; he doesn't want\n\t\tto play; he doesn't like other\n\t\tchildren, he doesn't like toys.\n\t\tIt's as though he's waiting for the\n\t\ttime he can take his Father's place.\n\t\t\t(almost in tears)\n\t\tYou know what he told me when he\n\t\twas four years old.  He said he had\n\t\tkilled his Grandfather...\n\nVIEW ON HAGEN\n\nlistening, calmly.\n\n\t\t\t\tKAY\n\t\t... He said he had shot his\n\t\tGrandfather with a gun, and then he\n\t\tdied in the garden.  And he asked\n\t\tme... he asked me, Tom, if that\n\t\tmeant now his father would shoot\n\t\thim out of... revenge.\n\t\t\t(she cries)\n\t\tHow does a four year old boy learn\n\t\tthe word... 'revenge'?\n\n\t\t\t\tHAGEN\n\t\tKay... Kay...\n\nVIEW ON KAY\n\n\t\t\t\tKAY\n\t\tWhat kind of a family is this...\n\t\tare we human beings?  He knows his\n\t\tFather killed his Uncle Carlo.  He\n\t\theard Connie.\n\n\t\t\t\tHAGEN\n\t\tYou don't know that's true.  But\n\t\tKay, just for the sake of an\n\t\targument, let's assume it is, I'm\n\t\tnot saying it is, remember, but...\n\t\tWhat if I gave you what might be\n\t\tsome justification for what he\n\t\tdid... or rather some possible\n\t\tjustification for what he possibly\n\t\tdid.\n\n\t\t\t\tKAY\n\t\tThat's the first time I've seen the\n\t\tlawyer side of you, Tom.  It's not\n\t\tyour best side.\n\n\t\t\t\tHAGEN\n\t\tOkay, just hear me out.  What if\n\t\tCarlo had been paid to help get\n\t\tSonny killed?  What if his beating\n\t\tof Connie that time was a deliberate\n\t\tplot to get Sonny out into the open?\n\t\tThen what?  And what if the Don, a\n\t\tgreat man, couldn't bring himself\n\t\tto do what he had to do, avenge his\n\t\tson's death by killing his\n\t\tdaughter's husband?  What if that,\n\t\tfinally, was too much for him, and\n\t\the made Michael his successor,\n\t\tknowing that Michael would take\n\t\tthat load off his shoulders, would\n\t\ttake that guilt?\n\n\t\t\t\tKAY\n\t\tHe's not the same as when I met him.\n\n\t\t\t\tHAGEN\n\t\tIf he were, he'd be dead by now.\n\t\tYou'd be a widow.  You'd have no\n\t\tproblem.\n\n\t\t\t\tKAY\n\t\tWhat the hell does that mean?  Come\n\t\ton, Tom, speak out straight once in\n\t\tyour life.  I know Michael can't,\n\t\tbut you're not Sicilian, you can\n\t\ttell a woman the truth; you can\n\t\ttreat her like an equal, a fellow\n\t\thuman being.\n\nThere is a long silence.\n\nThen Hagen shakes his head; he can tell her no more.\n\n\t\t\t\tHAGEN\n\t\tIf you told Michael what I've told\n\t\tyou today, I'm a dead man.\n\n\t\t\t\tKAY\n\t\tWhen is it finally over?  I want it\n\t\tto be over before my baby is born.\n\n\t\t\t\tHAGEN\n\t\tI don't know.  I hope soon; but\n\t\tit's not over yet, and that's why\n\t\tyou and the kids have to come back\n\t\tto me.\n\nHe looks at her; it's clear that he has been entrusted with\nher safety and her children's.\n\nHe is a kind, good man, and seems very nervous and\noverwrought.\n\nVIEW ON THE WINDOW\n\nLittle Anthony is pressing his face against the glass pane,\nas though he senses the adults are discussing something of\nimportance to him.\n\nINT. TROPICANA HOTEL-CASINO - MED. VIEW - NIGHT\n\nThe Baccarat table.  Busy, hundred dollar bills being played.\n\n\t\t\t\tLOUDSPEAKER\n\t\tMr. Corleone; Mr. Freddie Corleone,\n\t\ttelephone please.\n\n\t\t\t\tPIT BOSS\n\t\tNot here.\n\nVIEW ON THE CRAP TABLES\n\nThe play is fast; pit boss presiding; but no sign of Fredo.\n\n\t\t\t\tLOUDSPEAKER\n\t\tTelephone for Mr. Corleone.\n\nANOTHER PART OF THE CASINO\n\nWe see Neri, ominous, presiding over the entire store.  He\npicks up a pit telephone.\n\n\t\t\t\tNERI\n\t\tHe's backstage.\n\t\t\t(and hangs up disgustedly)\n\nINT. TROPICANA BACKSTAGE AREA - MED. VIEW - NIGHT\n\nFredo is entertaining two showgirls done up in feathers and\nwhat-have-you.\n\n\t\t\t\tFREDO\n\t\tC'mon, you got fifteen minutes\n\t\tbefore the finale!  I want to show\n\t\tyou a trick with feathers.\n\n\t\t\t\tSTAGEHAND\n\t\tPhone for you.\n\n\t\t\t\tFREDO\n\t\tDon't go away; wait a minute.\n\nHe takes the phone; we can catch a VIEW of the show going on\nfrom the wings.\n\n\t\t\t\tFREDO\n\t\t\t(on the phone)\n\t\tYeah.  Okay.  Who?  Mikey?  But...\n\t\tSi... si, caposco.\n\t\t\t(in Sicilian)\n\t\tSure... how much?  I understand.\n\t\tJesus, three million... I won't let\n\t\tyou down.  Sure.\n\nHe hangs up thoughtfully.\n\n\t\t\t\tONE OF THE GIRLS\n\t\tFreddie; we still got twelve\n\t\tminutes before the finale!\n\n\t\t\t\tFREDO\n\t\tYeah... some other time.\n\nEXT. NEW YORK BAR - DAY\n\nThere is a light rain.  Pentangeli steps out of his car;\npoints to Willy Cicci.\n\n\t\t\t\tPENTANGELI\n\t\tWait in the car.\n\nHe walks up the street, to the bar, where he is greeted by\nthe tall, handsome Carmine Rosato.  They shake hands.\nPentangeli looks in his hand.\n\nCLOSE VIEW\n\nRosato has put a crisp one hundred dollar bill in his hand,\nfolded sharply in two.\n\n\t\t\t\tPENTANGELI\n\t\tWhat's this?\n\n\t\t\t\tROSATO\n\t\tThat's a lucky C note for our new\n\t\tdeal.\n\nHe puts his arm around Pentangeli, and they walk into the bar.\n\nINT. THE BAR - DAY\n\nThe bar is fairly empty; and very dark.  Pentangeli and\nRosato step up to the bar; the bartender momentarily stops\npolishing glasses to pour a couple of drinks.\n\n\t\t\t\tROSATO\n\t\tWe were all real happy about your\n\t\tdecision, Frankie; you're not goin'\n\t\tto regret it.\n\nHe holds up the glass.\n\n\t\t\t\tPENTANGELI\n\t\tI don't like the C-note.  I take it\n\t\tlike an insult.\n\nSuddenly, a garrote is thrown around Pentangeli's throat;\nand he is forcefully yanked back into the shadows, all the\nway into a wooden telephone booth.\n\nCLOSE VIEW\n\nThe folded hundred dollar bill resting on the bar.\n\nMED. CLOSE - THE PHONE BOOTH\n\nWe see only Pentangeli's feet and legs, struggling.  We HEAR\nthe terrible sounds of a man being strangled.\n\nCLOSE ON ROSATO\n\nCalm, and then he sees something that disturbs him.\n\n\t\t\t\tROSATO\n\t\tShit, your friend the cop!\n\nSuddenly, the side door opens, and a shaft of sunlight cuts\nthrough the darkness.\n\n\t\t\t\tCOP\n\t\tEverything all right in there,\n\t\tRitchie?  The door was open.\n\nCLOSE ON THE PHONE BOOTH\n\nPentangeli's feet stop moving.\n\n\t\t\t\tRITCHIE\n\t\tJust cleaning up.\n\t\t\t(strained voice)\n\t\tYou okay?\n\n\t\t\t\tCOP\n\t\tIs that something on the floor?\n\n\t\t\t\tROSATO\n\t\tTake him!\n\n\t\t\t\tVOICE\n\t\tOkay.\n\n\t\t\t\tRITCHIE\n\t\tNot here; not a cop, not here!\n\nTwo figures race through the shadows and race through the\ndoors.\n\n\t\t\t\tCOP\n\t\t\t(shouting to his\n\t\t\tpartner, in uniform)\n\t\tStutz!  Watch out, Stutz!\n\nEXT. THE BAR - DAY\n\nWe see that a patrol car had stopped for its routine visit.\nSTUTZ, the second patrolman, is just stepping out of his\ncar; Pentangeli's bodyguard, seeing the commotion, leaps out.\nThree men, including Rosato, rush out.  There is gunfire;\nCicci is wounded.\n\nMED. CLOSE\n\nThe patrolmen is grazed across the face; trying to stop the\nflow of blood with his hand.\n\nNEW VIEW\n\nThe three assailants jump into the car and drive off.\n\nINT. THE BAR - DAY\n\nThe stricken Pentangeli comes back to life.  He can barely\nmove his lips.\n\n\t\t\t\tPENTANGELI\n\t\tThe bastard.  The dirty bastard, he\n\t\tgave me a C-note.  He gave me a C-\n\t\tnote.\n\nHe sees the patrolman leaning over him.\n\nEXT. PATROL CAR - DAY\n\nThe Sergeant is on the car radio.\n\n\t\t\t\tSERGEANT\n\t\tFrankie Pentangeli murder attempt.\n\t\tPatrolman Stutz shot.  Sahara\n\t\tLounge - Utica Avenue and Claredon\n\t\tRoad.  White Cadillac three or four\n\t\tmen took off from scene.  Need\n\t\tambulance; Stutz is bad.  Taking\n\t\tPentangeli into custody...\n\nINT. ROTH'S SUITE IN HAVANA - MED. CLOSE VIEW ON ROTH - DAY\n\nHis wizened face, pale.  Right now, though, his eyes have a\nsparkle as he watches three million dollars in cold cash\nbeing counted on a card table in front of him.\n\nHis brother Sam is present, and the sandy-haired Courier, a\nlittle nervous; the one who had left from the Tropicana with\nthe Corleone skim-money.  Also Johnny Ola.  The money is\nevidently all there; Roth picks up a packet; probably a\nhundred thousand dollars, and throws it over to the Courier.\n\n\t\t\t\tROTH\n\t\tMake it fast; I don't want to\n\t\tchance him being seen.\n\n\t\t\t\tCOURIER\n\t\t\t(frightened)\n\t\tWhat about the arrangements?  How\n\t\tcan I be sure about the arrangements?\n\n\t\t\t\tOLA\n\t\tRelax.  You're under our protection;\n\t\tthe Corleone family will never find\n\t\tyou.\n\nOla leads the Courier to the adjoining room where two\nsmartly dressed Military (Cuban) Police are standing, and a\ncivilian.  The Courier sees them, looks back to Ola.  One of\nthe police steps forward, placing the Courier under arrest;\nhandcuffing him.\n\n\t\t\t\tCOURIER\n\t\tHey, what's this?\n\nThe other takes the packet of money, and hands it to the\ncivilian, who places it in the briefcase he carries.  The\nother officer kneels down and fastens leg manacles.\n\n\t\t\t\tCOURIER\n\t\tThe arrangements... YOU BASTARDS!\n\t\tWhat...\n\nThe Captain strikes him expertly across the side of his head\nwith his pistol.\n\nOla closes the door on this scene.\n\nEXT. THE HAVANA CAPRI - DAY\n\nFredo Corleone steps out of a car, squints up at the sunshine\nand palm trees.  He is holding on tightly to a small satchel,\nwhich he won't let the bellman carry along with his other\nthings.\n\nINT. MICHAEL'S SUITE - MED. CLOSE VIEW - DAY\n\nMichael and Fredo in a brother's embrace; they kiss each\nother.  Fredo is still in his jacket, holding the satchel.\n\n\t\t\t\tFREDO\n\t\tMikey.  How are you?\n\nHe glances up at Bussetta, who doesn't say a word.  Fredo\nextends his hand.\n\n\t\t\t\tFREDO\n\t\tHiya, Freddie Corleone.\n\n\t\t\t\tMICHAEL\n\t\tMio fratello.\n\nThen Bussetta offers his hand back to Fredo.\n\n\t\t\t\tFREDO\n\t\t\t(taking off his jacket)\n\t\tWhat a trip, Jesus Christ, the\n\t\twhole time I'm thinking what if\n\t\tsomeone knew what I got in here.\n\nHe undoes the combination of the briefcase starts taking out\ncash.  Then he stops, remembering that there's a stranger in\nthe room.\n\n\t\t\t\tFREDO\n\t\tOh, 'scuse me.\n\n\t\t\t\tMICHAEL\n\t\tIt's all right.  He stays with me\n\t\tall the time.\n\n\t\t\t\tFREDO\n\t\tOh.  Mikey, what's up?  I'm totally\n\t\tin the dark.\n\n\t\t\t\tMICHAEL\n\t\tWe're making an investment in Havana.\n\n\t\t\t\tFREDO\n\t\tGreat, Havana's great.  Lots of\n\t\tactivity in Havana!  Anybody I know\n\t\there.  Five-Angels?  Anybody?\n\n\t\t\t\tMICHAEL\n\t\tJohnny Ola... Hyman Roth.\n\n\t\t\t\tFREDO\n\t\tI never met them.\n\n\t\t\t\tMICHAEL\n\t\tPentangeli's dead.  He was ambushed\n\t\tby the Rosato Brothers.\n\t\t\t(pause)\n\t\tDidn't you know that?\n\n\t\t\t\tFREDO\n\t\tNo.  No, I didn't.  Who tells me\n\t\tanything?  I been kept in the dark\n\t\tso long, I'm getting used to it.\n\n\t\t\t\tMICHAEL\n\t\tI want you to help me, Fredo.\n\n\t\t\t\tFREDO\n\t\tThat's what I'm here for.\n\n\t\t\t\tMICHAEL\n\t\tTonight I want to relax with you.\n\t\tThe Senator from Nevada is here\n\t\twith some people from Washington.\n\t\tI want to show them a good time in\n\t\tHavana.\n\n\t\t\t\tFREDO\n\t\tCount on me; that's my specialty.\n\n\t\t\t\tMICHAEL\n\t\tI'd like to come along.  There's\n\t\tbeen a lot of strain, and I've been\n\t\tcooped up in this room for three\n\t\tdays.\n\n\t\t\t\tFREDO\n\t\tMe and you, great!  Gimme an hour\n\t\tto wash my face and do my research\n\t\tand we'll have these Washington\n\t\tsuckers right where you want 'em.\n\t\t\t(then a thought\n\t\t\tstrikes him)\n\t\tPoor Frankie Five-Angels.  He\n\t\talways wanted to die in bed...with\n\t\ta broad.\n\nINT. ROTH'S SUITE - MED. VIEW - DAY\n\nMichael stands at Roth's door carrying the briefcase that\nFredo had brought.\n\nA hotel DOCTOR takes Hyman Roth's blood pressure, while his\nwife waits nervously.\n\n\t\t\t\tDOCTOR\n\t\t\t(Spanish)\n\t\tYou must not exert yourself; I will\n\t\twrite out a prescription and come\n\t\tback tomorrow.\n\n\t\t\t\tHOTEL MAN\n\t\tHe's going to write a prescription.\n\n\t\t\t\tROTH\n\t\tI want my own doctor; fly him in\n\t\tfrom Miami.  I don't trust a doctor\n\t\twho can't speak English.\n\nThe doctor is shown out.  Roth gestures to the hotel man,\nwho also leaves.  Then he looks to his wife.\n\n\t\t\t\tROTH\n\t\tHoney, go down to the casino?\n\n\t\t\t\tTERRI\n\t\tIf you feel better...\n\n\t\t\t\tROTH\n\t\tI do.  Play the Bingo game.\n\nThey kiss, and she leaves.  Also Bussetta and Ola remain.\n\n\t\t\t\tROTH\n\t\tMy sixth sense tells me you have a\n\t\tbag full of money in your hand.\n\nOla locks the door; Michael nods, and opens the bag, spilling\nits contents on the card table.\n\n\t\t\t\tMICHAEL\n\t\tThis doubles my investment.\n\n\t\t\t\tROTH\n\t\tStill no word of your courier?\n\t\tWe'll find him.  But at least this\n\t\twill satisfy our friends here.\n\t\tYou've been invited to the New Year\n\t\treception at the Presidential Home.\n\t\tI understand your brother is here\n\t\tas well; I hope he'll come.\n\n\t\t\t\tMICHAEL\n\t\tSix million dollars in cash is a\n\t\thigh price for a piece of a country\n\t\tin the middle of a revolution.\n\nRoth looks patiently at Michael, as though he were a child\nwho hadn't minded the lesson that he had been taught over\nand over again.\n\n\t\t\t\tROTH\n\t\tYou're a careful kid, and that's\n\t\tgood.  But look.  An international\n\t\tdispatch on the wire service.\n\t\tAmerican journalism, not propaganda.\n\t\tThe government troops have all but\n\t\teliminated the rebels.  All but\n\t\ttheir radio station.\n\n\t\t\t\tMICHAEL\n\t\tI've read it; I'm pleased that the\n\t\tgovernment is doing so well.  As a\n\t\theavy investor, I'm pleased.  How\n\t\tdid the doctor find you?\n\n\t\t\t\tROTH\n\t\tTerrible.  I'd give twice this\n\t\tamount to take a piss without it\n\t\thurting.\n\n\t\t\t\tMICHAEL\n\t\tWho had Frankie Pantangeli killed?\n\n\t\t\t\tROTH\n\t\t\t(taken a bit off-balance)\n\t\tWhy...the Rosato Brothers.\n\n\t\t\t\tMICHAEL\n\t\tI know that; but who gave the go\n\t\tahead.\n\nRoth glances to Ola; he is not a fool; he realizes Michael\nhas begun to suspect him.\n\n\t\t\t\tMICHAEL\n\t\tI know it wasn't me...so that\n\t\tleaves you.\n\n\t\t\t\tROTH\n\t\tThere was this kid that I grew up\n\t\twith; he was a couple years younger\n\t\tthan me, and sort of looked up to\n\t\tme, you know.  We did our first\n\t\twork together, worked our way out\n\t\tof the street.  Things were good\n\t\tand we made the most of it.  During\n\t\tprohibition, we ran molasses up to\n\t\tCanada and made a fortune; your\n\t\tfather too.  I guess as much as\n\t\tanyone, I loved him and trusted him.\n\t\tLater on he had an idea to make a\n\t\tcity out of a desert stop-over for\n\t\tG.I.'s on the way to the West Coast.\n\t\tThat kid's name was Moe Greene, and\n\t\tthe city he invented was Las Vegas.\n\t\tThis was a great man; a man with\n\t\tvision and guts; and there isn't\n\t\teven a plaque or a signpost or a\n\t\tstatue of him in that town.  Someone\n\t\tput a bullet through his eye; no\n\t\tone knows who gave the order.  When\n\t\tI heard about it I wasn't angry.  I\n\t\tknew Moe; I knew he was headstrong,\n\t\tand talking loud, and saying stupid\n\t\tthings.  So when he turned up dead,\n\t\tI let it go, and said to myself:\n\t\tthis is the business we've chosen.\n\t\tI never asked, who gave the go\n\t\tahead because it had nothing to do\n\t\twith business.\n\nHe regards Michael silently a moment.\n\n\t\t\t\tROTH\n\t\t\t(continuing)\n\t\tThere's three million dollars on\n\t\tthat table.  I'm going to lie down,\n\t\tmaybe take a nap.  When I wake up,\n\t\tif it's still there, I'll know I\n\t\thave a partner.  If it's gone, then\n\t\tI'll know I don't.\n\nThe old man turns, and moves in his slippers, toward his\nbedroom.\n\nINT. THE CORRIDOR - DAY\n\nMichael closes the door, and moves down the hallway.  He is\nfollowed by Bussetta, who had waited in the corridor.\n\n\t\t\t\tMICHAEL\n\t\t\t(Sicilian)\n\t\tHow sick do you think the old man is?\n\n\t\t\t\tBUSSETTA\n\t\t\t(Sicilian)\n\t\tHe'll live longer than me.\n\nINT. TROPICOR NIGHT CLUB - VIEW ON THE SHOW - NIGHT\n\nA Havana extravaganza, with tall, beautiful showgirls done\nup in flamboyant, 'South-of-the-Border' Carmen Miranda\ncostumes; the lead singer is a six foot blonde doing \"Rum\nand Coca Cola\" in that style.  Her name is YOLANDA.\n\nMED. VIEW\n\nAt a large round table, located in an obvious VIP section of\nthe high, tropically draped room with living ferns and other\ntropical planting with artificial stars.\n\nMichael rises, to be introduced by Fredo to some conservative\nlooking Senatorial types, including Senator Pat Geary of\nNevada.  We notice Bussetta standing nearby.\n\n\t\t\t\tFREDO\n\t\tDoes everyone know everyone, or\n\t\tnobody knows nobody.  Here, my\n\t\tbrother, Michael Corleone... well,\n\t\tyou know Senator Geary.\n\nGeary warmly shakes Michael's hand.\n\n\t\t\t\tSENATOR GEARY\n\t\tGood to see you, Mike; I'm glad we\n\t\tcan spend this time together.\n\n\t\t\t\tFREDO\n\t\tThis is Senator Payton from Florida;\n\t\tJudge DeMalco from New York...\n\t\tSenator Ream... Mr. Questadt from\n\t\tCalifornia, he's a lawyer with the\n\t\tPrice-Control Administration.  And\n\t\tFred Corngold of U T&T.\n\nThey all make themselves comfortable.  A waiter with a tray\nof drinks appears.\n\n\t\t\t\tFREDO\n\t\tGentlemen... your pleasure?  Cuba\n\t\tLibres, Pina Coladas, you name it.\n\n\t\t\t\tSENATOR GEARY\n\t\tI'll take a Yolanda.\n\nLaughter.\n\n\t\t\t\tFREDO\n\t\tLater, later.  All those girls look\n\t\tlike they're on stilts!\n\nThe various tropical drinks are distributed.\n\n\t\t\t\tSENATOR GEARY\n\t\tTo a night in Havana!\n\nThey all join in.\n\n\t\t\t\tFREDO\n\t\t\t(aside to Michael)\n\t\tJeeze, it's great you came along,\n\t\tMike... You know, we've never spent\n\t\ta night out on the town together.\n\t\tI always thought you looked down on\n\t\tme for liking a good time.\n\n\t\t\t\tMICHAEL\n\t\tI never looked down on you, Fredo.\n\t\tYou don't look down at a brother.\n\nINT. THE CASINO - NIGHT\n\nBy now the group has made its way into the casino.  Some of\nthem are crowded around the crap table; Senator Geary is\nwith the enormous and beautiful Yolanda, who barely speaks\nEnglish.  There are other girls with some of the men; not\nwith Michael, who gambles dollars while talking to Corngold.\n\n\t\t\t\tCORNGOLD\n\t\tOur information is that Castro is\n\t\tdead.  There are maybe a few\n\t\thundred die-hards in the Sierra\n\t\tMuestra; but government troops are\n\t\tgoing to clean them out any day.\n\nJohnny Ola approaches Michael.\n\n\t\t\t\tOLA\n\t\tMike, can I talk to you.\n\nMichael follows Ola toward the Baccarat table; a watchful\nBussetta moves, a distance away, with them.\n\n\t\t\t\tOLA\n\t\tListen, this Senator from Florida\n\t\talready has a hundred grand worth\n\t\tof markers on the table.\n\nWe can see Senator Ream at the table, making thousand dollar\nbets on the Bank.\n\n\t\t\t\tOLA\n\t\tThey asked him to sign paper to\n\t\ttake down the markers; but he got\n\t\tmad; told them to wait until he was\n\t\tfinished.\n\n\t\t\t\tMICHAEL\n\t\tLet him gamble.\n\n\t\t\t\tOLA\n\t\tOkay.  You know he doesn't have\n\t\tthat kind of money.\n\n\t\t\t\tFREDO\n\t\tMike said let him gamble.\n\nFredo puts his arm around his brother; he is high with the\nfirst attention Mike has ever given him, as though finally\nhe is being taken seriously; as though his brother needs him.\n\n\t\t\t\tFREDO\n\t\tMike, I got something special up my\n\t\tsleeve for these boys.  You ever\n\t\thear of \"Superman?\" And I don't\n\t\tmean the comic book.\n\n\t\t\t\tMICHAEL\n\t\tNo.\n\n\t\t\t\tFREDO\n\t\tWait'll you see!\n\nINT. HAVANA BAR - NIGHT\n\nOur group are in a large Havana bar; the walls totally\ncovered with hundreds of fifths of different types of rum\nand other liquor.\n\nA couple of the girls from the show are out with the men;\nYolanda herself is giving them a private song and dance.\n\nFredo is a little loaded, and especially attentive to\nMichael this night.\n\n\t\t\t\tFREDO\n\t\tMikey, why would they ever hit poor\n\t\told Frankie Five-Angels?  I loved\n\t\tthat ole sonuvabitch.  I remember\n\t\twhen he was just a 'button,' when\n\t\twe were kids.  We used to put\n\t\tbedsheets on our heads, you know,\n\t\tlike we were ghosts.  An' ole\n\t\tFrankie come peek into our room,\n\t\twe'd jump up, and he'd always\n\t\tpretend like he was really scared.\n\t\tYou remember?\n\n\t\t\t\tMICHAEL\n\t\tIt was hard to have him killed.\n\n\t\t\t\tFREDO\n\t\tYou?  What do you mean you, I\n\t\tthought...\n\n\t\t\t\tMICHAEL\n\t\tIt was hard to have him killed.\n\n\t\t\t\tFREDO\n\t\tYou?  What do you mean you, I\n\t\tthought...\n\n\t\t\t\tMICHAEL\n\t\tIt was Frankie tried to have me hit.\n\n\t\t\t\tFREDO\n\t\tNo.  I mean, are you sure?\n\n\t\t\t\tMICHAEL\n\t\tYou know otherwise, Freddie?\n\n\t\t\t\tFREDO\n\t\tMe?  NO, no, I don't know anything.\n\t\tFellas!  You're all falling asleep.\n\t\tWe got to see Superman.\n\nCLOSE ON MICHAEL\n\nA growing feeling about his brother.\n\nEXT. GARISH HAVANA STREET - NIGHT\n\nThe street is lit with tons of neon signs; it is alive with\npeople; some roving bands of musicians.  Everywhere are\nlittle boys running around, begging for money.  And in\ndoorways and windows are silent, dark-skinned women.\n\n\t\t\t\tSENATOR REAM\n\t\t\t(pushing away from\n\t\t\tthe palm outstretched\n\t\t\tlittle hands of the boys)\n\t\tGoddamn beggers.  Goddamn city of\n\t\tbeggars and pimps and whores.  And\n\t\twe bend over backwards to support\n\t\tthem with the goddamn sugar quota.\n\n\t\t\t\tFREDO\n\t\t\t(to Geary)\n\t\tWhat's eating him?\n\n\t\t\t\tSENATOR GEARY\n\t\tHe lost a quarter million dollars\n\t\tat the casino.\n\n\t\t\t\tSENATOR REAM\n\t\t...goddamn city of whores...\n\n\t\t\t\tSENATOR GEARY\n\t\tHe gave them a bad check.\n\nINT. 'SUPERMAN SHOW' - MED. VIEW - NIGHT\n\nA large room with a succession of platforms arranged step-\nlike around a circular area which becomes a stage.\n\nThere are a hundred or so people, practically all men,\ntourists and business men, standing on the different levels,\nforming the audience.\n\nIn the center of the stage is a thick, telephone type pole,\nto which is tied a young Cuban girl, in a flimsy white\nsacrificial slip.  A small band, mostly drummers, play some\nLatin music.\n\nMED. VIEW\n\nFredo's party standing on the ramp, looking down at the\nspectacle.  They're a little woozy from the drinks and late\nhour.  Michael is with them, but now we sense he is using\nthis time, with all exhausted and drunk, to come to some\nimportant conclusions.\n\n\t\t\t\tQUESTADT\n\t\tWhy do we have to stand?\n\n\t\t\t\tFREDO\n\t\tEveryone stands.  But it's worth\n\t\tit, watch!\n\nVIEW ON THE ARENA\n\nNow two high priestesses, scantily clad, bring in a tall and\nmuscular Cuban, done up in chains and loin cloth, as though\nhe were a captured slave.  This is SUPERMAN\n\nVIEW ACROSS THE MEN TO THE STAGE\n\n\t\t\t\tFREDO\n\t\tThat's him; that's Superman!\n\nSome preliminary pornographic proceedings go on, as the\npriestesses lead the slave to the virgin tied to the post.\nThe music is percussive and wild.\n\nMED. VIEW ON THE MEN\n\n\t\t\t\tSENATOR GEARY\n\t\tOhmygod.  I don't believe it.\n\n\t\t\t\tQUESTADT\n\t\tIt's got to be fake.\n\n\t\t\t\tFREDO\n\t\tThat's why they call him Superman.\n\t\tJohnny Ola told me about this; I\n\t\tdidn't believe it.\n\nCLOSE on Michael turning away.  Not because of the spectacle\nwhich he finds disgusting, but at what his brother is saying.\n\n\t\t\t\tFREDO (O.S.)\n\t\t... but seeing is believing.  Ole\n\t\tJohnny knows all the places.  I\n\t\ttol' you... can you believe it?\n\nIf Michael would ever allow himself to cry, it would be now.\n\n\t\t\t\tFREDO\n\t\t\t(continuing)\n\t\tThe old man Roth, would never come;\n\t\tbut Johnny knows these places like\n\t\tthe back of his hand...\n\n\t\t\t\t\t\t\tFADE OUT.\n\nFADE IN:\n\nINT. MICHAEL'S SUITE - MED. VIEW - MORNING\n\nMichael is alone in his bedroom; it seems as though he\nhasn't slept very much, but sits by his window, looking out\nat the city.  He is troubled and tired.\n\nHis radio is on:\n\n\t\t\t\tRADIO\n\t\t\t(Spanish)\n\t\t\"This is Rebel Radio: Rebel troops\n\t\tof Column Four 'Jose Marti' took\n\t\tthe town of Baire yesterday at 8:30\n\t\tp.m.  The enemy has retreated...\"\n\nEXT. CUBAN STREET - MORNING\n\nThis street in Havana is like a Caribbean tourist city with\nno indication of the revolution in progress.\n\nMichael walks along the street, alone, past the Cubans on\ntheir way to work; past the American ladies who have gotten\nup early for their shopping spree.\n\n\t\t\t\tRADIO\n\t\t\t(Spanish)\n\t\t\t (continuing)\n\t\t... An important military action is\n\t\tdeveloping along a 35-kilometer\n\t\tstretch of the Central Highway.\n\t\tNumerous enemy garrisons are left\n\t\twith two alternatives, surrender or\n\t\tannihilation...\n\nOne full block away, Bussetta rides in the front seat of the\ndark Mercury, driving slowly, giving Michael his privacy,\nbut never letting him out of Bussetta's sight.\n\nCLOSE ON MICHAEL\n\nwatching.\n\nMICHAEL'S VIEW\n\nShopkeepers happily luring the tourists into their shops in\nbroken English.  Havana is prosperous.\n\n\t\t\t\tRADIO\n\t\t\t(continuing)\n\t\t... Victories in war depend on a\n\t\tminimum on weapons and to a maximum\n\t\ton morale...\n\nVIEW ON MICHAEL\n\nglances back to the dark car following him.  In a moment, it\npulls up to him, and he gets into the back seat.\n\nEXT. AMERICAN MILITARY MISSION - VIEW ON MICHAEL - DAY\n\nstanding by his car, looking through the cyclone fencing\nthat borders this military training camp operated by the\nAmerican Army near the city.\n\n\t\t\t\tRADIO\n\t\t... War is not a simple question of\n\t\trifles, bullets, guns and planes...\n\nCLOSER VIEW INTO THE CAMP\n\nEXT. HAVANA STREET - DAY\n\nA street singer, followed by a guitarist sings Jose Marti's\nwords of \"Guantanamera.\" It is solemn, as though it is a\nsong of protest, a song of the revolution.\n\nNearby, in a restaurant, Michael has lunch with Fredo.\n\n\t\t\t\tMICHAEL\n\t\tHow is your wife, Fredo...your\n\t\tmarriage?\n\n\t\t\t\tFREDO\n\t\t\t(eating)\n\t\tYou know her; drives me crazy, one\n\t\tminute she's a popsicle, the next\n\t\tshe's all vinegar.  Sometimes I\n\t\tthink... I think - I should a\n\t\tmarried someone, like you did.  To\n\t\thave kids, to have a family.\n\nMichael turns, distracted for a moment at something the\nsinger has sung.\n\n\t\t\t\tMICHAEL\n\t\t\"Yo soy un hombre sincero...\"\n\t\tI am a sincere man,\n\t\tFrom the land of the palms...\n\n\t\t\t\tFREDO\n\t\tWhat's that?\n\n\t\t\t\tMICHAEL\n\t\tThe song.  Are you sincere with me,\n\t\tFredo?\n\n\t\t\t\tFREDO\n\t\tSincere.  What are you talking\n\t\tabout, of course I'm sincere with\n\t\tyou, Mike.\n\n\t\t\t\tMICHAEL\n\t\tThen I'm going to confide in you;\n\t\ttrust you with something.\n\n\t\t\t\tFREDO\n\t\t\t(Sicilian)\n\t\tMike, are you crazy, I'm your\n\t\tbrother.\n\n\t\t\t\tMICHAEL\n\t\tTonight we've been invited to a\n\t\treception at the Presidential\n\t\tPalace; to bring in the New Year.\n\t\tYou and I will go in a special car\n\t\tthat's being sent.  They'll have\n\t\tcocktails... then dinner, and a\n\t\treception with the President.  When\n\t\tit's over, it will be suggested\n\t\tthat you take Questadt and his\n\t\tfriends from Washington to spend\n\t\tthe night with some women.  I'll go\n\t\thome alone in the car; and before I\n\t\treach the hotel, I'll be\n\t\tassassinated.\n\n\t\t\t\tFREDO\n\t\t...Who?\n\n\t\t\t\tMICHAEL\n\t\tThe same man who tried in Nevada...\n\t\tHyman Roth, not Pentangeli.\n\n\t\t\t\tFREDO\n\t\tBut, you told me yourself...\n\n\t\t\t\tMICHAEL\n\t\tIt was never Pentangeli... I've\n\t\talways known that.  It was Roth all\n\t\talong.  He talks to me as a son; as\n\t\this successor, but the old man\n\t\tthinks he'll live forever.\n\n\t\t\t\tFREDO\n\t\tWhat do you want me to do?\n\n\t\t\t\tMICHAEL\n\t\tTo go tonight, with me, as though\n\t\twe know nothing.  I've already made\n\t\tmy move.\n\n\t\t\t\tFREDO\n\t\tWhat is it?  Can I help?\n\n\t\t\t\tMICHAEL\n\t\tThe old man will never bring in the\n\t\tNew Year.\n\nFredo realizes what he means; looks immediately to Bussetta,\nwho had been sitting near the door and the musicians.  He is\ngone.\n\nINT. HOTEL CORRIDOR - MOVING VIEW ON BUSSETTA - NIGHT\n\nThe first time ever away from Michael, moving toward us\nquickly.  He stops, knocks on the door of Roth's suite.\nThen quickly for a man his size, he moves without noise to\nthe adjoining door, opens it with a key, and disappears\ninside.\n\nA moment elapses on the empty corridor, and then a roused\nJohnny Ola, opens the first door.  He steps out into the\ncorridor, to see who had knocked.  Confused, he is about to\nreturn inside, when Bussetta easily breaks his neck in two\nfrom behind.\n\nINT. THE SUITE - MED. VIEW - NIGHT\n\nas Bussetta quietly pulls the limp body of Johnny Ola, his\nhead bent at an impossible angle, and lays it at the foot of\nthe couch.\n\nEXT. PRESIDENTIAL PALACE - NIGHT\n\nGuards who are regular troops patrol the Palace in twos,\ncarrying machine weapons.\n\nNow an elite officer, checks the identification of the\nvarious cars carrying dignitaries, as they are driven up to\nthe Palace.  The one being inspected at the moment contains\nFredo and Michael.  We can see the beautifully dressed\npeople on their way to the reception, and sense the cheerful\nmood of this New Year's Eve.\n\nINT. THE SUITE - NIGHT\n\nBussetta bends over Ola's body, tying the wrists and knees\nwith electrical extensions.  He then easily carries the body\nto the small balcony which all the rooms have.\n\nEXT. THE BALCONY - NIGHT\n\nBussetta swings the body over the side of the balcony\nrailing; tying the extension cord to the railing, and\nsuspending the body so that it is invisible both from the\ninside and out during the night.\n\nINT. PRESIDENTIAL PALACE - VIEW ON THE MAIN FOYER - NIGHT\n\nThe PRESIDENT, his WIFE and six oldest CHILDREN great\nformally the many beautifully and affluently dressed guests.\nHe speaks to them in Spanish, as one by one they file to him.\n\nMichael and Fredo are presented in a group with several\nother Americans, including several of the American\nbusinessmen with interests in Cuba.\n\nEXT. STREETS OF HAVANA - MED. VIEW - NIGHT\n\nThe excitement of the night is beginning to build; people\nare out in the streets; poor people, but they are\nenthusiastic and lively.\n\nNEW VIEW\n\nTraffic stops, as an ambulance speeds its way to a hospital;\nSIREN going.\n\nINT. THE SUITE - NIGHT\n\nBussetta delicately picks up a small satin cushion that had\nfallen from the couch, and replaces it as though nothing had\nhappened.  Slowly he cracks the door open which adjoins\nRoth's bedroom.  There is a slight commotion; whispered\nvoices.\n\nBUSSETTA'S VIEW\n\nTerri, Mrs. Roth, is crying.  A group of men lift Hyman\nRoth's frail body onto a stretcher.\n\nCLOSE ON BUSSETTA\n\nrealizes that this is the man he is to kill.\n\nCLOSER VIEW ON ROTH\n\nHe is alive; breathing hard with his mouth dry and open.\nThe doctor examines him, and then gives instructions to the\norderly who carries him out, presumably to the ambulance.\n\nBussetta closes the door on this VIEW.\n\nINT. PRESIDENTIAL PALACE - NIGHT\n\nAn orchestra plays for the guests, as an army of waiters\nserve champagne and hors d'oeuvres.  Michael relaxes with\nSenator Geary, Major Leon, and several of the Americans.\n\n\t\t\t\tQUESTADT\n\t\tThe embargo on arms shipments from\n\t\tthe U.S. to your government, was\n\t\tjust a necessary public relations\n\t\tmove... Only last month, your air\n\t\tforce received a major shipment of\n\t\trockets...\n\nMichael glances at his watch; Fredo concentrates on this.\n\n\t\t\t\tSENATOR GEARY\n\t\tWe believe in non-intervention...\n\t\tbut the agreement stipulates that\n\t\tour forces may be withdrawn... but\n\t\tas you've seen, we have not\n\t\twithdrawn them.\n\n\t\t\t\tCORNGOLD\n\t\tAnd my guess is that President\n\t\tEisenhower won't pull out while we\n\t\thave over three billion invested\n\t\tover here.\n\n\t\t\t\tMICHAEL\n\t\tFredo.  Where are you going?\n\n\t\t\t\tFREDO\n\t\tNowhere, Mike.  I wanted to get a\n\t\trefill.  How about you?\n\nEXT. HAVANA HOSPITAL - MED. VIEW - NIGHT\n\nThe ambulance makes its way up to the emergency section of\nthe hospital.  The orderlies quickly carry the old man\ninside.  His wife and the doctor, and several of his men,\nfollow in another car.\n\nTHE VIEW ALTERS\n\nand we see Bussetta waiting in the shadows.\n\nEXT. HAVANA STREETS - NIGHT\n\nThe growing crowds of Cubans begin their celebration.\n\nNEW VIEW\n\nA Cuban military detachment speeds along in the night,\nmotorcyclists clear a path through the celebrants.\n\nINT. PRESIDENTIAL PALACE - FULL VIEW - NIGHT\n\nA full sitdown dinner is being served the guests.  Michael\nsits at a table at dinner with several of the distinguished\nCubans, and some of the American businessmen.\n\n\t\t\t\tQUESTADT\n\t\tWhat's kept Mr. Roth?\n\nFredo looks up at Michael.\n\nIn the back of the room, we notice the detachment of military\nmoving quickly through the reception room on their way to\nthe President's private quarters.  Michael notices it as well.\n\nINT. THE HOSPITAL CORRIDOR - FULL VIEW - NIGHT\n\nThe activity at the end of the hall has come to rest; we can\ntell that the doctor tells Mrs. Roth that she should go, the\nold man will be taken to a room where he can rest.\nGradually, these people leave him in the care of the hospital\nstaff.\n\nBussetta watches from the distance of the hallway; after the\nold man has been moved, he quietly walks down the hallway to\nthe room.\n\nHIS VIEW\n\nA nurse sits in the room in attendance; Hyman Roth is\nasleep, his mouth wide open, breathing noisily.\n\nVIEW ON BUSSETTA\n\nhears footsteps, quickly steps away from the door, and into\nanother room.\n\nSome nurses and attendants speak to the nurse in the room in\nSpanish; one has brought a small bottle of wine, and\nobviously they are inviting the nurse to have a New Year's\ntoast with them.  They laugh; and the nurse steps away from\nthe room for a moment.\n\nBussetta moves slowly back into the room, alone with the\nhelpless Roth.\n\nINT. PRESIDENTIAL PALACE - FULL VIEW ON THE GUESTS - NIGHT\n\nseeing in the New Year; a great banner is hoisted up in\nSpanish, welcoming 1959.\n\nHands are shaken; kisses exchanged.\n\nMED. CLOSE VIEW\n\nMichael and Fredo in an embrace; they kiss one another.\n\n\t\t\t\tMICHAEL\n\t\tI've arranged for a plane; we're\n\t\tgoing to Miami in an hour.  Try not\n\t\tto make a big thing of it.\n\nHe kisses his brother once again.\n\n\t\t\t\tMICHAEL\n\t\t\t(Sicilian)\n\t\tI know it was you, Fredo.  You've\n\t\tbroken my heart.\n\nSlowly, understanding, Fredo backs away from his brother,\ntaking the kiss another way.\n\nA little distance away, Major Leon notices an old woman, one\nof the President's maids, moving across the alcove, carrying\nher suitcases.\n\n\t\t\t\tLEON\n\t\tWhat a pity; she's crying.  Must\n\t\thave been fired, and she's been\n\t\twith the President's family for\n\t\ttwenty years.\n\nEXT. HAVANA STREETS - NIGHT\n\nThe gathered crowd joyously welcomes the New Year.  We\nnotice the continual military movement.\n\nMED. VIEW\n\nA family surreptitiously leaves their home, carrying\nsuitcases and belongings.\n\nINT. ROTH'S HOSPITAL ROOM - NIGHT\n\nBussetta raises a hospital pillow, and easily begins to\nsmother the thin old man, who can barely struggle.\n\nOUT IN THE HALL\n\nA detachment of military move quickly, accompanied by some\nof Roth's men, as though they have important news that must\nbe dealt with.\n\nThey pass the small group of aides and nurses welcoming the\nNew Year.\n\nSeeing them, the nurse assigned to him, puts down her glass\nand moves quickly to the room.\n\nShe opens the door, and lays bare the sight of Bussetta\nsmothering Roth.  Bussetta turns quickly; and one of the\nmilitary takes out his pistol and shoots several times at\nhis head.\n\nINT. PRESIDENTIAL PALACE - NIGHT\n\nThe entire reception has been disrupted for an announcement;\nall the guests in their formal dress and evening gowns,\nstanding with frightened faces like first class passengers\non a doomed ship.  The President himself, his back to our\nVIEW, is making an announcement in Spanish.  While he\nspeaks, we notice continuous movement of his personal staff,\ncarrying suitcases and possessions.\n\n\t\t\t\tPRESIDENT\n\t\t...Because of serious setbacks of\n\t\tour troops in Guantanamo and\n\t\tSantiago, we feel reluctantly, that\n\t\twe must leave the Capital at once.\n\t\tMyself and my family must bid you\n\t\tgoodbye, and good fortune.  We will\n\t\tgo directly to Ciudad Trujillo.\n\nThe crowd is stunned; already whispers are moving throughout\nthe guests.\n\nThe only one who is not completely taken off guard is\nMichael, who quietly steps back, and disappears from the room.\n\n\t\t\t\tPRESIDENT\n\t\t...My only regret is that there\n\t\tcould not have been more warning...\n\t\tAs my last official act as\n\t\tPresident, I hereby appoint a\n\t\tprovisional government with Dr.\n\t\tCarlos M. Piedra, as its President.\n\nBy now, there is only one thought among the guests: how can\nthey get out, and with what.\n\nEXT. PRESIDENTIAL PALACE - NIGHT\n\nWe see evidence of the confusion at this late hour; already\ncars are beginning to move; people leaving the Palace in\nhaste.  Michael moves quickly toward his car.  He sees\nFredo, watching him in fear.\n\n\t\t\t\tMICHAEL\n\t\tCome with me.  It's your only way\n\t\tof getting out!\n\nVIEW ON FREDO\n\nTerrified of his brother, and what he knows; Fredo backs\naway into the growing noise and confusion of the crowd.\n\nVIEW ON MICHAEL\n\nFinally, he has to step into the car and it roars off.\n\nEXT. HAVANA STREETS - NIGHT\n\nRebel cars with loudspeakers have already picked up the news\nthat Batista has conceded...this throws the crowds already\ngathered for the New Year into cheers of joy.\n\nThey harass a wealthy family who are trying to get away in\ntheir car.\n\nThe people pull them out of the car, opening their suitcases,\nout of which spill piles of cash and jewelry into street.\n\nMichael's car makes its way as the crowd cheers: \"El animale\nse fue!\"\n\nEXT. THE UNITED STATES EMBASSY - MED. VIEW - NIGHT\n\nCrowds of panicked and frightened tourists, and Batistianos\nare trying to get to the safety of the Embassy with the\nfamilies and possessions.\n\nWe see Geary, and some of the Americans we had met, working\ntheir way through the crowds, shouting that they are\nAmericans in order to get preference on the line.  Often\nthat declaration brings 'boos' from the crowds.\n\nSometimes the joyous Cubans will let a family through, but\nagain, taking away the suitcases, rich leather, filled with\nmoney and valuables.  Money seems to be stuffed everywhere.\n\nEXT. THE YACHT CLUB - NIGHT\n\nAll forms of private transportation are jammed with people\ntrying to get out, holding cash in their hands for anyone\nwith a yacht or small boat to get them to Florida.\n\nA car pulls up; and we see Sam Roth, Terri Roth and some of\ntheir men, carry the sickly, but still alive Hyman Roth to a\nprivate cruiser which is protected by men with machine guns.\n\nWithin seconds, they are on their way to Miami.\n\nEXT. THE PRIVATE AIRPORT - NIGHT\n\nThings are no different at the airport; where anything that\ncan fly is being jammed with refugees and their money.\n\nA wealthy family is arguing with the pilot of a fast\nairplane; trying to force cash on him, and his family into\nthe plane.  The PILOT steadfastly refuses, although checking\nhis watch, as though his passengers are late.  He speaks\nonly English.\n\n\t\t\t\tPILOT\n\t\tNo, this is a private plane.  No,\n\t\tthis plane is taken.\n\nFinally Michael's Mercury pulls up, and Michael approaches\nthe Pilot.\n\n\t\t\t\tMICHAEL\n\t\tHe isn't here.\n\n\t\t\t\tPILOT\n\t\tWe've got to leave, they'll take\n\t\tthis thing apart.\n\n\t\t\t\tMICHAEL\n\t\tAll right.  Go now.\n\nThe Pilot lets Michael in, as the Cuban screams curses at\nthem, and begins searching for another plane for his family.\n\nINT. THE PLANE - VIEW ON THE PILOT - NIGHT\n\nas the propeller turns over.\n\nEXT. THE AIRPORT - FULL VIEW - NIGHT\n\nGroups of the cheering, celebrating Cubans sing\n\"Guantanamera,\" now as a song of triumph.\n\nINT. THE PLANE - MOVING VIEW - MICHAEL - NIGHT\n\nCloser to him, his personal and business life caught in the\nmiddle of history.\n\nEXT. NEW YORK STREET - MED. VIEW - DAY (1920)\n\nHe stops to pick out some choice oranges and peaches from a\nfruit stand.  Then he reaches into his pocket for change.\n\n\t\t\t\tVENDOR\n\t\tNo, no.  It is my pleasure to make\n\t\tthis a gift.\n\nCLOSE VIEW ON VITO\n\n\t\t\t\tVITO\n\t\tYou are kind.  If ever I can do\n\t\tsomething for you, in return,\n\t\tplease come to me.\n\nINT. VITO'S TENEMENT - DAY\n\nDespite his new position of 'respect,' there is little\nchanged about his home.  Only that they have lived there a\nwhile now, and the rooms are fuller with the inevitable\npossessions a young family acquires.\n\nHe kisses his wife, who seems a big apprehensive.  He shows\nher the fruit; and from her reaction knows she has something\non her mind.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tWhat is it?\n\n\t\t\t\tCARMELLA\n\t\t\t(Sicilian)\n\t\tCome...\n\nThey step into the tiny parlor, where we see an older woman,\nwaiting nervously.\n\n\t\t\t\tCARMELLA\n\t\tThe Signora is a friend of mine.\n\t\tShe has a favor to ask of you.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tWhy do you come to me?\n\n\t\t\t\tSIGNORA COLOMBO\n\t\t\t(Sicilian)\n\t\tShe told me to ask you.\n\nHe seems surprised; looks to his wife.\n\n\t\t\t\tCARMELLA\n\t\tShe is having some trouble.  Her\n\t\tlandlord has received complaints\n\t\tbecause of her dog.  He told her to\n\t\tget rid of it, but her boy loved\n\t\tit, so they tried to hide it.  When\n\t\tthe landlord found out, he was so\n\t\tangry, he ordered her to leave.\n\t\tEven if she truly will let the dog\n\t\tgo.\n\n\t\t\t\tSIGNORA COLOMBO\n\t\t\t(Sicilian)\n\t\tHe said he would have the police\n\t\tput us out.\n\n\t\t\t\tVITO\n\t\t\t(thoughtfully)\n\t\tI can give you some money to help\n\t\tyou move, is that what you want?\n\n\t\t\t\tSIGNORA COLOMBO\n\t\tMy friends are all here; how can I\n\t\tmove to another neighborhood with\n\t\tstrangers?  I want you to speak to\n\t\tthe landlord to let me stay.\n\nVito nods to the frightened old woman.\n\n\t\t\t\tVITO\n\t\tIt's done then.  You won't have to\n\t\tmove; I'll speak to him tomorrow\n\t\tmorning.\n\nCarmella breaks into a smile; which her husband des not\nacknowledge.\n\nThe old woman starts to leave the room; but she is not\nconvinced.\n\n\t\t\t\tSIGNORA COLOMBO\n\t\tYou're sure he'll say yes, the\n\t\tlandlord?\n\n\t\t\t\tVITO\n\t\tI'm sure he's a good-hearted fellow.\n\t\tOnce I explain how things are with\n\t\tyou, I'm sure he'll take pity on\n\t\tyour misfortunes.  Don't let it\n\t\ttrouble you any more.\n\t\t\t(as he shows her out)\n\t\tGuard your health, for the sake of\n\t\tyour children.\n\nEXT. TENEMENT BLOCK - DAY\n\nSIGNOR ROBERTO, a pompous, rather well-dressed Patrone\nangrily walks down the steps of one of his tenement buildings.\n\nHe carries a check list, and makes marks with a pencil\nconcerning the condition of his various buildings; a broken\nwindow here, some missing tile there.  He bends over to pick\nup some garbage left by a thoughtless tenant, muttering to\nhimself, when he sees the shoes and legs of a young worker.\n\n\t\t\t\tVITO (O.S.)\n\t\tSignore Roberto...\n\nHe rises to be face to face with a polite Vito Corleone.\n\n\t\t\t\tVITO\n\t\tThe friend of my wife, a poor widow\n\t\twith no man to protect her, tells\n\t\tme that for some reason she has\n\t\tbeen ordered to move from your\n\t\tbuilding.  She is in despair.  She\n\t\thas no money, she has no friends\n\t\texcept those that live here.\n\nSignor Roberto brusquely answers, and continues on his way.\n\n\t\t\t\tROBERTO\n\t\tI have already rented the apartment\n\t\tto another family.\n\nMOVING SHOT ON THE TWO\n\n\t\t\t\tVITO\n\t\tI told her I would speak to you,\n\t\tthat you are a reasonable man who\n\t\tacted out of some misunderstanding.\n\t\tShe has gotten rid of the animal\n\t\tthat caused all the trouble, so why\n\t\tshouldn't she stay.  As one Italian\n\t\tto another, I ask you the favor.\n\n\t\t\t\tROBERTO\n\t\tI've already rented it; I cannot\n\t\tdisappoint the new tenants.  They're\n\t\tpaying a higher rent.\n\n\t\t\t\tVITO\n\t\tHow much more a month?\n\n\t\t\t\tROBERTO\n\t\tEh...\n\t\t\t(we sense he is lying)\n\t\tFive dollars more.\n\nVito reaches into his pocket, and takes out a roll of bills.\n\n\t\t\t\tVITO\n\t\tHere is the six month's increase in\n\t\tadvance.  You needn't speak to her\n\t\tabout it, she's a proud woman.  See\n\t\tme again in another six months.\n\t\tBut of course, you'll let her keep\n\t\ther dog.\n\n\t\t\t\tROBERTO\n\t\tLike hell!  And who the hell are\n\t\tyou to give me orders.  Watch your\n\t\tmanners or you'll be on your\n\t\tSicilian ass in the street there.\n\nVito raises his hands in surprise; his voice is reasonable.\n\n\t\t\t\tVITO\n\t\tI'm asking you a favor, only that.\n\t\tOne never knows when one might need\n\t\ta friend, isn't that true?  Here,\n\t\ttake this money as a sign of my\n\t\tgood-will, and make your own\n\t\tdecision.  I won't quarrel with it.\n\t\t\t(he puts the money in\n\t\t\tRoberto's hand)\n\t\tDo me this little favor, just take\n\t\tit and think carefully.  Tomorrow\n\t\tmorning if you want to give me the\n\t\tmoney back, by all means do so.  If\n\t\tyou want the woman out of your\n\t\thouse, how can I stop you?  It's\n\t\tyour property, after all.  If you\n\t\tdon't want the dog in there, I can\n\t\tunderstand.  I dislike dogs myself.\n\t\t\t(he pats Roberto on\n\t\t\tthe shoulder)\n\t\tDo me this service, eh?  I won't\n\t\tforget it.  Ask your friends in\n\t\tthis neighborhood about me, they'll\n\t\ttell you I'm a man who believes in\n\t\tshowing his gratitude.\n\nWithout a word more, Vito leaves a hypnotized Roberto\nstanding in front of the tenement, his hand clasping the\nmoney.\n\nEXT. NEIGHBORHOOD STREET - DAY\n\nA thin young man, almost gawky, walks down the street in\nthis Italian neighborhood, his name is HYMAN SUCHOWSKY.\n\nHe carries his tools as he comes home from work.  He is\npursued and tormented by a couple of Italian youths, about\nhis own age, eighteen.\n\n\t\t\t\tITALIAN BOY\n\t\tKid, where do you live?\n\n\t\t\t\tANOTHER\n\t\tWhere'd you get those nigger lips?\n\nHe tries not to be intimidated; finally one of the boys,\nsteps in front of him and stops him.\n\n\t\t\t\tITALIAN BOY\n\t\tSay 'bread' in Italian.\n\n\t\t\t\tANOTHER\n\t\tHe dunno.\n\n\t\t\t\tITALIAN BOY\n\t\tGo on; how do you say 'bread' in\n\t\tItalian?  If you're from the\n\t\tneighborhood, you should know how\n\t\tto say 'bread' in Italian.\n\nAn amused Peter Clemenza steps forward from a local coffee\nhouse, to preside over the fuss.  He's a 'big' man in the\nneighborhood, and loves a fight.\n\n\t\t\t\tCLEMENZA\n\t\tWhat's up?\n\n\t\t\t\tITALIAN BOY\n\t\tThis kid lives around here, but he\n\t\tcan't say bread in Italian.\n\n\t\t\t\tCLEMENZA\n\t\tThat's 'cause he's Jew.  Look at\n\t\tthose pregnant lips!\n\nHe giggles at his own joke.\n\n\t\t\t\tITALIAN BOY\n\t\tAre you a Jewboy?\n\nThe boy doesn't answer, tries to keep going.\n\n\t\t\t\tITALIAN BOY\n\t\tWell, if you're not a Jew, say\n\t\t'bread' in Italian.  See, he can't.\n\nAnd with that, he rounds a blow squarely to the boy's face,\nsending him sprawling to the cement, his tools flying with a\nclatter.\n\nThe other Italian immediately joins in with a few kicks to\nthe boy's stomach.  Hyman tries to fight back; grabs a hold\nof his tormentor's foot, and brings him down on the cement\nas well.  For a moment, they are rolling around on the\nsidewalk, two against one, Hyman taking the worst of it.\n\n\t\t\t\tCLEMENZA\n\t\tAlright, alright, cut it out.\n\n\t\t\t\tSECOND ITALIAN\n\t\tWhat for?  He killed Jesus Christ!\n\nClemenza pulls him off, and kicks him in the ass.\n\n\t\t\t\tCLEMENZA\n\t\tI said cut it out!\n\t\t\t(to the beaten kid)\n\t\tWhat's your name?\n\n\t\t\t\tHYMAN\n\t\tHyman Suchowsky.\n\n\t\t\t\tITALIAN BOY\n\t\tI don't believe it.  In our\n\t\tneighborhood, with a name like that!\n\n\t\t\t\tCLEMENZA\n\t\tWhat are those tools?  You work on\n\t\tcars?\n\n\t\t\t\tHYMAN\n\t\tYeah.\n\n\t\t\t\tCLEMENZA\n\t\tMaybe I know how you can make a\n\t\tcouple of extra bucks working as a\n\t\tmechanic.\n\nThe boy seems agreeable.\n\n\t\t\t\tCLEMENZA\n\t\tBut you gotta know how to keep your\n\t\tmouth shut, and fer Chrissakes, get\n\t\trid of that name.  I'll call you\n\t\tJohnny Lips.\n\t\t\t(he giggles at his\n\t\t\town humor again)\n\t\tCome on...\n\nHe leads the boy down the street, whispering to him, on the\nside:\n\n\t\t\t\tCLEMENZA\n\t\tBread in Italian is pane.  P-A-N-E,\n\t\tpane.  Don't forget.\n\nINT. NEW GENCO WAREHOUSE - DAY\n\nA newly acquired warehouse, stocked with cases of the new\nproduct \"GENCO PURA\" olive oil.  It is the beginning of a\nnew business, in the American tradition.  Now they have one\nrattling old truck, and a few stock boys.\n\nGenco has become the accountant-business manager, based on\nthe experience working with his father.  But it is clear,\nthat Vito is the leader, and undisputed 'President' of the\nnew enterprise.\n\nGenco moves through the darkness of the warehouse, to the\nsmall divided area that Vito uses as his office.\n\n\t\t\t\tGENCO\n\t\t\t(Sicilian)\n\t\tThe 'patrone' is here.\n\n\t\t\t\tVITO\n\t\tChi?\n\n\t\t\t\tGENCO\n\t\tRoberto.  Who owns the 'rat-holes.'\n\nVito nods that he will see him; and soon Roberto enters, on\ntiptoe, his hat in his hand, and in a apologetic voice.\n\n\t\t\t\tROBERTO\n\t\tExcuse me, I hope I am not a\n\t\tdisturbance, Don Corleone.\n\n\t\t\t\tVITO\n\t\tYes.\n\n\t\t\t\tROBERTO\n\t\tWhat a terrible misunderstanding.\n\t\tOf course, Signora Colombo can stay\n\t\tin the flat.  Who were those\n\t\tmiserable tenants to complain about\n\t\tnoise from a poor animal...when\n\t\tthey pay such low rent.\n\nThen abruptly, he puts the roll of money on Vito's table,\nand steps back a respectful distance.\n\n\t\t\t\tROBERTO\n\t\tYour good heart in helping the poor\n\t\twidow has shamed me, and I want to\n\t\tshow that I, too, have some\n\t\tChristian charity.  Her rent will\n\t\tremain what it was.\n\n\t\t\t\tVITO\n\t\tWhat was that?\n\n\t\t\t\tROBERTO\n\t\tIn fact, reduced, bu five dollars!\n\nVito embraces him warmly.\n\n\t\t\t\tVITO\n\t\tI accept your generosity...\n\n\t\t\t\tROBERTO\n\t\tI won't keep you another minute...\n\nHe quickly takes his leave, bowing several times, and then\nmakes it back to the safety of the warehouse; he sighs,\ndeflates his lungs, and mops his brow; his bones have turned\ntoo jelly with fear at his narrow escape.  He all but runs\nout of the warehouse.\n\nGenco laughs as he watches.\n\n\t\t\t\tGENCO\n\t\tWe won't see him for weeks!  He'll\n\t\tstay in bed in the Bronx!\n\nClemenza has been waiting with his new mechanic.  We notice\nthe subtle difference in the way he treats Vito.  He is no\nlonger a junior apprentice in their petty crimes; but an\nimposing leader.\n\n\t\t\t\tCLEMENZA\n\t\tThis kid is good with cars; he\n\t\tkiijed at the truck, and says he\n\t\tcan keep it going.\n\nVito looks over the lanky young man.\n\n\t\t\t\tCLEMENZA\n\t\tWhat's your name?\n\n\t\t\t\tHYMAN\n\t\tSuchowsky.  Hyman Suchowsky.\n\n\t\t\t\tCLEMENZA\n\t\tHe's gonna dump that; I call him\n\t\tJohnny Lips.\n\n\t\t\t\tVITO\n\t\tWho is the greatest man you can\n\t\tthink of?\n\n\t\t\t\tCLEMENZA\n\t\tGo on, answer him when he talks to\n\t\tyou.  Tell him: Columbus, Marconi...\n\t\tGaribaldi.\n\n\t\t\t\tHYMAN\n\t\tArnold Rothstein.\n\n\t\t\t\tVITO\n\t\tThen take that as your name: Hyman\n\t\tRothstein.\n\nGenco is out in the alley; he calls out with glee.\n\n\t\t\t\tGENCO\n\t\tVitone!  Look at this!\n\nVito moves out to the smiling Genco; Clemenza and the newly\nchristened Hyman Rothstein follow a distance behind.\n\nEXT. THE ALLEY - DAY\n\nGenco stands beaming, as two workers raise up high, the\nfreshly painted sign: \"GENCO OLIVE OIL COMPANY.\"\n\n\t\t\t\tGENCO\n\t\t\t(enthusiastically)\n\t\tGod bless America!  We're in\n\t\tbusiness!\n\nThe young men watch as the sign is hoisted into place.  OUR\nVIEW goes from one to the other: Clemenza, Genco, Vito and\nHyman Rothstein.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nINT. SENATE CAUCUS ROOM - MED. CLOSE VIEW - DAY\n\nWilly Cicci, Pentangeli's associate and bodyguard takes a\ndrink of water.\n\n\t\t\t\tSENATOR (O.S.)\n\t\tMr. Cicci.  From the year 1927 to\n\t\tthe present time, you were an\n\t\temployee of the \"Genco Olive Oil\n\t\tCompany.\"\n\n\t\t\t\tCICCI\n\t\tThat's right.\n\n\t\t\t\tSENATOR (O.S.)\n\t\tBut in actuality, you were a member\n\t\tof the Corleone Crime organization.\n\n\t\t\t\tCICCI\n\t\tThe Corleone Family, Senator.  We\n\t\tcalled it, \"The Family.\"\n\n\t\t\t\tSENATOR (O.S.)\n\t\tWhat position did you occupy?\n\n\t\t\t\tCICCI\n\t\tAt first, like everybody, I was a\n\t\tsoldier.\n\nVIEW ON SENATOR KANE\n\nA thin, angular Baptist with a Mid-Western accent.\n\n\t\t\t\tSENATOR KANE\n\t\tWhat is that exactly?\n\n\t\t\t\tCICCI\n\t\tA button.  You know, Senator.\n\n\t\t\t\tSENATOR KANE\n\t\tNo, I don't know, explain that\n\t\texactly.\n\n\t\t\t\tCICCI\n\t\tWhen the boss says push the button\n\t\ton a guy, I push the button, see,\n\t\tSenator?\n\nThe Senators treat Cicci with a surface courtesy, as if he\nwere a curious kind of animal, not really human.  Cicci\nreacts to this by being even more brutally forthright than\nhe has to be, to show his contempt for what he considers a\nhypocrisy.\n\nThe VIEW ALTERS from Senator Kane to the Committee's\nattorney, Mr. Questadt.\n\n\t\t\t\tQUESTADT\n\t\tYou mean you killed people at the\n\t\tbehest of your superiors?\n\n\t\t\t\tCICCI\n\t\tThat's right, counsellor.\n\n\t\t\t\tQUESTADT\n\t\tAnd the head of your family was\n\t\tMichael Corleone.\n\n\t\t\t\tCICCI\n\t\tYeah, counsellor, Michael Corleone.\n\n\t\t\t\tSENATOR KANE\n\t\tDid you ever get such an order\n\t\tdirectly from Michael Corleone?\n\n\t\t\t\tCICCI\n\t\tNo, Senator, I never talked to him.\n\n\t\t\t\tSENATOR SAVOY\n\t\t\t(very autocratic,\n\t\t\tdeep South,\n\t\t\tgentlemanly man)\n\t\tThere was always a buffer, someone\n\t\tin between you who gave you orders.\n\n\t\t\t\tCICCI\n\t\tYeah, a buffer, the Family had a\n\t\tlot of buffers.\n\nEXT. THE TROPICANA IN VEGAS - MED. VIEW - DAY\n\nA limousine pulls up at a private area near the side of the\nhotel.  Michael exits the limousine followed by Hagen and\nNeri.\n\n\t\t\t\tMICHAEL\n\t\tDo you think they have somebody to\n\t\tback up Cicci?\n\n\t\t\t\tHAGEN\n\t\tNo.  But if they do have somebody,\n\t\tyou'll do three years for perjury\n\t\tif you give them so much as a wrong\n\t\tmiddle name.\n\nMichael smiles to him, but it's a cold, deadly smile.\n\n\t\t\t\tHAGEN\n\t\tMichael, take the Fifth all the\n\t\tway, that way you can't get into\n\t\ttrouble.\n\nEXT. PRIVATE BALCONY OF CORLEONE APARTMENT AT TROPICANA - DAY\n\nA Corleone bodyguard waits outside on the balcony overlooking\nthe pool area.  Through the translucent draperies, we see a\ngrouping of me.\n\nINT. CORLEONE APARTMENT AT THE TROPICANA - DAY\n\nMichael, Hagen, Neri and Rocco are seated in this luxury in\nthe hotel.  Michael sits in a comfortable chair in his\napartment.  Neri comes and brings him a drink without\nasking, but Michael refuses it.\n\n\t\t\t\tMICHAEL\n\t\tAl, get me a wet towel.  Does Kay\n\t\tknow I'm back?\n\nHagen nods.\n\n\t\t\t\tMICHAEL\n\t\tDid the boy get something from me\n\t\tfor Christmas?\n\n\t\t\t\tHAGEN\n\t\tI took care of it.\n\n\t\t\t\tMICHAEL\n\t\tWhat was it, so I'll know.\n\n\t\t\t\tHAGEN\n\t\tA little care he can ride in with\n\t\tan electric motor.\n\nNeri comes around with a wet face towel, which Michael uses\nto cool his eyes.  He puts the used towel down on the table.\n\n\t\t\t\tMICHAEL\n\t\tFellas, can you wait outside a\n\t\tminute?\n\nThey know what he means and leave the apartment, going out\nto the balcony where we can see them but they cannot hear.\nOnly Hagen remains.\n\n\t\t\t\tMICHAEL\n\t\tWhere's my brother?\n\n\t\t\t\tHAGEN\n\t\tRoth got out on a private boat.\n\t\tHe's in a hospital in Miami.  Had a\n\t\tstroke but he's recovered okay.\n\t\tBussetta's dead.\n\n\t\t\t\tMICHAEL\n\t\tI asked about Fredo?\n\n\t\t\t\tHAGEN\n\t\tThe new government arrested him,\n\t\theld him for a couple of days with\n\t\ta lot of the other casino people,\n\t\tincluding Roth's brother, Sam.  The\n\t\tAmerican Embassy arranged flights\n\t\tfor citizens; I'm not sure, but I\n\t\tthink he's somewhere in New York.\n\n\t\t\t\tMICHAEL\n\t\tI want you to reach Fredo.  I know\n\t\the's scared, but have one of our\n\t\tpeople reach him.  Assure him that\n\t\tthere will be no reprisals.  Tell\n\t\thim that I know Roth misled him.\n\n\t\t\t\tHAGEN\n\t\tMy information is that Fredo\n\t\tthought it was a kidnapping.  Roth\n\t\tassured him nothing would happen to\n\t\tyou.\n\n\t\t\t\tMICHAEL\n\t\t\t(indicating Rocco and\n\t\t\tNeri on the balcony)\n\t\tThey can come in now.\n\n\t\t\t\tHAGEN\n\t\tWait... there's something else.\n\n\t\t\t\tMICHAEL\n\t\tAlright.\n\nHagen pauses; doesn't know how to begin.\n\n\t\t\t\tMICHAEL\n\t\t\t(impatiently)\n\t\tGo on, tell me.\n\n\t\t\t\tHAGEN\n\t\tKay had a miscarriage; she lost the\n\t\tbaby.\n\nAfter a moment:\n\n\t\t\t\tMICHAEL\n\t\tWas it a boy or a girl?\n\n\t\t\t\tHAGEN\n\t\tMike, at three and a half...\n\n\t\t\t\tMICHAEL\n\t\tWhat is it, can't you give me\n\t\tstraight answers anymore!\n\n\t\t\t\tHAGEN\n\t\tIt was a boy.\n\n\t\t\t\tMICHAEL\n\t\tAnd Kay...she's all right?\n\n\t\t\t\tHAGEN\n\t\tShe took the Senate Investigation\n\t\tworse.\n\n\t\t\t\tMICHAEL\n\t\tDoes she blame it on me?  The baby?\n\n\t\t\t\tHAGEN\n\t\tI don't know.\n\nEXT. TAHOE ESTATE - DAY\n\nThe first snow of the New Year has fallen; the trees are\nbare, and there is hush all over this part of the Sierras.\nMichael is driven in his car, looking out at the familiar\nsight of the home he has been forced to be away from.\n\nVIEW ON MICHAEL\n\nlooking out from his window.  The last time he had seen the\nestate it was warm, and the trees were full.\n\nMOVING VIEW\n\napproaching the great stone gates; closed.  The bodyguards\nare not readily visible, but they are there.  The iron gates\nare opened, and one of the men makes a simple nod of respect,\nas the car pulls in.\n\nNEW VIEW\n\nInside the estate, the private roads have been freshly\nplowed, and occasionally a worker will pause to watch the\ncar as it passes.\n\nThe Grandchildren are in school now, and so the estate is\nespecially quiet.  Although there are signs that children\nlive here; a bicycle, a sled, a swing and gymnastic set, wet\nand with a rim of snow still on it.\n\nINT. MICHAEL'S HOUSE - VIEW FROM INSIDE THE HOUSE - DAY\n\nto the outside, where Michael walks slowly.  He stops and\nlooks at a little Italian red sportscar made for children.\n\nNEW VIEW\n\nThe front door opens, and Michael enters his own home.  It\nis very quiet, no one is at home to greet him.  He can see\nthe evidence of his family; things his wife and his children\nhave been using, and left on a sofa or a table.\n\nHe moves toward his and Kay's bedroom, where we can HEAR the\nSOUND of a sewing machine running.\n\nQuietly he opens the door.\n\nMICHAEL'S VIEW\n\ninto the bedroom.  Kay is sitting by the window, lit by the\ncold afternoon light, at work with her sewing machine.  She\nhasn't noticed that he's in the room yet, and goes on with\nher work.\n\nVIEW ON MICHAEL\n\nstands there a moment, watching, not making a sound.  And\nthen without a word, he steps back, and closes the door, so\nthat she doesn't see him.\n\nVIEW FROM INSIDE THE HOUSE\n\nonto Michael, moving outside, walking through the snow, he\nmoves to the house next to his own.\n\nINT. CONNIE'S HOUSE - DAY\n\nThis is the house where Mama lives with Connie's children,\nConnie so rarely is there.\n\nHe steps in; his mother is asleep in a chair in the living\nroom.  He moves to her, and bends low, whispers.\n\n\t\t\t\tMICHAEL\n\t\tMom... Mom...\n\nShe opens her eyes, which are red and small with age.\n\n\t\t\t\tMICHAEL\n\t\t\t(Sicilian)\n\t\tIt's Michael.  How are you, Mom?\n\n\t\t\t\tMAMA\n\t\t\t(Sicilian)\n\t\tI'm alright.  Will you stay home\n\t\tfor awhile?\n\n\t\t\t\tMICHAEL\n\t\t\t(Sicilian)\n\t\tThere are still things I have to do.\n\n\t\t\t\tMAMA\n\t\t\t(Sicilian)\n\t\tWell, we can all have a nice dinner\n\t\ttogether tonight.  How are your eyes?\n\n\t\t\t\tMICHAEL\n\t\tAlright.  They bother me once in\n\t\tawhile.\n\t\t\t(a pause as he thinks)\n\t\tTell me, when Pop had troubles...\n\t\tdid he ever think, even to himself,\n\t\tthat he had gone the wrong way;\n\t\tthat maybe by trying to be strong\n\t\tand trying to protect his family,\n\t\tthat he could... that he could...\n\t\tlose it instead?\n\n\t\t\t\tMAMA\n\t\t\t(Sicilian)\n\t\tYou talk about the baby.  She can\n\t\thave another baby.\n\n\t\t\t\tMICHAEL\n\t\t\t(Sicilian)\n\t\tNo, I meant lose his family.\n\n\t\t\t\tMAMA\n\t\t\t(as best she ever\n\t\t\tunderstood it)\n\t\tYour family?  How can you ever lose\n\t\tyour family?\n\n\t\t\t\tMICHAEL\n\t\t\t(almost to himself)\n\t\tBut times are different...\n\nFULL VIEW IN ROOM - MICHAEL AND HIS MOTHER\n\nQuietly we HEAR the music of a small band playing an Italian\nmarch.  From the orchestration, we know it is from the past.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nEXT. TRAIN STATION AT CORLEONE - DAY\n\nVitone and his young family: Mama, Santino, Fredo and the\nbaby Michael are met at the small station in Sicily by\nfriends, and Mama's relatives.  There is a small band,\nplaying for the occasion.  A small man has brought a motor\ncar to pick the family up; and there are certain dark men,\nwith shotguns slung over their shoulders to preside over the\noccasion.\n\nThe family is helped into the car; the luggage is packed on\nthe roof, and the car drives off.  The second car, with\nbodyguards following.\n\nEXT. DON TOMASINO'S VILLA OUTSIDE OF CORLEONE - DAY\n\nThe villa is bloomed with flowers and DON TOMASINO at this\npoint is a man in his late twenties.  He embraces Vitone and\npats the heads of his children, and leads them all into the\ngarden.\n\nINT. THE VILLA - SUMPTUOUS MED. VIEW - LATE DAY\n\nA sumptuous table is set for the visiting family from\nAmerica.  There is a warm atmosphere as Vito, his wife and\nchildren eat.  Tomasino and his family received presents\nfrom Carmella and to Tomasino's mother, and gifts are given\nto all of the children.\n\nAll typically American representing some of the prosperity\nand interests in the consumer goods that followed a great war.\n\nEXT. CORLEONE PLAZA - DAY\n\nThe family exits the church on the plaza of the town.  Vito\nshakes hands warmly with the priest.\n\nINT. VILLAGE COTTAGE - NIGHT\n\nThe door is open -- the footsteps of a man enter the room.\nWe follow these footsteps without quite knowing to whom they\nbelong.  They lead us to a bed, where we see asleep an OLD\nMAN.  He sleeps in his undershirt and is sweating, covered\nby mosquito netting.\n\nVIEW ALTERS\n\nand we realize that it is young Vito looking at the MAN.\n\nWe remember that the man is MOSCA, one of three men, who\nalmost twenty years before had hunted down Vito when he was\na boy.  With lightning speed, Vito slashes through the\nmosquito netting with a knife.  And with the movement\nprecise as a butcher's he disembowls this man.\n\nEXT. OLIVE OIL WAREHOUSE - FULL VIEW\n\nVito has brought his wife and children to see the Olive Oil\nDepot which is the link to his New York importing business.\nThey go inside.\n\nINT. OLIVE OIL WAREHOUSE - DAY\n\nThey are led by one of Vito's associates through rows and\nrows of large vats of olive oil.  Vito very proudly shows\nhis associates in Italy the olive oil can that will be used\nin the United States.  They all stand around at the link to\ntheir new importing business and share a toast of wine.\n\nEXT. THE BAY - DAY\n\nA team of Sicilian fisherman are at work mending their nets.\nOne sings accompanied by a guitar.\n\nVIEW MOVES TO ONE OF THE OLD FISHERMAN\n\nHe is recognized as the second of the men who had hunted\nVito down.  STROLLO.  As he walks we notice there is a\nfigure that is moving through the drying sails and barrels,\nit is Vito.  He moves quietly, stepping up behind the old\nman.  In an instant, he has thrown a garrote around his\nthroat, twisting it tight, so that there is very little\nsound.\n\nThen, almost silently dragging him through the space hidden\nby the drying sails.\n\nEXT. THE IMPRESSIVE ESTATE OF DON FRANCESCO - DAY\n\nWe see an old car approach.  Its driver is the young Tomasino.\nSitting in the car with him is Vito.\n\nThe car stops at the gates, and an old guard sees and\nrecognizes Tomasino, opens the gates allowing them to enter.\n\nMED. VIEW\n\non an almost decrepit DON FRANCESCO.  He must be in his\nearly nineties, sitting as powerful and as impressive as\never, in his throne-like chair from which he manages the\npower as the Mafia Chieftan of this village.  Young Don\nTomasino is speaking.\n\nWe notice in a little distance in the rear, there are some\nyounger shepherds with shotguns thrown over their shoulders.\n\n\t\t\t\tTOMASINO\n\t\t\t(Sicilian)\n\t\tDon Francesco, if you will honor\n\t\tme, by allowing me to introduce my\n\t\tassociate in America, in New York.\n\t\tHis name is Vito Corleone.\n\nThe old man and his eyes glance up at a notion of a man who\nhas taken the name of this town as his name.\n\n\t\t\t\tTOMASINO\n\t\tWe will supply him with olive oil\n\t\texclusively in the town of Corleone.\n\t\tHis company is called the \"Genco\n\t\tOlive Oil Company.\" Here we have\n\t\tbrought you an indication of how he\n\t\twill sell the product.\n\nTomasino respectfully puts a can of olive oil where the old\nman can look at it.  The old man nods, accepting the notion\nof this business.\n\n\t\t\t\tTOMASINO\n\t\t\t(Sicilian)\n\t\tWe have come to ask your blessing\n\t\tand permission to continue this\n\t\tenterprise.\n\n\t\t\t\tDON FRANCESCO\n\t\t\t(Sicilian)\n\t\t\t (in a shrill, high,\n\t\t\traspy voice)\n\t\tWhere is this young man?\n\n\t\t\t\tTOMASINO\n\t\tHe is right here, standing next to\n\t\tme, Don Francesco.\n\n\t\t\t\tDON FRANCESCO\n\t\t\t(Sicilian)\n\t\tHave him come closer, I can't see\n\t\tvery well.\n\nVito takes those several steps, so that he is standing right\nin front of the old man.\n\nVIEW ON DON FRANCESCO\n\nlooking up, squinting against the sun.\n\nDON FRANCESCO'S VIEW\n\nStrangely backlit, almost blurry image of the young man from\nAmerica.\n\n\t\t\t\tDON FRANCESCO\n\t\t\t(Sicilian)\n\t\tWhat is your name?\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tVito Corleone.\n\n\t\t\t\tDON FRANCESCO\n\t\t\t(Sicilian)\n\t\tYou took the name of this town, eh?\n\t\tWhat was your father's name?\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tAntonio Andolini.\n\nCLOSE VIEW ON THE OLD MAN\n\nThe recognition of the name throws a shudder through him.\nIt is as though he recognizes that this is the boy; the son\nof his old enemy, whom he had killed, and whose sons he had\ntried to wipe out.  The old man raises his feeble hands\nsignalling his guard, and in his weak voice, he shouts:\n\n\t\t\t\tDON FRANCESCO\n\t\t\t(Sicilian)\n\t\tKill him!  Kill him!\n\nBut he is too late; Vito steps forward.\n\n\t\t\t\tVITO\n\t\t\t(Sicilian)\n\t\tIn the name of my Father, and my\n\t\tBrother...\n\nAnd uses the knife, ritualistically plunging it into the old\nman's belly, and then up to his throat, which is severed.\n\nVIEW ON TOMASINO\n\nhas drawn his pistol and quickly shoots one of the guards,\nhelping Vito to escape back into the motor car.\n\nVIEW ON A GUARD\n\nraising his shotgun.\n\nVIEW ON THE MOTOR CAR\n\nJust as Tomasino is about to get into the car, the shotgun\nis fired, and he is hit in the legs.\n\nVito manages to pull him up into the car, and they make\ntheir escape.\n\nEXT. RAILROAD STATION IN CORLEONE - DAY\n\nSome of the townspeople have come bringing flowers and gifts\nfor Vito and his family.\n\nHis wife is radiant with the flowers given her.\n\nThe train has arrived and the crowd shout \"Ciao, come back\nsoon.\"\n\nTHE VIEW ALTERS\n\nrevealing his good friend Tomasino, waving from his\nwheelchair.\n\nVIEW ON VITO\n\nand his wife.  She holds up the baby Michael, and helps him\nwave his hand.\n\nINT. SENATE CAUCUS ROOM - MED. CLOSE VIEW ON MICHAEL - DAY\n\n\t\t\t\tSENATOR KANE (O.S.)\n\t\tAre you the son of Vito Corleone?\n\n\t\t\t\tMICHAEL\n\t\tYes.\n\n\t\t\t\tSENATOR KING\n\t\tDid he use at times an alias?  Was\n\t\tthis alias in certain circles\n\t\tGODFATHER?\n\n\t\t\t\tMICHAEL\n\t\tIt was not an alias.  GODFATHER was\n\t\ta term of affection, used by his\n\t\tfriends, one of respect.\n\n\t\t\t\tSENATOR WEEKLER\n\t\t\t(Senator from New\n\t\t\tYork, very smooth,\n\t\t\tpartly liberal,\n\t\t\tTammany Hall)\n\t\tLet me agree with that.  Many of my\n\t\tconstituents are Italian and have\n\t\tbeen honored with that certain\n\t\tfriendship by my close Italian\n\t\tfriends.  Up to this point before I\n\t\thave to leave this hearing to join\n\t\tmy own committee, let me say, that\n\t\tthis hearing on the Mafia is in no\n\t\tway a slur on the Italians by the\n\t\tSenate; nor is it meant to be; nor\n\t\twill I allow it to be.  Italian\n\t\tAmericans are the hardest working,\n\t\tmost law abiding patriotic Americans\n\t\tof our country.  It is a shame and\n\t\ta pity that a few rotten apples\n\t\tgive them a bad name.  We are here\n\t\tto weed those rotten apples out of\n\t\tthe vast healthy barrel of Italian\n\t\tAmericans, who are one of the\n\t\tbackbones of our country.\n\nThere is a pause for a while, while the New York Senator\nposes for the TV cameras and leaves the hearing so that he\nwill not be associated with hearing the rough stuff.\n\n\t\t\t\tSENATOR KANE\n\t\tI'm sure we all agree with our\n\t\testeemed colleague.  Now, Mr.\n\t\tCorleone, you have been advised as\n\t\tto your legal rights.  We have had\n\t\ttestimony from a preceding witness\n\t\twho states you are head of the most\n\t\tpowerful Mafia family in this\n\t\tcountry.  Are you?\n\n\t\t\t\tMICHAEL\n\t\tNo.\n\n\t\t\t\tSENATOR KANE\n\t\tThis witness has testified that you\n\t\tare personally responsible for the\n\t\tmurder of a New York Police Captain\n\t\tin the year 1947 and with him a man\n\t\tnamed Virgil Sollozzo.  Do you deny\n\t\tthis?\n\n\t\t\t\tMICHAEL\n\t\tI deny his every charge.\n\n\t\t\t\tSENATOR KANE\n\t\tIs it true that in the year 1950\n\t\tyou devised the murder of the heads\n\t\tof the Five Families in New York,\n\t\tto assume and consolidate your\n\t\tnefarious power?\n\n\t\t\t\tMICHAEL\n\t\tThat is a complete falsehood.\n\n\t\t\t\tSENATOR KANE\n\t\tIs it true that you own a\n\t\tcontrolling interest in three of\n\t\tthe major hotels in Las Vegas?\n\n\t\t\t\tMICHAEL\n\t\tThat is not true.  I own some stock\n\t\tin some of the hotels, but only\n\t\tvery small amounts.  I also own\n\t\tsome American Telephone and IBM\n\t\tstock.\n\nMichael had checked this point with Hagen, before answering,\nand then once again after the answer.\n\n\t\t\t\tSENATOR ROGERS\n\t\tWhy is it necessary for your\n\t\tcounsel to advise you on that\n\t\tquestion?\n\n\t\t\t\tMICHAEL\n\t\tSenator, I've observed the head of\n\t\tGeneral Motors before a Senate\n\t\tCommittee, and his lawyer whispered\n\t\tin his ear.  That was not commented\n\t\tupon in the way you have just done.\n\n\t\t\t\tSENATOR KANE\n\t\tMr. Corleone, do you have any hotel\n\t\tinterests in the state of Arizona?\n\t\tOr any gambling interests in that\n\t\tstate?\n\n\t\t\t\tMICHAEL\n\t\tI do not.\n\n\t\t\t\tSENATOR KANE\n\t\tDo you have interests or control\n\t\tover gambling and narcotics in the\n\t\tstate of New York.\n\n\t\t\t\tMICHAEL\n\t\tI do not.\n\nA pause.  Silence, as the Chairman whispers something to his\nassistant.\n\nTom Hagen takes a paper out of his briefcase, and addresses\nthe Chair.\n\n\t\t\t\tHAGEN\n\t\tSenator, my client would like to\n\t\tread a statement for the record.\n\n\t\t\t\tSENATOR KANE\n\t\tI don't think that's necessary.\n\n\t\t\t\tHAGEN\n\t\tSir, my client has answered every\n\t\tquestion asked by this committee\n\t\twith the utmost cooperation and\n\t\tsincerity.  He has not taken that\n\t\tFifth Amendment as it was his right\n\t\tto do, and which because of the\n\t\textreme legal complexity of this\n\t\thearing, counsel advised him to do.\n\t\tSo, I think in all fairness this\n\t\tcommittee should hear his statement\n\t\tand put it in the record.\n\n\t\t\t\tSENATOR KANE\n\t\tVery well.\n\nAt this point Senator Rogers contemptuously walks out of the\nhearing room.\n\n\t\t\t\tMICHAEL\n\t\t\t(reading)\n\t\tIn the hopes of clearing my family\n\t\tname, in the sincere desire to give\n\t\tmy children their fair share of the\n\t\tAmerican way of life without a\n\t\tblemish on their name and background\n\t\tI have appeared before this\n\t\tcommittee and given it all the\n\t\tcooperation in my power.  I consider\n\t\tmy being called before this\n\t\tcommittee an act of prejudice to\n\t\tall Americans of Italian extraction.\n\t\tI consider it a great dishonor to\n\t\tme personally to have to deny that\n\t\tI am a criminal.  I wish to have\n\t\tthe following noted for the record.\n\t\tThat I served my country faithfully\n\t\tand honorably in World War II and\n\t\twas awarded the Distinguished\n\t\tService Cross for actions in\n\t\tdefense of my country.  That I have\n\t\tnever been arrested or indicted for\n\t\tany crime whatsoever... that no\n\t\tproof linking me to any criminal\n\t\tconspiracy, whether it is called\n\t\tMafia or Cosa Nostra or whatever\n\t\tother name you wish to give, has\n\t\tever been made public.  Only one\n\t\tman has made charges against me,\n\t\tand that man is known to be a\n\t\tmurderer, arsonist and rapist.  And\n\t\tyet this committee had used this\n\t\tperson to besmirch my name.  My\n\t\tpersonal protest can only be made\n\t\tto the people of this country.  I\n\t\tcan only thank God that in this\n\t\tcountry we have a legal system and\n\t\tcourts of law to protect innocent\n\t\tpeople from wild accusation.  I\n\t\tthank God for our democratic due\n\t\tprocess of Law that shields me from\n\t\tthe false charges made by this\n\t\tcommittee's witness.  I have not\n\t\ttaken refuge behind the Fifth\n\t\tAmendment, though counsel advised\n\t\tme to do so.  I challenge this\n\t\tcommittee to produce any witness or\n\t\tevidence against me, and if they do\n\t\tnot, I hope they will have the\n\t\tdecency to clear my name with the\n\t\tsame publicity with which they have\n\t\tnow besmirched it.  I ask this\n\t\twithout malice, in the interests of\n\t\tfair play.\n\nThe television cameras have documented this moment, as Hagen\nhands the document over to the committee lawyer.\n\n\t\t\t\tSENATOR ROGERS\n\t\tWe are all impressed.  The committee\n\t\twill now recess over the weekend.\n\t\tHowever, it will continue Monday\n\t\tmorning, at eleven a.m.  At that\n\t\ttime, this committee will then\n\t\tproduce a witness directly linking\n\t\tMr. Corleone to the charges we have\n\t\tmade.  And then, Mr. Corleone may\n\t\tvery well by liable for indictments\n\t\tof perjury.  However, this document\n\t\twill be made a matter of record.\n\nEXT. ARMY POST - DAY\n\nAn army post somewhere in the East.  It is safely guarded.\n\nINT. HOUSE ON THE POST - DAY\n\nwhere Pentangeli is being held by his constant companions,\nthe two FBI MEN.\n\n\t\t\t\tPENTANGELI\n\t\tTen to one shot, you said.  Ten to\n\t\tone shot in my favor, and I lose.\n\n\t\t\t\tFBI MAN #1\n\t\tGet a good night's sleep.  We got a\n\t\tnew suit, new shirt, new tie, and\n\t\tI'm going to shave you myself.\n\t\tTomorrow we want you to look\n\t\trespectable for fifty million of\n\t\tyour fellow Americans.\n\n\t\t\t\tPENTANGELI\n\t\tMy life won't be worth a nickel\n\t\tafter tomorrow.\n\n\t\t\t\tFBI MAN #1\n\t\tWe have a special home for you for\n\t\tthe rest of your life.  Nobody gets\n\t\tnear you.  You're not going any\n\t\tplace.\n\n\t\t\t\tPENTANGELI\n\t\tYeah, some deal I made.\n\n\t\t\t\tFBI MAN #2\n\t\tYou live like a king.  You'll be a\n\t\thero.  You'll live better in here\n\t\tthan most people on the outside.\n\n\t\t\t\tPENTANGELI\n\t\tSome deal.\n\t\t\t(pause)\n\t\tI just wish Mike had took the Fifth.\n\n\t\t\t\tFBI MAN #1\n\t\tWhy'd you do it, Frankie?  After\n\t\tall these years, why'd you turn\n\t\tagainst him?\n\n\t\t\t\tPENTANGELI\n\t\tI didn't turn against nobody; he\n\t\tturned against me.\n\nEXT. THE BOATHOUSE ALCOVE - DAY\n\nA somewhat frightened Fredo Corleone sits in the easy chair\noverlooking the lake in this canopied section of the\nboathouse.  Rocco sits with him.\n\nINT. BOATHOUSE - DAY\n\nMichael is in the dark room with Hagen and Neri.\n\n\t\t\t\tMICHAEL\n\t\tHow did they get their hands on\n\t\tPentangeli?\n\n\t\t\t\tHAGEN\n\t\tRoth engineered it, Michael.  He\n\t\tmade Pentangeli think you hit him.\n\t\tDeliberately letting him get off\n\t\talive.  Then the New York detectives\n\t\tturned Frankie over to the FBI.  My\n\t\tinformants say he was half dead and\n\t\tscared stiff -- talking out loud\n\t\tthat you had turned on him and\n\t\ttried to kill him.  Anyway, they\n\t\thad him on possession, dealing in\n\t\theroin, murder one and a lot more.\n\t\tThere's no way we can get to him\n\t\tand you've opened yourself to five\n\t\tpoints of perjury.\n\n\t\t\t\tNERI\n\t\tThey've got him airtight.  He's in\n\t\ta military base, twenty-four hour\n\t\tguards.  Trying to kill him is like\n\t\ttrying to like the President --\n\t\tit's impossible.\n\n\t\t\t\tMICHAEL\n\t\tWhat does Fredo know?\n\n\t\t\t\tHAGEN\n\t\tHe says he doesn't know anything,\n\t\tand I believe him.  Roth played\n\t\tthis one beautifully.\n\n\t\t\t\tMICHAEL\n\t\tAlright.  I'm going to go outside\n\t\tand talk to Fredo.\n\nEXT. BOATHOUSE FOYER - DAY\n\nFredo sits on the couch.  When Rocco sees Michael, he\nautomatically takes his leave.  Michael sits in the chair\nopposite Fredo.\n\n\t\t\t\tFREDO\n\t\t\t(after a pause)\n\t\tI don't have a lot to say, Michael.\n\n\t\t\t\tMICHAEL\n\t\tWe have time.\n\n\t\t\t\tFREDO\n\t\tI was kept pretty much in the dark.\n\t\tI didn't know all that much.\n\n\t\t\t\tMICHAEL\n\t\tWhat about now, is there anything\n\t\tyou can help me out with?\n\n\t\t\t\tFREDO\n\t\tI know they get Pentangeli, that's\n\t\tall I know.\n\nFredo gets up, walks to the glass panel that separates the\nterrace from the lake.\n\n\t\t\t\tFREDO\n\t\tI didn't know it was a hit.  I\n\t\tswear to you I didn't know.  Johnny\n\t\tOla contacted me in Beverly Hills --\n\t\tsaid he wanted to talk.  He said\n\t\tyou and Roth were in on some big\n\t\tdeal, and there was a place for me\n\t\tin it if I could help them out.\n\t\tThey said you were being tough on\n\t\tthe negotiation, and if they had a\n\t\tlittle bit of help, they could\n\t\tclose it fast and it would be good\n\t\tfor you.\n\n\t\t\t\tMICHAEL\n\t\tAnd you believed that story.\n\n\t\t\t\tFREDO\n\t\tHe said there was something good in\n\t\tit for me...me on my own.\n\n\t\t\t\tMICHAEL\n\t\tI've always taken care of you.\n\n\t\t\t\tFREDO\n\t\tTaken care of me.  Mike, you're my\n\t\tkid brother, and you take care of\n\t\tmy.  Did you ever think of that.\n\t\tEver once?  Send Fredo off to do\n\t\tthis, send Fredo to take care of\n\t\tthat... take care of some little\n\t\tunimportant night club here, and\n\t\tthere; pick somebody up at the\n\t\tairport.  Mike, I'm your older\n\t\tbrother; I was stepped over!\n\n\t\t\t\tMICHAEL\n\t\tIt's the way Pop wanted it.\n\n\t\t\t\tFREDO\n\t\tIt wasn't the way I wanted it!  I\n\t\tcan handle things.  I'm not dumb\n\t\tChrist, not like everyone says.\n\t\tI'm smart; and I want respect.\n\n\t\t\t\tMICHAEL\n\t\tThere's nothing more you can tell\n\t\tme about this investigation?\n\n\t\t\t\tFREDO\n\t\tThe lawyer; Questadt, he belongs to\n\t\tRoth.\n\n\t\t\t\tMICHAEL\n\t\tYou're nothing to me now, Fredo;\n\t\tnot a brother, not a friend, I\n\t\tdon't want to know you, or what\n\t\thappens to you.  I don't want to\n\t\tsee you at the hotels, or near my\n\t\thome.  When you visit our Mother, I\n\t\twant to know a day in advance, so I\n\t\twon't be there.  Do you understand?\n\nMichael turns, and starts to leave.  A frightened voice\nbehind him:\n\n\t\t\t\tFREDO\n\t\tMikey?\n\nMichael doesn't stop, doesn't turn back.  He continues off\nthrough the veranda, and out the summer doors.\n\nNeri stops by him.\n\n\t\t\t\tMICHAEL\n\t\tI don't want anything to happen to\n\t\thim while my Mother's alive.\n\nMichael leaves.\n\nEXT. ARMY POST - DAY\n\nFive cars brimming with Army guards and Agents are waiting\nto move Pentangeli.  There is one empty car.\n\nINT. GUARDED HOUSE - DAY\n\nThe two FBI Agents are helping Pentangeli get dressed.  He's\nin brightly colored striped shorts and bare-chested.  The\nAgents help him with the shirt and tie.  One holds out the\ntrousers but Pentangeli ignores it and looks at himself in\nthe mirror.\n\n\t\t\t\tFBI MAN #1\n\t\tReady, Frankie.\n\n\t\t\t\tPENTANGELI\n\t\tLet's go.\n\nThe Agents open the door, and precede him, surveying the\narea.  They check the cars waiting, each with two Agents.\nThey check the gate and note the military sentries.  Then\nthey stand aside, and let Pentangeli come out.  They get\nclose to his side, and it is obvious they will protect his\nlife with their own.\n\nEXT. ARMY POST - DAY\n\nThe Agents put him in the front seat of the empty car, and\nget in with him, one at each side.  Another Agent drives.\nNow, the first cars start out; the Sentries opening the\ngates, and letting the caravan pass.\n\nAn Army supply truck comes very close to them, and the\nAgents next to Pentangeli become very tense.  Pentangeli\ngrins.  Then the truck passes on, and they relax.\n\nINT. SENATE CAUCUS ROOM - DAY\n\nThe room is crowded with TV journalists, cameras, etc.  We\npick Pentangeli up, closely guarded, being led to witness\nchair.\n\nPentangeli is seated, and made to take his oath.  FBI Agents\nare all around him.\n\nMED. VIEW\n\nAnyone given entrance to the caucus room is being frisked.\nThe five Senators take their places.\n\nVIEW ON HAGEN\n\nwaiting at his long table, very nervous.  He seems startled\nby the appearance of Pentangeli.\n\nVIEW ON PENTANGELI\n\ncatching Hagen's eye.  It's as though he is pleading for\nsome kind of understanding of the fact that he has become a\ntraitor.\n\nVIEW ON HAGEN\n\ncold; then he turns away.\n\nVIEW ON THE ENTRANCE\n\nThe bustle is settling down; then Michael Corleone enters,\nand with him is someone very peculiar and out of keeping for\nthis setting.  A burly-chested imposing man of middle age.\nVery powerful-looking with frightening magnetic eyes.  His\ndress is odd: boots, rough tie, and shirt.  He could be the\ntenor out of a Sicilian opera.  He is clearly a country Don,\ndirect from Sicily, and he dominates the room.\n\nVIEW ON PENTANGELI\n\nAt first his view is blocked.  Then he sees Michael and is a\nbit shamefaced, but still defiant.\n\nPENTANGELI'S POV\n\nMichael returns his glances without emotion.  Then the VIEW\nALTERS, revealing the Sicilian.\n\nVIEW ON PENTANGELI\n\nHe is terror stricken; obviously he recognizes the man.\n\nVIEW ON HAGEN'S TABLE\n\nMichael and the Sicilian sit by Hagen, where they can stare\ndirectly at Pentangeli; he is frozen with fear.\n\nVIEW ON THE SENATOR\n\nNotices the tension in the room.  The Chairman commences:\n\n\t\t\t\tSENATOR KANE\n\t\tWe have here a witness who will\n\t\ttestify further on Michael\n\t\tCorleone's rule of the criminal\n\t\tempire that controls gambling in\n\t\tthis country and perhaps in other\n\t\tcountries.  This witness had no\n\t\tbuffer between himself and Michael\n\t\tCorleone.  He can corroborate our\n\t\tcharges on enough counts for this\n\t\tcommittee to consider a charge of\n\t\tperjury against Michael Corleone.\n\t\t\t(then he turns to Pentangeli)\n\t\tYour name please, for the record.\n\n\t\t\t\tPENTANGELI\n\t\tFrank Pentangeli.\n\n\t\t\t\tSENATOR KANE\n\t\tWere you a member of the Corleone\n\t\tFamily?  Were you under the\n\t\tCaporegime Peter Clemenza, under\n\t\tVito Corleone, known as the\n\t\tGodfather?\n\nThere is a long silence.\n\nVIEW ON PENTANGELI\n\nHe seems unable to speak.\n\nVIEW ON THE SICILIAN\n\ngazing at him.\n\nVIEW ON PENTANGELI\n\n\t\t\t\tPENTANGELI\n\t\tI never knew no Godfather.  I got\n\t\tmy own family.\n\nSenator Kane is stunned.  The two FBI men are alert, their\neyes searching the room for what has intimidated their\nwitness at the last moment.\n\n\t\t\t\tSENATOR KANE\n\t\tMr. Pentangeli, you are\n\t\tcontradicting your confessions to\n\t\tour investigators; I ask you again,\n\t\twere you a member of a crime\n\t\torganization headed by Michael\n\t\tCorleone?\n\n\t\t\t\tPENTANGELI\n\t\tNo.  I never heard of it.  I never\n\t\theard of nothing like that.  I was\n\t\tin the olive oil business with his\n\t\tfather a long time ago.  That's all.\n\n\t\t\t\tSENATOR KANE\n\t\tWe have your confession that you\n\t\tmurdered on the orders of Michael\n\t\tCorleone.  Do you deny that\n\t\tconfession and do you know what\n\t\tdenying that confession will mean\n\t\tto you?\n\nThe die is cast and like a good soldier, Pentangeli will go\nall the way now.  So he is brazen in his defiance of the\nSenator.\n\n\t\t\t\tPENTANGELI\n\t\tThe FBI guys promised me a deal.\n\t\tSo I made up a lot of stuff about\n\t\tMichael Corleone.  Because then,\n\t\tthat's what they wanted.  But it\n\t\twas all lies.  Everything.  They\n\t\tsaid Michael Corleone did this,\n\t\tMichael Corleone did that.  So I\n\t\tsaid, \"Yeah, sure.\"\n\nHe makes a big grin to show how he has made fools of\neverybody.\n\nVIEW ON THE FBI AGENTS\n\nglancing around the room; their eyes have settled on the\nSicilian.  One of them scribbles a note on a piece of paper,\nand passes it to the Committee lawyer.  Then in turn it goes\nto Senator Kane.\n\n\t\t\t\tSENATOR KANE\n\t\tMr. Hagen, would you kindly identify\n\t\tto this committee that gentleman\n\t\tsitting on your right hand?\n\n\t\t\t\tHAGEN\n\t\t\t(coolly)\n\t\tYes, sir.  His name is Vincenzo\n\t\tPentangeli.\n\n\t\t\t\tSENATOR KANE\n\t\tIs he related to the witness?\n\n\t\t\t\tHAGEN\n\t\tHe is, I believe, a brother.\n\nVIEW ON MICHAEL AND VINCENZO PENTANGELI\n\nThey wait with no expression.\n\n\t\t\t\tSENATOR KANE\n\t\t\t(to Vincenzo Pentangeli)\n\t\tSir, I would like you to take the\n\t\tstand.\n\nVincenzo stares at him, uncomprehending.  There may just be\na shadow of contempt.  He doesn't answer.\n\n\t\t\t\tHAGEN\n\t\tSir, the gentleman does not\n\t\tunderstand English.  He would not\n\t\tin any case, take the stand.  He\n\t\tcame, at his own expense, to aid\n\t\this brother in his trouble.  He is\n\t\tnot under any jurisdiction of our\n\t\tgovernment and his reputation in\n\t\this own country is impeccable.\n\n\t\t\t\tSENATOR KANE\n\t\t\t(furious)\n\t\tThe witness is excused; take him out.\n\nThe guards and FBI Agents quickly remove Pentangeli, as\neverybody else in the room is required to sit still.\n\n\t\t\t\tHAGEN\n\t\tSenator Kane.\n\n\t\t\t\tSENATOR KANE\n\t\tThis meeting is adjourned.\n\n\t\t\t\tHAGEN\n\t\t\t(rising and shouting)\n\t\tThis committee owes an apology!\n\n\t\t\t\tSENATOR KANE\n\t\tThe committee is adjourned until\n\t\tfurther notice.\n\nFor the first time, in the midst of the confusion, Hagen\nsmiles.  A bitter, contemptuous smile.\n\nVIEW ON MICHAEL\n\nThe modest champion.  He rises and they take their leave.\n\nVIEW ON THE TWO FBI AGENTS\n\nThey watch the Corleone party as they exit.\n\nINT. WASHINGTON HOTEL CORRIDOR - DAY\n\nThe Corleone nurse is waiting, playing with the little girl\nMARY.  A distance away, the boy, Anthony, is standing by\nhimself.\n\nINT. MICHAEL'S SUITE - WASHINGTON HOTEL - DAY\n\nThe door to Michael's suite opens; Rocco leans in.\n\n\t\t\t\tROCCO\n\t\tIt's Kay.\n\nMichael is sitting in an easy chair; he seems to have\ndifficulty with his eyes.\n\n\t\t\t\tMICHAEL\n\t\tOn the phone?\n\n\t\t\t\tROCCO\n\t\tNo, she's here.\n\nMichael rises, surprised.  Rocco steps back, and Kay enters.\n\n\t\t\t\tMICHAEL\n\t\tI had no idea...\n\n\t\t\t\tKAY\n\t\tI wanted to see you before you went\n\t\tback to Nevada.  Also, the\n\t\tchildren - Michael, they're here.\n\n\t\t\t\tMICHAEL\n\t\tWhere?\n\n\t\t\t\tKAY\n\t\tIn a minute.  They're outside with\n\t\tEsther.  I'm very happy for you...\n\t\tI suppose I knew that you're simply\n\t\ttoo smart for anyone ever to beat\n\t\tyou.\n\n\t\t\t\tMICHAEL\n\t\tWhy don't you sit down?\n\n\t\t\t\tKAY\n\t\tI'm not going to stay long; I can't.\n\n\t\t\t\tMICHAEL\n\t\tThere are a lot of things I want to\n\t\ttalk to you about.  Things I've\n\t\tbeen thinking about -- changes I\n\t\twant to make.\n\n\t\t\t\tKAY\n\t\tI think it's too late for changes,\n\t\tMichael.  I promised myself I\n\t\twouldn't talk about it and I've\n\t\tgone and spoiled it.\n\n\t\t\t\tMICHAEL\n\t\tWhy too late?\n\n\t\t\t\tKAY\n\t\tTell me, Michael.  What really\n\t\thappened with Pentangeli?\n\n\t\t\t\tMICHAEL\n\t\tHis brother came to help him.\n\n\t\t\t\tKAY\n\t\tI didn't even know he had a brother.\n\t\tAnd where is he now?\n\n\t\t\t\tMICHAEL\n\t\tOn a plane back to Sicily.\n\n\t\t\t\tKAY\n\t\tAnd that's all he had to do.  Just\n\t\tshow his face.\n\n\t\t\t\tMICHAEL\n\t\tThat's all.  You see, in Sicily, in\n\t\tthe old days... there was only one\n\t\tlegitimate reason to kill a blood\n\t\trelative... only one.  IF he was a\n\t\ttraitor.\n\n\t\t\t\tKAY\n\t\tYou would have killed his brother?\n\n\t\t\t\tMICHAEL\n\t\tKay, you've got it wrong.  That\n\t\tkind of thing's all over, I promised\n\t\tyou.  This was between the two\n\t\tbrothers.  Years ago Frankie had a\n\t\tyoung girlfriend; he called her his\n\t\tco-wife.  That was his joke, but he\n\t\tmeant it.  He wouldn't divorce his\n\t\twife... because she was a great\n\t\tcook.  He said he girlfriend made a\n\t\tspaghetti sauce once and it was so\n\t\tterrible he knew he could never\n\t\tmarry her.  He set her up in a\n\t\thouse in Jersey.  She had to be\n\t\tfaithful... and she had to have kids.\n\t\tAnd she did, two, a boy and a girl.\n\t\tHe had her checked out and watched\n\t\tso she couldn't cheat... but the\n\t\tgirl couldn't stand that kind of\n\t\tlife.  She begged him to let her go.\n\t\tHe did.  He gave her money and made\n\t\ther give up the kids.  Then Frankie\n\t\ttook them to Italy, and had them\n\t\tbrought up by his brother Vincenzo.\n\t\tWhere he knew they'd by safe.\n\nKay begins to realize.\n\n\t\t\t\tMICHAEL\n\t\tWhen he saw his brother in the\n\t\thearing room, he knew what was at\n\t\tstake.\n\t\t\t(pause)\n\t\tI don't think Vincenzo would have\n\t\tdone it.  He loves the kids, too.\n\t\tOmerta, Kay.  Honor, silence.  It\n\t\thad nothing to do with me.  It was\n\t\tbetween those brothers.\n\n\t\t\t\tKAY\n\t\tI'll bring the children up now;\n\t\tthey want to say goodbye.\n\n\t\t\t\tMICHAEL\n\t\tKay, I told you...\n\n\t\t\t\tKAY\n\t\tGoodbye, Michael.\n\n\t\t\t\tMICHAEL\n\t\tI won't let you leave!  Christ, do\n\t\tyou think I'm going to let you leave.\n\n\t\t\t\tKAY\n\t\t\t(meekly)\n\t\tMichael.\n\n\t\t\t\tMICHAEL\n\t\tNo, I don't want to hear anything.\n\t\tThere are things between men and\n\t\twomen that will not change; things\n\t\tthat have been the same for\n\t\tthousands of years.  You are my\n\t\twife, and they are my children...\n\t\tand I love you and I will not let\n\t\tyou leave, because you are MINE!\n\n\t\t\t\tKAY\n\t\tOh, I do feel things for you,\n\t\tMichael; but now, I think it's pity.\n\t\tFor the first time since I've known\n\t\tyou, you seem so helpless.  You\n\t\theld me a prisoner once; will you\n\t\ttry again?\n\n\t\t\t\tMICHAEL\n\t\tIf that's what it takes; then yes,\n\t\tI will.\n\n\t\t\t\tKAY\n\t\tAt this moment, I feel no love for\n\t\tyou at all.  I never thought that\n\t\tcould happen, but it has.\n\n\t\t\t\tMICHAEL\n\t\tWe'll go back tonight.  Bring the\n\t\tchildren.\n\n\t\t\t\tKAY\n\t\tYou haven't heard me.\n\nHe moves to her; he does love her, and is tender with her.\n\n\t\t\t\tMICHAEL\n\t\tHow can I let you leave; how can I\n\t\tlet you take my children away?\n\t\tDon't you know me?  You understand,\n\t\tit's an impossibility.  I would\n\t\tnever let it happen; no, never, not\n\t\tif it took all my strength, all my\n\t\tcunning.  But in time, soon, you'll\n\t\tfeel differently.  You see, you'll\n\t\tbe happy that I stopped you.  I\n\t\tknow you.  You'll forget about\n\t\tthis; you'll forget about the baby\n\t\twe lost... and we'll go on, you and\n\t\tI.\n\n\t\t\t\tKAY\n\t\tThe baby I lost...\n\n\t\t\t\tMICHAEL\n\t\tI know what it meant... and I'm\n\t\tprepared to make it up to you.  I\n\t\twill make changes; I can.\n\t\t\t(he clenches his fist tightly)\n\t\tI CAN change; that I have learned,\n\t\tthat I have the strength to change...\n\t\tAnd we have another child, a boy...\n\t\tand you'll forget the miscarriage.\n\n\t\t\t\tKAY\n\t\tIt wasn't a miscarriage.  And you\n\t\twith your cunning, couldn't you\n\t\tfigure it out!  It was an abortion;\n\t\tan abortion, like our marriage is\n\t\tan abortion, something unholy and\n\t\tevil.  I don't want your son; I\n\t\twouldn't bring another of your sons\n\t\tinto this world.  An abortion,\n\t\tMichael... it was a son, and I had\n\t\tit killed, but this must all end!\n\nVIEW ON MICHAEL\n\nHe had no hint, not in his wildest imagination could he have\nguessed that she would do such a thing.\n\n\t\t\t\tKAY\n\t\tAnd I know that now it's over; I\n\t\tknew it then, there would be no way\n\t\tyou could ever forgive me, not with\n\t\tthis Sicilian thing that goes back\n\t\ttwo thousand years.\n\nHe is silent, though raging -- then, with all his passion,\nand his strength, he raises his arms, and strikes her across\nher neck, literally knocking her down to the floor, and\nhurting her badly.\n\n\t\t\t\tMICHAEL\n\t\t\t(coldly)\n\t\tYou won't take my children.\n\n\t\t\t\t\t\t\tFADE OUT.\n\nFADE IN:\n\nEXT. THE CORLEONE ESTATE AT TAHOE - FULL VIEW - DAY\n\nA collection of dark cars and black limousines are gathered\nto one side.  A few drivers wait quietly.\n\nAnd then, to the other extreme of the estate, is a small\ngrouping of about twenty to thirty people, gathered near\nMichael's house.\n\nMED. CLOSE SHOT\n\nConnie Corleone, dressed simply and now showing her age\nwithout the carefully applied makeup which we have been used\nto, kneels down before the shrine of Santa Theresa, and puts\ndown a bouquet of flowers, along with others that have been\nplaced there.  We see that some have the simple silk ribbon\nwith the word \"Mama\" hand-lettered upon it.\n\nHer two children stand close behind her; they had been\nraised by their Grandmother.\n\nConnie steps back, and moves through the small group of\nfriends and relatives, into Michael's house.\n\nINT. MICHAEL'S HOUSE - CONNIE'S VIEW - DAY\n\nFredo, kneeling by the coffin of his mother in a portion of\nthe house that has been set aside for the wake.  Fredo\nconcludes his prayer, wipes away the tears in his eyes and\nsteps away from the coffin.\n\nHe stops when he notices Neri, a little distance away,\nlooking at him.\n\nVIEW ON NERI\n\nAfter a moment, he nods respectfully to Fredo, and steps\nforward, moving to the old woman's coffin.  Fredo moves to\nHagen, who is there with his wife and children.\n\n\t\t\t\tFREDO\n\t\tTom.  Where's Mike?\n\n\t\t\t\tHAGEN\n\t\t\t(difficult to tell him)\n\t\tHe's waiting for you to leave.\n\n\t\t\t\tFREDO\n\t\tCan I talk to him?\n\n\t\t\t\tHAGEN\n\t\tNo chance.  I'm sorry, Freddie.\n\n\t\t\t\tCONNIE\n\t\t\t(who has heard this)\n\t\tCan I see him?\n\n\t\t\t\tHAGEN\n\t\tHe's in the boathouse.\n\nINT. THE BOATHOUSE - MED. VIEW - DAY\n\nMichael sits quietly in the darkened room in one of the big\nsofas, dressed immaculately in suit and tie.  His two\nchildren, also dressed for the wake sit opposite him in the\nother oversized sofa, their shoes not touching the floor.\nWe regard this tableau for a long moment.\n\n\t\t\t\tCONNIE (O.S.)\n\t\t\t(quietly)\n\t\tMichael?  It's Connie.\n\nShe comes in, and sits down by his knees.\n\n\t\t\t\tCONNIE\n\t\tI want to stay close to home now,\n\t\tis that alright?\n\nMichael nods.\n\n\t\t\t\tCONNIE\n\t\tIs Kay coming?\n\n\t\t\t\tMICHAEL\n\t\tNo.\n\n\t\t\t\tCONNIE\n\t\tMichael, Fredo's in the house with\n\t\tMama.  He asked for you, and Tom\n\t\tsaid he couldn't see you.\n\n\t\t\t\tMICHAEL\n\t\tTom is right.\n\n\t\t\t\tCONNIE\n\t\tKids, why don't you go outside for\n\t\ta while?\n\nThe children don't move; Connie realizes they will only\nlisten to Michael.\n\n\t\t\t\tCONNIE\n\t\tI want to talk to you, Michael.\n\n\t\t\t\tMICHAEL\n\t\tThe children can stay.\n\n\t\t\t\tCONNIE\n\t\tI hated you for so long, Michael;\n\t\tfor so many years.  I think I did\n\t\tthings to myself, to hurt myself,\n\t\tso that you would know -- and you\n\t\twould be hurt too.  But I understand\n\t\tyou now; I think I do.  You were\n\t\tbeing strong for all of us, like\n\t\tPapa was.  And I forgive you, and\n\t\twant to be close to you now.  Can't\n\t\tyou forgive Fredo; he's so sweet,\n\t\tand helpless without you.\n\nSlowly, Michael puts his hand on her hair, and touches her\ngently.\n\n\t\t\t\tCONNIE\n\t\tYou need me, Michael.  I want to be\n\t\twith you now.\n\nINT. MICHAEL'S HOUSE - DAY\n\nFriends, relatives; Francesca and her new husband, Gardner\nand their baby; Sandra Corleone; Teresa, her children; all\nthe familiar faces of the family are present, quietly paying\ntheir respects to Mama.\n\nSome of the men can be seen in the kitchen, drinking wine,\nand talking in low voices.\n\nFredo is there, broken-hearted over the loss of his Mother;\nlike some lost child with no friends.\n\nMED. VIEW\n\nMichael enters the room, followed by Connie, who tends\nlittle Mary and Anthony.\n\nHe approaches his brother, and then embraces.  Fredo breaks\ninto tears.\n\n\t\t\t\tFREDO\n\t\tChrist, Mike.  Jesus Christ, Mike.\n\nVIEW ON MICHAEL\n\nembracing his brother, he glances up.\n\nVIEW ON NERI\n\nquiet, and deadly.\n\nEXT. THE TAHOE ESTATE - MED. VIEW - DAY\n\nTom Hagen is talking in the distance to his wife, and one of\nhis older sons; he kisses, and moves toward the boathouse.\nAfter crossing the lawn, he stops.\n\nVIEW ON SANDRA CORLEONE\n\nwaiting there; obviously wanting to talk to him.  He\ncontinues, and she walks with him.\n\nMOVING VIEW ON THE TWO\n\nas they cross toward the boathouse.\n\n\t\t\t\tSANDRA\n\t\tYou're going to talk to him now.\n\n\t\t\t\tHAGEN\n\t\tYes.\n\n\t\t\t\tSANDRA\n\t\tWill you tell him?\n\n\t\t\t\tHAGEN\n\t\tI don't know.\n\nShe stops him.\n\n\t\t\t\tSANDRA\n\t\tTom, think of yourself for once.\n\t\tDon't let this opportunity slip\n\t\tthrough your fingers; don't do it.\n\t\tWe're all trapped here, don't you\n\t\tsee?\n\nHe continues past her, without answering her.  Continues up\nto the boathouse.  He stops before he enters.\n\nHAGEN'S VIEW\n\nFredo is sitting by the edge of the harbor with Michael's\nson Anthony; he is helping him with some fishing rig.\n\nINT. THE BOATHOUSE - VIEW ON MICHAEL - DAY\n\nlooking through the window at his son and brother.  Neri\nsits in the room, dressed informally.\n\n\t\t\t\tMICHAEL\n\t\t\t(without looking back)\n\t\tSit down, Tom.  Have you heard\n\t\tabout our friend and partner, Mr.\n\t\tHyman Roth?\n\n\t\t\t\tHAGEN\n\t\tI know he's in Israel.\n\n\t\t\t\tNERI\n\t\t\t(hands Hagen the paper)\n\t\tThe High Court of Israel turned\n\t\tdown his request to live as a\n\t\t'returned Jew.' His passport's been\n\t\tinvalidated except for return to\n\t\tthe U.S.  He landed in Buenos Aires\n\t\tyesterday, offered a gift of one\n\t\tmillion dollars if they would give\n\t\thim citizenship.  They turned him\n\t\tdown.\n\n\t\t\t\tHAGEN\n\t\t\t(reading)\n\t\tHe's going to try Panama...\n\n\t\t\t\tMICHAEL\n\t\tThey won't take him; not for a\n\t\tmillion, not for ten million.\n\n\t\t\t\tHAGEN\n\t\tHis medical condition is reported\n\t\tas... \"terminal.\"\n\n\t\t\t\tMICHAEL\n\t\tHe's been dying of the same heart\n\t\tattack for twenty years.\n\n\t\t\t\tHAGEN\n\t\tThat plane goes to Miami...\n\n\t\t\t\tMICHAEL\n\t\tI want it met.\n\n\t\t\t\tHAGEN\n\t\t\t(understanding)\n\t\tMike, it's impossible.  He'll be\n\t\tmet by the Internal Revenue; the\n\t\tCustoms Service, and half the FBI.\n\n\t\t\t\tMICHAEL\n\t\tI don't like it when you use the\n\t\tword impossible; nothing is\n\t\timpossible...\n\n\t\t\t\tHAGEN\n\t\tMike, it would be like trying to\n\t\tkill the President; there's no way\n\t\twe can get to him.\n\n\t\t\t\tMICHAEL\n\t\tI'm surprised at you, Tom.  If\n\t\tthere's anything certain; certain\n\t\tin life; if history has taught us\n\t\tanything, it's that you can kill...\n\t\t\t(he stops, then coldly)\n\t\tANYBODY.  But perhaps your\n\t\trelucatance is because you've come\n\t\tto tell me that you're moving your\n\t\tfamily to Vegas, that you've been\n\t\toffered the Vice-Presidency of the\n\t\tHoustan Hotels there.  Or weren't\n\t\tyou going to tell me at all?\n\n\t\t\t\tHAGEN\n\t\tAre you so hungry for traitors; do\n\t\tyou want to find them everywhere?\n\n\t\t\t\tMICHAEL\n\t\tThey are everywhere!\n\n\t\t\t\tHAGEN\n\t\tI turned Houstan down; I didn't see\n\t\twhy I should tell you about an\n\t\toffer I turned down.\n\t\t\t(Michael begins to\n\t\t\tconfuse him)\n\t\tAre you sure, Mikey?  Are you sure\n\t\tof what we're doing; what we'll\n\t\tgain; what does the family gain?\n\t\tForget that, Mike; I already know\n\t\tthe answer.\n\n\t\t\t\tMICHAEL\n\t\tI know you do, Tom.  Then I can\n\t\tcount on you to help me do the\n\t\tthings I have to do.  If not, call\n\t\tHoustan, and become a Vice-President.\n\t\tTake your family and your mistress\n\t\tand move them to Las Vegas.\n\n\t\t\t\tHAGEN\n\t\tWhy do you hurt me, Michael?  I've\n\t\talways been loyal to you.\n\n\t\t\t\tMICHAEL\n\t\tGood.  Then you're staying.\n\n\t\t\t\tHAGEN\n\t\tI'm staying.\n\t\t\t(he pauses...then,\n\t\t\twithout being asked)\n\t\tDon't ever enjoy the cruel part of\n\t\tall this; Sonny never listened to\n\t\tme about that.\n\t\t\t(then he sits down,\n\t\t\tand opens his briefcase)\n\t\tNow, explain everything to me.\n\nEXT. THE HARBOR - DAY\n\nFredo sits with Anthony, with a silly-looking fishing hat on\nhis head, covered with lure and flies.\n\n\t\t\t\tFREDO\n\t\tAnthony, ole buddy, your Uncle\n\t\tFredo's gonna teach you how to\n\t\tcatch the big fish.  You know, when\n\t\tI was a kid, I did this amazing\n\t\tthing.  I went out on a fishing\n\t\ttrip; me and my brothers and my\n\t\tPop, and no one could catch a fish\n\t\texcept me.  And this was my secret:\n\t\t\t(confidentially)\n\t\tEvery time I would put the line\n\t\tdown I would say a \"Hail Mary\" and\n\t\tevery time I said a \"Hail Mary\" I\n\t\twould catch a fish.  Now, when it's\n\t\tsunset, we're gonna go out on the\n\t\tlake, and we're gonna try it.\n\nINT. GUARDED HOUSE - DAY\n\nThe guards step aside as Tom Hagen enters the foyer of the\nhouse.  He shows a court order to them and they lead him up\nthe stairs where he knocks on the door.\n\nINT. GUARDED HOUSE - DAY\n\nThere is a KNOCK at the door.  The two guards show Hagen in\nand Hagen presents the court order to one of the FBI men.\n\n\t\t\t\tHAGEN\n\t\tI think I prefer to see my client\n\t\tprivately.\n\n\t\t\t\tPENTANGELI\n\t\tThe room has a bug in it.\n\n\t\t\t\tHAGEN\n\t\t\t(to the FBI men)\n\t\tI'd like to go outside with him, in\n\t\tthe open air.\n\n\t\t\t\tFBI MAN #1\n\t\tThis room is not bugged.\n\n\t\t\t\tHAGEN\n\t\tYou have guards outside and the\n\t\telectric fence.  There's no security\n\t\treason for not letting us talk in\n\t\tthe yard.\n\n\t\t\t\tFBI MAN #1\n\t\tOkay.\n\nThey pass out of the room.\n\nEXT. THE ARMY POST - DAY\n\nHagen and Pentangeli outside, by the electric fence.  They\ncannot be overheard.  Pentangeli takes out some cigars and\noffers Hagen one.  Hagen takes it and Pentangeli lights both\ntheir cigars.  They puff on them contentedly.  They are\ncomfortable together, almost.\n\n\t\t\t\tHAGEN\n\t\tEverything is going to be okay,\n\t\tFrankie, don't worry.\n\n\t\t\t\tPENTANGELI\n\t\tDid my brother go back?\n\n\t\t\t\tHAGEN\n\t\tYeah, but don't worry.\n\n\t\t\t\tPENTANGELI\n\t\tHe's ten times tougher than me, my\n\t\tbrother.  He's old-fashioned.\n\n\t\t\t\tHAGEN\n\t\tYeah.  He wouldn't even go out to\n\t\tdinner.  Just wanted to go home.\n\n\t\t\t\tPENTANGELI\n\t\tThat's my brother.  Nothing could\n\t\tget him away from that two mule\n\t\ttown.  He coulda been big over\n\t\there -- he could of had his own\n\t\tFamily.\n\n\t\t\t\tHAGEN\n\t\tYou're right.\n\n\t\t\t\tPENTANGELI\n\t\tTom, what do I do now?\n\nThe light is beginning to turn reddish as the sun falls.\n\n\t\t\t\tHAGEN\n\t\tFrankie, you were always interested\n\t\tin politics, in history.  I remember\n\t\tyou talking about Hitler back in\n\t\t'43.  We were young then.\n\n\t\t\t\tPENTANGELI\n\t\tYeah, I still read a lot.  They\n\t\tbring me stuff.\n\n\t\t\t\tHAGEN\n\t\tYou were around the old timers who\n\t\tdreamed up how the Families should\n\t\tbe organized, how they based it on\n\t\tthe old Roman Legions, and called\n\t\tthem 'Regimes'... with the 'Capos'\n\t\tand 'Soldiers,' and it worked.\n\n\t\t\t\tPENTANGELI\n\t\tYeah, it worked.  Those were great\n\t\told days.  We was like the Roman\n\t\tEmpire.  The Corleone family was\n\t\tlike the Roman Empire.\n\n\t\t\t\tHAGEN\n\t\t\t(sadly)\n\t\tYeah, it was once.\n\nThey both puff on their cigars.  Pentangeli lets himself be\ncarried away by thoughts of old days of glory; Hagen thinks\nof other days too.\n\n\t\t\t\tHAGEN\n\t\t\t(very gently)\n\t\tThe Roman Empire... when a plot\n\t\tagainst the Emperor failed, the\n\t\tplotters were always given a chance\n\t\tto let their families keep their\n\t\tfortunes.\n\n\t\t\t\tPENTANGELI\n\t\tYeah, but only the rich guys.  The\n\t\tlittle guys got knocked off.  If\n\t\tthey got arrested and executed, all\n\t\ttheir estate went to the Emperor.\n\t\tIf they just went home and killed\n\t\tthemselves, up front, nothing\n\t\thappened.\n\n\t\t\t\tHAGEN\n\t\tYeah, that was a good break.  A\n\t\tnice deal.\n\nPentangeli looks at Hagen; he understands.\n\n\t\t\t\tPENTANGELI\n\t\tThey went home and sat in a hot\n\t\tbath and opened their veins, and\n\t\tbled to death.  Sometimes they gave\n\t\ta little party before they did it.\n\nHagen throws away his cigar.  Pentangeli puffs on his.\n\n\t\t\t\tHAGEN\n\t\tDon't worry about anything, Frankie\n\t\tFive-Angels.\n\n\t\t\t\tPENTANGELI\n\t\tThanks, Tom.  Thanks.\n\nThey shake hands.  The FBI Agents come out to let Hagen out\nthe gate.  Pentangeli is led back to the house.\n\n\t\t\t\tFBI MAN #1\n\t\tYour lawyer tell you he can get\n\t\tthat 600 years reduced to 500?\n\nPentangeli puffs on his cigar and reflects.\n\n\t\t\t\tPENTANGELI\n\t\tYou boys sure you can't get me a\n\t\tbroad for tonight?  Give me a\n\t\tlittle party?\n\n\t\t\t\tFBI MAN #2\n\t\tWe got some nice books.\n\nPentangeli puffs on his cigar and gives the Agent a smile an\nold man gives a child.  He starts upstairs.\n\n\t\t\t\tPENTANGELI\n\t\tI guess I'll just take a hot bath.\n\nEXT. THE ARMY POST - DAY\n\nHagen walks away; glances back.  Then gets into his waiting\ncar and drives off.\n\nINT. THE BOATHOUSE - FULL VIEW - SUNSET\n\nMichael sits alone in the empty boathouse; in the shadows.\n\nINT. BOAT DOCK - SUNSET\n\nNeri stands by the dock area under the boathouse.  He pushes\nthe button which lowers a boat by winch and tackle.  He\nwears a fishing cap.\n\nHe steps into the boat, and pulls the small outboard, which\nglides the boat out into the harbor.\n\nMED. VIEW\n\nThe boat pulls up alongside Fredo and Anthony.\n\n\t\t\t\tFREDO\n\t\tHere we go; and remember the secret.\n\nHe lifts Anthony into the boat.\n\n\t\t\t\tCONNIE (O.S.)\n\t\tAnthony.\n\nTHEIR VIEW\n\nConnie, in houseclothes, is calling Anthony.\n\n\t\t\t\tFREDO\n\t\tHe's here; we're goin' fishing.\n\n\t\t\t\tCONNIE\n\t\tHe can't go; Michael wants to take\n\t\thim into Reno.\n\n\t\t\t\tFREDO\n\t\tAh.  Okay, kid, you got to go to\n\t\tReno with your Pop.\n\nHe lifts the boy out of the boat, and puts him on the shore.\n\n\t\t\t\tFREDO\n\t\tI'll catch one for you, with the\n\t\tsecret.\n\n\t\t\t\tCONNIE\n\t\tHurry, Anthony.\n\nNeri stands the motor; and the boat with the two fisherman\nglides off.\n\nVIEW ON MICHAEL\n\nwatching, from the dark window of the boathouse.\n\nINT. HIGH SECURITY HOUSE IN ARMY POST - NIGHT\n\nThe FBI man knocks on the bathroom door in the house where\nthey have kept Pentangeli.\n\n\t\t\t\tFBI MAN #1\n\t\tFrankie, open up.  You okay?\n\nNo answer; he hammers on the door.  Using his elbow, and\nthen a kick he breaks into the bathroom.\n\nHIS VIEW\n\nPentangeli lying in a tub of water.  His stomach shows above\nit.  His wrists are cut and covered with blood.  The bath\nwater has a purplish tone.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nEXT. LAKE TAHOE - MED. VIEW - SUNSET\n\nFredo and Neri are fishing, each with lines out.  The VIEW\nMOVES CLOSER, and we can hear Fredo as he holds the pole.\n\n\t\t\t\tFREDO\n\t\t... the Lord is with thee.  Blessed\n\t\tart thou amongst women, and blessed\n\t\tis the fruit of thy womb, Jesus.\n\nLONG SHOT\n\nThe boat on the shimmery lake.\n\n\t\t\t\tFREDO\n\t\t... Holy Mary, Mother of God, pray\n\t\tfor us...\n\nWe hear a quiet, echoing GUNSHOT; and then silence.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nINT. MIAMI AIRPORT - NIGHT\n\nAn exhausted Hyman Roth, ill-shaven, and in shirt-sleeves in\ntaken into custody by a swarm of Customs, and FBI men.  They\nallow him to be photographed by press people; and television\ncameramen.\n\n\t\t\t\tFBI MAN\n\t\tMr. Roth, we have to take you into\n\t\tcustody.\n\n\t\t\t\tROTH\n\t\tYes, I know.\n\nSome flashbulbs go off.\n\n\t\t\t\tREPORTER\n\t\tCan you give us your reaction to\n\t\tthe High Court of Israel's ruling.\n\n\t\t\t\tROTH\n\t\tI am a retired investor on a\n\t\tpension, and I wished to live there\n\t\tas a Jew  in the twilight of my\n\t\tlife...\n\n\t\t\t\tLAWYER\n\t\tMr. Roth is not a well man; he's\n\t\ttired of running.\n\n\t\t\t\tROTH\n\t\tI'm an old man; at my age, it's too\n\t\tlate to start worrying.\n\n\t\t\t\tREPORTER\n\t\tIs it true you are worth over three\n\t\thundred million dollars, Mr. Roth?\n\n\t\t\t\tROTH\n\t\tI'm a retired investor, living on a\n\t\tpension... I came home to vote in\n\t\tthe Presidential election, because\n\t\tthey wouldn't give me an absentee\n\t\tballot...\n\nThe newsmen and photographers all laugh, as the FBI men move\nhim away.\n\nCLOSE VIEW\n\nOne of the newspapermen laughing we recognize to be Rocco\nLampone.\n\nHe moves closer to Roth, and shoves his revolver right\nagainst his head, and in a second, on camera, assassinates\nRoth.  People scream, as Rocco attempts to run down the\nairport corridor, limping as he does.\n\nFBI men easily pick him off.\n\n\t\t\t\t\t\t\tFADE OUT.\n\nEXT. THE DRIVEWAY BY MICHAEL'S HOUSE - DAY\n\nA taxi cab waits by the house; its driver sleeping with a\nnewspaper over his face.\n\nINT. MICHAEL'S HOUSE - DAY\n\nThe cleaning woman, Esther, who had been with Kay for years,\nsits by the dining room table, weeping profusely.  Behind\nher, in the recreation room, we can see the tableau of Kay\nsitting on the couch, her little daughter Mary, between her\nknees, talking quietly about things we cannot hear.  Her son\nAnthony sits by himself, in another chair by the side of the\nroom.\n\nMED. VIEW\n\nConnie comes into the house quickly, and moves toward them.\n\n\t\t\t\tCONNIE\n\t\tKay, you have to go.\n\nThis prompts Esther to weep all the more.  Kay hugs her\ndaughter, and kisses her many times.\n\n\t\t\t\tCONNIE\n\t\tYou have to hurry; he's coming.\n\nKay puts her coat on; then stands, and reaches out for her\nson.\n\n\t\t\t\tKAY\n\t\tAnthony, kiss Mama goodbye.\n\nHe doesn't move.\n\n\t\t\t\tCONNIE\n\t\t\t(angrily)\n\t\tAnthony, you kiss your Mother\n\t\tgoodbye!\n\nHe rises, and walks to her.  Hugs her lifelessly.\n\nMED. CLOSE VIEW\n\non Kay, kissing her boy.\n\n\t\t\t\tKAY\n\t\tAnthony, say goodbye; your Mama\n\t\tloves you.\n\n\t\t\t\tANTHONY\n\t\tGoodbye.\n\nShe restrains any tears; she has become too strong for tears.\nKay starts to go; picks up Mary, kisses her, and starts to go.\n\nNEW VIEW\n\nShe steps out the kitchen door; then she cannot help herself.\nCrouches down, outside, and calls to her son.\n\n\t\t\t\tKAY\n\t\tAnthony, kiss me once.\n\nThen she looks up, and slowly rises.\n\nHER VIEW\n\nMichael has stepped into the dining room.  He seems older\nsomehow; as though some sickness has taken more years away\nfrom him.\n\nVIEW ON KAY\n\nlooks at him; instinctively, she takes a step back.\n\nVIEW ON MICHAEL\n\nslowly steps toward her.\n\nVIEW ON KAY\n\nAnother step back; the door is still open.\n\nVIEW ON MICHAEL\n\nHe moves closer to the door; stops, looks at her.  And then\ncloses it obscuring any view of her.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nEXT. TAHOE ESTATE - DAY\n\nIt is late fall -- most of the leaves have fallen on the\ngrounds and there is quite a wind.\n\nMED. VIEW\n\nThe water is whipped up by the wind, and the waves are high\nas they break against the pavilion.  We HEAR the MUSIC of\ntime passing, of Michael, of the Godfather over these images.\n\nVIEW ON THE SWIMMING POOLS\n\nThey have not been used in several months; they are drained\nand the bottoms are mossy and dark.\n\nVIEW ON THE MAIN GATE\n\nLeaves blowing past it; we don't see the button men; only a\nhint of someone in the gatehouse.\n\nVIEW ON THE HOUSES\n\nSome of the houses have had the summer awnings taken down,\nand put away.  Some of the windows have been boarded up.\n\nVIEW ON THE KENNELS\n\nThere are still the guard dogs; some sleeping, some moving\nimpatiently.\n\nAs the MUSIC concludes its statement.\n\nMED. VIEW\n\nThe peninsula of the private Corleone Harbor.  We see the\nfigures of two people, seated at a table.\n\nMED. VIEW\n\nMichael sits at a table having a sparse lunch.  He is\nattended by his sister Connie, who seems to be the closest\nperson now living on the estate with him.  We see from the\nway she pampers him with his lunch, that she has fallen into\nthe role of a surrogate Mother-Wife.  He seems older than\nhis years, as though his illness, diabetes, has taken its\ntoll.\n\n\t\t\t\tCONNIE\n\t\tDon't worry; I'm sure he got here\n\t\ton time.  The roads from the\n\t\tairport are so windy, it takes\n\t\tforever; I've driven them myself.\n\nShe picks up some of the serving plates that he has left\nuntouched.\n\n\t\t\t\tCONNIE\n\t\tI'll bring him out to you as soon\n\t\tas he comes.\n\nShe moves back to the main house.\n\nMED. CLOSE VIEW ON MICHAEL\n\nHe turns and looks at the rough water of the lake for a\nmoment.  He slowly takes a sip of wine.\n\nEXT. A PLACE IN THE GARDEN - DAY\n\nThere are a few chairs.\n\nMED. VIEW ON ANTHONY CORLEONE\n\nHe is eighteen years old.\n\n\t\t\t\tANTHONY\n\t\tHello, Dad.\n\nVIEW ON MICHAEL\n\nsquinting up at his son.\n\n\t\t\t\tMICHAEL\n\t\tAnthony.\n\nHe rises, and reaches up to his son, who is now taller than\nhe; he embraces him.\n\n\t\t\t\tMICHAEL\n\t\tYou've grown so tall... so tall in\n\t\tthe last year.  You're much taller\n\t\tthan me.\n\n\t\t\t\tANTHONY\n\t\tI was taller than you when I was\n\t\tfourteen.\n\n\t\t\t\tMICHAEL\n\t\tSit down.  Your Aunt Connie and I\n\t\twaited for you to have some lunch,\n\t\tbut now it's all dried out.\n\n\t\t\t\tANTHONY\n\t\tI'm not hungry.\n\n\t\t\t\tMICHAEL\n\t\tWell, that's alright... alright.\n\t\tGood.  You'll graduate in another\n\t\tyear, isn't that right?  You know...\n\t\tI never finished college.  I was a\n\t\tgood student, but I never finished.\n\t\tOf course, there was a war then.\n\nConnie approaches them.\n\n\t\t\t\tCONNIE\n\t\tDon't let me interrupt anything,\n\t\tthis will just take a second.  Here.\n\t\t\t(she takes out a\n\t\t\tsmall needle, and\n\t\t\tbegins to prepare it)\n\t\tYour father has to have his insulin\n\t\tshot.  Why don't you go to your\n\t\troom and put your things away,\n\t\tAnthony.\n\nShe begins to give Michael the shot.\n\n\t\t\t\tMICHAEL\n\t\tHurry back; we'll talk.  We'll talk.\n\nAnthony goes on his way to the house with his things.\nConnie gives Michael the shot.\n\n\t\t\t\tCONNIE\n\t\tWhenever I see that lake so cold, I\n\t\tthink of poor Fredo, drowned.  Lake\n\t\tTahoe is very cold.  They say if a\n\t\tperson drowns in it, that the body\n\t\twill remain mid-suspended --\n\t\tperfectly preserved.  Some say it\n\t\twill remain forever.\n\nShe finishes the shot, puts her things away.\n\n\t\t\t\tCONNIE\n\t\tYour boy will be right back.\n\nShe leaves.\n\nVIEW ON MICHAEL\n\nAlone in the garden.\n\nOUR VIEW begins to MOVE CLOSER to him.  We begin to HEAR\nMUSIC of the forties; happy music, swing music, as we move\nCLOSER to Michael.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nINT. OLD CORLEONE HOUSE - MED. VIEW - NIGHT\n\nSONNY CORLEONE, his arm wrapped around a smiling red-faced\nCarlo Rizzi, pulls him into the Corleone dining room.\n\n\t\t\t\tSONNY\n\t\tHey, who knows my buddy Carlo Rizzi.\n\t\tHere... my brother Fredo, here's my\n\t\tMom.  Mom, whatcha got cooking?\n\t\tAnd Carlo, this is my kid sister\n\t\tConnie.  Here, pull up a chair,\n\t\tCarol is sitting next to Connie.\n\t\tOh, the droopy kid over there is\n\t\tMike.  The college boy.\n\nAn older, lanky man enters the room, his arms laden with\npresents.  This is TESSIO.\n\n\t\t\t\tTESSIO\n\t\tBuon Natale, everybody.  Buon\n\t\tNatale...\n\t\t\t(he smiles at Tom Hagen)\n\t\tHi, Tom, how's every little thing?\n\n\t\t\t\tHAGEN\n\t\t\t(helping him with the presents)\n\t\tWonderful, Sal.\n\nNow the study door opens, and DON CORLEONE enters.\n\n\t\t\t\tDON CORLEONE\n\t\tIs dinner ready?\n\n\t\t\t\tMOM\n\t\tTwo minutes.\n\nThe Don happily regards his family; his sons and daughters\nand even some Grandchildren.  He raises a glass.\n\n\t\t\t\tDON CORLEONE\n\t\tA good life, a long life to all my\n\t\tchildren, and friends.  To my\n\t\tgrandchildren, and those that will\n\t\tbe.  To our family.\n\nThey all drink.\n\nThey refill glasses; then Tessio proposes a toast.\n\n\t\t\t\tTESSIO\n\t\tTo our Godfather.\n\nThey all drink.\n\nINT. THE DINING ROOM - MED. VIEW - NIGHT\n\nThe family is happily at Christmas dinner.  Don Corleone\nseated at the head of the table.\n\n\t\t\t\tSONNY\n\t\tWhat'd you think of those Japs, eh?\n\t\tThe nerve of those Japs, coming\n\t\tright here in our own backyard\n\t\tdropping bombs!\n\n\t\t\t\tHAGEN\n\t\tWell, we could have expected it\n\t\tafter the embargo.\n\n\t\t\t\tSONNY\n\t\tHey!  Expect it or not, those Japs\n\t\tdon't have a right to drop bombs in\n\t\tour backyard.  Whose side you on?\n\n\t\t\t\tMAMA\n\t\tPlease, do we have to talk about\n\t\tthe war at the table?  On Christmas,\n\t\tmuch less.\n\nVIEW ON MICHAEL\n\nHe has been listening to this discussion.\n\n\t\t\t\tMICHAEL\n\t\tPop, I've decided I'm going to\n\t\tenlist.\n\nA quiet hush descends over the table, as though everyone\nknows the effect this will have on the old man.  Sonny tries\nto make light of it.\n\n\t\t\t\tSONNY\n\t\tKid, stay in college.  The girls\n\t\tare cuter, if you know what I mean.\n\n\t\t\t\tHAGEN\n\t\tPop had to pull a lot of strings to\n\t\tget you your deferment.\n\n\t\t\t\tMICHAEL\n\t\tI never asked for it; I don't want\n\t\tit.\n\nVIEW ON DON CORLEONE\n\nDisturbed; but wise and prudent.\n\n\t\t\t\tDON CORLEONE\n\t\tMy son wants to talk about this,\n\t\tand so we'll talk, but not at the\n\t\tdinner table.\n\nHe rises, and starts across the room toward his study.  Then\nhe looks back.\n\n\t\t\t\tDON CORLEONE\n\t\tMichael.\n\nHe disappears into his study.  Michael rises, glances around.\nPeople are generally tense over the situation.  Michael\nfollows his father into the study.\n\nINT. DON CORLEONE'S OLD STUDY - NIGHT\n\nThe Don closes the door behind his son, and then moves\nacross the room.  He stops at the little bar there, and\npours himself a brandy.\n\n\t\t\t\tDON CORLEONE\n\t\tWould you like some?\n\n\t\t\t\tMICHAEL\n\t\tNo, Dad.\n\n\t\t\t\tDON CORLEONE\n\t\tNow what is this talk about joining\n\t\tthe army?  Eh?\n\n\t\t\t\tMICHAEL\n\t\tIt's not talk; I'm doing it.\n\n\t\t\t\tDON CORLEONE\n\t\tYou would risk your life for\n\t\tstrangers?\n\n\t\t\t\tMICHAEL\n\t\tNot for strangers; for my country.\n\n\t\t\t\tDON CORLEONE\n\t\tAnyone not in your family, is a\n\t\tstranger.  Believe me, when trouble\n\t\tcomes, your country won't take care\n\t\tof you.\n\n\t\t\t\tMICHAEL\n\t\tThat's how it was in the old world,\n\t\tPop, but this is not Sicily.\n\n\t\t\t\tDON CORLEONE\n\t\tI know.  I know, Michael.  It's\n\t\tChristmas, your brothers and sister\n\t\tare all here -- we are happy.\n\t\tLet's not spoil this.  Go your own\n\t\tway, but when you are ready, come\n\t\tto me the way a son should.  I have\n\t\thopes for you...\n\nCLOSE VIEW ON MICHAEL\n\nlooking at his father with a mixture of great love, and also\nfear, and confusion.\n\n\t\t\t\tMICHAEL\n\t\tI won't be a man like you.\n\n\t\t\t\t\t\t\tDISSOLVE TO:\n\nEXT. THE TAHOE ESTATE - HIGH FULL VIEW - DAY\n\nThe leaves are blowing.  MUSIC comes up.\n\nMichael and his young son, Anthony, walk through the grounds\nof the estate, talking about things we cannot hear.\n
\n \n
\n
\n\n\n" }, { "path": "samples/go/decent/data/godfather3.html", "content": "\n\n
\n
\n\n\n                       THE GODFATHER PART III\n\n\n          \n                             Written by\n\n                 Mario Puzo & Francis Ford Coppola\n\n\n\n                                         FIRST DRAFT. 3/22/79                       \n\n                         \n\n\n                         FADE IN:\n\n                         BEFORE TITLES:\n          The screen is black. First we hear the sound of a\n          single trumpet playing slowly and sadly, the notes\n          faintly resonant as if echoing through the narrow\n          streets of some old hill village in Sicily. Now,\n          confusingly, we see a slant of light move past us\n          and another, and as our surroundings become more\n          visible, we discover that we are moving through a\n          pine forest lit by shafts of morning sunlight. Now,.\n          suddenly, we come out of the trees and find ourselves\n          on the Nevada shore of Lake Tahoe.\n\n          It is a clear, cold morning in September of Nineteen\n          Fifty-nine. We now see the Corleone compound at\n          lakeside, half-hidden by pines and firs. We move\n          toward it, past the gates and guard houses, past\n          the guest houses, past the kennels for the guard\n          dogs, and finally to the front driveway of the main\n          house where a conservative sedan is being loaded by\n          a Chauffeur and a pair. of large dark-headed men named\n          Al Neri and Rocco Lampone. Rocco limps slightly.\n          As suitcases are being placed in the trunk of the\n          car, the front door to the main house opens and Tom\n          Hagen, a trim, serious, balding man in a business\n          suit, appears He crosses to the corner of the house\n          and looks off.\n\n          HAGEN'S POINT OF VIEW - THE LAKE AND LKSIDE\n          Standing by the shore of the lake is a little boy,.\n          Tony Adams Corleone, aged about ten. The boy, dressed\n          for travel, is looking off at the lake, his back to-\n          ward us.\n\n          REVERSE ANGLE - ON TONY\n           As he looks out at the lake we might sense that he\n           is troubled and puzzled, although he is managing to\n           keep his expression stoic. Hagen can be seen in the\n          background, by the house 9 After a moment:\n\n                         \n\n                         \n\n                         \n\n                         \n\n          .T\n\n          2.\n\n                         CONTINIIED S\n\n                         HAGEN\n          Tony.\n\n                         (THEN)\n          Time to go.\n          There is a beat and Tony composes himself, turns and\n          moves up toward the house and Tom Hagen.\n\n          EXT. FRONT OF TAEOE HOUSE AND DRIVEWAY - DAY\n\n          The heavy, black sedan is loaded. Connie Corleane\n          and a Housekeeper, in uniform, are bringing Mary,\n          about five, out to the car. She too is dressed for\n          travelling. As they put her into the car --\n\n                         CONNIE\n          In you get ---\n\n                         MARY\n          Will. Daddy be at the airport?\n\n          - ANOTHER ANGLE\n          as Tom and Tony come up.\n\n                         HAGEN\n          No. Ere wanted me to tell you both\n          how sorry he was.\n\n                         TONY\n\n                         (TO CONNIE)\n          Aunt Connie ..- ?\n          Connie would apparently prefer to avoid answering any\n          questions.\n\n                         CONNIE\n          8e sure and give my love to your\n          mother.\n\n                         NERI\n          It isn't like you won't be back\n          from time to time.\n\n                         LAMPCNE\n          I' 11 bet we' 11 all. be together\n          for Christmas. Wait and see.\n          Tony gems into the car. He locks out the window to--\n          ward the house. We begin XXI-N Ti. TZS 2 C 2=17\"s\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         E\n\n          3.\n\n                         TEE DRIVEWAY\n          Hagen is the last one into the car. As it starts down\n          the driveway on its way to the airport in Reno, Connie,\n          Lampone, Neri and the Housekeeper all wave. The auto-\n          matic gate at the foot of the driveway opens. The\n          Guard at the gate flicks a half-salute as the Limousine\n          passes through.\n          TEE GROUP r n FONT OF EOQSE\n          As the group breaks, Connie and the Housekeeper going\n          back inside, Neri and Lampone drifting off, we pan up-\n          ward to a window in the second-story and zoom in.\n          Michael Corleone has moved the curtain aside with one\n          hand and is looking after the disappearing limousine,\n          his expression unreadable-but somehow sad.\n\n          ZNT. AIRLMPR - DAY\n          We are close on Tony, looking out the window of the\n          airliner, his own expression matching that of his\n          father.\n          EXT. LOGAN I_?i'Z'E.NATIONAL AI72ORT (STOCE) - DAY\n          as an airliner of the period comes in for a landing.\n\n          . YEW ENGLAND COUNITRYSIDE - DAY\n          A limousine is moving northbound up Interstate 93.\n          Maples on either side of the road are turning' red\n          and yellow. We pan the limousine past and continue\n          to pan to a sign marking the stateline between\n          Massachusetts and New Hampshire.\n          r .M. L:MC SINNE - DAY\n          It is almost night. Tony is looking out the window\n          as the limousine enters the little town of Hanover,\n          moves down past the Da. Louth College green on Eleazer\n          Wheelock Street and t7 rns right on north Main-\n           E`{'.r.'. XAY DAMS' ECUSr. - VIGRI'\n           We are an a post box, the name \"Adams\" on _ts Side.\n           We cull back _c '..:cl ude a modes +shi -e, ?aa-story\n          ! (ccrrT+_`?-? J\n\n                         \n\n                         \n\n                         \n\n                         \n\n          4.\n\n                         0 CONTI` :\n          clapboard house with giant elms and maples in the front\n          yard. The limousine stops outside the house.\n\n                         4\n          Under its fanlight, the front door swings open and\n          Ray Adams comes out. As her children run to her she\n          kneels and gathers them into her arms. We move in\n          close on Tony and as we hold, the '' ?NN TITLES A.ND\n          CREDITS are over and we begin to hear\n\n          SPEAKER' S VOICE\n          our Nation is and has been histor-\n          ically the symbol of freedom, of\n          justice and opportunity and its\n          pecui.iar strength is that no matter\n          what our individual background --\n\n          EXT. NAVY AIM MA.RI Z CORPS STADIUM - DAY\n\n          We are close on a young man, Anthony Adams (Corleone),\n          in the uniform of a Midshipman of the Naval Academy\n          at Annapolis. We are pulling back as the speaker --\n          who is Arne Grundellius, the Secretary of State --\n          continues, his accent faintly Scandinavian.\n\n          • CRUNDELLIUS' VOICE\n          -- there are no limits to the goals\n          to which we can each legitimately\n          aspire. And now, as an unpopular war\n          is ended in East Asia and we set our\n          sights on new goals, I leave you with\n          the words of another Sailor ---\n          We continue to pull back to discover that we are at\n          the Navy and Marine Corps stadium. It is a late Spring\n          day and in the early-middle Nineteen Seventies. The\n          Midshipmen, their parents and guests, are gathered\n          for- the graduation ceremonies.\n\n                         GRQNDELLIIIS\n          ' Our will is to keep the torch of\n          freedom burning for all. To this\n          solemn purpose we call on the young,\n          the brave and the strong, and the\n          free. Heed my call. Come to the\n          sea. Come sail with me.'\n\n                         (THEN)\n          I'= sure the entire 3ricade of Mid-\n          shimen recogr zes the words o John\n          ^p ?aui Vo nes\n\n                         \n\n                         \n\n                         \n\n                         \n\n          S.\n\n                         CONTINUED:\n          During this speech we find we have been. moving through\n          the stadium. We find Tony's mother, Ray, among the\n          guests. We also see Tony's sister, Mary, about twenty,\n          Tom Hagen and Al Neri.\n\n                         ANOTHER ANGLE\n          The speech is over and as the audience applauds, the\n          Superintendent of the Academy crosses an to Grundellius.\n\n          SIIPER2NTE11DENT\n          Thank you, lister Secretary -\n          As the Secretary of State crosses back to his seat,\n          the Superintendent addresses the microphone.\n          SUPERZNT-\".j. DENT\n          The following First Classmen will\n          step forward to receive their\n          diplomas.\n          The Superintendent consults a list which an Aid has\n          supplied- The Superintendent reads off. the first\n          t'o names, then\n\n                         0\n\n                         SUPERINTENDENT\n          Trident Scholar Anthony Adams\n          At the sound of his name Tony rises and moves toward-\n          the Speaker's platform. We\n\n                         DISSOLVE TO:\n\n          EXT. TEE SPEAF.R'S PL?TFORM - DAY\n\n          The final First Classman -has received the final diploma\n          and is.moving off as a Midshipman runs up to center\n          stage, and.-in accordance with long tradition ---\n\n                         MIDS$IPMAN\n          I propose three cheers for those\n          about to leave us. Eig hip!\n          (the Brigade answers\n          with a roar)\n\n                         SIN- HIP:\n\n                         (AGAIN)\n\n                         HIP :LIP\n          The Brigade answers for the th:i.rd time aid as the Mid-\n          sh pma.. runs cff, his =lace is taken by a RepreSen- by\n          tat.?.ve of the graduating class.\n\n                         0\n\n          (CCNT TL\"ZD) -\n\n                         \n\n                         \n\n                         \n\n                         \n\n          6.\n\n                         CON IN = :\n\n                         FIRST CLASSMAN\n          I propose three cheers for those\n          we leave behind. Hip hip\n\n                         (HURRAY)\n          Hip hip!\n\n                         (HURRAY)\n\n                         SIP HIP\n          ?EA aR=G TONY\n          As the graduating Midshipmen give their last full-\n          throated response and skim their hats into the air,\n          Tony -- half a beat behind the others -- does like-\n          wise. We pull back and pan upward to the explosion\n          of white hats arching through the air.\n\n                         CAT TO:\n\n          EXT. NAVAL ACADEMY YA?W -- DAY\n\n          We pick up a man named Stu Palmateer moving th-rough\n          the group of strolling Midshipmen and their guests.\n          Palmateer, is a poised, pleasant, tough man abcut\n          forty, dressed in the unifora of a Marine Caotai.n_\n          He spots Tony in a group with his guests, near the\n          0 statue of Tecumseh.\n\n          TONY AND THE OT!ERS.\n          Tony is standing with Kay, Mara Hagen and Al Nexi.\n          Tony turns as --\n\n                         (COMING UP)\n          Congratulations, Tony.\n\n                         TONY\n          Thank you, Sir.\n\n                         (THEN)\n          Captain Palmateer, I'd like to\n          present you to my Mother, Mrs.\n          Adams -- my. sister, star,{ ---\n\n                         (THEN)\n          And this is Mister 3agen, a very\n          old friend, and Al Teri - T used\n          to ride on his shoulders when I\n          was a little 1:ov-\n          Pa: ateer ,as gzeeted the ladies, shaken hands with\n          he men, ad l bi.: c appropriately. Nice to .eet u,\n          so cn.\n\n           (CL?? T )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         K\n\n          7.\n\n                         PALMATEER\n\n                         (TO XAY\n          I'm sorry to drag him off this\n          way, Mam.\n\n                         (TO TONY)\n          Ready?\n\n                         TONY\n          (nods, then)\n          See you all tonight.\n          Tony kisses his mother and he and Palmateer move off.\n\n          EXC. GEORGE, WASEIYGTCN BELTWAY - DAY\n          We pick up a car coming along the Washington Beltway\n          approaching the CIA turnoff near Langley, Virginia.\n          We pan with it, then continue to pan over to a road\n          sign which reads, \"Central Intelligence Agency\".\n\n          LIT. T3E CAR - DAY\n          Palmateer is driving. Tony sits next to h.m. They\n          make the turnoff to the CIA, go up the access road,\n          0 come to a stop at the entrance. As the Marine Guard\n          comes out of the guardhouse and up to the car, Palmateer\n          is taking out his identification.\n\n          • EXT. TEE MAIN CIA 3UILDLNG - DAY\n\n          Palmateer and Tony, on foot now, move up to the build-\n          ing. They move through the front doors.\n\n          =4T. . CIA BUILD LNG (LOBBY). - DAY\n          We are in he gigantic foyer of the CIA building. The\n          camera is focused on the CIA motto, etched boldly into\n          the white marble wall. It reads: \"YE SHALL M TOW T=\n\n          TRUTH AND THE TRIIT$ SEA\" MA=- YOU FREE\" .\n          We pan off the motto to pick up Palnateer and Tony\n          as they move through the great, columned rcom toward\n          the desk at the end. The Guard at the desk, seeing\n          Paimateer' s identification, signals h 1m and Tony up\n          to the Badge Office which i.s up a flight of steps on\n          the ricnt.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          S.\n          0 I . CIA BU=SIT MA.DGE OF-vICF,) - DAY\n          as Palmateer shows his identification to a Woman\n          behind the desk:\n\n                         PPL4MATEER\n          Captain Palzaateer.\n          (then, indic-\n\n                         ATES TONY)\n          Mister- Adams. We have an appoint-\n          ment with the D.D.P.\n          The Woman gives Palmateer his badge, checking the photo\n          on the badge against his face before she does so.\n          Palmateer initials the form she gives him as she gives\n          Tony his pass and stamps it in large letters: \"rust\n\n                         BE ACCCIMPANIED%\n          I=. CIA BtJILAING (LOBBY) - DAY\n          Within the foyer is another crate separating the foyer\n          from the inner sanctum of the building itself. We are\n          on the Gate Guards as they check the badges and passes\n          of those entering.\n          They nod Palmateer and Tony through the gate. We pan\n          them toward the bank of varicolored elevators. As\n          they go to the elevators we hear ---\n\n          PALMATEER' S VC ICE\n          This is a. preliminary interview,\n          not binding on either party. The\n          next step, if it's agreed to take\n          that step, would be a session with\n          the Assessment and Evaluation sec-\n          tion. Psychological testing.\n          Biographical data. Ends with a\n          polygraph test.\n\n          SIM. CIA BUILDING (SIXTH FLOOR) - DAY\n          We are on the elevator doors as they open and Palmateer\n          and Tony emerge. They walk down the corridor with its\n          bare, off-white walls. The floors are covered; with\n          green vinyl. Only the office doors add color. They\n          are painted variously, red, blue and yellow. As they\n          go down the long ha. Tway:\n\n          PATZA,E\n          ?.ssum ._^_c no serious orobiams cc-me\n          to light, y cv will be g..ven pro-\n          visional operao_ons apz=va_, effec-\n          0 tive fors .x non zhs . di x. q rich\n\n          (C JNT =ED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          9.\n\n                         CONTINUED :\n          PALMATEER (Cont' d)\n          time you would be sent to our fac-\n          ility at Camp Peary for a special\n          training program --\n          Tony and Pal.mateer enter the office at the far- end\n          of the hall.\n          iT. MOREECUSE'S OFFICE - DAY\n          It is a reasonably large office as befits one of the\n          top echelon CIA men. Morehouse, himself, sits at a\n          large desk. The seal of the Agency is an the wall\n          behind him, flanked by the National flag and the\n          Agency flag, an standards.\n          We are close on Thomas Morehouse, about fifty-five,\n          an imposing, silver-haired mar.. He locks up from a\n          dossier he's been studying, then:\n\n                         =REHOUSE\n          Would you have any objections to\n          being assigned to us, Mister Adams?\n\n          ANOT: R ANCLH\n          including Tony and Pal steer who sit across the desk\n          from Morehouse.\n\n                         TO:JY\n          That would depend on the duty,\n          Sir.\n\n                         MOREHOUSE\n\n                         (TO PALMATEER)\n          How much have you told ii=, Stu?\n\n          PAL,`?ATEE.'\n          Just that there was an assignment\n          we thought he'd be suited for.\n          MOREHOQSr.\n          All right.\n\n                         (THEN)\n          We've been authorized and funded\n          to carry out a too priority covert\n          operation in Latin America. Captain\n          Palmateer w lZ be Field Coordinator.\n          Your job wou.l.d be s.iasor..\n          There is a beat, then:\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         -R\n\n          10.\n\n                         CONTMILTED :\n\n                         MOBS' OTJSE\n          Something the matter, Mister Adams?\n\n                         TONY\n          Before we go into polygraphs and\n          so on, there's something you might\n          not know.\n          MCB EOUS.c\n          Fact is, we know quite a bit.\n\n                         (READS FROM\n\n                         DOSSIER)\n          Adams, Anthony. No middle initial.\n          Born New York Nursery and Childs'\n          Hospital, rebruary 3rd, 1951.\n          Mother and father separated.\n\n                         - TONY\n          Divorced.\n\n                          MOREEOUSE\n          Divorced.\n          Morehouse. scratches the correction into the dossier.\n          with a pencil, then:\n\n          MO.'3EHOUSE.\n          You were raised in Hanover, New\n          Hampshire. Your mother teaches\n          school. Name legally changed in\n          1963. Attended Phillips Exeter.\n          Lettered in ice hockey.\n\n                         TONY\n          And baseball.\n\n                         MOREEOUSE\n          And baseball.\n\n                         (NODS)\n          Your father is Michael Co=l--one, a\n          resident of Nevada. Re's principal\n          stockholder of Genco International,\n          a corporation that deals mainly in\n          hotels and casinos, but they also\n          have interests in an alive cil com-\n          pany, a charter airline, laundromats,\n          nursing homes, so on.\n\n                         TONY\n          Does this assignment have scmet-h4 n5\n          to do wit nv =a;, er?\n          tc NT.?w.•'?VZD}\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          ?I.\n\n                         0 CONTIYU:\n\n                         MOREHOUSE\n          only in so far as he has certain\n          business connections that might\n          be willing to help us.\n\n                         TONY\n          Y haven't seen my father since I\n          was ten years old.\n\n                         MOREEOUSE\n          No contact at all?\n\n                         TONY\n          I get a Christmas present and a\n          check or. my birthday --that's\n          about it. As far as I know, he's\n          retired. Doesn't see anyone.\n\n                         OREEOUSE M\n          Yes. That's why I was anxious to\n          have this meeting today -- while\n          Mister Hagen was still in Washington.\n\n                         ANOTHER VGI2\n          0 as Tony pauses, thinking. After a moment:\n\n                         TONY\n          I really don't know if I can help\n          you very much, Mister :Korehouse.\n\n                         MOREHOUSE\n          Let me ask you a personal question,\n          Tony.. Do you love your Country ,-\n          (holds up hand)\n          I'm not talking about t.'::e Nathan\n          Hale kind of thug. Just simply,\n          warts and all, do you dish this\n          Nation well?\n\n                         TONY\n          Yes, of course.\n\n                         MOREHCUSE\n          Well, what if w told you -- and\n          trying nct to be grandiose -- that\n          this one operation might very ael'6\n          insure peace on this :iemisnhere for\n          the next: f.i t r vea s . Maybe Mora.\n\n                         \n\n                         \n\n                         \n\n                         \n\n          ?2.\n\n          EXT. A RESTAURANT TERRACE (WASHINGTON) RIGHT\n\n          We are an the terrace of a pleasant restaurant over-\n          looking Washington D.C. The dome of the Capitol\n          building is impressively floodlit as are the Washi:g-\n          ton monument and the White :louse.\n          Tony and Hagen are leaning on the terrace railing,\n          looking out. A Waiter has just finished pouring coffee\n          for them. As the Waiter moves off, Tony throws a look\n          at $agen, then:\n\n                         TONY\n          Well?\n\n          HA,EN\n          Let me make sure I'm clear on this.\n          You're telling me that the Govern-\n          ment would like the Corleone family\n          to perfatm a service for them --\n          probably involving some friends of\n\n                         R\n          ours in Latin America.\n\n                         TONY\n\n                          RIGHT_\n\n                          AAGEN\n          s Where? Who's concerned? What\n          would it entail, speci.-46ically?\n\n                         TOUR\n          I don't know that vet.\n\n                         EMMIT\n          When will you be able to tell me\n          these things?\n\n                         TONY\n          As soon as they decide I.'m not a\n          Russian spy.\n\n          A IOTSM XYGLE\n          as Eaaen smiles, shakes his head. Sureaucracy..\n\n                         TONY\n          What they'd like to fi:d out. right\n          now is: Wctzd you be interested?\n\n                         3AGEI\n          :' L... :.et you .c.-tct?•.\n\n                         TCN??Y\n          no ?4U save to cheCx i :tv fat er\n          (c N'\" =NL ?D )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         IT\n\n          13.\n\n                         CONTIWED\n\n                         HAGEN\n\n                         TONY\n\n                         HAGEN\n\n                         TONY\n          I think about Tahoe sometimes.\n          Is the fishing still good up\n          there?\n\n                         0 HAGZ\n          I suppose so. I'm in Vegas most\n          of the time..\n\n                         F\n\n                         TONY\n          Those were good days.\n\n                         EAGEN\n          Yes. They were.\n\n                         ANOTBER ANGLE\n          As Al Teri cones up, looking at his watch.\n\n                         NERI\n          You still want to catch that\n          shuttle to New York?\n\n                         HAGEN\n          Yes. Thank you, Al.\n\n                         (TO TONY)\n          Where can I reach you?\n          Tony scribbles an address on a card and hands it to\n          Hagen. As'he looks at it:\n\n                         HAGEZY\n          Q.S. Navy Research 'acili.ty.\n\n                         (THEN)\n          Research on what?\n\n                         TONY\n          Nothing. it-'s a dummy out: i,t.\n          That phone rings in Langley,\n          but they'll know where I am.\n          f f Agen nods and guts the card in his wallet a we\n\n                         TO:\n\n                         \n\n                         \n\n                         \n\n                         \n          l4.\n          M. tNTERF.OGATION ROOM ONE (CIA) -- DAV\n          Tony is completing a test, fitting blocks together\n          against time. The First•Interrogator is watching\n          impassively, stopwatch in hand.\n          Tony finishes, straightens. The Interrogator clicks\n          the stopwatch impassively, giving no indication\n          whether or not Tony has passed the test.\n\n                         CUT TO:\n\n          INT. INTERROGATION ROOM TWO (CIA) - DAY\n\n          The Second Interrogator, a psychiatrist, is seated\n\n                         I\n          behind a desk rocking at a little steeple that he's\n          made of his hands.\n\n                         2ND INTERROGATOR\n          And you were never curious?\n\n          ?NCTMM NGLE\n          including Tony who sits across the desk from the\n          Interrogator.\n\n                         TONY\n          About what?\n\n                         2ND INTEMOGATOR\n          Why your father sent you away.\n          You never wondered about it?\n          You must have thought something.\n\n                         TONY\n          I thought he had his reasons.\n\n                         2ND INTERROGATOR\n          And you don't feel any resentment?\n\n                         TONY\n\n          110.\n\n          2ND 7.NNT .'RR0GATOR\n          What do you feel?\n\n                         TONY\n          Nothi..ng\n          The Secor_d Iuterro,acor glandes •cver. The ~t ace of\n          anger in Tor_v`s _as- response has told hiz. and us\n          somethinc .\n          C\"^' TO: ui-\n\n                         \n\n                         \n\n                         \n\n                         \n\n          15.\n          0 T. L\"tT-_.RROGATION ROOM TER= - DAY\n          Tony sits in a chair in a small room with acoustical\n          tile on the walls and ceiling. Behind hint is a desk-\n          like structure with a built-in apparatus of dials,\n          graph paper and odd, narrow metal pens.\n          Tony is connected to the desk ensemble by three appara-\n          tuses: a blood pressure cuff attached to his arm,. an\n          accordian tube around the chest to measure changes in\n          breathing rhythms; a hand-held device with electrodes\n          which measures changes in perspiration or galvanic skin\n          response.\n          The Third Interrogator sits at the desk behind Tony,\n          asking questions slowly and checking the three styluses\n          on the rolling graphs.\n\n                         3RD INTERROGATOR\n          Have you ever visited a Ccnsmunist\n          Country?\n\n                         TONY\n          No.\n\n          3RD INTER3CGATOR\n          Have you ever belonged to a Commu-\n          9 aist Organization?\n\n                          TONY\n           No.\n\n          3RD IN`'ERROGATOR\n          Are you telling the ruth?\n\n                         TONY\n          Yes.\n\n                         3RD INTERROGATOR\n          Have you ever had a homosexual ex-\n          perience?\n          Tony turns and looks at the Third Interrogator.\n\n                         TONY\n          No. Save vou?\n          The ;\"hi=d Interrogator tenses angrily for a ncment,\n\n                         THEN:\n           31-I0 :N'_r'ERRCGATCR\n           Its esser zia1 that you face the\n          i f lar and answer the ClUest'_oi:s ?es\n          at 1o.\n          (CCNT'Z Nt D)\n\n                         \n\n                         \n\n                         \n\n                         \n\n          16.\n\n                         CCNTZ''' 'ITFED :\n          Tony turns back to the wall, then:\n\n          3RD I:YT RROGATOR\n          Do you have any friends in the\n          Communist Party?\n\n                         TONY\n          No.\n\n                         CTT TO:\n          INT. I:YT...c2ROGATZON ROOM TS-M2 - DAY\n          It is later. Tony is on his feet getting ready to\n          leave. as the Third Interrogator is checking over the\n          graphs with their red ink squiggles.\n\n                         3RD TERROGATOR\n          I get a high galvanic response on\n          question twenty-three.\n\n                         (CHECKS SECOND\n\n                         GRAPH)\n          Z also have agitation indicated\n          on your cardio tracing on the\n          same question: 'Do you have any\n          close friends in. the Communist\n          Party?' Your answer was negative.\n          Would you like to amend that?\n          Tony has rolled down his sleeve and out on his coat.\n          He pauses at the door, then:\n\n                         TONY\n          It's my roommate up at Eseter.\n          Phil Bodeen.\n\n          3RD INT.'RRCGATCR\n\n          He's a Ma.- xist?\n\n                         TONY\n          Z don't know.\n\n                         (GRINS)\n          But last time r saw him he had an\n          American flag sewn to --..e seat\n          of his pants.\n\n          3RD INT=.RCGATOR\n          And how did 'cu feel about tat?\n\n                         TONY\n          ( sh..?ug s\n           L gis y y r r 1 1 +I GrL MCI n\n          E (CON ` tZD)\n\n                         \n\n                         \n\n                         \n\n                         \n\n          17.\n\n                         WONT : F\n           3RD >arr.` 3ROGATOR\n           That's interesting.\n\n                          TONV\n          what?\n\n                         3RD INTERROGATOR\n          You think it's perfectly all right\n          to sew the Nation's flag to the\n          seat of your pants?\n\n                         TONY\n          As a matter of fact Z don't.\n\n                         (THEN)\n          But the Supreme Court does. Syza-\n          bclic freedom of speach. Protec\n          ted under the First Amendment.\n          As Tony tuffs to go :\n\n                         3RD INTERROGATOR\n          Adam me?\n\n                         TONY\n\n                         (TURNS BACK)\n          Sir?\n\n                         0\n\n                         3RD INTERROGATOR\n          I take it you're prepared to die for\n          that right?\n\n                         TOUR\n          No, Sir. Dying doesn't fit in with\n          my plans at all.\n\n          3RD M47TRROGATOR\n          Just what are your plans, Mister Adams?\n\n                         TONY\n          T plan on passing this thing\n\n                         CUT TO:\n\n          TNT. HANDBALL COURT - DAY\n          Tony and Stu Palszateer are in sweat clothes, playing\n          a hard, ma-mercy game Of handball. ?almateer is close\n          as he hits the ball:\n\n          PAT.MATE R\n\n                         YOU WILL\n\n                         \n\n                         \n\n                         \n\n                         \n          COHTZ.wE :\n          We whip pan to Tony, returning the shot.\n\n                         TONY\n          What makes you think so?\n\n          PALMAT Z.P..\n          You've got friends in high places,\n          chino(.\n          As Tony puts one away:\n          PAI.6VxT..,ER\n          Shot..\n\n                         P (THEN)\n          Believe me, you'll be reading-in\n          on the project by next week.\n\n                         CUT TO:\n          =T. IBO CATION BUILDING (St.PSEY CITY) - DAY\n          We are on the front entrance of the building owned by\n          the International Brotherhood of Dockworkers. This is\n          the National Headquarters Building, so indentified by\n          a plaque of some kind. Tom Hagen enters this buildinq\n          0 followed ?;y Al, Neri.\n\n          MM. BRA.DY' S OUTER OFT= - DAY\n          The President of the Union, Patrick Brady, a large,\n          red-faced, hearty man, comes bursting out of a door\n          and crosses up to Hagen and Neri.\n\n                         HAGEN\n          Tom! Alberto ! Come in. Come in\n          This way.\n          I (to Secretary)\n           No calls, :Maggie.\n\n                         1\n\n          =41Z. CCNFEP,.N=_ BOOM - DAY\n          This is a very plush room with heavy carpets, a long\n          polished table and a large portrait of the Union's ex-\n          president, Danny Devito. Under this portrait is a bar\n          at which Brady stands pouring drinks into crystal.\n          glasses. -\n          As he t r=ls the drin s, y Yea the w..T..rzh Ner+\n\n                          K\n           then as he gi\"Ies the secon d one to Iacer: °-\n           (CDNT t D)\n\n                         \n\n                         \n\n                         \n\n                         \n\n          18.\n\n                         CONTINVZD\n\n                         BRADY\n          You're looking good, Tom. How do\n          you keep your belly so flat?\n\n                         EAGEN\n          Mainly trying to rum you down.\n\n                         $RADY\n          Do You think this outfit =ins\n          itself?\n\n                         RAG=\n          I think you're stalling us, Pat.\n\n                         BRADY\n          StaLLJag?\n\n                         (TO UTERI)\n          Will you listen to this guy?\n\n                         (GRINS AT\n\n                         HAGEN)\n          You want to know when I got back\n          from New Orleans? Ten-thi.-ty last\n          night. Big problems.\n\n                         3AGE1\n          I understand and S sympathize, but\n          you've had our proposal for a month ---\n\n                         ANOTHER ANCE\n          Brady crosses to the head of the board table where\n          he opens a :older and starts riffling through some\n          papers. As he does so, Eagen opens his briefcase.\n\n                         BAGEN\n          '1f you've misplaced it, I have a\n          copy of the package plus. a summary\n          of Genco International's assets and\n          projected profits based on audited\n          financial statements with additional\n          data supplied by our Comptroller.\n          I also have the plans and estimates.\n\n                         BRADY\n          okay. okay. I found 4-\n\n                         (THEN)\n          You want a i'i,ty M :?lion do? la=\n          line of credit to be. granted is\n          full to Genco azternatiorai and\n          Subsidiaries and so on and so -73rth --\n\n                         (-OAKS AN)\n          For a. hotel in tlar tic City? 7-'s\n          a lot of money, '^o t.\n          (C.` CN'i _.dL: E^u )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          19.\n\n                         CONT'VCZD :\n\n                         BAG T\n          Not if they vote in local option\n          gambling. And they will.\n\n                         BRADY\n          That's beside the point. ?act\n          is, those days are over when Danny\n          used to hand it out like so much\n          free lunch. There's been a big\n          reduction in our c fitment to new\n          construction loans.\n\n                         RAGR`\n          How Long has that policy been in\n          effect?\n\n                         BRADY\n          Let me read you something.\n\n          ANGTIMR ANG '.E\n          as Brady withdraws a newspaper cut-out, obviously an\n          editorial, from the folder.\n\n                         3RADY\n          S It's headed2 Take the hcod out\n          of the Brotherhood.\n\n                         (LOOKS UP)\n          Cute?\n\n                         (READS)\n          'It. is precisely men like Patrick\n          Brady -- who took over the Pres-\n          idency of the I3D when the former\n          President, Danny Devito was packed\n          off to prison, who must be watched\n          by the SEC. Because of the tremen-\n          dous economic power of the anion\n          Pension Funds, these men -- with\n          their syndicate connections --- are\n          putting the Underworld in a position\n          to dominate the American economy\n\n                         (THEN)\n          What more can I tell vou?\n\n                         A\n          You can tell me if the answer is\n          yes or no.\n          As Brady crosses to the 4a!-! where ?.e ? arge srt dio\n          Por-trai.t of Danny Devito, a tcugh-lcok zc __ctle Tay,\n\n                         E\n          (CONTI ITv:.D )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          20.\n          CCNTI tUEI? :\n          is framed in a place of honor. After a moment Brady\n          shakes his head and turns back to Hagen.\n\n                         BRADY\n          t'm sorry, Tom. :Such as I love and\n          respect cur former president, I\n          don't have any burning urge to end\n          up rooming with him at Leavenworth.\n\n                         EAGW\n          Before you give me your final re-\n          fusal., I'd like to say one thing ---\n\n                         BRADY\n          You've just had my final refusal,\n          Hagen. The answer is no.\n          i ANOTHER ANGLZ\n          as Hagen looks at Brady for a moment, then starts\n          putting papers back into his briefcase. As he does so:\n\n                         BRADY\n          No hard feelings. It's just a\n          policy decision of the Board.\n\n                         BAGEN\n          I understand and I thank you for\n          your time ---\n\n                         NERI\n\n                         (TO BRADY)\n          I'll see you around, Pat.\n\n                         BRADY\n          What's that supposed to mean?\n\n                         M\n          It means I'll see you around.\n\n                         BRADY\n          What am t supposed to do? Get\n          scared? Piss in my pants?\n          ,,To one is trfing to intimidate you.\n          The Corleone :° ti.ly doesn't do\n          business that wacr,\n\n                         3RADY\n          T h e Cori eons 4ami? y doesn't do\n          (C ON\"r y zt, D )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          21.\n\n          CONTZ4\"D :.\n\n                         BRADY (CONT'D)\n          a hell of a lot of business, period.\n          The way Z hear it, you're finished.\n          With Mike locked away in a rubber\n          room someplace, and a two million\n          dollar tax lein on your Vegas prop-\n          erty, you. got the balls to come in\n          here and try to run muscle on me.\n          Get out of here. Both of you.\n\n                         SAGE N\n          We were just going.\n          Hagen has packed his papers into his briefcase. Now\n          he nods at Neri and as both turn, and start toward the\n\n                         DOOR:\n\n                         BRADY\n          Wait a minute.\n          Hagen turns back. Brady comes up with the Genco Inte.r-\n          national loan application.\n\n                         BRADY\n          Take this along with you in case\n          you run out of toilet paper on\n          0 the flight back to Vegas.\n          3agen takes the application from Brady, then cuietly:\n          E ,Gr 3\n          Don't ever think that the Corleone\n          Tamily is finished, Mister Brady.\n          That would be a mistake.\n          Hagen and Neri now turn and exit. Brady is looking\n          after them. Gradually the bravado drains from his\n          big, pink, Irish face and he crosses to the telephone\n          and picks it up.\n\n                         BRADY\n          Gat me Sam Maatrocina.\n\n                          CITT 1 '60\n          ELT . MAATRCCIii .' S YACET (LONG ZS LADIM MARIMTl) - DAY\n          We are close on the hatchway to the main Salon as\n          Sam iaatroci_na, the slick, sharp, middle-aged Don\n          of a. powerf :.L New York faoi i y, comes up into shot\n          and pauses, smiling cff:\n          CONT==,cm\n\n                         I\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         4\n\n          22.\n\n                         CONTSNVED :\n\n                         MAATROCINA\n          There's a trick to it, Pat.\n\n          ANOT$ER ANGLE\n          including the canopied afterdeck of %laatrocina's\n          lovely yacht, moored among others of its type in\n          a splendid :forth Shore Long Island marina.\n          Pat Brady, looking out of place in his suit and\n          city shoes,atands trying to light a cigar with a\n          table lighter.\n\n                         I BRADY\n          Hello Sam. I'm sorry to have\n          bothered you.\n          We pull back slightly as Maatrocina --- in neat yacht-\n          ing whites and deck shoes - comes up followed by a\n          cold-looking man, Ralph Augusto, Maatrocina's hood.\n\n                         TROC\n          No bother at all.\n\n                         (TO AUGUSTO)\n          Light the man's cigar for him,\n          Ralph..\n          As Augusta comes up, takes the lighter from Brady and\n          sets about the business of lighting the big, Union\n\n                         LEADER'S CIGAR:\n\n                         MAATRCCINA\n          So what's our friend Tcm Hagen sp\n          to these days?\n\n                         BRADY\n          Still shopping around for that loan.\n\n                         MAATROCINA\n          Lots of luck to him.\n\n                         BRADY\n          Thank you, Ralph.\n          Augusta, having lit Brady's cigar, nods expression-\n          lessly and sits as:\n\n                         3RADY\n          T h ey ;ri ed to =1=_-W a scare at me.\n          w^1TI\\Z'v.Z0 ;\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          23.\n\n                         CONT=NRED :\n\n                         MAATROCIIA\n          Don't worry about it.\n\n                         BRADY\n          Something about Hagen kind of\n          bothered me.\n          Maatrocina offers a platter of grapes to Brady.\n\n                         MAATRCC=TA\n          You like a grape?\n\n                         BRADY\n          No thank you.\n\n                         I\n\n                         M ATROCINA\n          I'll te12 you haw tough Hagen is --\n\n                         {EATS A\n\n                         GRAPE)\n          Ralph Auqusto will make in squat\n          down in the middle of Times Square\n          in the rush hour and take a shit.\n\n                         BRADY\n          Yeah? What about Al Neri?\n\n                         0\n\n                         MAATROCINA\n          I tell you don't worry -- don't\n          worry. You stick with the -Maatro-\n          ciza famly you're safe as church.\n\n                         (THEN)\n          That I promise you on my mother's\n          grave.\n          As Maatrocina leans across with the grapes again:\n\n                         MAATROCINA\n          Do me a favor.. One grape. I grow\n          'em at my own place. They're de-\n          licious.\n          As Brady takes a grape, Maatrocina looks over at Ralph\n\n                         AUGUSTO:\n          ANOTHER ANGI. - FEATURING AUGu STO\n          As he nods almost iimperceptibly, gets un and goes,\n\n                         I TO:\n\n                         E\n\n                         \n\n                         \n\n                         \n\n                         \n\n          24.\n\n          0 NETWORK NEWSROOM (WASBINGTON) - DAY\n          We pick up Elizabeth Ann Dunne, about 30. She is a\n          television personality, mainly an interviewer of\n          celebrities, although sometime a Newsweman and talk-\n          show performer. She is poised, attractive, warm,\n          hirp. She is coming out of an office. She crosses\n          through the newsroom with its teletypes and general\n          sense of activity. A Cameraman comes up to her:\n\n                         CAMERAMAN\n          You ready, Liz?\n\n                         ELIZA$ETE\n          f i fteen minutes. Out in front.\n          We follow her out of the newsroom, dawn a hallway and\n          around a corner and in through a. door rrar!ced, \"Projec-\n          tion Room One\".\n\n          =T. PROJECTION ROOM ONE\n          Elizabeth comes into the projection room, pausing in\n          the doorway as her eyes adjust to the change in light.\n\n          A? M OT R ANGLE\n          We see that Tony, in miform, is the only otter person\n          in the projection 'roem. Tony takes out a cigarette\n          lighter.\n\n                         TONY\n          Here.\n          As Tony flicks the lighter on:\n\n                         ELTZAAET\n          Thank you.\n\n                         TONY\n          My name's Adams. Tony Adams.\n\n                         (THEN)\n          Z know who you are Miss Dunne.\n          As Elizabeth finds a seat, we see that what is being\n          run i.n this Projection room is an interview ime-troreen\n          Elizabeth and a large, attractive, Latin-American\n          political leader named Arnando. ?Tidal. They sit to-\n          get er at the --col area of a lur..ar r hotel. Tidal wears\n          a fatigue uni.fc= without any insignia whatsoever- The\n          image of Elizabeth an screen is saving-\n\n                         (CONTINUED)\n\n                         C\n\n                         \n\n                         \n\n                         \n\n                         \n\n          25.\n\n                         0 CONTNUED:\n\n                         ELIZABETH'S VCIC\n          Senor Presidente, there are still\n          those who say that in spite of the\n          advances you've made, your Country\n          still falls short of the Democratic\n\n                         IDEAL ---\n\n                         VIDAL\n          Obviously. However, the convulsions\n          that my Countz went through four\n          years ago last February were not so\n          much a revolution as they were a --\n\n                         (PAUSES)\n          I am thinking of the labors of\n          I Herculi.o --- in the stable\n\n                          ELIZABETH'S VOZC\n           A cleansing.\n\n                         S VIDAL\n          Exactly. Muchas gracias. My Country\n          was befouled by the corruption of\n          its leaders and their exploitation\n          at the people. A hard cleansing was\n          needed and sometimes that is painful --\n          but the pain is over and now Z am\n          0 hoping. that qty little Cc=try and\n          your great Yaticn can once again be\n\n                         FRIENDS ----\n\n                         ELIZABETH'S VOICE\n          I'm sure a lot of pecple say Amen\n          to that, Senor Presidente.\n          On the screen the scene has shifted to an attractive\n          beach area where Vidal, in a wet suit, is adjusting\n          his'-scuba gear preparitory to diving. Admiring child--\n          red and Elizabeth Ann. Duane watch. Over this:\n\n                         J\n\n                         ELIZABETH'S VOICE\n          For a glimpse of another facet of\n          Armando Vidal's nat•.ire, we spent the\n          last day at the beach at Finca del\n          Sol Where El Presidente exhibited\n          his skills as a scuba diver -- one\n          of his favorite hobbies.\n          On the screen Vidal --ousels a kid' s ha..- and crosses\n          into the water. As he wades out, Tony turns to\n          Elizabeth.\n\n          (C CNT=IL EC ;\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         T\n\n          25.\n\n                         CONTZLVTIED :\n\n                         TONY\n          When will this be shown?\n\n                         ELIZABETH\n          Sometime in the Fall.\n          While Elizabeth, in the projection room, is saying\n          this. her image on the screen has turned to the T4\n          camera and is saying:\n\n                         ELIZABETB'S VOICE\n          This is Elizabeth Ann Dunne coming\n          to you from Finca del Sol where --\n          in an exclusive interview -- El\n          Presidents, Armando Vidal, has just\n          extended the hand of friendship\n          from quote, 'his little Country to\n          our great Nation . '\n\n                         I\n\n                         ANOTESR ANGLE\n          The reel is over. The screen goes blank and the\n          lights in the projection room go on.\n\n          ELIZA.BET?I\n          40 What is it that you're working on? 16\n          Some sort of'a psychological pro-\n          file on Vidal for the Navy Depart-\n          ment?\n\n                         TONY\n          Right. There's a couple of ques-\n          tions I wanted to ask you, if that's\n          okay.\n\n                         (THEN)\n          What are you doing about lunch?\n\n                         ELIZABETH\n          I ignoring it. I've got to pick up\n          some shots around town, but you're\n          welcome to come along.\n          They are at the projection: room door. As Tony cpens\n          it for her she causes, then:\n\n                         ELIZABET3\n          You ever play any baseball, :lister\n          Adams?\n\n                         TONY\n          Sow did you guess h.at?\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n                         I\n\n          26-A.\n\n                         CCNTTX= :\n\n                         ELIZABETH\n          I saw you pitch a no-hitter against\n          Army two years ago.\n\n                         TONY\n          You're kidding.\n\n          TI' ZA3ETH\n          L don't kid about no hitters..\n          Elizabeth exits. Tony follows. We ---\n\n          CII:' TO:\n\n          EX' T. MC= VE NQN - DAY\n          as a troup of Boy Scouts goes past us and clears the\n          Visitor's Gate, exposing the bowling green, the court-\n          yard and far down, framed by giant black oak and maple\n          trees, we see the `Mansion with its pure lines and\n\n                         SIMPLE ELEGANCE-\n          Tony and Elizabeth appear., followed by the Cameraman\n          and one or t'c more. As Tony and Elizabeth cross into\n          the beautifully kept grounds of the old Plantation ----\n\n                         TONY\n          None of my business, but what were\n          you. doing at an Army-Navy ball. game?\n\n                         ELIZABETH\n          We were putting together a special\n          on Arne Grundellius. He'd just been\n          appointed Secretary of State. Ee\n          threw the first ball that day.\n\n                         TONY\n          You've got a pretty good memory.\n\n                         ELIZABETH\n          And you.'ve got a pretty good slider.\n          But L've got to be honest, you threw\n          a lot of junk in the last two innings .\n          As Tony shoots her a look:.\n\n                         ELIZABETH\n          My old man was on the Spar rs' Desk\n          of the Boston Globe for wwent;r-five\n\n                         (CONTMM-S-0\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n                         CONTINUES:\n          ELIZABETH (Cent' d)\n          years. Z was practically raised at\n          Fenwick Park in the Carl Yastremski\n\n                         DAYS-\n          As Elizabeth crosses and starts working out a shot\n          with the Cameraman, Tony is eying her speculatively.\n\n                         CUT TO:\n\n          EXT. TEE CRYPT OF GEORGE WASHINGTON - DAY\n\n          We are at. the littleopen-fronted white marble burial\n          vault of George Washington. . We pull back to include\n          Tony as he looks at the sarcophagus.\n\n          ELIZABET3'S VOICE\n          Tony? We're finished ----\n          As Tony turns, we pull back and pan to include Eliza-\n          beth, who stands in the little leafy path leading\n          from the vault. As Tony crosses and ;tins her, we\n\n                         CUT TO:\n          0 T. SPACE AND FL GF ' M SEt M - DAY\n          We are on the balcony as Elizabeth sets up a shot\n          frog: the Wright Brother's \"Flyer\" panning to\n          \"The. Spirit of St. Louis\", and then to one of the\n          Space Capsules. Tony is watching Elizabeth. She\n          looks over, catches his eye, smiles.\n\n                         CUT TO :\n\n          EXT. GEORGETOWN - NIGET\n\n          as Tony and Elizabeth approach her apartment on a\n          quiet, Georgetown side street. The cold, faintly\n          blue light of the- street lamp at the corner throws\n          leaf shadows on their faces as they come up to the\n          front door..\n\n          CLOSER ---AT TEE DOOR\n          She opens the door, steps to one side and gestures\n          him in.\n\n                         MZZA3ETE\n          One dr iak, ckayr?\n\n                         \n\n                         \n\n                         \n\n                         \n\n          26-C.\n\n          INT. ELIZABETH'S APARTMENT - NIGHT\n\n          We are close on a picture of Elizabeth and Arne\n          Grundellius, the Secretary of state, whom we might\n          remember by sight as the Speaker at Tony's gradu-\n          ation. The picture is in.€orral, taken at an open-\n          ing night at Kennedy Center.\n          We pull back to include Tony, locking at the picture\n          as Elizabeth comes up with a couple of drinks.\n\n                         TONY\n          You and Grundeilias?\n\n          EZ,I ZA.3E'_'E\n          At Kennedy Center.\n\n                         TONY\n          Ee really gets around.\n\n                         ELIZABETH\n          Well, he got around nee anyway.\n\n                         TONY\n          Oaps.\n\n                         ELIZABE'-\"3\n          No ha=, no foul.\n\n                         (THEN)\n          r wasn't fighting him off very\n          hard.\n\n                         ANOTHER ANGLE\n          As Elizabeth picks up another picture, this one a\n          framed studio portrait, inscribed, of the Secretary.\n          As she locks at it:\n\n                         ELIZA3ETE\n          We had quite a little thing going\n          .for a while. I think he ac tuually\n          gave up two starlettes and a bare-\n          back rider -- temporarily.\n\n                         TONY\n          You stir see h,m?\n\n          EI,IZ.ABETE\n          Once in a while.\n\n                         (THEN LAUGHSY\n          Said she ?ri.s4f?i? r.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          26-D.\n\n                         CONTIDI = :\n\n                         TONY\n          Z was just going.\n\n                         ELIZABETH\n          Don't mind me. I'm just your basic\n          Boston Irish. We bruise easily and\n          heal slowly.\n\n                         CD TO:\n\n          EXT. ELIZABETH'S FRONT DOOR - NIGHT\n\n          as Tony cones out. Elizabeth stands in the doorway.\n\n                         ELIZABETH\n          Call. me.\n\n                         TONY\n\n                         (NODS)\n          Goodnight.\n          Tony goes down the street. She watches after him\n          as we -«\n\n                         CUT TO:\n          rXT THE BATTERY (NEW YORK CITY) - NIGHT\n          A limousine comes down past the Battery Park and moves\n          toward the Staten Island Ferry Building. It is about\n          ten o'clock at night.\n\n          I INT. THE L=!OUSINE - NIGHT\n\n          Frankie Rizzi, about 30, is driving. Frankie is the\n          son of Connie Corleone and Carlo Rizzi. In the back-\n          seat is Al Neri. Frankie looks off toward the river.\n          We can see. the ferry coming in.\n\n                         1\n\n                         FRANKII\n          Here it comes now.\n\n                         ANOTHER ANGLE\n          as the fear comes up. Noses into the slip. Creak\n          of pilings. The limousine drives onto the ferry.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          27.\n          IYT. : HE LS.' CUStNE - NIGHT\n          Frankie is listening to the baseball scores. Neri is\n          nervously checking his watch. A few passengers are\n          boarding. There are no other cars.\n\n          ANOTHER ANC LE\n          Salf a dozen loose, Puerto Rican kids come bopping\n          up the automobile deck. one is listening to a radio\n          that he holds to his ear. He crosses up to Frankie.\n\n          PUERTO RICAN RID\n          Hey Mister, you got a cigarette?\n\n                         PRANRIE\n          I don'-.t ;smcke.\n\n          PUERTO RICAN RID\n          You got a dollar?\n\n                         FRANRZIR\n          Beat it.\n\n          ANOTHER ANGT.E\n\n                         I\n          as Neri, who hasn't been paying attention to the kids,\n          now looks up in annoyance.\n\n                         NERI\n          Give him a _--\n          Suddenly veri breaks cg!, sensing something wrong.\n          The other Puerto Ricans are surrounding the limousine.\n          As Neri dives for the door of the car ---\n\n                         ;N1ERI\n\n                         LOOK OUT:\n\n                         ANOTHER ANGLE\n          Guns have appeared in the hands of the Puerto Ricans\n          and they start to blast at the limousine from outside.\n          The ferry whistle is blowing.\n\n                         MWTT ANGLZ\n          Neri :its the deck, a !4 na , cones t _t°? -- --u n :t a-\n          blast:.:.CT.\n\n                         \n\n                         \n\n                         \n\n                         \n\n          28.\n\n                         ANOTHER ANGLE\n          One of the Puerto Rican kids is hit squarely in the\n          chest. He flies back, azms up, legs sz readea_gI d,\n          like he was hit in the chest with a baseball bat.\n\n                         ON FRANRSE\n          Prankie is out of the car, reaching for his gun. Before\n          he can get it clear, he is hit in the shoulder and spun\n          to the splintery, oily deck.\n\n                         ANOTHER ANGZE\n          As a Puerto Rican goes to finish off Frankie, Al Neri\n          blows him away.\n\n          REVERSE Z.\n          as two other Puerto Ricans blast Neri simultaneously.\n          Neri's eyes go wide. He coughs.: A big, stringy sob\n          of blood appears in his mouth, vomits out onto his\n          shirtfront.\n\n                         ON NERI\n          He goes down to his knees, tries to raise his grin\n          for one last shot. A Puerto Rican grins and putting\n          his gun. an inch from Neri's face, pulls thetrigger.\n\n                         ANCTEER ANGLE\n          People are screaming, running. The'ferry is starting\n          to pull from the dock. The four remaining Puerto\n          Ricans run for the end of the boat, make the Leap\n          over the churning water from the ferry to the landing.\n\n          FULL SHOT - FMM LANCING\n           The ferry boat is still\n           pulling away as the `our\n           Puerto Ricans land on t h - e he pier and disappear into the\n           night. The ferry boat's whistle is blowing shrilly.\n           Off its starboard bow we can see the Statue c- Leber _r.\n           Over this we hear ---\n\n           -P?3T__' -.ST' S `JOIC\n           I am the ?aa^tt,-cc±±^; +n and the r. - ' 1\n           and he teat believeth in Me, al -\n           though he be dead, shall live ---\n\n                         E\n          DI SSO?.. iE TO :\n\n                         \n\n                         \n\n                         \n\n                         \n\n          29.\n          EXT. ITALIAN-CAT3CLiC GRAVEY-XI II (LONG ISLA `7D) -- DAY - -\n          We are on the ornate tombstone of Don Vito Carlecne-,\n          the Godfather. We are pulling back from it as\n\n                         PR2EST\n\n                         (CONTINUING)\n          And' everyone that liveth and be-\n          lieveth in pie shall not die forever.\n          We have pulled back to include the buria.L services\n          for Neri. At the graveside are Tony, in dress blues,\n          Hagen, Rocco Lampone and others.\n\n                         ANOTHER ANGLE\n          The services finish and the group breaks into smaller\n          informal, groupings. We move to Tom Hagen and Tony, who\n          have drifted to one side.\n          aNOTH.ER ANGLE\n          as Hagen indicates a moon-faced man, Imberto Croce,\n          about sixty, who is approaching them.\n\n                         HAGEN\n          Umberto Croce out of Tama. He\n          took over the whole Florida thing\n          after Hyman Roth and Johnny Ola\n          were retired.\n          Cmberto has come up.\n\n                         HAGZ\"N\n          Cmberto. Michael's son. Anthony.\n\n                         CROCZ\n          Youi father must be proud.\n\n                         (THEN)\n          What do you think, Tom? They're\n          saying it was Maatrocina.\n          3agen makes a little gesture, reminiscent of the God-\n          father; a kind of upward opening of the hand, as if\n          gently letting a tiny bird free.\n\n          ANOTEE . ANGLE\n           As Frank ,e R .zzi. his a in a sliaq, and Sant .._^.c\n          Cor' eone ?cnay ` s oldest son, rcW is awd . e W -i-r-\n          ties, =Cme UM together:\n          (CUNT:Nt7ZD )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         - 41\n\n          30.\n\n                         CONTNM :\n\n                          BAGZN\n           Say hello to your cousin,\n\n                          IR RANKIE\n           Rizzi - your Aunt Connie s by--\n           and this- is Santino, your uncle\n          Sonny's oldest.\n          As Tony shakes hands with Santino and ?rinkie:\n\n                         PRANKIE\n\n                         (TO TONV)\n          Excuse my left hand.\n\n                         SANTINO\n          Frankie was in that little shit\n          storm on the ferry boat.\n\n                         CROCE\n          The whole thing don't make sense\n          to me.\n\n                         FRANXIE\n          All Z know is Al got a phone call\n          from that, nephew of his, Tommy,\n          who runs numbers in Staten Island.\n\n                         SANTINO\n          Fucking punk.\n\n                         FRAHRIE\n          It was something about Al's sister\n          being sick bad with the ptomaine or\n          I something. We walked right into it.\n\n                         CROCZ\n          What about Tommy?\n\n                         FRANXIZ\n          The nephew? Nobody seen him since.\n\n                         SANTINO\n          Ask me he's out in the Narrows with\n          about eight slot machines tied\n          around his neck.\n\n                         ANOTHER ANGLE\n          As Rocco Lamm pone comes up, clearly agitated:\n\n                         LAAONE\n          How do you Like the ally oaf that\n          bastard? Snowing up here.\n\n                         I\n\n                         (CONTIEE\n\n                         \n\n                         \n\n                         \n\n                         \n\n          31.\n\n                         CONTIIUED :\n\n                         SANTI-1141O\n          Maatr-ocina?\n\n                         LAMP ONE\n          If your old man was alive, that\n          son of a bitch would be eating\n          his dinner in hell tonight.\n\n                         RAGE\n          Be patient, Rocco, and trust me.\n\n                         (SMILES OFF)\n          Sam.\n\n                         ANOTSER ANGLZ\n          as Sam Maat:ocina comes up, his expression suitably\n          somber.\n\n                         MAATROCLMA\n          Tom. E'ellas. Go figure life, huh?.\n          A guy like Al Neri. Who'd of fig-\n          c a red him to get mousetrapped like\n          that?\n          Maatrocima is shaking hands with Hagen,. Rocco and\n          0 Frankie. As he does so:\n\n                         EAGFN\n\n                         (INDICATES)\n          Mike's son, Anthony. Sam Maatrocina.\n          Maatrocina holds out his hand to Tony. Tony just looks\n          at him coldly for an insulting split second. Maatro-\n          cina's expression doesn't change, and the outstretched\n          hand moves to squeeze Tony's bicep. Now he shows his\n          teeth in a grin\n\n                         MAATROCINA\n          The arm on the guy.\n\n                         (THEN)\n          If you ever need a job come see me.\n          w!aatrocina ti ns and moves off. Tony Is looking after\n          him.\n\n                         HAGEN\n          Mistake, ';.'onv. ?never let a man\n          like that 'mow what you' :e thiak-\n          .ng .\n\n                         =;T TO\n\n                         11\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          32.\n          TNT. LIVING ROOM (CON coRLEONE's HOUSE) - MIGHT\n          The members of the Corleone Family and some of their\n          friends are gathered at the Godfather's old house in.\n          the Mall. The tenor of the group is subdued, although\n          the tensions of the day are beginning to ease.\n          We pick up Fraakie who is guiding Tony through the\n          room.\n\n                         PA N =\n          Santino always liked the old place.\n          He picked it up after Pentangeli\n          knocked himself off.\n\n          ANOTHER ANGLE.\n          as they pass Umberto Croce who is coming out of the\n          diming room where a buffet has been set up. Croce\n          has a heaping plate of, food. He pauses, shaking his\n          head.\n\n                         C:TOCE\n          Sad day, sad day.\n          As Croce moves on, shaking his head, _rankie locks\n          after him.\n\n                         IS\n\n                         FRAVRIE\n          Dania near ruined his appetite.\n\n                         (THEN)\n          Good man though. From the old days\n          before the Spics and all.\n\n                         TONY\n          How strong is the Corleone family\n          connected in Latin America?\n\n                         FRANKTE\n          We got some people doom there used\n          to work for us in the hotel. Now\n          they do odd jobs. Help with the\n          airline. Like that.\n\n                         TONY\n          Tell me about the airline.\n          It's what you call non-scheduled.\n          You 'now?\n\n                         (THEN)\n          Lf we ever go out of business, haL\n           the rock groups i -he ccunt-= r J\n\n                         40 (CCNT \" + D )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         S\n\n          33.\n\n                         CCNTINLTED :\n          ? R A N = (Cont'd)\n          be standing around with empty spoons\n          stuck up their noses.\n          As Sonny's daughter, Francesca, comes by talking with\n          her twin sister, Barbara, who is now a nun. They are\n          now about forty.\n\n                         FRANZIE\n          Hey, Francesca. Barbara.\n\n                         (TO TONY)\n          You remember the twins.\n          As the women greet Tony, old Mister Nazorine, the baker,\n          comes by.\n\n                         FRANBIE\n          And here' s Mister Nazori.ne -- still\n          makes the best tarelles in town.\n          As Tony is greeting the old man:\n\n                         FRANCESCA\n          Tony, Tony. I remember the day\n          you. were born. A blizzard. And\n          grandpa and Tessin and Clemenza\n          were sitting out in the backyard\n          in the snow with a five gallon jug\n          of grappa, celebrating.\n\n                         BARBARA\n\n                         (LAUGHS)\n          And grandma was out there yelling,\n          disgrazia.. Infamita! You could\n          hear her clear to Freeport.\n\n                         F RANXIZ\n          (tugs Tony off)\n          Later, huh?\n\n          M. DON CORLEONE'S OFFICE - NIGHT\n          Gathered in Don Corleone's old cotter office are Santino,\n          Tom Hagen, Rocco Lampone and Umberto Croce. Tony and\n          Frankie enter.\n\n                         BEN\n          CZ.ose the door, please, ?rankie\n          and make yourself comfor able.\n\n                         ( THEN)\n          First, Tony, t!lank you for want c.\n          We all aroreciate is.\n\n          (C ONT11-N ED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         A\n\n                         L\n\n          34.\n\n                         CONTINUED :\n          Tony nods. All are seating themselves as:\n\n                         HAGEN\n\n                         (TO TONY)\n          Would you care for a little wine?\n          A little Anisette? No? All right.\n\n                         (THEN)\n          I've invited L berto Croce to sit\n          in with us-because of his strong\n          connections in Latin America and\n          in the exile comarunity down in\n          Florida -- and because he's a\n          trusted and valued friend of the\n          Corl eone family.\n\n                         (THEN)\n          And now,, if your friends in Langley\n          have decided that you're not a\n          Russian spy, perhaps you can tell\n          as the nature of the service they'd\n          like us to perform.\n\n                         TONY\n          It's a political assassination.\n          The target is Asmanda Vidal.\n\n                         0\n\n          A,NQ1'8ER ANGLE\n          Santino, whose attitude has been somewhat sardonic\n          throughout, now breaks out in a bray of laughter.\n          Tony turns on his cousin, then coldly and quietly:\n\n                         TONY\n          If it's too much for you, just say\n          so. M can break this off right now ---\n          Santino, taken aback at the cold authority in Tony's\n          manner, turns for support.\n\n                         SANTINO\n          What did I say, for God sake.\n\n                         (TO TONY)\n          Whaddya so touchy?\n\n          E AG I\n          When is this planned for?\n\n          TONY.\n          Next February. ?!e' s having a week-\n          long celebration o the Fif_'h. nn?.zr-\n           ersar,7 of the Revolution.\n\n                          (THEN)\n           We're planning i t to ? cok like an\n\n                         40\n          accident -- or natural causes.\n\n                         (CJN'RINUZD )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          35.\n          CCNTI i ED\n          SA.NTTzYO\n          That's a hell of a trick if you\n          can do it.\n\n                         TONY\n          Our Technical Services Division is\n          experimenting in two areas. One is\n          a scuba: diving wet suit designed to\n          malfunction at a critical depth.\n          The other is a toxic biological\n          material -»- a strain of botulism\n          that's tasteless, colorless and\n          odorless -- and so lethal that\n          one drop on his food or an his\n          toothbrush would be fatal inside\n          of an hour.\n\n                         CROCE\n          That would mean getting somebody\n          close to him.\n\n                         RAG'N\n          Could that be done?\n\n                         CROCE\n          t think so.\n\n                         EAGEN\n\n                         (TO TONY)\n          And what's your part in all this?\n\n          TOW,\n          t'm the cut-out. The circuit\n          breaker.\n\n                         BAGZN\n          The only link between our people\n          and the Government'\n\n                         TONY\n          That's right.\n\n                         (THEN)\n          There's a second phase to this\n          operation. It consists a spread-\n          ing confusion -- planting explos-\n          ives -- the Police Barracks -- the\n          Central. Power Station -- so on.\n          which will hopefully trigger an\n          uprising of the Anti -Vi\"'aiista\n          forces.\n\n                         ONTI TCZ-\n\n                         \n\n                         \n\n                         \n\n                         \n\n          36.\n\n                         CON'RL'??IUED :\n\n                         SANTINO\n          I thought these bastards wanted a\n          hit --?-\n          .(then)\n          They're locking for World War Three.\n\n                         HAGEN\n\n                         (TO LAMPONE)\n          What do you think, Rocco?\n\n                         ANOTZR AY(=\n          As Rocco Lempone -- who walks with a limp as a souvenir\n          of his service in World War Two-- thinks about it, then:\n\n                         LAMPONE\n          You're starting to talk about a\n          pretty big operation. You're\n          going to need an assault team --\n          maybe two. Small arms. Ammo.\n          Hand grenades. Field radios. A\n          support network down there. You'll\n          have to have transportation. Stag-\n          ing areas.\n\n                         0 TONY\n          The supplies can be made available.\n\n                         LAMFONE\n          How about getting them down there?\n\n                         FRIAS=_\n          No problem. We can carry eight tons\n          a trip in the DC Six.\n\n                         HAGEN\n\n                         (TO UMBERTO)\n          What do you think, Umberto? Can\n          you get your hands on a few Anti-\n          Vidalistas who wouldn't mind going\n          doom there and raising a little\n          hell?\n\n                         UMBERTO\n          I'll talk to. Doctor Earcenas,\n\n                         SANTIO\n          And we'll all end :zn. Faith cu= balls -\n          in the g='av.\n\n          F I.R A, N&\n          You're cetti g old, Santi no.\n\n          (C\". NTINL=)\n\n                         \n\n                         \n\n                         \n\n                         \n\n          37.\n\n                         CONTLVIIED\n\n                         SAN'TINO\n          I'm still young enough to whip\n          your ass.\n          As x'rankie starts to rise, Hagen puts a stop to any\n          further discussion.\n\n                         HAGR\"`N\n          All-right. That's it.\n\n                         (THEN)\n          We all agree -- except Santino --\n          that what Tony proposes is possible\n          although not. easy.\n\n                         (THEN)\n          Of course, financial arrangements\n          will have to be worked out --\n\n                         TONY\n          There's no problem there. Every-\n          thing will be handled through me\n          and in cash.\n\n                         HAGZN\n          You can tell your friends that they\n          have a deal.\n\n          TONY.\n          Good.\n\n                         SAGE\n          Tell them this too --- what the\n          Corleone family wants for its part\n          in this operation is the uncondi-\n          tional pardon of Danny DeVito. And\n          that will have to come first.\n\n                         TONY\n          I don't know if they'll accept that.\n\n                         RAGE\n          It's a non-negotiable condition.\n\n                         (THHEN)\n          We'll start getting things lined\n          up, but we won't move until Danny\n          walks out of Leavenworth.\n\n          ANOTS Z-R NGLL\n          Sant-4-0 -a aagen:\n          Sr'?,N'T'I O\n\n                         YOU REALLY\n          Devito?\n\n                         ( CON'\"INCED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          38.\n\n                         11 CONIINEM\n\n                         HAGEN\n          Why not?\n\n                         SANTINO\n          That would have to come right from\n          the White Souse.\n\n          RA.GZN\n          Where do you suppose the hit order\n          came frog?\n\n                         CDT TO:\n\n          EXT. CAMP PEARY, VIRGINIA - DAY\n\n          Camp Peary is the clandestine training facility of\n          the CIA, under military cover. It is near Williams-\n          burg, Virginia, a couple of hours drive from. Washington.\n          .cwa by the official cryptonym \"ISOLATION\" it is un-\n          officially called The Farla\" -\n          The enormous, thickly-wooded area is divided intern-\n          ally into tightly controlled training areas. A high\n          chaintlink._fence topped with barbed wire surrounds\n          the base. Signs an it read: \"U.S. GOVE.SNMENT RESER-\n\n          VATION. NO TRESPASSING.\"\n          We are presently on such a sign. We hold for a moment\n          as we hear ----\n\n                         INSTRUCTOR'S VOICE\n          A doomsday car is a vehicle which\n          is loaded with a high explosive,\n          such as gelignite, and left in an\n          area where it will do the most dam-\n          age when detonated --\n\n          EXT. TRAINING AEA (CAMP P_.ARY) - DAY.\n\n          We are on the Instructor, a lean, mean-lacking A=Y\n          Sergeant is impeccable fatigues. His manner of speech\n          is Southern.\n\n                         INSTRUCTOR\n          Do I read disapproval on your face,\n          Mister Adams?\n          AsNOTR NN-- ~'\n          ..ncludina the tra:..'aing class. thirty or fort., men 4=\n          army fatigues. Score are dark men, mustachiced., pass\n          E :t iler Saudis cr Iranians. Amcnq t!?iese we finch Tony.\n\n          ( C O N T I N U ED)\n\n                         \n\n                         \n\n                         \n\n                         \n\n          39.\n\n                         CCNT NUED :\n          Although he doesn't answer the Instructor, we can\n          read something less than total approval on his face.\n\n                         INSTAUC'R'OR\n          In view of the fact that we're\n          getting our butts kicked rosey all\n          over the world, it might behoove\n          us to take a lesson from the IRA\n          and the PLO.\n\n                         (THEN)\n          Now, the first thing those boys'l1\n          do when they aim to spread a little\n          unhappiness in the ranks of the\n          righteous, is steal a motor vehicle --\n\n                         (THEN)\n          You know how to steal a motor\n          vehicle, Mister Adams?\n\n                         TONY\n          No I don't.\n\n                         \" = 4STRLICTOR\n          By the time you leave Camp Perry,\n          you' 11 be an expert.\n          The Sergeant turns to the others.\n\n                         INSTRUCTOR\n\n                         (CONTINUING)\n          All right, gentlemen, this is a\n          remote control detonator.. .on safe.\n          Be holds up a detonator in. his hand, then points Off.:\n\n                         INSTRUCTOR\n          And that yonder is a doomsday ca_r.\n\n                         ANOTHER ANA\n          including an old car set in a valley some two hundred\n          yards away. The Instructor takes the detonator off\n          safe and as he activates the charge in the doomsday\n          car, we zoom in. The explosion fills the screen as\n          the doomsday car is blown to hell.\n\n          JISSCL'JE .C :\n\n           \"NT. CT...ASSRCOM T (CAMP ?MARY) - DXZ\n           We are ..: a cl assrcOm in wooden a--=V ar-racks _ The\n           teacher\", a :Va .- Coxxna ,de: , s ? °_c t ing . -he= e are\n\n                         E\n\n                         (CONTINUED)\n\n                         \n\n                         \n\n                         \n\n                         \n\n          40.\n\n                         CONT IUED:\n          chalked o ptograms on the blackboard behind him.\n          They are variously, ODYOKc, 'ODACID, OD MM. AELAU ,,\n          AEJAMMER, AEBROOM. Tt3DOV, =ESR, .KUCAGE.\n\n                         COMMANDER\n          The cryptonym is a name used in\n          place of the true name. In the\n          company, crtonyms consist of\n          two letters that determine the\n          general catagory followed by a\n          word -- the United States Govern-\n          ment is designated by the letters\n          \"O\" and \"D\", and the word \"Yoke\".\n\n                         (POINTS)\n          O-D-yoke. The Department of State,\n          O -Z-acid.\n\n                         STUDENT\n          What's that last one?\n\n                         INSTRUCTOR\n          0-0--envy?\n\n                         (GRINS)\n          That's the FBI.\n           As the group of trainees laugh, we\n\n                          CUT TO:\n\n          EXT. A. CANYON ROAD (C+ '4P PAY) - DAY\n\n          Up'a dirt road, between the trees, comes a platoon of\n          trainees, in jungle gear wet with sweat, double-tizsing\n          as they chant:\n\n                         PLATOON\n          Hut two three four --\n\n                         REEP-BY-YA-LO'\n\n                         BEEP-BY-YA-LO-\n\n                         LO-RIGHTS-LO--\n          We move into the platoon, pick out Tony as he runs,\n          not cocnti.nq. The Drill Instructor runs up alongside\n          of Tony.\n\n                         INSTRUCTOR\n          You're not singing, Mister Adams.\n          Aren't you harpy is our little\n          croup?\n           As Tonv locks over at the : - acing Drill inst? ?c cor ,\n          h t en starts chanting a h --he zest\n          C:iT To :\n\n                         \n\n                         \n\n                         \n\n                         \n\n          41.\n\n          EXT. PISTOL RANGE (CAMP PEAR21 - DAY\n\n          Tony is at the pistol range, firing. As he finishes\n          up the clip and draws the target back to him on a\n          pully device, the Pistol Instructor comes up and in-\n          spects the target with the bullseye chewed out.\n\n                         PISTOL INSTRUCTOR\n          Good. shooting. You do a lot of\n          hunting?\n\n                         TONY\n          No.\n\n                         PISTOL INSTRUCTOR\n          Mast run is the family, then.\n\n                         TONY\n          You might be right.\n\n                         CUT TO:\n\n          INT.. LOCX P ICRITG CLASS (CAMP PEARY ) - DAY\n\n          There are diagrams on the blackboard. Tumblers,\n          locks, keys. There are big, half-sections of locks.\n          Half a dozen. members of the class, including Tony,\n          are working with picks an locks. As the Teacher\n          helps Tony.\n\n                         TEACHER\n          On the ordinary pin tumbler cylin-\n          der lock, the spring actuated\n          drivers are partly in the shell\n          and partly in the plug. The. trick\n          is to lift them up so the plug can\n          turn freely --\n\n                         (THEN)\n          That's right.\n          The lock has opened.\n\n                         TEACHER\n          Once you've gotten the lock picked\n          and the door opened the best thing\n          is to tape the bolt mechanism back\n          so you won't have to keep picking\n          the lock -\n          As the Teacher Instrructs the class - a la Watergate\n          break-inn -- how to tace back the bolt mechanism, we\n\n                         CUT TO:\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         L\n\n          42.\n\n          EXT. T: AIMING AREA (CAMP PEARY) DAY\n\n          We are on the Drill Instructor who brandishes a\n          Marine Corps knife, six-inch blade, brass knuckles\n          incorporated into the handle.\n\n                         INSTRUCTOR\n          This object is a United States\n          Marine Corps killing knife. I..\n          will now demonstrate that it is\n          not worth doodley shit if you don't\n          know how- to use it.\n\n                         ANA ANGLE\n          as the Instructor looks around the circle of Officer\n          Trainees and from them chooses Tony.\n\n                         INSTRUCTOR\n          You.. You'd like to kill me,\n          wouldn't you, Sir? Well, here's\n          your chance.\n          The Instructor tosses Tony the knife. Tony catches\n          it. As they circle,. the Instructor taunts Tony.\n\n                         INSTRUCTOR\n          Come on. Come on. Make a move,\n          Sir. Are you falling in love with\n          me? Then do something hostile.\n          Make a face. Stick out your tongue.\n          Do something, Mister Adams.\n          Tony swings the knife. The Instructor avoids him.\n\n                         INSTRUCTOR\n          My little bitty sister can make a\n          better move than that, Six.\n\n                         ANOTHER ANGLE\n          The Instructor offers a tempting target. Tony swings.\n          The Instructor slaps his cap across Tony's face, grabs\n          Tony's wrist and disarms him, throwing him to the\n          ground.\n\n          ANOTHER ANGLE - ON TONY\n          The Instructor turns his back on Tony, deliherately .\n          Tony gets to his feet and charges the .nst=,actorls back.\n\n          (C^NT? 2IUED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          43.\n\n                         CONTINUED\n          The Instructor has been waiting for this. Re flips\n          Tony again.-\n          Tony lies inert an the ground. The Instructor comes\n          up, beads to inspect Tony. As he does so:\n\n                         - INSTRUCTOR\n          All right, Sir. You ---\n          Re breaks off as. Tony has driven an upper cut into\n          the Instructor's balls. The Instructor grabs his\n          groin and goes down in a heap.\n\n                         ANOTHER ANGLE\n          Tony rolls to his feet, grabs up the killing knife\n          and presses it to the Instrutor's throat.\n\n                         TONY\n          All right you, son of a bitch, tell\n          me about it..\n          The Instructor is looking at Tony. Suddenly he grins;\n\n                         INSTRUCTOR\n          You're getting there, Mister Adams.\n\n          ANOT3ER ANGLE\n          Tony looks at the knife in his hand, then tosses it\n          away and starts off. At this point a jeep comes\n          bouncing over the hill and skids to a breadsiding\n          stop.. We see that PaJ.ateer is at the wheel.\n\n          PALHA2'E.=\n\n                         (TO TONY)\n          Jump in.\n          Tony is in the jeep. As Palmateer guns out, trailing\n          a plume of dust, we ---?\n\n                         CUT TO:\n          tNT. RECREATION ROOM - CAMP PEAwM - DAY\n          Palnateer is watching the Recreation Room television\n          set on which there is a newscast of the ze ease of\n          Danny DeVito frc n Leaveizwcrth..\n\n                         M ZUED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          44.\n\n                         CONTINUED:\n          Danny, almost as wide as he is tali., his broad pug-\n          natious face wreathed in a grin, camas cut of the\n          prison gate, his hands held up in the \"Victory\" sign.\n          He pushes his way through the Television Cameramen\n          and Newsmen, not saying anything.\n          We pan him over to a limousine which he gets into\n          quickly. The limousine drives off.\n\n                         NEWSCASTER'S VOICE\n          Today in Leavenworth, Kansas, one--\n          time International Brotherhood of\n          Dockworkers' President, Danny DeVito\n          i was. released. on pardon after serving\n          six and. a half years of a fifteen\n          year sentence an embezzlement and\n          conspiracy charges. DeVito didn't\n          state his future plans, but insiders\n          expect him to challenge the incum-\n          bent President, Pat Brady, in the up-\n          coming election.\n\n                         ANOTHER ANGLE\n          As Tony comes into the room, Palmateer gives him a\n          pen and a couple of pieces of paper.\n\n                         I\n\n          PAL.MATEER\n          A couple of things for you to sign\n          6 before you go operational. This is\n          for your piece --\n\n                         (SECOND PAPER)\n          And this is your resignation. It's\n          a technicality, but in case the shit\n          ever really hits the fan, we'd have\n          p to disclaim you.\n\n                         (GRINS)\n          It's called the principal of plaus-\n          ible denial. We never invoke it\n          until the cock crows thrice.\n          As Tony is signing the papers, Palmateer has produced\n          a service automatic from a canvas carryi+:g case. He\n          puts it on the table before Tony ---\n\n          E=. A WAS ZNGTCN D.C. SUZZZ G - DAY\n          As Tony, dressed in his blues, comes out of a build-\n          ing. He carries a briefcase. Cressinq to ie. curbs\n          gae stexas and looks up the s ?_. eet\n\n                         \n\n                         \n\n                         \n\n                         \n\n          45.\n\n                         ANOTHER ARGLZ\n          As a large car pulls up and stops in front of Tony,\n          he opens the door and gets into the backseat. We\n          now might be able to see that F'rankie is driving.\n          With him in the front seat is Santino. In the back-\n          seat is Umberto Croce.\n          INT'. THE L SOUS tNE - DAY\n          as Tony gets in and sits next to Croce. General\n          greetings, then Tony gives the briefcase to Croce.\n\n                         FRANRIE'\n          Where to?\n\n                         TONY\n          Take a right on Sixteenth Street.\n\n                         ANO ANGLE\n          As Croce sets the briefcase on his lap, Tony reaches\n          into his pocket and gives him the key. Now Croce\n          carefully unlocks the briefcase. As this is going on:\n\n                         TONY\n          How's. your progress?\n\n                         CROCE\n          Doctor Barcenas is getting an\n          assaS,ilt team together.\n\n                         TONY\n          'Barcenas?\n\n                         CROCE\n\n                         T\n          A leader in the exile community.\n          0 One of the early revolutionaries.\n          Broke with Vidal when El Presidente\n          went Marxist and abbrogated the\n          Constitution. He thinks we're a\n          group of business men backing him\n          to get our hotel and casino back.\n          Croce has gotten the briefcase open. It is full of\n          cash. Tony points ahead.\n\n          TONY .\n          Let use of-12 at the next corner.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          46/48.\n\n          INSE.BT - THE BRIEFCASE\n          As Croce picks up a packet of hundreds cut of the\n          suitcase, riffles it:\n\n                         TONY'S VOICE\n          It's all there.\n\n          HACK TO SCENE\n          As Santino grins:\n\n                         SANT=O\n          Straight frcm the East Coast\n          Distributor.\n          Tony looks over at Croce:\n\n                         CROCZ\n          We're in business.\n\n                         TONY\n          I' I L see you in Florida..\n\n          EXT. WASHINGTON STREET - DAY\n\n\n                         0\n          As Tony gets out of the limousine and crosses to a\n          phone booth. We are moving in as he puts a. dime into\n          the slot. and dials.\n\n                         TONY\n          Hiss Duane, please.\n\n                         CUT TO:\n\n          =T. LOBBY (ENNEDY CENTER) - NIGHT\n          We are shooting past the big, nubby sculptured head of\n          Sohn Kennedy which identifies, but in no way dominates\n          the long, high handsome lobby with its crimson carpets.\n          The Opera is breaking for intermission and the audience\n          is moving toward the bars and the terrace.\n          We move to the bar where, in the crush, we find Tony.\n          As everybody seems to be ordering at once---\n          TONY . i\n          Scotch-rocks, here. Two. Aadame,\n          I believe those were\n\n                         (THEN)\n          Scotch-rocks. Sere. Two.\n\n                         (CODITINUED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          49.\n\n                         CONTINUED\n          Tony turns to watch a woman move off with two drinks-\n          that were clearly his. 'Now, a. braided Admiral usurps\n          Tony's place.\n\n                         TONY\n          I believe I was next, Sir.\n          The Admiral throws a look in Tony's direction, then\n          pays for the bourbon and coke and exits.\n\n                         TONY\n          Scotch-rocks. Here. Two..\n          EXT? TEE TERRACE ( iYNEL'Y CENT~'R) - NZGET\n          On the broad, impressive terrace, overlooking the curve\n          of the river- as it mirrors the lights of Washington, we\n          find Elizabeth, dressed for evening. She is surrounded\n          by three urbane men, Morton, her Television Producer,\n          Swartzwalder, a Jurist and McKissick, a young Senator.\n          McKissick is lighting Elizabeth's cigarette as Tony\n          comes up with the two drinks.\n\n                         TONY\n          Finally.\n\n                         ELIZABETH\n          Thank God for 'the Navy.\n          As she takes her drink:\n\n                         ELIZABETH\n          Mister Adams,_ I'd like you to meet\n          Judge Swartzwalder -- Senator\n          McXissick and Jack Morton from\n          wham all blessings flow ---\n\n                         MORTON\n          As long as the ratings hold up.\n          As Tony is shaking hands around with the Judge, McKissick\n          turns to Elizabeth:\n\n                         MC EISSICK\n          Burning the midnight oil at State?\n\n                         ELIZABETH\n\n                         (SMILES)\n          I wouldn't know, s°qve broken off\n          relations with the State Decar ..went .\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          50.\n\n                         ANOTHER ANGLE\n          Tony has heard this last.\n\n                         SWARTZWALDER\n          You stationed in Washington?\n\n                         TONY\n          On temporary assignment.\n\n                         ELIZABETH\n          He's doing a background on Azmando\n          Vidal.\n\n                         MC RISSIC\n          A fine. man. And thank God we're\n          back on speaking terms -- or are\n          we?\n\n                         MORTON\n          If you don' t know, who?\n          A buzzer sounds from inside. Elizabeth hands Tony\n          her glass.\n\n                         ELIZABETH\n          Would you be an angel and\n          Tony takes the glass from her, crosses and puts it\n          on a nearby bench. As he does so, Elizabeth is Look-\n          ing after him, as to the others:\n\n                         ELIZABETH\n          I've always had a =ad thing for\n          sailors- They've got such neat\n          little asses.\n          As Tony returns and escorts her back into the theater:\n\n          CG\", TO:\n          EXT. R\"\" ?MY CE:TTm..,q DRIVEWAY - NZGHT\n          We are on the line of cars coming up to pick op their\n          passengers outside the Eall of States. Mi..ch honking\n          of Eioras, so on.\n\n          ON TON`I AND ELIZABETH\n          as they stand among the azagn= _coes , 1 oaki i for a\n          tax:.. Elizabeth spats -ne f_--st.\n          (CON°^Z `i?D ?\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         7\n\n          51.\n\n                         CONTMM=\n\n                         ELIZASET:\n\n                         THERE'S ONE\n          Tony hurries for the taxicab.\n\n                         ANOTSEB ANGLE\n          as. an Airforce General intercepts the cab.\n\n                         GENERAL\n          Believe this is mine.\n\n          .ON ELIZABETE\n          As she watches Tony get outranked for his cab, then\n          she spots another.\n\n                         ELIZABETH\n          f f ere comes another.\n\n          ANOTHER ANGT.E\n          as. Tony hurries for the. next cab, only to get beaten\n          cut by a. State Department type with two oil Sheiks\n          in tow.\n          STATE DEPART.'MiT TYPE\n          You're next, Ensign.\n          Tony turns, signals: to Elizabeth to wait, then exits.\n\n          CLOSE - ELIZABETE\n          as she looks after him., puzzled.\n\n          MT. . PARE ING AREA - NIGHT\n          as Tony moves into. the Eennedy Center Parking area,\n          looking around.\n\n          ON ELZZABET:E\n          She is pacing. She stops, looks at her watch, then\n          turns at an insistent bbonking from -.edam: veway. We\n          pull back and pan to include Tony s it ti_ng in a big,\n          beautiful, official-looking sedans. ae leans across\n          and opens the front door. She crosses and gets in.\n\n                         \n\n                         \n\n                         \n\n                         \n\n          52/53.\n          ON HALL OF STATES DOORWAY (Ba\"?° CENTER) - NIGHT\n          as an Admiral and his wife come out toward the drive-\n          way with their guests. All are chuckling merrily at\n          • something the old sea dog has said. Now his wife's\n          expression alters as she looks of.\n\n          ADMIRAL' S WIFE\n          Charles: Isn't that our car?\n          He looks off. His expression changes. He starts to\n          run..\n          ANOTHER ANGLEZ=n=Y• CENTER DRIVEWAY\n          As Tony drives off, the Admiral runs vainly after\n          his car, waving his hand..\n\n                         ADMIRAL\n          Stop: Stop!\n\n          INT. THE ADMI-RAL'S CAR_- NIGHT\n\n          Elizabeth has been looking back. Now she regards Tony\n          with. new and approving eyes..\n\n                         ELIZABETH\n          You have interesting talents, Mister\n          Adams ..\n\n                         TONY\n          You'll be astonished.\n\n                         ELIZABETH\n          i I'm looking forward to it.\n\n                         CUT TO:\n\n          INT. ELIZABETH' S BEDROOM - NIGHT\n\n          Tony and Elizabeth are in bed together making love.\n\n                         ANOTEER ANGLE\n          featuring Elizabeth as\n\n                         ELIZABETH\n          Now. Now. Yes. Now\n\n          = 14. TO :\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         Y\n\n          54.\n\n          INT. BEDROOM (ELIZABETH'S APT.) - NIGHT\n\n          Tony and Elizabeth are sitting up in bed. Tony is\n          lighting her cigarette. Suddenly he pauses.\n\n                         ELIZABETE\n          What's wrong?\n\n                         TONY\n          Shh.\n\n           L E IZAB£TE\n          What's the ?.»..\n          She breaks off because Tony has put his hand over her\n          mouth. Her eyes blaze as she reaches for his wrist.\n\n                         TONY -\n          Noise in the living- room.\n\n                         (THEN)\n          There it is again.\n          This time we too have heard something. He uncovers\n          her mouth.\n\n                         ELIZABETH\n          There's a window in the front room.\n          It rattles.\n\n                         TONY\n          I don't know.\n\n                         ELIZASETB\n          Well, I do.\n\n                         ANOTHER ANGLE\n          as Elizabeth swings out of bed, naked, crossing out\n          of the bedroom and into the hall.\n\n          INT. EALLWAY (ELIZASETS' S APT.) - NIGHT\n\n           We move with Elizabeth as she goes down the hallway.\n          SST. LIVING ROOM (EZIZABE:'B' S APT.) - \"r-GET\n          As Elizabeth comes into the darkened ? :.ring room, suddenly\n          a Man from out of \"-,-.e shadows, clamps one hand over her\n          .mouth and with the other hand he holds a :cr? a to her\n          .mroa t.\n\n                         0\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          55.\n\n                         ANOTHER ANGLE\n          As a. Second Man appears. This one has a gun. The First\n          Man turns to him, speaking in a whisper.\n\n                         FIRST MAIM\n          Bring him down here.\n          The- Second. Max. nods. We follow him as very softly he\n          goes down the hallway.\n\n                         P\n\n          CLOSE - THE SECOND MAN\n          as he pauses. outside the bedroom door which is half\n          P ajar. Now suddenly he moves ---\n          IDi'i:. BEDRCOM - NIGHT\n          0 We are angled on the hall. door as, in a single move\n          the Second Man kicks the door open and flicks the bedroom\n          light on. He has his gun pointed at the bed. We whip\n          pan to the bed.. It's empty.\n\n          CLOSE: - THE SECOND MAN\n          As for a frozen moment., surprise and consternation\n\n                         0\n          show an his face. At this point:\n\n                         TONY\n\n                         (VERY SOFTLY)\n          Don't make a noise. Just stand\n          where you are or I'll kill you.\n          We have pulled back and panned slightly to include Tony\n          who is sitting on the. floor, his back against the wall.\n\n                         I\n          The gun in. his hand is, trained on the Man who has just\n          -come in.\n\n                         ANOTHER ANGLE\n          As: the Second Man stands motionless, Tony is on. his\n          feet and up to him. Tony takes the qua from him and\n          tossing it an the bed gestures, forefinger to lips,\n          be quiet.\n          Now, Tony turns to the Man and. starts down the hall\n          with him. Tony has him by the back of the jacket, his\n          Tun pressed against the back of the Second Man's head.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          56.\n\n          INT. LIV -MIG ROOM (Z LIZABETH' S APT .) - NIGHT\n\n          As Tony enters with his prisoner, the First Man - the\n          one with the knife -- still has his weapon pressing\n          into Elizabeth's throat. He is looking the other way,\n          but turns as: _\n\n                         TONY'S VOICE\n          Turn very easy.\n          The First Man turns, sees Tony and the Other Man.\n\n                         TONY\n          S've got a. gun at your partner's\n          head.\n          There is 'a beat, then:\n\n                         FIRST MAN\n          Drop the gun or I slit her throat.\n\n                         TONY\n          You drop the knife.\n\n                         FIRST MAN\n          Don't you think I'll kill her?\n\n                         0 TONY\n          Z don't give a shit if you kill\n          her. I_ said crop the knife.\n\n          ANOTRER ANGLE.\n          As the Second Man -- overconfident at finding hi=self\n          still alive -- speaks to his partner.\n\n                         SECOND MAN\n          Cut her a little bit to convince\n          this asshcle. Open up her throat.\n          Suddenly the Second Man's knees buckle as. Tony, in a\n          lightning move, brings the gun barrel down across the\n          Man's head.\n          As the Second Man is on his hands and knees, Zi.?ce a\n          stunned ox, dripping blood on the carpet, Tony tuns\n          once more to the an with the knife.\n\n                         TONY\n          I'm going to give you one more\n          chance. Trot the k:i-`f:a and !'I--l\n          let you go .\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         -T\n\n          57.\n\n                         CONTINTED: -\n\n                          FIRST MAN\n\n                         (LAUGHS)\n          7. 111 cut her fucking head off,\n          first.\n\n                         ANOTSER ANGLE\n          as the Second Man, still on his hands and knees, now\n          pushes himself up to a kneeling position.\n\n                         SECOND MAN\n          Show him a little blood.\n          As the Man with the- knife nods grimly, Tony puts his\n          gun to the back of the kneeling man's head and calmly\n          pulls the trigger.\n\n                         ANOTBER ANGLE\n          The. kneeling man plunges for+qard, dead, the back of -\n          his head blown off.. Tony now turns, without emotion,\n          to the man with the knife.\n\n                         TONY\n          That's what you're going to look\n          like in two seconds if you harm\n          that girl..\n\n                         ANOTRER ANGLE\n          As the First Man drops his knife and releases Elizabeth,\n          she is locking at Tony, stunned, her aplomb totally\n          vanished.\n\n                         TONY\n          Go get dressed and call the Police.\n          Elizabeth nods and half-stumbles out of the room. Tony\n          crosses to the Man.\n\n                         (CONTIJ UED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n                         I\n\n          58.\n\n                         (CODITINT1EI1)\n\n                         FIRST XUX\n          when the cops come, you're going\n          to be up on a murder rap, baby.\n\n                         TONY\n          No, I'm not.\n          Tony lifts the gun and blasts the First Man three\n          times in the chest.\n\n                         ANOTHER ANGLE\n          as Elizabeth bursts bac k into the room, looks, then\n          starts to scream. Tony crosses up and taking her in\n          his arms, softly comforts her.\n\n                         TONY\n          It's all right.. He tried to get\n          my gum away and I had to shoot him.\n          As. Tony, still comforting her, moves her away, We --\n\n                         DISSOLVE TOT\n\n          EXT. EVERGLADES BAY - DAY\n\n          We are in a swampy backwater on the West Coast of\n          Florida, south at Fort Meyer. Hacked out of the\n          everglades is a clearing on which a tourist acccmmo-\n          dation has been built. It is called Tarpon Lodge\n          and signs announce \"Cold Beer\", \"Boat Rentals\" and\n          \"Tourist Accoammdations\". There is a small pier.\n\n                         ANOTHER ANGLE _\n          as a small, commercial fishing boat comes chugging\n          up the bayou and blows its whistle.\n\n          CLOSER - THE FISH=TG 3O\n          as a Latin American, Roberto Barcenas, about 3S,\n          comes out to the bow of the boat, locks off toward\n          the tourist cabins. Roberta is a capable, cool,\n          izstelligent man.\n\n          E<'CT. T9Y T RPON :IODGZ - DAY\n          as Umberto Croce, dressed in white and weariri a straw\n          hat against the sun,. ccmes out or she ramshackle lodge\n          and moves to the of\n\n                         0\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         59A\n\n                         THE PIER\n          Crewmen of the fishing boat are tieing up as Roberto\n          Barcenas comes down onto the pier followed by half a\n          dozen others. They are radio, Borracho, Padre Pepe,'\n          El Fararon, Arrigo and Roberto's younger brother, Jorge.\n          Croce and Roberto embrace, then :\n\n                         CROCE\n          This way.\n          M=. As AIRSTRIP - DAY\n          A rough airstrip has been bulldozed out of the scrib\n          pine and palmetto forest. At one end is a hanger and\n          on its side the words: INTER-CAR EAN CBAMTERS.\n          As Croce and the Latin Americans come up:\n\n          CROCE.\n\n                         TONY:\n\n                         1 0 ANOTHER ANGLE\n          1 Inside-the hanger we see a D.C. Six. Tony comes out\n          cleaning cosmolene off his hands with a rag.\n\n                         CROCE\n          Tony, I' d :like you to meet Doctor\n          Barcenas.\n\n                         TONY\n          I'm pleased to meet you, Doctor.\n          They shake hands, then:.\n\n                         ROBERRTO\n          Permit me to present my companions --\n          Indio, Borracho, Padre Pepe, El\n          Fararon, Arrigo and my brother, Jorge.\n          The members of the assault team -- the Indian, the\n          Drunk, the Priest, the Pharoah, Arrigo who looks\n          like a pimp, and the romantic, fire-eating younger\n          brother -- are introduced to Tony individually. Xd\n          lib greetings, then to Barcenas:\n\n                         TONY\n          Are you read-!?\n\n                         (CCINNTINUED 11\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         60-\n\n          CONT=T Z :\n\n                         ROBERTO\n          For many years.\n          As they move into the hanger:\n\n                         ROBERTO\n          Z understand from Senor Croca that\n          the idea with the diving suit did\n          not work out.\n\n                         I TONY\n          There were a few problems.\n\n                         ROBERTO\n          But the other is. fine?\n\n                         TONY\n          Yes. ,\n\n                         (THEN)\n\n                         D\n          You've been in. touch with your\n          underground?\n\n                         ROBERTO\n\n                         (NODS)\n          Everything is arranged.\n\n          INT THE EA2GAR - DAY\n          At one side of the hangar we see crates of various\n          weapons. Boxes of hand grenades, mortars and mortar\n          shells. Sub-machine guns caked in cosmolene, LAW\n          rockets, field radios. So on. As Tony and Roberto\n          come in, followed by the others:\n\n                         TONY\n          we got you BARS and Carbines\n          plus ten, thirty calibre light\n          machine guns, and ten, four-point\n          t o inch mortars with a thousand\n          rounds of high explosive and a\n          thousand, rounds of white phosphorous.\n          Tony is pointing out the various boxes and crates:\n\n                         JORGE\n          What about sub-machine guns?\n\n                         TONY -\n          Thirty or; them. Tarty-five calibre\n          nine millimeter. With ten thousand\n          rounds.\n          (po.ints\n          Over there.\n\n          (CONTT-.?_-D )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          61.\n\n                         CONT=TISW\n          As Jorge crosses. over to the box of weapons, picks\n          one up in his.hands:\n\n                         JORGE\n          Ten thousand rounds! Aye de mi!\n\n                         ANOTEER ANGLE\n          as. Tony picks up and hands Roberto a long, olive green\n          plastic tube.\n\n                         TONY\n          We also got you two hundred Sixty-\n          six, millimeter, M-72 LAW rockets.\n          They're lightweight, one-shot dis-\n          posable Bazookas.\n\n                         (THEN)\n          There is also field gear, medical\n          supplies, tentage, demolition mater-\n          ials, combat rations ---\n          Arrigo, the. pimp, comes up, takes the LAW rocket from\n          Roberto, then:\n\n                         ARRZGO\n          We make a little trouble for E:\n          Presidents, hey?\n\n                         JORGE\n          When do we leave?\n\n                         TONY\n          You, go in tomorrow to get it ready.\n          I'll meet you down there at the end\n          of the week.\n\n                         (THEN)\n          Who's your radio man?\n\n                         PEPE\n          t am.\n\n                         TONY\n          I'll show you the set up.\n          As Tony and Padre Pepe move off together toward. a\n          short wave radio set up in the corner, ae --\n\n                         CUT TO:\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         K\n\n          62.\n\n          EXT. LAS VEGAS HOTEL (SWT_NZU TG POOL AREA) - DAY.\n\n          We are in a long shot and angled downward on the swim-\n          ming pool. area of the Corleone Family's Las Vegas\n          flagship hotel. Into the shot, threading their way\n          between the girls in their bathing suits, came Sam\n          Xaatrocina and Ralph Augusta, walking purposefully.\n          We zoom back ---\n\n          INT. TOM HAGE\"N' S ©FFICE (LAS VEGAS HOTEL) - DAY\n\n          We are on Rocco La=pone, who is standing by the window,\n          looking out onto the pool area. As he turns:\n\n                         LA ONE\n          Here they came.\n          We are pulling back. Also present in the large, attrac-\n          tive office, are Tom Hagen, Danny DeVito and Sant no\n          Corleone, who is at the bar fixing drinks.\n\n                         SANTINO\n          Half an hour late.\n\n                         DEVIT0\n          0 I'm happy he's here at all. That\n          means he's willing to talk like\n          a reasonable man.\n\n          L.A ONE\n          Tell A3..ieri about it.\n\n                         HAG=\n          What you've got to understand,\n          Rocco, is that what happened to\n          Al was business. What's done is\n          done. The important thing is to\n          avoid trouble if we can. This\n          isn't the old days. The Five\n          'Family wars are over and done with.\n\n                         LAMPONE\n          Maybe that's too bad.\n\n                         ANOTEEQ ANGLE\n          as Santino comes over with the drinks io± Hagen and\n          DeVito. As Danny accepts his-drink and raises it to\n\n                         HAGEN:\n\n                         \n\n                         \n\n                         \n\n                         \n\n          63.\n\n                         CONTINUED:\n\n                         DEVITO\n          To you -- all of you. I'll never\n          be able to thank you for what you\n          done.\n          As DeVito drinks, the intercom is buzzing. Hagen\n          flicks it.\n\n                         SECRETARY'S VOICE\n          Mister Maatxocina and Mister Augusto\n          are here, Sir.\n\n                         HAGEN\n          Send them in, please.\n\n                         ANOTHER ANGLE\n          Eagen comes around his desk to the door to greet\n          Maatrocina as he enters, followed closely by the\n          cold-looking Ralph Augusto.\n\n                         EAGEN\n          Sam. Z'm glad. you could come.\n          Ralph. Good to see you. What\n          are you drinking?\n\n                         MAATROCINA\n          Z'11 take a little bourbon with\n          you._ On the rocks.\n          Sagem nods at Santino who crosses to the bar to make\n          the drink, as Hagen turns now to Augusto.\n\n                         HAGEN\n          Ralph?\n\n                         AUGCSTO\n          Nothing.\n\n                         HAGEN\n          Coke?\n\n                         AUGUSTO\n          Nothing.\n\n                         ANOTHER AN=\n          As Santin_o is making the drink; Kaat--ocL-za tax.,.--ns to\n          Danny DeVito, shaking hands.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          .7\n\n          64.\n\n                         CONTINUED:\n\n                         MAATRCCI4A\n          Let. me save us all some time. I\n          know the Hagen-Corleone Family got\n          Danny sprung and I'm glad about it ---\n\n                         (TO DEVITO)\n          From the heart.\n\n                         DEVII'O\n          Thank you, Sam. I :snow you mean\n          it.\n\n                         MAATROCI A\n\n                         (TO HAGEN)\n          I also know how you swung it --\n          (grins, holds\n\n                         UP HAND)\n          Yeah. That's right. You aren't\n          the only ones with a friend or\n          two in. Washington. Don't under-\n          estimate me.\n\n                         RAG=\n          We never have, Sam.\n\n                         MAATRCCINA\n          One more thing I know -- Danny' s\n\n          ? •\n          a big hero to the rank-and-file.\n          He'll leave Pat Brady for dead in\n          this Special Election that's coming\n          up. Okay? So that leaves one thing\n          to talk about.\n\n                         HAGZN\n          How we can all accommodate to this\n          new situation.\n\n                         MAATRCCINA\n\n                         (GRINS)\n          I don't know from accc=cdate --\n          just so we all get a chance to\n          drink from the well.\n\n                         HAGS\n          Problem being, for the vast five\n          years the well dried up an us as\n          you might say. It was almost as\n          it Pat Brady had scmethin q against\n          us personally.\n\n                         TROCT A\n          T=, Tcm. \"cu should have come\n          to me.\n\n                         E (CONTINCZD )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          65.\n\n                         CONTZY[JZD :\n\n                         EAGEN\n          You don't like to bother your\n          friends about these matters.\n\n                         MAATROCINA\n          What are friends for?\n\n                         HAGW\n          In any case, the problem is solved\n          ar will be soon. Tice well is ao\n          longer dry.\n\n                         MAATROCINA\n          And. everybody gets to fill his\n          bucket. That's as it should be.\n          After us.\n\n                         MAATRCCINA\n          Meaning what exactly?\n\n                         SAG=\n          We need a loan of fifty million\n          dollars. That has to be the first\n          order of business ---\n\n                         MAATROCI A\n          The Atlantic City Hotel?\n\n                         BABY\n          1 That's right.\n\n                         I\n\n                         ANOTHER ANGLZ\n          As Maatrocina makes flat paddles of his hands and points\n          them inward to his chest as he turns to DeVito.\n\n                         MAATROCLVA\n          And what am t? An orphan? I want\n          to get in on Atlantic City, too.\n\n          DEV.ITO\n          You had it all your way for the\n          last five years, Sam. Let some-\n          body else do business.\n\n          . AATROCINA\n          I got ao cb7ectio:t - ,a him doing\n           business --- i just want to do\n          business, too. Kay?\n          What Hagen gets, :: get.\n\n                         LI\n\n                         (CCNTI'TCZD )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          66.\n\n                         CONTINITE\n\n                         DEVITO\n          Are you talking about a hundred\n          million dollars?\n\n                         MAATRCCINA\n          If fifty and fifty still make a\n          hundred, that's what I'm talking\n          about.\n\n                         DEVITO\n          it can't be done.\n\n                         MAAT;RCCINA\n          Don`t you think I know how much\n          money you-'ve got in that goddamned\n          pension fund?\n\n                         HAGEN\n          He's also got the Department of\n          Labor, a board of trustees and\n          the SEC looking down his throat.\n          He can't make that large a commit-\n          ment.\n\n                         DEVITO\n          We` can't put more than twenty-five\n          percent of our assets into real\n\n                         ESTATE ---\n\n                         MAATROCINA\n          Is that what you brought me across\n          the country for?' To tell me I'm\n          getting frozen out?\n\n                         DEVITO\n          Nobody's freezing anybody.\n\n                         MAATROCIXA\n          Then why do I feel these chilly\n          winds nipping at my ass?\n\n                         DEVITO\n          You should felt the winds in that\n          joint. It was Sagen that got me\n          out.\n\n                         (THEN)\n          He comes first. You come second.\n          That's the way it is.\n\n                         MAATH(3CINA\n          How f2ar second?\n\n                         (CONTSNC?? )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I E\n\n                         Y\n\n          67.\n\n                         CQNTNMED\n\n           DEVI.TO\n          Youtve gat to. give me a year.\n\n                         MAATRCCIYA\n          We both want tc get into Atlantic\n          City. L can't give them a year's\n          head. start.\n\n                         ANOTEM ANGLE\n          asDe Vito looks over at Tom Eagen.\n\n                         DEVITO\n          Tom? What if-you people take thirty\n          and Sam takes twenty?\n\n                         SAG=\n          T have to have fifty. That`s defw\n          I inite.. You knew it and agreed to\n          it in front.\n\n                         DEV2TO\n          Sc that we can all part friends ---\n          what about forty-ten?\n\n                         MAATROCIA\n          I •st. not taking any ten. What\n          they get. I get. That is final.\n\n                         DEVITO\n          I All. right, All right. Maybe Z\n           can work something out. Z' 11\n          try`.\n          MAATROC n A\n          Try hard.\n\n                         ANOTSER ANGLZ\n          Maatrocina glances over at Augusta. They rise.\n\n                         EAG=\n          Thank you for coming, Dan Haatro--\n          cina. I'm. sure we can find a way\n          to live toget?..er in peace.\n\n                         4AATRCCI A\n          Of course, my old friend,\n           :Kaatrcc: na ad ? i s aoodbves a; ound and and Angus Lo\n\n                          LEAVED HE\n          49 (C Ci TIN-G\"ED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          68.\n\n                         CON'I INIIED :\n\n                         HAGE??1\n\n                         (TO LAMPONE)\n          You might have to go one on one\n          with Augusto before we're finished.\n\n                         LAN ONE\n          I'm counting on it.\n\n                         ANOTHER ANGLE\n          As Lampone crosses to the window to watch Augusto\n          and Maatrocina leave through the pool area below,\n          Hagen moves to Danny DeVito.\n\n                         HAGEN\n          I'd like to put a couple of people\n          with you,, Danny.\n\n                         DEVITO\n          I don't need your people, Tom.\n\n                         (THEN)\n          Sam and I go back a long ways. He\n          knows I' U. come up with something\n          for him.\n\n                         SANTINO\n\n                         (TO DANNY)\n          Why-don't you stick around? See\n          the show? Maybe the tooth fairy\n          will. stuff one of those long-legged.\n          blondes under your pillow.\n\n                         DEVITO\n          I'll take a rain check.\n\n                         ANOTHER ANGLE\n          As Devito grins, shakes hands around and exits, Hagen\n          turns to Santino.\n\n                         HAGEN\n          I want you to fly to Washington\n          tonight. Talk to Senator Geary.\n          Anybody you have to. But find out\n          who ' s on Maatrocina' s payroll.\n\n                         (THEN)\n          I don't like him knowing about Tony.\n\n                         UUMPCNE\n          Neither do 1.\n\n                         (CONTINMED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          69.\n\n                         CCNTI2MED :\n\n                         SANTINO\n          You think he's walking into some-\n          thinq down there?\n\n                         BALM\n          That's what I want you to find out.\n          As Santino turns and goes, we ---\n\n                         CUT TO:\n\n          EXZ. LATIX-AMERICAN CI\"-''T- NIGHT\n          We are as the sky over the bay of Armando Vidal's\n          Capitol. It is the Fifth Anniversary of his revol-\n          ution and a fireworks display is taking place over\n          the bay. Sky rockets are arching upward over the\n          waters, exploding in cascades of red, green and gold.\n\n                         CUT TO:\n\n          A STET COMM - NIGHT\n          We are on an impromptu exhibition on a street corner.\n          A big, marvelous looking black Girl, in almost nothing,\n          circles with a lean Latino almost touching but not quite,\n          in a hip grinding mambo .\n\n                         E\n\n                         ANOTSER ANGLE\n          as the crowd laughs and cheers them on.\n\n                         A WOMAN\n          Aqua!\n          We move up to a loud speaker attached to an ornate and\n          bunting festooned.lampost. From the speaker we hear:\n\n                         VIDAL'S VOL=\n          But make no mistake, any of you --\n          you of the North American delega-\n          tion in particular. We welcome\n          you to our Count=y -- but the days\n          of exploitation are over.\n\n                         CUT\" TO:\n\n          INT. BALL.COM - ECTE'L OE 3ZVOLCCICN - NIGHT'\n\n          Ei Presi.dente, A.. andc V .dal, is speaking to a crowd\n          j?n the Grand 3allrcomm. The usual tel.evisicn cameras,\n          so,or_.\n\n                         0\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         T\n\n          7Q.\n\n                         ANOTHER ANGLE\n          As the predominantly Latin American crowd cheers, we\n          pick out Tony, in civilian clothes, amongst. the' cheer-\n          ing people.\n\n                         VIRAL\n          The blood of our martyrs is still\n          too fresh on our pavement. We will\n          be friends and neighbors, but never\n          again slaves to Imperialism.\n\n                         1\n          A young Latino, and we will recognize him as Ar=igo,\n          has moved to Tony's side. As the cheers have erupted\n          once more, Amigo, with'a brief nod of the head, indic-\n          ates that Tony shouldfol:ow him.\n\n          ANOT'_R ANGLE\n          As Arrigo goes, Tony looks after him briefly, then over\n          to the American Delegation including Lucas, the Assistant\n          Secretary of State for Latin American Affairs; Cariock,\n          the Speaker of the House and several prominent liberal\n          Senators including McKissick of Utah and the elegant\n          Harthaiemew of Pennsylvania.\n          Also in the delegation, in uniform, is Stu Palmateer.\n          The older man's eyes meet with Tony's for a brief\n          moment. He has seen the exchange. He watches as\n          Tony leaves, then looks over toward u_agudo, Vidal's\n\n                         I\n          Chief of Police. The harsh, heavy Policeman is laugh-\n          1 ing at something Vidal has said, seemingly paying no\n          attention to Tony's exit.\n\n          INT. LOBBY ECT!L ME LA REVOLUTION - :TIGHT\n\n          As Tony is crossing the lobby, Elizabeth Ann Dunne,\n          followed by 'a small entourage, enters. Half a pace\n          behind her is Kenny Morton, her producer and behind\n          them are three men carrying portable TV equipment,\n          cameras, tripods, batteries, so on. She is speaking\n          to 8e.ny..\n\n                         ELIZABETH\n          With all due respect, that's bull-\n          shit.. The man's obviously inccm-\n          petent or the car wcu: d have been\n\n                         WAITING EXACTLY\n          She breaks cf-f . seei nc Tony.\n\n                         \n\n                         \n\n                         \n\n                         \n\n          71.\n\n                         TONY\n          Hi.\n          She stops. Ee crosses up:\n\n                         TONY\n          r thought you weren't coming down\n          here.\n          She doesn't answer.\n\n                         TONY\n          Aren't we speaking?\n\n          E L IZA3ET3\n          I. don't 'know.\n          Elizabeth crosses off toward the desk, Tony looks\n          after her for a moment, puzzled, then crosses out.\n\n          EXT. HOTEL DE LA REVOLCCION - NIGHT\n\n          As Tony comes out of the hotel and crosses the garden\n          grounds toward the street, we see that a fireworks\n          display is still in progress.\n\n          EXT. AVENIDA DE LA R VOLUCION - NIGHT\n\n          Throngs of Merry-makers are moving along the wide,\n          palm-lined avenue that follows the curve of the bay.\n          Many are in cost=e. Some play instruments. Others\n          have bongo drums. The atmosphere is frenetic. There\n          is. a pervasive beat. The whole city seems to throb\n          with it.\n          We are on Arrigo who stands outside the flaw of traffic,\n          cigarette in his mouth, patting his pockets for a match.\n          Tony comes- up and lights ArrigoIs cigarette. As he\n\n                         DOES SO\n\n          A.RASGO\n          Se sure you have your identification.\n          Maguda's pescadores -- you understand?\n          The Police -- they're out f.sh..:g\n          tonight.\n          We are on aaather c=oup Of G-4-is azd :Men i a ?rildly\n          sexual dance.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          72.\n\n                         CONTINT :\n          We pan off them to Arrigo passing. Tony follows. We\n          move. in closer to Tony. A big, fat Mama comes out of\n          a doorway, shakes everything she has at him. Tony grins,\n          pats her on the ass and -passes by\n\n          EXT. EARBOR AREA - NIGHT\n\n          We are on the part of the harbor where the big fishing,\n          boats tie up. The sound of the celebration is fainter\n          now, a couple of blocks distant. We are on an old,\n          but seaworthy fishing boat; the name on the stern is\n          \"Stelia.Maria\". Amigo comes up the pier, pauses by\n          the short gangway.\n\n                         ANOTEER ANGLE\n\n                         I\n          As Tony comes up and joins Axr-igo a Third man suddenly\n          appears out of the darkness. As he comes up to Tony,\n          we recognize Jorge Baxcenas.\n\n                         JORGE\n\n                         (SHAKING HANDS)\n          You bring it?\n\n                         TONY\n          Yes.\n\n                         JORGE\n          This way.\n          Jorge leads them aboard.\n\n          INT. MAIN CABIN (STELLA MAR15) - NIGHT\n\n          In the cabin are Roberto Barcenas, Indio, ?araron,\n          Padre Pepe'and Borracho. There is also a girl, Angelica,\n          25. She has dark eyes and blonde hair. There is some-\n          thing faintly flashy but also enormously sensual about\n          her..\n          One Man is cleaning a stripped automatic rifle. Another\n          is loading a banana. clip with 30 calibre bullets. Barcenas,\n          the girl and some others are bent over a map on the mess\n          table. All. cook up as Tony, Jorge and Arrigo enter.\n          T _tobertc smiles warm- , canes up and embraces Tony.\n           (C'J1?I'r ?`ILT'ED i\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         T\n\n          - 73.\n\n                         CONTLIUM\n\n                         ROBERTO\n          It's good to see you, my friend.\n\n                         (THEN)\n          All goes well?\n\n                         TON\n\n                         (NODS)\n          And you?\n\n                         ROBERTO\n          Everything is ready. Arms and\n          aumm=ition distributed. Now the\n          waiting.\n\n                         TONY\n          You've set a time?.\n\n                         RCBEATO\n          Day after tomorrow. Seven A.M.\n\n                         (CHECKS WATCH)\n          Th.zrty-four hours.\n\n                         (THEN)\n          This is Aujelica. A great good\n          friend of E1 2residente. She knows\n          where he keeps his toothbrush-\n\n                         TONY\n          Good.\n\n                         ANJELICA\n          You have something for me?\n\n                         ANOTHER AN=\n          as Tony unbuttons his shirt and strips off a money\n          belt which he puts on the table and opens. 'IIe takes\n          out some money.\n\n                         TONY\n          A hundred thousand pesos, cash.\n          Count it, please.\n          As Roberto takes up the money and counts it., Tony\n          takes out a small vial in a plastic container.\n\n                         TONY\n          And this.\n\n                         (THEN)\n          It's tasteless and odorless, but\n          it has a vet-,,? short period of tox--\n          ic.Z. ty --- do you understand? Once\n          s opened it must be used within\n          twenty-four hours.\n\n                         (CCNTI UZ )\n\n                         I\n\n                         \n\n                         \n\n                         \n\n                         \n\n          74.\n\n                         CONT2NUHD\n\n          ANJ ?ICA\n          How soon does it work?\n\n                         TONY\n          The symptoms are those of botulisat.\n          High fever, nausea, vomiting, and\n          death. Within an hour.\n\n                         ROBERTO\n\n                         (TO ANJELICA)\n          Hadn't you better be getting back?\n\n                         ANJELICA\n          Yes..\n\n                         ANOTHER ANGLE\n          As Anjelica turns, her coat swings open slightly and\n          we see that she is dressed in a costume.-- a typical\n          Tropicana showgirl kind of thing.\n\n          ANJELIC .\n\n                         (TO TONY)\n          Good night.\n\n                         TONY\n          Thank you.\n\n          ANJ%-\"LICA\n          For nothing. When we meet again\n          things will be better.\n\n                         JORGE\n          Be careful of 4agudo's fishermen.\n          They're out in force tonight.\n          Anjelica nods. Roberto hands her the money, then\n          kisses her on. the cheek.\n\n                         0\n\n                         ROBERTO\n          Vaya can Dios, Chica.\n\n                         ANOTSMR ALYGZZ\n          She staffs the money in the pocket of the overcoat and\n          exits. Roberto looks at A..-.rigo who nods and follows\n          her out. Vow Roberto looks back at Tony.\n\n                         TONY\n          Have the supporting omeratic n.s\n          been set urz?\n          E (CCNT? iUED)\n\n                         \n\n                         \n\n                         \n\n                         \n          .i\n\n          75.\n\n                         CONTINUED :\n          Jorge crosses to a map on the table.\n\n                         JORGE\n\n                         (POINTS)\n          Sere. Here. Here.\n           As Tony crosses and studies the map, we\n\n                          CAT TO:\n\n          A SMALL CITY PARK - NIGET\n          The celebration on Vidal's Capitol is still in progress\n          in this small city park. Suddenly, a police truck\n          appears in one of the streets, blocking it, and a fly-\n          ing squad of armed police piles out of the canvas-\n          covered rear and. The Sergente blows his whistle.\n\n                         SERGENTE\n\n                         (IN SPANISH)\n          identification check. Have your\n          cards ready.\n\n          ANOTTER ANGLE -- TE CROWD\n          We. see A.-rigo and Anj el ca moving through the crowd.\n          They stop, exchange a look and then turn- and Move the\n          other. way_ We follow them through the crowd to a small\n          alley. Amigo and Anjelica and a couple of others, who\n          1 would apparently just as soon not be stopped by the\n           police, move down the alley.\n\n          INT. ALLEY - NIGST\n\n          As Anjelica and Arrigo move down the alleyway, con-\n          gratulating themselves on their escape from Magudo's\n\n                         I\n          men, suddenly three Policemen materialize out of the\n          darkness ahead.\n\n                         POLICEMAN\n\n                         (IN SPANISH)\n          Not so fast my friends ---\n\n                         (THEN)\n          Against the wall, Pockets inside\n          out. You too, blondie.\n\n                         CTT TO :\n\n                         171\n\n                         \n\n                         \n\n                         \n\n                         \n\n          76.\n\n          I NT. POLICE STATION - ?SIGHT\n          Half a dozen of the disreputable fish that have been\n          caught in Magudo's net are sitting on a bench against\n          the bare, dirty walls of the bare, high-ceilinged room\n          with its one unshaded bulb hanging from the ceiling\n          illuminating the suspects and Police alike harshly.\n          On one side of the room is a barred \"tank\" or holding\n          cell. There is a fat Capitan behind the desk and\n          another Policeman, pockmarked and hawkish, looking,\n          working an some papers at another desk. The Sergente\n          comes in with Anjelica and Arrigo.\n\n                         ARRIGO\n\n                         (IN SPANISH)\n          This is an outrage: I protest:\n          SEr A?E\n\n                         (IN SPANISH)\n          Sit down!\n          The Sergente shoves Arrigo violently toward the bench\n          as we ---\n\n                         CQT TO:\n\n          INT'? HOTEL DE LA REVOLTJCICN LOBBY - NIGHT\n          We are on the doors of the hotel as Tony enters and\n          crosses the large lobby toward the elevators.\n\n                         ANTHER ANGLE\n          Over in one corner of the lobby we see Palmateer in\n          a group with several others of the United States\n          Delegation. Tony raises a hand to Palmateer who nods\n          back as Tony continues toward the elevators.\n\n                         I\n\n          ANOTHER ANGLE - BY DESK\n          as the Desk Clerk, who has been watching the door,\n          signals over to a uniformed Security Officer. The\n          Security Officer nods and crosses to intercept Tony.\n          As he does so:\n\n                          SECURITY OFFIC^11\n           Senor Adams.\n           Tony stops. The u::==sing Security Officer crosses\n          .\"p to m .\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         CONTZT ED:\n\n                         TONY\n          Yes?\n\n                         SECURITY CFFICZR\n          There is a message for you. A\n          friend. of yours wishes to see you\n          in the bar.\n          tN'I'. HOTEL DE LA REVOLUCION BAR - NIGHT\n          The place is reasonably crowded, even at this hour.\n          We are on Elizabeth Ann Dunne who is the center of\n          a small circle of admirers, including Kenny Morton.\n          She looks off, sees Tony entering, then excuses her-\n          sel-f and crosses up to Tony who stands in the doorway.\n\n                         CLOSER\n          as Elizabeth comas up to him.\n\n                         ELIZABETH\n          I want to talk to you.\n\n                         0 TONY\n           I take it this definitely means\n           we're speaking again.\n           She draws him to an empty booth and as they sit:\n\n                         I\n\n                         ELIZABETE\n          I'm going to tell you a secret.\n          It wouldn't take an awful lot for\n          me to get hung up on you. Okay?\n          But I've got this problem --\n\n                         (THEN)\n          You scare me.\n\n                         TONY\n          What are you talking about?\n\n                         ELIZABETH\n          You, sport. Adams, Anthony. No\n          middle initial.\n\n                         (THEN)\n          Are you really with the Navy or\n          was that just so much malarkey?\n\n                         TONY\n          What makes you thi:Lk --l'm not with\n          the Navy?\n          1 CO 1 .i...Y U 0\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n                         I\n          77-i.\n\n                         CONT+MME :\n\n                         ?? ELIZABETE\n          Tou know what I heard in Washington?\n          That something very interesting might\n          happen down here.\n\n                         TOONY\n          -Where did you hear that?\n\n                         ELIZABETS\n          You wouldn`t know anything about it,\n          would 'you?\n\n                         TONY\n          I want to know where you heard that.\n\n                         ELIZASETS\n          What difference does it make, it's --•-\n          Tony is on his feet and moving oui of the bar ae 2.\n\n          INT. EOTEL DE LA REVCLLCION LOBBY - NIT\n\n          The all group of Americans, of which Stu Palmateer\n          was part, has broken up. Palmateer is moving toward\n          the elevator, but turns as he hears -----\n\n                         TONY'S VOID\n          Stu?\n          As Tony comes up:\n\n                         PALMATEEE\n          Scmething wrong?\n\n                         TONY\n          I don' t know.\n          They wove casually toward the newsstand. Palmateer\n          Picks up a'Spanish language newspaper, idly scans it,\n\n                         THEN :\n\n          PALZ4AT'.'..E?B\n          What's the problem?\n\n                         TONY\n          Sow many people know about t:-his\n          mission?\n          oAL.u.ATv-ZR\n          Why?\n\n                         \n\n                         \n\n                         \n\n                         \n\n          78.\n\n                         TONY\n          Z think there's been a leak in\n          Washington.\n\n                         PAT MATER\n          There are only five people who have\n          any idea what's going on.\n\n                         TONY\n          is Artier Grundellius one of them?\n\n          PALMAT,.E'?\n          Yes.\n\n                         TONY\n          F Then maybe it's not that serious. -\n\n                         PALMSATEER\n          I If yon think the mission's compro-\n          mised, we'll abort.\n\n                         TONY\n          No. I think it's all right.\n\n                         (THEN)\n          See- you tomorrow..\n\n                         ANOTAR ANGLE\n          Palmateer crosses to the elevator as Tony moves back\n          toward the barroom. We move with Tony as a Bellboy\n          intercepts him.\n\n                         BELLBOY\n          Senor Adams? There is a phone call\n          for you. You can take it on the\n          house phone if you like.\n          Tony nods,.tips the Boy, then crosses to the house-\n          phone and picks it up.\n\n                         TONY\n          Yes?\n\n          IT. DC RSILE EATING PLACE - NIGHT\n          Jorge 3arcenas is at a public phone.\n\n                         JORGE\n          You zeccgnize my voice?\n\n                         (THEM\n          You better net down here right\n          away,\n\n                         0\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n                         S\n\n          79.\n\n          ANOTE R ANGLE\n          As Tony hangs up and turns, Elizabeth is coming up\n          from the bar.\n\n                         £LIZASETH\n          What in the hell's going on around\n          here?\n\n                         TONY\n          Z don't know..\n          Tony turns and crosses out of the lobby as we ---\n\n                         CUT TO:\n          INT. MAIN CABzX (STELLA MARTS) ,- NIGHT\n          we are close on the pack-marked, hawk-faced Policeman\n          who was present at the Police Station when Anjelica\n          and Arrigo were brought in.\n\n          PCLIC°..MAN\n          I'm positive, Doctor 3arcenas. AS\n          soon as they found the hundred\n          thousand pesos in her pocket, they\n          called Colonel Magudo In- T. came\n          as soon as I could.\n\n                         0\n          We have pulled back to include Roberto Harcenas, Tony,\n          Jorge and others, including the Captain of the fishing\n          boat, a man named Mezcurio.\n\n                         ROBERTO\n          She's being questioned now?\n          The Policeman nods. tRoberto turns to Tony.\n\n                         ROBERTO\n          You have to 'assume they'll be\n          tortured. You have to assume\n          that they'll talk.\n\n                         TONY\n          Then you're going to have to move\n          right now.\n\n          ANOTL ANGL.\n          Jorge start: passing out weapons and hand grenades.\n          Tony takes a su.?:-.machine gun and checks it, as we ---\n\n                         \n\n                         \n\n                         \n\n                         \n\n          82.\n\n          EXT. JAIL OUSE ROOF - NIGHT\n\n          A couple of Policemen are on duty on the jailhcuse\n          roof. Their names are Lopez and Ruiz. Lopez. is look-\n          ing into the square.\n\n          LOPEZ'S ANGLE (THE SQUARE)\n          We are shooting down into the square from behind Lopez\n          as Ruiz comes up and joins him. We see the two cars\n          stop outside the jail, and the occupants pile out. Tony,\n          Jorge, Roberto and Indio from the first car, Sorracho,\n          Pepe and 'araron from the second. They are all armed,\n          carrying LAW rockets and slung-sub-machine guns.\n\n                         ON RTXIZ\n          as he jumps up onto the parapet, cocks his own machine\n          gun and fires down into the square. As his bullets\n          stitch the cobblestones moving up toward the group.\n\n                         ON TONY\n          At the sound of Ruiz 's first shot, he's got his sub-\n          machine gun in his hands. He fires from the hip.\n          Lopez throws up his hands, falling back. Ruiz spins\n          on the parapet and falls into the street.\n\n          =T. THE JAII.HOUSE - NIGHT\n          Carbajal and Mosca have started firing out of the gun\n          ports in the doors.\n          i M=. TEE STREET OUTSIDE THE JAIL -- NIGHT\n          As Indio gets hit in the forehead and is jolted back-\n          wards, his face a bloody mask, Tony, Jorge and Roberto\n          have picked up their LAW's and fire almost simultaneously.\n\n          EXT. TEE JAILHOUSE - NIGHT\n\n          as the big, heavy doors blow away ---\n          INs°. ITS...-\" JAILEOL;S ?II(an\n          as - the second door is blown back into the roc=. The\n          ceiling is caving in, piaster is falling. The ai= is\n          thick with plaster dust and smoke.\n\n          (CCN'I'IVU ?,J )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n          a3.\n\n                         CONT IU :\n          Mosca sits against the wall, mouth open. His leg lies\n          across the room.\n          The Prisoners in the holding cage are screaming. A\n          Policeman runs in from another room just in time to\n          be cut down by Jorge who bursts through the door\n          followed by Tony and Roberto.\n\n          ANOTHER ANGLE. -- BAS = STAIRS\n          as Colonel Magudo runs up the basement stairs, pistol\n\n                         IN HAND:\n\n                         ANOTHER ANGLE\n          as Jorge, Tony and-Roberto all `ire at the same time,\n          blowing Magudo back down the basement stairs. Tony\n          and the others dive down the stairs.\n\n          EXT. THE CENTRAL SQUARE - NIGHT\n\n          As the occupants of the first car are attacking the\n          Jailhouse, Fa=axon, Pepe, and Borracho launch an\n          0 attack on the Presidential Palace.\n\n                         ANCTBER ANGLE\n          A.: the sleepy Soldiers come tumbling cut of the build-\n          ing in confusion, Barracho is spraying them with his\n          automatic weapon while the other two are blowing the\n          front doors away with their rockets. Smoke and plaster\n          dust is heavy in the square. rives start to lick and\n          flicker.\n\n                         ANOTHER ANGLE\n          As Borracho, Pepe and.Fararon run into the Presidential\n          Palace, somebody has started pealing the bells of the\n          Cathedral.\n\n          IY'S. THE PRESZDE. ITIA.L PALACZ - NIGHT\n          As Borracho, Pepe and Fararon rjn into the great\n          central hall, wit:-1 its cu gyring S--a.=Case, and great\n          crystal chan_delie=\n\n                         \n\n                         \n\n                         \n\n                         \n\n          84,\n\n                         CONTIYTJED :\n\n                         BORRACHO\n          This way.\n          EEO rugs up the stairway, by the others. At the top\n          of the stairs he turns, fires at the chaia that holds\n          the chandelier anchored. The ceiling chews away.\n          The chandelier falls with a crash of crystal.\n\n          CIIT TO,.\n          INT. BASMiENT (CI TRAL JAIL) - MIGHT\n          There are a dozen doors leading into a dozen basement-\n          cells- Tony, Jorge and Roberto are searching them.\n          Tony enters a cell, then we hear:\n\n                         TONY'S VOICE\n          Roberto!\n          As Roberto hurries into the cell\n\n          INT. THE CELL -- YIGET\n\n          There is a gully in the ceiling with a line led. through\n          it and tied off. The other end of the line is. tied\n          around Arrigo's ankles, suspending hi= upside down.\n          Eris hands are tied behind him. His head and shoulders\n          are not visible as he is immersed, head-down in a large\n          tuh of water.\n\n                         TONY\n          Quick!\n          Roberto runs in, whips out a knife and as Tony grabs\n          the motionless body, the other cuts the line.\n\n                         ANOTEFR ANGLE\n          As they gently lay Arrigo's body on the floor, it is\n          apparent he is dead.\n\n                         CQT TO:\n\n          INT. THE BASEMENT - YIGET\n\n          As Jorge throws open a door, his face goes white.\n\n                         FORGE\n          motheY of God.\n\n                         \n\n                         \n\n                         \n\n                         \n\n          85..\n\n                         CONTINDED:\n          We have panned over to the door. We are now shooting\n          toward the door with our view mainly blocked out, but\n          we can see that Anjelica is dead, naked and tied spread-\n          eagled, face down on a table.\n\n                         CUT TO:\n\n          ZNT. PRESIDENTIAL PALACE - NIGHT\n          As Bcrracho Fararon and Pepe are rune og down an\n          upstairs hallway, throwing open doors, a Soldier\n          appears, fires, 3orracho spins and falls.\n\n                         ANOTHER ANGLE\n          As E'ararcn cuts the Soldier down, then turns to Pepe:\n\n                         R ARARON\n\n                         (IN SPANISH)\n          Let's get out of here!\n          They turn and ran.\n\n          • CUT TO:\n\n                         S\n\n          EXT. THE JAILHOUSE AND SQUARE - NIGHT\n\n          A couple of Police vehicles and an Army truck careen\n          into the square, skid to a stop and the Soldiers and\n          Policemen pile out and take cover an the square. The\n          bodies of Ruiz and Indio are still, where they fall.\n          At a command from the Officer in charge, the Soldiers\n          and Policemen start moving forward, from cover to\n          cover, laying down a steady rattle of gunfire.\n\n                         CUT TO:\n\n          MM AN ALLEY BEHIND TEE JAIL - :NIGHT\n          Jorge, Roberto and Tony are running along an alley\n          in the darkness, bent over. One street away, in the\n          town square, apparently all hell is breaking loose.\n          As they run:\n\n                         FAARARON\n          Hey: This way.\n          They stop. Fara_-cn and 2epe are standing _ in -: mouth\n          of a i?.tp e n,ar-r-cw open -ig between houses\n\n                         0\n\n                         \n\n                         \n\n                         \n\n                         \n\n          86..\n\n                         ANOTHER ANGLE\n          as Pepe and Fararon disappear into the darkness follow-\n          ed by Tony, Roberto and Jorge.\n\n                         CUT TO:\n\n          E=. ANOTHER STREET - NIGHT\n          A Police car with the numerals \"22\" painted on the\n          side is abandoned at an angle in the street, the\n          doors open. -\n\n                         ANOTHER ANGLE\n          Pepe runs up, looks in. Tony and the others follow.\n\n                         PEPE\n          The keys are gone!\n\n                         TONY\n          Get in.\n          Tony is under the dashboard crossing the wires as\n          the others start piling in. The motor starts, Tony\n          slides behind the wheel. As they drive off ---\n\n                         CUT TO:\n\n          ZN' . POLICE CAR TAM=-TWO- NIGHT\n          Tony is driving. Roberto is next to him in the front\n          seat. In the back are Jorge, Pepe and rararon. As\n          they come to a crossroads:\n\n                         TONY\n          Which way?\n\n                         ROBERTO\n          Lett.\n\n                         (THEN)\n          When will the plane be coming?\n\n                         PEPE\n          Four. They will land at Quebrada.\n          Roberto looks at his watch, then\n\n                         ROBERTO\n          We can make it. Left again.\n\n                         E\n\n                         \n\n                         \n\n                         \n\n                         \n\n          87.\n\n          EXT. STREET CORNER - NIGHT\n\n          As Police Car, Number 22, comes left around the corner\n          a second Police Car is coming the other way. The two\n          cars barely miss each other, just kissing as they pass.\n\n          INT. POLICE..: CAR 22 - NIGHT\n\n          The comandeered Police Car swerves as Tony fights\n          the wheel, and finally steadies it.\n          TONY'S POINT OF Vt!W (THROUGE BACK WINDOW)\n\n                         I\n          The other police car has spun and stalled momentarily.\n          As the Driver of the car gets it started and straightened\n          out in pursuit, we can see the Second Policeman on the\n          hand mike. We can hear his excited voice in. Spanish\n          coming over the police radio.\n\n                         RADIO VOICE\n\n                         TIN SPANISH)\n          We have seen the terrorists. They\n          are in Police Car Twenty-too going\n          north on Avenue of the Martyrs.\n\n                         (THEN)\n\n                         0\n          All units. All units. Terrorists\n          seen going north on Avenue of the\n          martyrs .\n          Now other traffic can be heard on the Police radio as\n          other Police cars respond to the message.\n\n                         SEVERAL COTS\n          of various Police cars as they get the message. Some\n          swing around in Q-turns, Their sirens are winding up\n          to a howl.\n\n          INT. POLICE CAR NITMBER 22 -- NIGHT\n\n          We are shooting back through windshield. Pepe is look-\n          ing out the back window and we see the following Police\n          car. Now another joins it. Now still another. Police-\n          men start firing out of their. cars. The back window is\n          starred as a hole suddenly appears in it, only - inches\n          from Padre ?ece' s head. He looks at the ho l e and tu.-ns,\n          crossing 7:.msei f .\n\n                         PEPS\n          .;esus, Mary and Joseph.\n          (CO N'IiNII..D )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         K\n\n                         AS\n          `Cis;n7VED %\n          As another bullet hits the car somewhere with a clang.\n          of metal, Roberto turns td Tony:\n\n                         GO-III E1 0\n          Go right on Quebrada.\n\n          THE I BSPE= C3ASE\n          as the pursuing Police cars increase their numbers and\n          come closer, the-Comandeered car, carrying Tony and the\n          others, makes a right- turn on Avenue Quebrada, leading\n          out of town. Parked at the curb is a non-descript sedan.\n\n                         FEATURSITG TONY\n          As he drives, we see that Roberto has taken from his\n          knapsack-pouch a mall detonating device such as we\n          have seen in the demonstration of the doomsday car at\n          Camp Peary.\n\n          EXT. TEE STREET CORNER - NZGRT\n\n\n                         0\n          As the first of the pursuing Police cars starts around\n          the corner, suddenly the non-descript sedan parked at\n          the curb detonates -- disintigrates in a blinding dazzle\n          of light. In the jolt of the shock wave, every window\n          within half a mile radius shatters.\n          A vast ball of flame and black, heavy smoke billows\n          upwards from the corner where the doomsday vehicle and\n          the first car were immolated.\n          Now, the following Police cars, unable to stop and\n          unable to avoid the flames which have spread like\n          napalm all over the whole corner, skid into the flames\n          and smoke, plowing into the wreckage.\n          The buildings are in flames. A Policeman, his uniform\n          and hair on fire, runs screaming out of the inferno.\n\n                         DISSOLVE TO:\n\n          EX. A COUNTRY ROAD - N1 GET\n          as Pc?l.ce Car Nunber 22, tarns down a side road, through\n          a gate, then up and over a h-4-11. We pa.-n to the gate,\n          over which are the words: \"° i.-sca Quebrada\".\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         D\n\n          89.\n          EXT. AIR FIELD (F=CA. QUEBRADA) - NIGHT a\n          The Police Car comes to a stop at the side of what\n          is apparently a hacked-out -landing strip in the middle.\n          of a small valley. There is a cane field on one side,\n          the green stalks are give or six feet high.\n          Tony, Jorge, Roberto and Pepe get out of the car.\n          Roberto opens the back door and, to Fararon.\n\n                         ROBERTO\n          Come on! Do- you want to ---\n\n                         ANOTHER ANGLE\n          Roberto has broken off as he looks at-Fararon. We\n          realize that sometime during the chase the Pharoah\n          has taken a bullet in the chest. Se's dying, and pink\n          bubbles of blood fora and break on his lips as he\n          stiizggles for breath.\n\n                         ROBERTO\n          I'm sorry, old friend.\n          Tony crosses and gets back into the car to help\n          Fararon.\n\n                         ROBERTO\n          I'm afraid he's finished.\n          At this point we hear:\n\n                         JORGE'S VOICE\n          Here they come:\n\n                         ANOTHER ANGLE\n          as Jorge, Pepe and Roberto run to the center of the\n          airstrip, looking upward. We can hear the sound of\n          an approaching aircraft.\n\n                         ON MOSERTO\n          as he points a flashlight at the sky and signals a\n          short and a long, the letter Alpha.\n          AlNCTRER A0IGL.E - I:IC=1r--NG `\"HE OC-6\n          The aircraft is now visible. ?rcm the Pilot's coc pit\n          we see the answering signal , a long and th--ee shorts,\n          the letter Bravo.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          9Q.\n\n          PULL SECT - THE DC-6\n          as it swings in on the final approach and starts to\n          settle in for a landing.\n\n          CUT, TO:\n\n          EXT. COUNTRY ROAD - NIGHT\n\n          A couple of jeep loads of Soldiers turn down the\n          same side road that we saw Police Car 22 take a few\n          minutes earlier. As the jeep loads of Soldiers pass\n          through the gate on which are the words, \"Finca\n          Quebrada\", we ---\n          Ct?T' TO:\n\n          EXT. THE A2RFT L - NIGHT\n\n          The DC-6 lands and swings around, taxing bark. over\n          the rough ground.\n\n          CLOSE - TONY\n          He senses something wrong, and pausing by the edge\n          of the airfield, calls to the at-tars.\n\n                         0\n\n                         TONY\n          Wait a minute.\n\n          ON ROBERTO, PMPE AND JORGE\n          as they run for the plane.\n\n          ANOTHER ANGLE - ON THE PLANE\n          as the plane swings around again and the door opens.\n          We reveal E'rankie Rizzi in the doorway.\n\n          CLOSE ON RRANRI\"\n          as he looks out.\n\n          ON ROBERTO, P E AND J ORGE\n          as they ran toward. the plane.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         K\n\n                         91-\n          £Xfi. RIM OF THE ETT.r.\n          as the two jeeps come up to the rim of the hill. One\n          jeep turns a powerful spotlight onto the airfield.\n          The A.IRPT T-fl - NIGHT'\n          as the spotlight catches Roberto, Pepe and Jorge in\n          its beam.\n\n          ON THE SOLDIERS\n          as they fire.\n\n          ON ROBERTO, PEPE AND JORGE\n          as they are chopped down, one after the other.\n\n                         OK TONY\n          as he turns and fires at the jeep loads of Soldiers\n          with his automatic rifle.\n\n          ANQTEER ANGLE - THE SOLDIERS\n          as Tony's fire shatters the spotlight. Some Soldiers\n          fall, others fire at the DC-6.\n\n                         THE DC-6\n          as it starts to pick up speed, trundling over the.\n          rough ground.\n\n                         ON TONY\n          as he runs for the plane.\n          ON ?RAl?1Z\n          as he sees Tony.\n          A,NoTr.. ER ANGLZ\n          As Tony :'ims up -o the m lane , =rarekie reaches down,\n          drags him up and in. Bu-1-let hales are apvearing\n          the fuselage of the plane.\n\n                         \n\n                         \n\n                         \n\n                         \n\n          92.\n\n          ON THE SOL4IMRS\n          as they fire.\n\n          ON TIM DC-6\n          as it rises into the air and banks off to the Nor t. % ---\n\n                         CUT TO:\n\n          E,XT'. VZDAL' S PRES=ZNTIAL PALA - DAY\n          Armando Vidal, his face a harsh mask, stands in the\n          shattered doorway of the Presidential palace looking\n          off toward-the central Square..\n\n                         VIRAL\n          The cetinter-revolution has failed.\n          The. people once more did not -rise\n          up as expected.\n\n          VIRAL' S POV (THE SQUARE AN TAZLHCUSE) - DAY\n          In the. battle-torn square, outside the jailhouse, we\n          see an angry croard of Rioters care ing the dead and\n          bloody body of Roberto Barcenas.\n\n                         VIRAL\n\n                         (COMING OVER)\n          And once more the rich are bewilder-\n          ed by the fact that the poor are un-\n          I willing to die for them.\n          Somebody produces a rope, somebody else throws a loop\n          around his feet and Roberto is hoisted upside-down to\n          the top of an ornate old lam post.\n\n          NLMADT' S VOSCZ.\n          And while E1 Presidente was speak-\n          ing, in understandable bitterness,\n          crowds in the Capitol were running\n          rampant, stringing up the bodies of\n          the leaders of the failed coup d'etat.\n\n                         QIM CROWZ\n\n                         T\n          as the people cheer. Somebody produces an American\n          flag and sets it afire. it buns In the street. lids\n          kick at it, 5L it on it.\n\n                          \"\n\n           ,GCN'?'. .ZD?D )\n\n                          '\n\n                         \n\n                         \n\n                         \n\n                         \n\n          93.\n\n                         CONTIN= :\n\n                         NEWSMAN'S VOICE\n          They burned American flags and\n          threatened the safety of the Amer-\n\n                         ICAN DELEGATION\n\n          EXT. AIRFIELD - DAY\n\n          Uniformed Soldiers with fixed bayonets are guarding\n          the airfield. Crowds are gathered outside the gates.\n\n                         ANOTHER ANGLE\n          as limousines 4ith soldiers on the ;enders and roofs\n          come inching through the fist-waving, threatening mob.\n          As the gates are opened, Soldiers with bayoneted\n          rifles force the crowds back so that the cars can get\n          through,\n\n          NEWSMA2X' S VOICZ\n          -- who were taken to the airfield\n          under military escort.\n          We see the limousines stop by a waiting transport\n\n                         I\n          plane, and -- with the guns of the Soldiers holding\n          off the angry mob -- the American Delegation to Vidal's\n          celebration hurries into the big airliner..\n\n                         CLOSER\n          In the American group we see Elizabeth Ann Dunne; the\n          Senators McKissick and Barthalemew; Assistant Secretary\n          Lucas; Speaker of the House, Cari.ock; and Stu Palmateer,\n          looking cooler than he could possibly feel.- Over this\n\n                         WE HEAR:\n\n          NEWSMAN' S VOICE\n\n                         (CONTINUING)\n          Soldiers with bayonets were forced\n          to fight off the angry mob which\n          clearly blamed the United States\n          for the abortive coup.\n\n                         CUT TO:\n\n          !XT. THE STATED PART!-= -- DA ;\n          As Are r? z?de::.ias comes out of the State Department\n          and crosses to his car, he is surrounded by Newsmen.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          94.\n\n          CON'' I?7IIED\n\n          NEWSMAN 'S VOICE\n          Meanwhile in the Nation's Capitol,\n          Arne Grvndellius, the Secretary of\n          State, was beseiged by Newsmen as\n          he left the State Depa.tent after\n          an all-night session.\n          CLOSER - ON GRt'YDELL tD S\n          as the Newsmen are thrusting microphones into his face,\n          asking questions.\n\n          FI3ST NEWSMAN\n          Mister Secretary! Mister Secretary!\n          Can we get a statement?\n          Grnndelli.us stops.\n          GRIINDE,tLZUS\n          We will make an official statement\n          tomorrow.\n          NEWS ii TOGETRER\n          What about Vidal's accusations --\n          Have you Xosygia's statement? will\n          you speak to the United lations?\n          Was Doctor Barcenas an 1nerican\n          agent?\n\n          GR=EZS.ZC S\n          P One at a time. One at a time.\n\n          2ND NEWSMA,\n          According to world opinion, the\n          CIA was behind this.\n\n          - GRONDELL.\"II5\n          Gentlemen: Gentlemen: I am late\n          for a meeting at the White House\n          i but we catagorically deny these\n          allegations. The United States\n          Government does not use assassin-\n          ation as an iastrent of foreign\n          policy.\n\n                         ANOTHSR ANGLI\n          AS G-=de? ii us starts to move throuch --he crowd of\n          3eoorters,\n          (CCNT_-6 _qT '\n\n                         0\n\n                         \n\n                         \n\n                         \n\n                         \n\n          95.\n\n                         CONTMED :\n\n                         3RD NEWSMAN\n          What about the rumors' that the under-\n          world was involved in this?\n\n                         GRDNDELLIIIS\n          If the underworld was behind it\n          and I don't rule out the possibility\n          at all -- that fact will be brought\n          to light in open hearings before a-\n          special committee of Congress. It's\n          in the works right this minute, and\n          subpoenas will be coming out by the\n          weekend.\n\n          INT. ELIZABETH'S AP T - DAY\n\n          Tony, dressed as last we saw him at the airfield, is\n          in Elizabeth's apartment watching the news on. her liv-\n          ing room television set.\n\n          NEWSCASTER ° S. VOICE\n          And now for further reactions to\n          today's developments, we take you\n          to the United Nations where Sander\n\n                         VANOCUZ --\n          Tony has turned off the television set with a remote\n          control switch, having heard the. sound of a key in\n          the front door. He rises and turns as Elizabeth enters\n          carrying a paper bag.\n\n                         TONY\n          Listen, I --\n          He breaks off as Elizabeth's face goes pale with shock\n          and she drops the bag on the floor. A couple of oranges\n          roll across the rug as:\n\n                         TONY\n          I'm sorry, I had to talk to you\n          She looks at him her expression almost readable as anger;\n\n                         ELIZABETH\n          You're listed as missing. It's on\n          the wire.\n\n                         I\n\n          TONY.\n          Well, much as I.hate to disappoint\n          everybody.\n\n                         (CONTINCRM )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          96.\n\n                         CONT=M= :\n          Suddenly Elizabeth drops to her knees and starts\n          crawling around on the floor, blindly searching for\n          the oranges.\n\n          - ON TONY\n          as he locks at her for a puzzled moment, then drops\n          to his own knees and taking her by the shoulders,\n          straightens her. Tears are pouring down her cheeks.\n\n                         TONY\n          Bey, wait a minute\n          He kisses her.\n\n                         TONY\n          What's all. this crying shit?\n          She smiles, sniffles, blinks the tears out of her eyes,\n\n                         THEN:\n\n          ELZZ.A$ETE\n          You want an egg sandwich?\n\n                         CUT TO\n\n          INT. ELIZABETE' S EITCHEY - NIGHT\n\n          Tony and Elizabeth are in. the kitchen having sandwiches\n          and coffee as the kitchen wallphone rings. Elizabeth\n          gets up and answers it.\n\n                         ELI ZABETH\n          Hello?\n           NT. MoREHCIISE's O TI Y (LA.vGLEY) - NI zHT\n\n                         1\n          Stu Palmateer is at the desk. Morehouse is with him,\n          reading some reports.\n\n                         PAIMATE' R\n          This is Captain Pa.TLrnateer. I got\n          a message to call this number.\n\n                         ON ELIZABETH\n          as she speaks into the phane\n\n          LI2ABE:'H\n          Yes. Hold on for a minuts, will\n          you.\n          + ( ( ( to Tony??yy\n           Is your call.\n\n                          (CCNT E0 )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          97.\n\n                         CCNTINCTED\n          Tony crosses over and takes the phone.\n\n                         TONY\n          Stu?\n\n                         PATWMATEER\n          Are you all right?\n\n                         TONY\n          Considering.\n\n                         (THEN)\n          Some friends of mine flew me up\n          to Maryland this morning. I thought\n          maybe I'd better not go directly\n          home.\n\n                         PALMATEER\n          Good.\n\n                         TONY\n          How are you?\n\n          PAL?`SATEYR\n          Ten kinds of blue hell are breaking\n          loose out here. Are you all right\n          at that number till tomorrow?\n\n                         TONY\n          Fes.\n\n          PAL.MATEER\n          Then I' 11 get back to you.\n          Click, as Palmateer hangs up. Tony t•.irns to Elizabeth.\n\n                         TONY\n          I hope you don't mind a house guest.\n\n                         0 CU TO:\n\n          I INT. AN AUDITQRIGM (NEST ORLEANS) - NIGH\n\n           We are an a cheering audience of Longshoremen.\n           ANOi= AN= 1:,i.iii S2EA2=' S ?LA -'- Qi M\n           Danny'DeVito is holding up his hands to the crowd.\n           Behind him., red, white and blue bunti.^g. ?oste.zs\n           reading: 1 OTT YOR i=TO -- Gi V' Tom: CYZON . ACX\n           To : E VMSBERSHZP\" .\n\n                         \n\n                         \n\n                         \n\n                         \n\n          98.\n\n          EXT. AUDITORIIIM -- NIGH':'\n\n          As Danny DeVito comes out of the stage door of the\n          auditorium, Ralph Augusta comes up to him.\n\n                         AUGUSTO\n          Sam wants to see you.\n\n                         DEVITO\n          I'= at the Pcnchartrain.\n          Ralph takes Danny by the arm and moves him toward a\n          chauffeured limousine waiting at the curb. As he does\n\n                         SO :\n\n                         AUGUSTO\n          Now. Tonight. He's got an idea\n          how to get everybody off the hook.\n          As.Augusto opens the door and ushers Danny into the\n          back seat of the car, we ----\n\n                         CUT TO:\n\n           MCT. BOURBON STREET MEW ORLEANS) - NIGHT\n          0 As the limousine moves dawn Bourbon Street with its\n          hockey tonks and jazz joints.\n\n          INT. THE LZIMOUS INE - NIGHT\n\n          As they drive down Bourbon Street, Augusta leans for-\n\n                         I\n          ward to the Chauffeur.\n\n                         AUGUSTO\n          We'll go in the back way.\n\n                         CHAUFF EM\n          I Yes Sir.\n          The Chauffeur turns down a side street and up an alley.\n\n          E=. TEE ALLET -- NIGHT\n          A truck is blocking the alley. A big, cheerful-Looking\n          Laborer with a ;knit cap on his head, is sitting on a\n           big barrel by th rear of t h t= ck.\n          h T e Eli cusine pu e l ls ap be fer hi ? n d e 1 s the 2 trac /+ k, s c y/?Yy?? toys a :'+t^i /? i a\n           inside a nearby cltth we hear a Jazz t=,=zet on a long\n           ride.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         ??T\n\n          99.\n          MT. THE LnKOII&INE\n          as Augusta leans forward.\n\n                         AUGIISTO\n          Give him the horn.\n          The Chauffeur honks the horn. The Laborer grins over\n          at the limousine, gestures, palms up.\n\n                         AUGUSTO\n          So we walk. It's not far.\n\n                         ANOTHER ANGLE\n          as Augusta, Danny Devito and the Chauffeur move up\n          toward the rear of the truck, the Laborer jumps down\n          off the big barrel he's been sitting on.\n\n                         LABORER\n          Hello Danny.\n          At this point the Chauffeur wheels and grabs Devito\n          in an arm lock, with one gloved hand clamped over his\n          mouth.\n\n                         ON ABGUSTO\n          as a knife suddenly glitters in his hand and he plunges\n          it into Devito. Danny is kic-ting and st-aggling.\n\n                         ACWSTO\n          Hold the cocksucker still..\n          As Augusta plunges the knife into Danny again and\n          again and again.\n          AN WAVZ\"11 r?Nf:I.E\n          The Laborer has taken the lid of, the barrel. Danny\n          sinks to the pavement, convulses and dies.\n\n                         ANCTEER ANGLE\n          as the three men pick up Devito`s body, stuff him\n          into the barrel, then hoist the barrel into the back\n          of the truck.\n          The jazz tr ttpet is sti l playing, as we --\n\n                         CTT IRA\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         K\n\n          100.\n\n          =NT. ELIZABETH'S BEDROOM - DAY\n          It is the folloaring morning. Tony is in bed, asleep.\n          Elizabeth enters, turns on the bedroom TV set and\n          these crosses and shakes Tony. Tony comes awake fast.\n\n                         TONY\n          Wha_ is it?\n\n                         ELIZABETH\n          Your friend, DeVito ---\n          The TV` set has warmed up and now the Newscaster's voice\n          comes over the pict'..ire of a middle-class house in\n          Bayonne, New Jersey.\n\n                         NEWS CASTER\n          There have been no ransom demands\n          and the Devito family -- although\n          concerned -- are not yet alarmed.\n\n                         (THEN)\n          Police have stationed a guard on\n          the ex-union Leader's Bayonne, New\n          Jersey home were his 'rife and groom\n          daughter are in seclusion. Informed\n          sources fear an eruption of mob\n          violence if the popular Labor Leader\n          has met with foul play.\n\n                         I\n\n                         ANOTHER ANGLE\n          As the Television Newscaster switches to another item,\n          we see a burning house, fire engines, so on.\n\n                         NEWSCASTER\n          Long Beach, New York. In a possibly\n          related incident, the home of re-\n          puted Syndicate figure Santsno\n          Corleone was firebombed early this\n          morning. Corleone, thirty-five, is\n          in guarded condition at Saint\n          Catherine's hospital with first\n          } degree burns over two thirds of his ---\n           Tony is out of bed. Be has switched off the TV set\n           and crossed to the phone.\n\n                          TONY\n           Long distance =;fc_aa,ion, please.\n           h T e number of the Ve?as ?alms ---\n\n           C , D TO:\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         ??3\n\n          EXT. LAS VEGAS AIRPORT - MIGHT\n\n          It is early evening of the same day.\n\n          INT. LAS VEGAS AIRPORT - VIGET\n\n          as Tony comes out into the central area of the Vegas\n          Airport. He pauses, looks around.\n\n                         ANOTHER ANGLE\n          as Rocco Lampcne crosses up to him.\n\n                         ?? LAMPONE\n          Let's go.\n\n                         ANOTEER AUG=\n          as Tony follows Lampone.\n\n          EXT. LAS VEGAS AIRPORT - NIGHT\n\n          as a big car driven by a Button an named Fritz pu.Us\n          up. Rocco opens the door to the backseat, gestures\n          Tony in. As Tony gets in ---\n\n                         ON ROCCO\n          I He looks around. There is a car full of Button Men\n          in front of them. Another car full of button men\n          behind. Rocco nods. Gets a nod back >rom each driver.\n\n          INT. CAR - D1=GET\n\n          As Tony gets into the backseat, we see that Tom Hagen\n          is there. Rocco Lampone now gets into the front seat\n          alongside the driver, Fritz.\n\n          A L MP ORE.\n          Tony, this is Fritz.\n\n                         (THEN)\n          Let's move.\n\n          ?NOT= AIGI2\n           as the cars drive out om! the ai Ci .. a 1 e, 2Eagef' S\n          car in the middle.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          102.\n\n          INT. HAGEN `S CAR\n\n          as Hagen turns to. Tony.\n\n                         - SAGR'??1\n          Danny's dead. They found him in\n          a barrel at the mouth of the\n          Mississippi.\n\n                         TONY\n          Maatrocina?\n\n                         HAGEN\n\n                         (NODS }\n          It's going to be bad for a while,\n          that`s wby I wanted you out here.\n          It's time you-Jesus Christ Fritz\n          This last as another car swings out of a side street\n          and a Hood leans out of the rear window with a t•.reLve\n          gauge p=p gun, and blasts at Fritz.\n\n          ON RAG-EN'S CAR\n          as a blast of heavy shot takes out the windshield\n          and blows most of Fritz's head away.\n          ' : NT. SAGrzN ' S CAR\n          As a fountain of blood gushes up from the stalk of\n          Fritz's neck and Hagen's car starts to swerve into\n          the curbing, the Gunman in the other car sends two\n          more blasts of deer--load into Hagen's car.\n\n                         ANOTHER ANGLE\n          as the Corleone Button Men in the following car blasts\n          at the attackers, the Shotgun Man blasts away.\n\n          I ANOTT ER ANG=\n          as Hagen is hit by half a dozen buckshot and he flaps\n          over dead, onto Tony. The caw hits and tolls.\n\n                         ANOTSER ANGLE\n          The Hagen car comes to a s tor on its wheels again.\n          The Corleone Butt-an Men it the fol icwinc ca= -snt?\n          cut, run up.\n\n                         0\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          103.\n\n                         CLOSER\n          As Rocco Lampone comes out of the wreckage, the Button\n          men are pulling Tony -- who is unconscious -- out.\n          Lampone looks at Fritz and Hagen.\n\n                         LONE\n          Not a goddamned thug you can do\n          for them. Let's get out of here.\n\n                         ANOTBER ANGLE :\n          As The Corleone Men carry Tony to the other ca= and\n          get him inside, people are starting to gather.\n\n                         LAMPONE\n          It's all right. It's all right.\n          Gangway. We're getting him to\n          the hospital! Man's hurt here!\n          Clear the road:.\n          As the Bystanders move back, the Corleone car burns\n          rubber and digs out\n\n                         CQT TO:\n          MT. BEDROOM (CORLEONE COMPOIINf, TAHOE) - NIGTT\n          Camera is subjective, the screen is pitch black with\n          a single red-orange dot moving erratically in the\n          center. It is the coal of a cigarette as someone\n          takes a last puff, tamps it out, then scratches a\n          match and. lights another.\n          In the flare of the match we see a Nurse's bulldog\n          face.\n\n          ANOTHER ANGLE - INCLC1 ING TONY\n          He wears a bandage around his head like a burban. Be\n          is looking at the Nurse in the flicker of the match\n          flame.\n\n                         TONY\n          Who are you?\n\n                         XURSE\n          OY., gc?od, you're awake.\n\n                         TC Y\n          Wait a minute.\n\n                         S\n          t CfJI3'?'LD }\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         A\n\n          104.\n\n                         CONTD DZD :\n\n                         NU'RSZ\n          I'll be right back.\n          The nurse has risen. She switches on a lamp and\n          exits.\n          ANOThR ANGLE (MIRROR SHOT)\n          We are angled into a mirror over a chest of drawers,\n          holding on Tony as he sits up in bed, then gingerly\n          gets to his feet. He is wearing silk pajamas. Now\n          he crosses to the mirror and checks himself out.\n          Aside from the bandage he seems to be in one -piece.\n          Now, in the mirror we see the door open and silhou-\n          etted in the doorway, the figure of Michael. Tony\n          turns.\n          ANOTffER ANGL -\n          as Tony and Michael look at one another for. a long\n          moment.\n\n                         HICEAEL\n          How are you feeling?\n\n                         S TONY\n          What am I doing here?\n          As Michael comes into the room and closes the door:\n\n                         HICEAEL\n          I had Rocco bring you.\n\n                         (THEN)\n          How's the head? A little -pain?\n\n                         TONY\n          A little.\n\n                         MICHAEL\n          The Doctor says it's nothing.to\n          bother you, but I've always found\n          it's easier to be brave about some-\n          body else's headache.\n\n                         TONY\n          Did he say how soon T could leave?\n\n                         MIC'Z??\n          Sit dcwn. Sim down..\n\n                         S\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         ANO'TEMR ANGLE\n          Tony doesn't sit down.\n\n                         TONY\n          r'd like to get out of here as soon\n          as possible.\n\n                         MICHAEL\n          Let me ask you a question. Have\n          you any idea where you stand?\n\n                         TONY\n          I'm not totally stupid.\n\n                         MICHAEL\n          I aC 't think you're stupid. I\n          think you're smart. 3 t not smart\n          enough.\n\n                         TONY\n          t'n willing to learn.\n\n                         MICHAEL\n          Good.\n\n                         (THEN)\n          You've let your enemies get too\n          close to you.\n\n                         (THEN)\n          Those people who tried to kill you\n          in Las Vegas, they were Maatrocina's\n          people.\n\n                         TONY\n          Are 'you sure?\n          Michael answers that question with a look: Of course\n          in sure.\n\n          MICHAZZ,\n          The question to ask is this: Who\n          knew you were flying to Las Vegas?-\n\n                         TONY\n          The girl I was staying with and ---\n          Tony breaks off:\n\n                         HICI=L\n          Someone f_ cm Langley?\n\n                         TONY\n          Before I left, I called a man named\n          Stuart ?almateer.\n          (CONY r ED)\n\n                         \n\n                         \n\n                         \n\n                         \n\n          106.\n\n                         CONTINOEA\n\n                         MICHAEL\n          I told Tom Hagen -- God rest his\n          soul --- but I warned him. it was a.\n          mistake -- that whole Vidal business.\n          As the only contact between our people\n          and the Government you had no protec-\n          tion. You were naked.\n\n                         (THEN)\n          It they want to break the contact,\n          they elimirate you and they're clean.\n          As long as you're alive,. you're a\n          threat -- do you understand?\n\n                         ANOTHER ANGLE\n          As Tony sits. He's not sure whether he likes or\n          trusts or is ready to forgive his father, but he\n          knows the sound of good sense when he hears it.\n\n                         MICHAEL\n          How long do you think the-Administra-\n          tion would last if it were to come\n          out that the President used the\n          Corleone family to assassinate the\n          head of a foreign state.\n\n                         0 (THEN)\n          The question is rhetorical.\n\n                         ANOTHER ANGLE\n          Michael takes out a cigar and goes about the ceremony\n          of lighting it.\n\n                         - MICHAEL\n          I'm not supposed to smoke these\n          things, but it, isn't every day\n          a man's son comes home.\n\n                         TONY\n          You were saying?\n\n                         XICHAEL\n          Every year- -- on February third --\n          I've sent you a check. Those ohecks\n          were never cashed\n\n                         TCNY\n          Would you like to '.cacw why?\n\n                         \n\n                         \n\n                         \n\n                         \n\n          107.\n\n                         CONTINUED:\n\n                         MICHAEL\n          r've always know why -- and z res-\n          pect your reasons. But, neverthe-\n          less, I'm your lather and anything\n          t have is yours - if it's money you\n          want, my friendship, the benefit of\n          my experience -- if you'll accept it.\n          It's yours.\n\n                         TONY\n          I I need your advice.\n\n                         MICHAEL\n          (nods, then)\n          There are two ways I see to handle\n          this. First, I can make you dis-\n          appear in. Sicily until everything\n          blows over. Things change. Men get\n          old and angers cool. In four or\n          five years you could probably be\n          safe to come home again..\n\n                         TONY\n          What's the other way?\n\n                         1\n\n          MICE?\n          0 The other way is a gamble that you\n          could lose.\n\n                         (THEN)\n          And it would mean becoming a part\n          of the family -- for a while anyway.\n          Tony is just looking at Michael.\n\n                         MICHAEL\n          Think about it. Sleep on it.\n          We'll talk again in the morning.\n\n          I=. THE BOATHOUSE (CORLEONE COMPOUND) - DAE\n          Tony enters the glassed-in boathouse. Outside, cold\n          white winter and the deep, blue Lake. With hi= is a\n          Sezv'ant..\n\n                         SERVANT\n          Your father will be out in a moment.\n\n                         (INDICATES)\n          There's coffee on the sideboard.\n          Tony tads. The Ser-want exits. 'ror.?y crosses to the\n          sideboard, pours a cup of coffee. As he t:uzmns with\n          it, his eyes fall on a table in the corner on which\n          (CCdN V E )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          108.\n\n                         CONT??IUED :\n          are many Pictures, mostly in heavy silver frames.\n          Some are studio portraits. Some are grainy blow-ups\n          of snapshots. Tony crosses to the table of pictures.\n\n          VERY CLOSE - THE PICTURES\n          As Tony looks, we are moving from picture to picture.\n          The first is a wedding portrait taken on Connie's\n          wedding day. Carlo is kissing the Bride. The God-\n          father stands, proud and uncom.fortable in his tuxedo.\n          Michael in his uniform. The twins, - all heavy eyebrows\n          and baby fat.\n          '\"here is a picture of Sonny with his fists up and\n          laughing as if about to hit someone.\n          There is a photograph of the three brothers, Sonny,\n          Michael and 'redo, their arms around each other, at\n          an outing someplace. Sonny is in the middle of a\n          big grin. Fredo looks shy and scared. Michael is\n          staring straight ahead, a boyish smile on h 1s face\n          although his eyes are cold.\n          There is a picture of Michael and Ray at Tony's con-\n          firmation.\n\n                         0\n          There is a blown-up snapshot of the Godfather in his\n          tomato garden in the backyard..\n\n                         ON TONY\n          as he picks up the picture of the Godfather, remem-\n\n                         BERING ---\n\n                         DISSOLVE TO:\n\n          A FLLASHBACE SEQUENCE - {FROM GODFATHER I)\n          The old Don is tending his tomato vines. With him\n          is the little boy, Tony. aged three or four. They\n          have the special rapport that sometimes exists be-\n          tween the old and the very youzg. They play teasing\n          games with paper fangs and the bug spray can, then\n          suddenly, the old man's heart ;ailed him, and he falls\n          i n to the tomato vines. After a moment the lit :le boy\n          understands that the old man is no l anger playing a\n          game, and he beomes frightened:\n\n                         CUT TO:\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         L\n\n          LOS.\n          INT. TifE BOATHOIISZ•- DAY\n          We are on the door as Michael enters. He pauses for\n          a moment, then:\n\n          MICHAEL ' .\n          Do you remember your grandfather?\n\n                         ANOTHER ANC\n          as Tony turns from the table full of pictures with his\n          grandfather's framed portrait still in his hand.\n\n                         TONY\n          Yes..\n          As Tony puts down the picture, Michael comes up.\n\n                         1SICEAEL\n          Do you remember him with admira-\n          tion and respect?\n\n                         TONY\n          Z remember that I loved him.\n\n                         MILBAEL\n          0 So did I.\n\n                         TONY\n          I've been thinking about these\n          choices.\n          Michael holds up his hand to delay the decision.\n\n                         141C RAM\n          Talk with me for a moment.\n\n                         (THEN)\n          Let me learn something about my\n          son.. What's Trident Scholar?\n\n                         TONY\n          It's a special honors program for\n          First Classmen.\n\n                         MIC3AEL\n          You liked Annapolis?\n\n                         _ONY\n          Yes.\n\n                         AICZAEL\n          Enough to make a life in t e Navy?\n\n                         (CONT\"-RLGCZD)\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         ??I\n\n                         ILA\n\n                         CONTINUED\n\n                         TONY\n          No. Not that much.\n\n                         MICHAEL\n          2 was surprised you didn't go to\n          Dartmouth.\n\n                         TONY\n          I thought about it, but Kay wanted\n          me to go to Annapolis and the price\n          was right.\n\n                         MICHAEL\n          How's Hanover these days?\n\n                         TONY\n          About the same.\n\n                         MICHAEL\n          I always liked that town. That' S\n          where Z met your mother -- when I\n          was going to school up there.\n\n                         TONY\n          I know.\n\n                         0 MICHAEL\n          I'd planned to live there, you know.\n          Teach maybe -- or go into law. A.\n          little office on Wheelock Street.\n          Deeds and wills. I would have liked\n          that.\n\n                         TONY\n          Why didn't you do it?\n\n                         MICHAEL\n          Because one day a Sicilian pimp\n          and dope peddlar named Virgil\n          Solla2 O tried to assassinate my\n          father -- your grandfather -- and\n          I had to do something about it.\n\n                         (THEN)\n          It was a mistake. S took a road.\n          The wrong road for me. It ended\n          here.\n\n                         TONY\n          is this so bad? Sa\n          don't know. it depends or. what\n          you call terrib? e . You have to\n\n                         CCDN'TINV D)\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         CONTINUED:\n          MICHAEL (cant 'd)\n          live a certain way --- a cold way -\n          or you won't live long.\n\n                         (THEN)\n          Do you remember your uncle credo?\n\n                         TONY\n\n                         (SMILES)\n          ?redo, the fisherman. Yes. I'll\n          always remember, he had a secret\n          way to catch fish. 'He taught me.\n          You say a Hail. Mary' before you\n          put the line down. It never fails.\n\n                         MICHAEL\n          You know what happened to ?'redo?\n\n                         TONY\n          He died didn't he?\n\n                         MIS\n          I had him killed. My own brother.\n          It was something r had to do -- or\n          felt I had to.\n\n                         TONY\n          Why?\n\n                         MICHAEL\n          He went against the family. So I\n          waited until our mother died and\n          then I --\n          (breaks off ,\n\n                         THEN)\n          Not too many people are fitted for\n          this kind of a life. I've had to\n          do mazy hard things, but sending\n          you and your sister away -- that\n          was the hardest.\n\n                         ANOTHER ANGLE\n          There is a beat. To= and his father look at each\n          other and for a moment we feel that Tony is going\n          to cross to his father, then the moment passes:\n\n                         MYC\"\"L\n          And now, that I've said t~ at, I'd\n          like to hear your decision.\n\n                         ' :CN'Y\n          I don't th.iL k I'd ? ike Sicil v\n\n                         (CON'R 1MED )\n\n                         \n\n                         \n\n                         \n\n                         \n\n          112.\n\n                         CONTINOED:\n\n                         MICHAEL\n          Good.\n\n                         (THEN)\n          Come sit with me. Time is short\n          and I have a lot to tell you.\n\n                         CUT TO:\n          t N T. CIA BUILDING (LOBBY) - DAY\n          As Tony, in civilian clothes, passes through the\n          inner- checkpoint with his legitimate credentials.\n\n                         CUT TO:\n\n          ZNT. MOREHOUSE'S OFFICE - DAY\n          Morehouse and Palmateer are in the office. Their atti-\n          tudes are. less than cordial as Tony enters.\n\n                         MOREHOUSE\n          Come in Adams, sit down. I've\n          called Stu in on this --\n\n                         (GLANCES AT\n\n                         WATCH)\n          Though I'm afraid I can't give\n          you much time. I've got a brief-\n          ing with the Z-orty Committee at\n          noon.\n\n                         TONY\n          what I have to say won't take long.\n\n                         (THEN)\n          And I think we 111 all be happier\n          if it's not on tape.\n          Morehouse pauses a moment then openens a drawer and\n          switches ofd his tape machine.\n\n                         MOREHOUSE\n          Sow's your health? I understand\n          you got a crack on the head?\n\n                         TONY\n          I'm fine now.\n\n                         MOREBOUSE\n          you were up at your father's place\n          in Tahoe?\n          T::at d s rig\n\n                         (CONTINUED)\n\n                         \n\n                         \n\n                         \n\n                         \n\n          113.\n\n                         CONTINUED :\n\n                         MOREHOUSE\n          Has he changed much?\n\n                         TONY\n          Not as much as I' d. expected.\n\n                         MO 2 CUSZ\n          I remember him very well -- from\n          the Senate hearings on crime. Heat\n          little man. Very polite, spoke in\n          a soft, reasonable voice. He sat\n          there with two million dollars worth\n          of legal talent at his elbow and told\n          the United States Government to go\n          piss up a rope.\n\n                         (SMILES)\n          You had to admire the pretentious\n          little bastard.\n\n                         TONY\n          We're wasting each other's time\n          with this, Mister Morehouse.\n\n          ' M OREHOUSE\n          All right, Son. This is your party.\n          You've got five minutes.\n\n                         0 (THEN)\n          But before we start, I'm not going\n          to listen to a lot of recriminations\n          about that cveration. Your people\n          blew it, pure and sim le. It was\n          totally mishandled. A mistake from\n          beginning to end.\n\n                         TONY\n          The big mistake was that I'm still\n          alive.\n\n                         `SOREROUSE\n          Come again?\n\n                         TONY\n          11 you'd managed to get rid of me\n          down there - or in Vegas -- you' d\n          be all right. silt it's too Late\n          now. You've lost your chance.\n\n                         HOREZOUSE\n          I don't know what the hell you're\n          talking about.\n          (C 0 di': I?It )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          .7\n\n          114.\n\n                         CONTINU\n\n                         TONY\n          If anything should happen to me --\n          anything at all -- you can find out.\n          The full story will be all over the\n          six o'clock news.\n\n                         MOREHCQSE\n          I'd like to hear your conception of\n          the full story..\n\n                         TONY\n          You were running a no-lose operation.\n          if we succeeded, you were rid of\n          Vidal. -- if we failed, the Corleone\n          family was set up to take the blame.\n          Your hands were clean. I was the\n          only one who could dispute your\n          story and I wasn't supposed to come\n          back. I've found out that my res-\n          ignation from the Navy was processed\n          and accepted a full week before I\n          went down here ---\n\n                         MOPEEOUSE\n          Stu?\n\n                         PALMATE=\n          Well, yes, Sir. His resignation did\n\n                         E\n          go through, but it was a snafu -- a\n          Yeoman's mistake, that's all.\n\n                         MOREHOUSE\n          A clerical error.\n\n                         TONY\n          And I don't buy it.\n\n                         MOREBOUSE\n          don't give a shit if you buy it\n          or not. And I'll tell you some-\n          thing else, Mister Adams, or what-\n          ever your fucking name is, I don't\n          react favorably to blackmail.\n\n                         TONY\n          It's not blackmail. It's a simple\n          statement of fact. If anything\n          happens to me or if the Corleone\n          Family is damaged iz these Senate\n          Hearings, then the whistle blows\n          and the whole zdmin .st_ation aces.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         7\n\n          (CONTINUED) ?\n\n                         MOREHOUSE\n          And what makes you think T. give a\n          shit about what happens to the Admin-\n          istration?\n          Morehouse grins without humor. Tony's eyes flicker.\n          Morehouse has told something that Tony needed to know.\n          Morehouse is up and coming around the desk, as he\n          continues. _\n\n                         _14ORE8OVSE\n          Whatever I did was done under dix-\n          ect orders from the President of\n          the United States, and I. will. so\n          testify in open hearing. if that\n          upsets some of you= guinea gumbarrs\n          in Nevada, then so be it. The days\n          are over when your father and his\n          like could corrupt and intimidate\n          this Nation.\n\n                         TONY\n          No. You've taken over the job.\n\n                         MORE&OUSE\n          Your five minutes are up -- now get\n          the hell out of here.\n          As Tony turns and goes, Morehouse glares after him,\n          the glint of victory in his eyes.\n\n                         CST TO:\n\n          INT. SENATE BU=DING - FOYER - DAY\n\n          We axe outside the huge Senate Caucus room. The\n          double doors are open and inside we can see the\n          preparations for the Hearing. The Senators are\n          taking their seats -- the Press is being given the\n          Press hand-out (stamped: \"Embargoed until Witness\n          Testifies\"). Technicians are carrying cables for\n          the TV cameras and lights past Lawyers and Committee\n          Staff Members.\n          Planted In front of all this, speaking to a TV camera,\n          is Elizabeth Ann Dunne.\n\n                         ELIZABETH\n          This is Elizabeth Ann Dunne coming\n          to you from outside, the. Senate. Caucus\n          Room where the Senate Select Co=i.--tee\n          on Intelligence is meeting this morn--\n          ing ..,...\n\n                         (THEN)\n          Oh, Senator)\n          (CaNTI_ TU?:D )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n                         I\n\n          L16.\n          This last to the polished Senator Barthalemew of\n          Pennsylvania who pauses in front of the camera.\n\n                         ELIZABET3\n          This is Senator Paul Barthalemew\n          Of Pennsylvania who will chair the\n          Committee.\n\n                         (THEN)\n          Can you tell us the specific purpose\n          of this Committee, Senator?\n          This Committee has been mandated\n          to address two questiocis: One, did\n          the United States Government -- or\n          any of its officials --- authorize,\n          instigate or in any way-abet a plot\n          to assassinate the Head of a friendly\n          foreign State, Armando Vidal.\n\n                         (THEN)\n          And Two, if not -- who did?\n\n                         ELZZA3ETFF\n          Thank you, Senator.\n\n                         0\n          As Barthalazaew moves into the Caucus Room, shaking his\n          head, we ----\n\n                         CIIT TO:\n          ZNT. SE35MTw CAUCUS ROOM - DAY\n          Barthalemew is. pounding his gavel for order.\n\n                         HARTHALE EW\n          Take your seats, please. Sergeant\n          at Arms' Will. you see that every-\n          an takes his seat?\n          We pan over to the door as Tony enters and finds a\n          seat.\n\n                         CUT TO:\n\n          EXT. A WOODED AREA (PCMPTON LA=S, NNW SBRSEY) - DAY\n\n          Ralph •Augusto, bare to the waist, is half way up a\n          hill, in a stand of trees, working with an ax. Se\n          has stripped off his shirt and his coat. is shirt\n          and his gun are placed aver a fallen t=ee, ten yards\n          away.\n\n                         \n\n                         \n\n                         \n\n                         \n\n          117.\n\n                         ANCTHER ANGLE\n          as Prankie Rizzi and Rocco Lampone are coming up the\n          hill toward Augusto. They pause:\n\n                         LAMPONE\n          Hey Ralph?\n          Augusta torus. His eyes flicker to the gun, then\n          back to Rocco and Frankie. He smiles warmly:\n\n                         AGSTO\n          Hey Rocky. Whaddya say? Waddya\n          doin' up here?\n\n          ? R A N =\n          We just came up to say goodbye,\n          Ralph.\n\n          L?JWONE\n          for Neri and DeVito.\n          Frankie and Lampone have spoken almost simultaneously,\n          and as they speak, their guns are out and blasting.\n          ANCTHwR ANGLE - ON AaGt7STO\n\n                         10\n          As the bullets thwack into his body. he jerks, but\n          doesn't go down. His cold face twists in fury as\n          he is moving down toward Tony, raising the ax.\n\n          ON FRANI(TZ\n          as he stands his ground, blasting. Twice. 171A%ree\n          times. Four times.\n\n                         ANOTBER ANGLE\n          as Augusta wavers, his face goes slack and he plunges\n          into the ground at F'rankie' s feet. The ax has fallen\n          from his hands.\n\n                         CUT TO:\n          IN'r . S MIATE CAUCUS ROOM - OAT\n          ?Among the Senators an the Committee, we recognize\n          Mc:tissick and mossib? y we w i-11. remember old Geary,\n          from Nevada. Tonv is an interested spectator as\n          3artha? amew,r quest ions the witness, Martin Davideau.\n\n                         0\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n          CUNT t3ED\n\n                         3ARTKALZ W\n          And.'what light can you throw on\n          this matter, Mister Davideau?\n\n                         DAVIDEAII\n          S have. a memo here from the Director\n          of the FBI to the Director of the CIA\n          with copies to the Army, Air Force,\n          Navy and State Department Intelligence\n          Offices. It states that one of our\n          informants -- well, I' l1 read it (ro\n\n                         ADS)\n          'during. a recent conversation with\n          several friends, underworld figure\n          Sam Maatrocina, stated that there\n          was going to be an attempt on Armando\n          Vidal' s life and this attempt ---\n          this hit, in the argot -- was to\n          be carried out by another underworld\n\n                         FAMILY\n\n                         MC RISSICX\n          Did this informant identify the other\n          underworld family?\n\n                         DAVIDEAU\n          0 so Sir, he did not.\n\n                         MC KISSICX\n          Would you care to speculate,\n\n                         GEARY\n          Just a minute, Senator -- I whole-\n          heartedly object to this Committee\n          being used as a for,= for spec-\n          ulation based on an anonymous report\n          of an alleged conversation. Reput-\n          tations. could be recklessly and i --e-\n          grievably damaged.\n\n                         MC RISSICR\n          Very well, very well. We wouldn't\n          want to damage any of the constituency\n          of my esteemed Collegue from Nevada. --\n\n                         GEARY\n          Many thanks to the distinguished Sv..nior\n          Senator from our Wes tern Sister State\n          of Utah.\n\n          C7T TC :\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          119.\n          n ?T. MAATROCINA' S OFFICE (NEW YORK CITY) - DAY _\n          Sam Maatrccina is seated in his office on the thirty-\n          third,floor of an old downtown Manhattan office build-\n          ing. The buzzer sounds.\n\n                         MAATROCINA\n          Yes?\n\n                         SECRETARY'S VOICE\n          A couple of gentlemen here from the\n          Internal Revenue, Mister Maatrocina.\n\n                         MAATRCCIVA\n\n                         (SCOWLS)\n          The Internal Revenue!?\n\n                         SECRETARY\n          Yes sir.\n\n                         MAATROCINA\n          All right. Send 'em in.\n\n                         ANOTEER AN=\n          The door buzzes open and. two young,- rather conserva-\n          tive looking Men, enter with briefcases.\n\n          I FIRST MAN\n          Mister Maatrocina?\n          Maatrocina is coming around the desk:\n\n                         MAATROCI:YA\n          What- is this? Some kind of a roust?\n\n                         FIRST MAN\n          It's about your income tax, Sir.\n\n                         MAATROCITA\n          My taxes are handled by the biggest\n          firm of accountants in New York City.\n          It costs me a hundred thousand doll-\n          ars a year and you two assholes in\n          cheap- suits are going to come in here\n\n                         AND\n          Sam Maatroci.na breaks off. The Second Young man has\n          hit him an open-handed karate chop on the side of\n          the neck. Now, before Maatroctna can speak or cry\n          cut, the Aan has hit :taatrocina a seccad chop, shat-\n          ter .ag his adams apple.\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          120.\n\n                         ANOTHER ANGLE\n          As Maatrocina goes to his knees, his eyes goggling,\n          the First Young Man has crosses to the window and\n          opened it.\n          Now he and the Second Young Man carry the wide-eyed\n          New York Don to the window and throw him out.\n\n                         CUT T0:\n\n          EXT. D0WNTG\"N MAAJBATZAN STREET - DAY\n\n          as crowds are gathering around the body of Maatrocina\n          on the sidewalk, the Two Men come out of the building,\n          cross the street and go around the corner.. In the\n          distance, sound of approaching sirens.\n\n          EXT. TEE CORNER - DAY\n\n          as the Two Men come around the corner and get into a\n          waiting car. Driving the car is Frankie Rizzi. As\n          they drive off, we ---\n\n                         CDT TO:\n\n          LYT. SENATE CAUCUS. ROOM - DAY\n          it is late afternoon. A Uitness, General Vanderhorst,\n          the Director of the CIA, is at the table. Geary is\n          quizzing him.\n\n                         GEARY\n          No. No. What I'm trying to get\n          from you, General, is, as Director\n          of the Central Zntelligence Agency,\n          did you ever have any kind of order\n          in writing authorizing you to assass-\n          inate the head of a foreign state?\n\n                         VANDERHORST\n          No, Sir. Z did not.\n\n                         MC XISSICX\n          Well, come on now.. That's hardly\n          the thing that would be put in\n          writing, now is it?\n\n                         EXRY\n          Well what would it be put n 44\n          aol writ=q?\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          121.\n\n                         CONTIII=\n\n                         MC KISSICX\n          X suggest that it would be put into\n          the same kind of phraseology that\n          Henry used to his Barons when he\n          said, 'Will, no one rid me of this\n          turbulent priest?' He didn't tell\n          them to go kill Thomas --z6 Beckett.,\n          but the final result of it was\n          murder in the Cathedral ---\n\n                         GARY\n          What has Thomas 3.~ Beckett got to\n          do. witii the subject at hand? ---\n\n                         CSC RISSICX\n          If the Distinguished Senator from\n          Nevada will refrain from ----\n          Ba -halemew is beating with the gavel on his desk.\n\n                         BARTSALEMEW\n          Gentlemen: Gentlemen!\n\n                         CUT TO:\n\n          EXT.. THE MALL - DAY\n\n          Tony is buying a hot dog at one of the dog-wagons on\n          the Mall. as crosses and sits on a bench where Palmateer\n          is sunning himself, looking at a newspaper.\n\n                         CLOSER\n          as Tony takes a bite of his hot dog.\n\n                         TONY\n          These are good. You ought to\n          stave one.\n\n                         PALMATEER\n          I don't have a lot of time, Tony.\n\n                         TONY\n          First, I wanted to say that although\n          I'm sure that I was set--um to be\n          killed down there, I never thought\n          you were mixed up in i t.\n\n          PA M TEER\n          Pine. ' clad to hear that.Now,\n          what was it that you wanted?\n          (C©NT TC =rD )\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         I\n\n          122.\n\n                         CONTI': UED :\n\n                         TONY\n          Z hear that Morehouse is being called\n          in by the CommI tree to testify.\n\n                         PALMATEER\n          So they tell me.\n\n                         TONY\n          Well if he does -- and. I'm telling\n          you this as a friend --- he's going\n          to bring the roof down on him self ---\n          and you too.\n          As Pa.lmateer puts the newspaper down, looks at Tony:\n\n                         TONY\n          That's definite, Stu.\n\n                         (THEN)\n          VIM sorry.\n\n                         CUT TO:\n\n          ?.XT. A MARILAND ROAD (NEAR PATUXENT) - ?IIG3T\n          We pick up an automobile moving south along a road\n          that edges Chesapeake Bay.\n          is Q.\n\n          INT. THE CAR - NIGHT\n\n          We see that Tom Morehouse is driving. We are angling\n          through the front. windshield as he turns off into a\n          marina parking lot.\n\n          EXT. THE FLOATS - VIC T\n\n          as Morehouse comes out onto the float, then climbs\n          aboard. a nice little yawl, apparently his own.\n\n                         CLOSER\n          as Morehouse goes to the cabin. The snap lock has\n          been unlocked and the hatchway is open. There is a\n          dim light in the cabin.\n\n                         (OREEOUSE\n          Stu?\n          P Z;? yR\n          Down here.\n\n                         \n\n                         \n\n                         \n\n                         \n\n          123..\n\n          1 T. CAS= - NIGHT\n          as Morehouse comes down the four steps that lead into\n          the cabin, then stops.\n\n          ANOT R ANGLE\n          Palmateer and Rocco Lampone are waiting for him in the\n          cabin.\n\n                         MOREEOUSE\n          What the hell is this?\n\n          ANOT'E' ER ANGLE\n          as Rocco rises, drawing a twenty-two calibre pistol\n          with silencer, and shoots Morehouse three times in\n          the chest.\n          As Morehouse goes down:\n\n                         LAMPONE\n          Is he dead?\n          Palmateer kneels to check Morehouse.\n          PAL.yATEER\n          Yes.\n\n                         LAMPONE'\n          So are you.\n          i Lampone had leaned down, put the gun to Palmateer's\n          head and pulled the trigger.\n\n          EXT. CHESAPEAXE SAY -- DAWN\n\n          A small Coast Guard patrol boat moves up Chesapeake\n          Bay, flat calm reflecting a pearly pink sky. As Look-\n          out on the flying bridge scans the bay ahead, then.\n          i n to the sneaking tube.\n\n                         LOOKOUT\n          Bridge.\n\n          I . W'SMIZZOUSE - DAWN\n           As the Officer o-6' the deck, a yo:;.ng C: a 3cs,.:' .,\n\n                         ANSWERS:\n\n                         \n\n                         \n\n                         \n\n                         \n\n                         _T\n\n          124.\n\n                         CONTINGED:\n\n                         BOSUN\n          Bridge aye.\n\n                         LOOKOUT'S VOICE\n          That yawl looks like it's adrift.\n\n                         ANOTHER ANGLE\n          As Morehouse's sailboat drifts in the calm, it's\n          sails up and motionless, it's tiller untended. The\n          patrol boat noses up:\n\n                         BOSUN\n          Ahoy, the yawl. You all right? -\n          No answer. The Bosun jumps aboard.\n\n                         BOSUN\n          Hello? Avon Lady. if anybody\n          down there's doing anything they\n          shouldn't, now's the time to -\n          Tae Bosun has looked down into the cabin. 3e breaks\n          off and turns back to the Patrol boat.\n\n                         0 BOSUN\n          Get on the horn to base. We got\n          two bodies here.\n\n                         CUT TO:\n          k =. SFYLATE CAUCUS ROOM - DAY\n          Arne Grundellius is at the witness table.\n\n                         GRUNDELLIUS\n          And in conclusion, I'd like to say\n          that I believe these hearings have\n          served a great, good purpose. in\n          spite of the fears that the hear-\n          ings would do hartto the fabric of\n          democracy, our Nation's stronger to-\n          day in the knowledge that we do not\n          export revolution or use murder as\n          an extension of diplomacy.\n\n                         ANOTHER ANG:\n          as Grmdelli.us fi.'2is:?es, Geary And one or wwo others\n          rise, applatsd.i.r.g.\n\n                         \n\n                         \n\n                         \n\n                         \n\n          125.\n\n                         CONTIKTTED :\n\n                         GEARY\n          Hear. Hear.\n          ANOTHER ANGLE - THE At7D IENCE\n          Tony and Elizabeth are together. Elizabeth is applaud-\n          ing Grundellius. Tony looks over at her.\n\n                         CUT TO:\n\n          INT. THE LOBBY (SENATE BUILDING) = DAY\n\n          The meeting has broken up. Spectators, Legislators,\n          News Sian and Womea,. Staff and Committee Members are\n          leaving or standing in knots, gossiping.\n          We pick up Elizabeth and Tony as they cross out, pausing\n          to speak with Senator Geary and Grundellius who have\n          stopped to chat.\n\n                         ELIZABETH\n          Mister Secretary, Senator Geary, I'd\n          like to present Tony Adams.\n          There are general greetings, and then, as they move\n          S toward the doors.\n\n                         GEARY\n          I believe we have mutual friends\n          in Nevada, Mister Adams.\n\n                         TONY\n          Yes Sir. I believe so.\n\n                         GEARY\n          If there's ever anything I can do\n          you come see me.\n\n                         TCNY\n          I might just take you up on that,\n          Senator.\n          As they exit ----\n\n                         CUT TO:\n          EXT. WASHINGTON D.C. STREET\n          It is sunset. Tony and Elizabeth are walking toward\n          the capital building, outlined against a pink sky.\n          As they walk toward J.-_ we begin to hear t .he Gcdwa ther\n\n                         THEME\n          SLOW DISSOLVE\n\n                         \n\n                         \n\n                         \n\n                         \n\n          126.\n\n          EXT. CORLEONE COMPOUND - NIGHT\n\n          We pick up the headlights of a car coming up the\n          long approach driveway toward us. It comes past\n          the gate, past the kennels for the guard dogs, past\n          the guest houses and finally up to the main house\n          where it stops.\n          The front door to the house opens and Michael stands\n          silhouetted against the block of yellow light. Tony\n          gets out of the car and comes up to him. They embrace\n          briefly and move into the house together.\n          As the door closes behind them, we start to move up\n          and back. The sound of the single trumpet can still\n          be heard, playing slowly and sadly, the notes faintly\n          resonant as if echoing through the narrow streets of\n          some old hill village in Sicily.\n          We have pulled up and up and up c ntil everting is\n          darkness, as we ---\n\n          FADE OUT\n\n          THE END\n\n\n
\n
\n\n\n" }, { "path": "samples/go/decent/dbg/debug.go", "content": "// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage dbg\n\nimport (\n\t\"fmt\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"log\"\n\t\"os\"\n\t\"strconv\"\n)\n\nvar (\n\tFilepath = \"/tmp/noms-dbg.log\"\n\tlg = NewLogger(Filepath)\n)\n\nfunc NewLogger(fp string) *log.Logger {\n\tf, err := os.OpenFile(fp, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644)\n\td.PanicIfError(err)\n\tpid := strconv.FormatInt(int64(os.Getpid()), 10)\n\treturn log.New(f, pid+\": \", 0644)\n}\n\nfunc GetLogger() *log.Logger {\n\treturn lg\n}\n\nfunc SetLogger(newLg *log.Logger) {\n\tlg = newLg\n}\n\nfunc Debug(s string, args ...interface{}) {\n\ts1 := fmt.Sprintf(s, args...)\n\tlg.Println(s1)\n}\n\nfunc BoxF(s string, args ...interface{}) func() {\n\ts1 := fmt.Sprintf(s, args...)\n\tDebug(\"starting %s\", s1)\n\tf := func() {\n\t\tDebug(\"finished %s\", s1)\n\t}\n\treturn f\n}\n" }, { "path": "samples/go/decent/ipfs-chat/main.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"os/signal\"\n\t\"runtime\"\n\t\"syscall\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/ipfs\"\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/attic-labs/noms/samples/go/decent/dbg\"\n\t\"github.com/attic-labs/noms/samples/go/decent/lib\"\n\t\"github.com/ipfs/go-ipfs/core\"\n\t\"github.com/jroimartin/gocui\"\n\tkingpin \"gopkg.in/alecthomas/kingpin.v2\"\n)\n\nfunc main() {\n\t// allow short (-h) help\n\tkingpin.CommandLine.HelpFlag.Short('h')\n\n\tclientCmd := kingpin.Command(\"client\", \"runs the ipfs-chat client UI\")\n\tclientTopic := clientCmd.Flag(\"topic\", \"IPFS pubsub topic to publish and subscribe to\").Default(\"ipfs-chat\").String()\n\tusername := clientCmd.Flag(\"username\", \"username to sign in as\").String()\n\tnodeIdx := clientCmd.Flag(\"node-idx\", \"a single digit to be used as last digit in all port values: api, gateway and swarm (must be 0-9 inclusive)\").Default(\"-1\").Int()\n\tclientDS := clientCmd.Arg(\"dataset\", \"the dataset spec to store chat data in\").Required().String()\n\n\timportCmd := kingpin.Command(\"import\", \"imports data into a chat\")\n\timportDir := importCmd.Flag(\"dir\", \"directory that contains data to import\").Default(\"./data\").ExistingDir()\n\timportDS := importCmd.Arg(\"dataset\", \"the dataset spec to import chat data to\").Required().String()\n\n\tdaemonCmd := kingpin.Command(\"daemon\", \"runs a daemon that simulates filecoin, eagerly storing all chunks for a chat\")\n\tdaemonTopic := daemonCmd.Flag(\"topic\", \"IPFS pubsub topic to publish and subscribe to\").Default(\"ipfs-chat\").String()\n\tdaemonInterval := daemonCmd.Flag(\"interval\", \"amount of time to wait before publishing state to network\").Default(\"5s\").Duration()\n\tdaemonNodeIdx := daemonCmd.Flag(\"node-idx\", \"a single digit to be used as last digit in all port values: api, gateway and swarm (must be 0-9 inclusive)\").Default(\"-1\").Int()\n\tdaemonDS := daemonCmd.Arg(\"dataset\", \"the dataset spec indicating ipfs repo to use\").Required().String()\n\n\tkingpin.CommandLine.Help = \"A demonstration of using Noms to build a scalable multiuser collaborative application.\"\n\n\texpandRLimit()\n\tswitch kingpin.Parse() {\n\tcase \"client\":\n\t\tcInfo := lib.ClientInfo{\n\t\t\tTopic: *clientTopic,\n\t\t\tUsername: *username,\n\t\t\tIdx: *nodeIdx,\n\t\t\tIsDaemon: false,\n\t\t\tDelegate: lib.IPFSEventDelegate{},\n\t\t}\n\t\trunClient(*clientDS, cInfo)\n\tcase \"import\":\n\t\tlib.RunImport(*importDir, *importDS)\n\tcase \"daemon\":\n\t\tcInfo := lib.ClientInfo{\n\t\t\tTopic: *daemonTopic,\n\t\t\tUsername: \"daemon\",\n\t\t\tInterval: *daemonInterval,\n\t\t\tIdx: *daemonNodeIdx,\n\t\t\tIsDaemon: true,\n\t\t\tDelegate: lib.IPFSEventDelegate{},\n\t\t}\n\t\trunDaemon(*daemonDS, cInfo)\n\t}\n}\n\nfunc runClient(ipfsSpec string, cInfo lib.ClientInfo) {\n\tdbg.SetLogger(lib.NewLogger(cInfo.Username))\n\n\tsp, err := spec.ForDataset(ipfsSpec)\n\td.CheckErrorNoUsage(err)\n\n\tif !isIPFS(sp.Protocol) {\n\t\tfmt.Println(\"ipfs-chat requires an 'ipfs' dataset\")\n\t\tos.Exit(1)\n\t}\n\n\tnode, cs := initIPFSChunkStore(sp, cInfo.Idx)\n\tdb := datas.NewDatabase(cs)\n\n\t// Get the head of specified dataset.\n\tds := db.GetDataset(sp.Path.Dataset)\n\tds, err = lib.InitDatabase(ds)\n\td.PanicIfError(err)\n\n\tevents := make(chan lib.ChatEvent, 1024)\n\tt := lib.CreateTermUI(events)\n\tdefer t.Close()\n\n\td.PanicIfError(t.Layout())\n\tt.ResetAuthors(ds)\n\tt.UpdateMessages(ds, nil, nil)\n\n\tgo lib.ProcessChatEvents(node, ds, events, t, cInfo)\n\tgo lib.ReceiveMessages(node, events, cInfo)\n\n\tif err := t.Gui.MainLoop(); err != nil && err != gocui.ErrQuit {\n\t\tdbg.Debug(\"mainloop has exited, err:\", err)\n\t\tlog.Panicln(err)\n\t}\n}\n\nfunc runDaemon(ipfsSpec string, cInfo lib.ClientInfo) {\n\tdbg.SetLogger(log.New(os.Stdout, \"\", 0))\n\n\tsp, err := spec.ForDataset(ipfsSpec)\n\td.CheckErrorNoUsage(err)\n\n\tif !isIPFS(sp.Protocol) {\n\t\tfmt.Println(\"ipfs-chat requires an 'ipfs' dataset\")\n\t\tos.Exit(1)\n\t}\n\n\t// Create/Open a new network chunkstore\n\tnode, cs := initIPFSChunkStore(sp, cInfo.Idx)\n\tdb := datas.NewDatabase(cs)\n\n\t// Get the head of specified dataset.\n\tds := db.GetDataset(sp.Path.Dataset)\n\tds, err = lib.InitDatabase(ds)\n\td.PanicIfError(err)\n\n\tevents := make(chan lib.ChatEvent, 1024)\n\thandleSIGQUIT(events)\n\n\tgo lib.ReceiveMessages(node, events, cInfo)\n\tlib.ProcessChatEvents(node, ds, events, nil, cInfo)\n}\n\nfunc handleSIGQUIT(events chan<- lib.ChatEvent) {\n\tsigChan := make(chan os.Signal)\n\tgo func() {\n\t\tfor range sigChan {\n\t\t\tstacktrace := make([]byte, 1024*1024)\n\t\t\tlength := runtime.Stack(stacktrace, true)\n\t\t\tdbg.Debug(string(stacktrace[:length]))\n\t\t\tevents <- lib.ChatEvent{EventType: lib.QuitEvent}\n\t\t}\n\t}()\n\tsignal.Notify(sigChan, os.Interrupt)\n\tsignal.Notify(sigChan, syscall.SIGQUIT)\n}\n\n// IPFS can use a lot of file decriptors. There are several bugs in the IPFS\n// repo about this and plans to improve. For the time being, we bump the limits\n// for this process.\nfunc expandRLimit() {\n\tvar rLimit syscall.Rlimit\n\terr := syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit)\n\td.Chk.NoError(err, \"Unable to query file rlimit: %s\", err)\n\tif rLimit.Cur < rLimit.Max {\n\t\trLimit.Max = 64000\n\t\trLimit.Cur = 64000\n\t\terr = syscall.Setrlimit(syscall.RLIMIT_NOFILE, &rLimit)\n\t\td.Chk.NoError(err, \"Unable to increase number of open files limit: %s\", err)\n\t}\n\terr = syscall.Getrlimit(syscall.RLIMIT_NOFILE, &rLimit)\n\td.Chk.NoError(err)\n\n\terr = syscall.Getrlimit(8, &rLimit)\n\td.Chk.NoError(err, \"Unable to query thread rlimit: %s\", err)\n\tif rLimit.Cur < rLimit.Max {\n\t\trLimit.Max = 64000\n\t\trLimit.Cur = 64000\n\t\terr = syscall.Setrlimit(8, &rLimit)\n\t\td.Chk.NoError(err, \"Unable to increase number of threads limit: %s\", err)\n\t}\n\terr = syscall.Getrlimit(8, &rLimit)\n\td.Chk.NoError(err)\n}\n\nfunc initIPFSChunkStore(sp spec.Spec, nodeIdx int) (*core.IpfsNode, chunks.ChunkStore) {\n\t// recreate database so that we can have control of chunkstore's ipfs node\n\tnode := ipfs.OpenIPFSRepo(sp.DatabaseName, nodeIdx)\n\tcs := ipfs.ChunkStoreFromIPFSNode(sp.DatabaseName, sp.Protocol == \"ipfs-local\", node, 1)\n\treturn node, cs\n}\n\nfunc isIPFS(protocol string) bool {\n\treturn protocol == \"ipfs\" || protocol == \"ipfs-local\"\n}\n" }, { "path": "samples/go/decent/lib/datapager.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\ntype dataPager struct {\n\tdataset datas.Dataset\n\tmsgKeyChan chan types.String\n\tdoneChan chan struct{}\n\tmsgMap types.Map\n\tterms []string\n}\n\nfunc NewDataPager(ds datas.Dataset, mkChan chan types.String, doneChan chan struct{}, msgs types.Map, terms []string) *dataPager {\n\treturn &dataPager{\n\t\tdataset: ds,\n\t\tmsgKeyChan: mkChan,\n\t\tdoneChan: doneChan,\n\t\tmsgMap: msgs,\n\t\tterms: terms,\n\t}\n}\n\nfunc (dp *dataPager) Close() {\n\tdp.doneChan <- struct{}{}\n}\n\nfunc (dp *dataPager) Next() (string, bool) {\n\tmsgKey := <-dp.msgKeyChan\n\tif msgKey == \"\" {\n\t\treturn \"\", false\n\t}\n\tnm := dp.msgMap.Get(msgKey)\n\n\tvar m Message\n\terr := marshal.Unmarshal(nm, &m)\n\tif err != nil {\n\t\treturn fmt.Sprintf(\"ERROR: %s\", err.Error()), true\n\t}\n\n\ts1 := fmt.Sprintf(\"%s: %s\", m.Author, m.Body)\n\ts2 := highlightTerms(s1, dp.terms)\n\treturn s2, true\n}\n\nfunc (dp *dataPager) Prepend(lines []string, target int) ([]string, bool) {\n\tnew := []string{}\n\tm, ok := dp.Next()\n\tif !ok {\n\t\treturn lines, false\n\t}\n\tfor ; ok && len(new) < target; m, ok = dp.Next() {\n\t\tnew1 := strings.Split(m, \"\\n\")\n\t\tnew = append(new1, new...)\n\t}\n\treturn append(new, lines...), true\n}\n" }, { "path": "samples/go/decent/lib/event.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/ipfs\"\n\t\"github.com/attic-labs/noms/go/merge\"\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/math\"\n\t\"github.com/attic-labs/noms/samples/go/decent/dbg\"\n\t\"github.com/ipfs/go-ipfs/core\"\n)\n\nconst (\n\tInputEvent ChatEventType = \"input\"\n\tSearchEvent ChatEventType = \"search\"\n\tSyncEvent ChatEventType = \"sync\"\n\tQuitEvent ChatEventType = \"quit\"\n)\n\ntype ClientInfo struct {\n\tTopic string\n\tUsername string\n\tInterval time.Duration\n\tIdx int\n\tIsDaemon bool\n\tDir string\n\tSpec spec.Spec\n\tDelegate EventDelegate\n}\n\ntype ChatEventType string\n\ntype ChatEvent struct {\n\tEventType ChatEventType\n\tEvent string\n}\n\ntype EventDelegate interface {\n\tPinBlocks(node *core.IpfsNode, sourceDB, sinkDB datas.Database, sourceCommit types.Value)\n\tSourceCommitFromMsgData(db datas.Database, msgData string) (datas.Database, types.Value)\n\tHashFromMsgData(msgData string) (hash.Hash, error)\n\tGenMessageData(cInfo ClientInfo, h hash.Hash) string\n}\n\n// ProcessChatEvent reads events from the event channel and processes them\n// sequentially. Is ClientInfo.IsDaemon is true, it also publishes the current\n// head of the dataset continously.\nfunc ProcessChatEvents(node *core.IpfsNode, ds datas.Dataset, events chan ChatEvent, t *TermUI, cInfo ClientInfo) {\n\tstopChan := make(chan struct{})\n\tif cInfo.IsDaemon {\n\t\tgo func() {\n\t\t\ttickChan := time.NewTicker(cInfo.Interval).C\n\t\t\tfor {\n\t\t\t\tselect {\n\t\t\t\tcase <-stopChan:\n\t\t\t\t\tbreak\n\t\t\t\tcase <-tickChan:\n\t\t\t\t\tPublish(node, cInfo, ds.HeadRef().TargetHash())\n\t\t\t\t}\n\t\t\t}\n\t\t}()\n\t}\n\n\tfor event := range events {\n\t\tswitch event.EventType {\n\t\tcase SyncEvent:\n\t\t\tds = processHash(t, node, ds, event.Event, cInfo)\n\t\t\tPublish(node, cInfo, ds.HeadRef().TargetHash())\n\t\tcase InputEvent:\n\t\t\tds = processInput(t, node, ds, event.Event, cInfo)\n\t\t\tPublish(node, cInfo, ds.HeadRef().TargetHash())\n\t\tcase SearchEvent:\n\t\t\tprocessSearch(t, node, ds, event.Event, cInfo)\n\t\tcase QuitEvent:\n\t\t\tdbg.Debug(\"QuitEvent received, stopping program\")\n\t\t\tstopChan <- struct{}{}\n\t\t\treturn\n\t\t}\n\t}\n}\n\n// processHash processes msgs published by other chat nodes and does the work to\n// integrate new data into this nodes local database and display it as needed.\nfunc processHash(t *TermUI, node *core.IpfsNode, ds datas.Dataset, msgData string, cInfo ClientInfo) datas.Dataset {\n\th, err := cInfo.Delegate.HashFromMsgData(msgData)\n\td.PanicIfError(err)\n\tdefer dbg.BoxF(\"processHash, msgData: %s, hash: %s, cid: %s\", msgData, h, ipfs.NomsHashToCID(h))()\n\n\tsinkDB := ds.Database()\n\td.PanicIfFalse(ds.HasHead())\n\n\theadRef := ds.HeadRef()\n\tif h == headRef.TargetHash() {\n\t\tdbg.Debug(\"received hash same as current head, nothing to do\")\n\t\treturn ds\n\t}\n\n\tdbg.Debug(\"reading value for hash: %s\", h)\n\tsourceDB, sourceCommit := cInfo.Delegate.SourceCommitFromMsgData(sinkDB, msgData)\n\tif sourceCommit == nil {\n\t\tdbg.Debug(\"FAILED to read value for hash: %s\", h)\n\t\treturn ds\n\t}\n\n\tsourceRef := types.NewRef(sourceCommit)\n\n\t_, isP2P := cInfo.Delegate.(P2PEventDelegate)\n\tif cInfo.IsDaemon || isP2P {\n\t\tcInfo.Delegate.PinBlocks(node, sourceDB, sinkDB, sourceCommit)\n\t}\n\n\tdbg.Debug(\"Finding common ancestor for merge, sourceRef: %s, headRef: %s\", sourceRef.TargetHash(), headRef.TargetHash())\n\ta, ok := datas.FindCommonAncestor(sourceRef, headRef, sinkDB)\n\tif !ok {\n\t\tdbg.Debug(\"no common ancestor, cannot merge update!\")\n\t\treturn ds\n\t}\n\tdbg.Debug(\"Checking if source commit is ancestor\")\n\tif a.Equals(sourceRef) {\n\t\tdbg.Debug(\"source commit was ancestor, nothing to do\")\n\t\treturn ds\n\t}\n\tif a.Equals(headRef) {\n\t\tdbg.Debug(\"fast-forward to source commit\")\n\t\tds, err := sinkDB.SetHead(ds, sourceRef)\n\t\td.Chk.NoError(err)\n\t\tif !cInfo.IsDaemon {\n\t\t\tt.UpdateMessagesFromSync(ds)\n\t\t}\n\t\treturn ds\n\t}\n\n\tdbg.Debug(\"We have a mergeable commit\")\n\tleft := ds.HeadValue()\n\tright := sourceCommit.(types.Struct).Get(\"value\")\n\tparent := a.TargetValue(sinkDB).(types.Struct).Get(\"value\")\n\n\tdbg.Debug(\"Starting three-way commit\")\n\tmerged, err := merge.ThreeWay(left, right, parent, sinkDB, nil, nil)\n\tif err != nil {\n\t\tdbg.Debug(\"could not merge received data: \" + err.Error())\n\t\treturn ds\n\t}\n\n\tdbg.Debug(\"setting new datasetHead on localDB\")\n\tnewCommit := datas.NewCommit(merged, types.NewSet(sinkDB, ds.HeadRef(), sourceRef), types.EmptyStruct)\n\tcommitRef := sinkDB.WriteValue(newCommit)\n\tdbg.Debug(\"wrote new commit: %s\", commitRef.TargetHash())\n\tds, err = sinkDB.SetHead(ds, commitRef)\n\tif err != nil {\n\t\tdbg.Debug(\"call to db.SetHead on failed, err: %s\", err)\n\t}\n\tdbg.Debug(\"set new head ref: %s on ds.ID: %s\", commitRef.TargetHash(), ds.ID())\n\tnewH := ds.HeadRef().TargetHash()\n\tdbg.Debug(\"merged commit, dataset: %s, head: %s, cid: %s\", ds.ID(), newH, ipfs.NomsHashToCID(newH))\n\tif cInfo.IsDaemon {\n\t\tcInfo.Delegate.PinBlocks(node, sourceDB, sinkDB, newCommit)\n\t} else {\n\t\tt.UpdateMessagesFromSync(ds)\n\t}\n\treturn ds\n}\n\n// processInput adds a new msg (entered through the UI) updates it's dataset.\nfunc processInput(t *TermUI, node *core.IpfsNode, ds datas.Dataset, msg string, cInfo ClientInfo) datas.Dataset {\n\tdefer dbg.BoxF(\"processInput, msg: %s\", msg)()\n\tt.InSearch = false\n\tif msg != \"\" {\n\t\tvar err error\n\t\tds, err = AddMessage(msg, cInfo.Username, time.Now(), ds)\n\t\td.PanicIfError(err)\n\t}\n\tt.UpdateMessagesAsync(ds, nil, nil)\n\treturn ds\n}\n\n// updates the UI to display search results.\nfunc processSearch(t *TermUI, node *core.IpfsNode, ds datas.Dataset, terms string, cInfo ClientInfo) {\n\tdefer dbg.BoxF(\"processSearch\")()\n\tif terms == \"\" {\n\t\treturn\n\t}\n\tt.InSearch = true\n\tsearchTerms := TermsFromString(terms)\n\tsearchIds := SearchIndex(ds, searchTerms)\n\tt.UpdateMessagesAsync(ds, &searchIds, searchTerms)\n\treturn\n}\n\n// recurses over the chunks originating at 'h' and pins them to the IPFS repo.\nfunc pinBlocks(node *core.IpfsNode, h hash.Hash, db datas.Database, depth, cnt int) (maxDepth, newCnt int) {\n\tmaxDepth, newCnt = depth, cnt\n\n\tcid := ipfs.NomsHashToCID(h)\n\t_, pinned, err := node.Pinning.IsPinned(cid)\n\td.Chk.NoError(err)\n\tif pinned {\n\t\treturn\n\t}\n\n\tctx, cancel := context.WithCancel(context.Background())\n\tdefer cancel()\n\n\tv := db.ReadValue(h)\n\td.Chk.NotNil(v)\n\n\tv.WalkRefs(func(r types.Ref) {\n\t\tvar newDepth int\n\t\tnewDepth, newCnt = pinBlocks(node, r.TargetHash(), db, depth+1, newCnt)\n\t\tmaxDepth = math.MaxInt(newDepth, maxDepth)\n\t})\n\n\tn, err := node.DAG.Get(ctx, cid)\n\td.Chk.NoError(err)\n\terr = node.Pinning.Pin(ctx, n, false)\n\td.Chk.NoError(err)\n\tnewCnt++\n\treturn\n}\n\ntype IPFSEventDelegate struct{}\n\nfunc (d IPFSEventDelegate) PinBlocks(node *core.IpfsNode, sourceDB, sinkDB datas.Database, sourceCommit types.Value) {\n\th := sourceCommit.Hash()\n\tdbg.Debug(\"Starting pinBlocks\")\n\tdepth, cnt := pinBlocks(node, h, sinkDB, 0, 0)\n\tdbg.Debug(\"Finished pinBlocks, depth: %d, cnt: %d\", depth, cnt)\n\tnode.Pinning.Flush()\n}\n\nfunc (d IPFSEventDelegate) SourceCommitFromMsgData(db datas.Database, msgData string) (datas.Database, types.Value) {\n\th := hash.Parse(msgData)\n\tv := db.ReadValue(h)\n\treturn db, v\n}\n\nfunc (d IPFSEventDelegate) HashFromMsgData(msgData string) (hash.Hash, error) {\n\tvar err error\n\th, ok := hash.MaybeParse(msgData)\n\tif !ok {\n\t\terr = fmt.Errorf(\"Failed to parse hash from msgData: %s\", msgData)\n\t}\n\treturn h, err\n}\n\nfunc (d IPFSEventDelegate) GenMessageData(cInfo ClientInfo, h hash.Hash) string {\n\treturn h.String()\n}\n\ntype P2PEventDelegate struct{}\n\nfunc (d P2PEventDelegate) PinBlocks(node *core.IpfsNode, sourceDB, sinkDB datas.Database, sourceCommit types.Value) {\n\tsourceRef := types.NewRef(sourceCommit)\n\tdatas.Pull(sourceDB, sinkDB, sourceRef, nil)\n}\n\nfunc (d P2PEventDelegate) SourceCommitFromMsgData(db datas.Database, msgData string) (datas.Database, types.Value) {\n\tsp, _ := spec.ForPath(msgData)\n\tv := sp.GetValue()\n\treturn sp.GetDatabase(), v\n}\n\nfunc (d P2PEventDelegate) HashFromMsgData(msgData string) (hash.Hash, error) {\n\tsp, err := spec.ForPath(msgData)\n\treturn sp.Path.Hash, err\n}\n\nfunc (d P2PEventDelegate) GenMessageData(cInfo ClientInfo, h hash.Hash) string {\n\treturn fmt.Sprintf(\"%s::#%s\", cInfo.Spec, h)\n}\n" }, { "path": "samples/go/decent/lib/importer.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"regexp\"\n\t\"sort\"\n\t\"strings\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/merge\"\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/datetime\"\n\t\"golang.org/x/net/html\"\n)\n\nvar (\n\tcharacter = \"\"\n\tmsgs = []Message{}\n)\n\nfunc RunImport(dir, dsSpec string) error {\n\tfilepath.Walk(dir, func(path string, info os.FileInfo, err error) error {\n\t\tif path == dir {\n\t\t\treturn nil\n\t\t}\n\t\tif !strings.HasSuffix(info.Name(), \".html\") {\n\t\t\treturn nil\n\t\t}\n\t\tfmt.Println(\"importing:\", path)\n\t\tf, err := os.Open(path)\n\t\td.Chk.NoError(err)\n\t\tn, err := html.Parse(f)\n\t\td.Chk.NoError(err)\n\t\textractDialog(n)\n\t\treturn nil\n\t})\n\n\tif len(msgs) == 0 {\n\t\treturn errors.New(\"Failed to import any data\")\n\t}\n\tfmt.Println(\"Imported\", len(msgs), \"messages\")\n\n\tsp, err := spec.ForDataset(dsSpec)\n\td.CheckErrorNoUsage(err)\n\tds := sp.GetDataset()\n\tds, err = InitDatabase(ds)\n\td.PanicIfError(err)\n\tdb := ds.Database()\n\n\tfmt.Println(\"Creating msg map\")\n\tkvPairs := []types.Value{}\n\tfor _, msg := range msgs {\n\t\tkvPairs = append(kvPairs, types.String(msg.ID()), marshal.MustMarshal(db, msg))\n\t}\n\tm := types.NewMap(db, kvPairs...)\n\n\tfmt.Println(\"Creating index\")\n\tti := NewTermIndex(db, types.NewMap(db)).Edit()\n\tfor _, msg := range msgs {\n\t\tterms := GetTerms(msg)\n\t\tti.InsertAll(terms, types.String(msg.ID()))\n\t}\n\ttermDocs := ti.Value().TermDocs\n\n\tfmt.Println(\"Creating users\")\n\tusers := topUsers(msgs)\n\n\tfmt.Println(\"Docs:\", termDocs.Len(), \"Users:\", len(users))\n\troot := Root{Messages: m, Index: termDocs, Users: users}\n\tnroot := marshal.MustMarshal(db, root)\n\tif ds.HasHead() {\n\t\tleft := ds.HeadValue()\n\t\tparent := marshal.MustMarshal(db, Root{\n\t\t\tIndex: types.NewMap(db),\n\t\t\tMessages: types.NewMap(db),\n\t\t})\n\t\tfmt.Println(\"Merging data\")\n\t\tnroot, err = merge.ThreeWay(left, nroot, parent, db, nil, nil)\n\t\tfmt.Println(\"Merging complete\")\n\t\td.Chk.NoError(err)\n\t}\n\tfmt.Println(\"Committing data\")\n\t_, err = db.CommitValue(ds, nroot)\n\treturn err\n}\n\nfunc extractDialog(n *html.Node) {\n\tif c := characterName(n); c != \"\" {\n\t\t//fmt.Println(\"Character:\", character)\n\t\tcharacter = c\n\t\treturn\n\t}\n\tif character != \"\" && n.Type == html.TextNode {\n\t\t//fmt.Println(\"Dialog:\", strings.TrimSpace(n.Data))\n\t\tmsg := Message{\n\t\t\tOrdinal: uint64(len(msgs)),\n\t\t\tAuthor: character,\n\t\t\tBody: strings.TrimSpace(n.Data),\n\t\t\tClientTime: datetime.Now(),\n\t\t}\n\t\tmsgs = append(msgs, msg)\n\t\tcharacter = \"\"\n\t}\n\tfor c := n.FirstChild; c != nil; c = c.NextSibling {\n\t\textractDialog(c)\n\t}\n}\n\nfunc characterName(n *html.Node) string {\n\tif n.Type != html.ElementNode ||\n\t\tn.Data != \"b\" ||\n\t\tn.FirstChild == nil {\n\t\treturn \"\"\n\t}\n\n\tif hasSpaces, _ := regexp.MatchString(`^\\s+[^\\s]`, n.FirstChild.Data); !hasSpaces {\n\t\treturn \"\"\n\t}\n\treturn strings.TrimSpace(n.FirstChild.Data)\n}\n\ntype cpair struct {\n\tcharacter string\n\tcnt int\n}\n\nfunc topUsers(msgs []Message) []string {\n\tuserpat := regexp.MustCompile(`^[a-zA-Z][a-zA-Z\\s]*\\d*$`)\n\tusermap := map[string]int{}\n\tfor _, msg := range msgs {\n\t\tname := strings.TrimSpace(msg.Author)\n\t\tif userpat.MatchString(name) {\n\t\t\tusermap[name] += 1\n\t\t}\n\t}\n\tpairs := []cpair{}\n\tfor name, cnt := range usermap {\n\t\tif len(name) > 1 && !strings.HasPrefix(name, \"ANOTHER\") {\n\t\t\tpairs = append(pairs, cpair{character: strings.ToLower(name), cnt: cnt})\n\t\t}\n\t}\n\t// sort descending by cnt\n\tsort.Slice(pairs, func(i, j int) bool {\n\t\treturn pairs[j].cnt < pairs[i].cnt\n\t})\n\tusers := []string{}\n\tfor i, p := range pairs {\n\t\tif i >= 30 {\n\t\t\tbreak\n\t\t}\n\t\tusers = append(users, p.character)\n\t}\n\tsort.Strings(users)\n\treturn users\n}\n" }, { "path": "samples/go/decent/lib/logger.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/samples/go/decent/dbg\"\n)\n\nfunc NewLogger(username string) *log.Logger {\n\tf, err := os.OpenFile(dbg.Filepath, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0644)\n\td.PanicIfError(err)\n\tprefix := fmt.Sprintf(\"%d-%s: \", os.Getpid(), username)\n\treturn log.New(f, prefix, 0644)\n}\n" }, { "path": "samples/go/decent/lib/model.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"fmt\"\n\t\"regexp\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/datetime\"\n\t\"github.com/attic-labs/noms/samples/go/decent/dbg\"\n)\n\ntype Root struct {\n\t// Map\n\t// Keys are strings like: ,\n\t// This scheme allows:\n\t// - map is naturally sorted in the right order\n\t// - conflicts will generally be avoided\n\t// - messages are editable\n\tMessages types.Map\n\tIndex types.Map\n\tUsers []string `noms:\",set\"`\n}\n\ntype Message struct {\n\tOrdinal uint64\n\tAuthor string\n\tBody string\n\tClientTime datetime.DateTime\n}\n\nfunc (m Message) ID() string {\n\treturn fmt.Sprintf(\"%020x/%s\", m.ClientTime.UnixNano(), m.Author)\n}\n\nfunc AddMessage(body string, author string, clientTime time.Time, ds datas.Dataset) (datas.Dataset, error) {\n\tdefer dbg.BoxF(\"AddMessage, body: %s\", body)()\n\troot, err := getRoot(ds)\n\tif err != nil {\n\t\treturn datas.Dataset{}, err\n\t}\n\n\tdb := ds.Database()\n\n\tnm := Message{\n\t\tAuthor: author,\n\t\tBody: body,\n\t\tClientTime: datetime.DateTime{clientTime},\n\t\tOrdinal: root.Messages.Len(),\n\t}\n\troot.Messages = root.Messages.Edit().Set(types.String(nm.ID()), marshal.MustMarshal(db, nm)).Map()\n\tIndexNewMessage(db, &root, nm)\n\tnewRoot := marshal.MustMarshal(db, root)\n\tds, err = db.CommitValue(ds, newRoot)\n\treturn ds, err\n}\n\nfunc InitDatabase(ds datas.Dataset) (datas.Dataset, error) {\n\tif ds.HasHead() {\n\t\treturn ds, nil\n\t}\n\tdb := ds.Database()\n\troot := Root{\n\t\tIndex: types.NewMap(db),\n\t\tMessages: types.NewMap(db),\n\t}\n\treturn db.CommitValue(ds, marshal.MustMarshal(db, root))\n}\n\nfunc GetAuthors(ds datas.Dataset) []string {\n\tr, err := getRoot(ds)\n\td.PanicIfError(err)\n\treturn r.Users\n}\n\nfunc IndexNewMessage(vrw types.ValueReadWriter, root *Root, m Message) {\n\tdefer dbg.BoxF(\"IndexNewMessage\")()\n\n\tti := NewTermIndex(vrw, root.Index)\n\tid := types.String(m.ID())\n\troot.Index = ti.Edit().InsertAll(GetTerms(m), id).Value().TermDocs\n\troot.Users = append(root.Users, m.Author)\n}\n\nfunc SearchIndex(ds datas.Dataset, search []string) types.Map {\n\troot, err := getRoot(ds)\n\td.PanicIfError(err)\n\tidx := root.Index\n\tti := NewTermIndex(ds.Database(), idx)\n\tids := ti.Search(search)\n\tdbg.Debug(\"search for: %s, returned: %d\", strings.Join(search, \" \"), ids.Len())\n\treturn ids\n}\n\nvar (\n\tpunctPat = regexp.MustCompile(\"[[:punct:]]+\")\n\twsPat = regexp.MustCompile(\"\\\\s+\")\n)\n\nfunc TermsFromString(s string) []string {\n\ts1 := punctPat.ReplaceAllString(strings.TrimSpace(s), \" \")\n\tterms := wsPat.Split(s1, -1)\n\tclean := []string{}\n\tfor _, t := range terms {\n\t\tif t == \"\" {\n\t\t\tcontinue\n\t\t}\n\t\tclean = append(clean, strings.ToLower(t))\n\t}\n\treturn clean\n}\n\nfunc GetTerms(m Message) []string {\n\tterms := TermsFromString(m.Body)\n\tterms = append(terms, TermsFromString(m.Author)...)\n\treturn terms\n}\n\nfunc ListMessages(ds datas.Dataset, searchIds *types.Map, doneChan chan struct{}) (msgMap types.Map, mc chan types.String, err error) {\n\t//dbg.Debug(\"##### listMessages: entered\")\n\n\troot, err := getRoot(ds)\n\tdb := ds.Database()\n\tif err != nil {\n\t\treturn types.NewMap(db), nil, err\n\t}\n\tmsgMap = root.Messages\n\n\tmc = make(chan types.String)\n\tdone := false\n\tgo func() {\n\t\t<-doneChan\n\t\tdone = true\n\t\t<-mc\n\t\t//dbg.Debug(\"##### listMessages: exiting 'done' goroutine\")\n\t}()\n\n\tgo func() {\n\t\tkeyMap := msgMap\n\t\tif searchIds != nil {\n\t\t\tkeyMap = *searchIds\n\t\t}\n\t\ti := uint64(0)\n\t\tfor ; i < keyMap.Len() && !done; i++ {\n\t\t\tkey, _ := keyMap.At(keyMap.Len() - i - 1)\n\t\t\tmc <- key.(types.String)\n\t\t}\n\t\t//dbg.Debug(\"##### listMessages: exiting 'for loop' goroutine, examined: %d\", i)\n\t\tclose(mc)\n\t}()\n\treturn\n}\n\nfunc getRoot(ds datas.Dataset) (Root, error) {\n\tdefer dbg.BoxF(\"getRoot\")()\n\n\tdb := ds.Database()\n\troot := Root{\n\t\tMessages: types.NewMap(db),\n\t\tIndex: types.NewMap(db),\n\t}\n\t// TODO: It would be nice if Dataset.MaybeHeadValue() or HeadValue()\n\t// would return just , and it would be nil if not there, so you\n\t// could chain calls.\n\tif !ds.HasHead() {\n\t\treturn root, nil\n\t}\n\terr := marshal.Unmarshal(ds.HeadValue(), &root)\n\tif err != nil {\n\t\treturn Root{}, err\n\t}\n\treturn root, nil\n}\n" }, { "path": "samples/go/decent/lib/model_test.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/util/datetime\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestBasics(t *testing.T) {\n\ta := assert.New(t)\n\tdb := datas.NewDatabase(chunks.NewMemoryStoreFactory().CreateStore(\"\"))\n\tds := db.GetDataset(\"foo\")\n\tml, err := getAllMessages(ds)\n\ta.NoError(err)\n\ta.Equal(0, len(ml))\n\n\tds, err = AddMessage(\"body1\", \"aa\", time.Unix(0, 0), ds)\n\ta.NoError(err)\n\tml, err = getAllMessages(ds)\n\ta.NoError(err)\n\texpected := []Message{\n\t\tMessage{\n\t\t\tAuthor: \"aa\",\n\t\t\tBody: \"body1\",\n\t\t\tClientTime: datetime.DateTime{time.Unix(0, 0)},\n\t\t\tOrdinal: 0,\n\t\t},\n\t}\n\ta.Equal(expected, ml)\n\n\tds, err = AddMessage(\"body2\", \"bob\", time.Unix(1, 0), ds)\n\ta.NoError(err)\n\tml, err = getAllMessages(ds)\n\texpected = append(\n\t\t[]Message{\n\t\t\tMessage{\n\t\t\t\tAuthor: \"bob\",\n\t\t\t\tBody: \"body2\",\n\t\t\t\tClientTime: datetime.DateTime{time.Unix(1, 0)},\n\t\t\t\tOrdinal: 1,\n\t\t\t},\n\t\t},\n\t\texpected...,\n\t)\n\ta.NoError(err)\n\ta.Equal(expected, ml)\n}\n\nfunc getAllMessages(ds datas.Dataset) (r []Message, err error) {\n\tdoneChan := make(chan struct{})\n\tmm, keys, _ := ListMessages(ds, nil, doneChan)\n\tfor k := range keys {\n\t\tmv := mm.Get(k)\n\t\tvar m Message\n\t\tmarshal.MustUnmarshal(mv, &m)\n\t\tr = append(r, m)\n\t}\n\tdoneChan <- struct{}{}\n\treturn r, nil\n}\n" }, { "path": "samples/go/decent/lib/pubsub.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/samples/go/decent/dbg\"\n\t\"github.com/ipfs/go-ipfs/core\"\n\t\"github.com/mr-tron/base58/base58\"\n)\n\nvar (\n\tPubsubUser = \"default\"\n\tseenHash = map[hash.Hash]bool{}\n\tseenHashMutex = sync.Mutex{}\n)\n\nfunc lockSeenF() func() {\n\tseenHashMutex.Lock()\n\treturn func() {\n\t\tseenHashMutex.Unlock()\n\t}\n}\n\n// RecieveMessages listens for messages sent by other chat nodes. It filters out\n// any msgs that have already been received and adds events to teh events channel\n// for any msgs that it hasn't seen yet.\nfunc ReceiveMessages(node *core.IpfsNode, events chan ChatEvent, cInfo ClientInfo) {\n\tsub, err := node.Floodsub.Subscribe(cInfo.Topic)\n\td.Chk.NoError(err)\n\n\tlistenForAndHandleMessage := func() {\n\t\tmsg, err := sub.Next(context.Background())\n\t\td.PanicIfError(err)\n\t\tsender := base58.Encode(msg.From)\n\t\tmsgMap := map[string]string{}\n\t\terr = json.Unmarshal(msg.Data, &msgMap)\n\t\tif err != nil {\n\t\t\tdbg.Debug(\"ReceiveMessages: received non-json msg: %s from: %s, error: %s\", msg.Data, sender, err)\n\t\t\treturn\n\t\t}\n\t\tmsgData := msgMap[\"data\"]\n\t\th, err := cInfo.Delegate.HashFromMsgData(msgData)\n\t\tif err != nil {\n\t\t\tdbg.Debug(\"ReceiveMessages: received unknown msg: %s from: %s\", msgData, sender)\n\t\t\treturn\n\t\t}\n\n\t\tdefer lockSeenF()()\n\t\tif !seenHash[h] {\n\t\t\tevents <- ChatEvent{EventType: SyncEvent, Event: msgData}\n\t\t\tseenHash[h] = true\n\t\t\tdbg.Debug(\"got msgData: %s from: %s(%s)\", msgData, sender, msgMap[\"user\"])\n\t\t}\n\t}\n\n\tdbg.Debug(\"start listening for msgs on channel: %s\", cInfo.Topic)\n\tfor {\n\t\tlistenForAndHandleMessage()\n\t}\n\tpanic(\"unreachable\")\n}\n\n// Publish asks the delegate to format a hash/ClientInfo into a suitable msg\n// and publishes that using IPFS pubsub.\nfunc Publish(node *core.IpfsNode, cInfo ClientInfo, h hash.Hash) {\n\tdefer func() {\n\t\tif r := recover(); r != nil {\n\t\t\tdbg.Debug(\"Publish failed, error: %s\", r)\n\t\t}\n\t}()\n\tmsgData := cInfo.Delegate.GenMessageData(cInfo, h)\n\tm, err := json.Marshal(map[string]string{\"user\": cInfo.Username, \"data\": msgData})\n\tif err != nil {\n\n\t}\n\td.PanicIfError(err)\n\tdbg.Debug(\"publishing to topic: %s, msg: %s\", cInfo.Topic, m)\n\tnode.Floodsub.Publish(cInfo.Topic, append(m, []byte(\"\\r\\n\")...))\n\n\tdefer lockSeenF()()\n\tseenHash[h] = true\n}\n" }, { "path": "samples/go/decent/lib/term_index.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"sync\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\ntype TermIndex struct {\n\tTermDocs types.Map\n\tvrw types.ValueReadWriter\n}\n\nfunc NewTermIndex(vrw types.ValueReadWriter, TermDocs types.Map) TermIndex {\n\treturn TermIndex{TermDocs, vrw}\n}\n\nfunc (ti TermIndex) Edit() *TermIndexEditor {\n\treturn &TermIndexEditor{ti.TermDocs.Edit(), ti.vrw}\n}\n\nfunc (ti TermIndex) Search(terms []string) types.Map {\n\tseen := make(map[string]struct{}, len(terms))\n\titers := make([]types.SetIterator, 0, len(terms))\n\n\twg := sync.WaitGroup{}\n\tidx := 0\n\tfor _, t := range terms {\n\t\tif _, ok := seen[t]; ok {\n\t\t\tcontinue\n\t\t}\n\t\tseen[t] = struct{}{}\n\n\t\titers = append(iters, nil)\n\t\ti := idx\n\t\tt := t\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tts := ti.TermDocs.Get(types.String(t))\n\t\t\tif ts != nil {\n\t\t\t\titer := ts.(types.Set).Iterator()\n\t\t\t\titers[i] = iter\n\t\t\t}\n\t\t\twg.Done()\n\t\t}()\n\n\t\tidx++\n\t}\n\twg.Wait()\n\n\tvar si types.SetIterator\n\tfor _, iter := range iters {\n\t\tif iter == nil {\n\t\t\treturn types.NewMap(ti.vrw) // at least one term had no hits\n\t\t}\n\n\t\tif si == nil {\n\t\t\tsi = iter // first iter\n\t\t\tcontinue\n\t\t}\n\n\t\tsi = types.NewIntersectionIterator(si, iter)\n\t}\n\n\tch := make(chan types.Value)\n\trch := types.NewStreamingMap(ti.vrw, ch)\n\tfor next := si.Next(); next != nil; next = si.Next() {\n\t\tch <- next\n\t\tch <- types.Bool(true)\n\t}\n\tclose(ch)\n\n\treturn <-rch\n}\n\ntype TermIndexEditor struct {\n\tterms *types.MapEditor\n\tvrw types.ValueReadWriter\n}\n\n// Builds a new TermIndex\nfunc (te *TermIndexEditor) Value() TermIndex {\n\treturn TermIndex{te.terms.Map(), te.vrw}\n}\n\n// Indexes |v| by |term|\nfunc (te *TermIndexEditor) Insert(term string, v types.Value) *TermIndexEditor {\n\ttv := types.String(term)\n\thitSet := te.terms.Get(tv)\n\tif hitSet == nil {\n\t\thitSet = types.NewSet(te.vrw)\n\t}\n\thsEd, ok := hitSet.(*types.SetEditor)\n\tif !ok {\n\t\thsEd = hitSet.(types.Set).Edit()\n\t\tte.terms.Set(tv, hsEd)\n\t}\n\n\thsEd.Insert(v)\n\treturn te\n}\n\n// Indexes |v| by each unique term in |terms| (tolerates duplicate terms)\nfunc (te *TermIndexEditor) InsertAll(terms []string, v types.Value) *TermIndexEditor {\n\tvisited := map[string]struct{}{}\n\tfor _, term := range terms {\n\t\tif _, ok := visited[term]; ok {\n\t\t\tcontinue\n\t\t}\n\t\tvisited[term] = struct{}{}\n\t\tte.Insert(term, v)\n\t}\n\treturn te\n}\n\n// TODO: te.Remove\n" }, { "path": "samples/go/decent/lib/term_index_test.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestRun(t *testing.T) {\n\ta := assert.New(t)\n\n\tstorage := &chunks.MemoryStorage{}\n\tvs := types.NewValueStore(storage.NewView())\n\tdefer vs.Close()\n\n\tdocs := []struct {\n\t\tterms string\n\t\tid int\n\t}{\n\t\t{\"foo bar baz\", 1},\n\t\t{\"foo baz\", 2},\n\t\t{\"baz bat boo\", 3},\n\t}\n\n\tindexEditor := NewTermIndex(vs, types.NewMap(vs)).Edit()\n\tfor _, doc := range docs {\n\t\tindexEditor.InsertAll(strings.Split(doc.terms, \" \"), types.Number(doc.id))\n\t}\n\n\tindex := indexEditor.Value()\n\n\tgetMap := func(keys ...int) types.Map {\n\t\tm := types.NewMap(vs).Edit()\n\t\tfor _, k := range keys {\n\t\t\tm.Set(types.Number(k), types.Bool(true))\n\t\t}\n\t\treturn m.Map()\n\t}\n\n\ttest := func(search string, expect types.Map) {\n\t\tactual := index.Search(strings.Split(search, \" \"))\n\t\ta.True(expect.Equals(actual))\n\t}\n\n\ttest(\"foo\", getMap(1, 2))\n\ttest(\"baz\", getMap(1, 2, 3))\n\ttest(\"bar baz\", getMap(1))\n\ttest(\"boo\", getMap(3))\n\ttest(\"blarg\", getMap())\n}\n" }, { "path": "samples/go/decent/lib/termui.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage lib\n\nimport (\n\t\"fmt\"\n\t\"regexp\"\n\t\"runtime\"\n\t\"strings\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/math\"\n\t\"github.com/attic-labs/noms/samples/go/decent/dbg\"\n\t\"github.com/jroimartin/gocui\"\n)\n\nconst (\n\tallViews = \"\"\n\tusersView = \"users\"\n\tmessageView = \"messages\"\n\tinputView = \"input\"\n\tlinestofetch = 50\n\n\tsearchPrefix = \"/s\"\n\tquitPrefix = \"/q\"\n)\n\ntype TermUI struct {\n\tGui *gocui.Gui\n\tInSearch bool\n\tlines []string\n\tdp *dataPager\n}\n\nvar (\n\tviewNames = []string{usersView, messageView, inputView}\n\tfirstLayout = true\n)\n\nfunc CreateTermUI(events chan ChatEvent) *TermUI {\n\tg, err := gocui.NewGui(gocui.Output256)\n\td.PanicIfError(err)\n\n\tg.Highlight = true\n\tg.SelFgColor = gocui.ColorGreen\n\tg.Cursor = true\n\n\trelayout := func(g *gocui.Gui) error {\n\t\treturn layout(g)\n\t}\n\tg.SetManagerFunc(relayout)\n\n\ttermUI := new(TermUI)\n\ttermUI.Gui = g\n\n\td.PanicIfError(g.SetKeybinding(allViews, gocui.KeyF1, gocui.ModNone, debugInfo(termUI)))\n\td.PanicIfError(g.SetKeybinding(allViews, gocui.KeyCtrlC, gocui.ModNone, quit))\n\td.PanicIfError(g.SetKeybinding(allViews, gocui.KeyCtrlC, gocui.ModAlt, quitWithStack))\n\td.PanicIfError(g.SetKeybinding(allViews, gocui.KeyTab, gocui.ModNone, nextView))\n\td.PanicIfError(g.SetKeybinding(messageView, gocui.KeyArrowUp, gocui.ModNone, arrowUp(termUI)))\n\td.PanicIfError(g.SetKeybinding(messageView, gocui.KeyArrowDown, gocui.ModNone, arrowDown(termUI)))\n\td.PanicIfError(g.SetKeybinding(inputView, gocui.KeyEnter, gocui.ModNone, func(g *gocui.Gui, v *gocui.View) (err error) {\n\t\tdefer func() {\n\t\t\tv.Clear()\n\t\t\tv.SetCursor(0, 0)\n\t\t\tmsgView, err := g.View(messageView)\n\t\t\td.PanicIfError(err)\n\t\t\tmsgView.Title = \"messages\"\n\t\t\tmsgView.Autoscroll = true\n\t\t}()\n\t\tbuf := strings.TrimSpace(v.Buffer())\n\t\tif strings.HasPrefix(buf, searchPrefix) {\n\t\t\tevents <- ChatEvent{EventType: SearchEvent, Event: strings.TrimSpace(buf[len(searchPrefix):])}\n\t\t\treturn\n\t\t}\n\t\tif strings.HasPrefix(buf, quitPrefix) {\n\t\t\terr = gocui.ErrQuit\n\t\t\treturn\n\t\t}\n\t\tevents <- ChatEvent{EventType: InputEvent, Event: buf}\n\t\treturn\n\t}))\n\n\treturn termUI\n}\n\nfunc (t *TermUI) Close() {\n\tdbg.Debug(\"Closing gui\")\n\tt.Gui.Close()\n}\n\nfunc (t *TermUI) UpdateMessagesFromSync(ds datas.Dataset) {\n\tif t.InSearch || !t.textScrolledToEnd() {\n\t\tt.Gui.Execute(func(g *gocui.Gui) (err error) {\n\t\t\tupdateViewTitle(g, messageView, \"messages (NEW!)\")\n\t\t\treturn\n\t\t})\n\t} else {\n\t\tt.UpdateMessagesAsync(ds, nil, nil)\n\t}\n}\n\nfunc (t *TermUI) Layout() error {\n\treturn layout(t.Gui)\n}\n\nfunc layout(g *gocui.Gui) error {\n\tmaxX, maxY := g.Size()\n\tif v, err := g.SetView(usersView, 0, 0, 25, maxY-1); err != nil {\n\t\tif err != gocui.ErrUnknownView {\n\t\t\treturn err\n\t\t}\n\t\tv.Title = usersView\n\t\tv.Wrap = false\n\t\tv.Editable = false\n\t}\n\tif v, err := g.SetView(messageView, 25, 0, maxX-1, maxY-2-1); err != nil {\n\t\tif err != gocui.ErrUnknownView {\n\t\t\treturn err\n\t\t}\n\t\tv.Title = messageView\n\t\tv.Editable = false\n\t\tv.Wrap = true\n\t\tv.Autoscroll = true\n\t\treturn nil\n\t}\n\tif v, err := g.SetView(inputView, 25, maxY-2-1, maxX-1, maxY-1); err != nil {\n\t\tif err != gocui.ErrUnknownView {\n\t\t\treturn err\n\t\t}\n\t\tv.Wrap = true\n\t\tv.Editable = true\n\t\tv.Autoscroll = true\n\t}\n\tif firstLayout {\n\t\tfirstLayout = false\n\t\tg.SetCurrentView(inputView)\n\t\tdbg.Debug(\"started up\")\n\t}\n\treturn nil\n}\n\nfunc (t *TermUI) UpdateMessages(ds datas.Dataset, filterIds *types.Map, terms []string) error {\n\tdefer dbg.BoxF(\"updateMessages\")()\n\n\tt.ResetAuthors(ds)\n\tv, err := t.Gui.View(messageView)\n\td.PanicIfError(err)\n\tv.Clear()\n\tt.lines = []string{}\n\tv.SetOrigin(0, 0)\n\t_, winHeight := v.Size()\n\n\tif t.dp != nil {\n\t\tt.dp.Close()\n\t}\n\n\tdoneChan := make(chan struct{})\n\tmsgMap, msgKeyChan, err := ListMessages(ds, filterIds, doneChan)\n\td.PanicIfError(err)\n\tt.dp = NewDataPager(ds, msgKeyChan, doneChan, msgMap, terms)\n\tt.lines, _ = t.dp.Prepend(t.lines, math.MaxInt(linestofetch, winHeight+10))\n\n\tfor _, s := range t.lines {\n\t\tfmt.Fprintf(v, \"%s\\n\", s)\n\t}\n\treturn nil\n}\n\nfunc (t *TermUI) ResetAuthors(ds datas.Dataset) {\n\tv, err := t.Gui.View(usersView)\n\td.PanicIfError(err)\n\tv.Clear()\n\tfor _, u := range GetAuthors(ds) {\n\t\tfmt.Fprintln(v, u)\n\t}\n}\n\nfunc (t *TermUI) UpdateMessagesAsync(ds datas.Dataset, sids *types.Map, terms []string) {\n\tt.Gui.Execute(func(_ *gocui.Gui) error {\n\t\terr := t.UpdateMessages(ds, sids, terms)\n\t\td.PanicIfError(err)\n\t\treturn nil\n\t})\n}\n\nfunc (t *TermUI) scrollView(v *gocui.View, dy int) {\n\t// Get the size and position of the view.\n\tlineCnt := len(t.lines)\n\t_, windowHeight := v.Size()\n\tox, oy := v.Origin()\n\tcx, cy := v.Cursor()\n\n\t// maxCy will either be the height of the screen - 1, or in the case that\n\t// the there aren't enough lines to fill the screen, it will be the\n\t// lineCnt - origin\n\tnewCy := cy + dy\n\tmaxCy := math.MinInt(lineCnt-oy, windowHeight-1)\n\n\t// If the newCy doesn't require scrolling, then just move the cursor.\n\tif newCy >= 0 && newCy < maxCy {\n\t\tv.MoveCursor(cx, dy, false)\n\t\treturn\n\t}\n\n\t// If the cursor is already at the bottom of the screen and there are no\n\t// lines left to scroll up, then we're at the bottom.\n\tif newCy >= maxCy && oy >= lineCnt-windowHeight {\n\t\t// Set autoscroll to normal again.\n\t\tv.Autoscroll = true\n\t} else {\n\t\t// The cursor is already at the bottom or top of the screen so scroll\n\t\t// the text\n\t\tv.Autoscroll = false\n\t\tv.SetOrigin(ox, oy+dy)\n\t}\n}\n\nfunc quit(_ *gocui.Gui, _ *gocui.View) error {\n\tdbg.Debug(\"QUITTING #####\")\n\treturn gocui.ErrQuit\n}\n\nfunc quitWithStack(_ *gocui.Gui, _ *gocui.View) error {\n\tdbg.Debug(\"QUITTING WITH STACK\")\n\tstacktrace := make([]byte, 1024*1024)\n\tlength := runtime.Stack(stacktrace, true)\n\tdbg.Debug(string(stacktrace[:length]))\n\treturn gocui.ErrQuit\n}\n\nfunc arrowUp(t *TermUI) func(*gocui.Gui, *gocui.View) error {\n\treturn func(_ *gocui.Gui, v *gocui.View) error {\n\t\tlineCnt := len(t.lines)\n\t\tox, oy := v.Origin()\n\t\tif oy == 0 {\n\t\t\tvar ok bool\n\t\t\tt.lines, ok = t.dp.Prepend(t.lines, linestofetch)\n\t\t\tif ok {\n\t\t\t\tv.Clear()\n\t\t\t\tfor _, s := range t.lines {\n\t\t\t\t\tfmt.Fprintf(v, \"%s\\n\", s)\n\t\t\t\t}\n\t\t\t\tc1 := len(t.lines)\n\t\t\t\tv.SetOrigin(ox, c1-lineCnt)\n\t\t\t}\n\t\t}\n\t\tt.scrollView(v, -1)\n\t\treturn nil\n\t}\n}\n\nfunc arrowDown(t *TermUI) func(*gocui.Gui, *gocui.View) error {\n\treturn func(_ *gocui.Gui, v *gocui.View) error {\n\t\tt.scrollView(v, 1)\n\t\treturn nil\n\t}\n}\n\nfunc debugInfo(t *TermUI) func(*gocui.Gui, *gocui.View) error {\n\treturn func(g *gocui.Gui, _ *gocui.View) error {\n\t\tmsgView, _ := g.View(messageView)\n\t\tw, h := msgView.Size()\n\t\tdbg.Debug(\"info, window size:(%d, %d), lineCnt: %d\", w, h, len(t.lines))\n\t\tcx, cy := msgView.Cursor()\n\t\tox, oy := msgView.Origin()\n\t\tdbg.Debug(\"info, origin: (%d,%d), cursor: (%d,%d)\", ox, oy, cx, cy)\n\t\tdbg.Debug(\"info, view buffer:\\n%s\", highlightTerms(viewBuffer(msgView), t.dp.terms))\n\t\treturn nil\n\t}\n}\n\nfunc viewBuffer(v *gocui.View) string {\n\tbuf := strings.TrimSpace(v.ViewBuffer())\n\tif len(buf) > 0 && buf[len(buf)-1] != byte('\\n') {\n\t\tbuf = buf + \"\\n\"\n\t}\n\treturn buf\n}\n\nfunc nextView(g *gocui.Gui, v *gocui.View) (err error) {\n\tnextName := nextViewName(v.Name())\n\tif _, err = g.SetCurrentView(nextName); err != nil {\n\t\treturn\n\t}\n\t_, err = g.SetViewOnTop(nextName)\n\treturn\n}\n\nfunc nextViewName(currentView string) string {\n\tfor i, viewname := range viewNames {\n\t\tif currentView == viewname {\n\t\t\treturn viewNames[(i+1)%len(viewNames)]\n\t\t}\n\t}\n\treturn viewNames[0]\n}\n\nfunc (t *TermUI) textScrolledToEnd() bool {\n\tv, err := t.Gui.View(messageView)\n\tif err != nil {\n\t\t// doubt this will ever happen, if it does just assume we're at bottom\n\t\treturn true\n\t}\n\t_, oy := v.Origin()\n\t_, h := v.Size()\n\tlc := len(t.lines)\n\tdbg.Debug(\"textScrolledToEnd, oy: %d, h: %d, lc: %d, lc-oy: %d, res: %t\", oy, h, lc, lc-oy, lc-oy <= h)\n\treturn lc-oy <= h\n}\n\nfunc updateViewTitle(g *gocui.Gui, viewname, title string) (err error) {\n\tv, err := g.View(viewname)\n\tif err != nil {\n\t\treturn\n\t}\n\tv.Title = title\n\treturn\n}\n\nvar bgColors, fgColors = genColors()\n\nfunc genColors() ([]string, []string) {\n\tbg, fg := []string{}, []string{}\n\tfor i := 1; i <= 9; i++ {\n\t\t// skip dark blue & white\n\t\tif i != 4 && i != 7 {\n\t\t\tbg = append(bg, fmt.Sprintf(\"\\x1b[48;5;%dm\\x1b[30m%%s\\x1b[0m\", i))\n\t\t\tfg = append(fg, fmt.Sprintf(\"\\x1b[38;5;%dm%%s\\x1b[0m\", i))\n\t\t}\n\t}\n\treturn bg, fg\n}\n\nfunc colorTerm(color int, s string, background bool) string {\n\tc := fgColors[color]\n\tif background {\n\t\tc = bgColors[color]\n\t}\n\treturn fmt.Sprintf(c, s)\n}\n\nfunc highlightTerms(s string, terms []string) string {\n\tfor i, t := range terms {\n\t\tcolor := i % len(fgColors)\n\t\tre := regexp.MustCompile(fmt.Sprintf(\"(?i)%s\", regexp.QuoteMeta(t)))\n\t\ts = re.ReplaceAllStringFunc(s, func(s string) string {\n\t\t\treturn colorTerm(color, s, false)\n\t\t})\n\t}\n\treturn s\n}\n" }, { "path": "samples/go/decent/p2p-chat/README.md", "content": "This demo application is the simplest p2p chat app you could build using Noms.\n\nBasic idea:\n\n- Every node runs a Noms HTTP server (port controlled by --port) flag\n- Every node broadcasts its current commit and IP/port continuously\n- Every node continuously sync/merges with every other node\n (note that due to content addressing, most of these syncs will immediately exit)\n\n\n" }, { "path": "samples/go/decent/p2p-chat/main.go", "content": "// See: https://github.com/attic-labs/noms/issues/3808\n// +build ignore\n\n// Copyright 2017 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"net\"\n\t\"os\"\n\t\"os/signal\"\n\t\"path\"\n\t\"syscall\"\n\n\t\"github.com/attic-labs/noms/go/config\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/ipfs\"\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/attic-labs/noms/go/util/profile\"\n\t\"github.com/attic-labs/noms/samples/go/decent/dbg\"\n\t\"github.com/attic-labs/noms/samples/go/decent/lib\"\n\t\"github.com/jroimartin/gocui\"\n\tkingpin \"gopkg.in/alecthomas/kingpin.v2\"\n)\n\nfunc main() {\n\t// allow short (-h) help\n\tkingpin.CommandLine.HelpFlag.Short('h')\n\n\tclientCmd := kingpin.Command(\"client\", \"runs the ipfs-chat client UI\")\n\tclientTopic := clientCmd.Flag(\"topic\", \"IPFS pubsub topic to publish and subscribe to\").Default(\"noms-chat-p2p\").String()\n\tusername := clientCmd.Flag(\"username\", \"username to sign in as\").Required().String()\n\tnodeIdx := clientCmd.Flag(\"node-idx\", \"a single digit to be used as last digit in all port values: api, gateway and swarm (must be 0-9 inclusive)\").Default(\"-1\").Int()\n\tclientDir := clientCmd.Arg(\"path\", \"local directory to store data in\").Required().ExistingDir()\n\n\timportCmd := kingpin.Command(\"import\", \"imports data into a chat\")\n\timportSrc := importCmd.Flag(\"dir\", \"directory that contains data to import\").Default(\"../data\").ExistingDir()\n\timportDir := importCmd.Arg(\"path\", \"local directory to store data in\").Required().ExistingDir()\n\n\tkingpin.CommandLine.Help = \"A demonstration of using Noms to build a scalable multiuser collaborative application.\"\n\n\tswitch kingpin.Parse() {\n\tcase \"client\":\n\t\tcInfo := lib.ClientInfo{\n\t\t\tTopic: *clientTopic,\n\t\t\tUsername: *username,\n\t\t\tIdx: *nodeIdx,\n\t\t\tIsDaemon: false,\n\t\t\tDir: *clientDir,\n\t\t\tDelegate: lib.P2PEventDelegate{},\n\t\t}\n\t\trunClient(cInfo)\n\tcase \"import\":\n\t\terr := lib.RunImport(*importSrc, fmt.Sprintf(\"%s/noms::chat\", *importDir))\n\t\td.PanicIfError(err)\n\t}\n}\n\nfunc runClient(cInfo lib.ClientInfo) {\n\tdbg.SetLogger(lib.NewLogger(cInfo.Username))\n\n\tvar err error\n\thttpPort := 8000 + cInfo.Idx\n\tsp, err := spec.ForDatabase(fmt.Sprintf(\"http://%s:%d\", getIP(), httpPort))\n\td.PanicIfError(err)\n\tcInfo.Spec = sp\n\n\t<-runServer(path.Join(cInfo.Dir, \"noms\"), httpPort)\n\n\tdb := cInfo.Spec.GetDatabase()\n\tds := db.GetDataset(\"chat\")\n\tds, err = lib.InitDatabase(ds)\n\td.PanicIfError(err)\n\n\tnode := ipfs.OpenIPFSRepo(path.Join(cInfo.Dir, \"ipfs\"), cInfo.Idx)\n\tevents := make(chan lib.ChatEvent, 1024)\n\tt := lib.CreateTermUI(events)\n\tdefer t.Close()\n\n\td.PanicIfError(t.Layout())\n\tt.ResetAuthors(ds)\n\tt.UpdateMessages(ds, nil, nil)\n\n\tgo lib.ProcessChatEvents(node, ds, events, t, cInfo)\n\tgo lib.ReceiveMessages(node, events, cInfo)\n\n\tif err := t.Gui.MainLoop(); err != nil && err != gocui.ErrQuit {\n\t\tdbg.Debug(\"mainloop has exited, err:\", err)\n\t\tlog.Panicln(err)\n\t}\n}\n\nfunc getIP() string {\n\tifaces, err := net.Interfaces()\n\td.PanicIfError(err)\n\tfor _, i := range ifaces {\n\t\taddrs, err := i.Addrs()\n\t\td.PanicIfError(err)\n\t\tfor _, addr := range addrs {\n\t\t\tswitch v := addr.(type) {\n\t\t\tcase *net.IPNet:\n\t\t\t\tif !v.IP.IsLoopback() {\n\t\t\t\t\tip := v.IP.To4()\n\t\t\t\t\tif ip != nil {\n\t\t\t\t\t\treturn v.IP.String()\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\td.Panic(\"notreached\")\n\treturn \"\"\n}\n\nfunc runServer(atPath string, port int) (ready chan struct{}) {\n\tready = make(chan struct{})\n\t_ = os.Mkdir(atPath, 0755)\n\tcfg := config.NewResolver()\n\tcs, err := cfg.GetChunkStore(atPath)\n\td.CheckError(err)\n\tserver := datas.NewRemoteDatabaseServer(cs, port)\n\tserver.Ready = func() {\n\t\tready <- struct{}{}\n\t}\n\n\t// Shutdown server gracefully so that profile may be written\n\tc := make(chan os.Signal, 1)\n\tsignal.Notify(c, os.Interrupt)\n\tsignal.Notify(c, syscall.SIGTERM)\n\tgo func() {\n\t\t<-c\n\t\tserver.Stop()\n\t}()\n\n\tgo func() {\n\t\td.Try(func() {\n\t\t\tdefer profile.MaybeStartProfile().Stop()\n\t\t\tserver.Run()\n\t\t})\n\t}()\n\treturn\n}\n" }, { "path": "samples/go/nomdex/Readme.md", "content": "# Nomdex \n\nNomdex demonstrates how Noms maps can be used to index values in a database and provides a simple query language to search for objects.\n\n## Description\nThis program experiments with using ordinary Noms Maps as indexes. It leverages the fact that Maps in Noms are implemented by prolly-trees which are similar to B-Trees in many important ways that make them ideal for use as indexes. They are balanced, sorted, require relatively few accesses to find any leaf node and efficient to update.\n\n###Building Indexes\nNomdex constructs indexes as Maps that are keyed by either Strings or Numbers. The values in the index are sets of objects. The following command can be used to build an index:\n```shell\nnomdex up --in-path --by --out-ds \n```\nThe ***'in-path'*** argument must be a ValueSpec(see [Spelling In Noms](../../../doc/spelling.md#spelling-values)) that designates the root of an object hierarchy to be scanned for \"indexable\" objects.\n\nThe ***'by'*** argument must be a relative path. Nomdex traverses every value reachable from 'in-path' and attempts to resolve this relative ***'by'*** path from it. Any value that has a String, Number, or Bool index using the relative attribute as it's key.\n\nThe ***'out-ds'*** argument specifies a dataset name that will be used to store the new index.\n\nIn addition, there are arguments that allow values to be transformed before using them as keys in the index by applying regex expressions functions. Consult to the help text and code to see how those can be used.\n\n### Queries in Nomdex\nOnce an index is built, it can be queried against using the nomdex find command. For example, given a database that contains structs of the following type representing cities:\n```go\nstruct Row {\n City: String,\n State: String,\n GeoPos: struct {\n Latitude: Number,\n Longitude: Number,\n }\n}\n```\nThe following commands could be used to build indexes on the City, State, Latitude and Longitude attibutes.\n```shell\nnomdex up --in-path http://localhost:8000::cities --by .City --out-ds by-name\nnomdex up --in-path http://localhost:8000::cities --by .State --out-ds by-state\nnomdex up --in-path http://localhost:8000::cities --by .GeoPos.Latitude --out-ds by-lat\nnomdex up --in-path http://localhost:8000::cities --by .GeoPos.Longitude --out-ds by-lon\n```\nOnce these indexes are created, the following queries could be made using the find command:\n```shell\n// find all cities in California\nnomdex find --db http://localhost:8000 'by-state = \"California\"'\n\n// find all cities whose name begins with A, B, or C\nnomdex find --db http://localhost:8000 'by-name >= \"A\" and by-name < \"D\"'\n\n// Find all tropical cities whose name begins with A, B, or C\nnomdex find --db http://localhost:8000 '(by-name >= \"A\" and by-name < \"D\") and (by-lat >= -23.5 and by-lat <= 23.5)\n```\nThe nomdex query language is simple, it consists of comparison expressions which take the form of '*indexName comparisonOperator constantValue*'. Index names are the dataset given as the ***'out-ds'*** argument to the *build* command. Comparison operators can be one of: <, <=, >, >=, =, !=. Constants are either String values which are quoted: \"hi, I'm a string constant\", and Numbers which consist of digits and an optional decimal point and minus sign: 1, -1, 2.3, -3.2.\n\nIn addition, comparison expressions can be combined using \"and\" and \"or\". Parenthesis can, and should be used to express the order that evaluation should take place.\n\nNote: nomdex is not a complete query system. It's purpose is only to illustrate the fact that Noms maps have all the necessary properties to be used as indexes. A complete query system would have many additional features and the ability to optimize queries in an intelligent way.\n" }, { "path": "samples/go/nomdex/expr.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\ntype expr interface {\n\tranges() queryRangeSlice\n\tdbgPrintTree(w io.Writer, level int)\n\tindexName() string\n\titerator(im *indexManager) types.SetIterator\n}\n\n// logExpr represents a logical 'and' or 'or' expression between two other expressions.\n// e.g. logExpr would represent the and/or expressions in this query:\n// (index1 > 0 and index1 < 9) or (index1 > 100 and index < 109)\ntype logExpr struct {\n\top boolOp\n\texpr1 expr\n\texpr2 expr\n\tidxName string\n}\n\ntype compExpr struct {\n\tidxName string\n\top compOp\n\tv1 types.Value\n}\n\nfunc (le logExpr) indexName() string {\n\treturn le.idxName\n}\n\nfunc (le logExpr) iterator(im *indexManager) types.SetIterator {\n\tif le.idxName != \"\" {\n\t\treturn unionizeIters(iteratorsFromRanges(im.indexes[le.idxName], le.ranges()))\n\t}\n\n\ti1 := le.expr1.iterator(im)\n\ti2 := le.expr2.iterator(im)\n\tvar iter types.SetIterator\n\tswitch le.op {\n\tcase and:\n\t\tif i1 == nil || i2 == nil {\n\t\t\treturn nil\n\t\t}\n\t\titer = types.NewIntersectionIterator(le.expr1.iterator(im), le.expr2.iterator(im))\n\tcase or:\n\t\tif i1 == nil {\n\t\t\treturn i2\n\t\t}\n\t\tif i2 == nil {\n\t\t\treturn i1\n\t\t}\n\t\titer = types.NewUnionIterator(le.expr1.iterator(im), le.expr2.iterator(im))\n\t}\n\treturn iter\n}\n\nfunc (le logExpr) ranges() (ranges queryRangeSlice) {\n\trslice1 := le.expr1.ranges()\n\trslice2 := le.expr2.ranges()\n\trslice := queryRangeSlice{}\n\n\tswitch le.op {\n\tcase and:\n\t\tif len(rslice1) == 0 || len(rslice2) == 0 {\n\t\t\treturn rslice\n\t\t}\n\t\tfor _, r1 := range rslice1 {\n\t\t\tfor _, r2 := range rslice2 {\n\t\t\t\trslice = append(rslice, r1.and(r2)...)\n\t\t\t}\n\t\t}\n\t\tsort.Sort(rslice)\n\t\treturn rslice\n\tcase or:\n\t\tif len(rslice1) == 0 {\n\t\t\treturn rslice2\n\t\t}\n\t\tif len(rslice2) == 0 {\n\t\t\treturn rslice1\n\t\t}\n\t\tfor _, r1 := range rslice1 {\n\t\t\tfor _, r2 := range rslice2 {\n\t\t\t\trslice = append(rslice, r1.or(r2)...)\n\t\t\t}\n\t\t}\n\t\tsort.Sort(rslice)\n\t\treturn rslice\n\t}\n\treturn queryRangeSlice{}\n}\n\nfunc (le logExpr) dbgPrintTree(w io.Writer, level int) {\n\tfmt.Fprintf(w, \"%*s%s\\n\", 2*level, \"\", le.op)\n\tif le.expr1 != nil {\n\t\tle.expr1.dbgPrintTree(w, level+1)\n\t}\n\tif le.expr2 != nil {\n\t\tle.expr2.dbgPrintTree(w, level+1)\n\t}\n}\n\nfunc (re compExpr) indexName() string {\n\treturn re.idxName\n}\n\nfunc iteratorsFromRange(index types.Map, rd queryRange) []types.SetIterator {\n\tfirst := true\n\titerators := []types.SetIterator{}\n\tindex.IterFrom(rd.lower.value, func(k, v types.Value) bool {\n\t\tif first && rd.lower.value != nil && !rd.lower.include && rd.lower.value.Equals(k) {\n\t\t\treturn false\n\t\t}\n\t\tif rd.upper.value != nil {\n\t\t\tif !rd.upper.include && rd.upper.value.Equals(k) {\n\t\t\t\treturn true\n\t\t\t}\n\t\t\tif rd.upper.value.Less(k) {\n\t\t\t\treturn true\n\t\t\t}\n\t\t}\n\t\ts := v.(types.Set)\n\t\titerators = append(iterators, s.Iterator())\n\t\treturn false\n\t})\n\treturn iterators\n}\n\nfunc iteratorsFromRanges(index types.Map, ranges queryRangeSlice) []types.SetIterator {\n\titerators := []types.SetIterator{}\n\tfor _, r := range ranges {\n\t\titerators = append(iterators, iteratorsFromRange(index, r)...)\n\t}\n\treturn iterators\n}\n\nfunc unionizeIters(iters []types.SetIterator) types.SetIterator {\n\tif len(iters) == 0 {\n\t\treturn nil\n\t}\n\tif len(iters) <= 1 {\n\t\treturn iters[0]\n\t}\n\n\tunionIters := []types.SetIterator{}\n\tvar iter0 types.SetIterator\n\tfor i, iter := range iters {\n\t\tif i%2 == 0 {\n\t\t\titer0 = iter\n\t\t} else {\n\t\t\tunionIters = append(unionIters, types.NewUnionIterator(iter0, iter))\n\t\t\titer0 = nil\n\t\t}\n\t}\n\tif iter0 != nil {\n\t\tunionIters = append(unionIters, iter0)\n\t}\n\treturn unionizeIters(unionIters)\n}\n\nfunc (re compExpr) iterator(im *indexManager) types.SetIterator {\n\tindex := im.indexes[re.idxName]\n\titers := iteratorsFromRanges(index, re.ranges())\n\treturn unionizeIters(iters)\n}\n\nfunc (re compExpr) ranges() (ranges queryRangeSlice) {\n\tvar r queryRange\n\tswitch re.op {\n\tcase equals:\n\t\te := bound{value: re.v1, include: true}\n\t\tr = queryRange{lower: e, upper: e}\n\tcase gt:\n\t\tr = queryRange{lower: bound{re.v1, false, 0}, upper: bound{nil, true, 1}}\n\tcase gte:\n\t\tr = queryRange{lower: bound{re.v1, true, 0}, upper: bound{nil, true, 1}}\n\tcase lt:\n\t\tr = queryRange{lower: bound{nil, true, -1}, upper: bound{re.v1, false, 0}}\n\tcase lte:\n\t\tr = queryRange{lower: bound{nil, true, -1}, upper: bound{re.v1, true, 0}}\n\tcase ne:\n\t\treturn queryRangeSlice{\n\t\t\t{lower: bound{nil, true, -1}, upper: bound{re.v1, false, 0}},\n\t\t\t{lower: bound{re.v1, false, 0}, upper: bound{nil, true, 1}},\n\t\t}\n\t}\n\treturn queryRangeSlice{r}\n}\n\nfunc (re compExpr) dbgPrintTree(w io.Writer, level int) {\n\tbuf := bytes.Buffer{}\n\ttypes.WriteEncodedValue(&buf, re.v1)\n\tfmt.Fprintf(w, \"%*s%s %s %s\\n\", 2*level, \"\", re.idxName, re.op, buf.String())\n}\n" }, { "path": "samples/go/nomdex/nomdex.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\n\t\"github.com/attic-labs/kingpin\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/util/profile\"\n\t\"github.com/attic-labs/noms/go/util/verbose\"\n)\n\nfunc main() {\n\tregisterUpdate()\n\tregisterFind()\n\tverbose.RegisterVerboseFlags(kingpin.CommandLine)\n\tprofile.RegisterProfileFlags(kingpin.CommandLine)\n\n\tswitch kingpin.Parse() {\n\tcase \"up\":\n\t\trunUpdate()\n\tcase \"find\":\n\t\trunFind()\n\t}\n}\n\nfunc printError(err error, msgAndArgs ...interface{}) bool {\n\tif err != nil {\n\t\terr := d.Unwrap(err)\n\t\tswitch len(msgAndArgs) {\n\t\tcase 0:\n\t\t\tfmt.Fprintf(os.Stderr, \"error: %s\\n\", err)\n\t\tcase 1:\n\t\t\tfmt.Fprintf(os.Stderr, \"%s%s\\n\", msgAndArgs[0], err)\n\t\tdefault:\n\t\t\tformat, ok := msgAndArgs[0].(string)\n\t\t\tif ok {\n\t\t\t\ts1 := fmt.Sprintf(format, msgAndArgs[1:]...)\n\t\t\t\tfmt.Fprintf(os.Stderr, \"%s%s\\n\", s1, err)\n\t\t\t} else {\n\t\t\t\tfmt.Fprintf(os.Stderr, \"error: %s\\n\", err)\n\t\t\t}\n\t\t}\n\t}\n\treturn err != nil\n}\n" }, { "path": "samples/go/nomdex/nomdex_find.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\n\t\"github.com/attic-labs/kingpin\"\n\n\t\"github.com/attic-labs/noms/go/config\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/outputpager\"\n)\n\nvar longFindHelp = `'nomdex find' retrieves and prints objects that satisfy the 'query' argument.\n\nIndexes are built using the 'nomdex up' command. For information about building\nindexes, see: nomdex up -h\n\nObjects that have been indexed can be quickly found using the nomdex query\nlanguage. For example, consider objects with the following type:\n\nstruct Person {\n name String,\n geopos struct GeoPos {\n latitude Number,\n longitude Number,\n }\n}\n\nObjects of this type can be indexed on the name, latitude and longitude fields\nwith the following commands:\n nomdex up --in-path ~/nomsdb::people.value --by .name --out-ds by-name\n nomdex up --in-path ~/nomsdb::people.value --by .geopos.latitude --out-ds by-lat\n nomdex up --in-path ~/nomsdb::people.value --by .geopos.longitude --out-ds by-lng\n\nThe following query could be used to find all people with an address near the\nequator:\n nomdex find 'by-lat >= -1.0 and by-lat <= 1.0'\n\nWe could also get a list of all people who live near the equator whose name begins with \"A\":\n nomdex find '(by-name >= \"A\" and by-name < \"B\") and (by-lat >= -1.0 and by-lat <= 1.0)'\n\nThe query language is simple. It currently supports the following relational operators:\n <, <=, >, >=, =, !=\nRelational expressions are always of the form:\n e.g. personId >= 2000.\n\nIndexes are the name given by the --out-ds argument in the 'nomdex up' command.\nConstants are either \"strings\" (in quotes) or numbers (e.g. 3, 3000, -2, -2.5,\n3.147, etc).\n\nRelational expressions can be combined using the \"and\" and \"or\" operators.\nParentheses can (and should) be used to ensure that the evaluation is done in\nthe desired order.\n`\n\nvar dbPath = \"\"\nvar query = \"\"\n\nfunc registerFind() {\n\tcmd := kingpin.Command(\"find\", \"Search an index\")\n\tcmd.Flag(\"db\", \"Database containing index\").Required().StringVar(&dbPath)\n\tcmd.Arg(\"query\", \"query to evalute\").Required().StringVar(&query)\n\toutputpager.RegisterOutputpagerFlags(cmd)\n}\n\nfunc runFind() int {\n\tcfg := config.NewResolver()\n\tdb, err := cfg.GetDatabase(dbPath)\n\tif printError(err, \"Unable to open database\\n\\terror: \") {\n\t\treturn 1\n\t}\n\tdefer db.Close()\n\n\tim := &indexManager{db: db, indexes: map[string]types.Map{}}\n\texpr, err := parseQuery(query, im)\n\tif err != nil {\n\t\tfmt.Printf(\"err: %s\\n\", err)\n\t\treturn 1\n\t}\n\n\tpgr := outputpager.Start()\n\tdefer pgr.Stop()\n\n\titer := expr.iterator(im)\n\tcnt := 0\n\tif iter != nil {\n\t\tfor v := iter.Next(); v != nil; v = iter.Next() {\n\t\t\ttypes.WriteEncodedValue(pgr.Writer, v)\n\t\t\tfmt.Fprintf(pgr.Writer, \"\\n\")\n\t\t\tcnt++\n\t\t}\n\t}\n\tfmt.Fprintf(pgr.Writer, \"Found %d objects\\n\", cnt)\n\n\treturn 0\n}\n\nfunc printObjects(w io.Writer, index types.Map, ranges queryRangeSlice) {\n\tcnt := 0\n\tfirst := true\n\tprintObjectForRange := func(index types.Map, r queryRange) {\n\t\tindex.IterFrom(r.lower.value, func(k, v types.Value) bool {\n\t\t\tif first && r.lower.value != nil && !r.lower.include && r.lower.value.Equals(k) {\n\t\t\t\treturn false\n\t\t\t}\n\t\t\tif r.upper.value != nil {\n\t\t\t\tif !r.upper.include && r.upper.value.Equals(k) {\n\t\t\t\t\treturn true\n\t\t\t\t}\n\t\t\t\tif r.upper.value.Less(k) {\n\t\t\t\t\treturn true\n\t\t\t\t}\n\t\t\t}\n\t\t\ts := v.(types.Set)\n\t\t\ts.IterAll(func(v types.Value) {\n\t\t\t\ttypes.WriteEncodedValue(w, v)\n\t\t\t\tfmt.Fprintf(w, \"\\n\")\n\t\t\t\tcnt++\n\t\t\t})\n\t\t\treturn false\n\t\t})\n\t}\n\tfor _, r := range ranges {\n\t\tprintObjectForRange(index, r)\n\t}\n\tfmt.Fprintf(w, \"Found %d objects\\n\", cnt)\n}\n\nfunc openIndex(idxName string, im *indexManager) error {\n\tif _, hasIndex := im.indexes[idxName]; hasIndex {\n\t\treturn nil\n\t}\n\n\tds := im.db.GetDataset(idxName)\n\tcommit, ok := ds.MaybeHead()\n\tif !ok {\n\t\treturn fmt.Errorf(\"index '%s' not found\", idxName)\n\t}\n\n\tindex, ok := commit.Get(datas.ValueField).(types.Map)\n\tif !ok {\n\t\treturn fmt.Errorf(\"Value of commit at '%s' is not a valid index\", idxName)\n\t}\n\n\t// Todo: make this type be Map, Set> once Issue #2326 gets resolved and\n\t// IsSubtype() returns the correct value.\n\ttyp := types.MakeMapType(\n\t\ttypes.MakeUnionType(types.StringType, types.NumberType),\n\t\ttypes.ValueType)\n\n\tif !types.IsValueSubtypeOf(index, typ) {\n\t\treturn fmt.Errorf(\"%s does not point to a suitable index type:\", idxName)\n\t}\n\n\tim.indexes[idxName] = index\n\treturn nil\n}\n" }, { "path": "samples/go/nomdex/nomdex_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"regexp\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/marshal\"\n\t\"github.com/attic-labs/noms/go/nbs\"\n\t\"github.com/attic-labs/noms/go/spec\"\n\t\"github.com/attic-labs/noms/go/util/clienttest\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\ntype TestObj struct {\n\tKey int\n\tFname string\n\tLname string\n\tGender string\n\tAge int\n}\n\ntype testSuite struct {\n\tclienttest.ClientTestSuite\n}\n\nfunc TestNomdex(t *testing.T) {\n\tsuite.Run(t, &testSuite{})\n}\n\nfunc makeTestDb(s *testSuite, dsId string) datas.Database {\n\tdb := datas.NewDatabase(nbs.NewLocalStore(s.DBDir, clienttest.DefaultMemTableSize))\n\tl1 := []TestObj{\n\t\t{1, \"will\", \"smith\", \"m\", 40},\n\t\t{2, \"lana\", \"turner\", \"f\", 91},\n\t\t{3, \"john\", \"wayne\", \"m\", 86},\n\t\t{4, \"johnny\", \"depp\", \"m\", 50},\n\t\t{5, \"merrill\", \"streep\", \"f\", 60},\n\t\t{6, \"rob\", \"courdry\", \"m\", 45},\n\t\t{7, \"bruce\", \"lee\", \"m\", 72},\n\t\t{8, \"bruce\", \"willis\", \"m\", 36},\n\t\t{9, \"luis\", \"bunuel\", \"m\", 100},\n\t\t{10, \"andy\", \"sandberg\", \"m\", 32},\n\t\t{11, \"walter\", \"coggins\", \"m\", 28},\n\t\t{12, \"seth\", \"rogan\", \"m\", 29},\n\t}\n\n\tm1 := map[string]TestObj{\n\t\t\"lg\": {13, \"lady\", \"gaga\", \"f\", 39},\n\t\t\"ss\": {14, \"sam\", \"smith\", \"m\", 28},\n\t\t\"rp\": {15, \"robert\", \"plant\", \"m\", 69},\n\t\t\"ml\": {16, \"meat\", \"loaf\", \"m\", 65},\n\t\t\"gf\": {17, \"glenn\", \"frey\", \"m\", 60},\n\t\t\"jr\": {18, \"joey\", \"ramone\", \"m\", 55},\n\t\t\"rc\": {19, \"ray\", \"charles\", \"m\", 72},\n\t\t\"bk\": {20, \"bb\", \"king\", \"m\", 77},\n\t\t\"b\": {21, \"beck\", \"\", \"m\", 38},\n\t\t\"md\": {22, \"miles\", \"davis\", \"m\", 82},\n\t\t\"rd\": {23, \"roger\", \"daltry\", \"m\", 62},\n\t\t\"jf\": {24, \"john\", \"fogerty\", \"m\", 60},\n\t}\n\n\tm := map[string]interface{}{\"actors\": l1, \"musicians\": m1}\n\tv, err := marshal.Marshal(db, m)\n\ts.NoError(err)\n\t_, err = db.CommitValue(db.GetDataset(dsId), v)\n\ts.NoError(err)\n\treturn db\n}\n\nfunc (s *testSuite) TestNomdex() {\n\tdsId := \"data\"\n\tdb := makeTestDb(s, dsId)\n\ts.NotNil(db)\n\tdb.Close()\n\n\tfnameIdx := \"fname-idx\"\n\tdataSpec := spec.CreateValueSpecString(\"nbs\", s.DBDir, dsId)\n\tdbSpec := spec.CreateDatabaseSpecString(\"nbs\", s.DBDir)\n\tstdout, stderr := s.MustRun(main, []string{\"up\", \"--out-ds\", fnameIdx, \"--in-path\", dataSpec, \"--by\", \".fname\"})\n\ts.Contains(stdout, \"Indexed 24 objects\")\n\ts.Equal(\"\", stderr)\n\n\tgenderIdx := \"gender-idx\"\n\tstdout, stderr = s.MustRun(main, []string{\"up\", \"--out-ds\", genderIdx, \"--in-path\", dataSpec, \"--by\", \".gender\"})\n\ts.Contains(stdout, \"Indexed 24 objects\")\n\ts.Equal(\"\", stderr)\n\n\tstdout, stderr = s.MustRun(main, []string{\"find\", \"--db\", dbSpec, `fname-idx = \"lady\"`})\n\ts.Contains(stdout, \"Found 1 objects\")\n\ts.Equal(\"\", stderr)\n\n\tstdout, stderr = s.MustRun(main, []string{\"find\", \"--db\", dbSpec, `fname-idx = \"lady\" and gender-idx = \"f\"`})\n\ts.Contains(stdout, \"Found 1 objects\")\n\ts.Equal(\"\", stderr)\n\n\tstdout, stderr = s.MustRun(main, []string{\"find\", \"--db\", dbSpec, `fname-idx != \"lady\" and gender-idx != \"m\"`})\n\ts.Contains(stdout, \"Found 2 objects\")\n\ts.Equal(\"\", stderr)\n\n\tstdout, stderr = s.MustRun(main, []string{\"find\", \"--db\", dbSpec, `fname-idx != \"lady\" and fname-idx != \"john\"`})\n\ts.Contains(stdout, \"Found 21 objects\")\n\ts.Equal(\"\", stderr)\n\n\tstdout, stderr = s.MustRun(main, []string{\"find\", \"--db\", dbSpec, `fname-idx != \"lady\" or gender-idx != \"f\"`})\n\ts.Contains(stdout, \"Found 23 objects\")\n\ts.Equal(\"\", stderr)\n}\n\nfunc TestTransform(t *testing.T) {\n\tassert := assert.New(t)\n\n\ttcs := [][]string{\n\t\t[]string{`\"01/02/2003\"`, \"\\\"(\\\\d{2})/(\\\\d{2})/(\\\\d{4})\\\"\", \"$3/$2/$1\", \"2003/02/01\"},\n\t}\n\n\tfor _, tc := range tcs {\n\t\tbase, regex, replace, expected := tc[0], tc[1], tc[2], tc[3]\n\n\t\ttestRe := regexp.MustCompile(regex)\n\t\tresult := testRe.ReplaceAllString(base, replace)\n\t\tassert.Equal(expected, result)\n\t}\n\n\ttcs = [][]string{\n\t\t[]string{\"343 STATE ST\\nROCHESTER, NY 14650\\n(43.161276, -77.619386)\", \"43.161276\", \"-77.619386\"},\n\t\t[]string{\"TWO EMBARCADERO CENTER\\nPROMENADE LEVEL SAN FRANCISCO, CA 94111\\n\", \"\", \"\"},\n\t}\n\n\tfindLatRe := regexp.MustCompile(\"(?s)\\\\(([\\\\d.]+)\")\n\tfindLngRe := regexp.MustCompile(\"(?s)(-?[\\\\d.]+)\\\\)\")\n\tfor _, tc := range tcs {\n\t\tbase, expectedLat, expectedLng := tc[0], tc[1], tc[2]\n\n\t\tlat := findLatRe.FindStringSubmatch(base)\n\t\tassert.True(len(lat) == 0 && expectedLat == \"\" || (len(lat) == 2 && expectedLat == lat[1]))\n\n\t\tlng := findLngRe.FindStringSubmatch(base)\n\t\tassert.True(len(lng) == 0 && expectedLng == \"\" || (len(lng) == 2 && expectedLng == lng[1]))\n\t}\n}\n" }, { "path": "samples/go/nomdex/nomdex_update.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"fmt\"\n\t\"regexp\"\n\t\"strconv\"\n\t\"sync\"\n\t\"sync/atomic\"\n\n\t\"github.com/attic-labs/kingpin\"\n\thumanize \"github.com/dustin/go-humanize\"\n\n\t\"github.com/attic-labs/noms/go/config\"\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/hash\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/attic-labs/noms/go/util/profile\"\n\t\"github.com/attic-labs/noms/go/util/status\"\n)\n\nvar (\n\tinPathArg = \"\"\n\toutDsArg = \"\"\n\trelPathArg = \"\"\n\ttxRegexArg = \"\"\n\ttxReplaceArg = \"\"\n\ttxConvertArg = \"\"\n)\n\nvar longUpHelp = `'nomdex up' builds indexes that are useful for rapidly accessing objects.\n\nThis sample tool can index objects based on any string or number attribute of that\nobject. The 'up' command works by scanning all the objects reachable from the --in-path\ncommand line argument. It tests the object to determine if there is a string or number\nvalue reachable by applying the --by path argument to the object. If so, the object is\nadded to the index under that value.\n\nFor example, if there are objects in the database that contain a personId and a\ngender field, 'nomdex up' can scan all the objects in a given dataset and build\nan index on the specified field with the following commands:\n nomdex up --in-path .value --by .gender --out-ds gender-index\n nomdex up --in-path .value --by .address.city --out-ds personId-index\n\nThe previous commands can be understood as follows. The first command updates or\nbuilds an index by scanning all the objects that are reachable from |in-path| that\nhave a string or number value reachable using |by| and stores the root of the\nresulting index in a dataset specified by |out-ds|.\n\nNotice that the --in-path argument has a value of '.value'. The '.value'\nis not strictly necessary but it's normally useful when indexing. Since datasets\ngenerally point to Commit objects in Noms, they usually have parents which are\nprevious versions of the data. If you add .value to the end of the dataset, only\nthe most recent version of the data will be indexed. Without the '.value' all\nobjects in all previous commits will also be indexed which is most often not what\nis expected.\n\nThere are three additional commands that can be useful for transforming the value\nbeing indexed:\n * tx-replace: used to modify behavior of tx-regex, see below\n * tx-regex: the behavior for this argument depends on whether a tx-replace argument\n is present. If so, the go routine \"regexp.ReplaceAllString() is called:\n txRe := regex.MustCompile(|tx-regex|)\n txRe.ReplaceAllString(|index value|, |tx-replace|\n If tx-replace is not present then the following call is made on each value:\n txRe := regex.MustCompile(|tx-regex|)\n regex.FindStringSubmatch(|index value|)\n *tx-convert: attempts to convert the index value to the type specified.\n Currently the only value accepted for this arg is 'number'\n\nThe resulting indexes can be used by the 'nomdex find command' for help on that\nsee: nomdex find -h\n`\n\nfunc registerUpdate() {\n\tcmd := kingpin.Command(\"up\", \"Build/update an index.\")\n\tcmd.Flag(\"in-path\", \"a value to search for items to index within\").Required().StringVar(&inPathArg)\n\tcmd.Flag(\"out-ds\", \"name of dataset to save the results to\").Required().StringVar(&outDsArg)\n\tcmd.Flag(\"by\", \"a path relative to all the items in to index by\").Required().StringVar(&relPathArg)\n\tcmd.Flag(\"tx-regex\", \"perform a string transformation on value before putting it in index\").StringVar(&txRegexArg)\n\tcmd.Flag(\"tx-replace\", \"replace values matched by tx-regex\").StringVar(&txReplaceArg)\n\tcmd.Flag(\"tx-convert\", \"convert the result of a tx regex/replace to this type (only does 'number' currently)\").StringVar(&txConvertArg)\n}\n\ntype StreamingSetEntry struct {\n\tvalChan chan<- types.Value\n\tsetChan <-chan types.Set\n}\n\ntype IndexMap map[types.Value]StreamingSetEntry\n\ntype Index struct {\n\tm IndexMap\n\tindexedCnt int64\n\tseenCnt int64\n\tmutex sync.Mutex\n}\n\nfunc runUpdate() int {\n\tdefer profile.MaybeStartProfile().Stop()\n\n\tcfg := config.NewResolver()\n\tdb, rootObject, err := cfg.GetPath(inPathArg)\n\td.Chk.NoError(err)\n\n\tif rootObject == nil {\n\t\tfmt.Printf(\"Object not found: %s\\n\", inPathArg)\n\t\treturn 1\n\t}\n\n\toutDs := db.GetDataset(outDsArg)\n\trelPath, err := types.ParsePath(relPathArg)\n\tif printError(err, \"Error parsing -by value\\n\\t\") {\n\t\treturn 1\n\t}\n\n\tgb := types.NewGraphBuilder(db, types.MapKind)\n\taddElementsToGraphBuilder(gb, db, rootObject, relPath)\n\tindexMap := gb.Build().(types.Map)\n\n\toutDs, err = db.Commit(outDs, indexMap, datas.CommitOptions{})\n\td.Chk.NoError(err)\n\tfmt.Printf(\"Committed index with %d entries to dataset: %s\\n\", indexMap.Len(), outDsArg)\n\n\treturn 0\n}\n\nfunc addElementsToGraphBuilder(gb *types.GraphBuilder, db datas.Database, rootObject types.Value, relPath types.Path) {\n\ttypeCacheMutex := sync.Mutex{}\n\ttypeCache := map[hash.Hash]bool{}\n\n\tvar txRe *regexp.Regexp\n\tif txRegexArg != \"\" {\n\t\tvar err error\n\t\ttxRe, err = regexp.Compile(txRegexArg)\n\t\td.CheckError(err)\n\t}\n\n\tindex := Index{m: IndexMap{}}\n\ttypes.WalkValues(rootObject, db, func(v types.Value) bool {\n\t\ttyp := types.TypeOf(v)\n\t\ttypeCacheMutex.Lock()\n\t\thasPath, ok := typeCache[typ.Hash()]\n\t\ttypeCacheMutex.Unlock()\n\t\tif !ok || hasPath {\n\t\t\tpathResolved := false\n\t\t\ttv := relPath.Resolve(v, db)\n\t\t\tif tv != nil {\n\t\t\t\tindex.addToGraphBuilder(gb, tv, v, txRe)\n\t\t\t\tpathResolved = true\n\t\t\t}\n\t\t\tif !ok {\n\t\t\t\ttypeCacheMutex.Lock()\n\t\t\t\ttypeCache[typ.Hash()] = pathResolved\n\t\t\t\ttypeCacheMutex.Unlock()\n\t\t\t}\n\t\t}\n\t\treturn false\n\t})\n\n\tstatus.Done()\n}\n\nfunc (idx *Index) addToGraphBuilder(gb *types.GraphBuilder, k, v types.Value, txRe *regexp.Regexp) {\n\tatomic.AddInt64(&idx.seenCnt, 1)\n\tif txRe != nil {\n\t\tk1 := types.EncodedValue(k)\n\t\tk2 := \"\"\n\t\tif txReplaceArg != \"\" {\n\t\t\tk2 = txRe.ReplaceAllString(string(k1), txReplaceArg)\n\t\t} else {\n\t\t\tmatches := txRe.FindStringSubmatch(string(k1))\n\t\t\tif len(matches) > 0 {\n\t\t\t\tk2 = matches[len(matches)-1]\n\t\t\t}\n\t\t}\n\t\tif txConvertArg == \"number\" {\n\t\t\tif k2 == \"\" {\n\t\t\t\treturn\n\t\t\t}\n\t\t\tn, err := strconv.ParseFloat(k2, 64)\n\t\t\tif err != nil {\n\t\t\t\tfmt.Println(\"error converting to number: \", err)\n\t\t\t\treturn\n\t\t\t}\n\t\t\tk = types.Number(n)\n\t\t} else {\n\t\t\tk = types.String(k2)\n\t\t}\n\t}\n\tatomic.AddInt64(&idx.indexedCnt, 1)\n\tgb.SetInsert(types.ValueSlice{k}, v)\n\tstatus.Printf(\"Found %s objects, Indexed %s objects\", humanize.Comma(idx.seenCnt), humanize.Comma(idx.indexedCnt))\n}\n" }, { "path": "samples/go/nomdex/parser.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"fmt\"\n\t\"strconv\"\n\t\"strings\"\n\t\"text/scanner\"\n\t\"unicode\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\n/**** Query language BNF\n query := expr\n expr := expr boolOp compExpr | group\n compExpr := indexToken compOp value\n group := '(' expr ')' | compExpr\n boolOp := 'and' | 'or'\n compOp := '=' | '<' | '<=' | '>' | '>=' | !=\n value := \"\" | number\n number := '-' digits | digits\n digits := int | float\n\n*/\n\ntype compOp string\ntype boolOp string\n\ntype indexManager struct {\n\tdb datas.Database\n\tindexes map[string]types.Map\n}\n\nconst (\n\tequals compOp = \"=\"\n\tgt compOp = \">\"\n\tgte compOp = \">=\"\n\tlt compOp = \"<\"\n\tlte compOp = \"<=\"\n\tne compOp = \"!=\"\n\topenP = \"(\"\n\tcloseP = \")\"\n\tand boolOp = \"and\"\n\tor boolOp = \"or\"\n)\n\nvar (\n\tcompOps = []compOp{equals, gt, gte, lt, lte, ne}\n\tboolOps = []boolOp{and, or}\n)\n\ntype qScanner struct {\n\ts scanner.Scanner\n\tpeekedToken rune\n\tpeekedText string\n\tpeeked bool\n}\n\nfunc (qs *qScanner) Scan() rune {\n\tvar r rune\n\tif qs.peeked {\n\t\tr = qs.peekedToken\n\t\tqs.peeked = false\n\t} else {\n\t\tr = qs.s.Scan()\n\t}\n\treturn r\n}\n\nfunc (qs *qScanner) Peek() rune {\n\tvar r rune\n\n\tif !qs.peeked {\n\t\tqs.peekedToken = qs.s.Scan()\n\t\tqs.peekedText = qs.s.TokenText()\n\t\tqs.peeked = true\n\t}\n\tr = qs.peekedToken\n\treturn r\n}\n\nfunc (qs *qScanner) TokenText() string {\n\tvar text string\n\tif qs.peeked {\n\t\ttext = qs.peekedText\n\t} else {\n\t\ttext = qs.s.TokenText()\n\t}\n\treturn text\n}\n\nfunc (qs *qScanner) Pos() scanner.Position {\n\treturn qs.s.Pos()\n}\n\nfunc parseQuery(q string, im *indexManager) (expr, error) {\n\ts := NewQueryScanner(q)\n\tvar expr expr\n\terr := d.Try(func() {\n\t\texpr = s.parseExpr(0, im)\n\t})\n\treturn expr, err\n}\n\nfunc NewQueryScanner(query string) *qScanner {\n\tisIdentRune := func(r rune, i int) bool {\n\t\tidentChars := \":/.>=-\"\n\t\tstartIdentChars := \"!><\"\n\t\tif i == 0 {\n\t\t\treturn unicode.IsLetter(r) || strings.ContainsRune(startIdentChars, r)\n\t\t}\n\t\treturn unicode.IsLetter(r) || unicode.IsDigit(r) || strings.ContainsRune(identChars, r)\n\t}\n\n\terrorFunc := func(s *scanner.Scanner, msg string) {\n\t\td.PanicIfError(fmt.Errorf(\"%s, pos: %s\\n\", msg, s.Pos()))\n\t}\n\n\tvar s scanner.Scanner\n\ts.Mode = scanner.ScanIdents | scanner.ScanFloats | scanner.ScanStrings | scanner.SkipComments\n\ts.Init(strings.NewReader(query))\n\ts.IsIdentRune = isIdentRune\n\ts.Error = errorFunc\n\tqs := qScanner{s: s}\n\treturn &qs\n}\n\nfunc (qs *qScanner) parseExpr(level int, im *indexManager) expr {\n\ttok := qs.Scan()\n\tswitch tok {\n\tcase '(':\n\t\texpr1 := qs.parseExpr(level+1, im)\n\t\ttok := qs.Scan()\n\t\tif tok != ')' {\n\t\t\td.PanicIfError(fmt.Errorf(\"missing ending paren for expr\"))\n\t\t} else {\n\t\t\ttok = qs.Peek()\n\t\t\tif tok == ')' {\n\t\t\t\treturn expr1\n\t\t\t}\n\t\t\ttok = qs.Scan()\n\t\t\ttext := qs.TokenText()\n\t\t\tswitch {\n\t\t\tcase tok == scanner.Ident && isBoolOp(text):\n\t\t\t\top := boolOp(text)\n\t\t\t\texpr2 := qs.parseExpr(level+1, im)\n\t\t\t\treturn logExpr{op: op, expr1: expr1, expr2: expr2, idxName: idxNameIfSame(expr1, expr2)}\n\t\t\tcase tok == scanner.EOF:\n\t\t\t\treturn expr1\n\t\t\tdefault:\n\t\t\t\td.PanicIfError(fmt.Errorf(\"extra text found at end of expr, tok: %d, text: %s\", int(tok), qs.TokenText()))\n\t\t\t}\n\t\t}\n\tcase scanner.Ident:\n\t\terr := openIndex(qs.TokenText(), im)\n\t\td.PanicIfError(err)\n\t\texpr1 := qs.parseCompExpr(level+1, qs.TokenText(), im)\n\t\ttok := qs.Peek()\n\t\tswitch tok {\n\t\tcase ')':\n\t\t\treturn expr1\n\t\tcase rune(scanner.Ident):\n\t\t\t_ = qs.Scan()\n\t\t\ttext := qs.TokenText()\n\t\t\tif isBoolOp(text) {\n\t\t\t\top := boolOp(text)\n\t\t\t\texpr2 := qs.parseExpr(level+1, im)\n\t\t\t\treturn logExpr{op: op, expr1: expr1, expr2: expr2, idxName: idxNameIfSame(expr1, expr2)}\n\t\t\t} else {\n\t\t\t\td.PanicIfError(fmt.Errorf(\"expected boolean op, found: %s, level: %d\", text, level))\n\t\t\t}\n\t\tcase rune(scanner.EOF):\n\t\t\treturn expr1\n\t\tdefault:\n\t\t\t_ = qs.Scan()\n\t\t}\n\tdefault:\n\t\td.PanicIfError(fmt.Errorf(\"unexpected token in expr: %s, %d\", qs.TokenText(), tok))\n\t}\n\treturn logExpr{}\n}\n\nfunc (qs *qScanner) parseCompExpr(level int, indexName string, im *indexManager) compExpr {\n\tqs.Scan()\n\ttext := qs.TokenText()\n\tif !isCompOp(text) {\n\t\td.PanicIfError(fmt.Errorf(\"expected relop token but found: '%s'\", text))\n\t}\n\top := compOp(text)\n\tvalue := qs.parseValExpr()\n\treturn compExpr{indexName, op, value}\n}\n\nfunc (qs *qScanner) parseValExpr() types.Value {\n\ttok := qs.Scan()\n\ttext := qs.TokenText()\n\tisNeg := false\n\tif tok == '-' {\n\t\tisNeg = true\n\t\ttok = qs.Scan()\n\t\ttext = qs.TokenText()\n\t}\n\tswitch tok {\n\tcase scanner.String:\n\t\tif isNeg {\n\t\t\td.PanicIfError(fmt.Errorf(\"expected number after '-', found string: %s\", text))\n\t\t}\n\t\treturn valueFromString(text)\n\tcase scanner.Float:\n\t\tf, _ := strconv.ParseFloat(text, 64)\n\t\tif isNeg {\n\t\t\tf = -f\n\t\t}\n\t\treturn types.Number(f)\n\tcase scanner.Int:\n\t\ti, _ := strconv.ParseInt(text, 10, 64)\n\t\tif isNeg {\n\t\t\ti = -i\n\t\t}\n\t\treturn types.Number(i)\n\t}\n\td.PanicIfError(fmt.Errorf(\"expected value token, found: '%s'\", text))\n\treturn nil // for compiler\n}\n\nfunc valueFromString(t string) types.Value {\n\tl := len(t)\n\tif l < 2 && t[0] == '\"' && t[l-1] == '\"' {\n\t\td.PanicIfError(fmt.Errorf(\"Unable to get value from token: %s\", t))\n\t}\n\treturn types.String(t[1 : l-1])\n}\n\nfunc isCompOp(s string) bool {\n\tfor _, op := range compOps {\n\t\tif s == string(op) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc isBoolOp(s string) bool {\n\tfor _, op := range boolOps {\n\t\tif s == string(op) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc idxNameIfSame(expr1, expr2 expr) string {\n\tif expr1.indexName() == expr2.indexName() {\n\t\treturn expr1.indexName()\n\t}\n\treturn \"\"\n}\n" }, { "path": "samples/go/nomdex/parser_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"testing\"\n\t\"text/scanner\"\n\n\t\"github.com/attic-labs/noms/go/chunks\"\n\t\"github.com/attic-labs/noms/go/datas\"\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\ntype scannerResult struct {\n\ttok int\n\ttext string\n}\n\ntype parseResult struct {\n\tquery string\n\tex expr\n}\n\nfunc TestQueryScanner(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts := NewQueryScanner(`9 (99.9) -9 0x7F \"99.9\" and or http://localhost:8000/cli-tour::yo <= >= < > = _ !=`)\n\n\tscannerResults := []scannerResult{\n\t\t{tok: scanner.Int, text: \"9\"},\n\t\t{tok: int('('), text: \"(\"},\n\t\t{tok: scanner.Float, text: \"99.9\"},\n\t\t{tok: int(')'), text: \")\"},\n\t\t{tok: '-', text: \"-\"},\n\t\t{tok: scanner.Int, text: \"9\"},\n\t\t{tok: scanner.Int, text: \"0x7F\"},\n\t\t{tok: scanner.String, text: `\"99.9\"`},\n\t\t{tok: scanner.Ident, text: \"and\"},\n\t\t{tok: scanner.Ident, text: \"or\"},\n\t\t{tok: scanner.Ident, text: \"http://localhost:8000/cli-tour::yo\"},\n\t\t{tok: scanner.Ident, text: \"<=\"},\n\t\t{tok: scanner.Ident, text: \">=\"},\n\t\t{tok: scanner.Ident, text: \"<\"},\n\t\t{tok: scanner.Ident, text: \">\"},\n\t\t{tok: int('='), text: \"=\"},\n\t\t{tok: int('_'), text: \"_\"},\n\t\t{tok: scanner.Ident, text: \"!=\"},\n\t}\n\n\tfor _, sr := range scannerResults {\n\t\ttok := s.Scan()\n\t\tassert.Equal(sr.tok, int(tok), \"expected text: %s, found: %s, pos: %s\", sr.text, s.TokenText(), s.Pos())\n\t\tassert.Equal(sr.text, s.TokenText())\n\t}\n\ttok := s.Scan()\n\tassert.Equal(scanner.EOF, int(tok))\n}\n\nfunc TestPeek(t *testing.T) {\n\tassert := assert.New(t)\n\n\ts := NewQueryScanner(`_ < \"one\"`)\n\tscannerResults := []scannerResult{\n\t\t{tok: int('_'), text: \"_\"},\n\t\t{tok: scanner.Ident, text: \"<\"},\n\t\t{tok: scanner.String, text: `\"one\"`},\n\t\t{tok: scanner.EOF, text: \"\"},\n\t}\n\n\tfor _, sr := range scannerResults {\n\t\tassert.Equal(sr.tok, int(s.Peek()))\n\t\tassert.Equal(sr.text, s.TokenText())\n\t\tassert.Equal(sr.tok, int(s.Scan()))\n\t\tassert.Equal(sr.text, s.TokenText())\n\t}\n}\n\nfunc TestParsing(t *testing.T) {\n\tassert := assert.New(t)\n\n\tre1 := compExpr{\"index1\", equals, types.Number(2015)}\n\tre2 := compExpr{\"index1\", gte, types.Number(2020)}\n\tre3 := compExpr{\"index1\", lte, types.Number(2022)}\n\tre4 := compExpr{\"index1\", lt, types.Number(-2030)}\n\tre5 := compExpr{\"index1\", ne, types.Number(3.5)}\n\tre6 := compExpr{\"index1\", ne, types.Number(-3500.4536632)}\n\tre7 := compExpr{\"index1\", ne, types.String(\"whassup\")}\n\n\tqueries := []parseResult{\n\t\t{`index1 = 2015`, re1},\n\t\t{`(index1 = 2015 )`, re1},\n\t\t{`(((index1 = 2015 ) ))`, re1},\n\t\t{`index1 = 2015 or index1 >= 2020`, logExpr{or, re1, re2, \"index1\"}},\n\t\t{`(index1 = 2015) or index1 >= 2020`, logExpr{or, re1, re2, \"index1\"}},\n\t\t{`index1 = 2015 or (index1 >= 2020)`, logExpr{or, re1, re2, \"index1\"}},\n\t\t{`(index1 = 2015 or index1 >= 2020)`, logExpr{or, re1, re2, \"index1\"}},\n\t\t{`(index1 = 2015 or index1 >= 2020) and index1 <= 2022`, logExpr{and, logExpr{or, re1, re2, \"index1\"}, re3, \"index1\"}},\n\t\t{`index1 = 2015 or index1 >= 2020 and index1 <= 2022`, logExpr{or, re1, logExpr{and, re2, re3, \"index1\"}, \"index1\"}},\n\t\t{`index1 = 2015 or index1 >= 2020 and index1 <= 2022 or index1 < -2030`, logExpr{or, re1, logExpr{and, re2, logExpr{or, re3, re4, \"index1\"}, \"index1\"}, \"index1\"}},\n\t\t{`(index1 = 2015 or index1 >= 2020) and (index1 <= 2022 or index1 < -2030)`, logExpr{and, logExpr{or, re1, re2, \"index1\"}, logExpr{or, re3, re4, \"index1\"}, \"index1\"}},\n\t\t{`index1 != 3.5`, re5},\n\t\t{`index1 != -3500.4536632`, re6},\n\t\t{`index1 != \"whassup\"`, re7},\n\t}\n\n\tstorage := &chunks.MemoryStorage{}\n\tdb := datas.NewDatabase(storage.NewView())\n\t_, err := db.CommitValue(db.GetDataset(\"index1\"), types.NewMap(db, types.String(\"one\"), types.NewSet(db, types.String(\"two\"))))\n\tassert.NoError(err)\n\n\tim := &indexManager{db: db, indexes: map[string]types.Map{}}\n\tfor _, pr := range queries {\n\t\texpr, err := parseQuery(pr.query, im)\n\t\tassert.NoError(err)\n\t\tassert.Equal(pr.ex, expr, \"bad query: %s\", pr.query)\n\t}\n\n\tbadQueries := []string{\n\t\t`sdfsd = 2015`,\n\t\t`index1 = \"unfinished string`,\n\t\t`index1 and 2015`,\n\t\t`index1 < `,\n\t\t`index1 < 2015 and ()`,\n\t\t`index1 < 2015 an index1 > 2016`,\n\t\t`(index1 < 2015) what`,\n\t\t`(index1< 2015`,\n\t\t`(badIndexName < 2015)`,\n\t}\n\n\tim1 := &indexManager{db: db, indexes: map[string]types.Map{}}\n\tfor _, q := range badQueries {\n\t\texpr, err := parseQuery(q, im1)\n\t\tassert.Error(err)\n\t\tassert.Nil(expr)\n\t}\n}\n" }, { "path": "samples/go/nomdex/query_range.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"sort\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n)\n\ntype bound struct {\n\tvalue types.Value\n\tinclude bool\n\tinfinity int8\n}\n\nfunc (b bound) isLessThanOrEqual(o bound) (res bool) {\n\treturn b.equals(o) || b.isLessThan(o)\n}\n\nfunc (b bound) isLessThan(o bound) (res bool) {\n\tif b.infinity < o.infinity {\n\t\treturn true\n\t}\n\n\tif b.infinity > o.infinity {\n\t\treturn false\n\t}\n\n\tif b.infinity == o.infinity && b.infinity != 0 {\n\t\treturn false\n\t}\n\n\tif b.value.Less(o.value) {\n\t\treturn true\n\t}\n\n\tif b.value.Equals(o.value) {\n\t\tif !b.include && o.include {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc (b bound) isGreaterThanOrEqual(o bound) (res bool) {\n\treturn !b.isLessThan(o)\n}\n\nfunc (b bound) isGreaterThan(o bound) (res bool) {\n\treturn !b.equals(o) || !b.isLessThan(o)\n}\n\nfunc (b bound) equals(o bound) bool {\n\treturn b.infinity == o.infinity && b.include == o.include &&\n\t\t(b.value == nil && o.value == nil || (b.value != nil && o.value != nil && b.value.Equals(o.value)))\n}\n\nfunc (b bound) String() string {\n\tvar s1 string\n\tif b.value == nil {\n\t\ts1 = \"\"\n\t} else {\n\t\tbuf := bytes.Buffer{}\n\t\ttypes.WriteEncodedValue(&buf, b.value)\n\t\ts1 = buf.String()\n\t}\n\treturn fmt.Sprintf(\"bound{v: %s, include: %t, infinity: %d}\", s1, b.include, b.infinity)\n}\n\nfunc (b bound) minValue(o bound) (res bound) {\n\tif b.isLessThan(o) {\n\t\treturn b\n\t}\n\treturn o\n}\n\nfunc (b bound) maxValue(o bound) (res bound) {\n\tif b.isLessThan(o) {\n\t\treturn o\n\t}\n\treturn b\n}\n\ntype queryRange struct {\n\tlower bound\n\tupper bound\n}\n\nfunc (r queryRange) and(o queryRange) (rangeDescs queryRangeSlice) {\n\tif !r.intersects(o) {\n\t\treturn []queryRange{}\n\t}\n\n\tlower := r.lower.maxValue(o.lower)\n\tupper := r.upper.minValue(o.upper)\n\treturn []queryRange{{lower, upper}}\n}\n\nfunc (r queryRange) or(o queryRange) (rSlice queryRangeSlice) {\n\tif r.intersects(o) {\n\t\tv1 := r.lower.minValue(o.lower)\n\t\tv2 := r.upper.maxValue(o.upper)\n\t\treturn queryRangeSlice{queryRange{v1, v2}}\n\t}\n\trSlice = queryRangeSlice{r, o}\n\tsort.Sort(rSlice)\n\treturn rSlice\n}\n\nfunc (r queryRange) intersects(o queryRange) (res bool) {\n\tif r.lower.isGreaterThanOrEqual(o.lower) && r.lower.isLessThanOrEqual(o.upper) {\n\t\treturn true\n\t}\n\tif r.upper.isGreaterThanOrEqual(o.lower) && r.upper.isLessThanOrEqual(o.upper) {\n\t\treturn true\n\t}\n\tif o.lower.isGreaterThanOrEqual(r.lower) && o.lower.isLessThanOrEqual(r.upper) {\n\t\treturn true\n\t}\n\tif o.upper.isGreaterThanOrEqual(r.lower) && o.upper.isLessThanOrEqual(r.upper) {\n\t\treturn true\n\t}\n\treturn false\n}\n\nfunc (r queryRange) String() string {\n\treturn fmt.Sprintf(\"queryRange{lower: %s, upper: %s\", r.lower, r.upper)\n}\n\n// queryRangeSlice defines the sort.Interface. This implementation sorts queryRanges by\n// the lower bound in ascending order.\ntype queryRangeSlice []queryRange\n\nfunc (rSlice queryRangeSlice) Len() int {\n\treturn len(rSlice)\n}\n\nfunc (rSlice queryRangeSlice) Swap(i, j int) {\n\trSlice[i], rSlice[j] = rSlice[j], rSlice[i]\n}\n\nfunc (rSlice queryRangeSlice) Less(i, j int) bool {\n\treturn !rSlice[i].lower.equals(rSlice[j].lower) && rSlice[i].lower.isLessThanOrEqual(rSlice[j].lower)\n}\n\nfunc (rSlice queryRangeSlice) dbgPrint(w io.Writer) {\n\tfor i, rd := range rSlice {\n\t\tif i == 0 {\n\t\t\tfmt.Fprintf(w, \"\\n#################\\n\")\n\t\t}\n\t\tfmt.Fprintf(w, \"queryRange %d: %s\\n\", i, rd)\n\t}\n\tif len(rSlice) > 0 {\n\t\tfmt.Fprintf(w, \"\\n\")\n\t}\n}\n" }, { "path": "samples/go/nomdex/query_range_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/types\"\n\t\"github.com/stretchr/testify/assert\"\n)\n\nconst nilHolder = -1000000\n\nvar (\n\tr1 = qr(2, true, 5, true)\n\tr2 = qr(0, true, 8, true)\n\tr3 = qr(0, true, 3, true)\n\tr4 = qr(3, true, 8, true)\n\tr5 = qr(0, true, 1, true)\n\tr6 = qr(6, true, 10, true)\n\tr7 = qr(nilHolder, true, 10, true)\n\tr8 = qr(3, true, nilHolder, true)\n\tr10 = qr(2, true, 5, false)\n\tr11 = qr(5, true, 10, true)\n)\n\nfunc newBound(i int, include bool, infinity int) bound {\n\tvar v types.Value\n\tif i != nilHolder {\n\t\tv = types.Number(i)\n\t}\n\treturn bound{value: v, include: include, infinity: int8(infinity)}\n}\n\nfunc qr(lower int, lowerIncl bool, upper int, upperIncl bool) queryRange {\n\tlowerInf := 0\n\tif lower == nilHolder {\n\t\tlowerInf = -1\n\t}\n\tupperInf := 0\n\tif upper == nilHolder {\n\t\tupperInf = 1\n\t}\n\treturn queryRange{newBound(lower, lowerIncl, lowerInf), newBound(upper, upperIncl, upperInf)}\n}\n\nfunc TestRangeIntersects(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassert.True(r1.intersects(r2))\n\tassert.True(r1.intersects(r3))\n\tassert.True(r1.intersects(r4))\n\tassert.True(r2.intersects(r1))\n\tassert.True(r1.intersects(r7))\n\tassert.True(r1.intersects(r8))\n\tassert.True(r3.intersects(r4))\n\tassert.True(r3.intersects(r4))\n\n\tassert.False(r1.intersects(r5))\n\tassert.False(r1.intersects(r6))\n\tassert.False(r10.intersects(r11))\n}\n\nfunc TestRangeAnd(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassert.Empty(r1.and(r5))\n\tassert.Empty(r1.and(r6))\n\n\tassert.Equal(r1, r1.and(r2)[0])\n\tassert.Equal(r1, r2.and(r1)[0])\n\n\texpected := qr(3, true, 5, true)\n\tassert.Equal(expected, r1.and(r4)[0])\n}\n\nfunc TestRangeOr(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassert.Equal(r2, r1.or(r2)[0])\n\n\texpected := qr(0, true, 5, true)\n\tassert.Equal(expected, r1.or(r3)[0])\n\n\texpectedSlice := queryRangeSlice{r5, r1}\n\tassert.Equal(expectedSlice, r1.or(r5))\n\tassert.Equal(expectedSlice, r5.or(r1))\n}\n\nfunc TestIsLessThan(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassert.True(newBound(1, true, 0).isLessThanOrEqual(newBound(2, true, 0)))\n\tassert.False(newBound(2, true, 0).isLessThanOrEqual(newBound(1, true, 0)))\n\tassert.True(newBound(1, true, 0).isLessThanOrEqual(newBound(1, true, 0)))\n\n\tassert.True(newBound(1, false, 0).isLessThanOrEqual(newBound(2, false, 0)))\n\tassert.False(newBound(2, false, 0).isLessThanOrEqual(newBound(1, false, 0)))\n\tassert.True(newBound(1, false, 0).isLessThanOrEqual(newBound(1, false, 0)))\n\n\tassert.False(newBound(1, true, 0).isLessThanOrEqual(newBound(1, false, 0)))\n\tassert.True(newBound(1, false, 0).isLessThanOrEqual(newBound(1, true, 0)))\n\n\tassert.True(newBound(nilHolder, true, -1).isLessThanOrEqual(newBound(1, true, 0)))\n\tassert.False(newBound(1, false, 0).isLessThanOrEqual(newBound(nilHolder, true, -1)))\n}\n\nfunc TestIsGreaterThan(t *testing.T) {\n\tassert := assert.New(t)\n\n\tassert.True(newBound(2, true, 0).isGreaterThanOrEqual(newBound(1, true, 0)))\n\tassert.False(newBound(1, true, 0).isGreaterThanOrEqual(newBound(2, true, 0)))\n\tassert.True(newBound(1, true, 0).isGreaterThanOrEqual(newBound(1, true, 0)))\n\n\tassert.True(newBound(2, false, 0).isGreaterThanOrEqual(newBound(1, false, 0)))\n\tassert.False(newBound(1, false, 0).isGreaterThanOrEqual(newBound(2, false, 0)))\n\tassert.True(newBound(1, false, 0).isGreaterThanOrEqual(newBound(1, false, 0)))\n\n\tassert.True(newBound(1, true, 0).isGreaterThanOrEqual(newBound(1, false, 0)))\n\tassert.False(newBound(1, false, 0).isGreaterThanOrEqual(newBound(2, true, 0)))\n\n\tassert.True(newBound(nilHolder, true, 1).isGreaterThanOrEqual(newBound(1, true, 0)))\n\tassert.False(newBound(1, true, 0).isGreaterThanOrEqual(newBound(nilHolder, true, 1)))\n}\n\nfunc TestMinValue(t *testing.T) {\n\tassert := assert.New(t)\n\tve1 := newBound(5, false, 0)\n\tve2 := newBound(5, true, 0)\n\tve3 := newBound(nilHolder, true, -1)\n\tve4 := newBound(nilHolder, true, 1)\n\n\tassert.Equal(ve1, ve1.minValue(ve2))\n\tassert.Equal(ve3, ve1.minValue(ve3))\n\tassert.Equal(ve1, ve1.minValue(ve4))\n}\n\nfunc TestMaxValue(t *testing.T) {\n\tassert := assert.New(t)\n\tve1 := newBound(5, false, 0)\n\tve2 := newBound(5, true, 0)\n\tve3 := newBound(nilHolder, true, -1)\n\tve4 := newBound(nilHolder, true, 1)\n\n\tassert.Equal(ve2, ve1.maxValue(ve2))\n\tassert.Equal(ve1, ve1.maxValue(ve3))\n\tassert.Equal(ve4, ve1.maxValue(ve4))\n}\n" }, { "path": "tools/file/file.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage file\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"runtime\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// DumbCopy copies the contents of a regular file at srcPath (following symlinks) to a new regular file at dstPath. New file is created with same mode.\nfunc DumbCopy(srcPath, dstPath string) {\n\tchkClose := func(c io.Closer) { d.PanicIfError(c.Close()) }\n\n\tinfo, err := os.Stat(srcPath)\n\td.PanicIfError(err)\n\n\tif info.IsDir() {\n\t\td.PanicIfError(ErrNoCopyDir)\n\t}\n\n\tsrc, err := os.Open(srcPath)\n\td.PanicIfError(err)\n\tdefer chkClose(src)\n\n\tdst, err := os.OpenFile(dstPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, info.Mode())\n\td.PanicIfError(err)\n\tdefer chkClose(dst)\n\t_, err = io.Copy(dst, src)\n\td.PanicIfError(err)\n}\n\n// MyDir returns the directory in which the file containing the calling source code resides.\nfunc MyDir() string {\n\t_, path, _, ok := runtime.Caller(1)\n\tif !ok {\n\t\td.Panic(\"Should have been able to get Caller.\")\n\t}\n\treturn filepath.Dir(path)\n}\n\nvar ErrNoCopyDir = fmt.Errorf(\"attempted to copy a directory\")\n" }, { "path": "tools/file/file_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage file\n\nimport (\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"testing\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst (\n\tcontents = \"hey\"\n)\n\nfunc TestSerialRunnerTestSuite(t *testing.T) {\n\tsuite.Run(t, &FileTestSuite{})\n}\n\ntype FileTestSuite struct {\n\tsuite.Suite\n\tdir, src, exc string\n}\n\nfunc (suite *FileTestSuite) SetupTest() {\n\tvar err error\n\tsuite.dir, err = ioutil.TempDir(os.TempDir(), \"\")\n\tsuite.NoError(err)\n\tsuite.src = filepath.Join(suite.dir, \"srcfile\")\n\tsuite.exc = filepath.Join(suite.dir, \"excfile\")\n\tsuite.NoError(ioutil.WriteFile(suite.src, []byte(contents), 0644))\n\tsuite.NoError(ioutil.WriteFile(suite.exc, []byte(contents), 0755))\n}\n\nfunc (suite *FileTestSuite) TearDownTest() {\n\tos.Remove(suite.dir)\n}\n\nfunc (suite *FileTestSuite) TestCopyFile() {\n\tdst := filepath.Join(suite.dir, \"dstfile\")\n\n\ttest := func(src string, mode int) {\n\t\tDumbCopy(src, dst)\n\n\t\tinfo, err := os.Stat(src)\n\t\tsuite.NoError(err)\n\t\tsuite.Equal(mode, int(info.Mode()))\n\n\t\tout, err := ioutil.ReadFile(dst)\n\t\tsuite.NoError(err)\n\t\tsuite.Equal(contents, string(out))\n\t}\n\n\ttest(suite.src, 0644)\n\ttest(suite.exc, 0755)\n}\n\nfunc (suite *FileTestSuite) TestCopyLink() {\n\tlink := filepath.Join(suite.dir, \"link\")\n\tsuite.NoError(os.Symlink(suite.src, link))\n\n\tdst := filepath.Join(suite.dir, \"dstfile\")\n\tDumbCopy(link, dst)\n\n\tinfo, err := os.Lstat(dst)\n\tsuite.NoError(err)\n\tsuite.True(info.Mode().IsRegular())\n\n\tout, err := ioutil.ReadFile(dst)\n\tsuite.NoError(err)\n\tsuite.Equal(contents, string(out))\n}\n\nfunc (suite *FileTestSuite) TestNoCopyDir() {\n\tdir, err := ioutil.TempDir(suite.dir, \"\")\n\tsuite.NoError(err)\n\n\tdst := filepath.Join(suite.dir, \"dst\")\n\tsuite.Error(d.Try(func() { DumbCopy(dir, dst) }))\n}\n" }, { "path": "tools/licensify.py", "content": "#!/usr/bin/python\n\n# Copyright 2016 Attic Labs, Inc. All rights reserved.\n# Licensed under the Apache License, version 2.0:\n# http://www.apache.org/licenses/LICENSE-2.0\n\n# This script ensures our license header is present on all the files git knows\n# about within the current directory.\n#\n# It is safe to re-run this file on already-processed files.\n\nimport os\nimport re\nimport subprocess\nimport shutil\nimport tempfile\n\nlicenseRows = [\n 'Copyright 2016 Attic Labs, Inc. All rights reserved.',\n 'Licensed under the Apache License, version 2.0:',\n 'http://www.apache.org/licenses/LICENSE-2.0',\n]\n\ncomment_markers = {\n 'go': ('', '// ', ''),\n 'js': ('', '// ', ''),\n 'py': ('', '# ', ''),\n 'html': (''),\n 'css': ('/**', ' * ', ' */'),\n}\n\n\ndef main():\n files = subprocess.check_output(['git', 'ls-files']).split('\\n')\n for n in files:\n if n != '' and not n.startswith('vendor/') and (\n not n.endswith('.min.js')):\n _, ext = os.path.splitext(n)\n if ext == '':\n continue\n ext = ext[1:]\n pattern = buildLicensePattern(ext)\n if pattern != None:\n with open(n, 'r+') as f:\n processFile(f, ext, pattern)\n\n\ndef processFile(f, ext, pattern):\n '''Updates the license block in file |f|.'''\n content = f.read()\n f.seek(0)\n f.truncate()\n replacement = re.sub(pattern, getLicense(ext), content)\n f.write(replacement)\n\n\ndef buildLicensePattern(ext):\n '''\n Builds a regex pattern that matches license blocks in files with extension\n |ext|.\n '''\n markers = comment_markers.get(ext)\n if markers is None:\n return None\n\n (first, mark, last) = [re.escape(m) for m in markers]\n\n prefix = ''\n # The first line must include the copyright string to avoid picking up\n # random other comment blocks at head of file.\n head = mark + r'Copyright \\d+ (The Noms Authors|Attic Labs).*\\n'\n body = '(' + mark + r'.*\\n)*'\n suffix = ''\n\n if first != '':\n prefix = first + r'\\n'\n if last != '':\n suffix = last + r'\\n'\n\n # We want to make sure shebang files stay at head of file.\n shebang = r'(?P\\#\\!.+\\n+|)'\n\n # Allow flow annotations\n flow = r'(?P// @flow\\n+|)'\n\n # Doctype header\n doctype = r'(?P<\\!doctype html>\\n+|)'\n\n return ('^' + shebang + doctype +\n '(' + prefix + head + body + suffix + r'\\n)?' + flow)\n\n\ndef getLicense(ext):\n '''Gets the license block for files with extension |ext|.'''\n (first, mark, last) = comment_markers[ext]\n result = '\\n'.join([mark + line for line in licenseRows])\n if first != '':\n result = first + '\\n' + result\n if last != '':\n result = result + '\\n' + last\n return r'\\g' + r'\\g' + result + '\\n\\n\\g'\n\n\nif __name__ == '__main__':\n main()\n" }, { "path": "tools/loadtest/loadtest.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage main\n\nimport (\n\t\"bytes\"\n\t\"encoding/binary\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"math/rand\"\n\t\"net\"\n\t\"os\"\n\t\"os/exec\"\n\t\"strings\"\n\t\"time\"\n)\n\n// This script runs random noms commands against random datasets on a database.\n//\n// Example usage:\n// > go run path/to/loadtest.go http://demo.noms.io/cli-tour\n//\n// Imports should be Go builtin libraries only, so that this can be run with \"go run\".\n\ntype runnerFn func(db, ds string)\n\ntype runner struct {\n\tname string\n\tfn runnerFn\n}\n\nfunc main() {\n\trand.Seed(time.Now().UnixNano() + bestEffortGetIP())\n\n\tif len(os.Args) != 2 {\n\t\tfmt.Println(\"Usage: loadtest \")\n\t\tos.Exit(-1)\n\t}\n\n\tdb := os.Args[1]\n\n\trs := []runner{\n\t\t{\"diff\", runDiff},\n\t\t{\"log diff\", runLogDiff},\n\t\t{\"log show\", runLogShow},\n\t\t{\"show\", runShow},\n\t\t{\"sync\", runSync},\n\t}\n\n\tfor ds := range streamDs(db) {\n\t\tstart := time.Now()\n\t\tr := rs[rand.Intn(len(rs))]\n\t\tfmt.Println(time.Now().Format(time.Stamp), r.name, db, ds)\n\t\tr.fn(db, fmt.Sprintf(\"%s::%s\", db, ds))\n\t\tfmt.Println(\" took\", time.Since(start).String())\n\t}\n}\n\nfunc bestEffortGetIP() (asNum int64) {\n\taddrs, err := net.InterfaceAddrs()\n\tif err != nil {\n\t\treturn\n\t}\n\n\tfor _, a := range addrs {\n\t\tif ipnet, ok := a.(*net.IPNet); ok && !ipnet.IP.IsLoopback() {\n\t\t\tif ipnet.IP.To4() != nil {\n\t\t\t\tasNum = int64(binary.BigEndian.Uint32([]byte(ipnet.IP.To4())))\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}\n\treturn\n}\n\nfunc runDiff(db, ds string) {\n\tif parent := getParent(db, ds); parent != \"\" {\n\t\tcall(nil, \"noms\", \"diff\", ds, parent)\n\t} else {\n\t\tfmt.Println(\" (no parent, cannot diff)\")\n\t}\n}\n\nfunc runLogDiff(db, ds string) {\n\tcall(nil, \"noms\", \"log\", ds)\n}\n\nfunc runLogShow(db, ds string) {\n\tcall(nil, \"noms\", \"log\", \"--show-value\", ds)\n}\n\nfunc runShow(db, ds string) {\n\tif strings.HasSuffix(ds, \"/raw\") {\n\t\tfmt.Println(\" (skipping raw file, blobs are too slow)\")\n\t} else {\n\t\tcall(nil, \"noms\", \"show\", ds)\n\t}\n}\n\nfunc runSync(db, ds string) {\n\tdir, err := ioutil.TempDir(\"\", \"loadtest\")\n\tif err != nil {\n\t\tfmt.Fprintln(os.Stderr, \" ERROR: failed to create temp directory:\", err.Error())\n\t\treturn\n\t}\n\n\tdefer os.RemoveAll(dir)\n\t// Try to sync to parent, then from parent to head.\n\t// If there isn't a parent then just sync head.\n\tsyncDs := fmt.Sprintf(\"ldb:%s::sync\", dir)\n\tif parent := getParent(db, ds); parent != \"\" {\n\t\tcall(nil, \"noms\", \"sync\", parent, syncDs)\n\t}\n\tcall(nil, \"noms\", \"sync\", ds, syncDs)\n}\n\nfunc getParent(db, ds string) string {\n\tbuf := &bytes.Buffer{}\n\tcall(buf, \"noms\", \"log\", \"-n\", \"2\", \"--oneline\", ds)\n\t// Output will look like:\n\t// abc (Parent def)\n\t// def (Parent None)\n\t// We could use the first line and grab the Parent value from there, but it could also be Merge,\n\t// and it might be None, so easier to just get the 2nd row.\n\tlines := strings.SplitN(buf.String(), \"\\n\", 2)\n\tif len(lines) != 2 {\n\t\treturn \"\"\n\t}\n\thsh := strings.SplitN(lines[0], \" \", 2)[0]\n\treturn fmt.Sprintf(\"%s::#%s\", db, hsh)\n}\n\nfunc call(stdout io.Writer, name string, arg ...string) error {\n\tcmd := exec.Command(name, arg...)\n\tfmt.Println(\" >\", name, strings.Join(arg, \" \"))\n\tcmd.Stdout = stdout\n\tcmd.Stderr = os.Stderr\n\terr := cmd.Run()\n\tif err != nil {\n\t\tfmt.Fprintf(os.Stderr, \" ERROR: %s\\n\", err.Error())\n\t}\n\treturn err\n}\n\nfunc streamDs(db string) <-chan string {\n\tbuf := &bytes.Buffer{}\n\terr := call(buf, \"noms\", \"ds\", db)\n\tif err != nil {\n\t\tfmt.Fprintln(os.Stderr, \" ERROR: failed to get datasets\")\n\t\tos.Exit(-1)\n\t}\n\n\tout := strings.Trim(buf.String(), \" \\n\")\n\tif out == \"\" {\n\t\tfmt.Fprintln(os.Stderr, \" ERROR: no datasets at\", db)\n\t\tos.Exit(-1)\n\t}\n\n\tdatasets := strings.Split(out, \"\\n\")\n\n\tch := make(chan string)\n\tgo func() {\n\t\tfor {\n\t\t\tch <- datasets[rand.Intn(len(datasets))]\n\t\t}\n\t}()\n\treturn ch\n}\n" }, { "path": "tools/noms/README.md", "content": "# Noms build script helpers\n\nThese are helper functions for writing your Noms app build and staging scripts.\n\n## Writing your scripts\n\n### Build scripts\nYour build script must be named *build.py*. It will be discovered by the system and executed in the directory in which it's found. It must require no arguments, though environment variables will propagate in.\n\n### Staging scripts\nAfter your build script gets run, we'll run your staging script -- the purpose of which is to take your build products and put them in a directory that's ready to be packaged and deployed somewhere. This script must be called *stage.py* and take as its sole argument the path to a directory where all project code is being staged.\n\n### Libraries\nWe have provided a library to make writing your staging scripts easier. Example usage:\n```python\n\t#!/usr/bin/python\n\n\timport noms.staging as staging\n\n\tif __name__ == '__main__':\n\t\tstaging.Main('nerdosphere', staging.GlobCopier('index.html', 'styles.css', '*.js'))\n```\nImporting and using `noms.staging` handles determining where you should stage your code and creating the necessary directories for you. You just pass it the name of your project and a function that knows how to stage your build artifacts, given a path under which to put everything.\n\n\n## Develop\n\n* To run unittests: `python -m unittest discover -p \"*_test.py\" -s $GOPATH/src/github.com/attic-labs/noms/tools`\n" }, { "path": "tools/noms/__init__.py", "content": "# Copyright 2016 Attic Labs, Inc. All rights reserved.\n# Licensed under the Apache License, version 2.0:\n# http://www.apache.org/licenses/LICENSE-2.0\n\n" }, { "path": "tools/noms/copy.py", "content": "#!/usr/bin/python\n\n# Copyright 2016 Attic Labs, Inc. All rights reserved.\n# Licensed under the Apache License, version 2.0:\n# http://www.apache.org/licenses/LICENSE-2.0\n\nimport os.path, shutil\n\ndef Peers(me, dstDir):\n \"\"\"Peers copies the peers of me into dstDir.\n\n Peers looks for files, directories and symlinks next to me\n and copies them (with the same basenames) to dstDir, which is\n presumed to exist.\n \"\"\"\n myDir = os.path.dirname(os.path.abspath(me))\n names = os.listdir(myDir)\n for basename in names:\n src = os.path.join(myDir, basename)\n dst = os.path.join(dstDir, basename)\n if os.path.samefile(me, src):\n continue\n\n if os.path.islink(src):\n linkto = os.readlink(src)\n os.symlink(linkto, dst)\n elif os.path.isfile(src):\n shutil.copy2(src, dst)\n elif os.path.isdir(src):\n shutil.copytree(src, dst)\n else:\n raise Exception(\"Unknown file type at \" + src)\n" }, { "path": "tools/noms/copy_test.py", "content": "#!/usr/bin/python\n\n# Copyright 2016 Attic Labs, Inc. All rights reserved.\n# Licensed under the Apache License, version 2.0:\n# http://www.apache.org/licenses/LICENSE-2.0\n\nimport os, os.path, shutil, tempfile, unittest\nimport copy\n\nclass TestCopy(unittest.TestCase):\n def setUp(self):\n self.tempdir = os.path.realpath(tempfile.mkdtemp())\n\n\n def tearDown(self):\n shutil.rmtree(self.tempdir, ignore_errors=True)\n\n\n def test_CopyPeers(self):\n nested = tempfile.mkdtemp(dir=self.tempdir)\n otherNested = tempfile.mkdtemp(dir=self.tempdir)\n\n def mkfile():\n with tempfile.NamedTemporaryFile(dir=nested, delete=False) as f:\n return f.name\n\n me = mkfile()\n peerFile = os.path.basename(mkfile())\n peerDir = os.path.basename(tempfile.mkdtemp(dir=nested))\n peerLink = 'link'\n peerLinkAbs = os.path.join(nested, 'link')\n os.symlink(peerFile, peerLinkAbs)\n\n copy.Peers(me, otherNested)\n self.assertTrue(os.path.islink(os.path.join(otherNested, peerLink)))\n self.assertTrue(os.path.isfile(os.path.join(otherNested, peerFile)))\n self.assertTrue(os.path.isdir(os.path.join(otherNested, peerDir)))\n self.assertFalse(os.path.lexists(os.path.join(otherNested, os.path.basename(me))))\n\n\nif __name__ == '__main__':\n unittest.main()\n" }, { "path": "tools/noms/pushd.py", "content": "#!/usr/bin/python\n\n# Copyright 2016 Attic Labs, Inc. All rights reserved.\n# Licensed under the Apache License, version 2.0:\n# http://www.apache.org/licenses/LICENSE-2.0\n\nimport os\nfrom contextlib import contextmanager\n\n@contextmanager\ndef pushd(path):\n currentDir = os.getcwd()\n os.chdir(path)\n yield\n os.chdir(currentDir)\n" }, { "path": "tools/noms/staging.py", "content": "#!/usr/bin/python\n\n# Copyright 2016 Attic Labs, Inc. All rights reserved.\n# Licensed under the Apache License, version 2.0:\n# http://www.apache.org/licenses/LICENSE-2.0\n\nimport argparse\nimport glob\nimport hashlib\nimport os\nimport os.path\nimport re\nimport shutil\n\ndef Main(projectName, stagingFunction):\n \"\"\"Main should be called by all staging scripts when executed.\n\n Main takes a project name and a callable. It creates a staging directory for\n your project and then runs the callable, passing it the path to the\n newly-created staging directory.\n\n For the common case of simply copying a set of files into the staging\n directory, use GlobCopier:\n\n #!/usr/bin/python\n\n import noms.staging as staging\n\n if __name__ == '__main__':\n staging.Main('nerdosphere', staging.GlobCopier('index.html', 'styles.css', '*.js'))\n \"\"\"\n parser = argparse.ArgumentParser(description='Stage build products from this directory.')\n parser.add_argument('staging_dir',\n metavar='path/to/staging/directory',\n type=_dir_path,\n help='top-level dir into which project build products are staged')\n args = parser.parse_args()\n project_staging_dir = os.path.join(args.staging_dir, projectName)\n\n normalized = os.path.realpath(project_staging_dir)\n if not _is_sub_dir(project_staging_dir, args.staging_dir):\n raise Exception(project_staging_dir + ' must be a subdir of ' + args.staging_dir)\n\n if not os.path.exists(normalized):\n os.makedirs(normalized)\n stagingFunction(normalized)\n\n\ndef run_globs(staging_dir, globs, exclude):\n for pattern in globs:\n for f in glob.glob(pattern):\n if os.path.isdir(f):\n continue\n from_dir, name = os.path.split(f)\n if name in exclude:\n continue\n to_dir = os.path.join(staging_dir, from_dir)\n if not os.path.exists(to_dir):\n os.makedirs(to_dir)\n yield (f, to_dir)\n\n\ndef rename_with_hash(f, to_dir, rename_dict):\n with open(f) as fh:\n sha = hashlib.sha256()\n sha.update(fh.read())\n digest = sha.hexdigest()\n basename = os.path.basename(f)\n name, ext = os.path.splitext(basename)\n new_name = '%s.%s%s' % (name, digest[:20], ext)\n rename_dict[basename] = new_name\n shutil.move(os.path.join(to_dir, basename), os.path.join(to_dir, new_name))\n\n\ndef GlobCopier(*globs, **kwargs):\n '''\n Returns a function that copies files defined by globs into a staging dir.\n\n Arguments:\n - Zero or more globs used to determine which files to copy.\n\n Keyword arguments:\n - rename (bool) - If True then the files gets renamed to name.%%hash.ext\n - index_file (str) - If present then this file is copied to the staging dir\n and its content is updated where the paths to the files are updated to the\n renamed file paths.\n '''\n exclude = ('webpack.config.js',)\n rename = 'rename' in kwargs and kwargs['rename']\n def stage(staging_dir):\n if rename:\n rename_dict = dict()\n for f, to_dir in run_globs(staging_dir, globs, exclude):\n shutil.copy2(f, to_dir)\n if rename:\n rename_with_hash(f, to_dir, rename_dict)\n\n # Update index_file and write it to to_dir.\n if 'index_file' not in kwargs:\n return\n index_file = kwargs['index_file']\n from_dir, name = os.path.split(index_file)\n to_dir = os.path.join(staging_dir, from_dir)\n with open(index_file, 'r') as f:\n data = f.read()\n if rename:\n for old_name, new_name in rename_dict.iteritems():\n r = re.compile(r'\\b%s\\b' % re.escape(old_name))\n data = r.sub(new_name, data)\n with open(os.path.join(to_dir, name), 'w') as f:\n f.write(data)\n\n return stage\n\n\ndef _dir_path(arg):\n normalized = os.path.realpath(os.path.abspath(arg))\n if os.path.exists(normalized) and not os.path.isdir(normalized):\n raise ValueError(arg + ' is not a path to a directory.')\n return normalized\n\n\ndef _is_sub_dir(subdir, directory):\n # Need the path-sep at the end to ensure that commonprefix returns the correct result below.\n directory = os.path.join(os.path.realpath(directory), '')\n subdir = os.path.realpath(subdir)\n\n # return true, if the common prefix of both is equal to directory e.g. /a/b/c/d.rst and\n # directory is /a/b, the common prefix is /a/b\n return os.path.commonprefix([subdir, directory]) == directory\n" }, { "path": "tools/noms/staging_test.py", "content": "#!/usr/bin/python\n\n# Copyright 2016 Attic Labs, Inc. All rights reserved.\n# Licensed under the Apache License, version 2.0:\n# http://www.apache.org/licenses/LICENSE-2.0\n\nimport os, os.path, shutil, tempfile, unittest\nimport staging\n\nclass TestStaging(unittest.TestCase):\n def setUp(self):\n self.tempdir = os.path.realpath(tempfile.mkdtemp())\n self.nested = tempfile.mkdtemp(dir=self.tempdir)\n\n def tearDown(self):\n shutil.rmtree(self.tempdir, ignore_errors=True)\n\n def test_Nested(self):\n self.assertTrue(staging._is_sub_dir(self.nested, self.tempdir))\n\n def test_NotNested(self):\n otherNested = tempfile.mkdtemp(dir=self.tempdir)\n self.assertFalse(staging._is_sub_dir(self.nested, otherNested))\n\n def test_DotDotNotReallyNested(self):\n notReallyNested = os.path.join(self.tempdir, 'foo', os.pardir, 'bar')\n self.assertFalse(staging._is_sub_dir(self.nested, notReallyNested))\n\n def test_LinkNotReallyNested(self):\n otherNested = tempfile.mkdtemp(dir=self.tempdir)\n linkName = os.path.join(self.nested, 'link')\n os.symlink(otherNested, linkName)\n self.assertFalse(staging._is_sub_dir(linkName, self.nested))\n\n def test_DirPath(self):\n linkName = os.path.join(self.tempdir, 'link')\n os.symlink(self.nested, linkName)\n norm = staging._dir_path(linkName)\n self.assertEqual(self.nested, norm)\n\n def test_DirPathFails(self):\n f = tempfile.NamedTemporaryFile(dir=self.tempdir)\n try:\n staging._dir_path(f.name)\n except ValueError:\n pass\n\n def test_GlobCopier(self):\n files = (\n 'a.js',\n 'b.js',\n 'c.html',\n 'd.css',\n 'webpack.config.js',\n\n 'x/aa.js',\n 'x/bb.js',\n 'x/dd.css',\n 'x/webpack.config.js',\n\n 'x/xx/aaa.js',\n 'x/xx/bbb.js',\n 'x/xx/webpack.config.js',\n\n 'x/yy/aaa.js',\n 'x/yy/bbb.js',\n 'x/yy/webpack.config.js',\n\n 'y/aaaa.js',\n 'y/bbbb.js',\n 'y/webpack.config.js',\n\n 'y/xxx/a5.js',\n 'y/xxx/b5.js',\n 'y/xxx/webpack.config.js',\n\n 'z/a6.js',\n 'z/b6.js',\n 'z/webpack.config.js',\n )\n for d in ('x/xx', 'x/yy', 'y/xxx', 'z'):\n os.makedirs(os.path.join(self.tempdir, d))\n for name in files:\n with open(os.path.join(self.tempdir, name), 'w') as f:\n f.write('hi')\n\n cwd = os.getcwd()\n try:\n os.chdir(self.tempdir)\n staging.GlobCopier('*.js', 'c.html', 'x/*.js', 'x/xx/*', 'y/*', 'y/*')(self.nested)\n finally:\n os.chdir(cwd)\n\n self.assertEqual(sorted(['a.js', 'b.js', 'c.html', 'x', 'y']),\n sorted(os.listdir(self.nested)))\n self.assertEqual(sorted(['aa.js', 'bb.js', 'xx']),\n sorted(os.listdir(os.path.join(self.nested, 'x'))))\n self.assertEqual(sorted(['aaa.js', 'bbb.js']),\n sorted(os.listdir(os.path.join(self.nested, 'x/xx'))))\n self.assertEqual(sorted(['aaaa.js', 'bbbb.js']),\n sorted(os.listdir(os.path.join(self.nested, 'y'))))\n\n def test_GlobCopierWithRename(self):\n files = (\n 'a.js',\n 'b.js',\n 'c.html',\n 'd.css',\n 'webpack.config.js',\n\n 'x/aa.js',\n 'x/bb.js',\n 'x/dd.css',\n 'x/webpack.config.js',\n\n 'x/xx/aaa.js',\n 'x/xx/bbb.js',\n 'x/xx/webpack.config.js',\n\n 'x/yy/aaa.js',\n 'x/yy/bbb.js',\n 'x/yy/webpack.config.js',\n\n 'y/aaaa.js',\n 'y/bbbb.js',\n 'y/webpack.config.js',\n\n 'y/xxx/a5.js',\n 'y/xxx/b5.js',\n 'y/xxx/webpack.config.js',\n\n 'z/a6.js',\n 'z/b6.js',\n 'z/webpack.config.js',\n )\n with open(os.path.join(self.tempdir, 'index.html'), 'w') as f:\n f.write('index.html')\n for d in ('x/xx', 'x/yy', 'y/xxx', 'z'):\n os.makedirs(os.path.join(self.tempdir, d))\n for name in files:\n with open(os.path.join(self.tempdir, name), 'w') as f:\n f.write('hi! name: ' + name)\n\n cwd = os.getcwd()\n try:\n os.chdir(self.tempdir)\n staging.GlobCopier(\n '*.js', 'c.html', 'x/*.js', 'x/xx/*', 'y/*', 'y/*',\n index_file='index.html',\n rename=True)(self.nested)\n finally:\n os.chdir(cwd)\n\n self.assertEqual(sorted(['a.702f720d2b49bd41c30f.js', 'b.49cf685c13e7de516ebc.js',\n 'c.fe1a3b03473494234e2d.html', 'index.html', 'x', 'y']),\n sorted(os.listdir(self.nested)))\n self.assertEqual(sorted(['aa.eb0f5ae6432d325f9448.js',\n 'bb.480969faecf03a9eb729.js', 'xx']),\n sorted(os.listdir(os.path.join(self.nested, 'x'))))\n self.assertEqual(sorted(['aaa.a9810946370699474422.js',\n 'bbb.c06f75d2d61cb6717b2c.js']),\n sorted(os.listdir(os.path.join(self.nested, 'x/xx'))))\n self.assertEqual(sorted(['aaaa.a68d3caf6e0e971ab96f.js',\n 'bbbb.84bd5947630aca231726.js']),\n sorted(os.listdir(os.path.join(self.nested, 'y'))))\n\nif __name__ == '__main__':\n unittest.main()\n" }, { "path": "tools/noms/symlink.py", "content": "#!/usr/bin/python\n\n# Copyright 2016 Attic Labs, Inc. All rights reserved.\n# Licensed under the Apache License, version 2.0:\n# http://www.apache.org/licenses/LICENSE-2.0\n\nimport os\n\nclass LinkError(Exception):\n \"\"\"Raised when forcing a symlink fails for a non-OS reason.\"\"\"\n pass\n\ndef Force(source, linkName):\n \"\"\"\n Force forces linkName to be a symlink to source, as long as its not a dir.\n Creates a symlink from linkName to source, clobbering linkName as long as its not a directory.\n \"\"\"\n if not os.path.lexists(linkName):\n os.symlink(source, linkName)\n return\n\n if os.path.islink(linkName) or os.path.isfile(linkName):\n os.remove(linkName)\n os.symlink(source, linkName)\n return\n\n raise LinkError(\"Refusing to clobber \" + linkName)\n" }, { "path": "tools/noms/symlink_test.py", "content": "#!/usr/bin/python\n\n# Copyright 2016 Attic Labs, Inc. All rights reserved.\n# Licensed under the Apache License, version 2.0:\n# http://www.apache.org/licenses/LICENSE-2.0\n\nimport os, os.path, shutil, tempfile, unittest\nimport symlink\n\nclass TestForceSymlink(unittest.TestCase):\n CONTENTS = 'test file contents'\n\n def setUp(self):\n self.tempdir = tempfile.mkdtemp()\n self.source = tempfile.NamedTemporaryFile(dir=self.tempdir, delete=False)\n with self.source.file as f:\n f.write(self.CONTENTS)\n\n\n def tearDown(self):\n shutil.rmtree(self.tempdir, ignore_errors=True)\n\n\n def verifySymlink(self, linkName):\n with open(linkName, 'r') as f:\n actual = f.read()\n self.assertEqual(self.CONTENTS, actual)\n\n\n def test_ClobberFile(self):\n linkName = os.path.join(self.tempdir, 'link')\n with open(linkName, 'w') as f:\n f.write('foo')\n\n symlink.Force(self.source.name, linkName)\n self.verifySymlink(linkName)\n\n\n def test_ClobberSymlink(self):\n linkName = os.path.join(self.tempdir, 'link')\n os.symlink('nowhere', linkName)\n\n symlink.Force(self.source.name, linkName)\n self.verifySymlink(linkName)\n\n\n def test_NoClobberDir(self):\n linkName = os.path.join(self.tempdir, 'link')\n os.mkdir(linkName, 0777)\n\n try:\n symlink.Force(self.source.name, linkName)\n except symlink.LinkError:\n pass\n\n\nif __name__ == '__main__':\n unittest.main()\n" }, { "path": "tools/runner/serial.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage runner\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n\t\"os/exec\"\n\t\"path/filepath\"\n\t\"runtime\"\n\n\t\"github.com/attic-labs/noms/go/d\"\n)\n\n// Env is a map of env vars, mapping key string to value string.\ntype Env map[string]string\n\nfunc (e Env) toStrings() (out []string) {\n\tout = os.Environ()\n\t// Sadly, it seems like we need to force-set GOROOT in the environment to handle some funky runtime environments (e.g. on our Travis setup)\n\tif e == nil {\n\t\te = Env{}\n\t}\n\n\tif _, overridden := e[\"GOROOT\"]; !overridden {\n\t\te[\"GOROOT\"] = runtime.GOROOT()\n\t}\n\tfor n, v := range e {\n\t\tout = append(out, fmt.Sprintf(\"%s=%s\", n, v))\n\t}\n\treturn\n}\n\n// ForceRun runs 'exe [args...]' in current working directory, and d.Chk()s on failure. Inherits the environment of the current process.\nfunc ForceRun(exe string, args ...string) {\n\terr := runEnvDir(os.Stdout, os.Stderr, Env{}, \"\", exe, args...)\n\td.Chk.NoError(err)\n}\n\n// ForceRunInDir runs 'exe [args...]' in the given directory, and d.Chk()s on failure. Inherits the environment of the current process.\nfunc ForceRunInDir(dir string, env Env, exe string, args ...string) {\n\tinfo, err := os.Stat(dir)\n\tif err != nil {\n\t\td.Panic(\"Can't stat %s\", dir)\n\t}\n\tif !info.IsDir() {\n\t\td.Panic(\"%s must be a path to a directory.\", dir)\n\t}\n\td.Chk.NoError(runEnvDir(os.Stdout, os.Stderr, env, dir, exe, args...))\n}\n\n// RunInDir runs 'exe [args...]' in the given directory, returning any failure. The child's stdout and stderr are mapped to out and err respectively. Inherits the environment of the current process.\nfunc RunInDir(out, err io.Writer, dir, exe string, args ...string) error {\n\treturn runEnvDir(out, err, Env{}, dir, exe, args...)\n}\n\n// runEnvDir 'exe [args...]' in dir with the environment env overlaid on that of the current process. If dir == \"\", use the current working directory.\nfunc runEnvDir(out, err io.Writer, env Env, dir, exe string, args ...string) error {\n\tcmd := exec.Command(exe, args...)\n\tcmd.Dir = dir\n\tcmd.Env = env.toStrings()\n\tcmd.Stdout = out\n\tcmd.Stderr = err\n\treturn cmd.Run()\n}\n\n// Serial serially runs all instances of filename found under dir, mapping stdout and stderr to each subprocess in the obvious way. env is overlaid on the environment of the current process. If args are provided, they're passed en masse to each subprocess.\nfunc Serial(stdout, stderr io.Writer, env Env, dir, filename string, args ...string) bool {\n\tsuccess := true\n\terr := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {\n\t\tif os.IsNotExist(err) {\n\t\t\t// Some programs like npm create temporary log files which confuse filepath.Walk.\n\t\t\treturn nil\n\t\t}\n\t\tif err != nil {\n\t\t\td.Panic(\"Failed directory traversal at %s\", path)\n\t\t}\n\t\tif !info.IsDir() && filepath.Base(path) == filename {\n\t\t\tscriptAndArgs := append([]string{filepath.Base(path)}, args...)\n\t\t\trunErr := runEnvDir(stdout, stderr, env, filepath.Dir(path), \"python\", scriptAndArgs...)\n\t\t\tif runErr != nil {\n\t\t\t\tsuccess = false\n\t\t\t\tfmt.Fprintf(stderr, \"Running %s failed with %v\\n\", path, runErr)\n\t\t\t}\n\t\t}\n\t\treturn nil\n\t})\n\td.PanicIfError(err)\n\treturn success\n}\n" }, { "path": "tools/runner/serial_test.go", "content": "// Copyright 2016 Attic Labs, Inc. All rights reserved.\n// Licensed under the Apache License, version 2.0:\n// http://www.apache.org/licenses/LICENSE-2.0\n\npackage runner\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"io\"\n\t\"io/ioutil\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"runtime\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/suite\"\n)\n\nconst (\n\tboilerplate = `\nfrom __future__ import print_function\nimport os, sys\n\n%s\n`\n\tbuildFileBasename = \"build.py\"\n)\n\nfunc TestSerialRunnerTestSuite(t *testing.T) {\n\tsuite.Run(t, &SerialRunnerTestSuite{})\n}\n\ntype SerialRunnerTestSuite struct {\n\tsuite.Suite\n\tdir string\n\tindex int\n}\n\nfunc (suite *SerialRunnerTestSuite) SetupTest() {\n\tvar err error\n\tsuite.dir, err = ioutil.TempDir(os.TempDir(), \"\")\n\tsuite.NoError(err)\n}\n\nfunc (suite *SerialRunnerTestSuite) TearDownTest() {\n\tos.Remove(suite.dir)\n}\n\nfunc (suite *SerialRunnerTestSuite) TestForceRunInDir() {\n\tscriptPath := filepath.Join(suite.dir, buildFileBasename)\n\tsuite.makeTestBuildFile(scriptPath, []string{\"print(os.getcwd(), file=sys.stdout)\"})\n\n\told := os.Stdout // keep backup of the real stdout\n\tr, w, err := os.Pipe()\n\tsuite.NoError(err)\n\tos.Stdout = w\n\tdefer func() { os.Stdout = old }()\n\tdefer r.Close()\n\n\toutC := make(chan string)\n\t// copy the output in a separate goroutine so printing can't block indefinitely\n\tgo func() {\n\t\tbuf := &bytes.Buffer{}\n\t\tio.Copy(buf, r)\n\t\toutC <- buf.String()\n\t}()\n\n\tForceRunInDir(suite.dir, nil, \"python\", scriptPath)\n\n\tw.Close()\n\tout := strings.TrimSpace(<-outC)\n\tactualSuiteDir, err := filepath.EvalSymlinks(suite.dir)\n\tsuite.NoError(err)\n\tsuite.Equal(actualSuiteDir, out)\n}\n\nfunc (suite *SerialRunnerTestSuite) TestRunInDir() {\n\tscriptPath := filepath.Join(suite.dir, buildFileBasename)\n\tsuite.makeTestBuildFile(scriptPath, []string{\n\t\t\"print(os.getcwd(), file=sys.stdout)\",\n\t\t\"print('error', file=sys.stderr)\",\n\t})\n\n\tstdout := &bytes.Buffer{}\n\tstderr := &bytes.Buffer{}\n\tRunInDir(stdout, stderr, suite.dir, \"python\", scriptPath)\n\tactualSuiteDir, err := filepath.EvalSymlinks(suite.dir)\n\tsuite.NoError(err)\n\tsuite.Equal(actualSuiteDir, strings.TrimSpace(string(stdout.Bytes())))\n\tsuite.Equal(\"error\", strings.TrimSpace(string(stderr.Bytes())))\n}\n\nfunc (suite *SerialRunnerTestSuite) TestEnvVars() {\n\tmakeEnvVarPrintBuildFile := func(path, varname string) {\n\t\tfmtStatement := fmt.Sprintf(`print(os.environ['%s'], file=sys.stdout)`, varname)\n\t\tsuite.makeTestBuildFile(path, []string{fmtStatement})\n\t}\n\n\ttype testCase struct {\n\t\tpath, varname, expected string\n\t}\n\tenv := Env{\n\t\t\"PATH\": os.Getenv(\"PATH\"),\n\t\t\"GOPATH\": os.Getenv(\"GOPATH\"),\n\t\t\"NOMS_CHECKOUT_PATH\": \"/where/noms/is\",\n\t\t\"ATTIC_CHECKOUT_PATH\": \"/where/attic/is\",\n\t}\n\ttests := []testCase{}\n\tfor n, v := range env {\n\t\ttc := testCase{suite.uniqueBuildFile(), n, v}\n\t\tmakeEnvVarPrintBuildFile(tc.path, tc.varname)\n\t\ttests = append(tests, tc)\n\t}\n\tgorootTestCase := testCase{suite.uniqueBuildFile(), \"GOROOT\", runtime.GOROOT()}\n\tmakeEnvVarPrintBuildFile(gorootTestCase.path, gorootTestCase.varname)\n\ttests = append(tests, gorootTestCase)\n\n\tlog := &bytes.Buffer{}\n\tif suite.True(Serial(log, log, env, suite.dir, buildFileBasename), \"Serial() should have succeeded! logs:\\n%s\", string(log.Bytes())) {\n\t\tlogText := string(log.Bytes())\n\t\tfor _, tc := range tests {\n\t\t\tsuite.Contains(logText, tc.expected)\n\t\t}\n\t}\n}\n\nfunc (suite *SerialRunnerTestSuite) TestFailure() {\n\ttype testCase struct {\n\t\tpath, expected string\n\t}\n\ttests := []testCase{\n\t\t{suite.uniqueBuildFile(), \"Scoobaz\"},\n\t\t{suite.uniqueBuildFile(), \"at the disco\"},\n\t}\n\tgoodOne := testCase{suite.uniqueBuildFile(), \"All's well\"}\n\n\tsuite.makeTestBuildFile(tests[0].path, []string{\"Scoobaz() # Won't compile.\"})\n\tsuite.makeTestBuildFile(tests[1].path, []string{`assert(False, \"at the disco\") # Won't run.`})\n\tsuite.makeTestBuildFile(goodOne.path, []string{fmt.Sprintf(`print \"%s\"`, goodOne.expected)})\n\n\tlog := &bytes.Buffer{}\n\tsuite.False(Serial(log, log, Env{}, suite.dir, buildFileBasename))\n\tlogText := string(log.Bytes())\n\tsuite.Contains(logText, tests[0].expected)\n\tsuite.Contains(logText, tests[1].expected)\n}\n\nfunc (suite *SerialRunnerTestSuite) uniqueBuildFile() string {\n\tsuite.index++\n\treturn filepath.Join(suite.dir, fmt.Sprintf(\"%d\", suite.index), buildFileBasename)\n}\n\nfunc (suite *SerialRunnerTestSuite) makeTestBuildFile(path string, statements []string) {\n\tbuf := &bytes.Buffer{}\n\tfmt.Fprintf(buf, boilerplate, strings.Join(statements, \"\\n\"))\n\terr := os.MkdirAll(filepath.Dir(path), 0777)\n\tsuite.NoError(err)\n\terr = ioutil.WriteFile(path, buf.Bytes(), 0755)\n\tsuite.NoError(err)\n}\n" } ]