[
  {
    "path": ".gitignore",
    "content": "*.[568]\n[568].out\n_obj\n_test*\na.out\n*.gz\n"
  },
  {
    "path": "Makefile",
    "content": "# figure out what GOROOT is supposed to be\nGOROOT ?= $(shell printf 't:;@echo $$(GOROOT)\\n' | gomake -f -)\ninclude $(GOROOT)/src/Make.inc\n\nTARG=noeqd\nGOFILES=\\\n\tmain.go\\\n\ninclude $(GOROOT)/src/Make.cmd\n\nVERSION=$(shell git describe --tags --always)\n\ntar: clean $(TARG)\n\ttar -czf $(TARG)-$(VERSION)-$(GOOS)-$(GOARCH).tar.gz $(TARG) README.md\n"
  },
  {
    "path": "README.md",
    "content": "# noeqd - A fault-tolerant network service for meaningful GUID generation\n\nBased on [snowflake][].\n\n## Motivation\n\nGUIDs (Globally Unique IDs) are useful for a number a obvious reasons:\ndatabase keys, logging, etc.\n\nGenerating GUIDs with pure randomness is not always ideal because it doesn't\ncluster well, produces terrible locality, and no insight as to when it was\ngenerated.\n\nThis network service should also have these properties (Differences from [snowflake][]):\n\n* easy distribution with *no dependencies* and little to *no setup*\n* dirt simple wire-protocol (trivial to implement clients without added dependencies and complexity)\n* low memory footprint (starts and stays around ~1MB)\n* zero configuration\n* reduced network IO when multiple keys are needed at once\n\n## Glossary of terms to follow\n\n* `GUID`: Globally Unique Identifier\n* `datacenter`: A facility used to house computer systems.\n* `worker`: A single `noeqd` process with a worker and datacenter ID combination unique to their cohort.\n* `datacenter-id`: An integer representing a particular datacenter.\n* `worker-id`: An integer representing a particular worker.\n* `machine-id`: The comination of `datacenter-id` and `worker-id`\n* `twepoch`: custom epoch (same as [snowflake][])\n\n## Important note:\n\nReliability, and guarantees depend on:\n\n**System clock depedency and skew protection:** - (From [snowflake][] README and slightly modified)\n\nYou should use NTP to keep your system clock accurate. Noeq protects from\nnon-monotonic clocks, i.e. clocks that run backwards. If your clock is running\nfast and NTP tells it to repeat a few milliseconds, Noeq will refuse to\ngenerate ids until a time that is after the last time we generated an id. Even\nbetter, run in a mode where ntp won't move the clock backwards. See\n<http://wiki.dovecot.org/TimeMovedBackwards#Time_synchronization> for tips on how\nto do this.\n\n**Avoiding the reuse of a worker-id + datacenter-id too quickly**\n\nIt's important to know that a newly born process has no way of tracking its\nprevious life and where it left of. This means time could have moved\nbackwards while it was dead.\n\nIt's important to **not** use the same worker-id + datacenter-id without\ntelling the new process when to start generating new IDs to avoid duplicates.\n\nIt is only safe to reuse the same worker-id + datacenter-id when you can\nguarantee the current time is greater than the time of death. You can use the\n`-t` option to specifiy this.\n\nYou may have up to 1024 machine ids. It's generally safe to not reuse them\nuntil you've reached this limit.\n\n## Install\n\nYou can install noeqd by downloading the binary\n[here](http://github.com/bmizerany/noeqd/downloads) and putting it in your\n`PATH`.\n\n*or*\n\nClone the repo and build with [Go](http://golang.org/doc/install.html) (Requires Go `0beb796b4ef8 weekly/weekly.2011-12-02` or later)\n\n\t\t$ git clone http://github.com/bmizerany/noeqd\n\t\t$ cd noeqd\n\t\t$ make install\n\n## Run\n\n\t\t$ noeqd -h\n\t\tUsage of noeqd:\n\t\t  -d=0: datacenter id\n\t\t  -l=\"0.0.0.0:4444\": the address to listen on\n\t\t  -w=0: worker id\n\n**Coordinating machine-ids**\n\nNoeq does not assume you're using any automated coordination because it isn't\nalways correct to assume this. Its easy to do without baking it in. Here is an\nexample script in the repo for doing so if you need it (using [Doozer][]):\n\n\t\t#!/bin/sh\n\t\t# usage: ./coord-exec.sh <datacenter-id>\n\n\t\tdid=$1\n\t\twid=0\n\n\t\t[ -z \"$did\" ] && did=0\n\n\t\t_set() {\n\t\t  printf 1 | doozer set /goflake/$did/$wid 0\n\t\t}\n\n\t\twhile ! _set\n\t\tdo wid=`expr $wid + 1`\n\t\tdone\n\n\t\texec noeqd -w $wid -d $did\n\n## The Why\n\n**Uniqness**\n\nWe must know that a GUID, once generated, has never and will never be generated\nagain (i.e. Globally Unique) in our system.\n\n**Performance**\n\nHeroku serves many 10's of thousands of requests a second. Each request can\nrequire multiple actions that need their own id. To be on the safe side, we\nwill require a minimum of 100k ids/sec (without network latency); possibly more\nin the very near future. See benchmarks near the end of this README.\n\n**Uncoordinated**\n\nWe need all `noeqd`s to be able to generate GUIDs without coordinating with\nother `noeqd` processes. Coordination requires more time complexity than if\nwe didn't require it and reduces the amount of GUIDs we can generate during\nthat time. It also affects the yield (the probability the service will complete\na request).\n\n**Direcly sortable by time (roughly)**\n\nNoeq (like [snowflake][]) will guarantee the GUIDs will be k-sorted within\na reasonable bound (10's of ms to no more than 1s). More on this in \"How it works.\"\n\nReferences:\n\n<http://ci.nii.ac.jp/naid/110002673556>\n\n<http://ci.nii.ac.jp/naid/110002673489>\n\n# The \"Why not snowflake?\"\n\nAt Heroku, we value services that are simple, as self-contained as possible,\nand use nothing more than they can reasonably get away with. The setup and\ndistribution of an application should be as quick and painless as possible.\nThis means ruthlessly eliminating as much baggage, waste, and other overhead as\npossible.\n\n# How it works\n\n## GUID generation and guarantees\n\nGUIDs are represented as 64bit integers and are composed of (as described by the [snowflake][] README):\n\n* time - 41 bits (millisecond precision with a custom epoch gives us 69 years)\n* configured machine id - 10 bits - gives us up to 1024 machines\n* sequence number - 12 bits - rolls over every 4096 per machine (with protection to avoid rollover in the same ms)\n\n## Sorting - Time Ordered\n\n*Strictly sorted*:\n\n* GUIDs generated in a single request by a worker will strictly sort.\n* GUIDs generated one second or longer apart, by more than one worker, will strictly sort.\n* GUIDs generated over multiple requests by the same worker, will strictly sort.\n\n*Roughly sorted*:\n\n* GUIDs generated by multiple workers within a second could roughly sort.\n\nAn example of roughly sorted:\n\nIf client A requests three GUIDs from worker A in one request, and client B\nrequests three GUIDs from worker B in another request, and both requests are\nprocessed within the same second, together they may sort like:\n\n\t\tGUID-A1\n\t\tGUID-A2\n\t\tGUID-B1\n\t\tGUID-B2\n\t\tGUID-A3\n\t\tGUID-B3\n\nNOTE: The A GUIDs will strictly sort, as will B's.\n\n## Clients\n\nClients implement a simple wire-protocol that is specified below. Implementing\na client in your favorite language is trivial and should require no\ndependencies.\n\n**Failure Recovery**\n\nEach client should keep a list of addresses of all known worker process (or\nuse DNS) so that if one fails, it can move to another. To recover\nfrom a lost connection, a client should randomly select another address from\nits list, or in the case of DNS: reconnect using the same address allowing DNS\nto choose the next IP.\n\nSee [noeq.go](http://github.com/bmizerany/noeq.go) for a working\nexample.\n\n## Protocol\n\n*Auth Request*\n\nIf the server has its `NOEQ_TOKEN` environment variable set to an non-empty string, the server will require authentication.\n\n\t\t------------------------\n\t\t|0|<len byte>|token ...|\n\t\t------------------------\n\nAn auth request starts with byte(0) and then a byte that is the length of the\nfollwing token, followed by the token string. (NOTE: If a client and server\nhang during authentication, it's probably because the token is the client sent\nis too short.)\n\n*Id Request*:\n\n\t\t-------\n\t\t|uint8|\n\t\t-------\n\nA request must contain only one byte. The value of the byte tells the\nserver how many ids to respond with. A client can request up to 255 (or max uint8)\nids per request.\n\n*Response*:\n\n\t\t-------------------------------------------------- ...\n\t\t|uint8|uint8|uint8|uint8|uint8|uint8|uint8|uint8|  ...\n\t\t-------------------------------------------------- ...\n\nEach id comes as a 64bit integer in network byte order. The\nnumber of 64bit integers returned is the `request-byte * 8`\n\n*Errors*:\n\nErrors are logged by the server to stdout. Clients will have their\nconnection closed to signal the need to try elsewhere until the server can\nrecover. This generally happens if the servers clock is running backwards.\n\n## Benchmarks (fwiw)\n\n**MacAir 3, OS X 10.7.2, 2.13 GHz Intel Core 2 Duo, 4 GB 1067 MHz DDR3)**\n\n\n**Id Generation *without encoding* or network latency**\n\nThis is the benchmark done by [snowflake] and reported in their README.\n\n\t\tBenchmarkIdGeneration\t675 ns/op\t# 1.481 million ids/s\n\t\t\n**Id Generation *with* encoding and *without* network latency**\n\nI find these benchmarks more realistic. The ids must be encoded so we want to\nknow how fast an id can be generated and encoded in order to hit the wire.\nBenchmarks including a network are left as an exercise for the reader because\nall networks vary.\n\nThese show that when a client can safely ask for more one than one id at a time,\nthey can reduce time to wire and the expensive read/write operations.\n\n\t\tBenchmarkServe01\t 1677 ns/op\t# 596303 ids/sec\n\t\tBenchmarkServe02\t 2352 ns/op\t# 850340 ids/sec\n\t\tBenchmarkServe03\t 3067 ns/op\t# 978155 ids/sec\n\t\tBenchmarkServe05\t 4436 ns/op\t# 1.127 million ids/sec\n\t\tBenchmarkServe08\t 6436 ns/op\t# 1.243 million ids/sec\n\t\tBenchmarkServe13\t10169 ns/op\t# 1.278 million ids/sec\n\t\tBenchmarkServe21\t16257 ns/op\t# 1.292 million ids/sec\n\t\tBenchmarkServe34\t25603 ns/op\t# 1.328 million ids/sec\n\t\tBenchmarkServe55\t39693 ns/op\t# 1.386 million ids/sec\n\n## Contributing\n\nThis is Github. You know the drill. Please make sure you keep your changes in a\nbranch other than `master` and in nice, clean, atomic commits. If you modify a\n`.go` file, please use `gofmt` with no parameters to format it; then hit the\npull-request button.\n\n## Issues\n\nThese are tracked in this repos Github [issues tracker](http://github.com/bmizerany/noeqd/issues).\n\n## See Also\n\nNoeq command line util:\n<http://github.com/bmizerany/noeq>\n\nNoeq.go for Go:\n<http://github.com/bmizerany/noeq.go>\n\n## Thank you\n\nI want to make sure I give the Snowflake team at Twitter as much credit as\npossible. The heart of this program is their doing.\n\n## LICENSE\n\nCopyright (C) 2011 by Blake Mizerany ([@bmizerany](http://twitter.com/bmizerany))\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE. \n\n[Doozer]: http://github.com/ha/doozerd\n[snowflake]: http://github.com/twitter/snowflake\n"
  },
  {
    "path": "bench.sh",
    "content": "#!/bin/sh\ngotest -run=\"no tests\" -bench=\".*\"\n"
  },
  {
    "path": "bench_test.go",
    "content": "package main\n\nimport (\n\t\"bytes\"\n\t\"testing\"\n)\n\ntype nopWriter struct{}\n\nfunc (nw *nopWriter) Write(b []byte) (n int, err error) {\n\treturn\n}\n\nfunc BenchmarkIdGeneration(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tnextId()\n\t}\n}\n\nfunc BenchmarkServe01(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tb.StopTimer()\n\t\ti, o := bytes.NewBuffer([]byte{1}), new(nopWriter)\n\t\tb.StartTimer()\n\t\tserve(i, o)\n\t}\n}\n\nfunc BenchmarkServe02(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tb.StopTimer()\n\t\ti, o := bytes.NewBuffer([]byte{2}), new(nopWriter)\n\t\tb.StartTimer()\n\t\tserve(i, o)\n\t}\n}\n\nfunc BenchmarkServe03(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tb.StopTimer()\n\t\ti, o := bytes.NewBuffer([]byte{3}), new(nopWriter)\n\t\tb.StartTimer()\n\t\tserve(i, o)\n\t}\n}\n\nfunc BenchmarkServe05(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tb.StopTimer()\n\t\ti, o := bytes.NewBuffer([]byte{5}), new(nopWriter)\n\t\tb.StartTimer()\n\t\tserve(i, o)\n\t}\n}\n\nfunc BenchmarkServe08(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tb.StopTimer()\n\t\ti, o := bytes.NewBuffer([]byte{8}), new(nopWriter)\n\t\tb.StartTimer()\n\t\tserve(i, o)\n\t}\n}\n\nfunc BenchmarkServe13(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tb.StopTimer()\n\t\ti, o := bytes.NewBuffer([]byte{13}), new(nopWriter)\n\t\tb.StartTimer()\n\t\tserve(i, o)\n\t}\n}\n\nfunc BenchmarkServe21(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tb.StopTimer()\n\t\ti, o := bytes.NewBuffer([]byte{21}), new(nopWriter)\n\t\tb.StartTimer()\n\t\tserve(i, o)\n\t}\n}\n\nfunc BenchmarkServe34(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tb.StopTimer()\n\t\ti, o := bytes.NewBuffer([]byte{34}), new(nopWriter)\n\t\tb.StartTimer()\n\t\tserve(i, o)\n\t}\n}\n\nfunc BenchmarkServe55(b *testing.B) {\n\tfor n := 0; n < b.N; n++ {\n\t\tb.StopTimer()\n\t\ti, o := bytes.NewBuffer([]byte{55}), new(nopWriter)\n\t\tb.StartTimer()\n\t\tserve(i, o)\n\t}\n}\n"
  },
  {
    "path": "coord-exec.sh",
    "content": "#!/bin/sh\n\n#/ usage: exec.sh\n\ndid=$1\nwid=0\n\n[ -z \"$did\" ] && did=0\n\n_set() {\n  # TODO: use an id rather than 1\n  printf 1 | doozer set /noeq/$did/$wid 0\n}\n\nwhile ! _set\ndo wid=`expr $wid + 1`\ndone\n\nexec noeqd -w $wid -d $did\n"
  },
  {
    "path": "main.go",
    "content": "package main\n\nimport (\n\t\"errors\"\n\t\"flag\"\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"net\"\n\t\"os\"\n\t\"sync\"\n\t\"time\"\n)\n\nvar (\n\tErrInvalidRequest = errors.New(\"invalid request\")\n\tErrInvalidAuth    = errors.New(\"invalid auth\")\n)\n\nvar (\n\ttoken = os.Getenv(\"NOEQ_TOKEN\")\n)\n\nconst (\n\tworkerIdBits       = uint64(5)\n\tdatacenterIdBits   = uint64(5)\n\tmaxWorkerId        = int64(-1) ^ (int64(-1) << workerIdBits)\n\tmaxDatacenterId    = int64(-1) ^ (int64(-1) << datacenterIdBits)\n\tsequenceBits       = uint64(12)\n\tworkerIdShift      = sequenceBits\n\tdatacenterIdShift  = sequenceBits + workerIdBits\n\ttimestampLeftShift = sequenceBits + workerIdBits + datacenterIdBits\n\tsequenceMask       = int64(-1) ^ (int64(-1) << sequenceBits)\n\n\t// Tue, 21 Mar 2006 20:50:14.000 GMT\n\ttwepoch = int64(1288834974657)\n)\n\n// Flags\nvar (\n\twid   = flag.Int64(\"w\", 0, \"worker id\")\n\tdid   = flag.Int64(\"d\", 0, \"datacenter id\")\n\tladdr = flag.String(\"l\", \"0.0.0.0:4444\", \"the address to listen on\")\n\tlts   = flag.Int64(\"t\", -1, \"the last timestamp in milliseconds\")\n)\n\nvar (\n\tmu  sync.Mutex\n\tseq int64\n)\n\nfunc main() {\n\tparseFlags()\n\tacceptAndServe(mustListen())\n}\n\nfunc parseFlags() {\n\tflag.Parse()\n\tif *wid < 0 || *wid > maxWorkerId {\n\t\tlog.Fatalf(\"worker id must be between 0 and %d\", maxWorkerId)\n\t}\n\n\tif *did < 0 || *did > maxDatacenterId {\n\t\tlog.Fatalf(\"datacenter id must be between 0 and %d\", maxDatacenterId)\n\t}\n}\n\nfunc mustListen() net.Listener {\n\tl, err := net.Listen(\"tcp\", *laddr)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\treturn l\n}\n\nfunc acceptAndServe(l net.Listener) {\n\tfor {\n\t\tcn, err := l.Accept()\n\t\tif err != nil {\n\t\t\tlog.Println(err)\n\t\t}\n\n\t\tgo func() {\n\t\t\terr := serve(cn, cn)\n\t\t\tif err != io.EOF {\n\t\t\t\tlog.Println(err)\n\t\t\t}\n\t\t\tcn.Close()\n\t\t}()\n\t}\n}\n\nfunc serve(r io.Reader, w io.Writer) error {\n\tif token != \"\" {\n\t\terr := auth(r)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\tc := make([]byte, 1)\n\tfor {\n\t\t// Wait for 1 byte request\n\t\t_, err := io.ReadFull(r, c)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\n\t\tn := uint(c[0])\n\t\tif n == 0 {\n\t\t\t// No authing at this point\n\t\t\treturn ErrInvalidRequest\n\t\t}\n\n\t\tb := make([]byte, n*8)\n\t\tfor i := uint(0); i < n; i++ {\n\t\t\tid, err := nextId()\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\n\t\t\toff := i * 8\n\t\t\tb[off+0] = byte(id >> 56)\n\t\t\tb[off+1] = byte(id >> 48)\n\t\t\tb[off+2] = byte(id >> 40)\n\t\t\tb[off+3] = byte(id >> 32)\n\t\t\tb[off+4] = byte(id >> 24)\n\t\t\tb[off+5] = byte(id >> 16)\n\t\t\tb[off+6] = byte(id >> 8)\n\t\t\tb[off+7] = byte(id)\n\t\t}\n\n\t\t_, err = w.Write(b)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\tpanic(\"not reached\")\n}\n\nfunc milliseconds() int64 {\n\treturn time.Now().UnixNano() / 1e6\n}\n\nfunc nextId() (int64, error) {\n\tmu.Lock()\n\tdefer mu.Unlock()\n\n\tts := milliseconds()\n\n\tif ts < *lts {\n\t\treturn 0, fmt.Errorf(\"time is moving backwards, waiting until %d\\n\", *lts)\n\t}\n\n\tif *lts == ts {\n\t\tseq = (seq + 1) & sequenceMask\n\t\tif seq == 0 {\n\t\t\tfor ts <= *lts {\n\t\t\t\tts = milliseconds()\n\t\t\t}\n\t\t}\n\t} else {\n\t\tseq = 0\n\t}\n\n\t*lts = ts\n\n\tid := ((ts - twepoch) << timestampLeftShift) |\n\t\t(*did << datacenterIdShift) |\n\t\t(*wid << workerIdShift) |\n\t\tseq\n\n\treturn id, nil\n}\n\nfunc auth(r io.Reader) error {\n\tb := make([]byte, 2)\n\t_, err := io.ReadFull(r, b)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tif b[0] != 0 {\n\t\treturn ErrInvalidRequest\n\t}\n\n\tb = make([]byte, b[1])\n\t_, err = io.ReadFull(r, b)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tif string(b) != token {\n\t\treturn ErrInvalidAuth\n\t}\n\n\treturn nil\n}\n"
  },
  {
    "path": "main_test.go",
    "content": "package main\n\nimport (\n\t\"bytes\"\n\t\"github.com/bmizerany/assert\"\n\t\"testing\"\n)\n\nfunc TestServeZero(t *testing.T) {\n\ti, o := bytes.NewBuffer([]byte{0}), new(bytes.Buffer)\n\tserve(i, o)\n\tassert.Equal(t, 0, o.Len())\n}\n\nfunc TestServeMoreThanZero(t *testing.T) {\n\ti, o := bytes.NewBuffer([]byte{1}), new(bytes.Buffer)\n\tserve(i, o)\n\tassert.Equal(t, 8, o.Len())\n\n\ti, o = bytes.NewBuffer([]byte{2}), new(bytes.Buffer)\n\tserve(i, o)\n\tassert.Equal(t, 16, o.Len())\n\n\ti, o = bytes.NewBuffer([]byte{255}), new(bytes.Buffer)\n\tserve(i, o)\n\tassert.Equal(t, 255*8, o.Len())\n}\n"
  }
]