[ { "path": ".github/workflows/test.yml", "content": "name: tests\n\non:\n push: {branches: [master]}\n pull_request: {branches: [master]}\n\njobs:\n test:\n runs-on: ubuntu-20.04\n timeout-minutes: 30\n strategy:\n fail-fast: false\n matrix:\n go: [\"1.21.x\", \"1.22.x\", \"1.23.x\"]\n arch: [\"amd64\", \"386\"]\n\n env:\n GOARCH: \"${{matrix.arch}}\"\n\n steps:\n - uses: actions/checkout@v2\n\n - uses: WillAbides/setup-go-faster@v1.7.0\n with:\n go-version: ${{matrix.go}}\n\n - name: build\n run: make all\n\n - name: test\n run: ./test.sh\n\n staticcheck:\n runs-on: ubuntu-20.04\n steps:\n - uses: actions/checkout@v2\n\n - uses: dominikh/staticcheck-action@v1.3.1\n with:\n version: \"2024.1.1\"\n install-go: false\n\n code-coverage:\n runs-on: ubuntu-20.04\n steps:\n - uses: actions/checkout@v2\n\n - name: install goveralls\n run: go install github.com/mattn/goveralls@latest\n\n - name: send coverage\n env:\n COVERALLS_TOKEN: ${{secrets.GITHUB_TOKEN}}\n run: ./coverage.sh --coveralls\n" }, { "path": ".gitignore", "content": "/build/\ndist/\n.cover/\nprofile/\n\n# nsqd data from testing\n*.dat\n\n# nsqadmin\nnode_modules\n\n# apps\napps/nsqlookupd/nsqlookupd\napps/nsqd/nsqd\napps/nsqadmin/nsqadmin\napps/nsq_to_nsq/nsq_to_nsq\napps/nsq_to_file/nsq_to_file\napps/nsq_pubsub/nsq_pubsub\napps/nsq_to_http/nsq_to_http\napps/nsq_tail/nsq_tail\napps/nsq_stat/nsq_stat\napps/to_nsq/to_nsq\nbench/bench_reader/bench_reader\nbench/bench_writer/bench_writer\nbench/bench_channels/bench_channels\n\n# Go.gitignore\n\n# Compiled Object files, Static and Dynamic libs (Shared Objects)\n*.o\n*.a\n*.so\n\n# Folders\n_obj\n_test\n\n# Architecture specific extensions/prefixes\n*.[568vq]\n[568vq].out\n\n*.cgo1.go\n*.cgo2.c\n_cgo_defun.c\n_cgo_gotypes.go\n_cgo_export.*\n\n_testmain.go\n\n*.exe\n\n\n# vim stuff\n*.sw[op]\n" }, { "path": "AUTHORS", "content": "# For a complete listing, see https://github.com/nsqio/nsq/graphs/contributors\n\n# Original Authors\n\nMatt Reiferson \nJehiah Czebotar \n\n# Maintainers\n\nPierce Lopez \n\n# Disclaimer\n\nMatt Reiferson's contributions to this project are being made solely in a personal capacity\nand does not convey any rights to any intellectual property of any third parties.\n" }, { "path": "CODE_OF_CONDUCT.md", "content": "# Contributor Code of Conduct\n\nAs contributors and maintainers of this project, and in the interest of fostering an open and\nwelcoming community, we pledge to respect all people who contribute through reporting issues,\nposting feature requests, updating documentation, submitting pull requests or patches, and other\nactivities.\n\nWe are committed to making participation in this project a harassment-free experience for everyone,\nregardless of level of experience, gender, gender identity and expression, sexual orientation,\ndisability, personal appearance, body size, race, ethnicity, age, religion, or nationality.\n\nExamples of unacceptable behavior by participants include:\n\n* The use of sexualized language or imagery\n* Personal attacks\n* Trolling or insulting/derogatory comments\n* Public or private harassment\n* Publishing other's private information, such as physical or electronic addresses, without explicit permission\n* Other unethical or unprofessional conduct.\n\nProject maintainers have the right and responsibility to remove, edit, or reject comments, commits,\ncode, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. By\nadopting this Code of Conduct, project maintainers commit themselves to fairly and consistently\napplying these principles to every aspect of managing this project. Project maintainers who do not\nfollow or enforce the Code of Conduct may be permanently removed from the project team.\n\nThis code of conduct applies both within project spaces and in public spaces when an individual is\nrepresenting the project or its community.\n\nInstances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an\nissue or contacting one or more of the project maintainers.\n\nThis Code of Conduct is adapted from the [Contributor Covenant][1], version 1.2.0, available at\n[http://contributor-covenant.org/version/1/2/0/][2].\n\n[1]: http://contributor-covenant.org\n[2]: http://contributor-covenant.org/version/1/2/0/\n" }, { "path": "CONTRIBUTING.md", "content": "# Contributing\n\nThanks for your interest in contributing to NSQ!\n\n## Code of Conduct\n\nHelp us keep NSQ open and inclusive. Please read and follow our [Code of Conduct](CODE_OF_CONDUCT.md).\n\n## Getting Started\n\n* make sure you have a [GitHub account](https://github.com/signup/free)\n* submit a ticket for your issue, assuming one does not already exist\n * clearly describe the issue including steps to reproduce when it is a bug\n * identify specific versions of the binaries and client libraries\n* fork the repository on GitHub\n\n## Making Changes\n\n* create a branch from where you want to base your work\n * we typically name branches according to the following format: `helpful_name_`\n* make commits of logical units\n* make sure your commit messages are in a clear and readable format, example:\n \n```\nnsqd: fixed bug in protocol_v2\n \n* update the message pump to properly account for RDYness\n* cleanup variable names\n* ...\n```\n\n* if you're fixing a bug or adding functionality it probably makes sense to write a test\n* make sure to run `fmt.sh` and `test.sh` in the root of the repo to ensure that your code is\n properly formatted and that tests pass (we use GitHub Actions for continuous integration)\n\n## Submitting Changes\n\n* push your changes to your branch in your fork of the repository\n* submit a pull request against nsqio's repository\n* comment in the pull request when you're ready for the changes to be reviewed: `\"ready for review\"`\n" }, { "path": "ChangeLog.md", "content": "# NSQ Changelog\n\n## Releases\n\n### 1.3.0 - 2023-12-26\n\n**Upgrading**\n\n * #1427 / #1373 / #1371 - fix staticcheck warnings, remove support for gobindata / go 1.16\n\nFeatures:\n\n * #1473 - `nsqd`: use --tls-root-ca-file in nsqauth request (thanks @intellitrend-team)\n * #1470 / #1469 - `nsqadmin`: upgrade supported ECMA from ES5 to ES2020 (thanks @dudleycarr)\n * #1468 - `nsqadmin`: add paused label to topic within the node view (thanks @dudleycarr)\n * #1462 - `nsqadmin`: add admin check for topic/node thombstone endpoint (thanks @dudleycarr)\n * #1434 - `nsqd`: add support of unix sockets for tcp, http, https listeners (thanks @telepenin)\n * #1424 - `nsqd`: add /debug/freememory API (thanks @guozhao-coder)\n * #1421 - `nsqd`: nicer tls-min-version help text default\n * #1376 - `nsqd`: allow unbuffered memory chan if ephemeral or deferred\n * #1380 - `nsqd`: use metadata struct for both marshal and unmarshal (thanks @karalabe)\n * #1403 - `nsqd`: /info api returns more info (thanks @arshabbir)\n * #1384 - `nsqd`: allow disabling both HTTP and HTTPS interfaces (thanks @karalabe)\n * #1385 - `nsqd`: enable support for TLS1.3 (thanks @karalabe)\n * #1372 - `nsqadmin`: new flag --dev-static-dir instead of debug build tag\n\nBugs:\n\n * #1478 - `Dockerfile`: remove nsswitch.conf check (thanks @dudleycarr)\n * #1467 - `nsqadmin`: fix counter by bounding animation steps (thanks @dudleycarr)\n * #1466 - `nsqadmin`: fix broken graph template in nsqadmin node view (thanks @dudleycarr)\n * #1455 / #1387 - update dependencies\n * #1445 - `nsqd`: fix unsafe concurrency read in RemoveClient (thanks @gueFDF)\n * #1441 - `nsqd`: fix panic when statsd enabled and memstats disabled with no topics (thanks @carl-reverb)\n * #1428 - delete `validTopicChannelNameRegex` useless escape characters (thanks @sjatsh)\n * #1419 - contrib: update nsqadmin.cfg.example (thanks @StellarisW)\n\n### 1.2.1 - 2021-08-15\n\n**Upgrading**\n\n * #1227 - bump dependencies, officially drop `dep` support, drop Go `1.9` support\n\nFeatures:\n\n * #1347 - `nsqadmin`: switch to go:embed for static assets\n * #1355 / #1364 - arm64 builds (thanks @danbf)\n * #1346 - `nsqd`: ability to skip ephemeral topics/channels in statsd output \n * #1336 / #1341 / #1343 - `nsqd`: ability to configure auth endpoint path (thanks @tufanbarisyildirim)\n * #1307 - remove `Context` to use stdlib `context`\n * #1295 / #1296 - switch to GitHub Actions CI\n * #1292 - `nsqd`: minimize allocations on message send (thanks @imxyb)\n * #1289 - optimize `uniq` (thanks @donutloop)\n * #1230 / #1232 - `nsqd`: ability to omit memory stats from `/stats` (thanks @creker)\n * #1226 - `nsqd`: only update internal `RDY` count for client when it changes (thanks @andyxning)\n * #1221 / #1363 - test against more recent versions of Go\n * #1209 - `nsqd`: bump `go-diskqueue` (interface change) (thanks @bitpeng)\n * #1206 - prefer idiomatic `sort.Ints` over `sort.Sort` (thanks @lelenanam)\n * #1197 / #1362 - Dockerfile: update Alpine base image, use /data by default\n * #1178 - `nsqd`: configurable queue scan worker pool (thanks @andyxning)\n * #1159 - `nsqd`: don't buffer messages when `--mem-queue-size=0` (thanks @bitpeng)\n * #1073 / #1297 - `nsqd`: support separate broadcast ports for TCP and HTTP (thanks @shyam-king)\n\nBugs:\n\n * #1347 - `nsqadmin`: fix graphite key for ephemeral topics/channels\n * #765 / #1195 / #1203 / #1205 - fix build on illumos (thanks @i-sevostyanov)\n * #1333 - fix race detector tests on non-bash shells \n * #1330 - fix `log_level` support in configuration file (thanks @edoger)\n * #1319 / #1331 / #1361 - `nsqd`: handle SIGTERM \n * #1287 - `nsqadmin`: fix `--proxy-graphite` support (thanks @fanlix)\n * #1270 / #1271 - `nsqlookupd`: fix incorrect error message for HTTP listener (thanks @TangDH03)\n * #1264 - fix benchmark script\n * #1251 / #1314 / #1327 - `nsqd`: fix live lock for high churn ephemeral topic/channel reconnections (thanks @slayercat)\n * #1237 - Dockerfile: add `nsswitch.conf` to ensure go resolver uses `/etc/hosts` first\n * #1217 / #1220 - `nsqd`: improve error message when `--data-path` does not exist (thanks @mdh67899)\n * #1198 / #1190 / #1262 - synchronize close of all connections on Exit (thanks @benjsto)\n * #1188 / #1189 - `nsqadmin`: fix channel delete, fix source-maps in Firefox (thanks @avtrifanov)\n * #1186 - `nsqadmin`: fix nodes list with ipv6 addresses (thanks @andyxning)\n\n### 1.2.0 - 2019-08-26\n\n**Upgrading**\n\n * #1055 - `nsqd`: removed support for old metadata scheme used in v0.3.8 and earlier\n * you cannot upgrade directly from v0.3.8 to v1.2.0, you must go through v1.0.0-compat or v1.1.0\n * #1115 - manage dependencies with go modules\n * `dep` support still present for now, but deprecated\n\nFeatures:\n\n * #1136 - `nsqd`: add `--max-channel-consumers` (default unlimited) (thanks @andyxning)\n * #1133 - `nsqd`: add `--min-output-buffer-timeout` (default 25ms) to limit how low a timeout a consumer can request\n * and raise default `--max-output-buffer-timeout` to 30 seconds (lower timeout, more cpu usage)\n * #1127 - `nsqd`: add topic total message bytes to stats (thanks @andyxning)\n * #1125 - `nsqd`: add flag to adjust default `--output-buffer-timeout` (thanks @andyxning)\n * #1163 - `nsqd`: add random load balancing for authd requests (thanks @shenhui0509)\n * #1119 - `nsqd`: include client TLS cert CommonName in authd requests\n * #1147 - `nsq_to_file`: include topic/channel in most log messages\n * #1117 - `nsq_to_file`: add `--log-level` and `--log-prefix` flags\n * #1117/#1120/#1123 - `nsq_to_file`: big refactor, more robust file switching and syncing and error handling\n * #1118 - `nsqd`: add param to `/stats` endpoint to allow skipping per-client stats (much faster if many clients)\n * #1118 - `nsqadmin`, `nsq_stat`: use `include_clients` param for `/stats` for a big speedup for big clusters\n * #1110 - `nsq_to_file`: support for separate working directory with `--work-dir` (thanks @mccutchen)\n * #856 - `nsqadmin`: add `--base-path` flag (thanks @blinklv)\n * #1072 - `nsq_to_http`: add `--header` flag (thanks @alwindoss)\n * #881 - `nsqd`: add producer client tcp connections to stats (thanks @sparklxb)\n * #1071/#1074 - `nsq_to_file`: new flag `--sync-interval` (default same as previous behavior, 30 seconds) (thanks @alpaker)\n\nBugs:\n\n * #1153 - `nsqd`: close connections that don't send \"magic\" header (thanks @JoseFeng)\n * #1140 - `nsqd`: exit on all fatal Accept() errors - restart enables better recovery for some conditions (thanks @mdh67899)\n * #1140 - `nsqd`, `nsqlookupd`, `nsqadmin`: refactor LogLevel, general refactor to better exit on all fatal errors\n * #1140 - `nsqadmin`: switch to using `judwhite/go-svc` like `nsqd` and `nsqadmin` do\n * #1134 - `nsqadmin`: fix clients count and channel total message rate (new bugs introduced in this cycle)\n * #1132 - `nsqd`, `nsqlookupd`, `nsqadmin`: fix http error response unreliable json serialization\n * #1116 - `nsqlookupd`: fix orphaned ephemeral topics in registration DB\n * #1109 - `nsqd`: fix topic message mis-counting if there are backend write errors (thanks @SwanSpouse)\n * #1099 - `nsqlookupd`: optimize `/nodes` endpoint, much better for hundreds of nsqd (thanks @andyxning)\n * #1085 - switch `GOFLAGS` to `BLDFLAGS` in `Makefile` now that `GOFLAGS` is automatically used by go\n * #1080 - `nsqadmin`: eslint reported fixes/cleanups\n\n### 1.1.0 - 2018-08-19\n\n**Upgrading from 1.0.0-compat**: Just a few backwards incompatible changes:\n\n * #1056 - Removed the `nsq_pubsub` utility\n * #873 - `nsqd` flags `--msg-timeout` and `--statsd-interval` only take duration strings\n * plain integer no longer supported\n * #921 - `nsqd`: http `/mpub` endpoint `binary` param interprets \"0\" or \"false\" to mean text mode\n * previously any value meant to use binary mode instead of text mode - (thanks @andyxning)\n\nThe previous release, version \"1.0.0-compat\", was curiously-named to indicate an almost\n(but not quite) complete transition to a 1.0 api-stable release line. Confusingly, this\nfollow-up release which completes the transition comes more than a year later. Because there\nhave been a fair number of changes and improvements in the past year, an additional minor\nversion bump seems appropriate.\n\nFeatures:\n\n * #874 - `nsqd`: add memory stats to http `/stats` response (thanks @sparklxb)\n * #892 - `nsqd`, `nsqlookupd`, `nsqadmin`: add `--log-level` option (deprecating `--verbose`) (thanks @antihax)\n * #898 - `nsqd`, `nsqlookupd`, `nsqadmin`: logging refactor to use log levels everywhere\n * #914 - `nsqadmin`: `X-Forwarded-User` based \"admin\" permission (thanks @chen-anders)\n * #929 - `nsqd`: add topic/channel filter to `/stats`, use in `nsqadmin` and `nsq_stat` for efficiency (thanks @andyxning)\n * #936 - `nsq_to_file`: refactor/cleanup\n * #945 - `nsq_to_nsq`: support multiple `--topic` flags (thanks @jlr52)\n * #957 - `nsq_tail`: support multiple `--topic` flags (thanks @soar)\n * #946 - `nsqd`, `nsqadmin`: update internal http client with new go `http.Transport` features (keepalives, timeouts, dualstack)\n * affects metadata/stats requests between `nsqadmin`, `nsqd`, `nsqlookupd`\n * #954 - manage dependencies with `dep` (replacing `gpm`) (thanks @judwhite)\n * #957 - multi-stage docker image build (thanks @soar)\n * #996 - `nsqd`: better memory usage when messages have different sizes (thanks @andyxning)\n * #1019 - `nsqd`: optimize random channel selection in queueScanLoop (thanks @vearne)\n * #1025 - `nsqd`: buffer and spread statsd udp sends (avoid big burst of udp, less chance of loss)\n * #1038 - `nsqlookupd`: optimize for many producers (thousands) (thanks @andyxning)\n * #1050/#1053 - `nsqd`: new topic can be unlocked faster after creation\n * #1062 - `nsqadmin`: update JS deps\n\nBugs:\n\n * #753 - `nsqadmin`: fix missing channels in topic list\n * #867 - `to_nsq`: fix divide-by-zero issue when `--rate` not specified (thanks @adamweiner)\n * #868 - `nsqd`: clamp requeue timeout to range instead of dropping connection (thanks @tsholmes)\n * #891 - `nsqd`: fix race when client subscribes to ephemeral topic or channel while it is being cleaned up (reported by @slayercat)\n * #927 - `nsqd`: fix deflate level handling\n * #934 - `nsqd`: fix channel shutdown flush race\n * #935 - `nsq_to_file`: fix connection leaks when using `--topic-pattern` (thanks @jxskiss)\n * #951 - mention docker images and binaries for additional platforms in README (thanks @DAXaholic)\n * #950 - `nsqlookupd`: close connection when magic read fails (thanks @yc90s)\n * #971 - `nsqd`: fix some races getting ChannelStats (thanks @daroot)\n * #988 - `nsqd`: fix e2e timings config example, add range validation (thanks @protoss-player)\n * #991 - `nsq_tail`: logging to stderr (only nsq messages to stdout)\n * #1000 - `nsq_to_http`: fix http connect/request timeout flags (thanks @kamyanskiy)\n * #993/#1008 - `nsqd`: fix possible lookupd-identify-error busy-loop (reported by @andyxning)\n * #1005 - `nsqadmin`: fix typo \"Delfate\" in connection attrs list (thanks @arussellsaw)\n * #1032 - `nsqd`: fix loading metadata with messages queued on un-paused topic with multiple channels (thanks @michaelyou)\n * #1004 - `nsqlookupd`: exit with error when failed to listen on ports (thanks @stephens2424)\n * #1068 - `nsqadmin`: fix html escaping for large_graph url inside javascript\n * misc test suite improvements and updates (go versions, tls certs, ...)\n\n### 1.0.0-compat - 2017-03-21\n\n**Upgrading from 0.3.8**: Numerous backwards incompatible changes:\n\n * Deprecated `nsqd` features removed:\n * Pre-V1 HTTP endpoints / response format:\n * `/{m,}put` (use `/{m,}pub`)\n * `/{create,delete,empty,pause,unpause}_{topic,channel}` (use `/{topic,channel}/`)\n * `--max-message-size` flag (use `--max-msg-size`)\n * `V2` protocol `IDENTIFY` command `short_id` and `long_id` properties (use `client_id`, `hostname`, and `user_agent`)\n * `/stats` HTTP response `name` property (use `client_id`)\n * Deprecated `nsqlookupd` features removed:\n * Pre-V1 HTTP endpoints / response format:\n * `/{create,delete}_{topic,channel}` (use `/{topic,channel}/`)\n * `/tombstone_topic_producer` (use `/topic/tombstone`)\n * Deprecated `nsqadmin` features removed:\n * `--template-dir` flag (not required, templates are compiled into binary)\n * `--use-statsd-prefixes` flag (use `--statsd-counter-format` and `--statsd-gauge-format`)\n * `nsq_stat` `--status-every` flag (use `--interval`)\n * `--reader-opt` on all binaries that had this flag (use `--consumer-opt`)\n * `nsq_to_file` `--gzip-compression` flag (use `--gzip-level`)\n * `nsq_to_http` `--http-timeout` and `--http-timeout-ms` flags (use `--http-connect-timeout` and `--http-request-timeout`)\n * `nsq_to_http` `--round-robin` flag (use `--mode=round-robin`)\n * `nsq_to_http` `--max-backoff-duration` flag (use `--consumer-opt=max_backoff_duration,X`)\n * `nsq_to_http` `--throttle-fraction` flag (use `--sample=X`)\n * `nsq_to_nsq` `--max-backoff-duration` flag (use `--consumer-opt=max_backoff_duration,X`)\n * `nsqd` `--worker-id` deprecated in favor of `--node-id` (to be fully removed in subsequent release)\n\nThis is a compatibility release that drops a wide range of previously deprecated features (#367)\nwhile introducing some new deprecations (#844) that we intend to fully remove in a subsequent 1.0\nrelease.\n\nOf note, all of the pre-1.0 HTTP endpoints (and response formats) are gone. Any clients or tools\nthat use these endpoints/response formats won't work with this release. These changes have been\navailable since 0.2.29 (released in July of 2014). Clients wishing to forwards-compatibly upgrade\ncan either use the new endpoints or send the following header:\n\n Accept: application/vnd.nsq version=1.0\n\nAlso, many command line flags have been removed — in almost all cases an alternative is available\nwith a (hopefully) more obvious name. These changes have the same affect on config file option\nnames.\n\nOn Linux, this release will automatically migrate `nsq..dat` named metadata files to\n`nsq.dat` in a way that allows users to seamlessly _downgrade_ from this release back to 0.3.8, if\nnecessary. A subsequent release will clean up these convenience symlinks and observe only\n`nsq.dat`. See the discussion in #741 and the changes #844 for more details.\n\nPerformance wise, #741 landed which significantly reduces global contention on internal message ID\ngeneration, providing a ~1.75x speed improvement on multi-topic benchmarks.\n\nFinally, a number of minor issues were resolved spanning contributions from 9 community members!\nThanks!\n\nFeatures:\n\n * #766 - use `alpine` base image for official Docker container (thanks @kenjones-cisco)\n * #775 - `nsqadmin:` `/config` API (thanks @kenjones-cisco)\n * #776 - `nsqadmin`, `nsq_stat`, `nsq_to_file`, `nsq_to_http`: HTTP client connect/request timeouts (thanks @kenjones-cisco)\n * #777/#778/#783/#785 - improve test coverage (thanks @kenjones-cisco)\n * #788 - `to_nsq`: add `--rate` message throttling option\n * #367 - purge deprecated features (see above)\n * #741 - `nsqd`: per-topic message IDs (multi-topic pub benchmarks up to ~1.75x faster)\n * #850 - `nsqd`, `nsqlookupd`, `nsqadmin`: add `--log-prefix` option (thanks @ploxiln)\n * #844 - `nsqd`: deprecate `--worker-id` for `--node-id` and drop ID from `nsqd.dat` file (thanks @ploxiln)\n\nBugs:\n\n * #787 - `nsqlookupd`: properly close TCP connection in `IOLoop` (thanks @JackDrogon)\n * #792 - `nsqdmin`: fix root CA verification (thanks @joshuarubin)\n * #794 - `nsq_to_file`: require `--topic` or `--topic-pattern` (thanks @judwhite)\n * #816/#823 - `nsqadmin`: fix handling of IPv6 broadcast addresses (thanks @magnetised)\n * #805/#832 - `nsqd`: fix requeue and deferred message accounting (thanks @sdbaiguanghe)\n * #532/#830 - `nsqd`: switch to golang/snappy to fix snappy deadlock\n * #826/#831/#837/#839 - `nsqd`: fix default `--broadcast-address` and error when `nsqlookupd` reqs fail (thanks @ploxiln @stephensearles)\n * #822/#835 - `nsqd`: prevent panic in binary `/mpub` (thanks @yangyifeng01)\n * #841 - `nsqadmin`: allow ctrl/meta+click to open a new tab\n * #843 - `nsqd`: check for exit before requeing\n\n### 0.3.8 - 2016-05-26\n\n**Upgrading from 0.3.7**: Binaries contain no backwards incompatible changes.\n\nThis release fixes a critical regression in `0.3.7` that could result in message loss when\nattempting to cleanly shutdown `nsqd` by sending it a `SIGTERM`. The expected behavior was for it\nto flush messages in internal buffers to disk before exiting. See #757 and #759 for more details.\n\nA few performance improvements landed including #743, which improves channel throughput by ~17%,\nand #740, which reduces garbage when reading messages from disk.\n\nWe're now stripping debug info, reducing binary size, in the official binary downloads and Windows\nbinaries are now bundled with the appropriate `.exe` extension (#726 and #751).\n\nFeatures:\n\n * #743 - `nsqd`: remove channel `messagePump`\n * #751 - strip debug info from binaries (thanks @ploxiln)\n * #740 - `nsqd`: reduce garbage when reading from diskqueue (thanks @dieterbe)\n\nBugs:\n\n * #757/#759 - `nsqd`: properly handle `SIGTERM` (thanks @judwhite)\n * #738 - updates for latest `go-options`\n * #730 - `nsqd`: diskqueue sync count on both read/write\n * #734 - `nsqadmin`: make `rate` column work without `--proxy-graphite` (thanks @ploxiln)\n * #726 - add `.exe` extension to Windows binaries (thanks @ploxiln)\n * #722 - `nsqadmin`: fix connected duration > `1hr`\n\n### 0.3.7 - 2016-02-23\n\n**Upgrading from 0.3.6**: Binaries contain no backwards incompatible changes.\n\nThis release has been built with Go 1.6.\n\nHighlights include the various work done to reduce `nsqd` lock contention, significantly improving\nthe impact of high load on the `/stats` endpoint, addressing issues with timeouts and failures\nin `nsqadmin` (#700, #701, #703, #709).\n\nThanks to @judwhite, `nsqd` and `nsqlookupd` now natively support being run as a Windows service\n(#718). We're also now publishing official Windows releases.\n\n`nsqd` will now `flock` its data directory on linux, preventing two `nsqd` from running\nsimultaneously pointed at the same path (#583).\n\nOn the bugfix side, the most noteworthy change is that `nsqd` will now correctly reset health state\non a successful backend write (#671).\n\nFeatures:\n\n * #700/#701/#703/#709 - `nsqd`: reduce lock contention (thanks @zachbadgett @absolute8511)\n * #718 - `nsqd`/`nsqlookupd`: support running as a windows service (thanks @judwhite)\n * #706 - `nsqd`: support enabling/disabling block profile via HTTP (thanks @absolute8511)\n * #710 - `nsqd`: support `POST` `/debug/pprof/symbol` (thanks @absolute8511)\n * #662 - `nsqadmin`: add flags for formatting statsd keys (thanks @kesutton)\n * #583 - `nsqd`: `flock` `--data-path` on linux\n * #663 - `nsqd`: optimize GUID generation (thanks @ploxiln)\n\nBugs:\n\n * #672 - `nsqd`: fix max size accounting in `diskqueue` (thanks @judwhite)\n * #671 - `nsqd`: reset health on successful backend write (thanks @judwhite)\n * #615 - `nsqd`: prevent OOM when reading from `nsqlookupd` peer\n * #664/#666 - dist.sh/Makefile cleanup (thanks @ploxiln)\n\n### 0.3.6 - 2015-09-24\n\n**Upgrading from 0.3.5**: Binaries contain no backwards incompatible changes.\n\nWe've adopted the [Contributor Covenant 1.2 Code of Conduct](CODE_OF_CONDUCT.md) (#593). Help us\nkeep NSQ open and inclusive by reading and following this document.\n\nWe closed a few longstanding issues related to `nsqadmin`, namely (#323, et al.) converting it to\nan API and single-page app (so that it is _much_ easier to develop), displaying fine-grained errors\n(#421, #657), and enabling support for `--tls-required` configurations (#396).\n\nFor `nsqd`, we added support for deferred publishing aka `DPUB` (#293), which allows a producer to\nspecify a duration of time to delay initial delivery of the message. We also addressed performance\nissues relating to large numbers of topics/channels (#577) by removing some per-channel goroutines\nin favor of a centralized, periodic, garbage collection approach.\n\nIn order to provide more flexibility when deploying NSQ in dynamically orchestrated topologies,\n`nsqd` now supports the ability to configure `nsqlookupd` peers at runtime via HTTP (#601),\neliminating the need to restart the daemon.\n\nAs part of the large `nsqadmin` refactoring, we took the opportunity to cleanup the internals for\n_all_ of the daemon's HTTP code paths (#601, #610, #612, #641) as well as improving the test suite\nso that it doesn't leave around temporary files (#553).\n\nFeatures:\n\n * #593 - add code of conduct\n * #323/#631/#632/#642/#421/#649/#650/#651/#652/#654 - `nsqadmin`: convert to API / single-page app\n * #653 - `nsqadmin`: expand notification context\n * #293 - `nsqd`: add deferred pub (`DPUB`)\n * #577 - `nsqd`: drop per-channel queue workers in favor of centralized queue GC\n * #584 - `nsqlookupd`: improve registration DB performance (thanks @xiaost)\n * #601 - `nsqd`: HTTP endpoints to dynamically configure `nsqlookupd` peers\n * #608 - `nsqd`: support for filtering `/stats` to topic/channel (thanks @chrusty)\n * #601/#610/#612/#641 - improved HTTP internal routing / log HTTP requests\n * #628 - `nsqd`: clarify help text for `--e2e-processing-latency-percentile`\n * #640 - switch `--{consumer,producer}-opt` to `nsq.ConfigFlag`\n\nBugs:\n\n * #656 - `nsqadmin`: update `statsd` prefix to `stats.counters`\n * #421/#657 - `nsqadmin`: display upstream/partial errors\n * #396 - `nsqdamin`/`nsqd`: support for `--tls-required`\n * #558 - don't overwrite docker root FS\n * #582 - `nsqd`: ignore benign EOF errors\n * #587 - `nsqd`: GUID error handling / catch errors if GUID goes backwards (thanks @mpe)\n * #586 - `nsqd`: fix valid range for `--worker-id`\n * #550/#602/#617/#618/#619/#620/#622 - `nsqd`: fix benchmarks (thanks @Dieterbe)\n * #553 - cleanup test dirs\n * #600 - `nsqd`: enforce diskqueue min/max message size (thanks @twmb)\n\n### 0.3.5 - 2015-04-26\n\n**Upgrading from 0.3.3**: Binaries contain no backwards incompatible changes.\n\nThis is another quick bug fix release to address the broken `nsqadmin` binary in the distribution\n(see #578).\n\n### 0.3.4 - 2015-04-26\n\n**WARNING**: please upgrade to `v0.3.5` to address the broken `nsqadmin` binary.\n\n**Upgrading from 0.3.3**: Binaries contain no backwards incompatible changes.\n\nThis is a quick bug fix release to fix the outdated `go-nsq` dependency in `v0.3.3`\nfor the bundled utilities (see 6e8504e).\n\n### 0.3.3 - 2015-04-26\n\n**WARNING**: please upgrade to `v0.3.5` to address the outdated `go-nsq` dependency for the\nbundled utilities and the broken `nsqadmin` binary.\n\n**Upgrading from 0.3.2**: Binaries contain no backwards incompatible changes.\n\nThis release is primarily a bug fix release after cleaning up and reorganizing the codebase.\n`nsqadmin` is now importable, which paves the way for completing #323. The bundled utilities\nreceived a few feature additions and bug fixes (mostly from bug fixes on the `go-nsq` side).\n\nFeatures:\n\n * #569 - `nsqadmin`: re-org into importable package\n * #562 - `nsq_to_{nsq,http}`: add `epsilon-greedy` mode (thanks @twmb)\n * #547 - `nsqd`: adds `start_time` to `/stats` (thanks @ShawnSpooner)\n * #544 - `nsq_to_http`: accept any `200` response as success (thanks @mikedewar)\n * #548 - `nsq_to_http`: read entire request body (thanks @imgix)\n * #552/#554/#555/#556/#561 - code cleanup and `/internal` package re-org (thanks @cespare)\n\nBugs:\n\n * #573 - `nsqd`: don't persist metadata upon startup (thanks @xiaost)\n * #560 - `nsqd`: do not print `EOF` error when client closes cleanly (thanks @twmb)\n * #557 - `nsqd`: fix `--tls-required=tcp-https` with `--tls-client-auth-policy` (thanks @twmb)\n * #545 - enable shell expansion in official Docker image (thanks @paddyforan)\n\nNOTE: the bundled utilities are built against [`go-nsq` `v1.0.4`][go-nsq_104] and include all of\nthose features/fixes.\n\n[go-nsq_104]: https://github.com/nsqio/go-nsq/releases/tag/v1.0.4\n\n### 0.3.2 - 2015-02-08\n\n**Upgrading from 0.3.1**: Binaries contain no backwards incompatible changes however as of this\nrelease we've updated our official Docker images.\n\nWe now provide a single Docker image [`nsqio/nsq`](https://registry.hub.docker.com/r/nsqio/nsq/)\nthat includes *all* of the NSQ binaries. We did this for several reasons, primarily because the\ntagged versions in the previous incarnation were broken (and did not actually pin to a version!).\nThe new image is an order of magnitude smaller, weighing in around 70mb.\n\nIn addition, the impetus for this quick release is to address a slew of reconnect related bug fixes\nin the utility apps (`nsq_to_nsq`, `nsq_to_file`, etc.), for details see the [`go-nsq` `v1.0.3`\nrelease notes](https://github.com/nsqio/go-nsq/releases/tag/v1.0.3).\n\nFeatures:\n\n * #534/#539/#540 - single Dockerfile approach (thanks @paddyforan)\n\nBugs:\n\n * #529 - nsqadmin: fix more `#ephemeral` topic deletion issues\n * #530 - nsqd: fix the provided sample config file (thanks @jnewmano)\n * #538 - nsqd: fix orphaned ephemeral channels (thanks @adamsathailo)\n\n### 0.3.1 - 2015-01-21\n\n**Upgrading from 0.3.0**: No backwards incompatible changes.\n\nThis release contains minor bug fixes and feature additions.\n\nThere are a number of functionality improvements to the `nsq_stat` and `nsq_to_file` helper\napplications (and general support for `#ephemeral` topics, broken in `0.2.30`).\n\nAdditionally, the TLS options continue to improve with support for setting `--tls-min-version` and\na work-around for a bug relating to `TLS_FALLBACK_SCSV` ([to be fixed in Go\n1.5](https://go-review.googlesource.com/#/c/1776/)).\n\nFeatures:\n\n * #527 - nsq_stat: deprecate `--status-every` in favor of `--interval`\n * #524 - nsq_stat: add `--count` option (thanks @nordicdyno)\n * #518 - nsqd: set defaults for `--tls-min-version` and set TLS max version to 1.2\n * #475/#513/#518 - nsqd: `--tls-required` can be disabled for HTTP / add `--tls-min-version`\n (thanks @twmb)\n * #496 - nsq_to_file: add `` to filename and rotation by size/interval (thanks @xiaost)\n * #507 - nsq_stat: add rates (thanks @xiaost)\n * #505 - nsqd: speed up failure path of `BytesToBase10` (thanks @iand)\n\nBugs:\n\n * #522 - nsqadmin: fix `#ephemeral` topic deletion issues\n * #509 - nsqd: fix `diskqueue` atomic rename on Windows (thanks @allgeek)\n * #479 - nsqd: return `output_buffer_*` resolved settings in `IDENTIFY` response (thanks @tj)\n\n### 0.3.0 - 2014-11-18\n\n**Upgrading from 0.2.31**: No backwards incompatible changes.\n\nThis release includes a slew of bug fixes and few key feature additions.\n\nThe biggest functional change is that `nsqd` no longer decrements its `RDY` count for clients. This\nmeans that client libraries no longer have to periodically re-send `RDY`. For some context, `nsqd`\nalready provided back-pressure due to the fact that a client must respond to messages before\nreceiving new ones. The decremented `RDY` count only made the implementation of the server and\nclient more complex without additional benefit. Now the `RDY` command can be treated as an \"on/off\"\nswitch. For details see #404 and the associated changes in nsqio/go-nsq#83 and nsqio/pynsq#98.\n\nThe second biggest change (and oft-requested feature!) is `#ephemeral` topics. Their behavior\nmirrors that of channels. This feature is incredibly useful for situations where you're using\ntopics to \"route\" messages to consumers (like RPC) or when a backlog of messages is undesirable.\n\nThere are now scripts in the `bench` directory that automate the process of running a distributed\nbenchmark. This is a work-in-progress, but it already provides a closer-to-production setup and\ntherefore more accurate results. There's much work to do here!\n\nA whole bunch of bugs were fixed - notably all were 3rd-party contributions! Thanks!\n\n * #305 - `#ephemeral` topics\n * #404/#459 - don't decr `RDY` / send `RDY` before `FIN`/`REQ`\n * #472 - improve `nsqd` `diskqueue` sync strategies\n * #488 - ability to filter topics by regex in `nsq_to_file` (thanks @lxfontes)\n * #438 - distributed pub-sub benchmark scripts\n * #448 - better `nsqd` `IOLoop` logging (thanks @rexposadas)\n * #458 - switch to [gpm](https://github.com/pote/gpm) for builds\n\nBugs:\n\n * #493 - ensure all `nsqd` `Notify()` goroutines have exited prior to shutdown (thanks @allgeek)\n * #492 - ensure `diskqueue` syncs at end of benchmarks (thanks @Dieterbe)\n * #490 - de-flake `TestPauseMetadata` (thanks @allgeek)\n * #486 - require ports to be specified for daemons (thanks @jnewmano)\n * #482 - better bash in `dist.sh` (thanks @losinggeneration)\n * #480 - fix panic when `nsqadmin` checks stats for missing topic (thanks @jnewmano)\n * #469 - fix panic when misbehaving client sends corrupt command (thanks @prio)\n * #461 - fix panic when `nsqd` decodes corrupt message data (thanks @twmb)\n * #454/#455 - fix 32-bit atomic ops in `nsq_to_nsq`/`nsq_to_http` (thanks @leshik)\n * #451 - fix `go get` compatibility (thanks @adams-sarah)\n\n### 0.2.31 - 2014-08-26\n\n**Upgrading from 0.2.30**: No backwards incompatible changes.\n\nThis release includes a few key changes. First, we improved feedback and back-pressure when `nsqd`\nwrites to disk. Previously this was asynchronous and would result in clients not knowing that their\n`PUB` had failed. Interestingly, this refactoring improved performance of `PUB` by 41%, by removing\nthe topic's goroutine responsible for message routing in favor of `N:N` Go channel communication.\nFor details see #437.\n\n@paddyforan contributed official Dockerfiles that are now built automatically via Docker Hub.\nPlease begin to use (and improve these) as the various older images we had been maintaining will be\ndeprecated.\n\nThe utility apps deprecated the `--reader-opt` flag in favor of `--consumer-opt` and `nsq_to_nsq`\nand `to_nsq` received a `--producer-opt` flag, for configuring details of the connection publishing\nto `nsqd`. Additionally, it is now possible to configure client side TLS certificates via\n`tls_cert` and `tls_key` opts.\n\nAs usual, we fixed a few minor bugs, see below for details.\n\nNew Features / Enhancements:\n\n * #422/#437 - `nsqd`: diskqueue error feedback/backpressure (thanks @boyand)\n * #412 - official Dockerfiles for `nsqd`, `nsqlookupd`, `nsqadmin` (thanks @paddyforan)\n * #442 - utilities: add `--consumer-opt` alias for `--reader-opt` and\n add `--producer-opt` to `nsq_to_nsq` (also support configuration\n of `tls_cert` and `tls_key`)\n * #448 - `nsqd`: improve IOLoop error messages (thanks @rexposadas)\n\nBugs:\n\n * #440 - `nsqd`: fixed statsd GC stats reporting (thanks @jphines)\n * #434/#435 - refactored/stabilized tests and logging\n * #429 - `nsqd`: improve handling/documentation of `--worker-id` (thanks @bschwartz)\n * #428 - `nsqd`: `IDENTIFY` should respond with materialized `msg_timeout` (thanks @visionmedia)\n\n### 0.2.30 - 2014-07-28\n\n**Upgrading from 0.2.29**: No backwards incompatible changes.\n\n**IMPORTANT**: this is a quick bug-fix release to address a panic in `nsq_to_nsq` and\n`nsq_to_http`, see #425.\n\nNew Features / Enhancements:\n\n * #417 - `nsqadmin`/`nsqd`: expose TLS connection state\n * #425 - `nsq_to_nsq`/`nsq_to_file`: display per-destination-address timings\n\nBugs:\n\n * #425 - `nsq_to_nsq`/`nsq_to_file`: fix shared mutable state panic\n\n### 0.2.29 - 2014-07-25\n\n**Upgrading from 0.2.28**: No backwards incompatible changes.\n\nThis release includes a slew of new features and bug fixes, with contributions from 8\nmembers of the community, thanks!\n\nThe most important new feature is authentication (the `AUTH` command for `nsqd`), added in #356.\nWhen `nsqd` is configured with an `--auth-http-address` it will require clients to send the `AUTH`\ncommand. The `AUTH` command body is opaque to `nsqd`, it simply passes it along to the configured\nauth daemon which responds with well formed JSON, indicating which topics/channels and properties\non those entities are accessible to that client (rejecting the client if it accesses anything\nprohibited). For more details, see [the spec](https://nsq.io/clients/tcp_protocol_spec.html) or [the\n`nsqd` guide](https://nsq.io/components/nsqd.html#auth).\n\nAdditionally, we've improved performance in a few areas. First, we refactored in-flight handling in\n`nsqd` to reduce garbage creation and improve baseline performance 6%. End-to-end processing\nlatency calculations are also significantly faster, thanks to improvements in the\n[`perks`](https://github.com/bmizerany/perks/pulls/7) package.\n\nHTTP response formats have been improved (removing the redundant response wrapper) and cleaning up\nsome of the endpoint namespaces. This change is backwards compatible. Clients wishing to move\ntowards the new response format can either use the new endpoint names or send the following header:\n\n Accept: application/vnd.nsq version=1.0\n\nOther changes including officially bumping the character limit for topic and channel names to 64\n(thanks @svmehta), making the `REQ` timeout limit configurable in `nsqd` (thanks @AlphaB), and\ncompiling static asset dependencies into `nsqadmin` to simplify deployment (thanks @crossjam).\n\nFinally, `to_nsq` was added to the suite of bundled apps. It takes a stdin stream and publishes to\n`nsqd`, an extremely flexible solution (thanks @matryer)!\n\nAs for bugs, they're mostly minor, see the pull requests referenced in the section below for\ndetails.\n\nNew Features / Enhancements:\n\n * #304 - apps: added `to_nsq` for piping stdin to NSQ (thanks @matryer)\n * #406 - `nsqadmin`: embed external static asset dependencies (thanks @crossjam)\n * #389 - apps: report app name and version via `user_agent`\n * #378/#390 - `nsqd`: improve in-flight message handling (6% faster, GC reduction)\n * #356/#370/#386 - `nsqd`: introduce `AUTH`\n * #358 - increase topic/channel name max length to 64 (thanks @svmehta)\n * #357 - remove internal `go-nsq` dependencies (GC reduction)\n * #330/#366 - version HTTP endpoints, simplify response format\n * #352 - `nsqd`: make `REQ` timeout limit configurable (thanks @AlphaB)\n * #340 - `nsqd`: bump perks dependency (E2E performance improvement, see 25086e4)\n\nBugs:\n\n * #384 - `nsqd`: fix statsd GC time reporting\n * #407 - `nsqd`: fix double `TOUCH` and use of client's configured msg timeout\n * #392 - `nsqadmin`: fix HTTPS warning (thanks @juliangruber)\n * #383 - `nsqlookupd`: fix race on last update timestamp\n * #385 - `nsqd`: properly handle empty `FIN`\n * #365 - `nsqd`: fix `IDENTIFY` `msg_timeout` response (thanks @visionmedia)\n * #345 - `nsq_to_file`: set proper permissions on new directories (thanks @bschwartz)\n * #338 - `nsqd`: fix windows diskqueue filenames (thanks @politician)\n\n### 0.2.28 - 2014-04-28\n\n**Upgrading from 0.2.27**: No backwards incompatible changes. We've deprecated the `short_id`\nand `long_id` options in the `IDENTIFY` command in favor of `client_id` and `hostname`, which\nmore accurately reflect the data typically used.\n\nThis release includes a few important new features, in particular enhanced `nsqd`\nTLS support thanks to a big contribution by @chrisroberts.\n\nYou can now *require* that clients negotiate TLS with `--tls-required` and you can configure a\nclient certificate policy via `--tls-client-auth-policy` (`require` or `require-verify`):\n\n * `require` - the client must offer a certificate, otherwise rejected\n * `require-verify` - the client must offer a valid certificate according to the default CA or\n the chain specified by `--tls-root-ca-file`, otherwise rejected\n\nThis can be used as a form of client authentication.\n\nAdditionally, `nsqd` is now structured such that it is importable in other Go applications\nvia `github.com/nsqio/nsq/nsqd`, thanks to @kzvezdarov.\n\nFinally, thanks to @paddyforan, `nsq_to_file` can now archive *multiple* topics or\noptionally archive *all* discovered topics (by specifying no `--topic` params\nand using `--lookupd-http-address`).\n\nNew Features / Enhancements:\n\n * #334 - `nsq_to_file` can archive many topics (thanks @paddyforan)\n * #327 - add `nsqd` TLS client certificate verification policy, ability\n to require TLS, and HTTPS support (thanks @chrisroberts)\n * #325 - make `nsqd` importable (`github.com/nsqio/nsq/nsqd`) (thanks @kzvezdarov)\n * #321 - improve `IDENTIFY` options (replace `short_id` and `long_id` with\n `client_id` and `hostname`)\n * #319 - allow path separator in `nsq_to_file` filenames (thanks @jsocol)\n * #324 - display memory depth and total depth in `nsq_stat`\n\nBug Fixes:\n\n * nsqio/go-nsq#19 and nsqio/go-nsq#29 - fix deadlocks on `nsq.Reader` connection close/exit, this\n impacts the utilities packaged with the NSQ binary\n distribution such as `nsq_to_file`, `nsq_to_http`,\n `nsq_to_nsq` and `nsq_tail`.\n * #329 - use heartbeat interval for write deadline\n * #321/#326 - improve benchmarking tests\n * #315/#318 - fix test data races / flakiness\n\n### 0.2.27 - 2014-02-17\n\n**Upgrading from 0.2.26**: No backwards incompatible changes. We deprecated `--max-message-size`\nin favor of `--max-msg-size` for consistency with the rest of the flag names.\n\nIMPORTANT: this is another quick bug-fix release to address an issue in `nsqadmin` where templates\nwere incompatible with older versions of Go (pre-1.2).\n\n * #306 - fix `nsqadmin` template compatibility (and formatting)\n * #310 - fix `nsqadmin` behavior when E2E stats are disabled\n * #309 - fix `nsqadmin` `INVALID_ERROR` on node page tombstone link\n * #311/#312 - fix `nsqd` client metadata race condition and test flakiness\n * #314 - fix `nsqd` test races (run w/ `-race` and `GOMAXPROCS=4`) deprecate `--max-message-size`\n\n### 0.2.26 - 2014-02-06\n\n**Upgrading from 0.2.25**: No backwards incompatible changes.\n\nIMPORTANT: this is a quick bug-fix release to address a regression identified in `0.2.25` where\n`statsd` prefixes were broken when using the default (or any) prefix that contained a `%s` for\nautomatic host replacement.\n\n * #303 - fix `nsqd` `--statsd-prefix` when using `%s` host replacement\n\n### 0.2.25 - 2014-02-05\n\n**Upgrading from 0.2.24**: No backwards incompatible changes.\n\nThis release adds several commonly requested features.\n\nFirst, thanks to [@elubow](https://twitter.com/elubow) you can now configure your clients to sample\nthe stream they're subscribed to. To read more about the details of the implementation see #286 and\nthe original discussion in #223. Eric also contributed an improvement to `nsq_tail` to add\nthe ability to tail the last `N` messages and exit.\n\nWe added config file support ([TOML](https://github.com/mojombo/toml/blob/master/README.md)) for\n`nsqd`, `nsqlookupd`, and `nsqadmin` - providing even more deployment flexibility. Example configs\nare in the `contrib` directory. Command line arguments override the equivalent option in the config\nfile.\n\nWe added the ability to pause a *topic* (it is already possible to pause individual *channels*).\nThis functionality stops all message flow from topic to channel for *all channels* of a topic,\nqueueing at the topic level. This enables all kinds of interesting possibilities like atomic\nchannel renames and trivial infrastructure wide operations.\n\nFinally, we now compile the static assets used by `nsqadmin` into the binary, simplifying\ndeployment. This means that `--template-dir` is now deprecated and will be removed in a future\nrelease and you can remove the templates you previously deployed and maintained.\n\nNew Features / Enhancements:\n\n * #286 - add client `IDENTIFY` option to sample a % of messages\n * #279 - add TOML config file support to `nsqd`, `nsqlookupd`, and `nsqadmin`\n * #263 - add ability to pause a topic\n * #291 - compile templates into `nsqadmin` binary\n * #285/#288 - `nsq_tail` support for `-n #` to get recent # messages\n * #287/#294 - display client `IDENTIFY` attributes in `nsqadmin` (sample rate, TLS, compression)\n * #189/#296 - add client user agent to `nsqadmin``\n * #297 - add `nsq_to_nsq` JSON message filtering options\n\n### 0.2.24 - 2013-12-07\n\n**Upgrading from 0.2.23**: No backwards incompatible changes. However, as you'll see below, quite a\nfew command line flags to the utility apps (`nsq_to_http`, `nsq_to_file`, `nsq_to_http`) were\ndeprecated and will be removed in the next release. Please use this release to transition over to\nthe new ones.\n\nNOTE: we are now publishing additional binaries built against go1.2\n\nThe most prominent addition is the tracking of end-to-end message processing percentiles. This\nmeasures the amount of time it's taking from `PUB` to `FIN` per topic/channel. The percentiles are\nconfigurable and, because there is *some* overhead in collecting this data, it can be turned off\nentirely. Please see [the section in the docs](https://nsq.io/components/nsqd.html) for\nimplementation details.\n\nAdditionally, the utility apps received comprehensive support for all configurable reader options\n(including compression, which was previously missing). This necessitated a bit of command line flag\ncleanup, as follows:\n\n#### nsq_to_file\n\n * deprecated `--gzip-compression` in favor of `--gzip-level`\n * deprecated `--verbose` in favor of `--reader-opt=verbose`\n\n#### nsq_to_http\n\n * deprecated `--throttle-fraction` in favor of `--sample`\n * deprecated `--http-timeout-ms` in favor of `--http-timeout` (which is a\n *duration* flag)\n * deprecated `--verbose` in favor of `--reader-opt=verbose`\n * deprecated `--max-backoff-duration` in favor of\n `--reader-opt=max_backoff_duration=X`\n\n#### nsq_to_nsq\n\n * deprecated `--verbose` in favor of `--reader-opt=verbose`\n * deprecated `--max-backoff-duration` in favor of\n `--reader-opt=max_backoff_duration=X`\n\nNew Features / Enhancements:\n\n * #280 - add end-to-end message processing latency metrics\n * #267 - comprehensive reader command line flags for utilities\n\n### 0.2.23 - 2013-10-21\n\n**Upgrading from 0.2.22**: No backwards incompatible changes.\n\nWe now use [godep](https://github.com/kr/godep) in order to achieve reproducible builds with pinned\ndependencies. If you're on go1.1+ you can now just use `godep get github.com/nsqio/nsq/...`.\n\nThis release includes `nsqd` protocol compression feature negotiation.\n[Snappy](https://code.google.com/p/snappy/) and [Deflate](http://en.wikipedia.org/wiki/DEFLATE) are\nsupported, clients can choose their preferred format.\n\n`--statsd-prefix` can now be used to modify the prefix for the `statsd` keys generated by `nsqd`.\nThis is useful if you want to add datacenter prefixes or remove the default host prefix.\n\nFinally, this release includes a \"bug\" fix that reduces CPU usage for `nsqd` with many clients by\nchoosing a more reasonable default for a timer used in client output buffering. For more details\nsee #236.\n\nNew Features / Enhancements:\n\n * #266 - use godep for reproducible builds\n * #229 - compression (Snappy/Deflate) feature negotiation\n * #241 - binary support for HTTP /mput\n * #269 - add --statsd-prefix flag\n\nBug Fixes:\n\n * #278 - fix nsqd race for client subscription cleanup (thanks @simplereach)\n * #277 - fix nsqadmin counter page\n * #275 - stop accessing simplejson internals\n * #274 - nsqd channel pause state lost during unclean restart (thanks @hailocab)\n * #236 - reduce \"idle\" CPU usage by 90% with large # of clients\n\n### 0.2.22 - 2013-08-26\n\n**Upgrading from 0.2.21**: message timestamps are now officially nanoseconds. The protocol docs\nalways stated this however `nsqd` was actually sending seconds. This may cause some compatibility\nissues for client libraries/clients that were taking advantage of this field.\n\nThis release also introduces support for TLS feature negotiation in `nsqd`. Clients can optionally\nenable TLS by using the appropriate handshake via the `IDENTIFY` command. See #227.\n\nSignificant improvements were made to the HTTP publish endpoints and in flight message handling to\nreduce GC pressure and eliminate memory abuse vectors. See #242, #239, and #245.\n\nThis release also includes a new utility `nsq_to_nsq` for performant, low-latency, copying of an NSQ\ntopic over the TCP protocol.\n\nFinally, a whole suite of debug HTTP endpoints were added (and consolidated) under the\n`/debug/pprof` namespace. See #238, #248, and #252. As a result `nsqd` now supports *direct*\nprofiling via Go's `pprof` tool, ie:\n\n $ go tool pprof --web http://ip.address:4151/debug/pprof/heap\n\nNew Features / Enhancements:\n\n * #227 - TLS feature negotiation\n * #238/#248/#252 - support for more HTTP debug endpoints\n * #256 - `nsqadmin` single node view (with GC/mem graphs)\n * #255 - `nsq_to_nsq` utility for copying a topic over TCP\n * #230 - `nsq_to_http` takes `--content-type` flag (thanks @michaelhood)\n * #228 - `nsqadmin` displays tombstoned topics in the `/nodes` list\n * #242/#239/#245 - reduced GC pressure for inflight and `/mput`\n\nBug Fixes:\n\n * #260 - `tombstone_topic_producer` action in `nsqadmin` missing node info\n * #244 - fix 64bit atomic alignment issues on 32bit platforms\n * #251 - respect configured limits for HTTP publishing\n * #247 - publish methods should not allow 0 length messages\n * #231/#259 - persist `nsqd` metadata on topic/channel changes\n * #237 - fix potential memory leaks with retained channel references\n * #232 - message timestamps are now nano\n * #228 - `nsqlookupd`/`nsqadmin` would display inactive nodes in `/nodes` list\n * #216 - fix edge cases in `nsq_to_file` that caused empty files\n\n### 0.2.21 - 2013-06-07\n\n**Upgrading from 0.2.20**: there are no backward incompatible changes in this release.\n\nThis release introduces a significant new client feature as well as a slew of consistency and\nrecovery improvements to diskqueue.\n\nFirst, we expanded the feature negotiation options for clients. There are many cases where you want\ndifferent output buffering semantics from `nsqd` to your client. You can now control both\noutput buffer size and the output buffer timeout via new fields in the `IDENTIFY` command. You can\neven disable output buffering if low latency is a priority.\n\nYou can now specify a duration between fsyncs via `--sync-timeout`. This is a far better way to\nmanage when the process fsyncs messages to disk (vs the existing `--sync-every` which is based on #\nof messages). `--sync-every` is now considered a deprecated option and will be removed in a future\nrelease.\n\nFinally, `0.2.20` introduced a significant regression in #176 where a topic would not write messages\nto its channels. It is recommended that all users running `0.2.20` upgrade to this release. For\nadditional information see #217.\n\nNew Features / Enhancements:\n\n * #214 - add --sync-timeout for time based fsync, improve when diskqueue syncs\n * #196 - client configurable output buffering\n * #190 - nsq_tail generates a random #ephemeral channel\n\nBug Fixes:\n\n * #218/#220 - expose --statsd-interval for nsqadmin to handle non 60s statsd intervals\n * #217 - fix new topic channel creation regression from #176 (thanks @elubow)\n * #212 - dont use port in nsqadmin cookies\n * #214 - dont open diskqueue writeFile with O_APPEND\n * #203/#211 - diskqueue depth accounting consistency\n * #207 - failure to write a heartbeat is fatal / reduce error log noise\n * #206 - use broadcast address for statsd prefix\n * #205 - cleanup example utils exit\n\n### 0.2.20 - 2013-05-13\n\n**Upgrading from 0.2.19**: there are no backward incompatible changes in this release.\n\nThis release adds a couple of convenient features (such as adding the ability to empty a *topic*)\nand continues our work to reduce garbage produced at runtime to relieve GC pressure in the Go\nruntime.\n\n`nsqd` now has two new flags to control the max value clients can use to set their heartbeat\ninterval as well as adjust a clients maximum RDY count. This is all set/communicated via `IDENTIFY`.\n\n`nsqadmin` now displays `nsqd` -> `nsqlookupd` connections in the \"nodes\" view. This is useful for\nvisualizing how the topology is connected as well as situations where `--broadcast-address` is being\nused incorrectly.\n\n`nsq_to_http` now has a \"host pool\" mode where upstream state will be adjusted based on\nsuccessful/failed requests and for failures, upstreams will be exponentially backed off. This is an\nincredibly useful routing mode.\n\nAs for bugs, we fixed an issue where \"fatal\" client errors were not actually being treated as fatal.\nUnder certain conditions deleting a topic would not clean up all of its files on disk. There was a\nreported issue where the `--data-path` was not writable by the process and this was only discovered\nafter message flow began. We added a writability check at startup to improve feedback. Finally.\n`deferred_count` was being sent as a counter value to statsd, it should be a gauge.\n\nNew Features / Enhancements:\n\n * #197 - nsqadmin nodes list improvements (show nsqd -> lookupd conns)\n * #192 - add golang runtime version to daemon version output\n * #183 - ability to empty a topic\n * #176 - optimizations to reduce garbage, copying, locking\n * #184 - add table headers to nsqadmin channel view (thanks @elubow)\n * #174/#186 - nsq_to_http hostpool mode and backoff control\n * #173/#187 - nsq_stat utility for command line introspection\n * #175 - add nsqd --max-rdy-count configuration option\n * #178 - add nsqd --max-heartbeat-interval configuration option\n\nBug Fixes:\n\n * #198 - fix fatal errors not actually being fatal\n * #195 - fix delete topic does not delete all diskqueue files\n * #193 - fix data race in channel requeue\n * #185 - ensure that --data-path is writable on startup\n * #182 - fix topic deletion ordering to prevent race conditions with lookupd/diskqueue\n * #179 - deferred_count as gauge for statsd\n * #173/#188/#191 - fix nsqadmin counter template error; fix nsqadmin displaying negative rates\n\n### 0.2.19 - 2013-04-11\n\n**Upgrading from 0.2.18**: there are no backward incompatible changes in this release.\n\nThis release is a small release that introduces one major client side feature and resolves one\ncritical bug.\n\n`nsqd` clients can now configure their own heartbeat interval. This is important because as of\n`0.2.18` *all* clients (including producers) received heartbeats by default. In certain cases\nreceiving a heartbeat complicated \"simple\" clients that just wanted to produce messages and not\nhandle asynchronous responses. This gives flexibility for the client to decide how it would like\nbehave.\n\nA critical bug was discovered where emptying a channel would leave client in-flight state\ninconsistent (it would not zero) which limited deliverability of messages to those clients.\n\nNew Features / Enhancements:\n\n * #167 - 'go get' compatibility\n * #158 - allow nsqd clients to configure (or disable) heartbeats\n\nBug Fixes:\n\n * #171 - fix race conditions identified testing against go 1.1 (scheduler improvements)\n * #160 - empty channel left in-flight count inconsistent (thanks @dmarkham)\n\n### 0.2.18 - 2013-02-28\n\n**Upgrading from 0.2.17**: all V2 clients of nsqd now receive heartbeats (previously only clients\nthat subscribed would receive heartbeats, excluding TCP *producers*).\n\n**Upgrading from 0.2.16**: follow the notes in the 0.2.17 changelog for upgrading from 0.2.16.\n\nBeyond the important note above regarding heartbeats this release includes `nsq_tail`, an extremely\nuseful utility application that can be used to introspect a topic on the command line. If statsd is\nenabled (and graphite in `nsqadmin`) we added the ability to retrieve rates for display in\n`nsqadmin`.\n\nWe resolved a few critical issues with data consistency in `nsqlookupd` when channels and topics are\ndeleted. First, deleting a topic would cause that producer to disappear from `nsqlookupd` for all\ntopics. Second, deleting a channel would cause that producer to disappear from the topic list in\n`nsqlookupd`.\n\nNew Features / Enhancements:\n\n * #131 - all V2 nsqd clients get heartbeats\n * #154 - nsq_tail example reader\n * #143 - display message rates in nsqadmin\n\nBug Fixes:\n\n * #148 - store tombstone data per registration in nsqlookupd\n * #153 - fix large graph formulas in nsqadmin\n * #150/#151 - fix topics disappearing from nsqlookupd when channels are deleted\n\n### 0.2.17 - 2013-02-07\n\n**Upgrading from 0.2.16**: IDENTIFY and SUB now return success responses (they previously only\nresponded to errors). The official Go and Python libraries are forwards/backwards compatible with\nthis change however 3rd party client libraries may not be.\n\n**Upgrading from 0.2.15**: in #132 deprecations in SUB were removed as well as support for the old,\nline oriented, `nsqd` metadata file format. For these reasons you should upgrade to `0.2.16` first.\n\nNew Features / Enhancements:\n\n * #119 - add TOUCH command to nsqd\n * #142 - add --broadcast-address flag to nsqd/nsqadmin (thanks @dustismo)\n * #135 - atomic MPUB\n * #133 - improved protocol fatal error handling and responses; IDENTIFY/SUB success responses\n * #118 - switch nsqadmin actions to POST and require confirmation\n * #117/#147 - nsqadmin action POST notifications\n * #122 - configurable msg size limits\n * #132 - deprecate identify in SUB and old nsqd metadata file format\n\nBug Fixes:\n\n * #144 - empty channel should clear inflight/deferred messages\n * #140 - fix MPUB protocol documentation\n * #139 - fix nsqadmin handling of legacy statsd prefixes for graphs\n * #138/#145 - fix nsqadmin action redirect handling\n * #134 - nsqd to nsqlookupd registration fixes\n * #129 - nsq_to_file gzip file versioning\n * #106 - nsqlookupd topic producer tombstones\n * #100 - sane handling of diskqueue read errors\n * #123/#125 - fix notify related exit deadlock\n\n### 0.2.16 - 2013-01-07\n\n**Upgrading from 0.2.15**: there are no backward incompatible changes in this release.\n\nHowever, this release introduces the `IDENTIFY` command (which supersedes sending\nmetadata along with `SUB`) for clients of `nsqd`. The old functionality will be\nremoved in a future release.\n\n * #114 persist paused channels through restart\n * #121 fix typo preventing compile of bench_reader (thanks @datastream)\n * #120 fix nsqd crash when empty command is sent (thanks @michaelhood)\n * #115 nsq_to_file --filename-format --datetime-format parameter and fix\n * #101 fix topic/channel delete operations ordering\n * #98 nsqadmin fixes when not using lookupd\n * #90/#108 performance optimizations / IDENTIFY protocol support in nsqd. For\n a single consumer of small messages (< 4k) increases throughput ~400% and\n reduces # of allocations ~30%.\n * #105 strftime compatible datetime format\n * #103 nsq_to_http handler logging\n * #102 compatibility with Go tip\n * #99 nsq_to_file --gzip flag\n * #95 proxy graphite requests through nsqadmin\n * #93 fix nqd API response for no topics\n * #92 graph rendering options\n * #86 nsq_to_http Content-Length headers\n * #89 gopkg doc updates\n * #88 move pynsq to it's own repo\n * #81/#87 reader improvements / introduced MPUB. Fix bug for mem-queue-size < 10\n * #76 statsd/graphite support\n * #75 administrative ability to create topics and channels\n\n### 0.2.15 - 2012-10-25\n\n * #84 fix lookupd hanging on to ephemeral channels w/ no producers\n * #82 add /counter page to nsqadmin\n * #80 message size benchmark\n * #78 send Content-Length for nsq_to_http requests\n * #57/#83 documentation updates\n\n### 0.2.14 - 2012-10-19\n\n * #77 ability to pause a channel (includes bugfix for message pump/diskqueue)\n * #74 propagate all topic changes to lookupd\n * #65 create binary releases\n\n### 0.2.13 - 2012-10-15\n\n * #70 deadlined nsq_to_http outbound requests\n * #69/#72 improved nsq_to_file sync strategy\n * #58 diskqueue metadata bug and refactoring\n\n### 0.2.12 - 2012-10-10\n\n * #63 consolidated protocol V1 into V2 and fixed PUB bug\n * #61 added a makefile for simpler building\n * #55 allow topic/channel names with `.`\n * combined versions for all binaries\n\n### 0.2.7 - 0.2.11\n\n * Initial public release.\n\n## go-nsq Client Library\n\n * #264 moved **go-nsq** to its own [repository](https://github.com/nsqio/go-nsq)\n\n## pynsq Python Client Library\n\n * #88 moved **pynsq** to its own [repository](https://github.com/nsqio/pynsq)\n" }, { "path": "Dockerfile", "content": "FROM golang:alpine AS build\n\nRUN apk update && apk add make gcc musl-dev\n\nRUN mkdir -p /go/src/github.com/nsqio/nsq\nCOPY . /go/src/github.com/nsqio/nsq\nWORKDIR /go/src/github.com/nsqio/nsq\n\nRUN CGO_ENABLED=0 make BLDDIR=/tmp/nsq PREFIX=/opt/nsq BLDFLAGS='-ldflags=\"-s -w\"' install\n\nFROM alpine:latest\n\nEXPOSE 4150 4151 4160 4161 4170 4171\n\nRUN mkdir -p /data\nWORKDIR /data\n\n# Optional volumes (explicitly configure with \"docker run -v ...\")\n# /data - used by nsqd for persistent storage across restarts\n# /etc/ssl/certs - for SSL Root CA certificates from host\n\nCOPY --from=build /opt/nsq/bin/ /usr/local/bin/\nRUN ln -s /usr/local/bin/*nsq* / \\\n && ln -s /usr/local/bin/*nsq* /bin/\n" }, { "path": "LICENSE", "content": "Permission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE.\n" }, { "path": "Makefile", "content": "PREFIX=/usr/local\nBINDIR=${PREFIX}/bin\nDESTDIR=\nBLDDIR=build\nBLDFLAGS=\nEXT=\nifeq (${GOOS},windows)\n EXT=.exe\nendif\n\nAPPS = nsqd nsqlookupd nsqadmin nsq_to_nsq nsq_to_file nsq_to_http nsq_tail nsq_stat to_nsq\nall: $(APPS)\n\n$(BLDDIR)/nsqd: $(wildcard apps/nsqd/*.go nsqd/*.go nsq/*.go internal/*/*.go)\n$(BLDDIR)/nsqlookupd: $(wildcard apps/nsqlookupd/*.go nsqlookupd/*.go nsq/*.go internal/*/*.go)\n$(BLDDIR)/nsqadmin: $(wildcard apps/nsqadmin/*.go nsqadmin/*.go nsqadmin/templates/*.go internal/*/*.go)\n$(BLDDIR)/nsq_to_nsq: $(wildcard apps/nsq_to_nsq/*.go nsq/*.go internal/*/*.go)\n$(BLDDIR)/nsq_to_file: $(wildcard apps/nsq_to_file/*.go nsq/*.go internal/*/*.go)\n$(BLDDIR)/nsq_to_http: $(wildcard apps/nsq_to_http/*.go nsq/*.go internal/*/*.go)\n$(BLDDIR)/nsq_tail: $(wildcard apps/nsq_tail/*.go nsq/*.go internal/*/*.go)\n$(BLDDIR)/nsq_stat: $(wildcard apps/nsq_stat/*.go internal/*/*.go)\n$(BLDDIR)/to_nsq: $(wildcard apps/to_nsq/*.go internal/*/*.go)\n\n$(BLDDIR)/%:\n\t@mkdir -p $(dir $@)\n\tgo build ${BLDFLAGS} -o $@ ./apps/$*\n\n$(APPS): %: $(BLDDIR)/%\n\nclean:\n\trm -fr $(BLDDIR)\n\n.PHONY: install clean all\n.PHONY: $(APPS)\n\ninstall: $(APPS)\n\tinstall -m 755 -d ${DESTDIR}${BINDIR}\n\tfor APP in $^ ; do install -m 755 ${BLDDIR}/$$APP ${DESTDIR}${BINDIR}/$$APP${EXT} ; done\n" }, { "path": "README.md", "content": "

\n\n

    \n
  • Source: https://github.com/nsqio/nsq\n
  • Issues: https://github.com/nsqio/nsq/issues\n
  • Mailing List: nsq-users@googlegroups.com\n
  • IRC: #nsq on freenode\n
  • Docs: https://nsq.io\n
  • Twitter: @nsqio\n
\n

\n\n[![Build Status](https://github.com/nsqio/nsq/workflows/tests/badge.svg)](https://github.com/nsqio/nsq/actions) [![GitHub release](https://img.shields.io/github/release/nsqio/nsq.svg)](https://github.com/nsqio/nsq/releases/latest) [![Coverage Status](https://coveralls.io/repos/github/nsqio/nsq/badge.svg?branch=master)](https://coveralls.io/github/nsqio/nsq?branch=master)\n\n**NSQ** is a realtime distributed messaging platform designed to operate at scale, handling\nbillions of messages per day.\n\nIt promotes *distributed* and *decentralized* topologies without single points of failure,\nenabling fault tolerance and high availability coupled with a reliable message delivery\nguarantee. See [features & guarantees][features_guarantees].\n\nOperationally, **NSQ** is easy to configure and deploy (all parameters are specified on the command\nline and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to\ndata format (messages can be JSON, MsgPack, Protocol Buffers, or anything else). Official Go and\nPython libraries are available out of the box (as well as many other [client\nlibraries][client_libraries]), and if you're interested in building your own, there's a [protocol\nspec][protocol].\n\nWe publish [binary releases][installing] for Linux, Darwin, FreeBSD and Windows, as well as an official [Docker image][docker_deployment].\n\nNOTE: master is our *development* branch and may not be stable at all times.\n\n## In Production\n\n  \n  \n  \n  \n  \n
\n\n  \n  \n  \n  \n  \n
\n\n  \n  \n  \n  \n  \n
\n\n  \n  \n  \n  \n  \n
\n\n  \n  \n  \n  \n  \n
\n\n  \n  \n  \n
\n
\n\n## Code of Conduct\n\nHelp us keep NSQ open and inclusive. Please read and follow our [Code of Conduct](CODE_OF_CONDUCT.md).\n\n## Authors\n\nNSQ was designed and developed by Matt Reiferson ([@imsnakes][snakes_twitter]) and Jehiah Czebotar\n([@jehiah][jehiah_twitter]) but wouldn't have been possible without the support of [Bitly][bitly],\nmaintainers ([Pierce Lopez][pierce_github]), and all our [contributors][contributors].\n\nLogo created by Wolasi Konu ([@kisalow][wolasi_twitter]).\n\n[protocol]: https://nsq.io/clients/tcp_protocol_spec.html\n[installing]: https://nsq.io/deployment/installing.html\n[docker_deployment]: https://nsq.io/deployment/docker.html\n[snakes_twitter]: https://twitter.com/imsnakes\n[jehiah_twitter]: https://twitter.com/jehiah\n[bitly]: https://bitly.com\n[features_guarantees]: https://nsq.io/overview/features_and_guarantees.html\n[contributors]: https://github.com/nsqio/nsq/graphs/contributors\n[client_libraries]: https://nsq.io/clients/client_libraries.html\n[wolasi_twitter]: https://twitter.com/kisalow\n[pierce_github]: https://github.com/ploxiln\n" }, { "path": "apps/nsq_stat/nsq_stat.go", "content": "// This is a utility application that polls /stats for all the producers\n// of the specified topic/channel and displays aggregate stats\n\npackage main\n\nimport (\n\t\"errors\"\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"os/signal\"\n\t\"strconv\"\n\t\"strings\"\n\t\"syscall\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/app\"\n\t\"github.com/nsqio/nsq/internal/clusterinfo\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nvar (\n\tshowVersion = flag.Bool(\"version\", false, \"print version\")\n\ttopic = flag.String(\"topic\", \"\", \"NSQ topic\")\n\tchannel = flag.String(\"channel\", \"\", \"NSQ channel\")\n\tinterval = flag.Duration(\"interval\", 2*time.Second, \"duration of time between polling/printing output\")\n\thttpConnectTimeout = flag.Duration(\"http-client-connect-timeout\", 2*time.Second, \"timeout for HTTP connect\")\n\thttpRequestTimeout = flag.Duration(\"http-client-request-timeout\", 5*time.Second, \"timeout for HTTP request\")\n\tcountNum = numValue{}\n\tnsqdHTTPAddrs = app.StringArray{}\n\tlookupdHTTPAddrs = app.StringArray{}\n)\n\ntype numValue struct {\n\tisSet bool\n\tvalue int\n}\n\nfunc (nv *numValue) String() string { return \"N\" }\n\nfunc (nv *numValue) Set(s string) error {\n\tvalue, err := strconv.ParseInt(s, 10, 32)\n\tif err != nil {\n\t\treturn err\n\t}\n\tnv.value = int(value)\n\tnv.isSet = true\n\treturn nil\n}\n\nfunc init() {\n\tflag.Var(&nsqdHTTPAddrs, \"nsqd-http-address\", \"nsqd HTTP address (may be given multiple times)\")\n\tflag.Var(&lookupdHTTPAddrs, \"lookupd-http-address\", \"lookupd HTTP address (may be given multiple times)\")\n\tflag.Var(&countNum, \"count\", \"number of reports\")\n}\n\nfunc statLoop(interval time.Duration, connectTimeout time.Duration, requestTimeout time.Duration,\n\ttopic string, channel string, nsqdTCPAddrs []string, lookupdHTTPAddrs []string) {\n\tci := clusterinfo.New(nil, http_api.NewClient(nil, connectTimeout, requestTimeout))\n\tvar o *clusterinfo.ChannelStats\n\tfor i := 0; !countNum.isSet || countNum.value >= i; i++ {\n\t\tvar producers clusterinfo.Producers\n\t\tvar err error\n\n\t\tif len(lookupdHTTPAddrs) != 0 {\n\t\t\tproducers, err = ci.GetLookupdTopicProducers(topic, lookupdHTTPAddrs)\n\t\t} else {\n\t\t\tproducers, err = ci.GetNSQDTopicProducers(topic, nsqdHTTPAddrs)\n\t\t}\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"ERROR: failed to get topic producers - %s\", err)\n\t\t}\n\n\t\t_, channelStats, err := ci.GetNSQDStats(producers, topic, channel, false)\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"ERROR: failed to get nsqd stats - %s\", err)\n\t\t}\n\n\t\tc, ok := channelStats[channel]\n\t\tif !ok {\n\t\t\tlog.Fatalf(\"ERROR: failed to find channel(%s) in stats metadata for topic(%s)\", channel, topic)\n\t\t}\n\n\t\tif i%25 == 0 {\n\t\t\tfmt.Printf(\"%s+%s+%s\\n\",\n\t\t\t\t\"------rate------\",\n\t\t\t\t\"----------------depth----------------\",\n\t\t\t\t\"--------------metadata---------------\")\n\t\t\tfmt.Printf(\"%7s %7s | %7s %7s %7s %5s %5s | %7s %7s %12s %7s\\n\",\n\t\t\t\t\"ingress\", \"egress\",\n\t\t\t\t\"total\", \"mem\", \"disk\", \"inflt\",\n\t\t\t\t\"def\", \"req\", \"t-o\", \"msgs\", \"clients\")\n\t\t}\n\n\t\tif o == nil {\n\t\t\to = c\n\t\t\ttime.Sleep(interval)\n\t\t\tcontinue\n\t\t}\n\n\t\t// TODO: paused\n\t\tfmt.Printf(\"%7d %7d | %7d %7d %7d %5d %5d | %7d %7d %12d %7d\\n\",\n\t\t\tint64(float64(c.MessageCount-o.MessageCount)/interval.Seconds()),\n\t\t\tint64(float64(c.MessageCount-o.MessageCount-(c.Depth-o.Depth))/interval.Seconds()),\n\t\t\tc.Depth,\n\t\t\tc.MemoryDepth,\n\t\t\tc.BackendDepth,\n\t\t\tc.InFlightCount,\n\t\t\tc.DeferredCount,\n\t\t\tc.RequeueCount,\n\t\t\tc.TimeoutCount,\n\t\t\tc.MessageCount,\n\t\t\tc.ClientCount)\n\n\t\to = c\n\t\ttime.Sleep(interval)\n\t}\n\tos.Exit(0)\n}\n\nfunc checkAddrs(addrs []string) error {\n\tfor _, a := range addrs {\n\t\tif strings.HasPrefix(a, \"http\") {\n\t\t\treturn errors.New(\"address should not contain scheme\")\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc main() {\n\tflag.Parse()\n\n\tif *showVersion {\n\t\tfmt.Printf(\"nsq_stat v%s\\n\", version.Binary)\n\t\treturn\n\t}\n\n\tif *topic == \"\" || *channel == \"\" {\n\t\tlog.Fatal(\"--topic and --channel are required\")\n\t}\n\n\tintvl := *interval\n\tif int64(intvl) <= 0 {\n\t\tlog.Fatal(\"--interval should be positive\")\n\t}\n\n\tconnectTimeout := *httpConnectTimeout\n\tif int64(connectTimeout) <= 0 {\n\t\tlog.Fatal(\"--http-client-connect-timeout should be positive\")\n\t}\n\n\trequestTimeout := *httpRequestTimeout\n\tif int64(requestTimeout) <= 0 {\n\t\tlog.Fatal(\"--http-client-request-timeout should be positive\")\n\t}\n\n\tif countNum.isSet && countNum.value <= 0 {\n\t\tlog.Fatal(\"--count should be positive\")\n\t}\n\n\tif len(nsqdHTTPAddrs) == 0 && len(lookupdHTTPAddrs) == 0 {\n\t\tlog.Fatal(\"--nsqd-http-address or --lookupd-http-address required\")\n\t}\n\tif len(nsqdHTTPAddrs) > 0 && len(lookupdHTTPAddrs) > 0 {\n\t\tlog.Fatal(\"use --nsqd-http-address or --lookupd-http-address not both\")\n\t}\n\n\tif err := checkAddrs(nsqdHTTPAddrs); err != nil {\n\t\tlog.Fatalf(\"--nsqd-http-address error - %s\", err)\n\t}\n\n\tif err := checkAddrs(lookupdHTTPAddrs); err != nil {\n\t\tlog.Fatalf(\"--lookupd-http-address error - %s\", err)\n\t}\n\n\ttermChan := make(chan os.Signal, 1)\n\tsignal.Notify(termChan, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM)\n\n\tgo statLoop(intvl, connectTimeout, requestTimeout, *topic, *channel, nsqdHTTPAddrs, lookupdHTTPAddrs)\n\n\t<-termChan\n}\n" }, { "path": "apps/nsq_tail/nsq_tail.go", "content": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"math/rand\"\n\t\"os\"\n\t\"os/signal\"\n\t\"syscall\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/app\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nvar (\n\tshowVersion = flag.Bool(\"version\", false, \"print version string\")\n\n\tchannel = flag.String(\"channel\", \"\", \"NSQ channel\")\n\tmaxInFlight = flag.Int(\"max-in-flight\", 200, \"max number of messages to allow in flight\")\n\ttotalMessages = flag.Int(\"n\", 0, \"total messages to show (will wait if starved)\")\n\tprintTopic = flag.Bool(\"print-topic\", false, \"print topic name where message was received\")\n\n\tnsqdTCPAddrs = app.StringArray{}\n\tlookupdHTTPAddrs = app.StringArray{}\n\ttopics = app.StringArray{}\n)\n\nfunc init() {\n\tflag.Var(&nsqdTCPAddrs, \"nsqd-tcp-address\", \"nsqd TCP address (may be given multiple times)\")\n\tflag.Var(&lookupdHTTPAddrs, \"lookupd-http-address\", \"lookupd HTTP address (may be given multiple times)\")\n\tflag.Var(&topics, \"topic\", \"NSQ topic (may be given multiple times)\")\n}\n\ntype TailHandler struct {\n\ttopicName string\n\ttotalMessages int\n\tmessagesShown int\n}\n\nfunc (th *TailHandler) HandleMessage(m *nsq.Message) error {\n\tth.messagesShown++\n\n\tif *printTopic {\n\t\t_, err := os.Stdout.WriteString(th.topicName)\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"ERROR: failed to write to os.Stdout - %s\", err)\n\t\t}\n\t\t_, err = os.Stdout.WriteString(\" | \")\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"ERROR: failed to write to os.Stdout - %s\", err)\n\t\t}\n\t}\n\n\t_, err := os.Stdout.Write(m.Body)\n\tif err != nil {\n\t\tlog.Fatalf(\"ERROR: failed to write to os.Stdout - %s\", err)\n\t}\n\t_, err = os.Stdout.WriteString(\"\\n\")\n\tif err != nil {\n\t\tlog.Fatalf(\"ERROR: failed to write to os.Stdout - %s\", err)\n\t}\n\tif th.totalMessages > 0 && th.messagesShown >= th.totalMessages {\n\t\tos.Exit(0)\n\t}\n\treturn nil\n}\n\nfunc main() {\n\tcfg := nsq.NewConfig()\n\n\tflag.Var(&nsq.ConfigFlag{cfg}, \"consumer-opt\", \"option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)\")\n\tflag.Parse()\n\n\tif *showVersion {\n\t\tfmt.Printf(\"nsq_tail v%s\\n\", version.Binary)\n\t\treturn\n\t}\n\n\tif *channel == \"\" {\n\t\trand.Seed(time.Now().UnixNano())\n\t\t*channel = fmt.Sprintf(\"tail%06d#ephemeral\", rand.Int()%999999)\n\t}\n\n\tif len(nsqdTCPAddrs) == 0 && len(lookupdHTTPAddrs) == 0 {\n\t\tlog.Fatal(\"--nsqd-tcp-address or --lookupd-http-address required\")\n\t}\n\tif len(nsqdTCPAddrs) > 0 && len(lookupdHTTPAddrs) > 0 {\n\t\tlog.Fatal(\"use --nsqd-tcp-address or --lookupd-http-address not both\")\n\t}\n\tif len(topics) == 0 {\n\t\tlog.Fatal(\"--topic required\")\n\t}\n\n\tsigChan := make(chan os.Signal, 1)\n\tsignal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)\n\n\t// Don't ask for more messages than we want\n\tif *totalMessages > 0 && *totalMessages < *maxInFlight {\n\t\t*maxInFlight = *totalMessages\n\t}\n\n\tcfg.UserAgent = fmt.Sprintf(\"nsq_tail/%s go-nsq/%s\", version.Binary, nsq.VERSION)\n\tcfg.MaxInFlight = *maxInFlight\n\n\tconsumers := []*nsq.Consumer{}\n\tfor i := 0; i < len(topics); i++ {\n\t\tlog.Printf(\"Adding consumer for topic: %s\\n\", topics[i])\n\n\t\tconsumer, err := nsq.NewConsumer(topics[i], *channel, cfg)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\n\t\tconsumer.AddHandler(&TailHandler{topicName: topics[i], totalMessages: *totalMessages})\n\n\t\terr = consumer.ConnectToNSQDs(nsqdTCPAddrs)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\n\t\terr = consumer.ConnectToNSQLookupds(lookupdHTTPAddrs)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\n\t\tconsumers = append(consumers, consumer)\n\t}\n\n\t<-sigChan\n\n\tfor _, consumer := range consumers {\n\t\tconsumer.Stop()\n\t}\n\tfor _, consumer := range consumers {\n\t\t<-consumer.StopChan\n\t}\n}\n" }, { "path": "apps/nsq_to_file/file_logger.go", "content": "package main\n\nimport (\n\t\"compress/gzip\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n\t\"path\"\n\t\"path/filepath\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype FileLogger struct {\n\tlogf lg.AppLogFunc\n\topts *Options\n\ttopic string\n\tconsumer *nsq.Consumer\n\n\tout *os.File\n\twriter io.Writer\n\tgzipWriter *gzip.Writer\n\tlogChan chan *nsq.Message\n\tfilenameFormat string\n\n\ttermChan chan bool\n\thupChan chan bool\n\n\t// for rotation\n\tfilename string\n\topenTime time.Time\n\tfilesize int64\n\trev uint\n}\n\nfunc NewFileLogger(logf lg.AppLogFunc, opts *Options, topic string, cfg *nsq.Config) (*FileLogger, error) {\n\tcomputedFilenameFormat, err := computeFilenameFormat(opts, topic)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tconsumer, err := nsq.NewConsumer(topic, opts.Channel, cfg)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tf := &FileLogger{\n\t\tlogf: logf,\n\t\topts: opts,\n\t\ttopic: topic,\n\t\tconsumer: consumer,\n\t\tlogChan: make(chan *nsq.Message, 1),\n\t\tfilenameFormat: computedFilenameFormat,\n\t\ttermChan: make(chan bool),\n\t\thupChan: make(chan bool),\n\t}\n\tconsumer.AddHandler(f)\n\n\terr = consumer.ConnectToNSQDs(opts.NSQDTCPAddrs)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\terr = consumer.ConnectToNSQLookupds(opts.NSQLookupdHTTPAddrs)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn f, nil\n}\n\nfunc (f *FileLogger) HandleMessage(m *nsq.Message) error {\n\tm.DisableAutoResponse()\n\tf.logChan <- m\n\treturn nil\n}\n\nfunc (f *FileLogger) router() {\n\tpos := 0\n\toutput := make([]*nsq.Message, f.opts.MaxInFlight)\n\tsync := false\n\tticker := time.NewTicker(f.opts.SyncInterval)\n\tcloseFile := false\n\texit := false\n\n\tfor {\n\t\tselect {\n\t\tcase <-f.consumer.StopChan:\n\t\t\tsync = true\n\t\t\tcloseFile = true\n\t\t\texit = true\n\t\tcase <-f.termChan:\n\t\t\tticker.Stop()\n\t\t\tf.consumer.Stop()\n\t\t\tsync = true\n\t\tcase <-f.hupChan:\n\t\t\tsync = true\n\t\t\tcloseFile = true\n\t\tcase <-ticker.C:\n\t\t\tif f.needsRotation() {\n\t\t\t\tif f.opts.SkipEmptyFiles {\n\t\t\t\t\tcloseFile = true\n\t\t\t\t} else {\n\t\t\t\t\tf.updateFile()\n\t\t\t\t}\n\t\t\t}\n\t\t\tsync = true\n\t\tcase m := <-f.logChan:\n\t\t\tif f.needsRotation() {\n\t\t\t\tf.updateFile()\n\t\t\t\tsync = true\n\t\t\t}\n\t\t\t_, err := f.Write(m.Body)\n\t\t\tif err != nil {\n\t\t\t\tf.logf(lg.FATAL, \"[%s/%s] writing message to disk: %s\", f.topic, f.opts.Channel, err)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t\t_, err = f.Write([]byte(\"\\n\"))\n\t\t\tif err != nil {\n\t\t\t\tf.logf(lg.FATAL, \"[%s/%s] writing newline to disk: %s\", f.topic, f.opts.Channel, err)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t\toutput[pos] = m\n\t\t\tpos++\n\t\t\tif pos == cap(output) {\n\t\t\t\tsync = true\n\t\t\t}\n\t\t}\n\n\t\tif sync || f.consumer.IsStarved() {\n\t\t\tif pos > 0 {\n\t\t\t\tf.logf(lg.INFO, \"[%s/%s] syncing %d records to disk\", f.topic, f.opts.Channel, pos)\n\t\t\t\terr := f.Sync()\n\t\t\t\tif err != nil {\n\t\t\t\t\tf.logf(lg.FATAL, \"[%s/%s] failed syncing messages: %s\", f.topic, f.opts.Channel, err)\n\t\t\t\t\tos.Exit(1)\n\t\t\t\t}\n\t\t\t\tfor pos > 0 {\n\t\t\t\t\tpos--\n\t\t\t\t\tm := output[pos]\n\t\t\t\t\tm.Finish()\n\t\t\t\t\toutput[pos] = nil\n\t\t\t\t}\n\t\t\t}\n\t\t\tsync = false\n\t\t}\n\n\t\tif closeFile {\n\t\t\tf.Close()\n\t\t\tcloseFile = false\n\t\t}\n\n\t\tif exit {\n\t\t\tbreak\n\t\t}\n\t}\n}\n\nfunc (f *FileLogger) Close() {\n\tif f.out == nil {\n\t\treturn\n\t}\n\n\tif f.gzipWriter != nil {\n\t\terr := f.gzipWriter.Close()\n\t\tif err != nil {\n\t\t\tf.logf(lg.FATAL, \"[%s/%s] failed to close GZIP writer: %s\", f.topic, f.opts.Channel, err)\n\t\t\tos.Exit(1)\n\t\t}\n\t}\n\terr := f.out.Sync()\n\tif err != nil {\n\t\tf.logf(lg.FATAL, \"[%s/%s] failed to fsync output file: %s\", f.topic, f.opts.Channel, err)\n\t\tos.Exit(1)\n\t}\n\terr = f.out.Close()\n\tif err != nil {\n\t\tf.logf(lg.FATAL, \"[%s/%s] failed to close output file: %s\", f.topic, f.opts.Channel, err)\n\t\tos.Exit(1)\n\t}\n\n\t// Move file from work dir to output dir if necessary, taking care not\n\t// to overwrite existing files\n\tif f.opts.WorkDir != f.opts.OutputDir {\n\t\tsrc := f.out.Name()\n\t\tdst := filepath.Join(f.opts.OutputDir, strings.TrimPrefix(src, f.opts.WorkDir))\n\n\t\t// Optimistic rename\n\t\tf.logf(lg.INFO, \"[%s/%s] moving finished file %s to %s\", f.topic, f.opts.Channel, src, dst)\n\t\terr := exclusiveRename(src, dst)\n\t\tif err == nil {\n\t\t\treturn\n\t\t} else if !os.IsExist(err) {\n\t\t\tf.logf(lg.FATAL, \"[%s/%s] unable to move file from %s to %s: %s\", f.topic, f.opts.Channel, src, dst, err)\n\t\t\tos.Exit(1)\n\t\t}\n\n\t\t// Optimistic rename failed, so we need to generate a new\n\t\t// destination file name by bumping the revision number.\n\t\t_, filenameTmpl := filepath.Split(f.filename)\n\t\tdstDir, _ := filepath.Split(dst)\n\t\tdstTmpl := filepath.Join(dstDir, filenameTmpl)\n\n\t\tfor i := f.rev + 1; ; i++ {\n\t\t\tf.logf(lg.WARN, \"[%s/%s] destination file already exists: %s\", f.topic, f.opts.Channel, dst)\n\t\t\tdst := strings.Replace(dstTmpl, \"\", fmt.Sprintf(\"-%06d\", i), -1)\n\t\t\terr := exclusiveRename(src, dst)\n\t\t\tif err != nil {\n\t\t\t\tif os.IsExist(err) {\n\t\t\t\t\tcontinue // next rev\n\t\t\t\t}\n\t\t\t\tf.logf(lg.FATAL, \"[%s/%s] unable to rename file from %s to %s: %s\", f.topic, f.opts.Channel, src, dst, err)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t\tf.logf(lg.INFO, \"[%s/%s] renamed finished file %s to %s to avoid overwrite\", f.topic, f.opts.Channel, src, dst)\n\t\t\tbreak\n\t\t}\n\t}\n\n\tf.out = nil\n}\n\nfunc (f *FileLogger) Write(p []byte) (int, error) {\n\tn, err := f.writer.Write(p)\n\tf.filesize += int64(n)\n\treturn n, err\n}\n\nfunc (f *FileLogger) Sync() error {\n\tvar err error\n\tif f.gzipWriter != nil {\n\t\t// finish current gzip stream and start a new one (concatenated)\n\t\t// gzip stream trailer has checksum, and can indicate which messages were ACKed\n\t\terr = f.gzipWriter.Close()\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\terr = f.out.Sync()\n\t\tf.gzipWriter, _ = gzip.NewWriterLevel(f.out, f.opts.GZIPLevel)\n\t\tf.writer = f.gzipWriter\n\t} else {\n\t\terr = f.out.Sync()\n\t}\n\treturn err\n}\n\nfunc (f *FileLogger) currentFilename() string {\n\tt := time.Now()\n\tdatetime := strftime(f.opts.DatetimeFormat, t)\n\treturn strings.Replace(f.filenameFormat, \"\", datetime, -1)\n}\n\nfunc (f *FileLogger) needsRotation() bool {\n\tif f.out == nil {\n\t\treturn true\n\t}\n\n\tfilename := f.currentFilename()\n\tif filename != f.filename {\n\t\tf.logf(lg.INFO, \"[%s/%s] new filename %s, rotating...\", f.topic, f.opts.Channel, filename)\n\t\treturn true // rotate by filename\n\t}\n\n\tif f.opts.RotateInterval > 0 {\n\t\tif s := time.Since(f.openTime); s > f.opts.RotateInterval {\n\t\t\tf.logf(lg.INFO, \"[%s/%s] %s since last open, rotating...\", f.topic, f.opts.Channel, s)\n\t\t\treturn true // rotate by interval\n\t\t}\n\t}\n\n\tif f.opts.RotateSize > 0 && f.filesize > f.opts.RotateSize {\n\t\tf.logf(lg.INFO, \"[%s/%s] %s currently %d bytes (> %d), rotating...\",\n\t\t\tf.topic, f.opts.Channel, f.out.Name(), f.filesize, f.opts.RotateSize)\n\t\treturn true // rotate by size\n\t}\n\n\treturn false\n}\n\nfunc (f *FileLogger) updateFile() {\n\tf.Close() // uses current f.filename and f.rev to resolve rename dst conflict\n\n\tfilename := f.currentFilename()\n\tif filename != f.filename {\n\t\tf.rev = 0 // reset revision to 0 if it is a new filename\n\t} else {\n\t\tf.rev++\n\t}\n\tf.filename = filename\n\tf.openTime = time.Now()\n\n\tfullPath := path.Join(f.opts.WorkDir, filename)\n\terr := makeDirFromPath(f.logf, fullPath)\n\tif err != nil {\n\t\tf.logf(lg.FATAL, \"[%s/%s] unable to create dir: %s\", f.topic, f.opts.Channel, err)\n\t\tos.Exit(1)\n\t}\n\n\tvar fi os.FileInfo\n\tfor ; ; f.rev++ {\n\t\tabsFilename := strings.Replace(fullPath, \"\", fmt.Sprintf(\"-%06d\", f.rev), -1)\n\n\t\t// If we're using a working directory for in-progress files,\n\t\t// proactively check for duplicate file names in the output dir to\n\t\t// prevent conflicts on rename in the normal case\n\t\tif f.opts.WorkDir != f.opts.OutputDir {\n\t\t\toutputFileName := filepath.Join(f.opts.OutputDir, strings.TrimPrefix(absFilename, f.opts.WorkDir))\n\t\t\terr := makeDirFromPath(f.logf, outputFileName)\n\t\t\tif err != nil {\n\t\t\t\tf.logf(lg.FATAL, \"[%s/%s] unable to create dir: %s\", f.topic, f.opts.Channel, err)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\n\t\t\t_, err = os.Stat(outputFileName)\n\t\t\tif err == nil {\n\t\t\t\tf.logf(lg.WARN, \"[%s/%s] output file already exists: %s\", f.topic, f.opts.Channel, outputFileName)\n\t\t\t\tcontinue // next rev\n\t\t\t} else if !os.IsNotExist(err) {\n\t\t\t\tf.logf(lg.FATAL, \"[%s/%s] unable to stat output file %s: %s\", f.topic, f.opts.Channel, outputFileName, err)\n\t\t\t\tos.Exit(1)\n\t\t\t}\n\t\t}\n\n\t\topenFlag := os.O_WRONLY | os.O_CREATE\n\t\tif f.opts.GZIP || f.opts.RotateInterval > 0 {\n\t\t\topenFlag |= os.O_EXCL\n\t\t} else {\n\t\t\topenFlag |= os.O_APPEND\n\t\t}\n\t\tf.out, err = os.OpenFile(absFilename, openFlag, 0666)\n\t\tif err != nil {\n\t\t\tif os.IsExist(err) {\n\t\t\t\tf.logf(lg.WARN, \"[%s/%s] working file already exists: %s\", f.topic, f.opts.Channel, absFilename)\n\t\t\t\tcontinue // next rev\n\t\t\t}\n\t\t\tf.logf(lg.FATAL, \"[%s/%s] unable to open %s: %s\", f.topic, f.opts.Channel, absFilename, err)\n\t\t\tos.Exit(1)\n\t\t}\n\n\t\tf.logf(lg.INFO, \"[%s/%s] opening %s\", f.topic, f.opts.Channel, absFilename)\n\n\t\tfi, err = f.out.Stat()\n\t\tif err != nil {\n\t\t\tf.logf(lg.FATAL, \"[%s/%s] unable to stat file %s: %s\", f.topic, f.opts.Channel, f.out.Name(), err)\n\t\t}\n\t\tf.filesize = fi.Size()\n\n\t\tif f.opts.RotateSize > 0 && f.filesize > f.opts.RotateSize {\n\t\t\tf.logf(lg.INFO, \"[%s/%s] %s currently %d bytes (> %d), rotating...\",\n\t\t\t\tf.topic, f.opts.Channel, f.out.Name(), f.filesize, f.opts.RotateSize)\n\t\t\tcontinue // next rev\n\t\t}\n\n\t\tbreak // good file\n\t}\n\n\tif f.opts.GZIP {\n\t\tf.gzipWriter, _ = gzip.NewWriterLevel(f.out, f.opts.GZIPLevel)\n\t\tf.writer = f.gzipWriter\n\t} else {\n\t\tf.writer = f.out\n\t}\n}\n\nfunc makeDirFromPath(logf lg.AppLogFunc, path string) error {\n\tdir, _ := filepath.Split(path)\n\tif dir != \"\" {\n\t\treturn os.MkdirAll(dir, 0770)\n\t}\n\treturn nil\n}\n\nfunc exclusiveRename(src, dst string) error {\n\terr := os.Link(src, dst)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = os.Remove(src)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\nfunc computeFilenameFormat(opts *Options, topic string) (string, error) {\n\thostname, err := os.Hostname()\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\tshortHostname := strings.Split(hostname, \".\")[0]\n\n\tidentifier := shortHostname\n\tif len(opts.HostIdentifier) != 0 {\n\t\tidentifier = strings.Replace(opts.HostIdentifier, \"\", shortHostname, -1)\n\t\tidentifier = strings.Replace(identifier, \"\", hostname, -1)\n\t}\n\n\tcff := opts.FilenameFormat\n\tif opts.GZIP || opts.RotateSize > 0 || opts.RotateInterval > 0 || opts.WorkDir != opts.OutputDir {\n\t\tif !strings.Contains(cff, \"\") {\n\t\t\treturn \"\", errors.New(\"missing in --filename-format when gzip, rotation, or work dir enabled\")\n\t\t}\n\t} else {\n\t\t// remove as we don't need it\n\t\tcff = strings.Replace(cff, \"\", \"\", -1)\n\t}\n\tcff = strings.Replace(cff, \"\", topic, -1)\n\tcff = strings.Replace(cff, \"\", identifier, -1)\n\tcff = strings.Replace(cff, \"\", fmt.Sprintf(\"%d\", os.Getpid()), -1)\n\tif opts.GZIP && !strings.HasSuffix(cff, \".gz\") {\n\t\tcff = cff + \".gz\"\n\t}\n\n\treturn cff, nil\n}\n" }, { "path": "apps/nsq_to_file/nsq_to_file.go", "content": "// This is a client that writes out to a file, and optionally rolls the file\n\npackage main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"os/signal\"\n\t\"syscall\"\n\t\"time\"\n\n\t\"github.com/mreiferson/go-options\"\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/app\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nfunc flagSet() *flag.FlagSet {\n\tfs := flag.NewFlagSet(\"nsqd\", flag.ExitOnError)\n\n\tfs.Bool(\"version\", false, \"print version string\")\n\tfs.String(\"log-level\", \"info\", \"set log verbosity: debug, info, warn, error, or fatal\")\n\tfs.String(\"log-prefix\", \"[nsq_to_file] \", \"log message prefix\")\n\n\tfs.String(\"channel\", \"nsq_to_file\", \"nsq channel\")\n\tfs.Int(\"max-in-flight\", 200, \"max number of messages to allow in flight\")\n\n\tfs.String(\"output-dir\", \"/tmp\", \"directory to write output files to\")\n\tfs.String(\"work-dir\", \"\", \"directory for in-progress files before moving to output-dir\")\n\tfs.String(\"datetime-format\", \"%Y-%m-%d_%H\", \"strftime compatible format for in filename format\")\n\tfs.String(\"filename-format\", \"...log\", \"output filename format (, , , , are replaced. is increased when file already exists)\")\n\tfs.String(\"host-identifier\", \"\", \"value to output in log filename in place of hostname. and are valid replacement tokens\")\n\tfs.Int(\"gzip-level\", 6, \"gzip compression level (1-9, 1=BestSpeed, 9=BestCompression)\")\n\tfs.Bool(\"gzip\", false, \"gzip output files.\")\n\tfs.Bool(\"skip-empty-files\", false, \"skip writing empty files\")\n\tfs.Duration(\"topic-refresh\", time.Minute, \"how frequently the topic list should be refreshed\")\n\tfs.String(\"topic-pattern\", \"\", \"only log topics matching the following pattern\")\n\n\tfs.Int64(\"rotate-size\", 0, \"rotate the file when it grows bigger than `rotate-size` bytes\")\n\tfs.Duration(\"rotate-interval\", 0, \"rotate the file every duration\")\n\tfs.Duration(\"sync-interval\", 30*time.Second, \"sync file to disk every duration\")\n\n\tfs.Duration(\"http-client-connect-timeout\", 2*time.Second, \"timeout for HTTP connect\")\n\tfs.Duration(\"http-client-request-timeout\", 5*time.Second, \"timeout for HTTP request\")\n\n\tnsqdTCPAddrs := app.StringArray{}\n\tlookupdHTTPAddrs := app.StringArray{}\n\ttopics := app.StringArray{}\n\tconsumerOpts := app.StringArray{}\n\tfs.Var(&nsqdTCPAddrs, \"nsqd-tcp-address\", \"nsqd TCP address (may be given multiple times)\")\n\tfs.Var(&lookupdHTTPAddrs, \"lookupd-http-address\", \"lookupd HTTP address (may be given multiple times)\")\n\tfs.Var(&topics, \"topic\", \"nsq topic (may be given multiple times)\")\n\tfs.Var(&consumerOpts, \"consumer-opt\", \"option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)\")\n\n\treturn fs\n}\n\nfunc main() {\n\tfs := flagSet()\n\tfs.Parse(os.Args[1:])\n\n\tif args := fs.Args(); len(args) > 0 {\n\t\tlog.Fatalf(\"unknown arguments: %s\", args)\n\t}\n\n\topts := NewOptions()\n\toptions.Resolve(opts, fs, nil)\n\n\tlogger := log.New(os.Stderr, opts.LogPrefix, log.Ldate|log.Ltime|log.Lmicroseconds)\n\tlogLevel, err := lg.ParseLogLevel(opts.LogLevel)\n\tif err != nil {\n\t\tlog.Fatal(\"--log-level is invalid\")\n\t}\n\tlogf := func(lvl lg.LogLevel, f string, args ...interface{}) {\n\t\tlg.Logf(logger, logLevel, lvl, f, args...)\n\t}\n\n\tif fs.Lookup(\"version\").Value.(flag.Getter).Get().(bool) {\n\t\tfmt.Printf(\"nsq_to_file v%s\\n\", version.Binary)\n\t\treturn\n\t}\n\n\tif opts.Channel == \"\" {\n\t\tlog.Fatal(\"--channel is required\")\n\t}\n\n\tif opts.HTTPClientConnectTimeout <= 0 {\n\t\tlog.Fatal(\"--http-client-connect-timeout should be positive\")\n\t}\n\n\tif opts.HTTPClientRequestTimeout <= 0 {\n\t\tlog.Fatal(\"--http-client-request-timeout should be positive\")\n\t}\n\n\tif len(opts.NSQDTCPAddrs) == 0 && len(opts.NSQLookupdHTTPAddrs) == 0 {\n\t\tlog.Fatal(\"--nsqd-tcp-address or --lookupd-http-address required.\")\n\t}\n\tif len(opts.NSQDTCPAddrs) != 0 && len(opts.NSQLookupdHTTPAddrs) != 0 {\n\t\tlog.Fatal(\"use --nsqd-tcp-address or --lookupd-http-address not both\")\n\t}\n\n\tif opts.GZIPLevel < 1 || opts.GZIPLevel > 9 {\n\t\tlog.Fatalf(\"invalid --gzip-level value (%d), should be 1-9\", opts.GZIPLevel)\n\t}\n\n\tif len(opts.Topics) == 0 && len(opts.TopicPattern) == 0 {\n\t\tlog.Fatal(\"--topic or --topic-pattern required\")\n\t}\n\n\tif len(opts.Topics) == 0 && len(opts.NSQLookupdHTTPAddrs) == 0 {\n\t\tlog.Fatal(\"--lookupd-http-address must be specified when no --topic specified\")\n\t}\n\n\tif opts.WorkDir == \"\" {\n\t\topts.WorkDir = opts.OutputDir\n\t}\n\n\tcfg := nsq.NewConfig()\n\tcfgFlag := nsq.ConfigFlag{cfg}\n\tfor _, opt := range opts.ConsumerOpts {\n\t\tcfgFlag.Set(opt)\n\t}\n\tcfg.UserAgent = fmt.Sprintf(\"nsq_to_file/%s go-nsq/%s\", version.Binary, nsq.VERSION)\n\tcfg.MaxInFlight = opts.MaxInFlight\n\n\thupChan := make(chan os.Signal, 1)\n\ttermChan := make(chan os.Signal, 1)\n\tsignal.Notify(hupChan, syscall.SIGHUP)\n\tsignal.Notify(termChan, syscall.SIGINT, syscall.SIGTERM)\n\n\tdiscoverer := newTopicDiscoverer(logf, opts, cfg, hupChan, termChan)\n\tdiscoverer.run()\n}\n" }, { "path": "apps/nsq_to_file/options.go", "content": "package main\n\nimport \"time\"\n\ntype Options struct {\n\tTopics []string `flag:\"topic\"`\n\tTopicPattern string `flag:\"topic-pattern\"`\n\tTopicRefreshInterval time.Duration `flag:\"topic-refresh\"`\n\tChannel string `flag:\"channel\"`\n\n\tNSQDTCPAddrs []string `flag:\"nsqd-tcp-address\"`\n\tNSQLookupdHTTPAddrs []string `flag:\"lookupd-http-address\"`\n\tConsumerOpts []string `flag:\"consumer-opt\"`\n\tMaxInFlight int `flag:\"max-in-flight\"`\n\tHTTPClientConnectTimeout time.Duration `flag:\"http-client-connect-timeout\"`\n\tHTTPClientRequestTimeout time.Duration `flag:\"http-client-request-timeout\"`\n\n\tLogPrefix string `flag:\"log-prefix\"`\n\tLogLevel string `flag:\"log-level\"`\n\tOutputDir string `flag:\"output-dir\"`\n\tWorkDir string `flag:\"work-dir\"`\n\tDatetimeFormat string `flag:\"datetime-format\"`\n\tFilenameFormat string `flag:\"filename-format\"`\n\tHostIdentifier string `flag:\"host-identifier\"`\n\tGZIPLevel int `flag:\"gzip-level\"`\n\tGZIP bool `flag:\"gzip\"`\n\tSkipEmptyFiles bool `flag:\"skip-empty-files\"`\n\tRotateSize int64 `flag:\"rotate-size\"`\n\tRotateInterval time.Duration `flag:\"rotate-interval\"`\n\tSyncInterval time.Duration `flag:\"sync-interval\"`\n}\n\nfunc NewOptions() *Options {\n\treturn &Options{\n\t\tLogPrefix: \"[nsq_to_file] \",\n\t\tLogLevel: \"info\",\n\t\tChannel: \"nsq_to_file\",\n\t\tMaxInFlight: 200,\n\t\tOutputDir: \"/tmp\",\n\t\tDatetimeFormat: \"%Y-%m-%d_%H\",\n\t\tFilenameFormat: \"...log\",\n\t\tGZIPLevel: 6,\n\t\tTopicRefreshInterval: time.Minute,\n\t\tSyncInterval: 30 * time.Second,\n\t\tHTTPClientConnectTimeout: 2 * time.Second,\n\t\tHTTPClientRequestTimeout: 5 * time.Second,\n\t}\n}\n" }, { "path": "apps/nsq_to_file/strftime.go", "content": "// COPIED FROM https://github.com/jehiah/go-strftime\npackage main\n\nimport (\n\t\"time\"\n)\n\n// taken from time/format.go\nvar conversion = map[string]string{\n\t/*stdLongMonth */ \"B\": \"January\",\n\t/*stdMonth */ \"b\": \"Jan\",\n\t// stdNumMonth */ \"m\": \"1\",\n\t/*stdZeroMonth */ \"m\": \"01\",\n\t/*stdLongWeekDay */ \"A\": \"Monday\",\n\t/*stdWeekDay */ \"a\": \"Mon\",\n\t// stdDay */ \"d\": \"2\",\n\t// stdUnderDay */ \"d\": \"_2\",\n\t/*stdZeroDay */ \"d\": \"02\",\n\t/*stdHour */ \"H\": \"15\",\n\t// stdHour12 */ \"I\": \"3\",\n\t/*stdZeroHour12 */ \"I\": \"03\",\n\t// stdMinute */ \"M\": \"4\",\n\t/*stdZeroMinute */ \"M\": \"04\",\n\t// stdSecond */ \"S\": \"5\",\n\t/*stdZeroSecond */ \"S\": \"05\",\n\t/*stdLongYear */ \"Y\": \"2006\",\n\t/*stdYear */ \"y\": \"06\",\n\t/*stdPM */ \"p\": \"PM\",\n\t// stdpm */ \"p\": \"pm\",\n\t/*stdTZ */ \"Z\": \"MST\",\n\t// stdISO8601TZ */ \"z\": \"Z0700\", // prints Z for UTC\n\t// stdISO8601ColonTZ */ \"z\": \"Z07:00\", // prints Z for UTC\n\t/*stdNumTZ */ \"z\": \"-0700\", // always numeric\n\t// stdNumShortTZ */ \"b\": \"-07\", // always numeric\n\t// stdNumColonTZ */ \"b\": \"-07:00\", // always numeric\n\t\"%\": \"%\",\n}\n\n// This is an alternative to time.Format because no one knows\n// what date 040305 is supposed to create when used as a 'layout' string\n// this takes standard strftime format options. For a complete list\n// of format options see http://strftime.org/\nfunc strftime(format string, t time.Time) string {\n\tlayout := \"\"\n\tlength := len(format)\n\tfor i := 0; i < length; i++ {\n\t\tif format[i] == '%' && i <= length-2 {\n\t\t\tif layoutCmd, ok := conversion[format[i+1:i+2]]; ok {\n\t\t\t\tlayout = layout + layoutCmd\n\t\t\t\ti++\n\t\t\t\tcontinue\n\t\t\t}\n\t\t}\n\t\tlayout = layout + format[i:i+1]\n\t}\n\treturn t.Format(layout)\n}\n" }, { "path": "apps/nsq_to_file/topic_discoverer.go", "content": "package main\n\nimport (\n\t\"os\"\n\t\"regexp\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/clusterinfo\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype TopicDiscoverer struct {\n\tlogf lg.AppLogFunc\n\topts *Options\n\tci *clusterinfo.ClusterInfo\n\ttopics map[string]*FileLogger\n\thupChan chan os.Signal\n\ttermChan chan os.Signal\n\twg sync.WaitGroup\n\tcfg *nsq.Config\n}\n\nfunc newTopicDiscoverer(logf lg.AppLogFunc, opts *Options, cfg *nsq.Config, hupChan chan os.Signal, termChan chan os.Signal) *TopicDiscoverer {\n\tclient := http_api.NewClient(nil, opts.HTTPClientConnectTimeout, opts.HTTPClientRequestTimeout)\n\treturn &TopicDiscoverer{\n\t\tlogf: logf,\n\t\topts: opts,\n\t\tci: clusterinfo.New(nil, client),\n\t\ttopics: make(map[string]*FileLogger),\n\t\thupChan: hupChan,\n\t\ttermChan: termChan,\n\t\tcfg: cfg,\n\t}\n}\n\nfunc (t *TopicDiscoverer) updateTopics(topics []string) {\n\tfor _, topic := range topics {\n\t\tif _, ok := t.topics[topic]; ok {\n\t\t\tcontinue\n\t\t}\n\n\t\tif !t.isTopicAllowed(topic) {\n\t\t\tt.logf(lg.WARN, \"skipping topic %s (doesn't match pattern %s)\", topic, t.opts.TopicPattern)\n\t\t\tcontinue\n\t\t}\n\n\t\tfl, err := NewFileLogger(t.logf, t.opts, topic, t.cfg)\n\t\tif err != nil {\n\t\t\tt.logf(lg.ERROR, \"couldn't create logger for new topic %s: %s\", topic, err)\n\t\t\tcontinue\n\t\t}\n\t\tt.topics[topic] = fl\n\n\t\tt.wg.Add(1)\n\t\tgo func(fl *FileLogger) {\n\t\t\tfl.router()\n\t\t\tt.wg.Done()\n\t\t}(fl)\n\t}\n}\n\nfunc (t *TopicDiscoverer) run() {\n\tvar ticker <-chan time.Time\n\tif len(t.opts.Topics) == 0 {\n\t\tticker = time.Tick(t.opts.TopicRefreshInterval)\n\t}\n\tt.updateTopics(t.opts.Topics)\nforloop:\n\tfor {\n\t\tselect {\n\t\tcase <-ticker:\n\t\t\tnewTopics, err := t.ci.GetLookupdTopics(t.opts.NSQLookupdHTTPAddrs)\n\t\t\tif err != nil {\n\t\t\t\tt.logf(lg.ERROR, \"could not retrieve topic list: %s\", err)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tt.updateTopics(newTopics)\n\t\tcase <-t.termChan:\n\t\t\tfor _, fl := range t.topics {\n\t\t\t\tclose(fl.termChan)\n\t\t\t}\n\t\t\tbreak forloop\n\t\tcase <-t.hupChan:\n\t\t\tfor _, fl := range t.topics {\n\t\t\t\tfl.hupChan <- true\n\t\t\t}\n\t\t}\n\t}\n\tt.wg.Wait()\n}\n\nfunc (t *TopicDiscoverer) isTopicAllowed(topic string) bool {\n\tif t.opts.TopicPattern == \"\" {\n\t\treturn true\n\t}\n\tmatch, err := regexp.MatchString(t.opts.TopicPattern, topic)\n\tif err != nil {\n\t\treturn false\n\t}\n\treturn match\n}\n" }, { "path": "apps/nsq_to_http/http.go", "content": "package main\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"net/http\"\n\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nvar httpclient *http.Client\nvar userAgent string\n\nfunc init() {\n\tuserAgent = fmt.Sprintf(\"nsq_to_http v%s\", version.Binary)\n}\n\nfunc HTTPGet(endpoint string) (*http.Response, error) {\n\treq, err := http.NewRequest(\"GET\", endpoint, nil)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treq.Header.Set(\"User-Agent\", userAgent)\n\tfor key, val := range validCustomHeaders {\n\t\treq.Header.Set(key, val)\n\t}\n\treturn httpclient.Do(req)\n}\n\nfunc HTTPPost(endpoint string, body *bytes.Buffer) (*http.Response, error) {\n\treq, err := http.NewRequest(\"POST\", endpoint, body)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treq.Header.Set(\"User-Agent\", userAgent)\n\treq.Header.Set(\"Content-Type\", *contentType)\n\tfor key, val := range validCustomHeaders {\n\t\treq.Header.Set(key, val)\n\t}\n\treturn httpclient.Do(req)\n}\n" }, { "path": "apps/nsq_to_http/nsq_to_http.go", "content": "// This is an NSQ client that reads the specified topic/channel\n// and performs HTTP requests (GET/POST) to the specified endpoints\n\npackage main\n\nimport (\n\t\"bytes\"\n\t\"flag\"\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"math/rand\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"os\"\n\t\"os/signal\"\n\t\"strings\"\n\t\"sync/atomic\"\n\t\"syscall\"\n\t\"time\"\n\n\t\"github.com/bitly/go-hostpool\"\n\t\"github.com/bitly/timer_metrics\"\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/app\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nconst (\n\tModeAll = iota\n\tModeRoundRobin\n\tModeHostPool\n)\n\nvar (\n\tshowVersion = flag.Bool(\"version\", false, \"print version string\")\n\n\ttopic = flag.String(\"topic\", \"\", \"nsq topic\")\n\tchannel = flag.String(\"channel\", \"nsq_to_http\", \"nsq channel\")\n\tmaxInFlight = flag.Int(\"max-in-flight\", 200, \"max number of messages to allow in flight\")\n\n\tnumPublishers = flag.Int(\"n\", 100, \"number of concurrent publishers\")\n\tmode = flag.String(\"mode\", \"hostpool\", \"the upstream request mode options: round-robin, hostpool (default), epsilon-greedy\")\n\tsample = flag.Float64(\"sample\", 1.0, \"% of messages to publish (float b/w 0 -> 1)\")\n\thttpConnectTimeout = flag.Duration(\"http-client-connect-timeout\", 2*time.Second, \"timeout for HTTP connect\")\n\thttpRequestTimeout = flag.Duration(\"http-client-request-timeout\", 20*time.Second, \"timeout for HTTP request\")\n\tstatusEvery = flag.Int(\"status-every\", 250, \"the # of requests between logging status (per handler), 0 disables\")\n\tcontentType = flag.String(\"content-type\", \"application/octet-stream\", \"the Content-Type used for POST requests\")\n\n\tgetAddrs = app.StringArray{}\n\tpostAddrs = app.StringArray{}\n\tcustomHeaders = app.StringArray{}\n\tnsqdTCPAddrs = app.StringArray{}\n\tlookupdHTTPAddrs = app.StringArray{}\n\tvalidCustomHeaders map[string]string\n)\n\nfunc init() {\n\tflag.Var(&postAddrs, \"post\", \"HTTP address to make a POST request to. data will be in the body (may be given multiple times)\")\n\tflag.Var(&customHeaders, \"header\", \"Custom header for HTTP requests (may be given multiple times)\")\n\tflag.Var(&getAddrs, \"get\", \"HTTP address to make a GET request to. '%s' will be printf replaced with data (may be given multiple times)\")\n\tflag.Var(&nsqdTCPAddrs, \"nsqd-tcp-address\", \"nsqd TCP address (may be given multiple times)\")\n\tflag.Var(&lookupdHTTPAddrs, \"lookupd-http-address\", \"lookupd HTTP address (may be given multiple times)\")\n}\n\ntype Publisher interface {\n\tPublish(string, []byte) error\n}\n\ntype PublishHandler struct {\n\t// 64bit atomic vars need to be first for proper alignment on 32bit platforms\n\tcounter uint64\n\n\tPublisher\n\taddresses app.StringArray\n\tmode int\n\thostPool hostpool.HostPool\n\n\tperAddressStatus map[string]*timer_metrics.TimerMetrics\n\ttimermetrics *timer_metrics.TimerMetrics\n}\n\nfunc (ph *PublishHandler) HandleMessage(m *nsq.Message) error {\n\tif *sample < 1.0 && rand.Float64() > *sample {\n\t\treturn nil\n\t}\n\n\tstartTime := time.Now()\n\tswitch ph.mode {\n\tcase ModeAll:\n\t\tfor _, addr := range ph.addresses {\n\t\t\tst := time.Now()\n\t\t\terr := ph.Publish(addr, m.Body)\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tph.perAddressStatus[addr].Status(st)\n\t\t}\n\tcase ModeRoundRobin:\n\t\tcounter := atomic.AddUint64(&ph.counter, 1)\n\t\tidx := counter % uint64(len(ph.addresses))\n\t\taddr := ph.addresses[idx]\n\t\terr := ph.Publish(addr, m.Body)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tph.perAddressStatus[addr].Status(startTime)\n\tcase ModeHostPool:\n\t\thostPoolResponse := ph.hostPool.Get()\n\t\taddr := hostPoolResponse.Host()\n\t\terr := ph.Publish(addr, m.Body)\n\t\thostPoolResponse.Mark(err)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tph.perAddressStatus[addr].Status(startTime)\n\t}\n\tph.timermetrics.Status(startTime)\n\n\treturn nil\n}\n\ntype PostPublisher struct{}\n\nfunc (p *PostPublisher) Publish(addr string, msg []byte) error {\n\tbuf := bytes.NewBuffer(msg)\n\tresp, err := HTTPPost(addr, buf)\n\tif err != nil {\n\t\treturn err\n\t}\n\tio.Copy(io.Discard, resp.Body)\n\tresp.Body.Close()\n\n\tif resp.StatusCode < 200 || resp.StatusCode >= 300 {\n\t\treturn fmt.Errorf(\"got status code %d\", resp.StatusCode)\n\t}\n\treturn nil\n}\n\ntype GetPublisher struct{}\n\nfunc (p *GetPublisher) Publish(addr string, msg []byte) error {\n\tendpoint := fmt.Sprintf(addr, url.QueryEscape(string(msg)))\n\tresp, err := HTTPGet(endpoint)\n\tif err != nil {\n\t\treturn err\n\t}\n\tio.Copy(io.Discard, resp.Body)\n\tresp.Body.Close()\n\n\tif resp.StatusCode != 200 {\n\t\treturn fmt.Errorf(\"got status code %d\", resp.StatusCode)\n\t}\n\treturn nil\n}\n\nfunc main() {\n\tvar publisher Publisher\n\tvar addresses app.StringArray\n\tvar selectedMode int\n\n\tcfg := nsq.NewConfig()\n\n\tflag.Var(&nsq.ConfigFlag{cfg}, \"consumer-opt\", \"option to passthrough to nsq.Consumer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)\")\n\tflag.Parse()\n\n\thttpclient = &http.Client{Transport: http_api.NewDeadlineTransport(*httpConnectTimeout, *httpRequestTimeout), Timeout: *httpRequestTimeout}\n\n\tif *showVersion {\n\t\tfmt.Printf(\"nsq_to_http v%s\\n\", version.Binary)\n\t\treturn\n\t}\n\n\tif len(customHeaders) > 0 {\n\t\tvar err error\n\t\tvalidCustomHeaders, err = parseCustomHeaders(customHeaders)\n\t\tif err != nil {\n\t\t\tlog.Fatal(\"--header value format should be 'key=value'\")\n\t\t}\n\t}\n\n\tif *topic == \"\" || *channel == \"\" {\n\t\tlog.Fatal(\"--topic and --channel are required\")\n\t}\n\n\tif *contentType != flag.Lookup(\"content-type\").DefValue {\n\t\tif len(postAddrs) == 0 {\n\t\t\tlog.Fatal(\"--content-type only used with --post\")\n\t\t}\n\t\tif len(*contentType) == 0 {\n\t\t\tlog.Fatal(\"--content-type requires a value when used\")\n\t\t}\n\t}\n\n\tif len(nsqdTCPAddrs) == 0 && len(lookupdHTTPAddrs) == 0 {\n\t\tlog.Fatal(\"--nsqd-tcp-address or --lookupd-http-address required\")\n\t}\n\tif len(nsqdTCPAddrs) > 0 && len(lookupdHTTPAddrs) > 0 {\n\t\tlog.Fatal(\"use --nsqd-tcp-address or --lookupd-http-address not both\")\n\t}\n\n\tif len(getAddrs) == 0 && len(postAddrs) == 0 {\n\t\tlog.Fatal(\"--get or --post required\")\n\t}\n\tif len(getAddrs) > 0 && len(postAddrs) > 0 {\n\t\tlog.Fatal(\"use --get or --post not both\")\n\t}\n\tif len(getAddrs) > 0 {\n\t\tfor _, get := range getAddrs {\n\t\t\tif strings.Count(get, \"%s\") != 1 {\n\t\t\t\tlog.Fatal(\"invalid GET address - must be a printf string\")\n\t\t\t}\n\t\t}\n\t}\n\n\tswitch *mode {\n\tcase \"round-robin\":\n\t\tselectedMode = ModeRoundRobin\n\tcase \"hostpool\", \"epsilon-greedy\":\n\t\tselectedMode = ModeHostPool\n\t}\n\n\tif *sample > 1.0 || *sample < 0.0 {\n\t\tlog.Fatal(\"ERROR: --sample must be between 0.0 and 1.0\")\n\t}\n\n\ttermChan := make(chan os.Signal, 1)\n\tsignal.Notify(termChan, syscall.SIGINT, syscall.SIGTERM)\n\n\tif len(postAddrs) > 0 {\n\t\tpublisher = &PostPublisher{}\n\t\taddresses = postAddrs\n\t} else {\n\t\tpublisher = &GetPublisher{}\n\t\taddresses = getAddrs\n\t}\n\n\tcfg.UserAgent = fmt.Sprintf(\"nsq_to_http/%s go-nsq/%s\", version.Binary, nsq.VERSION)\n\tcfg.MaxInFlight = *maxInFlight\n\n\tconsumer, err := nsq.NewConsumer(*topic, *channel, cfg)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\tperAddressStatus := make(map[string]*timer_metrics.TimerMetrics)\n\tif len(addresses) == 1 {\n\t\t// disable since there is only one address\n\t\tperAddressStatus[addresses[0]] = timer_metrics.NewTimerMetrics(0, \"\")\n\t} else {\n\t\tfor _, a := range addresses {\n\t\t\tperAddressStatus[a] = timer_metrics.NewTimerMetrics(*statusEvery,\n\t\t\t\tfmt.Sprintf(\"[%s]:\", a))\n\t\t}\n\t}\n\n\thostPool := hostpool.New(addresses)\n\tif *mode == \"epsilon-greedy\" {\n\t\thostPool = hostpool.NewEpsilonGreedy(addresses, 0, &hostpool.LinearEpsilonValueCalculator{})\n\t}\n\n\thandler := &PublishHandler{\n\t\tPublisher: publisher,\n\t\taddresses: addresses,\n\t\tmode: selectedMode,\n\t\thostPool: hostPool,\n\t\tperAddressStatus: perAddressStatus,\n\t\ttimermetrics: timer_metrics.NewTimerMetrics(*statusEvery, \"[aggregate]:\"),\n\t}\n\tconsumer.AddConcurrentHandlers(handler, *numPublishers)\n\n\terr = consumer.ConnectToNSQDs(nsqdTCPAddrs)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\terr = consumer.ConnectToNSQLookupds(lookupdHTTPAddrs)\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\tfor {\n\t\tselect {\n\t\tcase <-consumer.StopChan:\n\t\t\treturn\n\t\tcase <-termChan:\n\t\t\tconsumer.Stop()\n\t\t}\n\t}\n}\n\nfunc parseCustomHeaders(strs []string) (map[string]string, error) {\n\tparsedHeaders := make(map[string]string)\n\tfor _, s := range strs {\n\t\tsp := strings.SplitN(s, \":\", 2)\n\t\tif len(sp) != 2 {\n\t\t\treturn nil, fmt.Errorf(\"invalid header: %q\", s)\n\t\t}\n\t\tkey := strings.TrimSpace(sp[0])\n\t\tval := strings.TrimSpace(sp[1])\n\t\tif key == \"\" || val == \"\" {\n\t\t\treturn nil, fmt.Errorf(\"invalid header: %q\", s)\n\t\t}\n\t\tparsedHeaders[key] = val\n\n\t}\n\treturn parsedHeaders, nil\n}\n" }, { "path": "apps/nsq_to_http/nsq_to_http_test.go", "content": "// This is an NSQ client that reads the specified topic/channel\n// and performs HTTP requests (GET/POST) to the specified endpoints\n\npackage main\n\nimport (\n\t\"reflect\"\n\t\"testing\"\n)\n\nfunc TestParseCustomHeaders(t *testing.T) {\n\ttype args struct {\n\t\tstrs []string\n\t}\n\ttests := []struct {\n\t\tname string\n\t\targs args\n\t\twant map[string]string\n\t\twantErr bool\n\t}{\n\t\t{\n\t\t\t\"Valid Custom Headers\",\n\t\t\targs{[]string{\"header1: value1\", \"header2:value2\", \"header3:value3\", \"header4:value4\"}},\n\t\t\tmap[string]string{\"header1\": \"value1\", \"header2\": \"value2\", \"header3\": \"value3\", \"header4\": \"value4\"},\n\t\t\tfalse,\n\t\t},\n\t\t{\n\t\t\t\"Invalid Custom Headers where key is present but no value\",\n\t\t\targs{[]string{\"header1:\", \"header2:value2\", \"header3: value3\", \"header4:value4\"}},\n\t\t\tnil,\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"Invalid Custom Headers where key is not present but value is present\",\n\t\t\targs{[]string{\"header1: value1\", \": value2\", \"header3:value3\", \"header4:value4\"}},\n\t\t\tnil,\n\t\t\ttrue,\n\t\t},\n\t\t{\n\t\t\t\"Invalid Custom Headers where key and value are not present but ':' is specified\",\n\t\t\targs{[]string{\"header1:value1\", \"header2:value2\", \":\", \"header4:value4\"}},\n\t\t\tnil,\n\t\t\ttrue,\n\t\t},\n\t}\n\tfor _, tt := range tests {\n\t\tt.Run(tt.name, func(t *testing.T) {\n\t\t\tgot, err := parseCustomHeaders(tt.args.strs)\n\t\t\tif (err != nil) != tt.wantErr {\n\t\t\t\tt.Errorf(\"parseCustomHeaders() error = %v, wantErr %v\", err, tt.wantErr)\n\t\t\t\treturn\n\t\t\t}\n\t\t\tif !reflect.DeepEqual(got, tt.want) {\n\t\t\t\tt.Errorf(\"parseCustomHeaders() = %v, want %v\", got, tt.want)\n\t\t\t}\n\t\t})\n\t}\n}\n" }, { "path": "apps/nsq_to_nsq/nsq_to_nsq.go", "content": "// This is an NSQ client that reads the specified topic/channel\n// and re-publishes the messages to destination nsqd via TCP\n\npackage main\n\nimport (\n\t\"encoding/json\"\n\t\"errors\"\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"os/signal\"\n\t\"strconv\"\n\t\"sync/atomic\"\n\t\"syscall\"\n\t\"time\"\n\n\t\"github.com/bitly/go-hostpool\"\n\t\"github.com/bitly/timer_metrics\"\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/app\"\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nconst (\n\tModeRoundRobin = iota\n\tModeHostPool\n)\n\nvar (\n\tshowVersion = flag.Bool(\"version\", false, \"print version string\")\n\tchannel = flag.String(\"channel\", \"nsq_to_nsq\", \"nsq channel\")\n\tdestTopic = flag.String(\"destination-topic\", \"\", \"use this destination topic for all consumed topics (default is consumed topic name)\")\n\tmaxInFlight = flag.Int(\"max-in-flight\", 200, \"max number of messages to allow in flight\")\n\n\tstatusEvery = flag.Int(\"status-every\", 250, \"the # of requests between logging status (per destination), 0 disables\")\n\tmode = flag.String(\"mode\", \"hostpool\", \"the upstream request mode options: round-robin, hostpool (default), epsilon-greedy\")\n\n\tnsqdTCPAddrs = app.StringArray{}\n\tlookupdHTTPAddrs = app.StringArray{}\n\tdestNsqdTCPAddrs = app.StringArray{}\n\twhitelistJSONFields = app.StringArray{}\n\ttopics = app.StringArray{}\n\n\trequireJSONField = flag.String(\"require-json-field\", \"\", \"for JSON messages: only pass messages that contain this field\")\n\trequireJSONValue = flag.String(\"require-json-value\", \"\", \"for JSON messages: only pass messages in which the required field has this value\")\n)\n\nfunc init() {\n\tflag.Var(&nsqdTCPAddrs, \"nsqd-tcp-address\", \"nsqd TCP address (may be given multiple times)\")\n\tflag.Var(&destNsqdTCPAddrs, \"destination-nsqd-tcp-address\", \"destination nsqd TCP address (may be given multiple times)\")\n\tflag.Var(&lookupdHTTPAddrs, \"lookupd-http-address\", \"lookupd HTTP address (may be given multiple times)\")\n\tflag.Var(&topics, \"topic\", \"nsq topic (may be given multiple times)\")\n\tflag.Var(&whitelistJSONFields, \"whitelist-json-field\", \"for JSON messages: pass this field (may be given multiple times)\")\n}\n\ntype PublishHandler struct {\n\t// 64bit atomic vars need to be first for proper alignment on 32bit platforms\n\tcounter uint64\n\n\taddresses app.StringArray\n\tproducers map[string]*nsq.Producer\n\tmode int\n\thostPool hostpool.HostPool\n\trespChan chan *nsq.ProducerTransaction\n\n\trequireJSONValueParsed bool\n\trequireJSONValueIsNumber bool\n\trequireJSONNumber float64\n\n\tperAddressStatus map[string]*timer_metrics.TimerMetrics\n\ttimermetrics *timer_metrics.TimerMetrics\n}\n\ntype TopicHandler struct {\n\tpublishHandler *PublishHandler\n\tdestinationTopic string\n}\n\nfunc (ph *PublishHandler) responder() {\n\tvar msg *nsq.Message\n\tvar startTime time.Time\n\tvar address string\n\tvar hostPoolResponse hostpool.HostPoolResponse\n\n\tfor t := range ph.respChan {\n\t\tswitch ph.mode {\n\t\tcase ModeRoundRobin:\n\t\t\tmsg = t.Args[0].(*nsq.Message)\n\t\t\tstartTime = t.Args[1].(time.Time)\n\t\t\thostPoolResponse = nil\n\t\t\taddress = t.Args[2].(string)\n\t\tcase ModeHostPool:\n\t\t\tmsg = t.Args[0].(*nsq.Message)\n\t\t\tstartTime = t.Args[1].(time.Time)\n\t\t\thostPoolResponse = t.Args[2].(hostpool.HostPoolResponse)\n\t\t\taddress = hostPoolResponse.Host()\n\t\t}\n\n\t\tsuccess := t.Error == nil\n\n\t\tif hostPoolResponse != nil {\n\t\t\tif !success {\n\t\t\t\thostPoolResponse.Mark(errors.New(\"failed\"))\n\t\t\t} else {\n\t\t\t\thostPoolResponse.Mark(nil)\n\t\t\t}\n\t\t}\n\n\t\tif success {\n\t\t\tmsg.Finish()\n\t\t} else {\n\t\t\tmsg.Requeue(-1)\n\t\t}\n\n\t\tph.perAddressStatus[address].Status(startTime)\n\t\tph.timermetrics.Status(startTime)\n\t}\n}\n\nfunc (ph *PublishHandler) shouldPassMessage(js map[string]interface{}) (bool, bool) {\n\tpass := true\n\tbackoff := false\n\n\tif *requireJSONField == \"\" {\n\t\treturn pass, backoff\n\t}\n\n\tif *requireJSONValue != \"\" && !ph.requireJSONValueParsed {\n\t\t// cache conversion in case needed while filtering json\n\t\tvar err error\n\t\tph.requireJSONNumber, err = strconv.ParseFloat(*requireJSONValue, 64)\n\t\tph.requireJSONValueIsNumber = (err == nil)\n\t\tph.requireJSONValueParsed = true\n\t}\n\n\tv, ok := js[*requireJSONField]\n\tif !ok {\n\t\tpass = false\n\t\tif *requireJSONValue != \"\" {\n\t\t\tlog.Printf(\"ERROR: missing field to check required value\")\n\t\t\tbackoff = true\n\t\t}\n\t} else if *requireJSONValue != \"\" {\n\t\t// if command-line argument can't convert to float, then it can't match a number\n\t\t// if it can, also integers (up to 2^53 or so) can be compared as float64\n\t\tif s, ok := v.(string); ok {\n\t\t\tif s != *requireJSONValue {\n\t\t\t\tpass = false\n\t\t\t}\n\t\t} else if ph.requireJSONValueIsNumber {\n\t\t\tf, ok := v.(float64)\n\t\t\tif !ok || f != ph.requireJSONNumber {\n\t\t\t\tpass = false\n\t\t\t}\n\t\t} else {\n\t\t\t// json value wasn't a plain string, and argument wasn't a number\n\t\t\t// give up on comparisons of other types\n\t\t\tpass = false\n\t\t}\n\t}\n\n\treturn pass, backoff\n}\n\nfunc filterMessage(js map[string]interface{}, rawMsg []byte) ([]byte, error) {\n\tif len(whitelistJSONFields) == 0 {\n\t\t// no change\n\t\treturn rawMsg, nil\n\t}\n\n\tnewMsg := make(map[string]interface{}, len(whitelistJSONFields))\n\n\tfor _, key := range whitelistJSONFields {\n\t\tvalue, ok := js[key]\n\t\tif ok {\n\t\t\t// avoid printing int as float (go 1.0)\n\t\t\tswitch tvalue := value.(type) {\n\t\t\tcase float64:\n\t\t\t\tivalue := int64(tvalue)\n\t\t\t\tif float64(ivalue) == tvalue {\n\t\t\t\t\tnewMsg[key] = ivalue\n\t\t\t\t} else {\n\t\t\t\t\tnewMsg[key] = tvalue\n\t\t\t\t}\n\t\t\tdefault:\n\t\t\t\tnewMsg[key] = value\n\t\t\t}\n\t\t}\n\t}\n\n\tnewRawMsg, err := json.Marshal(newMsg)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"unable to marshal filtered message %v\", newMsg)\n\t}\n\treturn newRawMsg, nil\n}\n\nfunc (t *TopicHandler) HandleMessage(m *nsq.Message) error {\n\treturn t.publishHandler.HandleMessage(m, t.destinationTopic)\n}\n\nfunc (ph *PublishHandler) HandleMessage(m *nsq.Message, destinationTopic string) error {\n\tvar err error\n\tmsgBody := m.Body\n\n\tif *requireJSONField != \"\" || len(whitelistJSONFields) > 0 {\n\t\tvar js map[string]interface{}\n\t\terr = json.Unmarshal(msgBody, &js)\n\t\tif err != nil {\n\t\t\tlog.Printf(\"ERROR: Unable to decode json: %s\", msgBody)\n\t\t\treturn nil\n\t\t}\n\n\t\tif pass, backoff := ph.shouldPassMessage(js); !pass {\n\t\t\tif backoff {\n\t\t\t\treturn errors.New(\"backoff\")\n\t\t\t}\n\t\t\treturn nil\n\t\t}\n\n\t\tmsgBody, err = filterMessage(js, msgBody)\n\n\t\tif err != nil {\n\t\t\tlog.Printf(\"ERROR: filterMessage() failed: %s\", err)\n\t\t\treturn err\n\t\t}\n\t}\n\n\tstartTime := time.Now()\n\n\tswitch ph.mode {\n\tcase ModeRoundRobin:\n\t\tcounter := atomic.AddUint64(&ph.counter, 1)\n\t\tidx := counter % uint64(len(ph.addresses))\n\t\taddr := ph.addresses[idx]\n\t\tp := ph.producers[addr]\n\t\terr = p.PublishAsync(destinationTopic, msgBody, ph.respChan, m, startTime, addr)\n\tcase ModeHostPool:\n\t\thostPoolResponse := ph.hostPool.Get()\n\t\tp := ph.producers[hostPoolResponse.Host()]\n\t\terr = p.PublishAsync(destinationTopic, msgBody, ph.respChan, m, startTime, hostPoolResponse)\n\t\tif err != nil {\n\t\t\thostPoolResponse.Mark(err)\n\t\t}\n\t}\n\n\tif err != nil {\n\t\treturn err\n\t}\n\tm.DisableAutoResponse()\n\treturn nil\n}\n\nfunc main() {\n\tvar selectedMode int\n\n\tcCfg := nsq.NewConfig()\n\tpCfg := nsq.NewConfig()\n\n\tflag.Var(&nsq.ConfigFlag{cCfg}, \"consumer-opt\", \"option to passthrough to nsq.Consumer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)\")\n\tflag.Var(&nsq.ConfigFlag{pCfg}, \"producer-opt\", \"option to passthrough to nsq.Producer (may be given multiple times, see http://godoc.org/github.com/nsqio/go-nsq#Config)\")\n\n\tflag.Parse()\n\n\tif *showVersion {\n\t\tfmt.Printf(\"nsq_to_nsq v%s\\n\", version.Binary)\n\t\treturn\n\t}\n\n\tif len(topics) == 0 || *channel == \"\" {\n\t\tlog.Fatal(\"--topic and --channel are required\")\n\t}\n\n\tfor _, topic := range topics {\n\t\tif !protocol.IsValidTopicName(topic) {\n\t\t\tlog.Fatal(\"--topic is invalid\")\n\t\t}\n\t}\n\n\tif *destTopic != \"\" && !protocol.IsValidTopicName(*destTopic) {\n\t\tlog.Fatal(\"--destination-topic is invalid\")\n\t}\n\n\tif !protocol.IsValidChannelName(*channel) {\n\t\tlog.Fatal(\"--channel is invalid\")\n\t}\n\n\tif len(nsqdTCPAddrs) == 0 && len(lookupdHTTPAddrs) == 0 {\n\t\tlog.Fatal(\"--nsqd-tcp-address or --lookupd-http-address required\")\n\t}\n\tif len(nsqdTCPAddrs) > 0 && len(lookupdHTTPAddrs) > 0 {\n\t\tlog.Fatal(\"use --nsqd-tcp-address or --lookupd-http-address not both\")\n\t}\n\n\tif len(destNsqdTCPAddrs) == 0 {\n\t\tlog.Fatal(\"--destination-nsqd-tcp-address required\")\n\t}\n\n\tswitch *mode {\n\tcase \"round-robin\":\n\t\tselectedMode = ModeRoundRobin\n\tcase \"hostpool\", \"epsilon-greedy\":\n\t\tselectedMode = ModeHostPool\n\t}\n\n\ttermChan := make(chan os.Signal, 1)\n\tsignal.Notify(termChan, syscall.SIGINT, syscall.SIGTERM)\n\n\tdefaultUA := fmt.Sprintf(\"nsq_to_nsq/%s go-nsq/%s\", version.Binary, nsq.VERSION)\n\n\tcCfg.UserAgent = defaultUA\n\tcCfg.MaxInFlight = *maxInFlight\n\tpCfg.UserAgent = defaultUA\n\n\tproducers := make(map[string]*nsq.Producer)\n\tfor _, addr := range destNsqdTCPAddrs {\n\t\tproducer, err := nsq.NewProducer(addr, pCfg)\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"failed creating producer %s\", err)\n\t\t}\n\t\tproducers[addr] = producer\n\t}\n\n\tperAddressStatus := make(map[string]*timer_metrics.TimerMetrics)\n\tif len(destNsqdTCPAddrs) == 1 {\n\t\t// disable since there is only one address\n\t\tperAddressStatus[destNsqdTCPAddrs[0]] = timer_metrics.NewTimerMetrics(0, \"\")\n\t} else {\n\t\tfor _, a := range destNsqdTCPAddrs {\n\t\t\tperAddressStatus[a] = timer_metrics.NewTimerMetrics(*statusEvery,\n\t\t\t\tfmt.Sprintf(\"[%s]:\", a))\n\t\t}\n\t}\n\n\thostPool := hostpool.New(destNsqdTCPAddrs)\n\tif *mode == \"epsilon-greedy\" {\n\t\thostPool = hostpool.NewEpsilonGreedy(destNsqdTCPAddrs, 0, &hostpool.LinearEpsilonValueCalculator{})\n\t}\n\n\tvar consumerList []*nsq.Consumer\n\n\tpublisher := &PublishHandler{\n\t\taddresses: destNsqdTCPAddrs,\n\t\tproducers: producers,\n\t\tmode: selectedMode,\n\t\thostPool: hostPool,\n\t\trespChan: make(chan *nsq.ProducerTransaction, len(destNsqdTCPAddrs)),\n\t\tperAddressStatus: perAddressStatus,\n\t\ttimermetrics: timer_metrics.NewTimerMetrics(*statusEvery, \"[aggregate]:\"),\n\t}\n\n\tfor _, topic := range topics {\n\t\tconsumer, err := nsq.NewConsumer(topic, *channel, cCfg)\n\t\tconsumerList = append(consumerList, consumer)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\n\t\tpublishTopic := topic\n\t\tif *destTopic != \"\" {\n\t\t\tpublishTopic = *destTopic\n\t\t}\n\t\ttopicHandler := &TopicHandler{\n\t\t\tpublishHandler: publisher,\n\t\t\tdestinationTopic: publishTopic,\n\t\t}\n\t\tconsumer.AddConcurrentHandlers(topicHandler, len(destNsqdTCPAddrs))\n\t}\n\tfor i := 0; i < len(destNsqdTCPAddrs); i++ {\n\t\tgo publisher.responder()\n\t}\n\n\tfor _, consumer := range consumerList {\n\t\terr := consumer.ConnectToNSQDs(nsqdTCPAddrs)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\t}\n\n\tfor _, consumer := range consumerList {\n\t\terr := consumer.ConnectToNSQLookupds(lookupdHTTPAddrs)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\t}\n\n\t<-termChan // wait for signal\n\n\tfor _, consumer := range consumerList {\n\t\tconsumer.Stop()\n\t}\n\tfor _, consumer := range consumerList {\n\t\t<-consumer.StopChan\n\t}\n}\n" }, { "path": "apps/nsqadmin/main.go", "content": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sync\"\n\t\"syscall\"\n\n\t\"github.com/BurntSushi/toml\"\n\t\"github.com/judwhite/go-svc\"\n\t\"github.com/mreiferson/go-options\"\n\t\"github.com/nsqio/nsq/internal/app\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/version\"\n\t\"github.com/nsqio/nsq/nsqadmin\"\n)\n\nfunc nsqadminFlagSet(opts *nsqadmin.Options) *flag.FlagSet {\n\tflagSet := flag.NewFlagSet(\"nsqadmin\", flag.ExitOnError)\n\n\tflagSet.String(\"config\", \"\", \"path to config file\")\n\tflagSet.Bool(\"version\", false, \"print version string\")\n\n\tlogLevel := opts.LogLevel\n\tflagSet.Var(&logLevel, \"log-level\", \"set log verbosity: debug, info, warn, error, or fatal\")\n\tflagSet.String(\"log-prefix\", \"[nsqadmin] \", \"log message prefix\")\n\tflagSet.Bool(\"verbose\", false, \"[deprecated] has no effect, use --log-level\")\n\n\tflagSet.String(\"http-address\", opts.HTTPAddress, \": to listen on for HTTP clients\")\n\tflagSet.String(\"base-path\", opts.BasePath, \"URL base path\")\n\tflagSet.String(\"dev-static-dir\", opts.DevStaticDir, \"(development use only)\")\n\n\tflagSet.String(\"graphite-url\", opts.GraphiteURL, \"graphite HTTP address\")\n\tflagSet.Bool(\"proxy-graphite\", false, \"proxy HTTP requests to graphite\")\n\n\tflagSet.String(\"statsd-counter-format\", opts.StatsdCounterFormat, \"The counter stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string.\")\n\tflagSet.String(\"statsd-gauge-format\", opts.StatsdGaugeFormat, \"The gauge stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string.\")\n\tflagSet.String(\"statsd-prefix\", opts.StatsdPrefix, \"prefix used for keys sent to statsd (%s for host replacement, must match nsqd)\")\n\tflagSet.Duration(\"statsd-interval\", opts.StatsdInterval, \"time interval nsqd is configured to push to statsd (must match nsqd)\")\n\n\tflagSet.String(\"notification-http-endpoint\", \"\", \"HTTP endpoint (fully qualified) to which POST notifications of admin actions will be sent\")\n\n\tflagSet.Duration(\"http-client-connect-timeout\", opts.HTTPClientConnectTimeout, \"timeout for HTTP connect\")\n\tflagSet.Duration(\"http-client-request-timeout\", opts.HTTPClientRequestTimeout, \"timeout for HTTP request\")\n\n\tflagSet.Bool(\"http-client-tls-insecure-skip-verify\", false, \"configure the HTTP client to skip verification of TLS certificates\")\n\tflagSet.String(\"http-client-tls-root-ca-file\", \"\", \"path to CA file for the HTTP client\")\n\tflagSet.String(\"http-client-tls-cert\", \"\", \"path to certificate file for the HTTP client\")\n\tflagSet.String(\"http-client-tls-key\", \"\", \"path to key file for the HTTP client\")\n\n\tflagSet.String(\"allow-config-from-cidr\", opts.AllowConfigFromCIDR, \"A CIDR from which to allow HTTP requests to the /config endpoint\")\n\tflagSet.String(\"acl-http-header\", opts.ACLHTTPHeader, \"HTTP header to check for authenticated admin users\")\n\n\tnsqlookupdHTTPAddresses := app.StringArray{}\n\tflagSet.Var(&nsqlookupdHTTPAddresses, \"lookupd-http-address\", \"lookupd HTTP address (may be given multiple times)\")\n\tnsqdHTTPAddresses := app.StringArray{}\n\tflagSet.Var(&nsqdHTTPAddresses, \"nsqd-http-address\", \"nsqd HTTP address (may be given multiple times)\")\n\tadminUsers := app.StringArray{}\n\tflagSet.Var(&adminUsers, \"admin-user\", \"admin user (may be given multiple times; if specified, only these users will be able to perform privileged actions; acl-http-header is used to determine the authenticated user)\")\n\n\treturn flagSet\n}\n\ntype program struct {\n\tonce sync.Once\n\tnsqadmin *nsqadmin.NSQAdmin\n}\n\nfunc main() {\n\tprg := &program{}\n\tif err := svc.Run(prg, syscall.SIGINT, syscall.SIGTERM); err != nil {\n\t\tlogFatal(\"%s\", err)\n\t}\n}\n\nfunc (p *program) Init(env svc.Environment) error {\n\tif env.IsWindowsService() {\n\t\tdir := filepath.Dir(os.Args[0])\n\t\treturn os.Chdir(dir)\n\t}\n\treturn nil\n}\n\nfunc (p *program) Start() error {\n\topts := nsqadmin.NewOptions()\n\n\tflagSet := nsqadminFlagSet(opts)\n\tflagSet.Parse(os.Args[1:])\n\n\tif flagSet.Lookup(\"version\").Value.(flag.Getter).Get().(bool) {\n\t\tfmt.Println(version.String(\"nsqadmin\"))\n\t\tos.Exit(0)\n\t}\n\n\tvar cfg config\n\tconfigFile := flagSet.Lookup(\"config\").Value.String()\n\tif configFile != \"\" {\n\t\t_, err := toml.DecodeFile(configFile, &cfg)\n\t\tif err != nil {\n\t\t\tlogFatal(\"failed to load config file %s - %s\", configFile, err)\n\t\t}\n\t}\n\tcfg.Validate()\n\n\toptions.Resolve(opts, flagSet, cfg)\n\tnsqadmin, err := nsqadmin.New(opts)\n\tif err != nil {\n\t\tlogFatal(\"failed to instantiate nsqadmin - %s\", err)\n\t}\n\tp.nsqadmin = nsqadmin\n\n\tgo func() {\n\t\terr := p.nsqadmin.Main()\n\t\tif err != nil {\n\t\t\tp.Stop()\n\t\t\tos.Exit(1)\n\t\t}\n\t}()\n\n\treturn nil\n}\n\nfunc (p *program) Stop() error {\n\tp.once.Do(func() {\n\t\tp.nsqadmin.Exit()\n\t})\n\treturn nil\n}\n\nfunc logFatal(f string, args ...interface{}) {\n\tlg.LogFatal(\"[nsqadmin] \", f, args...)\n}\n" }, { "path": "apps/nsqadmin/main_test.go", "content": "package main\n\nimport (\n\t\"testing\"\n\n\t\"github.com/mreiferson/go-options\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/test\"\n\t\"github.com/nsqio/nsq/nsqadmin\"\n)\n\nfunc TestConfigFlagParsing(t *testing.T) {\n\topts := nsqadmin.NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\n\tflagSet := nsqadminFlagSet(opts)\n\tflagSet.Parse([]string{})\n\n\tcfg := config{\"log_level\": \"debug\"}\n\tcfg.Validate()\n\n\toptions.Resolve(opts, flagSet, cfg)\n\tif opts.LogLevel != lg.DEBUG {\n\t\tt.Fatalf(\"log level: want debug, got %s\", opts.LogLevel.String())\n\t}\n}\n" }, { "path": "apps/nsqadmin/options.go", "content": "package main\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype config map[string]interface{}\n\n// Validate settings in the config file, and fatal on errors\nfunc (cfg config) Validate() {\n\tif v, exists := cfg[\"log_level\"]; exists {\n\t\tvar t lg.LogLevel\n\t\terr := t.Set(fmt.Sprintf(\"%v\", v))\n\t\tif err == nil {\n\t\t\tcfg[\"log_level\"] = t\n\t\t} else {\n\t\t\tlogFatal(\"failed parsing log_level %+v\", v)\n\t\t}\n\t}\n}\n" }, { "path": "apps/nsqd/README.md", "content": "## nsqd\r\n\r\n`nsqd` is the daemon that receives, queues, and delivers messages to clients.\r\n\r\nRead the [docs](https://nsq.io/components/nsqd.html).\r\n" }, { "path": "apps/nsqd/main.go", "content": "package main\n\nimport (\n\t\"context\"\n\t\"flag\"\n\t\"fmt\"\n\t\"math/rand\"\n\t\"os\"\n\t\"sync\"\n\t\"syscall\"\n\t\"time\"\n\n\t\"github.com/BurntSushi/toml\"\n\t\"github.com/judwhite/go-svc\"\n\t\"github.com/mreiferson/go-options\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/version\"\n\t\"github.com/nsqio/nsq/nsqd\"\n)\n\ntype program struct {\n\tonce sync.Once\n\tnsqd *nsqd.NSQD\n}\n\nfunc main() {\n\tprg := &program{}\n\tif err := svc.Run(prg, syscall.SIGINT, syscall.SIGTERM); err != nil {\n\t\tlogFatal(\"%s\", err)\n\t}\n}\n\nfunc (p *program) Init(env svc.Environment) error {\n\topts := nsqd.NewOptions()\n\n\tflagSet := nsqdFlagSet(opts)\n\tflagSet.Parse(os.Args[1:])\n\n\trand.Seed(time.Now().UTC().UnixNano())\n\n\tif flagSet.Lookup(\"version\").Value.(flag.Getter).Get().(bool) {\n\t\tfmt.Println(version.String(\"nsqd\"))\n\t\tos.Exit(0)\n\t}\n\n\tvar cfg config\n\tconfigFile := flagSet.Lookup(\"config\").Value.String()\n\tif configFile != \"\" {\n\t\t_, err := toml.DecodeFile(configFile, &cfg)\n\t\tif err != nil {\n\t\t\tlogFatal(\"failed to load config file %s - %s\", configFile, err)\n\t\t}\n\t}\n\tcfg.Validate()\n\n\toptions.Resolve(opts, flagSet, cfg)\n\tapplyBackwardCompatibility(opts, flagSet)\n\n\tnsqd, err := nsqd.New(opts)\n\tif err != nil {\n\t\tlogFatal(\"failed to instantiate nsqd - %s\", err)\n\t}\n\tp.nsqd = nsqd\n\n\treturn nil\n}\n\nfunc (p *program) Start() error {\n\terr := p.nsqd.LoadMetadata()\n\tif err != nil {\n\t\tlogFatal(\"failed to load metadata - %s\", err)\n\t}\n\terr = p.nsqd.PersistMetadata()\n\tif err != nil {\n\t\tlogFatal(\"failed to persist metadata - %s\", err)\n\t}\n\n\tgo func() {\n\t\terr := p.nsqd.Main()\n\t\tif err != nil {\n\t\t\tp.Stop()\n\t\t\tos.Exit(1)\n\t\t}\n\t}()\n\n\treturn nil\n}\n\nfunc (p *program) Stop() error {\n\tp.once.Do(func() {\n\t\tp.nsqd.Exit()\n\t})\n\treturn nil\n}\n\nfunc (p *program) Handle(s os.Signal) error {\n\treturn svc.ErrStop\n}\n\n// Context returns a context that will be canceled when nsqd initiates the shutdown\nfunc (p *program) Context() context.Context {\n\treturn p.nsqd.Context()\n}\n\nfunc logFatal(f string, args ...interface{}) {\n\tlg.LogFatal(\"[nsqd] \", f, args...)\n}\n\n// applyBackwardCompatibility applies backward compatibility rules to options after flag resolution\nfunc applyBackwardCompatibility(opts *nsqd.Options, flagSet *flag.FlagSet) {\n\t// when max-defer-timeout was not explicitly set, refer to the max-req-timeout value\n\tif flag := flagSet.Lookup(\"max-defer-timeout\"); flag != nil && flag.Value.String() == flag.DefValue {\n\t\topts.MaxDeferTimeout = opts.MaxReqTimeout\n\t}\n\n\t// ... other backward compatibility rules can be added here\n}\n" }, { "path": "apps/nsqd/main_test.go", "content": "package main\n\nimport (\n\t\"crypto/tls\"\n\t\"os\"\n\t\"testing\"\n\n\t\"github.com/BurntSushi/toml\"\n\t\"github.com/mreiferson/go-options\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/test\"\n\t\"github.com/nsqio/nsq/nsqd\"\n)\n\nfunc TestConfigFlagParsing(t *testing.T) {\n\topts := nsqd.NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\n\tflagSet := nsqdFlagSet(opts)\n\tflagSet.Parse([]string{})\n\n\tvar cfg config\n\tf, err := os.Open(\"../../contrib/nsqd.cfg.example\")\n\tif err != nil {\n\t\tt.Fatalf(\"%s\", err)\n\t}\n\tdefer f.Close()\n\ttoml.NewDecoder(f).Decode(&cfg)\n\tcfg[\"log_level\"] = \"debug\"\n\tcfg.Validate()\n\n\toptions.Resolve(opts, flagSet, cfg)\n\tnsqd.New(opts)\n\n\tif opts.TLSMinVersion != tls.VersionTLS10 {\n\t\tt.Errorf(\"min %#v not expected %#v\", opts.TLSMinVersion, tls.VersionTLS10)\n\t}\n\tif opts.LogLevel != lg.DEBUG {\n\t\tt.Fatalf(\"log level: want debug, got %s\", opts.LogLevel.String())\n\t}\n}\n" }, { "path": "apps/nsqd/options.go", "content": "package main\n\nimport (\n\t\"crypto/tls\"\n\t\"flag\"\n\t\"fmt\"\n\t\"strconv\"\n\t\"strings\"\n\n\t\"github.com/nsqio/nsq/internal/app\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/nsqd\"\n)\n\ntype tlsRequiredOption int\n\nfunc (t *tlsRequiredOption) Set(s string) error {\n\ts = strings.ToLower(s)\n\tif s == \"tcp-https\" {\n\t\t*t = nsqd.TLSRequiredExceptHTTP\n\t\treturn nil\n\t}\n\trequired, err := strconv.ParseBool(s)\n\tif required {\n\t\t*t = nsqd.TLSRequired\n\t} else {\n\t\t*t = nsqd.TLSNotRequired\n\t}\n\treturn err\n}\n\nfunc (t *tlsRequiredOption) Get() interface{} { return int(*t) }\n\nfunc (t *tlsRequiredOption) String() string {\n\treturn strconv.FormatInt(int64(*t), 10)\n}\n\nfunc (t *tlsRequiredOption) IsBoolFlag() bool { return true }\n\ntype tlsMinVersionOption uint16\n\nvar tlsVersionTable = []struct {\n\tval uint16\n\tstr string\n}{\n\t{tls.VersionTLS10, \"tls1.0\"},\n\t{tls.VersionTLS11, \"tls1.1\"},\n\t{tls.VersionTLS12, \"tls1.2\"},\n\t{tls.VersionTLS13, \"tls1.3\"},\n}\n\nfunc (t *tlsMinVersionOption) Set(s string) error {\n\ts = strings.ToLower(s)\n\tif s == \"\" {\n\t\treturn nil\n\t}\n\tfor _, v := range tlsVersionTable {\n\t\tif s == v.str {\n\t\t\t*t = tlsMinVersionOption(v.val)\n\t\t\treturn nil\n\t\t}\n\t}\n\treturn fmt.Errorf(\"unknown tlsVersionOption %q\", s)\n}\n\nfunc (t *tlsMinVersionOption) Get() interface{} { return uint16(*t) }\n\nfunc (t *tlsMinVersionOption) String() string {\n\tfor _, v := range tlsVersionTable {\n\t\tif uint16(*t) == v.val {\n\t\t\treturn v.str\n\t\t}\n\t}\n\treturn strconv.FormatInt(int64(*t), 10)\n}\n\ntype config map[string]interface{}\n\n// Validate settings in the config file, and fatal on errors\nfunc (cfg config) Validate() {\n\t// special validation/translation\n\tif v, exists := cfg[\"tls_required\"]; exists {\n\t\tvar t tlsRequiredOption\n\t\terr := t.Set(fmt.Sprintf(\"%v\", v))\n\t\tif err == nil {\n\t\t\tcfg[\"tls_required\"] = t.String()\n\t\t} else {\n\t\t\tlogFatal(\"failed parsing tls_required %+v\", v)\n\t\t}\n\t}\n\tif v, exists := cfg[\"tls_min_version\"]; exists {\n\t\tvar t tlsMinVersionOption\n\t\terr := t.Set(fmt.Sprintf(\"%v\", v))\n\t\tif err == nil {\n\t\t\tnewVal := fmt.Sprintf(\"%v\", t.Get())\n\t\t\tif newVal != \"0\" {\n\t\t\t\tcfg[\"tls_min_version\"] = newVal\n\t\t\t} else {\n\t\t\t\tdelete(cfg, \"tls_min_version\")\n\t\t\t}\n\t\t} else {\n\t\t\tlogFatal(\"failed parsing tls_min_version %+v\", v)\n\t\t}\n\t}\n\tif v, exists := cfg[\"log_level\"]; exists {\n\t\tvar t lg.LogLevel\n\t\terr := t.Set(fmt.Sprintf(\"%v\", v))\n\t\tif err == nil {\n\t\t\tcfg[\"log_level\"] = t\n\t\t} else {\n\t\t\tlogFatal(\"failed parsing log_level %+v\", v)\n\t\t}\n\t}\n}\n\nfunc nsqdFlagSet(opts *nsqd.Options) *flag.FlagSet {\n\tflagSet := flag.NewFlagSet(\"nsqd\", flag.ExitOnError)\n\n\t// basic options\n\tflagSet.Bool(\"version\", false, \"print version string\")\n\tflagSet.String(\"config\", \"\", \"path to config file\")\n\n\tlogLevel := opts.LogLevel\n\tflagSet.Var(&logLevel, \"log-level\", \"set log verbosity: debug, info, warn, error, or fatal\")\n\tflagSet.String(\"log-prefix\", \"[nsqd] \", \"log message prefix\")\n\tflagSet.Bool(\"verbose\", false, \"[deprecated] has no effect, use --log-level\")\n\n\tflagSet.Int64(\"node-id\", opts.ID, \"unique part for message IDs, (int) in range [0,1024) (default is hash of hostname)\")\n\tflagSet.Bool(\"worker-id\", false, \"[deprecated] use --node-id\")\n\n\tflagSet.String(\"https-address\", opts.HTTPSAddress, \": to listen on for HTTPS clients\")\n\tflagSet.String(\"http-address\", opts.HTTPAddress, \"address to listen on for HTTP clients (: for TCP/IP or for unix socket)\")\n\tflagSet.String(\"tcp-address\", opts.TCPAddress, \"address to listen on for TCP clients (: for TCP/IP or for unix socket)\")\n\n\tauthHTTPAddresses := app.StringArray{}\n\tflagSet.Var(&authHTTPAddresses, \"auth-http-address\", \": or a full url to query auth server (may be given multiple times)\")\n\tflagSet.String(\"auth-http-request-method\", opts.AuthHTTPRequestMethod, \"HTTP method to use for auth server requests\")\n\tflagSet.String(\"broadcast-address\", opts.BroadcastAddress, \"address that will be registered with lookupd (defaults to the OS hostname)\")\n\tflagSet.Int(\"broadcast-tcp-port\", opts.BroadcastTCPPort, \"TCP port that will be registered with lookupd (defaults to the TCP port that this nsqd is listening on)\")\n\tflagSet.Int(\"broadcast-http-port\", opts.BroadcastHTTPPort, \"HTTP port that will be registered with lookupd (defaults to the HTTP port that this nsqd is listening on)\")\n\tlookupdTCPAddrs := app.StringArray{}\n\tflagSet.Var(&lookupdTCPAddrs, \"lookupd-tcp-address\", \"lookupd TCP address (may be given multiple times)\")\n\tflagSet.Duration(\"http-client-connect-timeout\", opts.HTTPClientConnectTimeout, \"timeout for HTTP connect\")\n\tflagSet.Duration(\"http-client-request-timeout\", opts.HTTPClientRequestTimeout, \"timeout for HTTP request\")\n\tflagSet.String(\"topology-region\", opts.TopologyRegion, \"A region represents a larger domain, made up of one or more zones for preferring closer consumer\")\n\tflagSet.String(\"topology-zone\", opts.TopologyZone, \"A zone represents a logical failure domain for preferring closer consumer\")\n\n\t// diskqueue options\n\tflagSet.String(\"data-path\", opts.DataPath, \"path to store disk-backed messages\")\n\tflagSet.Int64(\"mem-queue-size\", opts.MemQueueSize, \"number of messages to keep in memory (per topic/channel)\")\n\tflagSet.Int64(\"max-bytes-per-file\", opts.MaxBytesPerFile, \"number of bytes per diskqueue file before rolling\")\n\tflagSet.Int64(\"sync-every\", opts.SyncEvery, \"number of messages per diskqueue fsync\")\n\tflagSet.Duration(\"sync-timeout\", opts.SyncTimeout, \"duration of time per diskqueue fsync\")\n\n\tflagSet.Int(\"queue-scan-worker-pool-max\", opts.QueueScanWorkerPoolMax, \"max concurrency for checking in-flight and deferred message timeouts\")\n\tflagSet.Int(\"queue-scan-selection-count\", opts.QueueScanSelectionCount, \"number of channels to check per cycle (every 100ms) for in-flight and deferred timeouts\")\n\n\t// msg and command options\n\tflagSet.Duration(\"msg-timeout\", opts.MsgTimeout, \"default duration to wait before auto-requeing a message\")\n\tflagSet.Duration(\"max-msg-timeout\", opts.MaxMsgTimeout, \"maximum duration before a message will timeout\")\n\tflagSet.Int64(\"max-msg-size\", opts.MaxMsgSize, \"maximum size of a single message in bytes\")\n\tflagSet.Duration(\"max-req-timeout\", opts.MaxReqTimeout, \"maximum requeuing timeout for a message\")\n\tflagSet.Int64(\"max-body-size\", opts.MaxBodySize, \"maximum size of a single command body\")\n\tflagSet.Duration(\"max-defer-timeout\", opts.MaxDeferTimeout, \"maximum duration when deferring a message\")\n\n\t// client overridable configuration options\n\tflagSet.Duration(\"max-heartbeat-interval\", opts.MaxHeartbeatInterval, \"maximum client configurable duration of time between client heartbeats\")\n\tflagSet.Int64(\"max-rdy-count\", opts.MaxRdyCount, \"maximum RDY count for a client\")\n\tflagSet.Int64(\"max-output-buffer-size\", opts.MaxOutputBufferSize, \"maximum client configurable size (in bytes) for a client output buffer\")\n\tflagSet.Duration(\"max-output-buffer-timeout\", opts.MaxOutputBufferTimeout, \"maximum client configurable duration of time between flushing to a client\")\n\tflagSet.Duration(\"min-output-buffer-timeout\", opts.MinOutputBufferTimeout, \"minimum client configurable duration of time between flushing to a client\")\n\tflagSet.Duration(\"output-buffer-timeout\", opts.OutputBufferTimeout, \"default duration of time between flushing data to clients\")\n\tflagSet.Int(\"max-channel-consumers\", opts.MaxChannelConsumers, \"maximum channel consumer connection count per nsqd instance (default 0, i.e., unlimited)\")\n\n\t// statsd integration options\n\tflagSet.String(\"statsd-address\", opts.StatsdAddress, \"UDP : of a statsd daemon for pushing stats\")\n\tflagSet.Duration(\"statsd-interval\", opts.StatsdInterval, \"duration between pushing to statsd\")\n\tflagSet.Bool(\"statsd-mem-stats\", opts.StatsdMemStats, \"toggle sending memory and GC stats to statsd\")\n\tflagSet.String(\"statsd-prefix\", opts.StatsdPrefix, \"prefix used for keys sent to statsd (%s for host replacement)\")\n\tflagSet.Int(\"statsd-udp-packet-size\", opts.StatsdUDPPacketSize, \"the size in bytes of statsd UDP packets\")\n\tflagSet.Bool(\"statsd-exclude-ephemeral\", opts.StatsdExcludeEphemeral, \"Skip ephemeral topics and channels when sending stats to statsd\")\n\n\t// End to end percentile flags\n\te2eProcessingLatencyPercentiles := app.FloatArray{}\n\tflagSet.Var(&e2eProcessingLatencyPercentiles, \"e2e-processing-latency-percentile\", \"message processing time percentiles (as float (0, 1.0]) to track (can be specified multiple times or comma separated '1.0,0.99,0.95', default none)\")\n\tflagSet.Duration(\"e2e-processing-latency-window-time\", opts.E2EProcessingLatencyWindowTime, \"calculate end to end latency quantiles for this duration of time (ie: 60s would only show quantile calculations from the past 60 seconds)\")\n\n\t// TLS config\n\tflagSet.String(\"tls-cert\", opts.TLSCert, \"path to certificate file\")\n\tflagSet.String(\"tls-key\", opts.TLSKey, \"path to key file\")\n\tflagSet.String(\"tls-client-auth-policy\", opts.TLSClientAuthPolicy, \"client certificate auth policy ('require' or 'require-verify')\")\n\tflagSet.String(\"tls-root-ca-file\", opts.TLSRootCAFile, \"path to certificate authority file\")\n\ttlsRequired := tlsRequiredOption(opts.TLSRequired)\n\ttlsMinVersion := tlsMinVersionOption(opts.TLSMinVersion)\n\tflagSet.Var(&tlsRequired, \"tls-required\", \"require TLS for client connections (true, false, tcp-https)\")\n\tflagSet.Var(&tlsMinVersion, \"tls-min-version\", \"minimum SSL/TLS version acceptable ('ssl3.0', 'tls1.0', 'tls1.1', 'tls1.2' or 'tls1.3')\")\n\n\t// compression\n\tflagSet.Bool(\"deflate\", opts.DeflateEnabled, \"enable deflate feature negotiation (client compression)\")\n\tflagSet.Int(\"max-deflate-level\", opts.MaxDeflateLevel, \"max deflate compression level a client can negotiate (> values == > nsqd CPU usage)\")\n\tflagSet.Bool(\"snappy\", opts.SnappyEnabled, \"enable snappy feature negotiation (client compression)\")\n\n\texperiments := app.StringArray{}\n\tvar validExperiments []string\n\tfor _, e := range nsqd.AllExperiments {\n\t\tvalidExperiments = append(validExperiments, fmt.Sprintf(\"%q\", string(e)))\n\t}\n\tflagSet.Var(&experiments, \"enable-experiment\", fmt.Sprintf(\"enable experimental feature (may be given multiple times) (valid options: %s)\", strings.Join(validExperiments, \", \")))\n\n\treturn flagSet\n}\n" }, { "path": "apps/nsqlookupd/README.md", "content": "## nsqlookupd\r\n\r\n`nsqlookupd` is the daemon that manages topology metadata and serves client requests to\r\ndiscover the location of topics at runtime.\r\n\r\nRead the [docs](https://nsq.io/components/nsqlookupd.html).\r\n" }, { "path": "apps/nsqlookupd/main.go", "content": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sync\"\n\t\"syscall\"\n\n\t\"github.com/BurntSushi/toml\"\n\t\"github.com/judwhite/go-svc\"\n\t\"github.com/mreiferson/go-options\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/version\"\n\t\"github.com/nsqio/nsq/nsqlookupd\"\n)\n\nfunc nsqlookupdFlagSet(opts *nsqlookupd.Options) *flag.FlagSet {\n\tflagSet := flag.NewFlagSet(\"nsqlookupd\", flag.ExitOnError)\n\n\tflagSet.String(\"config\", \"\", \"path to config file\")\n\tflagSet.Bool(\"version\", false, \"print version string\")\n\n\tlogLevel := opts.LogLevel\n\tflagSet.Var(&logLevel, \"log-level\", \"set log verbosity: debug, info, warn, error, or fatal\")\n\tflagSet.String(\"log-prefix\", \"[nsqlookupd] \", \"log message prefix\")\n\tflagSet.Bool(\"verbose\", false, \"[deprecated] has no effect, use --log-level\")\n\n\tflagSet.String(\"tcp-address\", opts.TCPAddress, \": to listen on for TCP clients\")\n\tflagSet.String(\"http-address\", opts.HTTPAddress, \": to listen on for HTTP clients\")\n\tflagSet.String(\"broadcast-address\", opts.BroadcastAddress, \"address of this lookupd node, (default to the OS hostname)\")\n\n\tflagSet.Duration(\"inactive-producer-timeout\", opts.InactiveProducerTimeout, \"duration of time a producer will remain in the active list since its last ping\")\n\tflagSet.Duration(\"tombstone-lifetime\", opts.TombstoneLifetime, \"duration of time a producer will remain tombstoned if registration remains\")\n\n\treturn flagSet\n}\n\ntype program struct {\n\tonce sync.Once\n\tnsqlookupd *nsqlookupd.NSQLookupd\n}\n\nfunc main() {\n\tprg := &program{}\n\tif err := svc.Run(prg, syscall.SIGINT, syscall.SIGTERM); err != nil {\n\t\tlogFatal(\"%s\", err)\n\t}\n}\n\nfunc (p *program) Init(env svc.Environment) error {\n\tif env.IsWindowsService() {\n\t\tdir := filepath.Dir(os.Args[0])\n\t\treturn os.Chdir(dir)\n\t}\n\treturn nil\n}\n\nfunc (p *program) Start() error {\n\topts := nsqlookupd.NewOptions()\n\n\tflagSet := nsqlookupdFlagSet(opts)\n\tflagSet.Parse(os.Args[1:])\n\n\tif flagSet.Lookup(\"version\").Value.(flag.Getter).Get().(bool) {\n\t\tfmt.Println(version.String(\"nsqlookupd\"))\n\t\tos.Exit(0)\n\t}\n\n\tvar cfg config\n\tconfigFile := flagSet.Lookup(\"config\").Value.String()\n\tif configFile != \"\" {\n\t\t_, err := toml.DecodeFile(configFile, &cfg)\n\t\tif err != nil {\n\t\t\tlogFatal(\"failed to load config file %s - %s\", configFile, err)\n\t\t}\n\t}\n\tcfg.Validate()\n\n\toptions.Resolve(opts, flagSet, cfg)\n\tnsqlookupd, err := nsqlookupd.New(opts)\n\tif err != nil {\n\t\tlogFatal(\"failed to instantiate nsqlookupd\", err)\n\t}\n\tp.nsqlookupd = nsqlookupd\n\n\tgo func() {\n\t\terr := p.nsqlookupd.Main()\n\t\tif err != nil {\n\t\t\tp.Stop()\n\t\t\tos.Exit(1)\n\t\t}\n\t}()\n\n\treturn nil\n}\n\nfunc (p *program) Stop() error {\n\tp.once.Do(func() {\n\t\tp.nsqlookupd.Exit()\n\t})\n\treturn nil\n}\n\nfunc logFatal(f string, args ...interface{}) {\n\tlg.LogFatal(\"[nsqlookupd] \", f, args...)\n}\n" }, { "path": "apps/nsqlookupd/main_test.go", "content": "package main\n\nimport (\n\t\"testing\"\n\n\t\"github.com/mreiferson/go-options\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/test\"\n\t\"github.com/nsqio/nsq/nsqlookupd\"\n)\n\nfunc TestConfigFlagParsing(t *testing.T) {\n\topts := nsqlookupd.NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\n\tflagSet := nsqlookupdFlagSet(opts)\n\tflagSet.Parse([]string{})\n\n\tcfg := config{\"log_level\": \"debug\"}\n\tcfg.Validate()\n\n\toptions.Resolve(opts, flagSet, cfg)\n\tif opts.LogLevel != lg.DEBUG {\n\t\tt.Fatalf(\"log level: want debug, got %s\", opts.LogLevel.String())\n\t}\n}\n" }, { "path": "apps/nsqlookupd/options.go", "content": "package main\n\nimport (\n\t\"fmt\"\n\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype config map[string]interface{}\n\n// Validate settings in the config file, and fatal on errors\nfunc (cfg config) Validate() {\n\tif v, exists := cfg[\"log_level\"]; exists {\n\t\tvar t lg.LogLevel\n\t\terr := t.Set(fmt.Sprintf(\"%v\", v))\n\t\tif err == nil {\n\t\t\tcfg[\"log_level\"] = t\n\t\t} else {\n\t\t\tlogFatal(\"failed parsing log_level %+v\", v)\n\t\t}\n\t}\n}\n" }, { "path": "apps/to_nsq/README.md", "content": "# to_nsq\n\nA tool for publishing to an nsq topic with data from `stdin`.\n\n## Usage\n\n```\nUsage of ./to_nsq:\n -delimiter string\n \tcharacter to split input from stdin (default \"\\n\")\n -nsqd-tcp-address value\n \tdestination nsqd TCP address (may be given multiple times)\n -producer-opt value\n \toption to passthrough to nsq.Producer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)\n -rate int\n \tThrottle messages to n/second. 0 to disable\n -topic string\n \tNSQ topic to publish to\n```\n \n### Examples\n\nPublish each line of a file:\n\n```bash\n$ cat source.txt | to_nsq -topic=\"topic\" -nsqd-tcp-address=\"127.0.0.1:4150\"\n```\n\nPublish three messages, in one go:\n\n```bash\n$ echo \"one,two,three\" | to_nsq -delimiter=\",\" -topic=\"topic\" -nsqd-tcp-address=\"127.0.0.1:4150\"\n```" }, { "path": "apps/to_nsq/to_nsq.go", "content": "// This is an NSQ client that publishes incoming messages from\n// stdin to the specified topic.\n\npackage main\n\nimport (\n\t\"bufio\"\n\t\"flag\"\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"os\"\n\t\"os/signal\"\n\t\"sync/atomic\"\n\t\"syscall\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/app\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nvar (\n\ttopic = flag.String(\"topic\", \"\", \"NSQ topic to publish to\")\n\tdelimiter = flag.String(\"delimiter\", \"\\n\", \"character to split input from stdin\")\n\n\tdestNsqdTCPAddrs = app.StringArray{}\n)\n\nfunc init() {\n\tflag.Var(&destNsqdTCPAddrs, \"nsqd-tcp-address\", \"destination nsqd TCP address (may be given multiple times)\")\n}\n\nfunc main() {\n\tcfg := nsq.NewConfig()\n\tflag.Var(&nsq.ConfigFlag{cfg}, \"producer-opt\", \"option to passthrough to nsq.Producer (may be given multiple times, http://godoc.org/github.com/nsqio/go-nsq#Config)\")\n\trate := flag.Int64(\"rate\", 0, \"Throttle messages to n/second. 0 to disable\")\n\n\tflag.Parse()\n\n\tif len(*topic) == 0 {\n\t\tlog.Fatal(\"--topic required\")\n\t}\n\n\tif len(*delimiter) != 1 {\n\t\tlog.Fatal(\"--delimiter must be a single byte\")\n\t}\n\n\tstopChan := make(chan bool)\n\ttermChan := make(chan os.Signal, 1)\n\tsignal.Notify(termChan, syscall.SIGINT, syscall.SIGTERM)\n\n\tcfg.UserAgent = fmt.Sprintf(\"to_nsq/%s go-nsq/%s\", version.Binary, nsq.VERSION)\n\n\t// make the producers\n\tproducers := make(map[string]*nsq.Producer)\n\tfor _, addr := range destNsqdTCPAddrs {\n\t\tproducer, err := nsq.NewProducer(addr, cfg)\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"failed to create nsq.Producer - %s\", err)\n\t\t}\n\t\tproducers[addr] = producer\n\t}\n\n\tif len(producers) == 0 {\n\t\tlog.Fatal(\"--nsqd-tcp-address required\")\n\t}\n\n\tthrottleEnabled := *rate >= 1\n\tbalance := int64(1)\n\t// avoid divide by 0 if !throttleEnabled\n\tvar interval time.Duration\n\tif throttleEnabled {\n\t\tinterval = time.Second / time.Duration(*rate)\n\t}\n\tgo func() {\n\t\tif !throttleEnabled {\n\t\t\treturn\n\t\t}\n\t\tlog.Printf(\"Throttling messages rate to max:%d/second\", *rate)\n\t\t// every tick increase the number of messages we can send\n\t\tfor range time.Tick(interval) {\n\t\t\tn := atomic.AddInt64(&balance, 1)\n\t\t\t// if we build up more than 1s of capacity just bound to that\n\t\t\tif n > int64(*rate) {\n\t\t\t\tatomic.StoreInt64(&balance, int64(*rate))\n\t\t\t}\n\t\t}\n\t}()\n\n\tr := bufio.NewReader(os.Stdin)\n\tdelim := (*delimiter)[0]\n\tgo func() {\n\t\tfor {\n\t\t\tvar err error\n\t\t\tif throttleEnabled {\n\t\t\t\tcurrentBalance := atomic.LoadInt64(&balance)\n\t\t\t\tif currentBalance <= 0 {\n\t\t\t\t\ttime.Sleep(interval)\n\t\t\t\t}\n\t\t\t\terr = readAndPublish(r, delim, producers)\n\t\t\t\tatomic.AddInt64(&balance, -1)\n\t\t\t} else {\n\t\t\t\terr = readAndPublish(r, delim, producers)\n\t\t\t}\n\t\t\tif err != nil {\n\t\t\t\tif err != io.EOF {\n\t\t\t\t\tlog.Fatal(err)\n\t\t\t\t}\n\t\t\t\tclose(stopChan)\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}()\n\n\tselect {\n\tcase <-termChan:\n\tcase <-stopChan:\n\t}\n\n\tfor _, producer := range producers {\n\t\tproducer.Stop()\n\t}\n}\n\n// readAndPublish reads to the delim from r and publishes the bytes\n// to the map of producers.\nfunc readAndPublish(r *bufio.Reader, delim byte, producers map[string]*nsq.Producer) error {\n\tline, readErr := r.ReadBytes(delim)\n\n\tif len(line) > 0 {\n\t\t// trim the delimiter\n\t\tline = line[:len(line)-1]\n\t}\n\n\tif len(line) == 0 {\n\t\treturn readErr\n\t}\n\n\tfor _, producer := range producers {\n\t\terr := producer.Publish(*topic, line)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\treturn readErr\n}\n" }, { "path": "bench/bench.py", "content": "#!/usr/bin/env python3\n\n#\n# This script bootstraps an NSQ cluster in EC2 and runs benchmarks.\n#\n# Requires python3 and the following packages:\n# - boto3\n# - paramiko\n# - tornado\n#\n# AWS authentication is delegated entirely to the boto3 environment, see:\n#\n# https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html\n#\n# EC2 instances are launched into EC2 Classic, expecting a 'default' security group\n# that allows allows SSH (port 22) from 0.0.0.0/0 and an EC2 key pair created\n# (named 'default', but configurable via --ssh-key-name).\n#\n\nimport sys\nimport logging\nimport time\nimport datetime\nimport socket\nimport warnings\nimport hashlib\n\nimport boto3\nimport paramiko.client\nimport paramiko.ssh_exception\nimport tornado.options\n\n\ndef ssh_connect_with_retries(host, retries=3, timeout=30):\n for i in range(retries):\n try:\n ssh_client = paramiko.client.SSHClient()\n ssh_client.set_missing_host_key_policy(paramiko.client.WarningPolicy())\n ssh_client.connect(host, username='ubuntu', timeout=timeout)\n return ssh_client\n except (socket.error, paramiko.ssh_exception.SSHException):\n if i == retries - 1:\n raise\n logging.warning('... re-trying to connect to %s:%d in 15s', host, 22)\n time.sleep(15)\n\n\ndef ssh_cmd_async(ssh_client, cmd):\n transport = ssh_client.get_transport()\n chan = transport.open_session()\n chan.exec_command(cmd)\n return chan\n\n\ndef ssh_cmd(ssh_client, cmd, timeout=2):\n transport = ssh_client.get_transport()\n chan = transport.open_session()\n chan.settimeout(timeout)\n chan.exec_command(cmd)\n\n stdout = b''\n stderr = b''\n while True:\n if chan.recv_ready():\n stdout += chan.recv(4096)\n continue\n if chan.recv_stderr_ready():\n stderr += chan.recv_stderr(4096)\n continue\n if chan.exit_status_ready():\n exit_status = chan.recv_exit_status()\n break\n time.sleep(0.1)\n\n if exit_status != 0:\n raise Exception('%r' % stderr)\n\n return stdout, stderr\n\n\ndef get_session():\n return boto3.session.Session(region_name=tornado.options.options.region)\n\n\ndef _bootstrap(addr):\n commit = tornado.options.options.commit\n golang_version = tornado.options.options.golang_version\n ssh_client = ssh_connect_with_retries(addr)\n for cmd in [\n 'wget https://storage.googleapis.com/golang/go%s.linux-amd64.tar.gz' % golang_version,\n 'sudo -S tar -C /usr/local -xzf go%s.linux-amd64.tar.gz' % golang_version,\n 'sudo -S apt-get update',\n 'sudo -S apt-get -y install git mercurial',\n 'mkdir -p go/src/github.com/nsqio',\n 'cd go/src/github.com/nsqio && git clone https://github.com/nsqio/nsq',\n 'cd go/src/github.com/nsqio/nsq && git checkout %s' % commit,\n 'cd go/src/github.com/nsqio/nsq/apps/nsqd && GO111MODULE=on /usr/local/go/bin/go build',\n 'cd go/src/github.com/nsqio/nsq/bench/bench_writer && GO111MODULE=on /usr/local/go/bin/go build',\n 'cd go/src/github.com/nsqio/nsq/bench/bench_reader && GO111MODULE=on /usr/local/go/bin/go build',\n 'sudo -S mkdir -p /mnt/nsq',\n 'sudo -S chmod 777 /mnt/nsq']:\n ssh_cmd(ssh_client, cmd, timeout=10)\n\n\ndef bootstrap():\n session = get_session()\n\n ec2 = session.resource('ec2')\n\n total_count = tornado.options.options.nsqd_count + tornado.options.options.worker_count\n logging.info('launching %d instances', total_count)\n instances = ec2.create_instances(\n ImageId=tornado.options.options.ami,\n MinCount=total_count,\n MaxCount=total_count,\n KeyName=tornado.options.options.ssh_key_name,\n InstanceType=tornado.options.options.instance_type,\n SecurityGroups=['default'])\n\n logging.info('waiting for instances to launch...')\n\n while any(i.state['Name'] != 'running' for i in instances):\n waiting_for = [i.id for i in instances if i.state['Name'] != 'running']\n logging.info('... sleeping for 5s (waiting for %s)', ', '.join(waiting_for))\n time.sleep(5)\n for instance in instances:\n instance.load()\n\n for instance in instances:\n if not instance.tags:\n instance.create_tags(Tags=[{'Key': 'nsq_bench', 'Value': '1'}])\n\n try:\n c = 0\n for i in instances:\n c += 1\n logging.info('(%d) bootstrapping %s (%s)', c, i.public_dns_name, i.id)\n _bootstrap(i.public_dns_name)\n except Exception:\n logging.exception('bootstrap failed')\n decomm()\n\n\ndef run():\n instances = _find_instances()\n\n logging.info('launching nsqd on %d host(s)', tornado.options.options.nsqd_count)\n\n nsqd_chans = []\n nsqd_hosts = instances[:tornado.options.options.nsqd_count]\n for instance in nsqd_hosts:\n try:\n ssh_client = ssh_connect_with_retries(instance.public_dns_name)\n for cmd in [\n 'sudo -S pkill -f nsqd',\n 'sudo -S rm -f /mnt/nsq/*.dat',\n 'GOMAXPROCS=32 ./go/src/github.com/nsqio/nsq/apps/nsqd/nsqd \\\n --data-path=/mnt/nsq \\\n --mem-queue-size=10000000 \\\n --max-rdy-count=%s' % (tornado.options.options.rdy)]:\n nsqd_chans.append((ssh_client, ssh_cmd_async(ssh_client, cmd)))\n except Exception:\n logging.exception('failed')\n\n nsqd_tcp_addrs = [i.public_dns_name for i in nsqd_hosts]\n\n dt = datetime.datetime.utcnow()\n deadline = dt + datetime.timedelta(seconds=30)\n\n logging.info('launching %d producer(s) on %d host(s)',\n tornado.options.options.nsqd_count * tornado.options.options.worker_count,\n tornado.options.options.worker_count)\n\n worker_chans = []\n\n producer_hosts = instances[tornado.options.options.nsqd_count:]\n for instance in producer_hosts:\n for nsqd_tcp_addr in nsqd_tcp_addrs:\n topic = hashlib.md5(instance.public_dns_name.encode('utf-8')).hexdigest()\n try:\n ssh_client = ssh_connect_with_retries(instance.public_dns_name)\n for cmd in [\n 'GOMAXPROCS=2 \\\n ./go/src/github.com/nsqio/nsq/bench/bench_writer/bench_writer \\\n --topic=%s --nsqd-tcp-address=%s:4150 --deadline=\\'%s\\' --size=%d' % (\n topic, nsqd_tcp_addr, deadline.strftime('%Y-%m-%d %H:%M:%S'),\n tornado.options.options.msg_size)]:\n worker_chans.append((ssh_client, ssh_cmd_async(ssh_client, cmd)))\n except Exception:\n logging.exception('failed')\n\n if tornado.options.options.mode == 'pubsub':\n logging.info('launching %d consumer(s) on %d host(s)',\n tornado.options.options.nsqd_count * tornado.options.options.worker_count,\n tornado.options.options.worker_count)\n\n consumer_hosts = instances[tornado.options.options.nsqd_count:]\n for instance in consumer_hosts:\n for nsqd_tcp_addr in nsqd_tcp_addrs:\n topic = hashlib.md5(instance.public_dns_name.encode('utf-8')).hexdigest()\n try:\n ssh_client = ssh_connect_with_retries(instance.public_dns_name)\n for cmd in [\n 'GOMAXPROCS=8 \\\n ./go/src/github.com/nsqio/nsq/bench/bench_reader/bench_reader \\\n --topic=%s --nsqd-tcp-address=%s:4150 --deadline=\\'%s\\' --size=%d \\\n --rdy=%d' % (\n topic, nsqd_tcp_addr, deadline.strftime('%Y-%m-%d %H:%M:%S'),\n tornado.options.options.msg_size, tornado.options.options.rdy)]:\n worker_chans.append((ssh_client, ssh_cmd_async(ssh_client, cmd)))\n except Exception:\n logging.exception('failed')\n\n stats = {\n 'bench_reader': {\n 'durations': [],\n 'mbytes': [],\n 'ops': []\n },\n 'bench_writer': {\n 'durations': [],\n 'mbytes': [],\n 'ops': []\n }\n }\n while worker_chans:\n for ssh_client, chan in worker_chans[:]:\n if chan.recv_ready():\n sys.stdout.write(chan.recv(4096))\n sys.stdout.flush()\n continue\n if chan.recv_stderr_ready():\n line = chan.recv_stderr(4096).decode('utf-8')\n if 'duration:' in line:\n kind = line.split(' ')[0][1:-1]\n parts = line.rsplit('duration:')[1].split('-')\n stats[kind]['durations'].append(float(parts[0].strip()[:-1]))\n stats[kind]['mbytes'].append(float(parts[1].strip()[:-4]))\n stats[kind]['ops'].append(float(parts[2].strip()[:-5]))\n sys.stdout.write(line)\n sys.stdout.flush()\n continue\n if chan.exit_status_ready():\n worker_chans.remove((ssh_client, chan))\n time.sleep(0.1)\n\n for kind, data in stats.items():\n if not data['durations']:\n continue\n\n max_duration = max(data['durations'])\n total_mb = sum(data['mbytes'])\n total_ops = sum(data['ops'])\n\n logging.info('[%s] %fs - %fmb/s - %fops/s - %fus/op',\n kind, max_duration, total_mb, total_ops,\n max_duration / total_ops * 1000 * 1000)\n\n for ssh_client, chan in nsqd_chans:\n chan.close()\n\n\ndef _find_instances():\n session = get_session()\n ec2 = session.resource('ec2')\n return [i for i in ec2.instances.all() if\n i.state['Name'] == 'running' and any(t['Key'] == 'nsq_bench' for t in i.tags)]\n\n\ndef decomm():\n instances = _find_instances()\n logging.info('terminating instances %s' % ','.join(i.id for i in instances))\n for instance in instances:\n instance.terminate()\n\n\nif __name__ == '__main__':\n tornado.options.define('region', type=str, default='us-east-1',\n help='EC2 region to launch instances')\n tornado.options.define('nsqd_count', type=int, default=3,\n help='how many nsqd instances to launch')\n tornado.options.define('worker_count', type=int, default=3,\n help='how many worker instances to launch')\n # ubuntu 18.04 HVM instance store us-east-1\n tornado.options.define('ami', type=str, default='ami-0938f2289b3fa3f5b',\n help='AMI ID')\n tornado.options.define('ssh_key_name', type=str, default='default',\n help='SSH key name')\n tornado.options.define('instance_type', type=str, default='c3.2xlarge',\n help='EC2 instance type')\n tornado.options.define('msg_size', type=int, default=200,\n help='size of message')\n tornado.options.define('rdy', type=int, default=10000,\n help='RDY count to use for bench_reader')\n tornado.options.define('mode', type=str, default='pubsub',\n help='the benchmark mode (pub, pubsub)')\n tornado.options.define('commit', type=str, default='master',\n help='the git commit')\n tornado.options.define('golang_version', type=str, default='1.14.3',\n help='the go version')\n tornado.options.parse_command_line()\n\n logging.getLogger('paramiko').setLevel(logging.WARNING)\n warnings.simplefilter('ignore')\n\n cmd_name = sys.argv[-1]\n cmd_map = {\n 'bootstrap': bootstrap,\n 'run': run,\n 'decomm': decomm\n }\n cmd = cmd_map.get(cmd_name, bootstrap)\n\n sys.exit(cmd())\n" }, { "path": "bench/bench_channels/bench_channels.go", "content": "package main\n\nimport (\n\t\"bufio\"\n\t\"flag\"\n\t\"fmt\"\n\t\"net\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n)\n\nvar (\n\tnum = flag.Int(\"num\", 10000, \"num channels\")\n\ttcpAddress = flag.String(\"nsqd-tcp-address\", \"127.0.0.1:4150\", \": to connect to nsqd\")\n)\n\nfunc main() {\n\tflag.Parse()\n\tvar wg sync.WaitGroup\n\n\tgoChan := make(chan int)\n\trdyChan := make(chan int)\n\tfor j := 0; j < *num; j++ {\n\t\twg.Add(1)\n\t\tgo func(id int) {\n\t\t\tsubWorker(*num, *tcpAddress, fmt.Sprintf(\"t%d\", j), \"ch\", rdyChan, goChan, id)\n\t\t\twg.Done()\n\t\t}(j)\n\t\t<-rdyChan\n\t\ttime.Sleep(5 * time.Millisecond)\n\t}\n\n\tclose(goChan)\n\twg.Wait()\n}\n\nfunc subWorker(n int, tcpAddr string,\n\ttopic string, channel string,\n\trdyChan chan int, goChan chan int, id int) {\n\tconn, err := net.DialTimeout(\"tcp\", tcpAddr, time.Second)\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\tconn.Write(nsq.MagicV2)\n\trw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))\n\tci := make(map[string]interface{})\n\tci[\"client_id\"] = \"test\"\n\tcmd, _ := nsq.Identify(ci)\n\tcmd.WriteTo(rw)\n\tnsq.Subscribe(topic, channel).WriteTo(rw)\n\trdyCount := 1\n\trdy := rdyCount\n\trdyChan <- 1\n\t<-goChan\n\tnsq.Ready(rdyCount).WriteTo(rw)\n\trw.Flush()\n\tnsq.ReadResponse(rw)\n\tnsq.ReadResponse(rw)\n\tfor {\n\t\tresp, err := nsq.ReadResponse(rw)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tframeType, data, err := nsq.UnpackResponse(resp)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tif frameType == nsq.FrameTypeError {\n\t\t\tpanic(string(data))\n\t\t} else if frameType == nsq.FrameTypeResponse {\n\t\t\tnsq.Nop().WriteTo(rw)\n\t\t\trw.Flush()\n\t\t\tcontinue\n\t\t}\n\t\tmsg, err := nsq.DecodeMessage(data)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tnsq.Finish(msg.ID).WriteTo(rw)\n\t\trdy--\n\t\tif rdy == 0 {\n\t\t\tnsq.Ready(rdyCount).WriteTo(rw)\n\t\t\trdy = rdyCount\n\t\t\trw.Flush()\n\t\t}\n\t}\n}\n" }, { "path": "bench/bench_reader/bench_reader.go", "content": "package main\n\nimport (\n\t\"bufio\"\n\t\"errors\"\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"net\"\n\t\"runtime\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n)\n\nvar (\n\trunfor = flag.Duration(\"runfor\", 10*time.Second, \"duration of time to run\")\n\ttcpAddress = flag.String(\"nsqd-tcp-address\", \"127.0.0.1:4150\", \": to connect to nsqd\")\n\tsize = flag.Int(\"size\", 200, \"size of messages\")\n\ttopic = flag.String(\"topic\", \"sub_bench\", \"topic to receive messages on\")\n\tchannel = flag.String(\"channel\", \"ch\", \"channel to receive messages on\")\n\tdeadline = flag.String(\"deadline\", \"\", \"deadline to start the benchmark run\")\n\trdy = flag.Int(\"rdy\", 2500, \"RDY count to use\")\n)\n\nvar totalMsgCount int64\n\nfunc main() {\n\tflag.Parse()\n\tvar wg sync.WaitGroup\n\n\tlog.SetPrefix(\"[bench_reader] \")\n\n\tgoChan := make(chan int)\n\trdyChan := make(chan int)\n\tworkers := runtime.GOMAXPROCS(0)\n\tfor j := 0; j < workers; j++ {\n\t\twg.Add(1)\n\t\tgo func(id int) {\n\t\t\tsubWorker(*runfor, workers, *tcpAddress, *topic, *channel, rdyChan, goChan, id)\n\t\t\twg.Done()\n\t\t}(j)\n\t\t<-rdyChan\n\t}\n\n\tif *deadline != \"\" {\n\t\tt, err := time.Parse(\"2006-01-02 15:04:05\", *deadline)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\t\td := time.Until(t)\n\t\tlog.Printf(\"sleeping until %s (%s)\", t, d)\n\t\ttime.Sleep(d)\n\t}\n\n\tstart := time.Now()\n\tclose(goChan)\n\twg.Wait()\n\tend := time.Now()\n\tduration := end.Sub(start)\n\ttmc := atomic.LoadInt64(&totalMsgCount)\n\tlog.Printf(\"duration: %s - %.03fmb/s - %.03fops/s - %.03fus/op\",\n\t\tduration,\n\t\tfloat64(tmc*int64(*size))/duration.Seconds()/1024/1024,\n\t\tfloat64(tmc)/duration.Seconds(),\n\t\tfloat64(duration/time.Microsecond)/float64(tmc))\n}\n\nfunc subWorker(td time.Duration, workers int, tcpAddr string, topic string, channel string, rdyChan chan int, goChan chan int, id int) {\n\tconn, err := net.DialTimeout(\"tcp\", tcpAddr, time.Second)\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\tconn.Write(nsq.MagicV2)\n\trw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))\n\tci := make(map[string]interface{})\n\tci[\"client_id\"] = \"reader\"\n\tci[\"hostname\"] = \"reader\"\n\tci[\"user_agent\"] = fmt.Sprintf(\"bench_reader/%s\", nsq.VERSION)\n\tcmd, _ := nsq.Identify(ci)\n\tcmd.WriteTo(rw)\n\tnsq.Subscribe(topic, channel).WriteTo(rw)\n\trdyChan <- 1\n\t<-goChan\n\tnsq.Ready(*rdy).WriteTo(rw)\n\trw.Flush()\n\tnsq.ReadResponse(rw)\n\tnsq.ReadResponse(rw)\n\tvar msgCount int64\n\tgo func() {\n\t\ttime.Sleep(td)\n\t\tconn.Close()\n\t}()\n\tfor {\n\t\tresp, err := nsq.ReadResponse(rw)\n\t\tif err != nil {\n\t\t\tif errors.Is(err, net.ErrClosed) {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tframeType, data, err := nsq.UnpackResponse(resp)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tif frameType == nsq.FrameTypeError {\n\t\t\tpanic(string(data))\n\t\t} else if frameType == nsq.FrameTypeResponse {\n\t\t\tcontinue\n\t\t}\n\t\tmsg, err := nsq.DecodeMessage(data)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tnsq.Finish(msg.ID).WriteTo(rw)\n\t\tmsgCount++\n\t\tif float64(msgCount%int64(*rdy)) > float64(*rdy)*0.75 {\n\t\t\trw.Flush()\n\t\t}\n\t}\n\tatomic.AddInt64(&totalMsgCount, msgCount)\n}\n" }, { "path": "bench/bench_writer/bench_writer.go", "content": "package main\n\nimport (\n\t\"bufio\"\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"net\"\n\t\"runtime\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n)\n\nvar (\n\trunfor = flag.Duration(\"runfor\", 10*time.Second, \"duration of time to run\")\n\ttcpAddress = flag.String(\"nsqd-tcp-address\", \"127.0.0.1:4150\", \": to connect to nsqd\")\n\ttopic = flag.String(\"topic\", \"sub_bench\", \"topic to receive messages on\")\n\tsize = flag.Int(\"size\", 200, \"size of messages\")\n\tbatchSize = flag.Int(\"batch-size\", 200, \"batch size of messages\")\n\tdeadline = flag.String(\"deadline\", \"\", \"deadline to start the benchmark run\")\n)\n\nvar totalMsgCount int64\n\nfunc main() {\n\tflag.Parse()\n\tvar wg sync.WaitGroup\n\n\tlog.SetPrefix(\"[bench_writer] \")\n\n\tmsg := make([]byte, *size)\n\tbatch := make([][]byte, *batchSize)\n\tfor i := range batch {\n\t\tbatch[i] = msg\n\t}\n\n\tgoChan := make(chan int)\n\trdyChan := make(chan int)\n\tfor j := 0; j < runtime.GOMAXPROCS(0); j++ {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tpubWorker(*runfor, *tcpAddress, *batchSize, batch, *topic, rdyChan, goChan)\n\t\t\twg.Done()\n\t\t}()\n\t\t<-rdyChan\n\t}\n\n\tif *deadline != \"\" {\n\t\tt, err := time.Parse(\"2006-01-02 15:04:05\", *deadline)\n\t\tif err != nil {\n\t\t\tlog.Fatal(err)\n\t\t}\n\t\td := time.Until(t)\n\t\tlog.Printf(\"sleeping until %s (%s)\", t, d)\n\t\ttime.Sleep(d)\n\t}\n\n\tstart := time.Now()\n\tclose(goChan)\n\twg.Wait()\n\tend := time.Now()\n\tduration := end.Sub(start)\n\ttmc := atomic.LoadInt64(&totalMsgCount)\n\tlog.Printf(\"duration: %s - %.03fmb/s - %.03fops/s - %.03fus/op\",\n\t\tduration,\n\t\tfloat64(tmc*int64(*size))/duration.Seconds()/1024/1024,\n\t\tfloat64(tmc)/duration.Seconds(),\n\t\tfloat64(duration/time.Microsecond)/float64(tmc))\n}\n\nfunc pubWorker(td time.Duration, tcpAddr string, batchSize int, batch [][]byte, topic string, rdyChan chan int, goChan chan int) {\n\tconn, err := net.DialTimeout(\"tcp\", tcpAddr, time.Second)\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\tconn.Write(nsq.MagicV2)\n\trw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))\n\tci := make(map[string]interface{})\n\tci[\"client_id\"] = \"writer\"\n\tci[\"hostname\"] = \"writer\"\n\tci[\"user_agent\"] = fmt.Sprintf(\"bench_writer/%s\", nsq.VERSION)\n\tcmd, _ := nsq.Identify(ci)\n\tcmd.WriteTo(rw)\n\trdyChan <- 1\n\t<-goChan\n\trw.Flush()\n\tnsq.ReadResponse(rw)\n\tvar msgCount int64\n\tendTime := time.Now().Add(td)\n\tfor {\n\t\tcmd, _ := nsq.MultiPublish(topic, batch)\n\t\t_, err := cmd.WriteTo(rw)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\terr = rw.Flush()\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tresp, err := nsq.ReadResponse(rw)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tframeType, data, err := nsq.UnpackResponse(resp)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tif frameType == nsq.FrameTypeError {\n\t\t\tpanic(string(data))\n\t\t}\n\t\tmsgCount += int64(len(batch))\n\t\tif time.Now().After(endTime) {\n\t\t\tbreak\n\t\t}\n\t}\n\tatomic.AddInt64(&totalMsgCount, msgCount)\n}\n" }, { "path": "bench/requirements.txt", "content": "tornado==4.3\nparamiko==1.16.0\nboto==2.39.0\n" }, { "path": "bench.sh", "content": "#!/bin/bash\nreadonly messageSize=\"${1:-200}\"\nreadonly batchSize=\"${2:-200}\"\nreadonly memQueueSize=\"${3:-1000000}\"\nreadonly dataPath=\"${4:-}\"\nset -e\nset -u\n\necho \"# using --mem-queue-size=$memQueueSize --data-path=$dataPath --size=$messageSize --batch-size=$batchSize\"\necho \"# compiling/running nsqd\"\npushd apps/nsqd >/dev/null\ngo build\nrm -f *.dat\n./nsqd --mem-queue-size=$memQueueSize --data-path=$dataPath >/dev/null 2>&1 &\nnsqd_pid=$!\npopd >/dev/null\n\ncleanup() {\n kill -9 $nsqd_pid\n rm -f nsqd/*.dat\n}\ntrap cleanup INT TERM EXIT\n\nsleep 0.3\necho \"# creating topic/channel\"\ncurl --silent 'http://127.0.0.1:4151/create_topic?topic=sub_bench' >/dev/null 2>&1\ncurl --silent 'http://127.0.0.1:4151/create_channel?topic=sub_bench&channel=ch' >/dev/null 2>&1\n\necho \"# compiling bench_reader/bench_writer\"\npushd bench >/dev/null\nfor app in bench_reader bench_writer; do\n pushd $app >/dev/null\n go build\n popd >/dev/null\ndone\npopd >/dev/null\n\necho -n \"PUB: \"\nbench/bench_writer/bench_writer --size=$messageSize --batch-size=$batchSize 2>&1\n\ncurl -s -o cpu.pprof http://127.0.0.1:4151/debug/pprof/profile &\npprof_pid=$!\n\necho -n \"SUB: \"\nbench/bench_reader/bench_reader --size=$messageSize --channel=ch 2>&1\n\necho \"waiting for pprof...\"\nwait $pprof_pid\n" }, { "path": "contrib/nsq.spec", "content": "%define name nsq\n%define version 1.1.1-alpha\n%define release 1\n%define path usr/local\n%define group Database/Applications\n%define __os_install_post %{nil}\n\nSummary: nsq\nName: %{name}\nVersion: %{version}\nRelease: %{release}\nGroup: %{group}\nPackager: Matt Reiferson \nLicense: Apache\nBuildRoot: %{_tmppath}/%{name}-%{version}-%{release}\nAutoReqProv: no\n# we just assume you have go installed. You may or may not have an RPM to depend on.\n# BuildRequires: go\n\n%description \nNSQ - A realtime distributed messaging platform\nhttps://github.com/nsqio/nsq\n\n%prep\nmkdir -p $RPM_BUILD_DIR/%{name}-%{version}-%{release}\ncd $RPM_BUILD_DIR/%{name}-%{version}-%{release}\ngit clone git@github.com:nsqio/nsq.git\n\n%build\ncd $RPM_BUILD_DIR/%{name}-%{version}-%{release}/nsq\nmake PREFIX=/%{path}\n\n%install\nexport DONT_STRIP=1\nrm -rf $RPM_BUILD_ROOT\ncd $RPM_BUILD_DIR/%{name}-%{version}-%{release}/nsq\nmake PREFIX=/${path} DESTDIR=$RPM_BUILD_ROOT install\n\n%files\n/%{path}/bin/nsqadmin\n/%{path}/bin/nsqd\n/%{path}/bin/nsqlookupd\n/%{path}/bin/nsq_to_file\n/%{path}/bin/nsq_to_http\n/%{path}/bin/nsq_to_nsq\n/%{path}/bin/nsq_tail\n/%{path}/bin/nsq_stat\n/%{path}/bin/to_nsq\n" }, { "path": "contrib/nsqadmin.cfg.example", "content": "## log verbosity level: debug, info, warn, error, or fatal\nlog_level = \"info\"\n\n## log message prefix (default \"[nsqadmin] \")\n# log_prefix = \"\"\n\n## HTTP header to check for authenticated admin users (default \"X_Forwarded_User\")\n# acl_http_header = \"\"\n\n## admin user (may be given multiple times; if specified, only these users will be able to perform privileged actions)\n# admin_users = [\n# \"admin\"\n# ]\n\n## A CIDR from which to allow HTTP requests to the /config endpoint (default \"127.0.0.1/8\")\n# allow_config_from_cidr = \"\"\n\n## URL base path (default \"/\")\n# base_path = \"\"\n\n## timeout for HTTP connect (default 2s)\n# http_client_connect_timeout = \"2s\"\n\n## timeout for HTTP request (default 5s)\n# http_client_request_timeout = \"5s\"\n\n## path to certificate file for the HTTP client\n# http_client_tls_cert = \"\"\n\n## configure the HTTP client to skip verification of TLS certificates\n# http_client_tls_insecure_skip_verify = false\n\n## path to key file for the HTTP client\n# http_client_tls_key = \"\"\n\n## path to CA file for the HTTP client\n# http_client_tls_root_ca_file = \"\"\n\n## : to listen on for HTTP clients\nhttp_address = \"0.0.0.0:4171\"\n\n## graphite HTTP address\ngraphite_url = \"\"\n\n## proxy HTTP requests to graphite\nproxy_graphite = false\n\n## prefix used for keys sent to statsd (%s for host replacement, must match nsqd)\nstatsd_prefix = \"nsq.%s\"\n\n## format of statsd counter stats\nstatsd_counter_format = \"stats.counters.%s.count\"\n\n## format of statsd gauge stats\nstatsd_gauge_format = \"stats.gauges.%s\"\n\n## time interval nsqd is configured to push to statsd (must match nsqd)\nstatsd_interval = \"60s\"\n\n## HTTP endpoint (fully qualified) to which POST notifications of admin actions will be sent\nnotification_http_endpoint = \"\"\n\n\n## nsqlookupd HTTP addresses\nnsqlookupd_http_addresses = [\n \"127.0.0.1:4161\"\n]\n\n## nsqd HTTP addresses (optional)\nnsqd_http_addresses = [\n \"127.0.0.1:4151\"\n]\n" }, { "path": "contrib/nsqd.cfg.example", "content": "## log verbosity level: debug, info, warn, error, or fatal\nlog_level = \"info\"\n\n## unique identifier (int) for this worker (will default to a hash of hostname)\n# id = 5150\n\n## : to listen on for TCP clients\ntcp_address = \"0.0.0.0:4150\"\n\n## : to listen on for HTTP clients\nhttp_address = \"0.0.0.0:4151\"\n\n## : to listen on for HTTPS clients\n# https_address = \"0.0.0.0:4152\"\n\n## address that will be registered with lookupd (defaults to the OS hostname)\n# broadcast_address = \"\"\n\n## cluster of nsqlookupd TCP addresses\nnsqlookupd_tcp_addresses = [\n \"127.0.0.1:4160\"\n]\n\n## duration to wait before HTTP client connection timeout\nhttp_client_connect_timeout = \"2s\"\n\n## duration to wait before HTTP client request timeout\nhttp_client_request_timeout = \"5s\"\n\n## path to store disk-backed messages\n# data_path = \"/var/lib/nsq\"\n\n## number of messages to keep in memory (per topic/channel)\nmem_queue_size = 10000\n\n## number of bytes per diskqueue file before rolling\nmax_bytes_per_file = 104857600\n\n## number of messages per diskqueue fsync\nsync_every = 2500\n\n## duration of time per diskqueue fsync (time.Duration)\nsync_timeout = \"2s\"\n\n\n## duration to wait before auto-requeing a message\nmsg_timeout = \"60s\"\n\n## maximum duration before a message will timeout\nmax_msg_timeout = \"15m\"\n\n## maximum size of a single message in bytes\nmax_msg_size = 1024768\n\n## maximum requeuing timeout for a message\nmax_req_timeout = \"1h\"\n\n## maximum size of a single command body\nmax_body_size = 5123840\n\n\n## maximum client configurable duration of time between client heartbeats\nmax_heartbeat_interval = \"60s\"\n\n## maximum RDY count for a client\nmax_rdy_count = 2500\n\n## maximum client configurable size (in bytes) for a client output buffer\nmax_output_buffer_size = 65536\n\n## maximum client configurable duration of time between flushing to a client (time.Duration)\nmax_output_buffer_timeout = \"1s\"\n\n\n## UDP : of a statsd daemon for pushing stats\n# statsd_address = \"127.0.0.1:8125\"\n\n## prefix used for keys sent to statsd (%s for host replacement)\nstatsd_prefix = \"nsq.%s\"\n\n## duration between pushing to statsd (time.Duration)\nstatsd_interval = \"60s\"\n\n## toggle sending memory and GC stats to statsd\nstatsd_mem_stats = true\n\n## the size in bytes of statsd UDP packets\n# statsd_udp_packet_size = 508\n\n\n## message processing time percentiles to keep track of (float)\ne2e_processing_latency_percentiles = [\n 1.0,\n 0.99,\n 0.95\n]\n\n## calculate end to end latency quantiles for this duration of time (time.Duration)\ne2e_processing_latency_window_time = \"10m\"\n\n\n## path to certificate file\ntls_cert = \"\"\n\n## path to private key file\ntls_key = \"\"\n\n## set policy on client certificate (require - client must provide certificate,\n## require-verify - client must provide verifiable signed certificate)\n# tls_client_auth_policy = \"require-verify\"\n\n## set custom root Certificate Authority\n# tls_root_ca_file = \"\"\n\n## require client TLS upgrades\ntls_required = false\n\n## minimum TLS version (\"ssl3.0\", \"tls1.0,\" \"tls1.1\", \"tls1.2\")\ntls_min_version = \"\"\n\n## enable deflate feature negotiation (client compression)\ndeflate = true\n\n## max deflate compression level a client can negotiate (> values == > nsqd CPU usage)\nmax_deflate_level = 6\n\n## enable snappy feature negotiation (client compression)\nsnappy = true\n" }, { "path": "contrib/nsqlookupd.cfg.example", "content": "## log verbosity level: debug, info, warn, error, or fatal\nlog_level = \"info\"\n\n## : to listen on for TCP clients\ntcp_address = \"0.0.0.0:4160\"\n\n## : to listen on for HTTP clients\nhttp_address = \"0.0.0.0:4161\"\n\n## address that will be registered with lookupd (defaults to the OS hostname)\n# broadcast_address = \"\"\n\n\n## duration of time a producer will remain in the active list since its last ping\ninactive_producer_timeout = \"300s\"\n\n## duration of time a producer will remain tombstoned if registration remains\ntombstone_lifetime = \"45s\"\n" }, { "path": "coverage.sh", "content": "#!/bin/bash\n# Generate test coverage statistics for Go packages.\n#\n# Works around the fact that `go test -coverprofile` currently does not work\n# with multiple packages, see https://code.google.com/p/go/issues/detail?id=6909\n#\n# Usage: coverage.sh [--html|--coveralls]\n#\n# --html Additionally create HTML report\n# --coveralls Push coverage statistics to coveralls.io\n#\n\nset -e\n\nworkdir=.cover\nprofile=\"$workdir/cover.out\"\nmode=count\n\ngenerate_cover_data() {\n rm -rf \"$workdir\"\n mkdir \"$workdir\"\n\n for pkg in \"$@\"; do\n f=\"$workdir/$(echo $pkg | tr / -).cover\"\n go test -covermode=\"$mode\" -coverprofile=\"$f\" \"$pkg\"\n done\n\n echo \"mode: $mode\" >\"$profile\"\n grep -h -v \"^mode:\" \"$workdir\"/*.cover >>\"$profile\"\n}\n\nshow_html_report() {\n go tool cover -html=\"$profile\" -o=\"$workdir\"/coverage.html\n}\n\nshow_csv_report() {\n go tool cover -func=\"$profile\" -o=\"$workdir\"/coverage.csv\n}\n\npush_to_coveralls() {\n echo \"Pushing coverage statistics to coveralls.io\"\n # ignore failure to push - it happens\n $GOPATH/bin/goveralls -coverprofile=\"$profile\" \\\n -service=github \\\n -ignore=\"nsqadmin/bindata.go\" || true\n}\n\ngenerate_cover_data $(go list ./... | grep -v /vendor/)\nshow_csv_report\n\ncase \"$1\" in\n\"\")\n ;;\n--html)\n show_html_report ;;\n--coveralls)\n push_to_coveralls ;;\n*)\n echo >&2 \"error: invalid option: $1\"; exit 1 ;;\nesac\n" }, { "path": "dist.sh", "content": "#!/bin/bash\n\n# 1. commit to bump the version and update the changelog/readme\n# 2. tag that commit\n# 3. use dist.sh to produce tar.gz for all platforms\n# 4. aws s3 cp dist s3://bitly-downloads/nsq/ --recursive --exclude \"*\" --include \"nsq-1.3.0*\" --profile bitly --acl public-read\n# 5. docker manifest push nsqio/nsq:latest\n# 6. push to nsqio/master\n# 7. update the release metadata on github / upload the binaries\n# 8. update nsqio/nsqio.github.io/_posts/2014-03-01-installing.md\n# 9. send release announcement emails\n# 10. update IRC channel topic\n# 11. tweet\n\nset -e\n\nDIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nrm -rf $DIR/dist/docker\nmkdir -p $DIR/dist/docker\n\nGOFLAGS='-ldflags=\"-s -w\"'\nversion=$(awk '/const Binary/ {print $NF}' < $DIR/internal/version/binary.go | sed 's/\"//g')\ngoversion=$(go version | awk '{print $3}')\n\necho \"... running tests\"\n./test.sh\n\nexport GO111MODULE=on\nfor target in \"linux/amd64\" \"linux/arm64\" \"darwin/amd64\" \"darwin/arm64\" \"freebsd/amd64\" \"windows/amd64\"; do\n os=${target%/*}\n arch=${target##*/}\n echo \"... building v$version for $os/$arch\"\n BUILD=$(mktemp -d ${TMPDIR:-/tmp}/nsq-XXXXX)\n TARGET=\"nsq-$version.$os-$arch.$goversion\"\n GOOS=$os GOARCH=$arch CGO_ENABLED=0 \\\n make DESTDIR=$BUILD PREFIX=/$TARGET BLDFLAGS=\"$GOFLAGS\" install\n pushd $BUILD\n sudo chown -R 0:0 $TARGET\n tar czvf $TARGET.tar.gz $TARGET\n mv $TARGET.tar.gz $DIR/dist\n popd\n make clean\n sudo rm -r $BUILD\ndone\n\nrnd=$(LC_ALL=C tr -dc 'a-zA-Z0-9' < /dev/urandom | head -c10)\ndocker buildx create --use --name nsq-$rnd\ndocker buildx build .\\\n --tag nsqio/nsq:v$version \\\n --platform linux/amd64,linux/arm64 \\\n --output type=image,push=true\nif [[ ! $version == *\"-\"* ]]; then\n echo \"Tagging nsqio/nsq:v$version as the latest release\"\n shas=$(docker manifest inspect nsqio/nsq:v$version |\\\n grep digest | awk '{print $2}' | sed 's/[\",]//g' | sed 's/^/nsqio\\/nsq@/')\n docker manifest create --amend nsqio/nsq:latest $shas\nfi\n" }, { "path": "fmt.sh", "content": "#!/bin/bash\nfind . -name \"*.go\" | xargs goimports -w\n" }, { "path": "go.mod", "content": "module github.com/nsqio/nsq\n\ngo 1.17\n\nrequire (\n\tgithub.com/BurntSushi/toml v1.3.2\n\tgithub.com/bitly/go-hostpool v0.1.0\n\tgithub.com/bitly/timer_metrics v1.0.0\n\tgithub.com/blang/semver v3.5.1+incompatible\n\tgithub.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b\n\tgithub.com/golang/snappy v0.0.4\n\tgithub.com/judwhite/go-svc v1.2.1\n\tgithub.com/julienschmidt/httprouter v1.3.0\n\tgithub.com/mreiferson/go-options v1.0.0\n\tgithub.com/nsqio/go-diskqueue v1.1.0\n\tgithub.com/nsqio/go-nsq v1.1.0\n)\n\nrequire (\n\tgithub.com/stretchr/testify v1.9.0 // indirect\n\tgolang.org/x/sys v0.10.0 // indirect\n)\n\nreplace github.com/judwhite/go-svc => github.com/mreiferson/go-svc v1.2.2-0.20210815184239-7a96e00010f6\n" }, { "path": "go.sum", "content": "github.com/BurntSushi/toml v1.3.2 h1:o7IhLm0Msx3BaB+n3Ag7L8EVlByGnpq14C4YWiu/gL8=\ngithub.com/BurntSushi/toml v1.3.2/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=\ngithub.com/bitly/go-hostpool v0.1.0 h1:XKmsF6k5el6xHG3WPJ8U0Ku/ye7njX7W81Ng7O2ioR0=\ngithub.com/bitly/go-hostpool v0.1.0/go.mod h1:4gOCgp6+NZnVqlKyZ/iBZFTAJKembaVENUpMkpg42fw=\ngithub.com/bitly/timer_metrics v1.0.0 h1:bbszVIl0vT5+/cdZx8L4KOQmM8mC/0y3EBICGSxyhCk=\ngithub.com/bitly/timer_metrics v1.0.0/go.mod h1:87z4/LSg3f++tMqZwZlsLwPuJu6xloyJ7Qm40NyEkLs=\ngithub.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ=\ngithub.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk=\ngithub.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b h1:AP/Y7sqYicnjGDfD5VcY4CIfh1hRXBUavxrvELjTiOE=\ngithub.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b/go.mod h1:ac9efd0D1fsDb3EJvhqgXRbFx7bs2wqZ10HQPeU8U/Q=\ngithub.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=\ngithub.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=\ngithub.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=\ngithub.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=\ngithub.com/julienschmidt/httprouter v1.3.0 h1:U0609e9tgbseu3rBINet9P48AI/D3oJs4dN7jwJOQ1U=\ngithub.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=\ngithub.com/mreiferson/go-options v1.0.0 h1:RMLidydGlDWpL+lQTXo0bVIf/XT2CTq7AEJMoz5/VWs=\ngithub.com/mreiferson/go-options v1.0.0/go.mod h1:zHtCks/HQvOt8ATyfwVe3JJq2PPuImzXINPRTC03+9w=\ngithub.com/mreiferson/go-svc v1.2.2-0.20210815184239-7a96e00010f6 h1:NbuBXARvEXrYZ1SzN53ZpObeuwGhl1zvs/C+kzCggrQ=\ngithub.com/mreiferson/go-svc v1.2.2-0.20210815184239-7a96e00010f6/go.mod h1:mo/P2JNX8C07ywpP9YtO2gnBgnUiFTHqtsZekJrUuTk=\ngithub.com/nsqio/go-diskqueue v1.1.0 h1:r0dJ0DMXT3+2mOq+79cvCjnhoBxyGC2S9O+OjQrpe4Q=\ngithub.com/nsqio/go-diskqueue v1.1.0/go.mod h1:INuJIxl4ayUsyoNtHL5+9MFPDfSZ0zY93hNY6vhBRsI=\ngithub.com/nsqio/go-nsq v1.1.0 h1:PQg+xxiUjA7V+TLdXw7nVrJ5Jbl3sN86EhGCQj4+FYE=\ngithub.com/nsqio/go-nsq v1.1.0/go.mod h1:vKq36oyeVXgsS5Q8YEO7WghqidAVXQlcFxzQbQTuDEY=\ngithub.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=\ngithub.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=\ngithub.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=\ngithub.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=\ngithub.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=\ngithub.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=\ngithub.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=\ngithub.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=\ngithub.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=\ngithub.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=\ngithub.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=\ngithub.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=\ngolang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.10.0 h1:SqMFp9UcQJZa+pmYuAKjd9xq1f0j5rLcDIk0mj4qAsA=\ngolang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=\ngopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=\ngopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=\ngopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=\n" }, { "path": "internal/app/float_array.go", "content": "package app\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"sort\"\n\t\"strconv\"\n\t\"strings\"\n)\n\ntype FloatArray []float64\n\nfunc (a *FloatArray) Get() interface{} { return []float64(*a) }\n\nfunc (a *FloatArray) Set(param string) error {\n\tfor _, s := range strings.Split(param, \",\") {\n\t\tv, err := strconv.ParseFloat(s, 64)\n\t\tif err != nil {\n\t\t\tlog.Fatalf(\"Could not parse: %s\", s)\n\t\t\treturn nil\n\t\t}\n\t\t*a = append(*a, v)\n\t}\n\tsort.Sort(*a)\n\treturn nil\n}\n\nfunc (a FloatArray) Swap(i, j int) { a[i], a[j] = a[j], a[i] }\nfunc (a FloatArray) Less(i, j int) bool { return a[i] > a[j] }\nfunc (a FloatArray) Len() int { return len(a) }\n\nfunc (a *FloatArray) String() string {\n\tvar s []string\n\tfor _, v := range *a {\n\t\ts = append(s, fmt.Sprintf(\"%f\", v))\n\t}\n\treturn strings.Join(s, \",\")\n}\n" }, { "path": "internal/app/string_array.go", "content": "package app\n\nimport (\n\t\"strings\"\n)\n\ntype StringArray []string\n\nfunc (a *StringArray) Get() interface{} { return []string(*a) }\n\nfunc (a *StringArray) Set(s string) error {\n\t*a = append(*a, s)\n\treturn nil\n}\n\nfunc (a *StringArray) String() string {\n\treturn strings.Join(*a, \",\")\n}\n" }, { "path": "internal/auth/authorizations.go", "content": "package auth\n\nimport (\n\t\"crypto/tls\"\n\t\"errors\"\n\t\"fmt\"\n\t\"math/rand\"\n\t\"net/url\"\n\t\"regexp\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/http_api\"\n)\n\ntype Authorization struct {\n\tTopic string `json:\"topic\"`\n\tChannels []string `json:\"channels\"`\n\tPermissions []string `json:\"permissions\"`\n}\n\ntype State struct {\n\tTTL int `json:\"ttl\"`\n\tAuthorizations []Authorization `json:\"authorizations\"`\n\tIdentity string `json:\"identity\"`\n\tIdentityURL string `json:\"identity_url\"`\n\tExpires time.Time\n}\n\nfunc (a *Authorization) HasPermission(permission string) bool {\n\tfor _, p := range a.Permissions {\n\t\tif permission == p {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc (a *Authorization) IsAllowed(topic, channel string) bool {\n\tif channel != \"\" {\n\t\tif !a.HasPermission(\"subscribe\") {\n\t\t\treturn false\n\t\t}\n\t} else {\n\t\tif !a.HasPermission(\"publish\") {\n\t\t\treturn false\n\t\t}\n\t}\n\n\ttopicRegex := regexp.MustCompile(a.Topic)\n\n\tif !topicRegex.MatchString(topic) {\n\t\treturn false\n\t}\n\n\tfor _, c := range a.Channels {\n\t\tchannelRegex := regexp.MustCompile(c)\n\t\tif channelRegex.MatchString(channel) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc (a *State) IsAllowed(topic, channel string) bool {\n\tfor _, aa := range a.Authorizations {\n\t\tif aa.IsAllowed(topic, channel) {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc (a *State) IsExpired() bool {\n\treturn a.Expires.Before(time.Now())\n}\n\nfunc QueryAnyAuthd(authd []string, remoteIP string, tlsEnabled bool, commonName string, authSecret string,\n\tclientTLSConfig *tls.Config, connectTimeout time.Duration, requestTimeout time.Duration, httpRequestMethod string) (*State, error) {\n\tvar retErr error\n\tstart := rand.Int()\n\tn := len(authd)\n\tfor i := 0; i < n; i++ {\n\t\ta := authd[(i+start)%n]\n\t\tauthState, err := QueryAuthd(a, remoteIP, tlsEnabled, commonName, authSecret, clientTLSConfig, connectTimeout, requestTimeout, httpRequestMethod)\n\t\tif err != nil {\n\t\t\tes := fmt.Sprintf(\"failed to auth against %s - %s\", a, err)\n\t\t\tif retErr != nil {\n\t\t\t\tes = fmt.Sprintf(\"%s; %s\", retErr, es)\n\t\t\t}\n\t\t\tretErr = errors.New(es)\n\t\t\tcontinue\n\t\t}\n\t\treturn authState, nil\n\t}\n\treturn nil, retErr\n}\n\nfunc QueryAuthd(authd string, remoteIP string, tlsEnabled bool, commonName string, authSecret string,\n\tclientTLSConfig *tls.Config, connectTimeout time.Duration, requestTimeout time.Duration, httpRequestMethod string) (*State, error) {\n\tvar authState State\n\tv := url.Values{}\n\tv.Set(\"remote_ip\", remoteIP)\n\tif tlsEnabled {\n\t\tv.Set(\"tls\", \"true\")\n\t} else {\n\t\tv.Set(\"tls\", \"false\")\n\t}\n\tv.Set(\"secret\", authSecret)\n\tv.Set(\"common_name\", commonName)\n\n\tvar endpoint string\n\tif strings.Contains(authd, \"://\") {\n\t\tendpoint = authd\n\t} else {\n\t\tendpoint = fmt.Sprintf(\"http://%s/auth\", authd)\n\t}\n\n\tclient := http_api.NewClient(clientTLSConfig, connectTimeout, requestTimeout)\n\tif httpRequestMethod == \"post\" {\n\t\tif err := client.POSTV1(endpoint, v, &authState); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t} else {\n\t\tendpoint = fmt.Sprintf(\"%s?%s\", endpoint, v.Encode())\n\t\tif err := client.GETV1(endpoint, &authState); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t}\n\n\t// validation on response\n\tfor _, auth := range authState.Authorizations {\n\t\tfor _, p := range auth.Permissions {\n\t\t\tswitch p {\n\t\t\tcase \"subscribe\", \"publish\":\n\t\t\tdefault:\n\t\t\t\treturn nil, fmt.Errorf(\"unknown permission %s\", p)\n\t\t\t}\n\t\t}\n\n\t\tif _, err := regexp.Compile(auth.Topic); err != nil {\n\t\t\treturn nil, fmt.Errorf(\"unable to compile topic %q %s\", auth.Topic, err)\n\t\t}\n\n\t\tfor _, channel := range auth.Channels {\n\t\t\tif _, err := regexp.Compile(channel); err != nil {\n\t\t\t\treturn nil, fmt.Errorf(\"unable to compile channel %q %s\", channel, err)\n\t\t\t}\n\t\t}\n\t}\n\n\tif authState.TTL <= 0 {\n\t\treturn nil, fmt.Errorf(\"invalid TTL %d (must be >0)\", authState.TTL)\n\t}\n\n\tauthState.Expires = time.Now().Add(time.Duration(authState.TTL) * time.Second)\n\treturn &authState, nil\n}\n" }, { "path": "internal/clusterinfo/data.go", "content": "package clusterinfo\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"net/url\"\n\t\"sort\"\n\t\"strconv\"\n\t\"strings\"\n\t\"sync\"\n\n\t\"github.com/blang/semver\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/stringy\"\n)\n\ntype PartialErr interface {\n\terror\n\tErrors() []error\n}\n\ntype ErrList []error\n\nfunc (l ErrList) Error() string {\n\tvar es []string\n\tfor _, e := range l {\n\t\tes = append(es, e.Error())\n\t}\n\treturn strings.Join(es, \"\\n\")\n}\n\nfunc (l ErrList) Errors() []error {\n\treturn l\n}\n\ntype ClusterInfo struct {\n\tlog lg.AppLogFunc\n\tclient *http_api.Client\n}\n\nfunc New(log lg.AppLogFunc, client *http_api.Client) *ClusterInfo {\n\treturn &ClusterInfo{\n\t\tlog: log,\n\t\tclient: client,\n\t}\n}\n\nfunc (c *ClusterInfo) logf(f string, args ...interface{}) {\n\tif c.log != nil {\n\t\tc.log(lg.INFO, f, args...)\n\t}\n}\n\n// GetVersion returns a semver.Version object by querying /info\nfunc (c *ClusterInfo) GetVersion(addr string) (semver.Version, error) {\n\tendpoint := fmt.Sprintf(\"http://%s/info\", addr)\n\tvar resp struct {\n\t\tVersion string `json:\"version\"`\n\t}\n\terr := c.client.GETV1(endpoint, &resp)\n\tif err != nil {\n\t\treturn semver.Version{}, err\n\t}\n\tif resp.Version == \"\" {\n\t\tresp.Version = \"unknown\"\n\t}\n\treturn semver.Parse(resp.Version)\n}\n\n// GetLookupdTopics returns a []string containing a union of all the topics\n// from all the given nsqlookupd\nfunc (c *ClusterInfo) GetLookupdTopics(lookupdHTTPAddrs []string) ([]string, error) {\n\tvar topics []string\n\tvar lock sync.Mutex\n\tvar wg sync.WaitGroup\n\tvar errs []error\n\n\ttype respType struct {\n\t\tTopics []string `json:\"topics\"`\n\t}\n\n\tfor _, addr := range lookupdHTTPAddrs {\n\t\twg.Add(1)\n\t\tgo func(addr string) {\n\t\t\tdefer wg.Done()\n\n\t\t\tendpoint := fmt.Sprintf(\"http://%s/topics\", addr)\n\t\t\tc.logf(\"CI: querying nsqlookupd %s\", endpoint)\n\n\t\t\tvar resp respType\n\t\t\terr := c.client.GETV1(endpoint, &resp)\n\t\t\tif err != nil {\n\t\t\t\tlock.Lock()\n\t\t\t\terrs = append(errs, err)\n\t\t\t\tlock.Unlock()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tlock.Lock()\n\t\t\tdefer lock.Unlock()\n\t\t\ttopics = append(topics, resp.Topics...)\n\t\t}(addr)\n\t}\n\twg.Wait()\n\n\tif len(errs) == len(lookupdHTTPAddrs) {\n\t\treturn nil, fmt.Errorf(\"failed to query any nsqlookupd: %s\", ErrList(errs))\n\t}\n\n\ttopics = stringy.Uniq(topics)\n\tsort.Strings(topics)\n\n\tif len(errs) > 0 {\n\t\treturn topics, ErrList(errs)\n\t}\n\treturn topics, nil\n}\n\n// GetLookupdTopicChannels returns a []string containing a union of all the channels\n// from all the given lookupd for the given topic\nfunc (c *ClusterInfo) GetLookupdTopicChannels(topic string, lookupdHTTPAddrs []string) ([]string, error) {\n\tvar channels []string\n\tvar lock sync.Mutex\n\tvar wg sync.WaitGroup\n\tvar errs []error\n\n\ttype respType struct {\n\t\tChannels []string `json:\"channels\"`\n\t}\n\n\tfor _, addr := range lookupdHTTPAddrs {\n\t\twg.Add(1)\n\t\tgo func(addr string) {\n\t\t\tdefer wg.Done()\n\n\t\t\tendpoint := fmt.Sprintf(\"http://%s/channels?topic=%s\", addr, url.QueryEscape(topic))\n\t\t\tc.logf(\"CI: querying nsqlookupd %s\", endpoint)\n\n\t\t\tvar resp respType\n\t\t\terr := c.client.GETV1(endpoint, &resp)\n\t\t\tif err != nil {\n\t\t\t\tlock.Lock()\n\t\t\t\terrs = append(errs, err)\n\t\t\t\tlock.Unlock()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tlock.Lock()\n\t\t\tdefer lock.Unlock()\n\t\t\tchannels = append(channels, resp.Channels...)\n\t\t}(addr)\n\t}\n\twg.Wait()\n\n\tif len(errs) == len(lookupdHTTPAddrs) {\n\t\treturn nil, fmt.Errorf(\"failed to query any nsqlookupd: %s\", ErrList(errs))\n\t}\n\n\tchannels = stringy.Uniq(channels)\n\tsort.Strings(channels)\n\n\tif len(errs) > 0 {\n\t\treturn channels, ErrList(errs)\n\t}\n\treturn channels, nil\n}\n\n// GetLookupdProducers returns Producers of all the nsqd connected to the given lookupds\nfunc (c *ClusterInfo) GetLookupdProducers(lookupdHTTPAddrs []string) (Producers, error) {\n\tvar producers []*Producer\n\tvar lock sync.Mutex\n\tvar wg sync.WaitGroup\n\tvar errs []error\n\n\tproducersByAddr := make(map[string]*Producer)\n\tmaxVersion, _ := semver.Parse(\"0.0.0\")\n\n\ttype respType struct {\n\t\tProducers []*Producer `json:\"producers\"`\n\t}\n\n\tfor _, addr := range lookupdHTTPAddrs {\n\t\twg.Add(1)\n\t\tgo func(addr string) {\n\t\t\tdefer wg.Done()\n\n\t\t\tendpoint := fmt.Sprintf(\"http://%s/nodes\", addr)\n\t\t\tc.logf(\"CI: querying nsqlookupd %s\", endpoint)\n\n\t\t\tvar resp respType\n\t\t\terr := c.client.GETV1(endpoint, &resp)\n\t\t\tif err != nil {\n\t\t\t\tlock.Lock()\n\t\t\t\terrs = append(errs, err)\n\t\t\t\tlock.Unlock()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tlock.Lock()\n\t\t\tdefer lock.Unlock()\n\t\t\tfor _, producer := range resp.Producers {\n\t\t\t\tkey := producer.TCPAddress()\n\t\t\t\tp, ok := producersByAddr[key]\n\t\t\t\tif !ok {\n\t\t\t\t\tproducersByAddr[key] = producer\n\t\t\t\t\tproducers = append(producers, producer)\n\t\t\t\t\tif maxVersion.LT(producer.VersionObj) {\n\t\t\t\t\t\tmaxVersion = producer.VersionObj\n\t\t\t\t\t}\n\t\t\t\t\tsort.Sort(producer.Topics)\n\t\t\t\t\tp = producer\n\t\t\t\t}\n\t\t\t\tp.RemoteAddresses = append(p.RemoteAddresses,\n\t\t\t\t\tfmt.Sprintf(\"%s/%s\", addr, producer.Address()))\n\t\t\t}\n\t\t}(addr)\n\t}\n\twg.Wait()\n\n\tif len(errs) == len(lookupdHTTPAddrs) {\n\t\treturn nil, fmt.Errorf(\"failed to query any nsqlookupd: %s\", ErrList(errs))\n\t}\n\n\tfor _, producer := range producersByAddr {\n\t\tif producer.VersionObj.LT(maxVersion) {\n\t\t\tproducer.OutOfDate = true\n\t\t}\n\t}\n\tsort.Sort(ProducersByHost{producers})\n\n\tif len(errs) > 0 {\n\t\treturn producers, ErrList(errs)\n\t}\n\treturn producers, nil\n}\n\n// GetLookupdTopicProducers returns Producers of all the nsqd for a given topic by\n// unioning the nodes returned from the given lookupd\nfunc (c *ClusterInfo) GetLookupdTopicProducers(topic string, lookupdHTTPAddrs []string) (Producers, error) {\n\tvar producers Producers\n\tvar lock sync.Mutex\n\tvar wg sync.WaitGroup\n\tvar errs []error\n\n\ttype respType struct {\n\t\tProducers Producers `json:\"producers\"`\n\t}\n\n\tfor _, addr := range lookupdHTTPAddrs {\n\t\twg.Add(1)\n\t\tgo func(addr string) {\n\t\t\tdefer wg.Done()\n\n\t\t\tendpoint := fmt.Sprintf(\"http://%s/lookup?topic=%s\", addr, url.QueryEscape(topic))\n\t\t\tc.logf(\"CI: querying nsqlookupd %s\", endpoint)\n\n\t\t\tvar resp respType\n\t\t\terr := c.client.GETV1(endpoint, &resp)\n\t\t\tif err != nil {\n\t\t\t\tlock.Lock()\n\t\t\t\terrs = append(errs, err)\n\t\t\t\tlock.Unlock()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tlock.Lock()\n\t\t\tdefer lock.Unlock()\n\t\t\tfor _, p := range resp.Producers {\n\t\t\t\tfor _, pp := range producers {\n\t\t\t\t\tif p.HTTPAddress() == pp.HTTPAddress() {\n\t\t\t\t\t\tgoto skip\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tproducers = append(producers, p)\n\t\t\tskip:\n\t\t\t}\n\t\t}(addr)\n\t}\n\twg.Wait()\n\n\tif len(errs) == len(lookupdHTTPAddrs) {\n\t\treturn nil, fmt.Errorf(\"failed to query any nsqlookupd: %s\", ErrList(errs))\n\t}\n\tif len(errs) > 0 {\n\t\treturn producers, ErrList(errs)\n\t}\n\treturn producers, nil\n}\n\n// GetNSQDTopics returns a []string containing all the topics produced by the given nsqd\nfunc (c *ClusterInfo) GetNSQDTopics(nsqdHTTPAddrs []string) ([]string, error) {\n\tvar topics []string\n\tvar lock sync.Mutex\n\tvar wg sync.WaitGroup\n\tvar errs []error\n\n\ttype respType struct {\n\t\tTopics []struct {\n\t\t\tName string `json:\"topic_name\"`\n\t\t} `json:\"topics\"`\n\t}\n\n\tfor _, addr := range nsqdHTTPAddrs {\n\t\twg.Add(1)\n\t\tgo func(addr string) {\n\t\t\tdefer wg.Done()\n\n\t\t\tendpoint := fmt.Sprintf(\"http://%s/stats?format=json\", addr)\n\t\t\tc.logf(\"CI: querying nsqd %s\", endpoint)\n\n\t\t\tvar resp respType\n\t\t\terr := c.client.GETV1(endpoint, &resp)\n\t\t\tif err != nil {\n\t\t\t\tlock.Lock()\n\t\t\t\terrs = append(errs, err)\n\t\t\t\tlock.Unlock()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tlock.Lock()\n\t\t\tdefer lock.Unlock()\n\t\t\tfor _, topic := range resp.Topics {\n\t\t\t\ttopics = stringy.Add(topics, topic.Name)\n\t\t\t}\n\t\t}(addr)\n\t}\n\twg.Wait()\n\n\tif len(errs) == len(nsqdHTTPAddrs) {\n\t\treturn nil, fmt.Errorf(\"failed to query any nsqd: %s\", ErrList(errs))\n\t}\n\n\tsort.Strings(topics)\n\n\tif len(errs) > 0 {\n\t\treturn topics, ErrList(errs)\n\t}\n\treturn topics, nil\n}\n\n// GetNSQDProducers returns Producers of all the given nsqd\nfunc (c *ClusterInfo) GetNSQDProducers(nsqdHTTPAddrs []string) (Producers, error) {\n\tvar producers Producers\n\tvar lock sync.Mutex\n\tvar wg sync.WaitGroup\n\tvar errs []error\n\n\ttype infoRespType struct {\n\t\tVersion string `json:\"version\"`\n\t\tBroadcastAddress string `json:\"broadcast_address\"`\n\t\tHostname string `json:\"hostname\"`\n\t\tHTTPPort int `json:\"http_port\"`\n\t\tTCPPort int `json:\"tcp_port\"`\n\t\tTopologyZone string `json:\"topology_zone,omitempty\"`\n\t\tTopologyRegion string `json:\"topology_region,omitempty\"`\n\t}\n\n\ttype statsRespType struct {\n\t\tTopics []struct {\n\t\t\tName string `json:\"topic_name\"`\n\t\t} `json:\"topics\"`\n\t}\n\n\tfor _, addr := range nsqdHTTPAddrs {\n\t\twg.Add(1)\n\t\tgo func(addr string) {\n\t\t\tdefer wg.Done()\n\n\t\t\tendpoint := fmt.Sprintf(\"http://%s/info\", addr)\n\t\t\tc.logf(\"CI: querying nsqd %s\", endpoint)\n\n\t\t\tvar infoResp infoRespType\n\t\t\terr := c.client.GETV1(endpoint, &infoResp)\n\t\t\tif err != nil {\n\t\t\t\tlock.Lock()\n\t\t\t\terrs = append(errs, err)\n\t\t\t\tlock.Unlock()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tendpoint = fmt.Sprintf(\"http://%s/stats?format=json&include_clients=false\", addr)\n\t\t\tc.logf(\"CI: querying nsqd %s\", endpoint)\n\n\t\t\tvar statsResp statsRespType\n\t\t\terr = c.client.GETV1(endpoint, &statsResp)\n\t\t\tif err != nil {\n\t\t\t\tlock.Lock()\n\t\t\t\terrs = append(errs, err)\n\t\t\t\tlock.Unlock()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar producerTopics ProducerTopics\n\t\t\tfor _, t := range statsResp.Topics {\n\t\t\t\tproducerTopics = append(producerTopics, ProducerTopic{Topic: t.Name})\n\t\t\t}\n\n\t\t\tversion, err := semver.Parse(infoResp.Version)\n\t\t\tif err != nil {\n\t\t\t\tversion, _ = semver.Parse(\"0.0.0\")\n\t\t\t}\n\n\t\t\tlock.Lock()\n\t\t\tdefer lock.Unlock()\n\t\t\tproducers = append(producers, &Producer{\n\t\t\t\tVersion: infoResp.Version,\n\t\t\t\tVersionObj: version,\n\t\t\t\tBroadcastAddress: infoResp.BroadcastAddress,\n\t\t\t\tHostname: infoResp.Hostname,\n\t\t\t\tHTTPPort: infoResp.HTTPPort,\n\t\t\t\tTCPPort: infoResp.TCPPort,\n\t\t\t\tTopics: producerTopics,\n\t\t\t\tTopologyZone: infoResp.TopologyZone,\n\t\t\t\tTopologyRegion: infoResp.TopologyRegion,\n\t\t\t})\n\t\t}(addr)\n\t}\n\twg.Wait()\n\n\tif len(errs) == len(nsqdHTTPAddrs) {\n\t\treturn nil, fmt.Errorf(\"failed to query any nsqd: %s\", ErrList(errs))\n\t}\n\tif len(errs) > 0 {\n\t\treturn producers, ErrList(errs)\n\t}\n\treturn producers, nil\n}\n\n// GetNSQDTopicProducers returns Producers containing the addresses of all the nsqd\n// that produce the given topic\nfunc (c *ClusterInfo) GetNSQDTopicProducers(topic string, nsqdHTTPAddrs []string) (Producers, error) {\n\tvar producers Producers\n\tvar lock sync.Mutex\n\tvar wg sync.WaitGroup\n\tvar errs []error\n\n\ttype infoRespType struct {\n\t\tVersion string `json:\"version\"`\n\t\tBroadcastAddress string `json:\"broadcast_address\"`\n\t\tHostname string `json:\"hostname\"`\n\t\tHTTPPort int `json:\"http_port\"`\n\t\tTCPPort int `json:\"tcp_port\"`\n\t\tTopologyZone string `json:\"topology_zone,omitempty\"`\n\t\tTopologyRegion string `json:\"topology_region,omitempty\"`\n\t}\n\n\ttype statsRespType struct {\n\t\tTopics []struct {\n\t\t\tName string `json:\"topic_name\"`\n\t\t} `json:\"topics\"`\n\t}\n\n\tfor _, addr := range nsqdHTTPAddrs {\n\t\twg.Add(1)\n\t\tgo func(addr string) {\n\t\t\tdefer wg.Done()\n\n\t\t\tendpoint := fmt.Sprintf(\"http://%s/stats?format=json&topic=%s&include_clients=false\",\n\t\t\t\taddr, url.QueryEscape(topic))\n\t\t\tc.logf(\"CI: querying nsqd %s\", endpoint)\n\n\t\t\tvar statsResp statsRespType\n\t\t\terr := c.client.GETV1(endpoint, &statsResp)\n\t\t\tif err != nil {\n\t\t\t\tlock.Lock()\n\t\t\t\terrs = append(errs, err)\n\t\t\t\tlock.Unlock()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar producerTopics ProducerTopics\n\t\t\tfor _, t := range statsResp.Topics {\n\t\t\t\tproducerTopics = append(producerTopics, ProducerTopic{Topic: t.Name})\n\t\t\t}\n\n\t\t\tfor _, t := range statsResp.Topics {\n\t\t\t\tif t.Name == topic {\n\t\t\t\t\tendpoint := fmt.Sprintf(\"http://%s/info\", addr)\n\t\t\t\t\tc.logf(\"CI: querying nsqd %s\", endpoint)\n\n\t\t\t\t\tvar infoResp infoRespType\n\t\t\t\t\terr := c.client.GETV1(endpoint, &infoResp)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tlock.Lock()\n\t\t\t\t\t\terrs = append(errs, err)\n\t\t\t\t\t\tlock.Unlock()\n\t\t\t\t\t\treturn\n\t\t\t\t\t}\n\n\t\t\t\t\tversion, err := semver.Parse(infoResp.Version)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tversion, _ = semver.Parse(\"0.0.0\")\n\t\t\t\t\t}\n\n\t\t\t\t\t// if BroadcastAddress/HTTPPort are missing, use the values from `addr` for\n\t\t\t\t\t// backwards compatibility\n\n\t\t\t\t\tif infoResp.BroadcastAddress == \"\" {\n\t\t\t\t\t\tvar p string\n\t\t\t\t\t\tinfoResp.BroadcastAddress, p, _ = net.SplitHostPort(addr)\n\t\t\t\t\t\tinfoResp.HTTPPort, _ = strconv.Atoi(p)\n\t\t\t\t\t}\n\t\t\t\t\tif infoResp.Hostname == \"\" {\n\t\t\t\t\t\tinfoResp.Hostname, _, _ = net.SplitHostPort(addr)\n\t\t\t\t\t}\n\n\t\t\t\t\tlock.Lock()\n\t\t\t\t\tproducers = append(producers, &Producer{\n\t\t\t\t\t\tVersion: infoResp.Version,\n\t\t\t\t\t\tVersionObj: version,\n\t\t\t\t\t\tBroadcastAddress: infoResp.BroadcastAddress,\n\t\t\t\t\t\tHostname: infoResp.Hostname,\n\t\t\t\t\t\tHTTPPort: infoResp.HTTPPort,\n\t\t\t\t\t\tTCPPort: infoResp.TCPPort,\n\t\t\t\t\t\tTopics: producerTopics,\n\t\t\t\t\t\tTopologyZone: infoResp.TopologyZone,\n\t\t\t\t\t\tTopologyRegion: infoResp.TopologyRegion,\n\t\t\t\t\t})\n\t\t\t\t\tlock.Unlock()\n\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t}\n\t\t}(addr)\n\t}\n\twg.Wait()\n\n\tif len(errs) == len(nsqdHTTPAddrs) {\n\t\treturn nil, fmt.Errorf(\"failed to query any nsqd: %s\", ErrList(errs))\n\t}\n\tif len(errs) > 0 {\n\t\treturn producers, ErrList(errs)\n\t}\n\treturn producers, nil\n}\n\n// GetNSQDStats returns aggregate topic and channel stats from the given Producers\n//\n// if selectedChannel is empty, this will return stats for topic/channel\n// if selectedTopic is empty, this will return stats for *all* topic/channels\n// if includeClients is false, this will *not* return client stats for channels\n// and the ChannelStats dict will be keyed by topic + ':' + channel\nfunc (c *ClusterInfo) GetNSQDStats(producers Producers,\n\tselectedTopic string, selectedChannel string,\n\tincludeClients bool) ([]*TopicStats, map[string]*ChannelStats, error) {\n\tvar lock sync.Mutex\n\tvar wg sync.WaitGroup\n\tvar topicStatsList TopicStatsList\n\tvar errs []error\n\n\tchannelStatsMap := make(map[string]*ChannelStats)\n\n\ttype respType struct {\n\t\tTopics []*TopicStats `json:\"topics\"`\n\t}\n\n\tfor _, p := range producers {\n\t\twg.Add(1)\n\t\tgo func(p *Producer) {\n\t\t\tdefer wg.Done()\n\n\t\t\taddr := p.HTTPAddress()\n\n\t\t\tendpoint := fmt.Sprintf(\"http://%s/stats?format=json\", addr)\n\t\t\tif selectedTopic != \"\" {\n\t\t\t\tendpoint += \"&topic=\" + url.QueryEscape(selectedTopic)\n\t\t\t\tif selectedChannel != \"\" {\n\t\t\t\t\tendpoint += \"&channel=\" + url.QueryEscape(selectedChannel)\n\t\t\t\t}\n\t\t\t}\n\t\t\tif !includeClients {\n\t\t\t\tendpoint += \"&include_clients=false\"\n\t\t\t}\n\n\t\t\tc.logf(\"CI: querying nsqd %s\", endpoint)\n\n\t\t\tvar resp respType\n\t\t\terr := c.client.GETV1(endpoint, &resp)\n\t\t\tif err != nil {\n\t\t\t\tlock.Lock()\n\t\t\t\terrs = append(errs, err)\n\t\t\t\tlock.Unlock()\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tlock.Lock()\n\t\t\tdefer lock.Unlock()\n\t\t\tfor _, topic := range resp.Topics {\n\t\t\t\ttopic.Node = addr\n\t\t\t\ttopic.Hostname = p.Hostname\n\t\t\t\ttopic.MemoryDepth = topic.Depth - topic.BackendDepth\n\t\t\t\ttopic.DeliveryMsgCount = topic.ZoneLocalMsgCount + topic.RegionLocalMsgCount + topic.GlobalMsgCount\n\t\t\t\tif selectedTopic != \"\" && topic.TopicName != selectedTopic {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\ttopicStatsList = append(topicStatsList, topic)\n\n\t\t\t\tfor _, channel := range topic.Channels {\n\t\t\t\t\tchannel.Node = addr\n\t\t\t\t\tchannel.Hostname = p.Hostname\n\t\t\t\t\tchannel.TopicName = topic.TopicName\n\t\t\t\t\tchannel.MemoryDepth = channel.Depth - channel.BackendDepth\n\t\t\t\t\tchannel.DeliveryMsgCount = channel.ZoneLocalMsgCount + channel.RegionLocalMsgCount + channel.GlobalMsgCount\n\t\t\t\t\tkey := channel.ChannelName\n\t\t\t\t\tif selectedTopic == \"\" {\n\t\t\t\t\t\tkey = fmt.Sprintf(\"%s:%s\", topic.TopicName, channel.ChannelName)\n\t\t\t\t\t}\n\t\t\t\t\tchannelStats, ok := channelStatsMap[key]\n\t\t\t\t\tif !ok {\n\t\t\t\t\t\tchannelStats = &ChannelStats{\n\t\t\t\t\t\t\tNode: addr,\n\t\t\t\t\t\t\tTopicName: topic.TopicName,\n\t\t\t\t\t\t\tChannelName: channel.ChannelName,\n\t\t\t\t\t\t}\n\t\t\t\t\t\tchannelStatsMap[key] = channelStats\n\t\t\t\t\t}\n\t\t\t\t\tfor _, c := range channel.Clients {\n\t\t\t\t\t\tc.Node = addr\n\t\t\t\t\t\tc.NodeTopologyRegion = p.TopologyRegion\n\t\t\t\t\t\tc.NodeTopologyZone = p.TopologyZone\n\t\t\t\t\t}\n\t\t\t\t\tchannelStats.Add(channel)\n\t\t\t\t}\n\t\t\t}\n\t\t}(p)\n\t}\n\twg.Wait()\n\n\tif len(errs) == len(producers) {\n\t\treturn nil, nil, fmt.Errorf(\"failed to query any nsqd: %s\", ErrList(errs))\n\t}\n\n\tsort.Sort(TopicStatsByHost{topicStatsList})\n\n\tif len(errs) > 0 {\n\t\treturn topicStatsList, channelStatsMap, ErrList(errs)\n\t}\n\treturn topicStatsList, channelStatsMap, nil\n}\n\n// TombstoneNodeForTopic tombstones the given node for the given topic on all the given nsqlookupd\n// and deletes the topic from the node\nfunc (c *ClusterInfo) TombstoneNodeForTopic(topic string, node string, lookupdHTTPAddrs []string) error {\n\tvar errs []error\n\n\t// tombstone the topic on all the lookupds\n\tqs := fmt.Sprintf(\"topic=%s&node=%s\", url.QueryEscape(topic), url.QueryEscape(node))\n\terr := c.nsqlookupdPOST(lookupdHTTPAddrs, \"topic/tombstone\", qs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\tproducers, err := c.GetNSQDProducers([]string{node})\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\t// delete the topic on the producer\n\tqs = fmt.Sprintf(\"topic=%s\", url.QueryEscape(topic))\n\terr = c.producersPOST(producers, \"topic/delete\", qs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\tif len(errs) > 0 {\n\t\treturn ErrList(errs)\n\t}\n\treturn nil\n}\n\nfunc (c *ClusterInfo) CreateTopicChannel(topicName string, channelName string, lookupdHTTPAddrs []string) error {\n\tvar errs []error\n\n\t// create the topic on all the nsqlookupd\n\tqs := fmt.Sprintf(\"topic=%s\", url.QueryEscape(topicName))\n\terr := c.nsqlookupdPOST(lookupdHTTPAddrs, \"topic/create\", qs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\tif len(channelName) > 0 {\n\t\tqs := fmt.Sprintf(\"topic=%s&channel=%s\", url.QueryEscape(topicName), url.QueryEscape(channelName))\n\n\t\t// create the channel on all the nsqlookupd\n\t\terr := c.nsqlookupdPOST(lookupdHTTPAddrs, \"channel/create\", qs)\n\t\tif err != nil {\n\t\t\tpe, ok := err.(PartialErr)\n\t\t\tif !ok {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\terrs = append(errs, pe.Errors()...)\n\t\t}\n\n\t\t// create the channel on all the nsqd that produce the topic\n\t\tproducers, err := c.GetLookupdTopicProducers(topicName, lookupdHTTPAddrs)\n\t\tif err != nil {\n\t\t\tpe, ok := err.(PartialErr)\n\t\t\tif !ok {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\terrs = append(errs, pe.Errors()...)\n\t\t}\n\t\terr = c.producersPOST(producers, \"channel/create\", qs)\n\t\tif err != nil {\n\t\t\tpe, ok := err.(PartialErr)\n\t\t\tif !ok {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\terrs = append(errs, pe.Errors()...)\n\t\t}\n\t}\n\n\tif len(errs) > 0 {\n\t\treturn ErrList(errs)\n\t}\n\treturn nil\n}\n\nfunc (c *ClusterInfo) DeleteTopic(topicName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) error {\n\tvar errs []error\n\n\t// for topic removal, you need to get all the producers _first_\n\tproducers, err := c.GetTopicProducers(topicName, lookupdHTTPAddrs, nsqdHTTPAddrs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\tqs := fmt.Sprintf(\"topic=%s\", url.QueryEscape(topicName))\n\n\t// remove the topic from all the nsqlookupd\n\terr = c.nsqlookupdPOST(lookupdHTTPAddrs, \"topic/delete\", qs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\t// remove the topic from all the nsqd that produce this topic\n\terr = c.producersPOST(producers, \"topic/delete\", qs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\tif len(errs) > 0 {\n\t\treturn ErrList(errs)\n\t}\n\treturn nil\n}\n\nfunc (c *ClusterInfo) DeleteChannel(topicName string, channelName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) error {\n\tvar errs []error\n\n\tproducers, err := c.GetTopicProducers(topicName, lookupdHTTPAddrs, nsqdHTTPAddrs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\tqs := fmt.Sprintf(\"topic=%s&channel=%s\", url.QueryEscape(topicName), url.QueryEscape(channelName))\n\n\t// remove the channel from all the nsqlookupd\n\terr = c.nsqlookupdPOST(lookupdHTTPAddrs, \"channel/delete\", qs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\t// remove the channel from all the nsqd that produce this topic\n\terr = c.producersPOST(producers, \"channel/delete\", qs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\tif len(errs) > 0 {\n\t\treturn ErrList(errs)\n\t}\n\treturn nil\n}\n\nfunc (c *ClusterInfo) PauseTopic(topicName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) error {\n\tqs := fmt.Sprintf(\"topic=%s\", url.QueryEscape(topicName))\n\treturn c.actionHelper(topicName, lookupdHTTPAddrs, nsqdHTTPAddrs, \"topic/pause\", qs)\n}\n\nfunc (c *ClusterInfo) UnPauseTopic(topicName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) error {\n\tqs := fmt.Sprintf(\"topic=%s\", url.QueryEscape(topicName))\n\treturn c.actionHelper(topicName, lookupdHTTPAddrs, nsqdHTTPAddrs, \"topic/unpause\", qs)\n}\n\nfunc (c *ClusterInfo) PauseChannel(topicName string, channelName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) error {\n\tqs := fmt.Sprintf(\"topic=%s&channel=%s\", url.QueryEscape(topicName), url.QueryEscape(channelName))\n\treturn c.actionHelper(topicName, lookupdHTTPAddrs, nsqdHTTPAddrs, \"channel/pause\", qs)\n}\n\nfunc (c *ClusterInfo) UnPauseChannel(topicName string, channelName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) error {\n\tqs := fmt.Sprintf(\"topic=%s&channel=%s\", url.QueryEscape(topicName), url.QueryEscape(channelName))\n\treturn c.actionHelper(topicName, lookupdHTTPAddrs, nsqdHTTPAddrs, \"channel/unpause\", qs)\n}\n\nfunc (c *ClusterInfo) EmptyTopic(topicName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) error {\n\tqs := fmt.Sprintf(\"topic=%s\", url.QueryEscape(topicName))\n\treturn c.actionHelper(topicName, lookupdHTTPAddrs, nsqdHTTPAddrs, \"topic/empty\", qs)\n}\n\nfunc (c *ClusterInfo) EmptyChannel(topicName string, channelName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) error {\n\tqs := fmt.Sprintf(\"topic=%s&channel=%s\", url.QueryEscape(topicName), url.QueryEscape(channelName))\n\treturn c.actionHelper(topicName, lookupdHTTPAddrs, nsqdHTTPAddrs, \"channel/empty\", qs)\n}\n\nfunc (c *ClusterInfo) actionHelper(topicName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string, uri string, qs string) error {\n\tvar errs []error\n\n\tproducers, err := c.GetTopicProducers(topicName, lookupdHTTPAddrs, nsqdHTTPAddrs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\terr = c.producersPOST(producers, uri, qs)\n\tif err != nil {\n\t\tpe, ok := err.(PartialErr)\n\t\tif !ok {\n\t\t\treturn err\n\t\t}\n\t\terrs = append(errs, pe.Errors()...)\n\t}\n\n\tif len(errs) > 0 {\n\t\treturn ErrList(errs)\n\t}\n\treturn nil\n}\n\nfunc (c *ClusterInfo) GetProducers(lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) (Producers, error) {\n\tif len(lookupdHTTPAddrs) != 0 {\n\t\treturn c.GetLookupdProducers(lookupdHTTPAddrs)\n\t}\n\treturn c.GetNSQDProducers(nsqdHTTPAddrs)\n}\n\nfunc (c *ClusterInfo) GetTopicProducers(topicName string, lookupdHTTPAddrs []string, nsqdHTTPAddrs []string) (Producers, error) {\n\tif len(lookupdHTTPAddrs) != 0 {\n\t\treturn c.GetLookupdTopicProducers(topicName, lookupdHTTPAddrs)\n\t}\n\treturn c.GetNSQDTopicProducers(topicName, nsqdHTTPAddrs)\n}\n\nfunc (c *ClusterInfo) nsqlookupdPOST(addrs []string, uri string, qs string) error {\n\tvar errs []error\n\tfor _, addr := range addrs {\n\t\tendpoint := fmt.Sprintf(\"http://%s/%s?%s\", addr, uri, qs)\n\t\tc.logf(\"CI: querying nsqlookupd %s\", endpoint)\n\t\terr := c.client.POSTV1(endpoint, nil, nil)\n\t\tif err != nil {\n\t\t\terrs = append(errs, err)\n\t\t}\n\t}\n\tif len(errs) > 0 {\n\t\treturn ErrList(errs)\n\t}\n\treturn nil\n}\n\nfunc (c *ClusterInfo) producersPOST(pl Producers, uri string, qs string) error {\n\tvar errs []error\n\tfor _, p := range pl {\n\t\tendpoint := fmt.Sprintf(\"http://%s/%s?%s\", p.HTTPAddress(), uri, qs)\n\t\tc.logf(\"CI: querying nsqd %s\", endpoint)\n\t\terr := c.client.POSTV1(endpoint, nil, nil)\n\t\tif err != nil {\n\t\t\terrs = append(errs, err)\n\t\t}\n\t}\n\tif len(errs) > 0 {\n\t\treturn ErrList(errs)\n\t}\n\treturn nil\n}\n" }, { "path": "internal/clusterinfo/producer_test.go", "content": "package clusterinfo\n\nimport \"testing\"\n\nfunc TestHostNameAddresses(t *testing.T) {\n\tp := &Producer{\n\t\tBroadcastAddress: \"host.domain.com\",\n\t\tTCPPort: 4150,\n\t\tHTTPPort: 4151,\n\t}\n\n\tif p.HTTPAddress() != \"host.domain.com:4151\" {\n\t\tt.Errorf(\"Incorrect HTTPAddress: %s\", p.HTTPAddress())\n\t}\n\tif p.TCPAddress() != \"host.domain.com:4150\" {\n\t\tt.Errorf(\"Incorrect TCPAddress: %s\", p.TCPAddress())\n\t}\n}\n\nfunc TestIPv4Addresses(t *testing.T) {\n\tp := &Producer{\n\t\tBroadcastAddress: \"192.168.1.17\",\n\t\tTCPPort: 4150,\n\t\tHTTPPort: 4151,\n\t}\n\n\tif p.HTTPAddress() != \"192.168.1.17:4151\" {\n\t\tt.Errorf(\"Incorrect IPv4 HTTPAddress: %s\", p.HTTPAddress())\n\t}\n\tif p.TCPAddress() != \"192.168.1.17:4150\" {\n\t\tt.Errorf(\"Incorrect IPv4 TCPAddress: %s\", p.TCPAddress())\n\t}\n}\n\nfunc TestIPv6Addresses(t *testing.T) {\n\tp := &Producer{\n\t\tBroadcastAddress: \"fd4a:622f:d2f2::1\",\n\t\tTCPPort: 4150,\n\t\tHTTPPort: 4151,\n\t}\n\tif p.HTTPAddress() != \"[fd4a:622f:d2f2::1]:4151\" {\n\t\tt.Errorf(\"Incorrect IPv6 HTTPAddress: %s\", p.HTTPAddress())\n\t}\n\tif p.TCPAddress() != \"[fd4a:622f:d2f2::1]:4150\" {\n\t\tt.Errorf(\"Incorrect IPv6 TCPAddress: %s\", p.TCPAddress())\n\t}\n}\n" }, { "path": "internal/clusterinfo/types.go", "content": "package clusterinfo\n\nimport (\n\t\"encoding/json\"\n\t\"net\"\n\t\"sort\"\n\t\"strconv\"\n\t\"time\"\n\n\t\"github.com/blang/semver\"\n\t\"github.com/nsqio/nsq/internal/quantile\"\n)\n\ntype ProducerTopic struct {\n\tTopic string `json:\"topic\"`\n\tTombstoned bool `json:\"tombstoned\"`\n}\n\ntype ProducerTopics []ProducerTopic\n\nfunc (pt ProducerTopics) Len() int { return len(pt) }\nfunc (pt ProducerTopics) Swap(i, j int) { pt[i], pt[j] = pt[j], pt[i] }\nfunc (pt ProducerTopics) Less(i, j int) bool { return pt[i].Topic < pt[j].Topic }\n\ntype Producer struct {\n\tRemoteAddresses []string `json:\"remote_addresses\"`\n\tRemoteAddress string `json:\"remote_address\"`\n\tHostname string `json:\"hostname\"`\n\tBroadcastAddress string `json:\"broadcast_address\"`\n\tTCPPort int `json:\"tcp_port\"`\n\tHTTPPort int `json:\"http_port\"`\n\tVersion string `json:\"version\"`\n\tTopologyZone string `json:\"topology_zone,omitempty\"`\n\tTopologyRegion string `json:\"topology_region,omitempty\"`\n\tVersionObj semver.Version `json:\"-\"`\n\tTopics ProducerTopics `json:\"topics\"`\n\tOutOfDate bool `json:\"out_of_date\"`\n}\n\n// UnmarshalJSON implements json.Unmarshaler and postprocesses of ProducerTopics and VersionObj\nfunc (p *Producer) UnmarshalJSON(b []byte) error {\n\tvar r struct {\n\t\tRemoteAddress string `json:\"remote_address\"`\n\t\tHostname string `json:\"hostname\"`\n\t\tBroadcastAddress string `json:\"broadcast_address\"`\n\t\tTCPPort int `json:\"tcp_port\"`\n\t\tHTTPPort int `json:\"http_port\"`\n\t\tVersion string `json:\"version\"`\n\t\tTopics []string `json:\"topics\"`\n\t\tTombstoned []bool `json:\"tombstones\"`\n\t\tTopologyZone string `json:\"topology_zone,omitempty\"`\n\t\tTopologyRegion string `json:\"topology_region,omitempty\"`\n\t}\n\tif err := json.Unmarshal(b, &r); err != nil {\n\t\treturn err\n\t}\n\t*p = Producer{\n\t\tRemoteAddress: r.RemoteAddress,\n\t\tHostname: r.Hostname,\n\t\tBroadcastAddress: r.BroadcastAddress,\n\t\tTCPPort: r.TCPPort,\n\t\tHTTPPort: r.HTTPPort,\n\t\tVersion: r.Version,\n\t\tTopologyZone: r.TopologyZone,\n\t\tTopologyRegion: r.TopologyRegion,\n\t}\n\tfor i, t := range r.Topics {\n\t\tp.Topics = append(p.Topics, ProducerTopic{Topic: t, Tombstoned: r.Tombstoned[i]})\n\t}\n\tversion, err := semver.Parse(p.Version)\n\tif err != nil {\n\t\tversion, _ = semver.Parse(\"0.0.0\")\n\t}\n\tp.VersionObj = version\n\treturn nil\n}\n\nfunc (p *Producer) Address() string {\n\tif p.RemoteAddress == \"\" {\n\t\treturn \"N/A\"\n\t}\n\treturn p.RemoteAddress\n}\n\nfunc (p *Producer) HTTPAddress() string {\n\treturn net.JoinHostPort(p.BroadcastAddress, strconv.Itoa(p.HTTPPort))\n}\n\nfunc (p *Producer) TCPAddress() string {\n\treturn net.JoinHostPort(p.BroadcastAddress, strconv.Itoa(p.TCPPort))\n}\n\n// IsInconsistent checks for cases where an unexpected number of nsqd connections are\n// reporting the same information to nsqlookupd (ie: multiple instances are using the\n// same broadcast address), or cases where some nsqd are not reporting to all nsqlookupd.\nfunc (p *Producer) IsInconsistent(numLookupd int) bool {\n\treturn len(p.RemoteAddresses) != numLookupd\n}\n\ntype TopicStats struct {\n\tNode string `json:\"node\"`\n\tHostname string `json:\"hostname\"`\n\tTopicName string `json:\"topic_name\"`\n\tDepth int64 `json:\"depth\"`\n\tMemoryDepth int64 `json:\"memory_depth\"`\n\tBackendDepth int64 `json:\"backend_depth\"`\n\tMessageCount int64 `json:\"message_count\"`\n\tDeliveryMsgCount int64 `json:\"delivery_msg_count\"`\n\tZoneLocalMsgCount int64 `json:\"zone_local_msg_count,omitempty\"`\n\tRegionLocalMsgCount int64 `json:\"region_local_msg_count,omitempty\"`\n\tGlobalMsgCount int64 `json:\"global_msg_count,omitempty\"`\n\tNodeStats []*TopicStats `json:\"nodes\"`\n\tChannels []*ChannelStats `json:\"channels\"`\n\tPaused bool `json:\"paused\"`\n\n\tE2eProcessingLatency *quantile.E2eProcessingLatencyAggregate `json:\"e2e_processing_latency\"`\n}\n\nfunc (t *TopicStats) Add(a *TopicStats) {\n\tt.Node = \"*\"\n\tt.Depth += a.Depth\n\tt.MemoryDepth += a.MemoryDepth\n\tt.BackendDepth += a.BackendDepth\n\tt.MessageCount += a.MessageCount\n\tt.DeliveryMsgCount += a.DeliveryMsgCount\n\tt.ZoneLocalMsgCount += a.ZoneLocalMsgCount\n\tt.RegionLocalMsgCount += a.RegionLocalMsgCount\n\tt.GlobalMsgCount += a.GlobalMsgCount\n\tif a.Paused {\n\t\tt.Paused = a.Paused\n\t}\n\tfor _, aChannelStats := range a.Channels {\n\t\tfound := false\n\t\tfor _, channelStats := range t.Channels {\n\t\t\tif aChannelStats.ChannelName == channelStats.ChannelName {\n\t\t\t\tfound = true\n\t\t\t\tchannelStats.Add(aChannelStats)\n\t\t\t}\n\t\t}\n\t\tif !found {\n\t\t\tt.Channels = append(t.Channels, aChannelStats)\n\t\t}\n\t}\n\tt.NodeStats = append(t.NodeStats, a)\n\tsort.Sort(TopicStatsByHost{t.NodeStats})\n\tif t.E2eProcessingLatency == nil {\n\t\tt.E2eProcessingLatency = &quantile.E2eProcessingLatencyAggregate{\n\t\t\tAddr: t.Node,\n\t\t\tTopic: t.TopicName,\n\t\t}\n\t}\n\tt.E2eProcessingLatency.Add(a.E2eProcessingLatency)\n}\n\ntype ChannelStats struct {\n\tNode string `json:\"node\"`\n\tHostname string `json:\"hostname\"`\n\tTopicName string `json:\"topic_name\"`\n\tChannelName string `json:\"channel_name\"`\n\tDepth int64 `json:\"depth\"`\n\tMemoryDepth int64 `json:\"memory_depth\"`\n\tBackendDepth int64 `json:\"backend_depth\"`\n\tInFlightCount int64 `json:\"in_flight_count\"`\n\tDeferredCount int64 `json:\"deferred_count\"`\n\tRequeueCount int64 `json:\"requeue_count\"`\n\tTimeoutCount int64 `json:\"timeout_count\"`\n\tMessageCount int64 `json:\"message_count\"`\n\tDeliveryMsgCount int64 `json:\"delivery_msg_count,omitempty\"`\n\tZoneLocalMsgCount int64 `json:\"zone_local_msg_count,omitempty\"`\n\tRegionLocalMsgCount int64 `json:\"region_local_msg_count,omitempty\"`\n\tGlobalMsgCount int64 `json:\"global_msg_count,omitempty\"`\n\tClientCount int `json:\"client_count\"`\n\tSelected bool `json:\"-\"`\n\tNodeStats []*ChannelStats `json:\"nodes\"`\n\tClients []*ClientStats `json:\"clients\"`\n\tPaused bool `json:\"paused\"`\n\n\tE2eProcessingLatency *quantile.E2eProcessingLatencyAggregate `json:\"e2e_processing_latency\"`\n}\n\nfunc (c *ChannelStats) Add(a *ChannelStats) {\n\tc.Node = \"*\"\n\tc.Depth += a.Depth\n\tc.MemoryDepth += a.MemoryDepth\n\tc.BackendDepth += a.BackendDepth\n\tc.InFlightCount += a.InFlightCount\n\tc.DeferredCount += a.DeferredCount\n\tc.RequeueCount += a.RequeueCount\n\tc.TimeoutCount += a.TimeoutCount\n\tc.MessageCount += a.MessageCount\n\tc.DeliveryMsgCount += a.DeliveryMsgCount\n\tc.ZoneLocalMsgCount += a.ZoneLocalMsgCount\n\tc.RegionLocalMsgCount += a.RegionLocalMsgCount\n\tc.GlobalMsgCount += a.GlobalMsgCount\n\tc.ClientCount += a.ClientCount\n\tif a.Paused {\n\t\tc.Paused = a.Paused\n\t}\n\tc.NodeStats = append(c.NodeStats, a)\n\tsort.Sort(ChannelStatsByHost{c.NodeStats})\n\tif c.E2eProcessingLatency == nil {\n\t\tc.E2eProcessingLatency = &quantile.E2eProcessingLatencyAggregate{\n\t\t\tAddr: c.Node,\n\t\t\tTopic: c.TopicName,\n\t\t\tChannel: c.ChannelName,\n\t\t}\n\t}\n\tc.E2eProcessingLatency.Add(a.E2eProcessingLatency)\n\tc.Clients = append(c.Clients, a.Clients...)\n\tsort.Sort(ClientsByHost{c.Clients})\n}\n\ntype ClientStats struct {\n\tNode string `json:\"node\"`\n\tRemoteAddress string `json:\"remote_address\"`\n\tVersion string `json:\"version\"`\n\tClientID string `json:\"client_id\"`\n\tHostname string `json:\"hostname\"`\n\tUserAgent string `json:\"user_agent\"`\n\tConnectTs int64 `json:\"connect_ts\"`\n\tConnectedDuration time.Duration `json:\"connected\"`\n\tInFlightCount int `json:\"in_flight_count\"`\n\tReadyCount int `json:\"ready_count\"`\n\tFinishCount int64 `json:\"finish_count\"`\n\tRequeueCount int64 `json:\"requeue_count\"`\n\tMessageCount int64 `json:\"message_count\"`\n\tSampleRate int32 `json:\"sample_rate\"`\n\tDeflate bool `json:\"deflate\"`\n\tSnappy bool `json:\"snappy\"`\n\tAuthed bool `json:\"authed\"`\n\tAuthIdentity string `json:\"auth_identity\"`\n\tAuthIdentityURL string `json:\"auth_identity_url\"`\n\tNodeTopologyRegion string `json:\"node_topology_region,omitempty\"`\n\tNodeTopologyZone string `json:\"node_topology_zone,omitempty\"`\n\tTopologyRegion string `json:\"topology_region,omitempty\"`\n\tTopologyZone string `json:\"topology_zone,omitempty\"`\n\n\tTLS bool `json:\"tls\"`\n\tCipherSuite string `json:\"tls_cipher_suite\"`\n\tTLSVersion string `json:\"tls_version\"`\n\tTLSNegotiatedProtocol string `json:\"tls_negotiated_protocol\"`\n\tTLSNegotiatedProtocolIsMutual bool `json:\"tls_negotiated_protocol_is_mutual\"`\n}\n\n// UnmarshalJSON implements json.Unmarshaler and postprocesses ConnectedDuration\nfunc (s *ClientStats) UnmarshalJSON(b []byte) error {\n\ttype locaClientStats ClientStats // re-typed to prevent recursion from json.Unmarshal\n\tvar ss locaClientStats\n\tif err := json.Unmarshal(b, &ss); err != nil {\n\t\treturn err\n\t}\n\t*s = ClientStats(ss)\n\ts.ConnectedDuration = time.Now().Truncate(time.Second).Sub(time.Unix(s.ConnectTs, 0))\n\treturn nil\n}\n\nfunc (s *ClientStats) HasUserAgent() bool {\n\treturn s.UserAgent != \"\"\n}\n\nfunc (s *ClientStats) HasSampleRate() bool {\n\treturn s.SampleRate > 0\n}\n\ntype ChannelStatsList []*ChannelStats\n\nfunc (c ChannelStatsList) Len() int { return len(c) }\nfunc (c ChannelStatsList) Swap(i, j int) { c[i], c[j] = c[j], c[i] }\n\ntype ChannelStatsByHost struct {\n\tChannelStatsList\n}\n\nfunc (c ChannelStatsByHost) Less(i, j int) bool {\n\treturn c.ChannelStatsList[i].Hostname < c.ChannelStatsList[j].Hostname\n}\n\ntype ClientStatsList []*ClientStats\n\nfunc (c ClientStatsList) Len() int { return len(c) }\nfunc (c ClientStatsList) Swap(i, j int) { c[i], c[j] = c[j], c[i] }\n\ntype ClientsByHost struct {\n\tClientStatsList\n}\n\nfunc (c ClientsByHost) Less(i, j int) bool {\n\treturn c.ClientStatsList[i].Hostname < c.ClientStatsList[j].Hostname\n}\n\ntype ClientStatsByNodeTopology struct {\n\tClientStatsList\n}\n\nfunc (c ClientStatsByNodeTopology) Less(i, j int) bool {\n\t// if its the same node, sort by topology\n\tif c.ClientStatsList[i].Node == c.ClientStatsList[j].Node {\n\t\tregion := c.ClientStatsList[i].NodeTopologyRegion\n\t\tzone := c.ClientStatsList[i].NodeTopologyZone\n\n\t\tswitch {\n\t\tcase c.ClientStatsList[i].TopologyRegion == region && c.ClientStatsList[i].TopologyZone == zone:\n\t\t\treturn true\n\t\tcase c.ClientStatsList[j].TopologyRegion == region && c.ClientStatsList[j].TopologyZone == zone:\n\t\t\treturn false\n\t\tcase c.ClientStatsList[i].TopologyRegion == region:\n\t\t\treturn true\n\t\tcase c.ClientStatsList[j].TopologyRegion == region:\n\t\t\treturn false\n\t\tdefault:\n\t\t\tif c.ClientStatsList[i].TopologyRegion == c.ClientStatsList[j].TopologyRegion {\n\t\t\t\treturn c.ClientStatsList[i].TopologyZone < c.ClientStatsList[j].TopologyZone\n\t\t\t}\n\t\t\treturn c.ClientStatsList[i].TopologyRegion < c.ClientStatsList[j].TopologyRegion\n\t\t}\n\t}\n\treturn c.ClientStatsList[i].Node < c.ClientStatsList[j].Node\n}\n\ntype TopicStatsList []*TopicStats\n\nfunc (t TopicStatsList) Len() int { return len(t) }\nfunc (t TopicStatsList) Swap(i, j int) { t[i], t[j] = t[j], t[i] }\n\ntype TopicStatsByHost struct {\n\tTopicStatsList\n}\n\nfunc (c TopicStatsByHost) Less(i, j int) bool {\n\treturn c.TopicStatsList[i].Hostname < c.TopicStatsList[j].Hostname\n}\n\ntype Producers []*Producer\n\nfunc (t Producers) Len() int { return len(t) }\nfunc (t Producers) Swap(i, j int) { t[i], t[j] = t[j], t[i] }\n\nfunc (t Producers) HTTPAddrs() []string {\n\tvar addrs []string\n\tfor _, p := range t {\n\t\taddrs = append(addrs, p.HTTPAddress())\n\t}\n\treturn addrs\n}\n\nfunc (t Producers) Search(needle string) *Producer {\n\tfor _, producer := range t {\n\t\tif needle == producer.HTTPAddress() {\n\t\t\treturn producer\n\t\t}\n\t}\n\treturn nil\n}\n\ntype ProducersByHost struct {\n\tProducers\n}\n\nfunc (c ProducersByHost) Less(i, j int) bool {\n\treturn c.Producers[i].Hostname < c.Producers[j].Hostname\n}\n" }, { "path": "internal/dirlock/dirlock.go", "content": "//go:build !windows && !illumos\n// +build !windows,!illumos\n\npackage dirlock\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"syscall\"\n)\n\ntype DirLock struct {\n\tdir string\n\tf *os.File\n}\n\nfunc New(dir string) *DirLock {\n\treturn &DirLock{\n\t\tdir: dir,\n\t}\n}\n\nfunc (l *DirLock) Lock() error {\n\tf, err := os.Open(l.dir)\n\tif err != nil {\n\t\treturn err\n\t}\n\tl.f = f\n\terr = syscall.Flock(int(f.Fd()), syscall.LOCK_EX|syscall.LOCK_NB)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"cannot flock directory %s - %s (possibly in use by another instance of nsqd)\", l.dir, err)\n\t}\n\treturn nil\n}\n\nfunc (l *DirLock) Unlock() error {\n\tdefer l.f.Close()\n\treturn syscall.Flock(int(l.f.Fd()), syscall.LOCK_UN)\n}\n" }, { "path": "internal/dirlock/dirlock_illumos.go", "content": "//go:build illumos\n// +build illumos\n\npackage dirlock\n\ntype DirLock struct {\n\tdir string\n}\n\nfunc New(dir string) *DirLock {\n\treturn &DirLock{\n\t\tdir: dir,\n\t}\n}\n\nfunc (l *DirLock) Lock() error {\n\treturn nil\n}\n\nfunc (l *DirLock) Unlock() error {\n\treturn nil\n}\n" }, { "path": "internal/dirlock/dirlock_windows.go", "content": "//go:build windows\n// +build windows\n\npackage dirlock\n\ntype DirLock struct {\n\tdir string\n}\n\nfunc New(dir string) *DirLock {\n\treturn &DirLock{\n\t\tdir: dir,\n\t}\n}\n\nfunc (l *DirLock) Lock() error {\n\treturn nil\n}\n\nfunc (l *DirLock) Unlock() error {\n\treturn nil\n}\n" }, { "path": "internal/http_api/api_request.go", "content": "package http_api\n\nimport (\n\t\"bytes\"\n\t\"crypto/tls\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"strconv\"\n\t\"strings\"\n\t\"time\"\n)\n\n// A custom http.Transport with support for deadline timeouts\nfunc NewDeadlineTransport(connectTimeout time.Duration, requestTimeout time.Duration) *http.Transport {\n\t// arbitrary values copied from http.DefaultTransport\n\ttransport := &http.Transport{\n\t\tDialContext: (&net.Dialer{\n\t\t\tTimeout: connectTimeout,\n\t\t\tKeepAlive: 30 * time.Second,\n\t\t\tDualStack: true,\n\t\t}).DialContext,\n\t\tResponseHeaderTimeout: requestTimeout,\n\t\tMaxIdleConns: 100,\n\t\tIdleConnTimeout: 90 * time.Second,\n\t\tTLSHandshakeTimeout: 10 * time.Second,\n\t}\n\treturn transport\n}\n\ntype Client struct {\n\tc *http.Client\n}\n\nfunc NewClient(tlsConfig *tls.Config, connectTimeout time.Duration, requestTimeout time.Duration) *Client {\n\ttransport := NewDeadlineTransport(connectTimeout, requestTimeout)\n\ttransport.TLSClientConfig = tlsConfig\n\treturn &Client{\n\t\tc: &http.Client{\n\t\t\tTransport: transport,\n\t\t\tTimeout: requestTimeout,\n\t\t},\n\t}\n}\n\n// GETV1 is a helper function to perform a V1 HTTP request\n// and parse our NSQ daemon's expected response format, with deadlines.\nfunc (c *Client) GETV1(endpoint string, v interface{}) error {\nretry:\n\treq, err := http.NewRequest(\"GET\", endpoint, nil)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treq.Header.Add(\"Accept\", \"application/vnd.nsq; version=1.0\")\n\n\tresp, err := c.c.Do(req)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tbody, err := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\tif err != nil {\n\t\treturn err\n\t}\n\tif resp.StatusCode != 200 {\n\t\tif resp.StatusCode == 403 && !strings.HasPrefix(endpoint, \"https\") {\n\t\t\tendpoint, err = httpsEndpoint(endpoint, body)\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tgoto retry\n\t\t}\n\t\treturn fmt.Errorf(\"got response %s %q\", resp.Status, body)\n\t}\n\terr = json.Unmarshal(body, &v)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\n// PostV1 is a helper function to perform a V1 HTTP request\n// and parse our NSQ daemon's expected response format, with deadlines.\nfunc (c *Client) POSTV1(endpoint string, data url.Values, v interface{}) error {\nretry:\n\tvar reqBody io.Reader\n\tif data != nil {\n\t\tjs, err := json.Marshal(data)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"failed to marshal POST data to endpoint: %v\", endpoint)\n\t\t}\n\t\treqBody = bytes.NewBuffer(js)\n\t}\n\n\treq, err := http.NewRequest(\"POST\", endpoint, reqBody)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treq.Header.Add(\"Accept\", \"application/vnd.nsq; version=1.0\")\n\tif reqBody != nil {\n\t\treq.Header.Add(\"Content-Type\", \"application/json\")\n\t}\n\n\tresp, err := c.c.Do(req)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tbody, err := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\tif err != nil {\n\t\treturn err\n\t}\n\tif resp.StatusCode != 200 {\n\t\tif resp.StatusCode == 403 && !strings.HasPrefix(endpoint, \"https\") {\n\t\t\tendpoint, err = httpsEndpoint(endpoint, body)\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tgoto retry\n\t\t}\n\t\treturn fmt.Errorf(\"got response %s %q\", resp.Status, body)\n\t}\n\n\tif v != nil {\n\t\treturn json.Unmarshal(body, &v)\n\t}\n\n\treturn nil\n}\n\nfunc httpsEndpoint(endpoint string, body []byte) (string, error) {\n\tvar forbiddenResp struct {\n\t\tHTTPSPort int `json:\"https_port\"`\n\t}\n\terr := json.Unmarshal(body, &forbiddenResp)\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\n\tu, err := url.Parse(endpoint)\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\n\thost, _, err := net.SplitHostPort(u.Host)\n\tif err != nil {\n\t\treturn \"\", err\n\t}\n\n\tu.Scheme = \"https\"\n\tu.Host = net.JoinHostPort(host, strconv.Itoa(forbiddenResp.HTTPSPort))\n\treturn u.String(), nil\n}\n" }, { "path": "internal/http_api/api_response.go", "content": "package http_api\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"github.com/julienschmidt/httprouter\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype Decorator func(APIHandler) APIHandler\n\ntype APIHandler func(http.ResponseWriter, *http.Request, httprouter.Params) (interface{}, error)\n\ntype Err struct {\n\tCode int\n\tText string\n}\n\nfunc (e Err) Error() string {\n\treturn e.Text\n}\n\nfunc PlainText(f APIHandler) APIHandler {\n\treturn func(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t\tcode := 200\n\t\tdata, err := f(w, req, ps)\n\t\tif err != nil {\n\t\t\tcode = err.(Err).Code\n\t\t\tdata = err.Error()\n\t\t}\n\t\tswitch d := data.(type) {\n\t\tcase string:\n\t\t\tw.WriteHeader(code)\n\t\t\tio.WriteString(w, d)\n\t\tcase []byte:\n\t\t\tw.WriteHeader(code)\n\t\t\tw.Write(d)\n\t\tdefault:\n\t\t\tpanic(fmt.Sprintf(\"unknown response type %T\", data))\n\t\t}\n\t\treturn nil, nil\n\t}\n}\n\nfunc V1(f APIHandler) APIHandler {\n\treturn func(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t\tdata, err := f(w, req, ps)\n\t\tif err != nil {\n\t\t\tRespondV1(w, err.(Err).Code, err)\n\t\t\treturn nil, nil\n\t\t}\n\t\tRespondV1(w, 200, data)\n\t\treturn nil, nil\n\t}\n}\n\nfunc RespondV1(w http.ResponseWriter, code int, data interface{}) {\n\tvar response []byte\n\tvar err error\n\tvar isJSON bool\n\n\tif code == 200 {\n\t\tswitch data := data.(type) {\n\t\tcase string:\n\t\t\tresponse = []byte(data)\n\t\tcase []byte:\n\t\t\tresponse = data\n\t\tcase nil:\n\t\t\tresponse = []byte{}\n\t\tdefault:\n\t\t\tisJSON = true\n\t\t\tresponse, err = json.Marshal(data)\n\t\t\tif err != nil {\n\t\t\t\tcode = 500\n\t\t\t\tdata = err\n\t\t\t}\n\t\t}\n\t}\n\n\tif code != 200 {\n\t\tisJSON = true\n\t\tresponse, _ = json.Marshal(struct {\n\t\t\tMessage string `json:\"message\"`\n\t\t}{fmt.Sprintf(\"%s\", data)})\n\t}\n\n\tif isJSON {\n\t\tw.Header().Set(\"Content-Type\", \"application/json; charset=utf-8\")\n\t}\n\tw.Header().Set(\"X-NSQ-Content-Type\", \"nsq; version=1.0\")\n\tw.WriteHeader(code)\n\tw.Write(response)\n}\n\nfunc Decorate(f APIHandler, ds ...Decorator) httprouter.Handle {\n\tdecorated := f\n\tfor _, decorate := range ds {\n\t\tdecorated = decorate(decorated)\n\t}\n\treturn func(w http.ResponseWriter, req *http.Request, ps httprouter.Params) {\n\t\tdecorated(w, req, ps)\n\t}\n}\n\nfunc Log(logf lg.AppLogFunc) Decorator {\n\treturn func(f APIHandler) APIHandler {\n\t\treturn func(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t\t\tstart := time.Now()\n\t\t\tresponse, err := f(w, req, ps)\n\t\t\telapsed := time.Since(start)\n\t\t\tstatus := 200\n\t\t\tif e, ok := err.(Err); ok {\n\t\t\t\tstatus = e.Code\n\t\t\t}\n\t\t\tlogf(lg.INFO, \"%d %s %s (%s) %s\",\n\t\t\t\tstatus, req.Method, req.URL.RequestURI(), req.RemoteAddr, elapsed)\n\t\t\treturn response, err\n\t\t}\n\t}\n}\n\nfunc LogPanicHandler(logf lg.AppLogFunc) func(w http.ResponseWriter, req *http.Request, p interface{}) {\n\treturn func(w http.ResponseWriter, req *http.Request, p interface{}) {\n\t\tlogf(lg.ERROR, \"panic in HTTP handler - %s\", p)\n\t\tDecorate(func(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t\t\treturn nil, Err{500, \"INTERNAL_ERROR\"}\n\t\t}, Log(logf), V1)(w, req, nil)\n\t}\n}\n\nfunc LogNotFoundHandler(logf lg.AppLogFunc) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {\n\t\tDecorate(func(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t\t\treturn nil, Err{404, \"NOT_FOUND\"}\n\t\t}, Log(logf), V1)(w, req, nil)\n\t})\n}\n\nfunc LogMethodNotAllowedHandler(logf lg.AppLogFunc) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {\n\t\tDecorate(func(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t\t\treturn nil, Err{405, \"METHOD_NOT_ALLOWED\"}\n\t\t}, Log(logf), V1)(w, req, nil)\n\t})\n}\n" }, { "path": "internal/http_api/compress.go", "content": "// Copyright 2013 The Gorilla Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// license that can be found in the LICENSE file.\n\n// copied from https://github.com/gorilla/handlers/blob/master/compress.go\n\npackage http_api\n\nimport (\n\t\"compress/flate\"\n\t\"compress/gzip\"\n\t\"io\"\n\t\"net/http\"\n\t\"strings\"\n)\n\ntype compressResponseWriter struct {\n\tio.Writer\n\thttp.ResponseWriter\n\thttp.Hijacker\n}\n\nfunc (w *compressResponseWriter) Header() http.Header {\n\treturn w.ResponseWriter.Header()\n}\n\nfunc (w *compressResponseWriter) WriteHeader(c int) {\n\tw.ResponseWriter.Header().Del(\"Content-Length\")\n\tw.ResponseWriter.WriteHeader(c)\n}\n\nfunc (w *compressResponseWriter) Write(b []byte) (int, error) {\n\th := w.ResponseWriter.Header()\n\tif h.Get(\"Content-Type\") == \"\" {\n\t\th.Set(\"Content-Type\", http.DetectContentType(b))\n\t}\n\th.Del(\"Content-Length\")\n\treturn w.Writer.Write(b)\n}\n\n// CompressHandler gzip compresses HTTP responses for clients that support it\n// via the 'Accept-Encoding' header.\nfunc CompressHandler(h http.Handler) http.Handler {\n\treturn http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\tL:\n\t\tfor _, enc := range strings.Split(r.Header.Get(\"Accept-Encoding\"), \",\") {\n\t\t\tswitch strings.TrimSpace(enc) {\n\t\t\tcase \"gzip\":\n\t\t\t\tw.Header().Set(\"Content-Encoding\", \"gzip\")\n\t\t\t\tw.Header().Add(\"Vary\", \"Accept-Encoding\")\n\n\t\t\t\tgw := gzip.NewWriter(w)\n\t\t\t\tdefer gw.Close()\n\n\t\t\t\th, hok := w.(http.Hijacker)\n\t\t\t\tif !hok { /* w is not Hijacker... oh well... */\n\t\t\t\t\th = nil\n\t\t\t\t}\n\n\t\t\t\tw = &compressResponseWriter{\n\t\t\t\t\tWriter: gw,\n\t\t\t\t\tResponseWriter: w,\n\t\t\t\t\tHijacker: h,\n\t\t\t\t}\n\n\t\t\t\tbreak L\n\t\t\tcase \"deflate\":\n\t\t\t\tw.Header().Set(\"Content-Encoding\", \"deflate\")\n\t\t\t\tw.Header().Add(\"Vary\", \"Accept-Encoding\")\n\n\t\t\t\tfw, _ := flate.NewWriter(w, flate.DefaultCompression)\n\t\t\t\tdefer fw.Close()\n\n\t\t\t\th, hok := w.(http.Hijacker)\n\t\t\t\tif !hok { /* w is not Hijacker... oh well... */\n\t\t\t\t\th = nil\n\t\t\t\t}\n\n\t\t\t\tw = &compressResponseWriter{\n\t\t\t\t\tWriter: fw,\n\t\t\t\t\tResponseWriter: w,\n\t\t\t\t\tHijacker: h,\n\t\t\t\t}\n\n\t\t\t\tbreak L\n\t\t\t}\n\t\t}\n\n\t\th.ServeHTTP(w, r)\n\t})\n}\n" }, { "path": "internal/http_api/http_server.go", "content": "package http_api\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"log\"\n\t\"net\"\n\t\"net/http\"\n\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype logWriter struct {\n\tlogf lg.AppLogFunc\n}\n\nfunc (l logWriter) Write(p []byte) (int, error) {\n\tl.logf(lg.WARN, \"%s\", string(p))\n\treturn len(p), nil\n}\n\nfunc Serve(listener net.Listener, handler http.Handler, proto string, logf lg.AppLogFunc) error {\n\tlogf(lg.INFO, \"%s: listening on %s\", proto, listener.Addr())\n\n\tserver := &http.Server{\n\t\tHandler: handler,\n\t\tErrorLog: log.New(logWriter{logf}, \"\", 0),\n\t}\n\terr := server.Serve(listener)\n\t// theres no direct way to detect this error because it is not exposed\n\tif err != nil && !errors.Is(err, net.ErrClosed) {\n\t\treturn fmt.Errorf(\"http.Serve() error - %s\", err)\n\t}\n\n\tlogf(lg.INFO, \"%s: closing %s\", proto, listener.Addr())\n\n\treturn nil\n}\n" }, { "path": "internal/http_api/req_params.go", "content": "package http_api\n\nimport (\n\t\"errors\"\n\t\"io\"\n\t\"net/http\"\n\t\"net/url\"\n)\n\ntype ReqParams struct {\n\turl.Values\n\tBody []byte\n}\n\nfunc NewReqParams(req *http.Request) (*ReqParams, error) {\n\treqParams, err := url.ParseQuery(req.URL.RawQuery)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tdata, err := io.ReadAll(req.Body)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn &ReqParams{reqParams, data}, nil\n}\n\nfunc (r *ReqParams) Get(key string) (string, error) {\n\tv, ok := r.Values[key]\n\tif !ok {\n\t\treturn \"\", errors.New(\"key not in query params\")\n\t}\n\treturn v[0], nil\n}\n\nfunc (r *ReqParams) GetAll(key string) ([]string, error) {\n\tv, ok := r.Values[key]\n\tif !ok {\n\t\treturn nil, errors.New(\"key not in query params\")\n\t}\n\treturn v, nil\n}\n" }, { "path": "internal/http_api/topic_channel_args.go", "content": "package http_api\n\nimport (\n\t\"errors\"\n\n\t\"github.com/nsqio/nsq/internal/protocol\"\n)\n\ntype getter interface {\n\tGet(key string) (string, error)\n}\n\nfunc GetTopicChannelArgs(rp getter) (string, string, error) {\n\ttopicName, err := rp.Get(\"topic\")\n\tif err != nil {\n\t\treturn \"\", \"\", errors.New(\"MISSING_ARG_TOPIC\")\n\t}\n\n\tif !protocol.IsValidTopicName(topicName) {\n\t\treturn \"\", \"\", errors.New(\"INVALID_ARG_TOPIC\")\n\t}\n\n\tchannelName, err := rp.Get(\"channel\")\n\tif err != nil {\n\t\treturn \"\", \"\", errors.New(\"MISSING_ARG_CHANNEL\")\n\t}\n\n\tif !protocol.IsValidChannelName(channelName) {\n\t\treturn \"\", \"\", errors.New(\"INVALID_ARG_CHANNEL\")\n\t}\n\n\treturn topicName, channelName, nil\n}\n" }, { "path": "internal/lg/lg.go", "content": "// Package lg provides leveled logging\npackage lg\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"strings\"\n)\n\nconst (\n\tDEBUG = LogLevel(1)\n\tINFO = LogLevel(2)\n\tWARN = LogLevel(3)\n\tERROR = LogLevel(4)\n\tFATAL = LogLevel(5)\n)\n\ntype AppLogFunc func(lvl LogLevel, f string, args ...interface{})\n\ntype Logger interface {\n\tOutput(maxdepth int, s string) error\n}\n\ntype NilLogger struct{}\n\nfunc (l NilLogger) Output(maxdepth int, s string) error {\n\treturn nil\n}\n\ntype LogLevel int\n\nfunc (l *LogLevel) Get() interface{} { return *l }\n\nfunc (l *LogLevel) Set(s string) error {\n\tlvl, err := ParseLogLevel(s)\n\tif err != nil {\n\t\treturn err\n\t}\n\t*l = lvl\n\treturn nil\n}\n\nfunc (l *LogLevel) String() string {\n\tswitch *l {\n\tcase DEBUG:\n\t\treturn \"DEBUG\"\n\tcase INFO:\n\t\treturn \"INFO\"\n\tcase WARN:\n\t\treturn \"WARNING\"\n\tcase ERROR:\n\t\treturn \"ERROR\"\n\tcase FATAL:\n\t\treturn \"FATAL\"\n\t}\n\treturn \"invalid\"\n}\n\nfunc ParseLogLevel(levelstr string) (LogLevel, error) {\n\tswitch strings.ToLower(levelstr) {\n\tcase \"debug\":\n\t\treturn DEBUG, nil\n\tcase \"info\":\n\t\treturn INFO, nil\n\tcase \"warn\":\n\t\treturn WARN, nil\n\tcase \"error\":\n\t\treturn ERROR, nil\n\tcase \"fatal\":\n\t\treturn FATAL, nil\n\t}\n\treturn 0, fmt.Errorf(\"invalid log level '%s' (debug, info, warn, error, fatal)\", levelstr)\n}\n\nfunc Logf(logger Logger, cfgLevel LogLevel, msgLevel LogLevel, f string, args ...interface{}) {\n\tif cfgLevel > msgLevel {\n\t\treturn\n\t}\n\tlogger.Output(3, fmt.Sprintf(msgLevel.String()+\": \"+f, args...))\n}\n\nfunc LogFatal(prefix string, f string, args ...interface{}) {\n\tlogger := log.New(os.Stderr, prefix, log.Ldate|log.Ltime|log.Lmicroseconds)\n\tLogf(logger, FATAL, FATAL, f, args...)\n\tos.Exit(1)\n}\n" }, { "path": "internal/lg/lg_test.go", "content": "package lg\n\nimport (\n\t\"testing\"\n\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\ntype mockLogger struct {\n\tCount int\n}\n\nfunc (l *mockLogger) Output(maxdepth int, s string) error {\n\tl.Count++\n\treturn nil\n}\n\nfunc TestLogging(t *testing.T) {\n\tlogger := &mockLogger{}\n\n\t// Test only fatal get through\n\tlogger.Count = 0\n\tfor i := 1; i <= 5; i++ {\n\t\tLogf(logger, FATAL, LogLevel(i), \"Test\")\n\t}\n\ttest.Equal(t, 1, logger.Count)\n\n\t// Test only warnings or higher get through\n\tlogger.Count = 0\n\tfor i := 1; i <= 5; i++ {\n\t\tLogf(logger, WARN, LogLevel(i), \"Test\")\n\t}\n\ttest.Equal(t, 3, logger.Count)\n\n\t// Test everything gets through\n\tlogger.Count = 0\n\tfor i := 1; i <= 5; i++ {\n\t\tLogf(logger, DEBUG, LogLevel(i), \"Test\")\n\t}\n\ttest.Equal(t, 5, logger.Count)\n}\n" }, { "path": "internal/pqueue/pqueue.go", "content": "package pqueue\n\nimport (\n\t\"container/heap\"\n)\n\ntype Item struct {\n\tValue interface{}\n\tPriority int64\n\tIndex int\n}\n\n// this is a priority queue as implemented by a min heap\n// ie. the 0th element is the *lowest* value\ntype PriorityQueue []*Item\n\nfunc New(capacity int) PriorityQueue {\n\treturn make(PriorityQueue, 0, capacity)\n}\n\nfunc (pq PriorityQueue) Len() int {\n\treturn len(pq)\n}\n\nfunc (pq PriorityQueue) Less(i, j int) bool {\n\treturn pq[i].Priority < pq[j].Priority\n}\n\nfunc (pq PriorityQueue) Swap(i, j int) {\n\tpq[i], pq[j] = pq[j], pq[i]\n\tpq[i].Index = i\n\tpq[j].Index = j\n}\n\nfunc (pq *PriorityQueue) Push(x interface{}) {\n\tn := len(*pq)\n\tc := cap(*pq)\n\tif n+1 > c {\n\t\tnpq := make(PriorityQueue, n, c*2)\n\t\tcopy(npq, *pq)\n\t\t*pq = npq\n\t}\n\t*pq = (*pq)[0 : n+1]\n\titem := x.(*Item)\n\titem.Index = n\n\t(*pq)[n] = item\n}\n\nfunc (pq *PriorityQueue) Pop() interface{} {\n\tn := len(*pq)\n\tc := cap(*pq)\n\tif n < (c/2) && c > 25 {\n\t\tnpq := make(PriorityQueue, n, c/2)\n\t\tcopy(npq, *pq)\n\t\t*pq = npq\n\t}\n\titem := (*pq)[n-1]\n\titem.Index = -1\n\t*pq = (*pq)[0 : n-1]\n\treturn item\n}\n\nfunc (pq *PriorityQueue) PeekAndShift(max int64) (*Item, int64) {\n\tif pq.Len() == 0 {\n\t\treturn nil, 0\n\t}\n\n\titem := (*pq)[0]\n\tif item.Priority > max {\n\t\treturn nil, item.Priority - max\n\t}\n\theap.Remove(pq, 0)\n\n\treturn item, 0\n}\n" }, { "path": "internal/pqueue/pqueue_test.go", "content": "package pqueue\n\nimport (\n\t\"container/heap\"\n\t\"math/rand\"\n\t\"path/filepath\"\n\t\"reflect\"\n\t\"runtime\"\n\t\"sort\"\n\t\"testing\"\n)\n\nfunc equal(t *testing.T, act, exp interface{}) {\n\tif !reflect.DeepEqual(exp, act) {\n\t\t_, file, line, _ := runtime.Caller(1)\n\t\tt.Logf(\"\\033[31m%s:%d:\\n\\n\\texp: %#v\\n\\n\\tgot: %#v\\033[39m\\n\\n\",\n\t\t\tfilepath.Base(file), line, exp, act)\n\t\tt.FailNow()\n\t}\n}\n\nfunc TestPriorityQueue(t *testing.T) {\n\tc := 100\n\tpq := New(c)\n\n\tfor i := 0; i < c+1; i++ {\n\t\theap.Push(&pq, &Item{Value: i, Priority: int64(i)})\n\t}\n\tequal(t, pq.Len(), c+1)\n\tequal(t, cap(pq), c*2)\n\n\tfor i := 0; i < c+1; i++ {\n\t\titem := heap.Pop(&pq)\n\t\tequal(t, item.(*Item).Value.(int), i)\n\t}\n\tequal(t, cap(pq), c/4)\n}\n\nfunc TestUnsortedInsert(t *testing.T) {\n\tc := 100\n\tpq := New(c)\n\tints := make([]int, 0, c)\n\n\tfor i := 0; i < c; i++ {\n\t\tv := rand.Int()\n\t\tints = append(ints, v)\n\t\theap.Push(&pq, &Item{Value: i, Priority: int64(v)})\n\t}\n\tequal(t, pq.Len(), c)\n\tequal(t, cap(pq), c)\n\n\tsort.Ints(ints)\n\n\tfor i := 0; i < c; i++ {\n\t\titem, _ := pq.PeekAndShift(int64(ints[len(ints)-1]))\n\t\tequal(t, item.Priority, int64(ints[i]))\n\t}\n}\n\nfunc TestRemove(t *testing.T) {\n\tc := 100\n\tpq := New(c)\n\n\tfor i := 0; i < c; i++ {\n\t\tv := rand.Int()\n\t\theap.Push(&pq, &Item{Value: \"test\", Priority: int64(v)})\n\t}\n\n\tfor i := 0; i < 10; i++ {\n\t\theap.Remove(&pq, rand.Intn((c-1)-i))\n\t}\n\n\tlastPriority := heap.Pop(&pq).(*Item).Priority\n\tfor i := 0; i < (c - 10 - 1); i++ {\n\t\titem := heap.Pop(&pq)\n\t\tequal(t, lastPriority < item.(*Item).Priority, true)\n\t\tlastPriority = item.(*Item).Priority\n\t}\n}\n" }, { "path": "internal/protocol/byte_base10.go", "content": "package protocol\n\nimport (\n\t\"errors\"\n)\n\nvar errBase10 = errors.New(\"failed to convert to Base10\")\n\nfunc ByteToBase10(b []byte) (n uint64, err error) {\n\tbase := uint64(10)\n\n\tn = 0\n\tfor i := 0; i < len(b); i++ {\n\t\tvar v byte\n\t\td := b[i]\n\t\tswitch {\n\t\tcase '0' <= d && d <= '9':\n\t\t\tv = d - '0'\n\t\tdefault:\n\t\t\tn = 0\n\t\t\terr = errBase10\n\t\t\treturn\n\t\t}\n\t\tn *= base\n\t\tn += uint64(v)\n\t}\n\n\treturn n, err\n}\n" }, { "path": "internal/protocol/byte_base10_test.go", "content": "package protocol\n\nimport (\n\t\"testing\"\n)\n\nvar result uint64\n\nfunc BenchmarkByteToBase10Valid(b *testing.B) {\n\tbt := []byte{'3', '1', '4', '1', '5', '9', '2', '5'}\n\tvar n uint64\n\tfor i := 0; i < b.N; i++ {\n\t\tn, _ = ByteToBase10(bt)\n\t}\n\tresult = n\n}\n\nfunc BenchmarkByteToBase10Invalid(b *testing.B) {\n\tbt := []byte{'?', '1', '4', '1', '5', '9', '2', '5'}\n\tvar n uint64\n\tfor i := 0; i < b.N; i++ {\n\t\tn, _ = ByteToBase10(bt)\n\t}\n\tresult = n\n}\n" }, { "path": "internal/protocol/errors.go", "content": "package protocol\n\ntype ChildErr interface {\n\tParent() error\n}\n\n// ClientErr provides a way for NSQ daemons to log a human reabable\n// error string and return a machine readable string to the client.\n//\n// see docs/protocol.md for error codes by command\ntype ClientErr struct {\n\tParentErr error\n\tCode string\n\tDesc string\n}\n\n// Error returns the machine readable form\nfunc (e *ClientErr) Error() string {\n\treturn e.Code + \" \" + e.Desc\n}\n\n// Parent returns the parent error\nfunc (e *ClientErr) Parent() error {\n\treturn e.ParentErr\n}\n\n// NewClientErr creates a ClientErr with the supplied human and machine readable strings\nfunc NewClientErr(parent error, code string, description string) *ClientErr {\n\treturn &ClientErr{parent, code, description}\n}\n\ntype FatalClientErr struct {\n\tParentErr error\n\tCode string\n\tDesc string\n}\n\n// Error returns the machine readable form\nfunc (e *FatalClientErr) Error() string {\n\treturn e.Code + \" \" + e.Desc\n}\n\n// Parent returns the parent error\nfunc (e *FatalClientErr) Parent() error {\n\treturn e.ParentErr\n}\n\n// NewFatalClientErr creates a ClientErr with the supplied human and machine readable strings\nfunc NewFatalClientErr(parent error, code string, description string) *FatalClientErr {\n\treturn &FatalClientErr{parent, code, description}\n}\n" }, { "path": "internal/protocol/names.go", "content": "package protocol\n\nimport (\n\t\"regexp\"\n)\n\nvar validTopicChannelNameRegex = regexp.MustCompile(`^[.a-zA-Z0-9_-]+(#ephemeral)?$`)\n\n// IsValidTopicName checks a topic name for correctness\nfunc IsValidTopicName(name string) bool {\n\treturn isValidName(name)\n}\n\n// IsValidChannelName checks a channel name for correctness\nfunc IsValidChannelName(name string) bool {\n\treturn isValidName(name)\n}\n\nfunc isValidName(name string) bool {\n\tif len(name) > 64 || len(name) < 1 {\n\t\treturn false\n\t}\n\treturn validTopicChannelNameRegex.MatchString(name)\n}\n" }, { "path": "internal/protocol/protocol.go", "content": "package protocol\n\nimport (\n\t\"encoding/binary\"\n\t\"io\"\n\t\"net\"\n)\n\ntype Client interface {\n\tClose() error\n}\n\n// Protocol describes the basic behavior of any protocol in the system\ntype Protocol interface {\n\tNewClient(net.Conn) Client\n\tIOLoop(Client) error\n}\n\n// SendResponse is a server side utility function to prefix data with a length header\n// and write to the supplied Writer\nfunc SendResponse(w io.Writer, data []byte) (int, error) {\n\terr := binary.Write(w, binary.BigEndian, int32(len(data)))\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\n\tn, err := w.Write(data)\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\n\treturn (n + 4), nil\n}\n\n// SendFramedResponse is a server side utility function to prefix data with a length header\n// and frame header and write to the supplied Writer\nfunc SendFramedResponse(w io.Writer, frameType int32, data []byte) (int, error) {\n\tbeBuf := make([]byte, 4)\n\tsize := uint32(len(data)) + 4\n\n\tbinary.BigEndian.PutUint32(beBuf, size)\n\tn, err := w.Write(beBuf)\n\tif err != nil {\n\t\treturn n, err\n\t}\n\n\tbinary.BigEndian.PutUint32(beBuf, uint32(frameType))\n\tn, err = w.Write(beBuf)\n\tif err != nil {\n\t\treturn n + 4, err\n\t}\n\n\tn, err = w.Write(data)\n\treturn n + 8, err\n}\n" }, { "path": "internal/protocol/tcp_server.go", "content": "package protocol\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"net\"\n\t\"runtime\"\n\t\"sync\"\n\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype TCPHandler interface {\n\tHandle(net.Conn)\n}\n\nfunc TCPServer(listener net.Listener, handler TCPHandler, logf lg.AppLogFunc) error {\n\tlogf(lg.INFO, \"TCP: listening on %s\", listener.Addr())\n\n\tvar wg sync.WaitGroup\n\n\tfor {\n\t\tclientConn, err := listener.Accept()\n\t\tif err != nil {\n\t\t\t// net.Error.Temporary() is deprecated, but is valid for accept\n\t\t\t// this is a hack to avoid a staticcheck error\n\t\t\tif te, ok := err.(interface{ Temporary() bool }); ok && te.Temporary() {\n\t\t\t\tlogf(lg.WARN, \"temporary Accept() failure - %s\", err)\n\t\t\t\truntime.Gosched()\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\t// theres no direct way to detect this error because it is not exposed\n\t\t\tif !errors.Is(err, net.ErrClosed) {\n\t\t\t\treturn fmt.Errorf(\"listener.Accept() error - %s\", err)\n\t\t\t}\n\t\t\tbreak\n\t\t}\n\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\thandler.Handle(clientConn)\n\t\t\twg.Done()\n\t\t}()\n\t}\n\n\t// wait to return until all handler goroutines complete\n\twg.Wait()\n\n\tlogf(lg.INFO, \"TCP: closing %s\", listener.Addr())\n\n\treturn nil\n}\n" }, { "path": "internal/quantile/aggregate.go", "content": "package quantile\n\nimport (\n\t\"encoding/json\"\n\t\"math\"\n\t\"sort\"\n)\n\ntype E2eProcessingLatencyAggregate struct {\n\tCount int `json:\"count\"`\n\tPercentiles []map[string]float64 `json:\"percentiles\"`\n\tTopic string `json:\"topic\"`\n\tChannel string `json:\"channel\"`\n\tAddr string `json:\"host\"`\n}\n\nfunc (e *E2eProcessingLatencyAggregate) UnmarshalJSON(b []byte) error {\n\tvar resp struct {\n\t\tCount int `json:\"count\"`\n\t\tPercentiles []map[string]float64 `json:\"percentiles\"`\n\t\tTopic string `json:\"topic\"`\n\t\tChannel string `json:\"channel\"`\n\t\tAddr string `json:\"host\"`\n\t}\n\terr := json.Unmarshal(b, &resp)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tfor _, p := range resp.Percentiles {\n\t\tp[\"min\"] = p[\"value\"]\n\t\tp[\"max\"] = p[\"value\"]\n\t\tp[\"average\"] = p[\"value\"]\n\t\tp[\"count\"] = float64(resp.Count)\n\t}\n\n\te.Count = resp.Count\n\te.Percentiles = resp.Percentiles\n\te.Topic = resp.Topic\n\te.Channel = resp.Channel\n\te.Addr = resp.Addr\n\n\treturn nil\n}\n\nfunc (e *E2eProcessingLatencyAggregate) Len() int { return len(e.Percentiles) }\nfunc (e *E2eProcessingLatencyAggregate) Swap(i, j int) {\n\te.Percentiles[i], e.Percentiles[j] = e.Percentiles[j], e.Percentiles[i]\n}\nfunc (e *E2eProcessingLatencyAggregate) Less(i, j int) bool {\n\treturn e.Percentiles[i][\"percentile\"] > e.Percentiles[j][\"percentile\"]\n}\n\n// Add merges e2 into e by averaging the percentiles\nfunc (e *E2eProcessingLatencyAggregate) Add(e2 *E2eProcessingLatencyAggregate) {\n\te.Addr = \"*\"\n\tp := e.Percentiles\n\te.Count += e2.Count\n\tfor _, value := range e2.Percentiles {\n\t\ti := -1\n\t\tfor j, v := range p {\n\t\t\tif value[\"quantile\"] == v[\"quantile\"] {\n\t\t\t\ti = j\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t\tif i == -1 {\n\t\t\ti = len(p)\n\t\t\te.Percentiles = append(p, make(map[string]float64))\n\t\t\tp = e.Percentiles\n\t\t\tp[i][\"quantile\"] = value[\"quantile\"]\n\t\t}\n\t\tp[i][\"max\"] = math.Max(value[\"max\"], p[i][\"max\"])\n\t\tp[i][\"min\"] = math.Min(value[\"max\"], p[i][\"max\"])\n\t\tp[i][\"count\"] += value[\"count\"]\n\t\tif p[i][\"count\"] == 0 {\n\t\t\tp[i][\"average\"] = 0\n\t\t\tcontinue\n\t\t}\n\t\tdelta := value[\"average\"] - p[i][\"average\"]\n\t\tR := delta * value[\"count\"] / p[i][\"count\"]\n\t\tp[i][\"average\"] = p[i][\"average\"] + R\n\t}\n\tsort.Sort(e)\n}\n" }, { "path": "internal/quantile/quantile.go", "content": "package quantile\n\nimport (\n\t\"strings\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/bmizerany/perks/quantile\"\n\t\"github.com/nsqio/nsq/internal/stringy\"\n)\n\ntype Result struct {\n\tCount int `json:\"count\"`\n\tPercentiles []map[string]float64 `json:\"percentiles\"`\n}\n\nfunc (r *Result) String() string {\n\tvar s []string\n\tfor _, item := range r.Percentiles {\n\t\ts = append(s, stringy.NanoSecondToHuman(item[\"value\"]))\n\t}\n\treturn strings.Join(s, \", \")\n}\n\ntype Quantile struct {\n\tsync.Mutex\n\tstreams [2]quantile.Stream\n\tcurrentIndex uint8\n\tlastMoveWindow time.Time\n\tcurrentStream *quantile.Stream\n\n\tPercentiles []float64\n\tMoveWindowTime time.Duration\n}\n\nfunc New(WindowTime time.Duration, Percentiles []float64) *Quantile {\n\tq := Quantile{\n\t\tcurrentIndex: 0,\n\t\tlastMoveWindow: time.Now(),\n\t\tMoveWindowTime: WindowTime / 2,\n\t\tPercentiles: Percentiles,\n\t}\n\tfor i := 0; i < 2; i++ {\n\t\tq.streams[i] = *quantile.NewTargeted(Percentiles...)\n\t}\n\tq.currentStream = &q.streams[0]\n\treturn &q\n}\n\nfunc (q *Quantile) Result() *Result {\n\tif q == nil {\n\t\treturn &Result{}\n\t}\n\tqueryHandler := q.QueryHandler()\n\tresult := Result{\n\t\tCount: queryHandler.Count(),\n\t\tPercentiles: make([]map[string]float64, len(q.Percentiles)),\n\t}\n\tfor i, p := range q.Percentiles {\n\t\tvalue := queryHandler.Query(p)\n\t\tresult.Percentiles[i] = map[string]float64{\"quantile\": p, \"value\": value}\n\t}\n\treturn &result\n}\n\nfunc (q *Quantile) Insert(msgStartTime int64) {\n\tq.Lock()\n\n\tnow := time.Now()\n\tfor q.IsDataStale(now) {\n\t\tq.moveWindow()\n\t}\n\n\tq.currentStream.Insert(float64(now.UnixNano() - msgStartTime))\n\tq.Unlock()\n}\n\nfunc (q *Quantile) QueryHandler() *quantile.Stream {\n\tq.Lock()\n\tnow := time.Now()\n\tfor q.IsDataStale(now) {\n\t\tq.moveWindow()\n\t}\n\n\tmerged := quantile.NewTargeted(q.Percentiles...)\n\tmerged.Merge(q.streams[0].Samples())\n\tmerged.Merge(q.streams[1].Samples())\n\tq.Unlock()\n\treturn merged\n}\n\nfunc (q *Quantile) IsDataStale(now time.Time) bool {\n\treturn now.After(q.lastMoveWindow.Add(q.MoveWindowTime))\n}\n\nfunc (q *Quantile) Merge(them *Quantile) {\n\tq.Lock()\n\tthem.Lock()\n\tiUs := q.currentIndex\n\tiThem := them.currentIndex\n\n\tq.streams[iUs].Merge(them.streams[iThem].Samples())\n\n\tiUs ^= 0x1\n\tiThem ^= 0x1\n\tq.streams[iUs].Merge(them.streams[iThem].Samples())\n\n\tif q.lastMoveWindow.Before(them.lastMoveWindow) {\n\t\tq.lastMoveWindow = them.lastMoveWindow\n\t}\n\tq.Unlock()\n\tthem.Unlock()\n}\n\nfunc (q *Quantile) moveWindow() {\n\tq.currentIndex ^= 0x1\n\tq.currentStream = &q.streams[q.currentIndex]\n\tq.lastMoveWindow = q.lastMoveWindow.Add(q.MoveWindowTime)\n\tq.currentStream.Reset()\n}\n" }, { "path": "internal/statsd/client.go", "content": "package statsd\n\nimport (\n\t\"fmt\"\n\t\"io\"\n)\n\ntype Client struct {\n\tw io.Writer\n\tprefix string\n}\n\nfunc NewClient(w io.Writer, prefix string) *Client {\n\treturn &Client{\n\t\tw: w,\n\t\tprefix: prefix,\n\t}\n}\n\nfunc (c *Client) Incr(stat string, count int64) error {\n\treturn c.send(stat, \"%d|c\", count)\n}\n\nfunc (c *Client) Decr(stat string, count int64) error {\n\treturn c.send(stat, \"%d|c\", -count)\n}\n\nfunc (c *Client) Timing(stat string, delta int64) error {\n\treturn c.send(stat, \"%d|ms\", delta)\n}\n\nfunc (c *Client) Gauge(stat string, value int64) error {\n\treturn c.send(stat, \"%d|g\", value)\n}\n\nfunc (c *Client) send(stat string, format string, value int64) error {\n\tformat = fmt.Sprintf(\"%s%s:%s\\n\", c.prefix, stat, format)\n\t_, err := fmt.Fprintf(c.w, format, value)\n\treturn err\n}\n" }, { "path": "internal/statsd/host.go", "content": "package statsd\n\nimport (\n\t\"strings\"\n)\n\nfunc HostKey(h string) string {\n\treturn strings.Replace(strings.Replace(h, \".\", \"_\", -1), \":\", \"_\", -1)\n}\n" }, { "path": "internal/stringy/slice.go", "content": "package stringy\n\nfunc Add(s []string, a string) []string {\n\tfor _, existing := range s {\n\t\tif a == existing {\n\t\t\treturn s\n\t\t}\n\t}\n\treturn append(s, a)\n\n}\n\nfunc Union(s []string, a []string) []string {\n\tfor _, entry := range a {\n\t\tfound := false\n\t\tfor _, existing := range s {\n\t\t\tif entry == existing {\n\t\t\t\tfound = true\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t\tif !found {\n\t\t\ts = append(s, entry)\n\t\t}\n\t}\n\treturn s\n}\n\nfunc Uniq(s []string) (r []string) {\nouterLoop:\n\tfor _, entry := range s {\n\t\tfor _, existing := range r {\n\t\t\tif existing == entry {\n\t\t\t\tcontinue outerLoop\n\t\t\t}\n\t\t}\n\t\tr = append(r, entry)\n\t}\n\treturn\n}\n" }, { "path": "internal/stringy/slice_test.go", "content": "package stringy_test\n\nimport (\n\t\"testing\"\n\n\t\"github.com/nsqio/nsq/internal/stringy\"\n)\n\nfunc BenchmarkUniq(b *testing.B) {\n\tvalues := []string{\"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"a\", \"b\"}\n\tfor i := 0; i < b.N; i++ {\n\t\tvalues = stringy.Uniq(values)\n\t\tif len(values) != 2 {\n\t\t\tb.Fatal(\"values len is incorrect\")\n\t\t}\n\t}\n}\n\nfunc TestUniq(t *testing.T) {\n\tvalues := []string{\"a\", \"a\", \"a\", \"b\", \"b\", \"b\", \"c\", \"c\", \"c\"}\n\tvalues = stringy.Uniq(values)\n\tif len(values) != 3 {\n\t\tt.Fatal(\"values len is incorrect\")\n\t}\n}\n" }, { "path": "internal/stringy/template.go", "content": "package stringy\n\nimport (\n\t\"fmt\"\n)\n\nfunc NanoSecondToHuman(v float64) string {\n\tvar suffix string\n\tswitch {\n\tcase v > 1000000000:\n\t\tv /= 1000000000\n\t\tsuffix = \"s\"\n\tcase v > 1000000:\n\t\tv /= 1000000\n\t\tsuffix = \"ms\"\n\tcase v > 1000:\n\t\tv /= 1000\n\t\tsuffix = \"us\"\n\tdefault:\n\t\tsuffix = \"ns\"\n\t}\n\treturn fmt.Sprintf(\"%0.1f%s\", v, suffix)\n}\n" }, { "path": "internal/test/assertions.go", "content": "package test\n\nimport (\n\t\"path/filepath\"\n\t\"reflect\"\n\t\"runtime\"\n\t\"testing\"\n)\n\nfunc Equal(t *testing.T, expected, actual interface{}) {\n\tif !reflect.DeepEqual(expected, actual) {\n\t\t_, file, line, _ := runtime.Caller(1)\n\t\tt.Logf(\"\\033[31m%s:%d:\\n\\n\\t %#v (expected)\\n\\n\\t!= %#v (actual)\\033[39m\\n\\n\",\n\t\t\tfilepath.Base(file), line, expected, actual)\n\t\tt.FailNow()\n\t}\n}\n\nfunc NotEqual(t *testing.T, expected, actual interface{}) {\n\tif reflect.DeepEqual(expected, actual) {\n\t\t_, file, line, _ := runtime.Caller(1)\n\t\tt.Logf(\"\\033[31m%s:%d:\\n\\n\\tnexp: %#v\\n\\n\\tgot: %#v\\033[39m\\n\\n\",\n\t\t\tfilepath.Base(file), line, expected, actual)\n\t\tt.FailNow()\n\t}\n}\n\nfunc Nil(t *testing.T, object interface{}) {\n\tif !isNil(object) {\n\t\t_, file, line, _ := runtime.Caller(1)\n\t\tt.Logf(\"\\033[31m%s:%d:\\n\\n\\t (expected)\\n\\n\\t!= %#v (actual)\\033[39m\\n\\n\",\n\t\t\tfilepath.Base(file), line, object)\n\t\tt.FailNow()\n\t}\n}\n\nfunc NotNil(t *testing.T, object interface{}) {\n\tif isNil(object) {\n\t\t_, file, line, _ := runtime.Caller(1)\n\t\tt.Logf(\"\\033[31m%s:%d:\\n\\n\\tExpected value not to be \\033[39m\\n\\n\",\n\t\t\tfilepath.Base(file), line)\n\t\tt.FailNow()\n\t}\n}\n\nfunc isNil(object interface{}) bool {\n\tif object == nil {\n\t\treturn true\n\t}\n\n\tvalue := reflect.ValueOf(object)\n\tkind := value.Kind()\n\tif kind >= reflect.Chan && kind <= reflect.Slice && value.IsNil() {\n\t\treturn true\n\t}\n\n\treturn false\n}\n" }, { "path": "internal/test/fakes.go", "content": "package test\n\nimport (\n\t\"net\"\n\t\"time\"\n)\n\ntype FakeNetConn struct {\n\tReadFunc func([]byte) (int, error)\n\tWriteFunc func([]byte) (int, error)\n\tCloseFunc func() error\n\tLocalAddrFunc func() net.Addr\n\tRemoteAddrFunc func() net.Addr\n\tSetDeadlineFunc func(time.Time) error\n\tSetReadDeadlineFunc func(time.Time) error\n\tSetWriteDeadlineFunc func(time.Time) error\n}\n\nfunc (f FakeNetConn) Read(b []byte) (int, error) { return f.ReadFunc(b) }\nfunc (f FakeNetConn) Write(b []byte) (int, error) { return f.WriteFunc(b) }\nfunc (f FakeNetConn) Close() error { return f.CloseFunc() }\nfunc (f FakeNetConn) LocalAddr() net.Addr { return f.LocalAddrFunc() }\nfunc (f FakeNetConn) RemoteAddr() net.Addr { return f.RemoteAddrFunc() }\nfunc (f FakeNetConn) SetDeadline(t time.Time) error { return f.SetDeadlineFunc(t) }\nfunc (f FakeNetConn) SetReadDeadline(t time.Time) error { return f.SetReadDeadlineFunc(t) }\nfunc (f FakeNetConn) SetWriteDeadline(t time.Time) error { return f.SetWriteDeadlineFunc(t) }\n\ntype fakeNetAddr struct{}\n\nfunc (fakeNetAddr) Network() string { return \"\" }\nfunc (fakeNetAddr) String() string { return \"\" }\n\nfunc NewFakeNetConn() FakeNetConn {\n\tnetAddr := fakeNetAddr{}\n\treturn FakeNetConn{\n\t\tReadFunc: func(b []byte) (int, error) { return 0, nil },\n\t\tWriteFunc: func(b []byte) (int, error) { return len(b), nil },\n\t\tCloseFunc: func() error { return nil },\n\t\tLocalAddrFunc: func() net.Addr { return netAddr },\n\t\tRemoteAddrFunc: func() net.Addr { return netAddr },\n\t\tSetDeadlineFunc: func(time.Time) error { return nil },\n\t\tSetWriteDeadlineFunc: func(time.Time) error { return nil },\n\t\tSetReadDeadlineFunc: func(time.Time) error { return nil },\n\t}\n}\n" }, { "path": "internal/test/logger.go", "content": "package test\n\ntype Logger interface {\n\tOutput(maxdepth int, s string) error\n}\n\ntype tbLog interface {\n\tLog(...interface{})\n}\n\ntype testLogger struct {\n\ttbLog\n}\n\nfunc (tl *testLogger) Output(maxdepth int, s string) error {\n\ttl.Log(s)\n\treturn nil\n}\n\nfunc NewTestLogger(tbl tbLog) Logger {\n\treturn &testLogger{tbl}\n}\n" }, { "path": "internal/util/rand.go", "content": "package util\n\nimport (\n\t\"math/rand\"\n)\n\nfunc UniqRands(quantity int, maxval int) []int {\n\tif maxval < quantity {\n\t\tquantity = maxval\n\t}\n\n\tintSlice := make([]int, maxval)\n\tfor i := 0; i < maxval; i++ {\n\t\tintSlice[i] = i\n\t}\n\n\tfor i := 0; i < quantity; i++ {\n\t\tj := rand.Int()%maxval + i\n\t\t// swap\n\t\tintSlice[i], intSlice[j] = intSlice[j], intSlice[i]\n\t\tmaxval--\n\n\t}\n\treturn intSlice[0:quantity]\n}\n" }, { "path": "internal/util/unix_socket.go", "content": "package util\n\nimport (\n\t\"net\"\n)\n\nfunc TypeOfAddr(addr string) string {\n\tif _, _, err := net.SplitHostPort(addr); err == nil {\n\t\treturn \"tcp\"\n\t}\n\treturn \"unix\"\n}\n" }, { "path": "internal/util/util_test.go", "content": "package util\n\nimport (\n\t\"testing\"\n\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\nfunc BenchmarkUniqRands5of5(b *testing.B) {\n\tfor i := 0; i < b.N; i++ {\n\t\tUniqRands(5, 5)\n\t}\n}\nfunc BenchmarkUniqRands20of20(b *testing.B) {\n\tfor i := 0; i < b.N; i++ {\n\t\tUniqRands(20, 20)\n\t}\n}\n\nfunc BenchmarkUniqRands20of50(b *testing.B) {\n\tfor i := 0; i < b.N; i++ {\n\t\tUniqRands(20, 50)\n\t}\n}\n\nfunc TestUniqRands(t *testing.T) {\n\tvar x []int\n\tx = UniqRands(3, 10)\n\ttest.Equal(t, 3, len(x))\n\n\tx = UniqRands(10, 5)\n\ttest.Equal(t, 5, len(x))\n\n\tx = UniqRands(10, 20)\n\ttest.Equal(t, 10, len(x))\n}\n\nfunc TestTypeOfAddr(t *testing.T) {\n\tvar x string\n\tx = TypeOfAddr(\"127.0.0.1:80\")\n\ttest.Equal(t, \"tcp\", x)\n\n\tx = TypeOfAddr(\"test:80\")\n\ttest.Equal(t, \"tcp\", x)\n\n\tx = TypeOfAddr(\"/var/run/nsqd.sock\")\n\ttest.Equal(t, \"unix\", x)\n\n\tx = TypeOfAddr(\"[::1%lo0]:80\")\n\ttest.Equal(t, \"tcp\", x)\n}\n" }, { "path": "internal/util/wait_group_wrapper.go", "content": "package util\n\nimport (\n\t\"sync\"\n)\n\ntype WaitGroupWrapper struct {\n\tsync.WaitGroup\n}\n\nfunc (w *WaitGroupWrapper) Wrap(cb func()) {\n\tw.Add(1)\n\tgo func() {\n\t\tcb()\n\t\tw.Done()\n\t}()\n}\n" }, { "path": "internal/version/binary.go", "content": "package version\n\nimport (\n\t\"fmt\"\n\t\"runtime\"\n)\n\nconst Binary = \"1.3.0\"\n\nfunc String(app string) string {\n\treturn fmt.Sprintf(\"%s v%s (built w/%s)\", app, Binary, runtime.Version())\n}\n" }, { "path": "internal/writers/boundary_buffered_writer.go", "content": "package writers\n\nimport (\n\t\"bufio\"\n\t\"io\"\n)\n\ntype BoundaryBufferedWriter struct {\n\tbw *bufio.Writer\n}\n\nfunc NewBoundaryBufferedWriter(w io.Writer, size int) *BoundaryBufferedWriter {\n\treturn &BoundaryBufferedWriter{\n\t\tbw: bufio.NewWriterSize(w, size),\n\t}\n}\n\nfunc (b *BoundaryBufferedWriter) Write(p []byte) (int, error) {\n\tif len(p) > b.bw.Available() {\n\t\terr := b.bw.Flush()\n\t\tif err != nil {\n\t\t\treturn 0, err\n\t\t}\n\t}\n\treturn b.bw.Write(p)\n}\n\nfunc (b *BoundaryBufferedWriter) Flush() error {\n\treturn b.bw.Flush()\n}\n" }, { "path": "internal/writers/spread_writer.go", "content": "package writers\n\nimport (\n\t\"io\"\n\t\"time\"\n)\n\ntype SpreadWriter struct {\n\tw io.Writer\n\tinterval time.Duration\n\tbuf [][]byte\n\texitCh chan int\n}\n\nfunc NewSpreadWriter(w io.Writer, interval time.Duration, exitCh chan int) *SpreadWriter {\n\treturn &SpreadWriter{\n\t\tw: w,\n\t\tinterval: interval,\n\t\tbuf: make([][]byte, 0),\n\t\texitCh: exitCh,\n\t}\n}\n\nfunc (s *SpreadWriter) Write(p []byte) (int, error) {\n\tb := make([]byte, len(p))\n\tcopy(b, p)\n\ts.buf = append(s.buf, b)\n\treturn len(p), nil\n}\n\nfunc (s *SpreadWriter) Flush() {\n\tif len(s.buf) == 0 {\n\t\t// nothing to write, just wait\n\t\tselect {\n\t\tcase <-time.After(s.interval):\n\t\tcase <-s.exitCh:\n\t\t}\n\t\treturn\n\t}\n\tsleep := s.interval / time.Duration(len(s.buf))\n\tticker := time.NewTicker(sleep)\n\tfor _, b := range s.buf {\n\t\ts.w.Write(b)\n\t\tselect {\n\t\tcase <-ticker.C:\n\t\tcase <-s.exitCh: // skip sleeps finish writes\n\t\t}\n\t}\n\tticker.Stop()\n\ts.buf = s.buf[:0]\n}\n" }, { "path": "nsqadmin/.eslintrc", "content": "{\n \"env\": {\n \"browser\": true,\n \"es2020\": true\n },\n // Rule docs: http://eslint.org/docs/rules/\n \"rules\": {\n \"block-scoped-var\": [1],\n \"brace-style\": [1, \"1tbs\", {\"allowSingleLine\": true}],\n \"camelcase\": [1],\n \"comma-spacing\": [1],\n \"comma-style\": [1],\n \"computed-property-spacing\": [1, \"never\"],\n \"consistent-return\": [1],\n \"consistent-this\": [1, \"self\"],\n \"curly\": [2],\n \"dot-notation\": [0],\n \"eol-last\": [1],\n \"eqeqeq\": [1],\n \"indent\": [1, 4],\n \"key-spacing\": [1],\n \"max-len\": [1, 100],\n \"max-nested-callbacks\": [2, 3], // ????\n \"new-cap\": [1],\n \"new-parens\": [1],\n \"no-caller\": [2],\n \"no-console\": [0],\n \"no-eval\": [2],\n \"no-extend-native\": [2],\n \"no-extra-bind\": [1],\n \"no-floating-decimal\": [1],\n \"no-iterator\": [1],\n \"no-lone-blocks\": [1],\n \"no-lonely-if\": [1],\n \"no-mixed-requires\": [0],\n \"no-mixed-spaces-and-tabs\": [1],\n \"no-multi-spaces\": [1],\n \"no-multi-str\": [1],\n \"no-multiple-empty-lines\": [2, {\"max\": 2}],\n \"no-native-reassign\": [1],\n \"no-new\": [0],\n \"no-redeclare\": [1],\n \"no-shadow\": [1],\n \"no-spaced-func\": [1],\n \"no-throw-literal\": [1],\n \"no-trailing-spaces\": [1],\n \"no-undef\": [1],\n \"no-underscore-dangle\": [0],\n \"no-unneeded-ternary\": [1],\n \"no-unused-vars\": [1],\n \"no-use-before-define\": [1, \"nofunc\"],\n \"no-with\": [2],\n \"one-var\": [1, \"never\"],\n \"quotes\": [1, \"single\"],\n \"radix\": [1],\n \"semi\": [1, \"always\"],\n \"semi-spacing\": [1],\n \"keyword-spacing\": [1, {\"before\": true, \"after\": true}],\n \"space-before-blocks\": [1, \"always\"],\n \"space-before-function-paren\": [1, {\"anonymous\": \"never\", \"named\": \"never\", \"asyncArrow\": \"always\"}],\n \"space-in-parens\": [1, \"never\"],\n \"space-infix-ops\": [1],\n \"space-unary-ops\": [1],\n \"strict\": [0],\n \"wrap-iife\": [1]\n },\n \"globals\": {\n \"BASE_PATH\": true,\n \"GRAPHITE_URL\": true,\n \"GRAPH_ENABLED\": true,\n \"IS_ADMIN\": true,\n \"NSQLOOKUPD\": true,\n \"STATSD_COUNTER_FORMAT\": true,\n \"STATSD_GAUGE_FORMAT\": true,\n \"STATSD_INTERVAL\": true,\n \"STATSD_PREFIX\": true,\n \"USER_AGENT\": true,\n \"VERSION\": true,\n \"module\": true,\n \"require\": true\n }\n}\n" }, { "path": "nsqadmin/README.md", "content": "## nsqadmin\n\n`nsqadmin` is a Web UI to view aggregated cluster stats in realtime and perform various\nadministrative tasks.\n\nRead the [docs](https://nsq.io/components/nsqadmin.html)\n\n\n## Local Development\n\n### Dependencies\n\n 1. Install NodeJS 16.x (includes `npm`)\n\n### Live Reload Workflow\n\n 1. `$ npm install`\n 2. `$ ./gulp --series clean watch`\n 3. `$ cd .. && make && ./build/nsqadmin --dev-static-dir=nsqadmin/static/build --lookupd-http-address=<...>`\n 4. make changes to static assets (repeat step 3 only if you make changes to any Go code)\n\n### Build\n\n 1. `$ ./gulp --series clean build`\n" }, { "path": "nsqadmin/gulp", "content": "#!/bin/sh\nexec ./node_modules/.bin/gulp \"$@\"\n" }, { "path": "nsqadmin/gulpfile.js", "content": "var browserify = require('browserify');\nvar clean = require('gulp-clean');\nvar gulp = require('gulp');\nvar notify = require('gulp-notify');\nvar path = require('path');\nvar sass = require('gulp-dart-sass');\nvar source = require('vinyl-source-stream');\nvar taskListing = require('gulp-task-listing');\nvar uglify = require('gulp-uglify');\nvar sourcemaps = require('gulp-sourcemaps');\nvar buffer = require('vinyl-buffer');\n\n\nvar ROOT = 'static';\n\nvar VENDOR_CONFIG = {\n 'src': [\n 'backbone',\n 'jquery',\n 'underscore',\n 'bootbox',\n ],\n 'target': 'vendor.js',\n 'targetDir': './static/build/'\n};\n\nfunction excludeVendor(b) {\n VENDOR_CONFIG.src.forEach(function(vendorLib) {\n b.exclude(vendorLib);\n });\n}\n\nfunction bytesToKB(bytes) { return Math.floor(+bytes/1024); }\n\nfunction logBundle(filename, watching) {\n return function (err, buf) {\n if (err) {\n console.error(err.toString());\n if (!watching) {\n process.exit(1);\n }\n }\n if (!watching) {\n console.log(filename + ' ' + bytesToKB(buf.length) + ' KB written');\n }\n }\n}\n\n\nfunction sassTask(root, inputFile) {\n return function sassing() {\n var onError = function(err) {\n notify({'title': 'Sass Compile Error'}).write(err);\n };\n return gulp.src(path.join(root, 'css', inputFile))\n .pipe(sass({\n 'sourceComments': 'map',\n }).on('error', onError))\n .pipe(gulp.dest(path.join(root, 'build/')));\n };\n}\n\n\nfunction browserifyTask(root, inputFile) {\n return function browserifying() {\n var onError = function() {\n var args = Array.prototype.slice.call(arguments);\n notify.onError({\n 'title': 'JS Compile Error',\n 'message': '<%= error.message %>'\n }).apply(this, args);\n // Keep gulp from hanging on this task\n this.emit('end');\n };\n\n // Browserify needs a node module to import as its arg, so we need to\n // force the leading \"./\" to be included.\n var b = browserify({\n entries: './' + path.join(root, 'js', inputFile),\n debug: true\n })\n\n excludeVendor(b);\n\n return b.bundle()\n .pipe(source(inputFile))\n .pipe(buffer())\n .pipe(sourcemaps.init({'loadMaps': true, 'debug': true}))\n // Add transformation tasks to the pipeline here.\n .pipe(uglify())\n .on('error', onError)\n .pipe(sourcemaps.write('./'))\n .pipe(gulp.dest(path.join(root, 'build/')));\n };\n}\n\n\nfunction watchTask(root) {\n return function watching() {\n gulp.watch(path.join(root, 'sass/**/*.scss'), gulp.series('sass'));\n gulp.watch([\n path.join(root, 'js/**/*.js'),\n path.join(root, 'js/**/*.hbs')\n ], gulp.series('browserify'));\n gulp.watch([\n path.join(root, 'html/**'),\n path.join(root, 'fonts/**')\n ], gulp.series('sync-static-assets'))\n };\n}\n\n\nfunction cleanTask() {\n var paths = Array.prototype.slice.apply(arguments);\n return function cleaning() {\n return gulp.src(paths, {allowEmpty: true}).pipe(clean());\n };\n}\n\ngulp.task('vendor-build-js', function() {\n var onError = function() {\n var args = Array.prototype.slice.call(arguments);\n notify.onError({\n 'title': 'JS Compile Error',\n 'message': '<%= error.message %>'\n }).apply(this, args);\n // Keep gulp from hanging on this task\n this.emit('end');\n };\n\n var b = browserify()\n .require(VENDOR_CONFIG.src);\n\n return b.bundle(logBundle(VENDOR_CONFIG.target))\n .pipe(source(VENDOR_CONFIG.target))\n .pipe(buffer())\n // Add transformation tasks to the pipeline here.\n .pipe(uglify())\n .on('error', onError)\n .pipe(gulp.dest(VENDOR_CONFIG.targetDir));\n});\n\ngulp.task('help', taskListing);\n\ngulp.task('sync-static-assets', function() {\n return gulp.src([\n path.join(ROOT, 'html/**'),\n path.join(ROOT, 'fonts/**'),\n path.join(ROOT, 'img/**')\n ]).pipe(gulp.dest(path.join(ROOT, 'build')));\n});\n\ngulp.task('sass', sassTask(ROOT, '*.*css'));\ngulp.task('browserify', browserifyTask(ROOT, 'main.js'));\ngulp.task('build', gulp.parallel('sass', 'browserify', 'sync-static-assets', 'vendor-build-js'));\ngulp.task('watch', gulp.series('build', watchTask(ROOT)));\ngulp.task('clean', gulp.series(cleanTask(path.join(ROOT, 'build'))));\n\ngulp.task('default', gulp.series('help'));\n" }, { "path": "nsqadmin/http.go", "content": "package nsqadmin\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"html/template\"\n\t\"io\"\n\t\"mime\"\n\t\"net\"\n\t\"net/http\"\n\t\"net/http/httputil\"\n\t\"net/url\"\n\t\"os\"\n\t\"path\"\n\t\"reflect\"\n\t\"sort\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/julienschmidt/httprouter\"\n\t\"github.com/nsqio/nsq/internal/clusterinfo\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nfunc maybeWarnMsg(msgs []string) string {\n\tif len(msgs) > 0 {\n\t\treturn \"WARNING: \" + strings.Join(msgs, \"; \")\n\t}\n\treturn \"\"\n}\n\n// this is similar to httputil.NewSingleHostReverseProxy except it passes along basic auth\nfunc NewSingleHostReverseProxy(target *url.URL, connectTimeout time.Duration, requestTimeout time.Duration) *httputil.ReverseProxy {\n\tdirector := func(req *http.Request) {\n\t\treq.URL.Scheme = target.Scheme\n\t\treq.URL.Host = target.Host\n\t\tif target.User != nil {\n\t\t\tpasswd, _ := target.User.Password()\n\t\t\treq.SetBasicAuth(target.User.Username(), passwd)\n\t\t}\n\t}\n\treturn &httputil.ReverseProxy{\n\t\tDirector: director,\n\t\tTransport: http_api.NewDeadlineTransport(connectTimeout, requestTimeout),\n\t}\n}\n\ntype httpServer struct {\n\tnsqadmin *NSQAdmin\n\trouter http.Handler\n\tclient *http_api.Client\n\tci *clusterinfo.ClusterInfo\n\tbasePath string\n\tdevStaticDir string\n}\n\nfunc NewHTTPServer(nsqadmin *NSQAdmin) *httpServer {\n\tlog := http_api.Log(nsqadmin.logf)\n\n\tclient := http_api.NewClient(nsqadmin.httpClientTLSConfig, nsqadmin.getOpts().HTTPClientConnectTimeout,\n\t\tnsqadmin.getOpts().HTTPClientRequestTimeout)\n\n\trouter := httprouter.New()\n\trouter.HandleMethodNotAllowed = true\n\trouter.PanicHandler = http_api.LogPanicHandler(nsqadmin.logf)\n\trouter.NotFound = http_api.LogNotFoundHandler(nsqadmin.logf)\n\trouter.MethodNotAllowed = http_api.LogMethodNotAllowedHandler(nsqadmin.logf)\n\n\ts := &httpServer{\n\t\tnsqadmin: nsqadmin,\n\t\trouter: router,\n\t\tclient: client,\n\t\tci: clusterinfo.New(nsqadmin.logf, client),\n\n\t\tbasePath: nsqadmin.getOpts().BasePath,\n\t\tdevStaticDir: nsqadmin.getOpts().DevStaticDir,\n\t}\n\n\tbp := func(p string) string {\n\t\treturn path.Join(s.basePath, p)\n\t}\n\n\trouter.Handle(\"GET\", bp(\"/\"), http_api.Decorate(s.indexHandler, log))\n\trouter.Handle(\"GET\", bp(\"/ping\"), http_api.Decorate(s.pingHandler, log, http_api.PlainText))\n\n\trouter.Handle(\"GET\", bp(\"/topics\"), http_api.Decorate(s.indexHandler, log))\n\trouter.Handle(\"GET\", bp(\"/topics/:topic\"), http_api.Decorate(s.indexHandler, log))\n\trouter.Handle(\"GET\", bp(\"/topics/:topic/:channel\"), http_api.Decorate(s.indexHandler, log))\n\trouter.Handle(\"GET\", bp(\"/nodes\"), http_api.Decorate(s.indexHandler, log))\n\trouter.Handle(\"GET\", bp(\"/nodes/:node\"), http_api.Decorate(s.indexHandler, log))\n\trouter.Handle(\"GET\", bp(\"/counter\"), http_api.Decorate(s.indexHandler, log))\n\trouter.Handle(\"GET\", bp(\"/lookup\"), http_api.Decorate(s.indexHandler, log))\n\n\trouter.Handle(\"GET\", bp(\"/static/:asset\"), http_api.Decorate(s.staticAssetHandler, log, http_api.PlainText))\n\trouter.Handle(\"GET\", bp(\"/fonts/:asset\"), http_api.Decorate(s.staticAssetHandler, log, http_api.PlainText))\n\tif s.nsqadmin.getOpts().ProxyGraphite {\n\t\tproxy := NewSingleHostReverseProxy(nsqadmin.graphiteURL, nsqadmin.getOpts().HTTPClientConnectTimeout,\n\t\t\tnsqadmin.getOpts().HTTPClientRequestTimeout)\n\t\trouter.Handler(\"GET\", bp(\"/render\"), proxy)\n\t}\n\n\t// v1 endpoints\n\trouter.Handle(\"GET\", bp(\"/api/topics\"), http_api.Decorate(s.topicsHandler, log, http_api.V1))\n\trouter.Handle(\"GET\", bp(\"/api/topics/:topic\"), http_api.Decorate(s.topicHandler, log, http_api.V1))\n\trouter.Handle(\"GET\", bp(\"/api/topics/:topic/:channel\"), http_api.Decorate(s.channelHandler, log, http_api.V1))\n\trouter.Handle(\"GET\", bp(\"/api/nodes\"), http_api.Decorate(s.nodesHandler, log, http_api.V1))\n\trouter.Handle(\"GET\", bp(\"/api/nodes/:node\"), http_api.Decorate(s.nodeHandler, log, http_api.V1))\n\trouter.Handle(\"POST\", bp(\"/api/topics\"), http_api.Decorate(s.createTopicChannelHandler, log, http_api.V1))\n\trouter.Handle(\"POST\", bp(\"/api/topics/:topic\"), http_api.Decorate(s.topicActionHandler, log, http_api.V1))\n\trouter.Handle(\"POST\", bp(\"/api/topics/:topic/:channel\"), http_api.Decorate(s.channelActionHandler, log, http_api.V1))\n\trouter.Handle(\"DELETE\", bp(\"/api/nodes/:node\"), http_api.Decorate(s.tombstoneNodeForTopicHandler, log, http_api.V1))\n\trouter.Handle(\"DELETE\", bp(\"/api/topics/:topic\"), http_api.Decorate(s.deleteTopicHandler, log, http_api.V1))\n\trouter.Handle(\"DELETE\", bp(\"/api/topics/:topic/:channel\"), http_api.Decorate(s.deleteChannelHandler, log, http_api.V1))\n\trouter.Handle(\"GET\", bp(\"/api/counter\"), http_api.Decorate(s.counterHandler, log, http_api.V1))\n\trouter.Handle(\"GET\", bp(\"/api/graphite\"), http_api.Decorate(s.graphiteHandler, log, http_api.V1))\n\trouter.Handle(\"GET\", bp(\"/config/:opt\"), http_api.Decorate(s.doConfig, log, http_api.V1))\n\trouter.Handle(\"PUT\", bp(\"/config/:opt\"), http_api.Decorate(s.doConfig, log, http_api.V1))\n\n\treturn s\n}\n\nfunc (s *httpServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {\n\ts.router.ServeHTTP(w, req)\n}\n\nfunc (s *httpServer) pingHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treturn \"OK\", nil\n}\n\nfunc (s *httpServer) indexHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tasset, _ := staticAsset(\"index.html\")\n\tt, _ := template.New(\"index\").Funcs(template.FuncMap{\n\t\t\"basePath\": func(p string) string {\n\t\t\treturn path.Join(s.basePath, p)\n\t\t},\n\t}).Parse(string(asset))\n\n\tw.Header().Set(\"Content-Type\", \"text/html\")\n\tt.Execute(w, struct {\n\t\tVersion string\n\t\tProxyGraphite bool\n\t\tGraphEnabled bool\n\t\tGraphiteURL string\n\t\tStatsdInterval int\n\t\tStatsdCounterFormat string\n\t\tStatsdGaugeFormat string\n\t\tStatsdPrefix string\n\t\tNSQLookupd []string\n\t\tIsAdmin bool\n\t}{\n\t\tVersion: version.Binary,\n\t\tProxyGraphite: s.nsqadmin.getOpts().ProxyGraphite,\n\t\tGraphEnabled: s.nsqadmin.getOpts().GraphiteURL != \"\",\n\t\tGraphiteURL: s.nsqadmin.getOpts().GraphiteURL,\n\t\tStatsdInterval: int(s.nsqadmin.getOpts().StatsdInterval / time.Second),\n\t\tStatsdCounterFormat: s.nsqadmin.getOpts().StatsdCounterFormat,\n\t\tStatsdGaugeFormat: s.nsqadmin.getOpts().StatsdGaugeFormat,\n\t\tStatsdPrefix: s.nsqadmin.getOpts().StatsdPrefix,\n\t\tNSQLookupd: s.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\tIsAdmin: s.isAuthorizedAdminRequest(req),\n\t})\n\n\treturn nil, nil\n}\n\nfunc (s *httpServer) staticAssetHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tassetName := ps.ByName(\"asset\")\n\n\tvar (\n\t\tasset []byte\n\t\terr error\n\t)\n\tif s.devStaticDir != \"\" {\n\t\ts.nsqadmin.logf(LOG_DEBUG, \"using dev dir %q for static asset %q\", s.devStaticDir, assetName)\n\t\tfsPath := path.Join(s.devStaticDir, assetName)\n\t\tasset, err = os.ReadFile(fsPath)\n\t} else {\n\t\tasset, err = staticAsset(assetName)\n\t}\n\tif err != nil {\n\t\treturn nil, http_api.Err{404, \"NOT_FOUND\"}\n\t}\n\n\text := path.Ext(assetName)\n\tct := mime.TypeByExtension(ext)\n\tif ct == \"\" {\n\t\tswitch ext {\n\t\tcase \".map\":\n\t\t\tct = \"application/json\"\n\t\tcase \".svg\":\n\t\t\tct = \"image/svg+xml\"\n\t\tcase \".woff\":\n\t\t\tct = \"application/font-woff\"\n\t\tcase \".ttf\":\n\t\t\tct = \"application/font-sfnt\"\n\t\tcase \".eot\":\n\t\t\tct = \"application/vnd.ms-fontobject\"\n\t\tcase \".woff2\":\n\t\t\tct = \"application/font-woff2\"\n\t\t}\n\t}\n\tif ct != \"\" {\n\t\tw.Header().Set(\"Content-Type\", ct)\n\t}\n\n\treturn string(asset), nil\n}\n\nfunc (s *httpServer) topicsHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, err.Error()}\n\t}\n\n\tvar topics []string\n\tif len(s.nsqadmin.getOpts().NSQLookupdHTTPAddresses) != 0 {\n\t\ttopics, err = s.ci.GetLookupdTopics(s.nsqadmin.getOpts().NSQLookupdHTTPAddresses)\n\t} else {\n\t\ttopics, err = s.ci.GetNSQDTopics(s.nsqadmin.getOpts().NSQDHTTPAddresses)\n\t}\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get topics - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\tinactive, _ := reqParams.Get(\"inactive\")\n\tif inactive == \"true\" {\n\t\ttopicChannelMap := make(map[string][]string)\n\t\tif len(s.nsqadmin.getOpts().NSQLookupdHTTPAddresses) == 0 {\n\t\t\tgoto respond\n\t\t}\n\t\tfor _, topicName := range topics {\n\t\t\tproducers, _ := s.ci.GetLookupdTopicProducers(\n\t\t\t\ttopicName, s.nsqadmin.getOpts().NSQLookupdHTTPAddresses)\n\t\t\tif len(producers) == 0 {\n\t\t\t\ttopicChannels, _ := s.ci.GetLookupdTopicChannels(\n\t\t\t\t\ttopicName, s.nsqadmin.getOpts().NSQLookupdHTTPAddresses)\n\t\t\t\ttopicChannelMap[topicName] = topicChannels\n\t\t\t}\n\t\t}\n\trespond:\n\t\treturn struct {\n\t\t\tTopics map[string][]string `json:\"topics\"`\n\t\t\tMessage string `json:\"message\"`\n\t\t}{topicChannelMap, maybeWarnMsg(messages)}, nil\n\t}\n\n\treturn struct {\n\t\tTopics []string `json:\"topics\"`\n\t\tMessage string `json:\"message\"`\n\t}{topics, maybeWarnMsg(messages)}, nil\n}\n\nfunc (s *httpServer) topicHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\n\ttopicName := ps.ByName(\"topic\")\n\n\tproducers, err := s.ci.GetTopicProducers(topicName,\n\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get topic producers - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\ttopicStats, _, err := s.ci.GetNSQDStats(producers, topicName, \"\", false)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get topic metadata - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\tallNodesTopicStats := &clusterinfo.TopicStats{TopicName: topicName}\n\tfor _, t := range topicStats {\n\t\tallNodesTopicStats.Add(t)\n\t}\n\n\treturn struct {\n\t\t*clusterinfo.TopicStats\n\t\tMessage string `json:\"message\"`\n\t}{allNodesTopicStats, maybeWarnMsg(messages)}, nil\n}\n\nfunc (s *httpServer) channelHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\n\ttopicName := ps.ByName(\"topic\")\n\tchannelName := ps.ByName(\"channel\")\n\n\tproducers, err := s.ci.GetTopicProducers(topicName,\n\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get topic producers - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\t_, channelStats, err := s.ci.GetNSQDStats(producers, topicName, channelName, true)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get channel metadata - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\tsort.Sort(clusterinfo.ClientStatsByNodeTopology{channelStats[channelName].Clients})\n\n\treturn struct {\n\t\t*clusterinfo.ChannelStats\n\t\tMessage string `json:\"message\"`\n\t}{channelStats[channelName], maybeWarnMsg(messages)}, nil\n}\n\nfunc (s *httpServer) nodesHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\n\tproducers, err := s.ci.GetProducers(s.nsqadmin.getOpts().NSQLookupdHTTPAddresses, s.nsqadmin.getOpts().NSQDHTTPAddresses)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get nodes - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\treturn struct {\n\t\tNodes clusterinfo.Producers `json:\"nodes\"`\n\t\tMessage string `json:\"message\"`\n\t}{producers, maybeWarnMsg(messages)}, nil\n}\n\nfunc (s *httpServer) nodeHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\n\tnode := ps.ByName(\"node\")\n\n\tproducers, err := s.ci.GetProducers(s.nsqadmin.getOpts().NSQLookupdHTTPAddresses, s.nsqadmin.getOpts().NSQDHTTPAddresses)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get producers - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\tproducer := producers.Search(node)\n\tif producer == nil {\n\t\treturn nil, http_api.Err{404, \"NODE_NOT_FOUND\"}\n\t}\n\n\ttopicStats, _, err := s.ci.GetNSQDStats(clusterinfo.Producers{producer}, \"\", \"\", true)\n\tif err != nil {\n\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get nsqd stats - %s\", err)\n\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t}\n\n\tvar totalClients int64\n\tvar totalMessages int64\n\tfor _, ts := range topicStats {\n\t\tfor _, cs := range ts.Channels {\n\t\t\ttotalClients += int64(len(cs.Clients))\n\t\t}\n\t\ttotalMessages += ts.MessageCount\n\t}\n\n\treturn struct {\n\t\tNode string `json:\"node\"`\n\t\tTopicStats []*clusterinfo.TopicStats `json:\"topics\"`\n\t\tTotalMessages int64 `json:\"total_messages\"`\n\t\tTotalClients int64 `json:\"total_clients\"`\n\t\tMessage string `json:\"message\"`\n\t}{\n\t\tNode: node,\n\t\tTopicStats: topicStats,\n\t\tTotalMessages: totalMessages,\n\t\tTotalClients: totalClients,\n\t\tMessage: maybeWarnMsg(messages),\n\t}, nil\n}\n\nfunc (s *httpServer) tombstoneNodeForTopicHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\n\tif !s.isAuthorizedAdminRequest(req) {\n\t\treturn nil, http_api.Err{403, \"FORBIDDEN\"}\n\t}\n\n\tnode := ps.ByName(\"node\")\n\n\tvar body struct {\n\t\tTopic string `json:\"topic\"`\n\t}\n\terr := json.NewDecoder(req.Body).Decode(&body)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_BODY\"}\n\t}\n\n\tif !protocol.IsValidTopicName(body.Topic) {\n\t\treturn nil, http_api.Err{400, \"INVALID_TOPIC\"}\n\t}\n\n\terr = s.ci.TombstoneNodeForTopic(body.Topic, node,\n\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to tombstone node for topic - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\ts.notifyAdminAction(\"tombstone_topic_producer\", body.Topic, \"\", node, req)\n\n\treturn struct {\n\t\tMessage string `json:\"message\"`\n\t}{maybeWarnMsg(messages)}, nil\n}\n\nfunc (s *httpServer) createTopicChannelHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\n\tvar body struct {\n\t\tTopic string `json:\"topic\"`\n\t\tChannel string `json:\"channel\"`\n\t}\n\n\tif !s.isAuthorizedAdminRequest(req) {\n\t\treturn nil, http_api.Err{403, \"FORBIDDEN\"}\n\t}\n\n\terr := json.NewDecoder(req.Body).Decode(&body)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, err.Error()}\n\t}\n\n\tif !protocol.IsValidTopicName(body.Topic) {\n\t\treturn nil, http_api.Err{400, \"INVALID_TOPIC\"}\n\t}\n\n\tif len(body.Channel) > 0 && !protocol.IsValidChannelName(body.Channel) {\n\t\treturn nil, http_api.Err{400, \"INVALID_CHANNEL\"}\n\t}\n\n\terr = s.ci.CreateTopicChannel(body.Topic, body.Channel,\n\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to create topic/channel - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\ts.notifyAdminAction(\"create_topic\", body.Topic, \"\", \"\", req)\n\tif len(body.Channel) > 0 {\n\t\ts.notifyAdminAction(\"create_channel\", body.Topic, body.Channel, \"\", req)\n\t}\n\n\treturn struct {\n\t\tMessage string `json:\"message\"`\n\t}{maybeWarnMsg(messages)}, nil\n}\n\nfunc (s *httpServer) deleteTopicHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\n\tif !s.isAuthorizedAdminRequest(req) {\n\t\treturn nil, http_api.Err{403, \"FORBIDDEN\"}\n\t}\n\n\ttopicName := ps.ByName(\"topic\")\n\n\terr := s.ci.DeleteTopic(topicName,\n\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to delete topic - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\ts.notifyAdminAction(\"delete_topic\", topicName, \"\", \"\", req)\n\n\treturn struct {\n\t\tMessage string `json:\"message\"`\n\t}{maybeWarnMsg(messages)}, nil\n}\n\nfunc (s *httpServer) deleteChannelHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\n\tif !s.isAuthorizedAdminRequest(req) {\n\t\treturn nil, http_api.Err{403, \"FORBIDDEN\"}\n\t}\n\n\ttopicName := ps.ByName(\"topic\")\n\tchannelName := ps.ByName(\"channel\")\n\n\terr := s.ci.DeleteChannel(topicName, channelName,\n\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to delete channel - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\ts.notifyAdminAction(\"delete_channel\", topicName, channelName, \"\", req)\n\n\treturn struct {\n\t\tMessage string `json:\"message\"`\n\t}{maybeWarnMsg(messages)}, nil\n}\n\nfunc (s *httpServer) topicActionHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\ttopicName := ps.ByName(\"topic\")\n\treturn s.topicChannelAction(req, topicName, \"\")\n}\n\nfunc (s *httpServer) channelActionHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\ttopicName := ps.ByName(\"topic\")\n\tchannelName := ps.ByName(\"channel\")\n\treturn s.topicChannelAction(req, topicName, channelName)\n}\n\nfunc (s *httpServer) topicChannelAction(req *http.Request, topicName string, channelName string) (interface{}, error) {\n\tvar messages []string\n\n\tvar body struct {\n\t\tAction string `json:\"action\"`\n\t}\n\n\tif !s.isAuthorizedAdminRequest(req) {\n\t\treturn nil, http_api.Err{403, \"FORBIDDEN\"}\n\t}\n\n\terr := json.NewDecoder(req.Body).Decode(&body)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, err.Error()}\n\t}\n\n\tswitch body.Action {\n\tcase \"pause\":\n\t\tif channelName != \"\" {\n\t\t\terr = s.ci.PauseChannel(topicName, channelName,\n\t\t\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\t\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\n\t\t\ts.notifyAdminAction(\"pause_channel\", topicName, channelName, \"\", req)\n\t\t} else {\n\t\t\terr = s.ci.PauseTopic(topicName,\n\t\t\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\t\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\n\t\t\ts.notifyAdminAction(\"pause_topic\", topicName, \"\", \"\", req)\n\t\t}\n\tcase \"unpause\":\n\t\tif channelName != \"\" {\n\t\t\terr = s.ci.UnPauseChannel(topicName, channelName,\n\t\t\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\t\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\n\t\t\ts.notifyAdminAction(\"unpause_channel\", topicName, channelName, \"\", req)\n\t\t} else {\n\t\t\terr = s.ci.UnPauseTopic(topicName,\n\t\t\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\t\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\n\t\t\ts.notifyAdminAction(\"unpause_topic\", topicName, \"\", \"\", req)\n\t\t}\n\tcase \"empty\":\n\t\tif channelName != \"\" {\n\t\t\terr = s.ci.EmptyChannel(topicName, channelName,\n\t\t\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\t\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\n\t\t\ts.notifyAdminAction(\"empty_channel\", topicName, channelName, \"\", req)\n\t\t} else {\n\t\t\terr = s.ci.EmptyTopic(topicName,\n\t\t\t\ts.nsqadmin.getOpts().NSQLookupdHTTPAddresses,\n\t\t\t\ts.nsqadmin.getOpts().NSQDHTTPAddresses)\n\n\t\t\ts.notifyAdminAction(\"empty_topic\", topicName, \"\", \"\", req)\n\t\t}\n\tdefault:\n\t\treturn nil, http_api.Err{400, \"INVALID_ACTION\"}\n\t}\n\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to %s topic/channel - %s\", body.Action, err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\treturn struct {\n\t\tMessage string `json:\"message\"`\n\t}{maybeWarnMsg(messages)}, nil\n}\n\ntype counterStats struct {\n\tNode string `json:\"node\"`\n\tTopicName string `json:\"topic_name\"`\n\tChannelName string `json:\"channel_name\"`\n\tMessageCount int64 `json:\"message_count\"`\n}\n\nfunc (s *httpServer) counterHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar messages []string\n\tstats := make(map[string]*counterStats)\n\n\tproducers, err := s.ci.GetProducers(s.nsqadmin.getOpts().NSQLookupdHTTPAddresses, s.nsqadmin.getOpts().NSQDHTTPAddresses)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get counter producer list - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\t_, channelStats, err := s.ci.GetNSQDStats(producers, \"\", \"\", false)\n\tif err != nil {\n\t\tpe, ok := err.(clusterinfo.PartialErr)\n\t\tif !ok {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to get nsqd stats - %s\", err)\n\t\t\treturn nil, http_api.Err{502, fmt.Sprintf(\"UPSTREAM_ERROR: %s\", err)}\n\t\t}\n\t\ts.nsqadmin.logf(LOG_WARN, \"%s\", err)\n\t\tmessages = append(messages, pe.Error())\n\t}\n\n\tfor _, channelStats := range channelStats {\n\t\tfor _, hostChannelStats := range channelStats.NodeStats {\n\t\t\tkey := fmt.Sprintf(\"%s:%s:%s\", channelStats.TopicName, channelStats.ChannelName, hostChannelStats.Node)\n\t\t\ts, ok := stats[key]\n\t\t\tif !ok {\n\t\t\t\ts = &counterStats{\n\t\t\t\t\tNode: hostChannelStats.Node,\n\t\t\t\t\tTopicName: channelStats.TopicName,\n\t\t\t\t\tChannelName: channelStats.ChannelName,\n\t\t\t\t}\n\t\t\t\tstats[key] = s\n\t\t\t}\n\t\t\ts.MessageCount += hostChannelStats.MessageCount\n\t\t}\n\t}\n\n\treturn struct {\n\t\tStats map[string]*counterStats `json:\"stats\"`\n\t\tMessage string `json:\"message\"`\n\t}{stats, maybeWarnMsg(messages)}, nil\n}\n\nfunc (s *httpServer) graphiteHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\tmetric, err := reqParams.Get(\"metric\")\n\tif err != nil || metric != \"rate\" {\n\t\treturn nil, http_api.Err{400, \"INVALID_ARG_METRIC\"}\n\t}\n\n\ttarget, err := reqParams.Get(\"target\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_ARG_TARGET\"}\n\t}\n\n\tparams := url.Values{}\n\tparams.Set(\"from\", fmt.Sprintf(\"-%dsec\", s.nsqadmin.getOpts().StatsdInterval*2/time.Second))\n\tparams.Set(\"until\", fmt.Sprintf(\"-%dsec\", s.nsqadmin.getOpts().StatsdInterval/time.Second))\n\tparams.Set(\"format\", \"json\")\n\tparams.Set(\"target\", target)\n\tquery := fmt.Sprintf(\"/render?%s\", params.Encode())\n\turl := s.nsqadmin.getOpts().GraphiteURL + query\n\n\ts.nsqadmin.logf(LOG_INFO, \"GRAPHITE: %s\", url)\n\n\tvar response []struct {\n\t\tTarget string `json:\"target\"`\n\t\tDataPoints [][]*float64 `json:\"datapoints\"`\n\t}\n\terr = s.client.GETV1(url, &response)\n\tif err != nil {\n\t\ts.nsqadmin.logf(LOG_ERROR, \"graphite request failed - %s\", err)\n\t\treturn nil, http_api.Err{500, \"INTERNAL_ERROR\"}\n\t}\n\n\tvar rateStr string\n\trate := *response[0].DataPoints[0][0]\n\tif rate < 0 {\n\t\trateStr = \"N/A\"\n\t} else {\n\t\trateDivisor := s.nsqadmin.getOpts().StatsdInterval / time.Second\n\t\trateStr = fmt.Sprintf(\"%.2f\", rate/float64(rateDivisor))\n\t}\n\treturn struct {\n\t\tRate string `json:\"rate\"`\n\t}{rateStr}, nil\n}\n\nfunc (s *httpServer) doConfig(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\topt := ps.ByName(\"opt\")\n\n\tallowConfigFromCIDR := s.nsqadmin.getOpts().AllowConfigFromCIDR\n\tif allowConfigFromCIDR != \"\" {\n\t\t_, ipnet, _ := net.ParseCIDR(allowConfigFromCIDR)\n\t\taddr, _, err := net.SplitHostPort(req.RemoteAddr)\n\t\tif err != nil {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to parse RemoteAddr %s\", req.RemoteAddr)\n\t\t\treturn nil, http_api.Err{400, \"INVALID_REMOTE_ADDR\"}\n\t\t}\n\t\tip := net.ParseIP(addr)\n\t\tif ip == nil {\n\t\t\ts.nsqadmin.logf(LOG_ERROR, \"failed to parse RemoteAddr %s\", req.RemoteAddr)\n\t\t\treturn nil, http_api.Err{400, \"INVALID_REMOTE_ADDR\"}\n\t\t}\n\t\tif !ipnet.Contains(ip) {\n\t\t\treturn nil, http_api.Err{403, \"FORBIDDEN\"}\n\t\t}\n\t}\n\n\tif req.Method == \"PUT\" {\n\t\t// add 1 so that it's greater than our max when we test for it\n\t\t// (LimitReader returns a \"fake\" EOF)\n\t\treadMax := int64(1024*1024 + 1)\n\t\tbody, err := io.ReadAll(io.LimitReader(req.Body, readMax))\n\t\tif err != nil {\n\t\t\treturn nil, http_api.Err{500, \"INTERNAL_ERROR\"}\n\t\t}\n\t\tif int64(len(body)) == readMax || len(body) == 0 {\n\t\t\treturn nil, http_api.Err{413, \"INVALID_VALUE\"}\n\t\t}\n\n\t\topts := *s.nsqadmin.getOpts()\n\t\tswitch opt {\n\t\tcase \"nsqlookupd_http_addresses\":\n\t\t\terr := json.Unmarshal(body, &opts.NSQLookupdHTTPAddresses)\n\t\t\tif err != nil {\n\t\t\t\treturn nil, http_api.Err{400, \"INVALID_VALUE\"}\n\t\t\t}\n\t\tcase \"log_level\":\n\t\t\tlogLevelStr := string(body)\n\t\t\tlogLevel, err := lg.ParseLogLevel(logLevelStr)\n\t\t\tif err != nil {\n\t\t\t\treturn nil, http_api.Err{400, \"INVALID_VALUE\"}\n\t\t\t}\n\t\t\topts.LogLevel = logLevel\n\t\tdefault:\n\t\t\treturn nil, http_api.Err{400, \"INVALID_OPTION\"}\n\t\t}\n\t\ts.nsqadmin.swapOpts(&opts)\n\t}\n\n\tv, ok := getOptByCfgName(s.nsqadmin.getOpts(), opt)\n\tif !ok {\n\t\treturn nil, http_api.Err{400, \"INVALID_OPTION\"}\n\t}\n\n\treturn v, nil\n}\n\nfunc (s *httpServer) isAuthorizedAdminRequest(req *http.Request) bool {\n\tadminUsers := s.nsqadmin.getOpts().AdminUsers\n\tif len(adminUsers) == 0 {\n\t\treturn true\n\t}\n\taclHTTPHeader := s.nsqadmin.getOpts().ACLHTTPHeader\n\tuser := req.Header.Get(aclHTTPHeader)\n\tfor _, v := range adminUsers {\n\t\tif v == user {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc getOptByCfgName(opts interface{}, name string) (interface{}, bool) {\n\tval := reflect.ValueOf(opts).Elem()\n\ttyp := val.Type()\n\tfor i := 0; i < typ.NumField(); i++ {\n\t\tfield := typ.Field(i)\n\t\tflagName := field.Tag.Get(\"flag\")\n\t\tcfgName := field.Tag.Get(\"cfg\")\n\t\tif flagName == \"\" {\n\t\t\tcontinue\n\t\t}\n\t\tif cfgName == \"\" {\n\t\t\tcfgName = strings.Replace(flagName, \"-\", \"_\", -1)\n\t\t}\n\t\tif name != cfgName {\n\t\t\tcontinue\n\t\t}\n\t\treturn val.FieldByName(field.Name).Interface(), true\n\t}\n\treturn nil, false\n}\n" }, { "path": "nsqadmin/http_test.go", "content": "package nsqadmin\n\nimport (\n\t\"bytes\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net\"\n\t\"net/http\"\n\t\"os\"\n\t\"strconv\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/clusterinfo\"\n\t\"github.com/nsqio/nsq/internal/test\"\n\t\"github.com/nsqio/nsq/internal/version\"\n\t\"github.com/nsqio/nsq/nsqd\"\n\t\"github.com/nsqio/nsq/nsqlookupd\"\n)\n\ntype TopicsDoc struct {\n\tTopics []interface{} `json:\"topics\"`\n}\n\ntype TopicStatsDoc struct {\n\t*clusterinfo.TopicStats\n\tMessage string `json:\"message\"`\n}\n\ntype NodesDoc struct {\n\tNodes clusterinfo.Producers `json:\"nodes\"`\n\tMessage string `json:\"message\"`\n}\n\ntype NodeStatsDoc struct {\n\tNode string `json:\"node\"`\n\tTopicStats []*clusterinfo.TopicStats `json:\"topics\"`\n\tTotalMessages int64 `json:\"total_messages\"`\n\tTotalClients int64 `json:\"total_clients\"`\n\tMessage string `json:\"message\"`\n}\n\ntype ChannelStatsDoc struct {\n\t*clusterinfo.ChannelStats\n\tMessage string `json:\"message\"`\n}\n\nfunc mustStartNSQLookupd(opts *nsqlookupd.Options) (*net.TCPAddr, *net.TCPAddr, *nsqlookupd.NSQLookupd) {\n\topts.TCPAddress = \"127.0.0.1:0\"\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\tlookupd, err := nsqlookupd.New(opts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := lookupd.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\treturn lookupd.RealTCPAddr(), lookupd.RealHTTPAddr(), lookupd\n}\n\nfunc bootstrapNSQCluster(t *testing.T) (string, []*nsqd.NSQD, []*nsqlookupd.NSQLookupd, *NSQAdmin) {\n\treturn bootstrapNSQClusterWithAuth(t, false)\n}\n\nfunc bootstrapNSQClusterWithAuth(t *testing.T, withAuth bool) (string, []*nsqd.NSQD, []*nsqlookupd.NSQLookupd, *NSQAdmin) {\n\tlgr := test.NewTestLogger(t)\n\n\tnsqlookupdOpts := nsqlookupd.NewOptions()\n\tnsqlookupdOpts.TCPAddress = \"127.0.0.1:0\"\n\tnsqlookupdOpts.HTTPAddress = \"127.0.0.1:0\"\n\tnsqlookupdOpts.BroadcastAddress = \"127.0.0.1\"\n\tnsqlookupdOpts.Logger = lgr\n\tnsqlookupd1, err := nsqlookupd.New(nsqlookupdOpts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := nsqlookupd1.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\tnsqdOpts := nsqd.NewOptions()\n\tnsqdOpts.TCPAddress = \"127.0.0.1:0\"\n\tnsqdOpts.HTTPAddress = \"127.0.0.1:0\"\n\tnsqdOpts.BroadcastAddress = \"127.0.0.1\"\n\tnsqdOpts.NSQLookupdTCPAddresses = []string{nsqlookupd1.RealTCPAddr().String()}\n\tnsqdOpts.Logger = lgr\n\ttmpDir, err := os.MkdirTemp(\"\", \"nsq-test-\")\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tnsqdOpts.DataPath = tmpDir\n\tnsqd1, err := nsqd.New(nsqdOpts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := nsqd1.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\n\tnsqadminOpts := NewOptions()\n\tnsqadminOpts.HTTPAddress = \"127.0.0.1:0\"\n\tnsqadminOpts.NSQLookupdHTTPAddresses = []string{nsqlookupd1.RealHTTPAddr().String()}\n\tnsqadminOpts.Logger = lgr\n\tif withAuth {\n\t\tnsqadminOpts.AdminUsers = []string{\"matt\"}\n\t}\n\tnsqadmin1, err := New(nsqadminOpts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := nsqadmin1.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\treturn tmpDir, []*nsqd.NSQD{nsqd1}, []*nsqlookupd.NSQLookupd{nsqlookupd1}, nsqadmin1\n}\n\nfunc TestPing(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/ping\", nsqadmin1.RealHTTPAddr())\n\treq, _ := http.NewRequest(\"GET\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\ttest.Equal(t, []byte(\"OK\"), body)\n}\n\nfunc TestHTTPTopicsGET(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_topics_get\" + strconv.Itoa(int(time.Now().Unix()))\n\tnsqds[0].GetTopic(topicName)\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics\", nsqadmin1.RealHTTPAddr())\n\treq, _ := http.NewRequest(\"GET\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\ttr := TopicsDoc{}\n\terr = json.Unmarshal(body, &tr)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 1, len(tr.Topics))\n\ttest.Equal(t, topicName, tr.Topics[0])\n}\n\nfunc TestHTTPTopicGET(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_topic_get\" + strconv.Itoa(int(time.Now().Unix()))\n\tnsqds[0].GetTopic(topicName)\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics/%s\", nsqadmin1.RealHTTPAddr(), topicName)\n\treq, _ := http.NewRequest(\"GET\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\tts := TopicStatsDoc{}\n\terr = json.Unmarshal(body, &ts)\n\ttest.Nil(t, err)\n\ttest.Equal(t, topicName, ts.TopicName)\n\ttest.Equal(t, 0, int(ts.Depth))\n\ttest.Equal(t, 0, int(ts.MemoryDepth))\n\ttest.Equal(t, 0, int(ts.BackendDepth))\n\ttest.Equal(t, 0, int(ts.MessageCount))\n\ttest.Equal(t, false, ts.Paused)\n}\n\nfunc TestHTTPNodesGET(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/nodes\", nsqadmin1.RealHTTPAddr())\n\treq, _ := http.NewRequest(\"GET\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\thostname, _ := os.Hostname()\n\n\tt.Logf(\"%s\", body)\n\tns := NodesDoc{}\n\terr = json.Unmarshal(body, &ns)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 1, len(ns.Nodes))\n\ttestNode := ns.Nodes[0]\n\ttest.Equal(t, hostname, testNode.Hostname)\n\ttest.Equal(t, \"127.0.0.1\", testNode.BroadcastAddress)\n\ttest.Equal(t, nsqds[0].RealTCPAddr().(*net.TCPAddr).Port, testNode.TCPPort)\n\ttest.Equal(t, nsqds[0].RealHTTPAddr().(*net.TCPAddr).Port, testNode.HTTPPort)\n\ttest.Equal(t, version.Binary, testNode.Version)\n\ttest.Equal(t, 0, len(testNode.Topics))\n}\n\nfunc TestHTTPChannelGET(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_channel_get\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqds[0].GetTopic(topicName)\n\ttopic.GetChannel(\"ch\")\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics/%s/ch\", nsqadmin1.RealHTTPAddr(), topicName)\n\treq, _ := http.NewRequest(\"GET\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\tcs := ChannelStatsDoc{}\n\terr = json.Unmarshal(body, &cs)\n\ttest.Nil(t, err)\n\ttest.Equal(t, topicName, cs.TopicName)\n\ttest.Equal(t, \"ch\", cs.ChannelName)\n\ttest.Equal(t, 0, int(cs.Depth))\n\ttest.Equal(t, 0, int(cs.MemoryDepth))\n\ttest.Equal(t, 0, int(cs.BackendDepth))\n\ttest.Equal(t, 0, int(cs.MessageCount))\n\ttest.Equal(t, false, cs.Paused)\n\ttest.Equal(t, 0, int(cs.InFlightCount))\n\ttest.Equal(t, 0, int(cs.DeferredCount))\n\ttest.Equal(t, 0, int(cs.RequeueCount))\n\ttest.Equal(t, 0, int(cs.TimeoutCount))\n\ttest.Equal(t, 0, len(cs.Clients))\n}\n\nfunc TestHTTPNodesSingleGET(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_nodes_single_get\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqds[0].GetTopic(topicName)\n\ttopic.GetChannel(\"ch\")\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/nodes/%s\", nsqadmin1.RealHTTPAddr(),\n\t\tnsqds[0].RealHTTPAddr().String())\n\treq, _ := http.NewRequest(\"GET\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\tns := NodeStatsDoc{}\n\terr = json.Unmarshal(body, &ns)\n\ttest.Nil(t, err)\n\ttest.Equal(t, nsqds[0].RealHTTPAddr().String(), ns.Node)\n\ttest.Equal(t, 1, len(ns.TopicStats))\n\ttestTopic := ns.TopicStats[0]\n\ttest.Equal(t, topicName, testTopic.TopicName)\n\ttest.Equal(t, 0, int(testTopic.Depth))\n\ttest.Equal(t, 0, int(testTopic.MemoryDepth))\n\ttest.Equal(t, 0, int(testTopic.BackendDepth))\n\ttest.Equal(t, 0, int(testTopic.MessageCount))\n\ttest.Equal(t, false, testTopic.Paused)\n}\n\nfunc TestHTTPCreateTopicPOST(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\ttopicName := \"test_create_topic_post\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics\", nsqadmin1.RealHTTPAddr())\n\tbody, _ := json.Marshal(map[string]interface{}{\n\t\t\"topic\": topicName,\n\t})\n\treq, _ := http.NewRequest(\"POST\", url, bytes.NewBuffer(body))\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n}\n\nfunc TestHTTPCreateTopicChannelPOST(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\ttopicName := \"test_create_topic_channel_post\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics\", nsqadmin1.RealHTTPAddr())\n\tbody, _ := json.Marshal(map[string]interface{}{\n\t\t\"topic\": topicName,\n\t\t\"channel\": \"ch\",\n\t})\n\treq, _ := http.NewRequest(\"POST\", url, bytes.NewBuffer(body))\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n}\n\nfunc TestHTTPTombstoneTopicNodePOST(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_tombstone_topic_node_post\" + strconv.Itoa(int(time.Now().Unix()))\n\tnsqds[0].GetTopic(topicName)\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/nodes/%s\", nsqadmin1.RealHTTPAddr(), nsqds[0].RealHTTPAddr())\n\tbody, _ := json.Marshal(map[string]interface{}{\n\t\t\"topic\": topicName,\n\t})\n\treq, _ := http.NewRequest(\"DELETE\", url, bytes.NewBuffer(body))\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n}\n\nfunc TestHTTPDeleteTopicPOST(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_delete_topic_post\" + strconv.Itoa(int(time.Now().Unix()))\n\tnsqds[0].GetTopic(topicName)\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics/%s\", nsqadmin1.RealHTTPAddr(), topicName)\n\treq, _ := http.NewRequest(\"DELETE\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n}\n\nfunc TestHTTPDeleteChannelPOST(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_delete_channel_post\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqds[0].GetTopic(topicName)\n\ttopic.GetChannel(\"ch\")\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics/%s/ch\", nsqadmin1.RealHTTPAddr(), topicName)\n\treq, _ := http.NewRequest(\"DELETE\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n}\n\nfunc TestHTTPPauseTopicPOST(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_pause_topic_post\" + strconv.Itoa(int(time.Now().Unix()))\n\tnsqds[0].GetTopic(topicName)\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics/%s\", nsqadmin1.RealHTTPAddr(), topicName)\n\tbody, _ := json.Marshal(map[string]interface{}{\n\t\t\"action\": \"pause\",\n\t})\n\treq, _ := http.NewRequest(\"POST\", url, bytes.NewBuffer(body))\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\t_, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n\n\turl = fmt.Sprintf(\"http://%s/api/topics/%s\", nsqadmin1.RealHTTPAddr(), topicName)\n\tbody, _ = json.Marshal(map[string]interface{}{\n\t\t\"action\": \"unpause\",\n\t})\n\treq, _ = http.NewRequest(\"POST\", url, bytes.NewBuffer(body))\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n}\n\nfunc TestHTTPPauseChannelPOST(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_pause_channel_post\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqds[0].GetTopic(topicName)\n\ttopic.GetChannel(\"ch\")\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics/%s/ch\", nsqadmin1.RealHTTPAddr(), topicName)\n\tbody, _ := json.Marshal(map[string]interface{}{\n\t\t\"action\": \"pause\",\n\t})\n\treq, _ := http.NewRequest(\"POST\", url, bytes.NewBuffer(body))\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\t_, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n\n\turl = fmt.Sprintf(\"http://%s/api/topics/%s/ch\", nsqadmin1.RealHTTPAddr(), topicName)\n\tbody, _ = json.Marshal(map[string]interface{}{\n\t\t\"action\": \"unpause\",\n\t})\n\treq, _ = http.NewRequest(\"POST\", url, bytes.NewBuffer(body))\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n}\n\nfunc TestHTTPEmptyTopicPOST(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_empty_topic_post\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqds[0].GetTopic(topicName)\n\ttopic.PutMessage(nsqd.NewMessage(nsqd.MessageID{}, []byte(\"1234\")))\n\ttest.Equal(t, int64(1), topic.Depth())\n\ttime.Sleep(100 * time.Millisecond)\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics/%s\", nsqadmin1.RealHTTPAddr(), topicName)\n\tbody, _ := json.Marshal(map[string]interface{}{\n\t\t\"action\": \"empty\",\n\t})\n\treq, _ := http.NewRequest(\"POST\", url, bytes.NewBuffer(body))\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\t_, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n\n\ttest.Equal(t, int64(0), topic.Depth())\n}\n\nfunc TestHTTPEmptyChannelPOST(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\ttopicName := \"test_empty_channel_post\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqds[0].GetTopic(topicName)\n\tchannel := topic.GetChannel(\"ch\")\n\tchannel.PutMessage(nsqd.NewMessage(nsqd.MessageID{}, []byte(\"1234\")))\n\n\ttime.Sleep(100 * time.Millisecond)\n\ttest.Equal(t, int64(1), channel.Depth())\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/api/topics/%s/ch\", nsqadmin1.RealHTTPAddr(), topicName)\n\tbody, _ := json.Marshal(map[string]interface{}{\n\t\t\"action\": \"empty\",\n\t})\n\treq, _ := http.NewRequest(\"POST\", url, bytes.NewBuffer(body))\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\t_, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tresp.Body.Close()\n\n\ttest.Equal(t, int64(0), channel.Depth())\n}\n\nfunc TestHTTPconfig(t *testing.T) {\n\tdataPath, nsqds, nsqlookupds, nsqadmin1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupds[0].Exit()\n\tdefer nsqadmin1.Exit()\n\n\tlopts := nsqlookupd.NewOptions()\n\tlopts.Logger = test.NewTestLogger(t)\n\n\tlopts1 := *lopts\n\t_, _, lookupd1 := mustStartNSQLookupd(&lopts1)\n\tdefer lookupd1.Exit()\n\tlopts2 := *lopts\n\t_, _, lookupd2 := mustStartNSQLookupd(&lopts2)\n\tdefer lookupd2.Exit()\n\n\turl := fmt.Sprintf(\"http://%s/config/nsqlookupd_http_addresses\", nsqadmin1.RealHTTPAddr())\n\tresp, err := http.Get(url)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\torigaddrs := fmt.Sprintf(`[\"%s\"]`, nsqlookupds[0].RealHTTPAddr().String())\n\ttest.Equal(t, origaddrs, string(body))\n\n\tclient := http.Client{}\n\taddrs := fmt.Sprintf(`[\"%s\",\"%s\"]`, lookupd1.RealHTTPAddr().String(), lookupd2.RealHTTPAddr().String())\n\turl = fmt.Sprintf(\"http://%s/config/nsqlookupd_http_addresses\", nsqadmin1.RealHTTPAddr())\n\treq, err := http.NewRequest(\"PUT\", url, bytes.NewBuffer([]byte(addrs)))\n\ttest.Nil(t, err)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\ttest.Equal(t, addrs, string(body))\n\n\turl = fmt.Sprintf(\"http://%s/config/log_level\", nsqadmin1.RealHTTPAddr())\n\treq, err = http.NewRequest(\"PUT\", url, bytes.NewBuffer([]byte(`fatal`)))\n\ttest.Nil(t, err)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\t_, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\ttest.Equal(t, LOG_FATAL, nsqadmin1.getOpts().LogLevel)\n\n\turl = fmt.Sprintf(\"http://%s/config/log_level\", nsqadmin1.RealHTTPAddr())\n\treq, err = http.NewRequest(\"PUT\", url, bytes.NewBuffer([]byte(`bad`)))\n\ttest.Nil(t, err)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\t_, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 400, resp.StatusCode)\n}\n\nfunc TestHTTPconfigCIDR(t *testing.T) {\n\topts := NewOptions()\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\topts.NSQLookupdHTTPAddresses = []string{\"127.0.0.1:4161\"}\n\topts.Logger = test.NewTestLogger(t)\n\topts.AllowConfigFromCIDR = \"10.0.0.0/8\"\n\tnsqadmin, err := New(opts)\n\ttest.Nil(t, err)\n\tgo func() {\n\t\terr := nsqadmin.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\tdefer nsqadmin.Exit()\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\turl := fmt.Sprintf(\"http://%s/config/nsqlookupd_http_addresses\", nsqadmin.RealHTTPAddr())\n\tresp, err := http.Get(url)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\t_, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 403, resp.StatusCode)\n}\n" }, { "path": "nsqadmin/logger.go", "content": "package nsqadmin\n\nimport (\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype Logger lg.Logger\n\nconst (\n\tLOG_DEBUG = lg.DEBUG\n\tLOG_INFO = lg.INFO\n\tLOG_WARN = lg.WARN\n\tLOG_ERROR = lg.ERROR\n\tLOG_FATAL = lg.FATAL\n)\n\nfunc (n *NSQAdmin) logf(level lg.LogLevel, f string, args ...interface{}) {\n\topts := n.getOpts()\n\tlg.Logf(opts.Logger, opts.LogLevel, level, f, args...)\n}\n" }, { "path": "nsqadmin/notify.go", "content": "package nsqadmin\n\nimport (\n\t\"encoding/base64\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"os\"\n\t\"strings\"\n\t\"time\"\n)\n\ntype AdminAction struct {\n\tAction string `json:\"action\"`\n\tTopic string `json:\"topic\"`\n\tChannel string `json:\"channel,omitempty\"`\n\tNode string `json:\"node,omitempty\"`\n\tTimestamp int64 `json:\"timestamp\"`\n\tUser string `json:\"user,omitempty\"`\n\tRemoteIP string `json:\"remote_ip\"`\n\tUserAgent string `json:\"user_agent\"`\n\tURL string `json:\"url\"` // The URL of the HTTP request that triggered this action\n\tVia string `json:\"via\"` // the Hostname of the nsqadmin performing this action\n}\n\nfunc basicAuthUser(req *http.Request) string {\n\ts := strings.SplitN(req.Header.Get(\"Authorization\"), \" \", 2)\n\tif len(s) != 2 || s[0] != \"Basic\" {\n\t\treturn \"\"\n\t}\n\tb, err := base64.StdEncoding.DecodeString(s[1])\n\tif err != nil {\n\t\treturn \"\"\n\t}\n\tpair := strings.SplitN(string(b), \":\", 2)\n\tif len(pair) != 2 {\n\t\treturn \"\"\n\t}\n\treturn pair[0]\n}\n\nfunc (s *httpServer) notifyAdminAction(action, topic, channel, node string, req *http.Request) {\n\tif s.nsqadmin.getOpts().NotificationHTTPEndpoint == \"\" {\n\t\treturn\n\t}\n\tvia, _ := os.Hostname()\n\n\tu := url.URL{\n\t\tScheme: \"http\",\n\t\tHost: req.Host,\n\t\tPath: req.URL.Path,\n\t\tRawQuery: req.URL.RawQuery,\n\t}\n\tif req.TLS != nil || req.Header.Get(\"X-Scheme\") == \"https\" {\n\t\tu.Scheme = \"https\"\n\t}\n\n\ta := &AdminAction{\n\t\tAction: action,\n\t\tTopic: topic,\n\t\tChannel: channel,\n\t\tNode: node,\n\t\tTimestamp: time.Now().Unix(),\n\t\tUser: basicAuthUser(req),\n\t\tRemoteIP: req.RemoteAddr,\n\t\tUserAgent: req.UserAgent(),\n\t\tURL: u.String(),\n\t\tVia: via,\n\t}\n\t// Perform all work in a new goroutine so this never blocks\n\tgo func() { s.nsqadmin.notifications <- a }()\n}\n" }, { "path": "nsqadmin/nsqadmin.go", "content": "package nsqadmin\n\nimport (\n\t\"bytes\"\n\t\"crypto/tls\"\n\t\"crypto/x509\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"log\"\n\t\"net\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"os\"\n\t\"path\"\n\t\"sync\"\n\t\"sync/atomic\"\n\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/util\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\ntype NSQAdmin struct {\n\tsync.RWMutex\n\topts atomic.Value\n\thttpListener net.Listener\n\twaitGroup util.WaitGroupWrapper\n\tnotifications chan *AdminAction\n\tgraphiteURL *url.URL\n\thttpClientTLSConfig *tls.Config\n}\n\nfunc New(opts *Options) (*NSQAdmin, error) {\n\tif opts.Logger == nil {\n\t\topts.Logger = log.New(os.Stderr, opts.LogPrefix, log.Ldate|log.Ltime|log.Lmicroseconds)\n\t}\n\n\tn := &NSQAdmin{\n\t\tnotifications: make(chan *AdminAction),\n\t}\n\tn.swapOpts(opts)\n\n\tif len(opts.NSQDHTTPAddresses) == 0 && len(opts.NSQLookupdHTTPAddresses) == 0 {\n\t\treturn nil, errors.New(\"--nsqd-http-address or --lookupd-http-address required\")\n\t}\n\n\tif len(opts.NSQDHTTPAddresses) != 0 && len(opts.NSQLookupdHTTPAddresses) != 0 {\n\t\treturn nil, errors.New(\"use --nsqd-http-address or --lookupd-http-address not both\")\n\t}\n\n\tif opts.HTTPClientTLSCert != \"\" && opts.HTTPClientTLSKey == \"\" {\n\t\treturn nil, errors.New(\"--http-client-tls-key must be specified with --http-client-tls-cert\")\n\t}\n\n\tif opts.HTTPClientTLSKey != \"\" && opts.HTTPClientTLSCert == \"\" {\n\t\treturn nil, errors.New(\"--http-client-tls-cert must be specified with --http-client-tls-key\")\n\t}\n\n\tn.httpClientTLSConfig = &tls.Config{\n\t\tInsecureSkipVerify: opts.HTTPClientTLSInsecureSkipVerify,\n\t}\n\tif opts.HTTPClientTLSCert != \"\" && opts.HTTPClientTLSKey != \"\" {\n\t\tcert, err := tls.LoadX509KeyPair(opts.HTTPClientTLSCert, opts.HTTPClientTLSKey)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to LoadX509KeyPair %s, %s - %s\",\n\t\t\t\topts.HTTPClientTLSCert, opts.HTTPClientTLSKey, err)\n\t\t}\n\t\tn.httpClientTLSConfig.Certificates = []tls.Certificate{cert}\n\t}\n\tif opts.HTTPClientTLSRootCAFile != \"\" {\n\t\ttlsCertPool := x509.NewCertPool()\n\t\tcaCertFile, err := os.ReadFile(opts.HTTPClientTLSRootCAFile)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to read TLS root CA file %s - %s\",\n\t\t\t\topts.HTTPClientTLSRootCAFile, err)\n\t\t}\n\t\tif !tlsCertPool.AppendCertsFromPEM(caCertFile) {\n\t\t\treturn nil, fmt.Errorf(\"failed to AppendCertsFromPEM %s\", opts.HTTPClientTLSRootCAFile)\n\t\t}\n\t\tn.httpClientTLSConfig.RootCAs = tlsCertPool\n\t}\n\n\tfor _, address := range opts.NSQLookupdHTTPAddresses {\n\t\t_, err := net.ResolveTCPAddr(\"tcp\", address)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to resolve --lookupd-http-address (%s) - %s\", address, err)\n\t\t}\n\t}\n\n\tfor _, address := range opts.NSQDHTTPAddresses {\n\t\t_, err := net.ResolveTCPAddr(\"tcp\", address)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to resolve --nsqd-http-address (%s) - %s\", address, err)\n\t\t}\n\t}\n\n\tif opts.ProxyGraphite {\n\t\turl, err := url.Parse(opts.GraphiteURL)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to parse --graphite-url (%s) - %s\", opts.GraphiteURL, err)\n\t\t}\n\t\tn.graphiteURL = url\n\t}\n\n\tif opts.AllowConfigFromCIDR != \"\" {\n\t\t_, _, err := net.ParseCIDR(opts.AllowConfigFromCIDR)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"failed to parse --allow-config-from-cidr (%s) - %s\", opts.AllowConfigFromCIDR, err)\n\t\t}\n\t}\n\n\topts.BasePath = normalizeBasePath(opts.BasePath)\n\n\tn.logf(LOG_INFO, version.String(\"nsqadmin\"))\n\n\tvar err error\n\tn.httpListener, err = net.Listen(\"tcp\", n.getOpts().HTTPAddress)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"listen (%s) failed - %s\", n.getOpts().HTTPAddress, err)\n\t}\n\n\treturn n, nil\n}\n\nfunc normalizeBasePath(p string) string {\n\tif len(p) == 0 {\n\t\treturn \"/\"\n\t}\n\t// add leading slash\n\tif p[0] != '/' {\n\t\tp = \"/\" + p\n\t}\n\treturn path.Clean(p)\n}\n\nfunc (n *NSQAdmin) getOpts() *Options {\n\treturn n.opts.Load().(*Options)\n}\n\nfunc (n *NSQAdmin) swapOpts(opts *Options) {\n\tn.opts.Store(opts)\n}\n\nfunc (n *NSQAdmin) RealHTTPAddr() *net.TCPAddr {\n\treturn n.httpListener.Addr().(*net.TCPAddr)\n}\n\nfunc (n *NSQAdmin) handleAdminActions() {\n\tfor action := range n.notifications {\n\t\tcontent, err := json.Marshal(action)\n\t\tif err != nil {\n\t\t\tn.logf(LOG_ERROR, \"failed to serialize admin action - %s\", err)\n\t\t}\n\t\thttpclient := &http.Client{\n\t\t\tTransport: http_api.NewDeadlineTransport(n.getOpts().HTTPClientConnectTimeout, n.getOpts().HTTPClientRequestTimeout),\n\t\t}\n\t\tn.logf(LOG_INFO, \"POSTing notification to %s\", n.getOpts().NotificationHTTPEndpoint)\n\t\tresp, err := httpclient.Post(n.getOpts().NotificationHTTPEndpoint,\n\t\t\t\"application/json\", bytes.NewBuffer(content))\n\t\tif err != nil {\n\t\t\tn.logf(LOG_ERROR, \"failed to POST notification - %s\", err)\n\t\t}\n\t\tresp.Body.Close()\n\t}\n}\n\nfunc (n *NSQAdmin) Main() error {\n\texitCh := make(chan error)\n\tvar once sync.Once\n\texitFunc := func(err error) {\n\t\tonce.Do(func() {\n\t\t\tif err != nil {\n\t\t\t\tn.logf(LOG_FATAL, \"%s\", err)\n\t\t\t}\n\t\t\texitCh <- err\n\t\t})\n\t}\n\n\thttpServer := NewHTTPServer(n)\n\tn.waitGroup.Wrap(func() {\n\t\texitFunc(http_api.Serve(n.httpListener, http_api.CompressHandler(httpServer), \"HTTP\", n.logf))\n\t})\n\tn.waitGroup.Wrap(n.handleAdminActions)\n\n\terr := <-exitCh\n\treturn err\n}\n\nfunc (n *NSQAdmin) Exit() {\n\tif n.httpListener != nil {\n\t\tn.httpListener.Close()\n\t}\n\tclose(n.notifications)\n\tn.waitGroup.Wait()\n}\n" }, { "path": "nsqadmin/nsqadmin_test.go", "content": "package nsqadmin\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"os\"\n\t\"testing\"\n\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/test\"\n\t\"github.com/nsqio/nsq/nsqd\"\n)\n\nfunc TestNeitherNSQDAndNSQLookup(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = lg.NilLogger{}\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\t_, err := New(opts)\n\ttest.NotNil(t, err)\n\ttest.Equal(t, \"--nsqd-http-address or --lookupd-http-address required\", fmt.Sprintf(\"%s\", err))\n}\n\nfunc TestBothNSQDAndNSQLookup(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = lg.NilLogger{}\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\topts.NSQLookupdHTTPAddresses = []string{\"127.0.0.1:4161\"}\n\topts.NSQDHTTPAddresses = []string{\"127.0.0.1:4151\"}\n\t_, err := New(opts)\n\ttest.NotNil(t, err)\n\ttest.Equal(t, \"use --nsqd-http-address or --lookupd-http-address not both\", fmt.Sprintf(\"%s\", err))\n}\n\nfunc TestTLSHTTPClient(t *testing.T) {\n\tlgr := test.NewTestLogger(t)\n\n\tnsqdOpts := nsqd.NewOptions()\n\tnsqdOpts.TLSCert = \"./test/server.pem\"\n\tnsqdOpts.TLSKey = \"./test/server.key\"\n\tnsqdOpts.TLSRootCAFile = \"./test/ca.pem\"\n\tnsqdOpts.TLSClientAuthPolicy = \"require-verify\"\n\tnsqdOpts.Logger = lgr\n\t_, nsqdHTTPAddr, nsqd := mustStartNSQD(nsqdOpts)\n\tdefer os.RemoveAll(nsqdOpts.DataPath)\n\tdefer nsqd.Exit()\n\n\topts := NewOptions()\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\topts.NSQDHTTPAddresses = []string{nsqdHTTPAddr.String()}\n\topts.HTTPClientTLSRootCAFile = \"./test/ca.pem\"\n\topts.HTTPClientTLSCert = \"./test/client.pem\"\n\topts.HTTPClientTLSKey = \"./test/client.key\"\n\topts.Logger = lgr\n\tnsqadmin, err := New(opts)\n\ttest.Nil(t, err)\n\tgo func() {\n\t\terr := nsqadmin.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\tdefer nsqadmin.Exit()\n\n\thttpAddr := nsqadmin.RealHTTPAddr()\n\tu := url.URL{\n\t\tScheme: \"http\",\n\t\tHost: httpAddr.String(),\n\t\tPath: \"/api/nodes/\" + nsqdHTTPAddr.String(),\n\t}\n\n\tresp, err := http.Get(u.String())\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\n\ttest.Equal(t, resp.StatusCode < 500, true)\n}\n\nfunc mustStartNSQD(opts *nsqd.Options) (net.Addr, net.Addr, *nsqd.NSQD) {\n\topts.TCPAddress = \"127.0.0.1:0\"\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\topts.HTTPSAddress = \"127.0.0.1:0\"\n\tif opts.DataPath == \"\" {\n\t\ttmpDir, err := os.MkdirTemp(\"\", \"nsq-test-\")\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t\topts.DataPath = tmpDir\n\t}\n\tnsqd, err := nsqd.New(opts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := nsqd.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\treturn nsqd.RealTCPAddr(), nsqd.RealHTTPAddr(), nsqd\n}\n" }, { "path": "nsqadmin/options.go", "content": "package nsqadmin\n\nimport (\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype Options struct {\n\tLogLevel lg.LogLevel `flag:\"log-level\"`\n\tLogPrefix string `flag:\"log-prefix\"`\n\tLogger Logger\n\n\tHTTPAddress string `flag:\"http-address\"`\n\tBasePath string `flag:\"base-path\"`\n\n\tDevStaticDir string `flag:\"dev-static-dir\"`\n\n\tGraphiteURL string `flag:\"graphite-url\"`\n\tProxyGraphite bool `flag:\"proxy-graphite\"`\n\n\tStatsdPrefix string `flag:\"statsd-prefix\"`\n\tStatsdCounterFormat string `flag:\"statsd-counter-format\"`\n\tStatsdGaugeFormat string `flag:\"statsd-gauge-format\"`\n\n\tStatsdInterval time.Duration `flag:\"statsd-interval\"`\n\n\tNSQLookupdHTTPAddresses []string `flag:\"lookupd-http-address\" cfg:\"nsqlookupd_http_addresses\"`\n\tNSQDHTTPAddresses []string `flag:\"nsqd-http-address\" cfg:\"nsqd_http_addresses\"`\n\n\tHTTPClientConnectTimeout time.Duration `flag:\"http-client-connect-timeout\"`\n\tHTTPClientRequestTimeout time.Duration `flag:\"http-client-request-timeout\"`\n\n\tHTTPClientTLSInsecureSkipVerify bool `flag:\"http-client-tls-insecure-skip-verify\"`\n\tHTTPClientTLSRootCAFile string `flag:\"http-client-tls-root-ca-file\"`\n\tHTTPClientTLSCert string `flag:\"http-client-tls-cert\"`\n\tHTTPClientTLSKey string `flag:\"http-client-tls-key\"`\n\n\tAllowConfigFromCIDR string `flag:\"allow-config-from-cidr\"`\n\n\tNotificationHTTPEndpoint string `flag:\"notification-http-endpoint\"`\n\n\tACLHTTPHeader string `flag:\"acl-http-header\"`\n\tAdminUsers []string `flag:\"admin-user\" cfg:\"admin_users\"`\n}\n\nfunc NewOptions() *Options {\n\treturn &Options{\n\t\tLogPrefix: \"[nsqadmin] \",\n\t\tLogLevel: lg.INFO,\n\t\tHTTPAddress: \"0.0.0.0:4171\",\n\t\tBasePath: \"/\",\n\t\tStatsdPrefix: \"nsq.%s\",\n\t\tStatsdCounterFormat: \"stats.counters.%s.count\",\n\t\tStatsdGaugeFormat: \"stats.gauges.%s\",\n\t\tStatsdInterval: 60 * time.Second,\n\t\tHTTPClientConnectTimeout: 2 * time.Second,\n\t\tHTTPClientRequestTimeout: 5 * time.Second,\n\t\tAllowConfigFromCIDR: \"127.0.0.1/8\",\n\t\tACLHTTPHeader: \"X-Forwarded-User\",\n\t\tAdminUsers: []string{},\n\t}\n}\n" }, { "path": "nsqadmin/package.json", "content": "{\n \"name\": \"nsqadmin\",\n \"version\": \"0.3.0\",\n \"description\": \"operational dashboard for NSQ (https://nsq.io/)\",\n \"repository\": {\n \"type\": \"git\",\n \"url\": \"\"\n },\n \"scripts\": {\n \"lint\": \"eslint static/js\"\n },\n \"devDependencies\": {\n \"browserify\": \"^17.0.0\",\n \"eslint\": \"^8.52.0\",\n \"gulp\": \"^4.0.2\",\n \"gulp-clean\": \"^0.4.0\",\n \"gulp-cli\": \"^2.3.0\",\n \"gulp-dart-sass\": \"^1.0.2\",\n \"gulp-notify\": \"^4.0.0\",\n \"gulp-sourcemaps\": \"^3.0.0\",\n \"gulp-task-listing\": \"^1.1.0\",\n \"gulp-uglify\": \"^3.0.2\",\n \"handlebars\": \"^4.7.7\",\n \"hbsfy\": \"^2.8.1\",\n \"vinyl-buffer\": \"^1.0.1\",\n \"vinyl-source-stream\": \"^2.0.0\"\n },\n \"dependencies\": {\n \"backbone\": \"^1.4.0\",\n \"bootbox\": \"^5.5.2\",\n \"bootstrap\": \"^3.4.1\",\n \"jquery\": \"^3.6.0\",\n \"moment\": \"^2.29.1\",\n \"underscore\": \"^1.13.1\"\n },\n \"browserify\": {\n \"transform\": [\n \"hbsfy\"\n ]\n }\n}\n" }, { "path": "nsqadmin/static/build/base.css", "content": ".red {\n color: #c30;\n}\n\n.red:hover {\n color: #f30;\n}\n\n.bold {\n font-weight: bold;\n}\n\n.graph-row td {\n text-align: center;\n}\n\n.image-preview {\n display: none;\n position: absolute;\n z-index: 100;\n height: 240px;\n width: 480px;\n}\n\n.white {\n color: #fff;\n}\n\n.bubblingG {\n text-align: center;\n width: 125px;\n height: 78px;\n}\n\n.bubblingG span {\n display: inline-block;\n vertical-align: middle;\n width: 16px;\n height: 16px;\n margin: 39px auto;\n background: #006FC4;\n -moz-border-radius: 79px;\n -moz-animation: bubblingG 0.9s infinite alternate;\n -webkit-border-radius: 79px;\n -webkit-animation: bubblingG 0.9s infinite alternate;\n -ms-border-radius: 79px;\n -ms-animation: bubblingG 0.9s infinite alternate;\n -o-border-radius: 79px;\n -o-animation: bubblingG 0.9s infinite alternate;\n border-radius: 79px;\n animation: bubblingG 0.9s infinite alternate;\n}\n\n#bubblingG_1 {\n -moz-animation-delay: 0s;\n -webkit-animation-delay: 0s;\n -ms-animation-delay: 0s;\n -o-animation-delay: 0s;\n animation-delay: 0s;\n}\n\n#bubblingG_2 {\n -moz-animation-delay: 0.27s;\n -webkit-animation-delay: 0.27s;\n -ms-animation-delay: 0.27s;\n -o-animation-delay: 0.27s;\n animation-delay: 0.27s;\n}\n\n#bubblingG_3 {\n -moz-animation-delay: 0.54s;\n -webkit-animation-delay: 0.54s;\n -ms-animation-delay: 0.54s;\n -o-animation-delay: 0.54s;\n animation-delay: 0.54s;\n}\n\n@-moz-keyframes bubblingG {\n 0% {\n width: 16px;\n height: 16px;\n background-color: #006FC4;\n -moz-transform: translateY(0);\n }\n 100% {\n width: 38px;\n height: 38px;\n background-color: #FFFFFF;\n -moz-transform: translateY(-33px);\n }\n}\n@-webkit-keyframes bubblingG {\n 0% {\n width: 16px;\n height: 16px;\n background-color: #006FC4;\n -webkit-transform: translateY(0);\n }\n 100% {\n width: 38px;\n height: 38px;\n background-color: #FFFFFF;\n -webkit-transform: translateY(-33px);\n }\n}\n@-ms-keyframes bubblingG {\n 0% {\n width: 16px;\n height: 16px;\n background-color: #006FC4;\n -ms-transform: translateY(0);\n }\n 100% {\n width: 38px;\n height: 38px;\n background-color: #FFFFFF;\n -ms-transform: translateY(-33px);\n }\n}\n@-o-keyframes bubblingG {\n 0% {\n width: 16px;\n height: 16px;\n background-color: #006FC4;\n -o-transform: translateY(0);\n }\n 100% {\n width: 38px;\n height: 38px;\n background-color: #FFFFFF;\n -o-transform: translateY(-33px);\n }\n}\n@keyframes bubblingG {\n 0% {\n width: 16px;\n height: 16px;\n background-color: #006FC4;\n transform: translateY(0);\n }\n 100% {\n width: 38px;\n height: 38px;\n background-color: #FFFFFF;\n transform: translateY(-33px);\n }\n}\n.bubblingG {\n position: absolute;\n left: 0;\n right: 0;\n top: 0;\n bottom: 0;\n margin: auto;\n max-width: 100%;\n max-height: 100%;\n overflow: auto;\n}\n\n.navbar-brand > img {\n width: 30px;\n height: 30px;\n margin-right: 5px;\n margin-top: -5px;\n display: inline;\n}\n\n.bg-zone-local {\n background-color: #ddffdd;\n}\n\n.bg-region-local {\n background-color: #fefec2;\n}\n\n.bg-global {\n background-color: white;\n}\n\n.popup {\n position: relative;\n display: inline-block;\n cursor: pointer;\n}\n\n/* The actual popup (appears on top) */\n.popup .popuptext {\n visibility: hidden;\n width: 180px;\n height: 27px;\n background-color: white;\n color: #4b4b4b;\n text-align: center;\n border-radius: 6px;\n border-right: 1px solid #777;\n border-left: 1px solid #777;\n border-top: 1px solid #777;\n border-bottom: 1px solid #777;\n padding: 2px 0;\n position: absolute;\n z-index: 1;\n top: 60%;\n left: 50%;\n margin-left: -85%;\n}\n\n/* Popup arrow */\n.popup .popuptext::after {\n content: \"\";\n position: absolute;\n top: -5px;\n left: 50%;\n margin-left: -5px;\n width: 0;\n height: 0;\n border-left: 5px solid transparent;\n border-right: 5px solid transparent;\n border-bottom: 5px solid #777;\n}\n\n/* Toggle this class when clicking on the popup container (hide and show the popup) */\n.popup .show {\n visibility: visible;\n -webkit-animation: fadeIn 1s;\n animation: fadeIn 1s;\n}\n\n/* Add animation (fade in the popup) */\n@-webkit-keyframes fadeIn {\n from {\n opacity: 0;\n }\n to {\n opacity: 1;\n }\n}\n@keyframes fadeIn {\n from {\n opacity: 0;\n }\n to {\n opacity: 1;\n }\n}" }, { "path": "nsqadmin/static/build/index.html", "content": "\n\n\n nsqadmin\n \n \n \n \n \n\n\n \n\n
\n\n \n \n \n\n\n" }, { "path": "nsqadmin/static/build/main.js", "content": "!function l(a,o,r){function i(e,n){if(!o[e]){if(!a[e]){var t=\"function\"==typeof require&&require;if(!n&&t)return t(e,!0);if(s)return s(e,!0);throw(t=new Error(\"Cannot find module '\"+e+\"'\")).code=\"MODULE_NOT_FOUND\",t}t=o[e]={exports:{}},a[e][0].call(t.exports,function(n){return i(a[e][1][n]||n)},t,t.exports,l,a,o,r)}return o[e].exports}for(var s=\"function\"==typeof require&&require,n=0;nthis.$items.length-1||n<0))return this.sliding?this.$element.one(\"slid.bs.carousel\",function(){e.to(n)}):t==n?this.pause().cycle():this.slide(tdocument.documentElement.clientHeight;this.$element.css({paddingLeft:!this.bodyIsOverflowing&&n?this.scrollbarWidth:\"\",paddingRight:this.bodyIsOverflowing&&!n?this.scrollbarWidth:\"\"})},r.prototype.resetAdjustments=function(){this.$element.css({paddingLeft:\"\",paddingRight:\"\"})},r.prototype.checkScrollbar=function(){var n,e=window.innerWidth;e||(e=(n=document.documentElement.getBoundingClientRect()).right-Math.abs(n.left)),this.bodyIsOverflowing=document.body.clientWidth

'}),((o.prototype=a.extend({},a.fn.tooltip.Constructor.prototype)).constructor=o).prototype.getDefaults=function(){return o.DEFAULTS},o.prototype.setContent=function(){var n,e=this.tip(),t=this.getTitle(),l=this.getContent();this.options.html?(n=typeof l,this.options.sanitize&&(t=this.sanitizeHtml(t),\"string\"==n&&(l=this.sanitizeHtml(l))),e.find(\".popover-title\").html(t),e.find(\".popover-content\").children().detach().end()[\"string\"==n?\"html\":\"append\"](l)):(e.find(\".popover-title\").text(t),e.find(\".popover-content\").children().detach().end().text(l)),e.removeClass(\"fade top bottom left right in\"),e.find(\".popover-title\").html()||e.find(\".popover-title\").hide()},o.prototype.hasContent=function(){return this.getTitle()||this.getContent()},o.prototype.getContent=function(){var n=this.$element,e=this.options;return n.attr(\"data-content\")||(\"function\"==typeof e.content?e.content.call(n[0]):e.content)},o.prototype.arrow=function(){return this.$arrow=this.$arrow||this.tip().find(\".arrow\")};var n=a.fn.popover;a.fn.popover=function(l){return this.each(function(){var n=a(this),e=n.data(\"bs.popover\"),t=\"object\"==typeof l&&l;!e&&/destroy|hide/.test(l)||(e||n.data(\"bs.popover\",e=new o(this,t)),\"string\"==typeof l&&e[l]())})},a.fn.popover.Constructor=o,a.fn.popover.noConflict=function(){return a.fn.popover=n,this}}(jQuery)},{}],10:[function(n,e,t){!function(a){\"use strict\";function l(n,e){this.$body=a(document.body),this.$scrollElement=a(n).is(document.body)?a(window):a(n),this.options=a.extend({},l.DEFAULTS,e),this.selector=(this.options.target||\"\")+\" .nav li > a\",this.offsets=[],this.targets=[],this.activeTarget=null,this.scrollHeight=0,this.$scrollElement.on(\"scroll.bs.scrollspy\",a.proxy(this.process,this)),this.refresh(),this.process()}function e(t){return this.each(function(){var n=a(this),e=n.data(\"bs.scrollspy\");e||n.data(\"bs.scrollspy\",e=new l(this,\"object\"==typeof t&&t)),\"string\"==typeof t&&e[t]()})}l.VERSION=\"3.4.1\",l.DEFAULTS={offset:10},l.prototype.getScrollHeight=function(){return this.$scrollElement[0].scrollHeight||Math.max(this.$body[0].scrollHeight,document.documentElement.scrollHeight)},l.prototype.refresh=function(){var n=this,t=\"offset\",l=0;this.offsets=[],this.targets=[],this.scrollHeight=this.getScrollHeight(),a.isWindow(this.$scrollElement[0])||(t=\"position\",l=this.$scrollElement.scrollTop()),this.$body.find(this.selector).map(function(){var n=a(this),e=n.data(\"target\")||n.attr(\"href\"),n=/^#./.test(e)&&a(e);return n&&n.length&&n.is(\":visible\")?[[n[t]().top+l,e]]:null}).sort(function(n,e){return n[0]-e[0]}).each(function(){n.offsets.push(this[0]),n.targets.push(this[1])})},l.prototype.process=function(){var n,e=this.$scrollElement.scrollTop()+this.options.offset,t=this.getScrollHeight(),l=this.options.offset+t-this.$scrollElement.height(),a=this.offsets,o=this.targets,r=this.activeTarget;if(this.scrollHeight!=t&&this.refresh(),l<=e)return r!=(n=o[o.length-1])&&this.activate(n);if(r&&e=a[n]&&(void 0===a[n+1]||e .active\"),a=t&&r.support.transition&&(l.length&&l.hasClass(\"fade\")||!!e.find(\"> .fade\").length);function o(){l.removeClass(\"active\").find(\"> .dropdown-menu > .active\").removeClass(\"active\").end().find('[data-toggle=\"tab\"]').attr(\"aria-expanded\",!1),n.addClass(\"active\").find('[data-toggle=\"tab\"]').attr(\"aria-expanded\",!0),a?(n[0].offsetWidth,n.addClass(\"in\")):n.removeClass(\"fade\"),n.parent(\".dropdown-menu\").length&&n.closest(\"li.dropdown\").addClass(\"active\").end().find('[data-toggle=\"tab\"]').attr(\"aria-expanded\",!0),t&&t()}l.length&&a?l.one(\"bsTransitionEnd\",o).emulateTransitionEnd(i.TRANSITION_DURATION):o(),l.removeClass(\"in\")};var n=r.fn.tab;r.fn.tab=e,r.fn.tab.Constructor=i,r.fn.tab.noConflict=function(){return r.fn.tab=n,this};function t(n){n.preventDefault(),e.call(r(this),\"show\")}r(document).on(\"click.bs.tab.data-api\",'[data-toggle=\"tab\"]',t).on(\"click.bs.tab.data-api\",'[data-toggle=\"pill\"]',t)}(jQuery)},{}],12:[function(n,e,t){!function(d){\"use strict\";var l=[\"sanitize\",\"whiteList\",\"sanitizeFn\"],m=[\"background\",\"cite\",\"href\",\"itemtype\",\"longdesc\",\"poster\",\"src\",\"xlink:href\"],n={\"*\":[\"class\",\"dir\",\"id\",\"lang\",\"role\",/^aria-[\\w-]*$/i],a:[\"target\",\"href\",\"title\",\"rel\"],area:[],b:[],br:[],col:[],code:[],div:[],em:[],hr:[],h1:[],h2:[],h3:[],h4:[],h5:[],h6:[],i:[],img:[\"src\",\"alt\",\"title\",\"width\",\"height\"],li:[],ol:[],p:[],pre:[],s:[],small:[],span:[],sub:[],sup:[],strong:[],u:[],ul:[]},f=/^(?:(?:https?|mailto|ftp|tel|file):|[^&:/?#]*(?:[/?#]|$))/gi,g=/^data:(?:image\\/(?:bmp|gif|jpeg|jpg|png|tiff|webp)|video\\/(?:mpeg|mp4|ogg|webm)|audio\\/(?:mp3|oga|ogg|opus));base64,[a-z0-9+/]+=*$/i;function a(n,e,t){if(0===n.length)return n;if(t&&\"function\"==typeof t)return t(n);if(!document.implementation||!document.implementation.createHTMLDocument)return n;t=document.implementation.createHTMLDocument(\"sanitization\");t.body.innerHTML=n;for(var l=d.map(e,function(n,e){return e}),a=d(t.body).find(\"*\"),o=0,r=a.length;o
',trigger:\"hover focus\",title:\"\",delay:0,html:!1,container:!1,viewport:{selector:\"body\",padding:0},sanitize:!0,sanitizeFn:null,whiteList:n},s.prototype.init=function(n,e,t){if(this.enabled=!0,this.type=n,this.$element=d(e),this.options=this.getOptions(t),this.$viewport=this.options.viewport&&d(document).find(d.isFunction(this.options.viewport)?this.options.viewport.call(this,this.$element):this.options.viewport.selector||this.options.viewport),this.inState={click:!1,hover:!1,focus:!1},this.$element[0]instanceof document.constructor&&!this.options.selector)throw new Error(\"`selector` option must be specified when initializing \"+this.type+\" on the window.document object!\");for(var l=this.options.trigger.split(\" \"),a=l.length;a--;){var o,r=l[a];\"click\"==r?this.$element.on(\"click.\"+this.type,this.options.selector,d.proxy(this.toggle,this)):\"manual\"!=r&&(o=\"hover\"==r?\"mouseleave\":\"focusout\",this.$element.on((\"hover\"==r?\"mouseenter\":\"focusin\")+\".\"+this.type,this.options.selector,d.proxy(this.enter,this)),this.$element.on(o+\".\"+this.type,this.options.selector,d.proxy(this.leave,this)))}this.options.selector?this._options=d.extend({},this.options,{trigger:\"manual\",selector:\"\"}):this.fixTitle()},s.prototype.getDefaults=function(){return s.DEFAULTS},s.prototype.getOptions=function(n){var e,t=this.$element.data();for(e in t)t.hasOwnProperty(e)&&-1!==d.inArray(e,l)&&delete t[e];return(n=d.extend({},this.getDefaults(),t,n)).delay&&\"number\"==typeof n.delay&&(n.delay={show:n.delay,hide:n.delay}),n.sanitize&&(n.template=a(n.template,n.whiteList,n.sanitizeFn)),n},s.prototype.getDelegateOptions=function(){var t={},l=this.getDefaults();return this._options&&d.each(this._options,function(n,e){l[n]!=e&&(t[n]=e)}),t},s.prototype.enter=function(n){var e=n instanceof this.constructor?n:d(n.currentTarget).data(\"bs.\"+this.type);if(e||(e=new this.constructor(n.currentTarget,this.getDelegateOptions()),d(n.currentTarget).data(\"bs.\"+this.type,e)),n instanceof d.Event&&(e.inState[\"focusin\"==n.type?\"focus\":\"hover\"]=!0),e.tip().hasClass(\"in\")||\"in\"==e.hoverState)e.hoverState=\"in\";else{if(clearTimeout(e.timeout),e.hoverState=\"in\",!e.options.delay||!e.options.delay.show)return e.show();e.timeout=setTimeout(function(){\"in\"==e.hoverState&&e.show()},e.options.delay.show)}},s.prototype.isInStateTrue=function(){for(var n in this.inState)if(this.inState[n])return!0;return!1},s.prototype.leave=function(n){var e=n instanceof this.constructor?n:d(n.currentTarget).data(\"bs.\"+this.type);if(e||(e=new this.constructor(n.currentTarget,this.getDelegateOptions()),d(n.currentTarget).data(\"bs.\"+this.type,e)),n instanceof d.Event&&(e.inState[\"focusout\"==n.type?\"focus\":\"hover\"]=!1),!e.isInStateTrue()){if(clearTimeout(e.timeout),e.hoverState=\"out\",!e.options.delay||!e.options.delay.hide)return e.hide();e.timeout=setTimeout(function(){\"out\"==e.hoverState&&e.hide()},e.options.delay.hide)}},s.prototype.show=function(){var e,n,t,l,a,o,r,i=d.Event(\"show.bs.\"+this.type);this.hasContent()&&this.enabled&&(this.$element.trigger(i),t=d.contains(this.$element[0].ownerDocument.documentElement,this.$element[0]),!i.isDefaultPrevented()&&t&&(n=(e=this).tip(),o=this.getUID(this.type),this.setContent(),n.attr(\"id\",o),this.$element.attr(\"aria-describedby\",o),this.options.animation&&n.addClass(\"fade\"),r=\"function\"==typeof this.options.placement?this.options.placement.call(this,n[0],this.$element[0]):this.options.placement,(a=(l=/\\s?auto?\\s?/i).test(r))&&(r=r.replace(l,\"\")||\"top\"),n.detach().css({top:0,left:0,display:\"block\"}).addClass(r).data(\"bs.\"+this.type,this),this.options.container?n.appendTo(d(document).find(this.options.container)):n.insertAfter(this.$element),this.$element.trigger(\"inserted.bs.\"+this.type),i=this.getPosition(),t=n[0].offsetWidth,o=n[0].offsetHeight,a&&(l=r,a=this.getPosition(this.$viewport),r=\"bottom\"==r&&i.bottom+o>a.bottom?\"top\":\"top\"==r&&i.top-oa.width?\"left\":\"left\"==r&&i.left-ti.top+i.height&&(a.top=i.top+i.height-o)):(o=e.left-r,t=e.left+r+t,oi.right&&(a.left=i.left+i.width-t)),a},s.prototype.getTitle=function(){var n=this.$element,e=this.options;return n.attr(\"data-original-title\")||(\"function\"==typeof e.title?e.title.call(n[0]):e.title)},s.prototype.getUID=function(n){for(;n+=~~(1e6*Math.random()),document.getElementById(n););return n},s.prototype.tip=function(){if(!this.$tip&&(this.$tip=d(this.options.template),1!=this.$tip.length))throw new Error(this.type+\" `template` option must consist of exactly 1 top-level element!\");return this.$tip},s.prototype.arrow=function(){return this.$arrow=this.$arrow||this.tip().find(\".tooltip-arrow\")},s.prototype.enable=function(){this.enabled=!0},s.prototype.disable=function(){this.enabled=!1},s.prototype.toggleEnabled=function(){this.enabled=!this.enabled},s.prototype.toggle=function(n){var e=this;n&&((e=d(n.currentTarget).data(\"bs.\"+this.type))||(e=new this.constructor(n.currentTarget,this.getDelegateOptions()),d(n.currentTarget).data(\"bs.\"+this.type,e))),n?(e.inState.click=!e.inState.click,e.isInStateTrue()?e.enter(e):e.leave(e)):e.tip().hasClass(\"in\")?e.leave(e):e.enter(e)},s.prototype.destroy=function(){var n=this;clearTimeout(this.timeout),this.hide(function(){n.$element.off(\".\"+n.type).removeData(\"bs.\"+n.type),n.$tip&&n.$tip.detach(),n.$tip=null,n.$arrow=null,n.$viewport=null,n.$element=null})},s.prototype.sanitizeHtml=function(n){return a(n,this.options.whiteList,this.options.sanitizeFn)};var e=d.fn.tooltip;d.fn.tooltip=function(l){return this.each(function(){var n=d(this),e=n.data(\"bs.tooltip\"),t=\"object\"==typeof l&&l;!e&&/destroy|hide/.test(l)||(e||n.data(\"bs.tooltip\",e=new s(this,t)),\"string\"==typeof l&&e[l]())})},d.fn.tooltip.Constructor=s,d.fn.tooltip.noConflict=function(){return d.fn.tooltip=e,this}}(jQuery)},{}],13:[function(n,e,t){!function(l){\"use strict\";l.fn.emulateTransitionEnd=function(n){var e=!1,t=this;l(this).one(\"bsTransitionEnd\",function(){e=!0});return setTimeout(function(){e||l(t).trigger(l.support.transition.end)},n),this},l(function(){l.support.transition=function(){var n,e=document.createElement(\"bootstrap\"),t={WebkitTransition:\"webkitTransitionEnd\",MozTransition:\"transitionend\",OTransition:\"oTransitionEnd otransitionend\",transition:\"transitionend\"};for(n in t)if(void 0!==e.style[n])return{end:t[n]};return!1}(),l.support.transition&&(l.event.special.bsTransitionEnd={bindType:l.support.transition.end,delegateType:l.support.transition.end,handle:function(n){if(l(n.target).is(this))return n.handleObj.handler.apply(this,arguments)}})})}(jQuery)},{}],14:[function(n,e,t){\"use strict\";function l(n){return n&&n.__esModule?n:{default:n}}function a(n){if(n&&n.__esModule)return n;var e={};if(null!=n)for(var t in n)Object.prototype.hasOwnProperty.call(n,t)&&(e[t]=n[t]);return e.default=n,e}t.__esModule=!0;var o=a(n(\"./handlebars/base\")),r=l(n(\"./handlebars/safe-string\")),i=l(n(\"./handlebars/exception\")),s=a(n(\"./handlebars/utils\")),c=a(n(\"./handlebars/runtime\")),u=l(n(\"./handlebars/no-conflict\"));function h(){var e=new o.HandlebarsEnvironment;return s.extend(e,o),e.SafeString=r.default,e.Exception=i.default,e.Utils=s,e.escapeExpression=s.escapeExpression,e.VM=c,e.template=function(n){return c.template(n,e)},e}n=h();n.create=h,u.default(n),n.default=n,t.default=n,e.exports=t.default},{\"./handlebars/base\":15,\"./handlebars/exception\":18,\"./handlebars/no-conflict\":31,\"./handlebars/runtime\":32,\"./handlebars/safe-string\":33,\"./handlebars/utils\":34}],15:[function(n,e,t){\"use strict\";function l(n){return n&&n.__esModule?n:{default:n}}t.__esModule=!0,t.HandlebarsEnvironment=h;var a=n(\"./utils\"),o=l(n(\"./exception\")),r=n(\"./helpers\"),i=n(\"./decorators\"),s=l(n(\"./logger\")),c=n(\"./internal/proto-access\");t.VERSION=\"4.7.7\";t.COMPILER_REVISION=8;t.LAST_COMPATIBLE_COMPILER_REVISION=7;t.REVISION_CHANGES={1:\"<= 1.0.rc.2\",2:\"== 1.0.0-rc.3\",3:\"== 1.0.0-rc.4\",4:\"== 1.x.x\",5:\"== 2.0.0-alpha.x\",6:\">= 2.0.0-beta.1\",7:\">= 4.0.0 <4.3.0\",8:\">= 4.3.0\"};var u=\"[object Object]\";function h(n,e,t){this.helpers=n||{},this.partials=e||{},this.decorators=t||{},r.registerDefaultHelpers(this),i.registerDefaultDecorators(this)}h.prototype={constructor:h,logger:s.default,log:s.default.log,registerHelper:function(n,e){if(a.toString.call(n)===u){if(e)throw new o.default(\"Arg not supported with multiple helpers\");a.extend(this.helpers,n)}else this.helpers[n]=e},unregisterHelper:function(n){delete this.helpers[n]},registerPartial:function(n,e){if(a.toString.call(n)===u)a.extend(this.partials,n);else{if(void 0===e)throw new o.default('Attempting to register a partial called \"'+n+'\" as undefined');this.partials[n]=e}},unregisterPartial:function(n){delete this.partials[n]},registerDecorator:function(n,e){if(a.toString.call(n)===u){if(e)throw new o.default(\"Arg not supported with multiple decorators\");a.extend(this.decorators,n)}else this.decorators[n]=e},unregisterDecorator:function(n){delete this.decorators[n]},resetLoggedPropertyAccesses:function(){c.resetLoggedProperties()}};n=s.default.log;t.log=n,t.createFrame=a.createFrame,t.logger=s.default},{\"./decorators\":16,\"./exception\":18,\"./helpers\":19,\"./internal/proto-access\":28,\"./logger\":30,\"./utils\":34}],16:[function(n,e,t){\"use strict\";t.__esModule=!0,t.registerDefaultDecorators=function(n){o.default(n)};var l,a=n(\"./decorators/inline\"),o=(l=a)&&l.__esModule?l:{default:l}},{\"./decorators/inline\":17}],17:[function(n,e,t){\"use strict\";t.__esModule=!0;var r=n(\"../utils\");t.default=function(n){n.registerDecorator(\"inline\",function(l,a,o,n){var e=l;return a.partials||(a.partials={},e=function(n,e){var t=o.partials;o.partials=r.extend({},t,a.partials);e=l(n,e);return o.partials=t,e}),a.partials[n.args[0]]=n.fn,e})},e.exports=t.default},{\"../utils\":34}],18:[function(n,e,t){\"use strict\";t.__esModule=!0;var s=[\"description\",\"fileName\",\"lineNumber\",\"endLineNumber\",\"message\",\"name\",\"number\",\"stack\"];function c(n,e){var t=e&&e.loc,l=void 0,a=void 0,o=void 0,e=void 0;t&&(l=t.start.line,a=t.end.line,o=t.start.column,e=t.end.column,n+=\" - \"+l+\":\"+o);for(var r=Error.prototype.constructor.call(this,n),i=0;i=p.LAST_COMPATIBLE_COMPILER_REVISION&&e<=p.COMPILER_REVISION)){if(e\":\">\",'\"':\""\",\"'\":\"'\",\"`\":\"`\",\"=\":\"=\"},a=/[&<>\"'`=]/g,o=/[&<>\"'`=]/;function r(n){return l[n]}function i(n){for(var e=1;etombstone \"+e+\"?\",function(n){!0===n&&new b({name:e}).tombstoneTopic(t).done(function(){window.location.reload(!0)}).fail(this.handleAJAXError.bind(this))}.bind(this))}});e.exports=n},{\"../app_state\":36,\"../lib/pubsub\":41,\"../models/channel\":43,\"../models/node\":44,\"../models/topic\":45,\"../router\":46,\"./base\":48,\"./channel\":50,\"./counter\":52,\"./header\":55,\"./lookup\":57,\"./node\":59,\"./nodes\":61,\"./topic\":64,\"./topics\":66,bootbox:void 0,bootstrap:1,jquery:void 0}],48:[function(n,e,t){var l=n(\"jquery\"),a=n(\"underscore\"),o=n(\"backbone\"),r=n(\"../app_state\"),i=n(\"./error.hbs\"),n=o.View.extend({constructor:function(n){return this.options=n||{},o.View.prototype.constructor.apply(this,arguments)},initialize:function(){this.subviews=[],this.rendered=!1},template:function(){},skippedRender:function(){},render:function(n){if(this.renderOnce&&this.rendered)return this.skippedRender(),this;this.removeSubviews();var e=this.getRenderCtx(n),n=this.template(e);return this.removed||(this.$el.empty(),this.$el.append(n),this.postRender(e)),this.rendered=!0,this},getRenderCtx:function(n){var e={graph_enabled:r.get(\"GRAPH_ENABLED\"),graph_interval:r.get(\"graph_interval\"),graph_active:r.get(\"GRAPH_ENABLED\")&&\"off\"!==r.get(\"graph_interval\"),nsqlookupd:r.get(\"NSQLOOKUPD\"),version:r.get(\"VERSION\")};return this.model?e=a.extend(e,this.model.toJSON()):this.collection&&(e=a.extend(e,{collection:this.collection.toJSON()})),e=n?a.extend(e,n):e},postRender:function(){},appendSubview:function(n,e){return this.appendSubviews([n],e)},appendSubviews:function(n,e){this.subviews.push.apply(this.subviews,n),(e?this.$(e):this.$el).append(n.map(function(n){return n.render().delegateEvents().el}))},removeSubviews:function(){for(;this.subviews.length;)this.subviews.pop().remove()},remove:function(){this.removed=!0,this.removeSubviews(),o.View.prototype.remove.apply(this,arguments)},parseErrorMessage:function(n){var e=\"ERROR: failed to connect to nsqadmin\";if(4===n.readyState)try{e=JSON.parse(n.responseText).message}catch(n){e=\"ERROR: failed to decode JSON - \"+n.message}return e},handleAJAXError:function(n){l(\"#warning, #error\").hide(),l(\"#error .alert\").text(this.parseErrorMessage(n)),l(\"#error\").show()},handleViewError:function(n){this.removeSubviews(),this.$el.html(i({message:this.parseErrorMessage(n)}))}});e.exports=n},{\"../app_state\":36,\"./error.hbs\":53,backbone:void 0,jquery:void 0,underscore:void 0}],49:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
\\n
\\n
\\n

Notice

No producers exist for this topic/channel.\\n

See Lookup for more information.\\n

\\n
\\n
\\n'},3:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=n.escapeExpression,h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"isAdmin\"):e,{name:\"if\",hash:{},fn:n.program(4,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:29,column:0},end:{line:45,column:7}}}))?i:\"\")+'\\n
\\n
\\n

Channel

\\n \\n \\n \\n \\n \\n'+(null!=(i=h(t,\"if\").call(s,null!=(i=null!=(i=null!=e?h(e,\"e2e_processing_latency\"):e)?h(i,\"percentiles\"):i)?h(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(13,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:55,column:12},end:{line:57,column:19}}}))?i:\"\")+\" \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(15,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:68,column:12},end:{line:68,column:52}}}))?i:\"\")+\"\\n \\n \"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"delivery_msg_count\"):e,{name:\"if\",hash:{},fn:n.program(17,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:70,column:12},end:{line:70,column:83}}}))?i:\"\")+\"\\n\"+(null!=(i=h(t,\"each\").call(s,null!=(i=null!=e?h(e,\"e2e_processing_latency\"):e)?h(i,\"percentiles\"):i,{name:\"each\",hash:{},fn:n.program(19,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:71,column:12},end:{line:73,column:21}}}))?i:\"\")+\" \\n\"+(null!=(i=h(t,\"each\").call(s,null!=e?h(e,\"nodes\"):e,{name:\"each\",hash:{},fn:n.program(21,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:75,column:8},end:{line:142,column:17}}}))?i:\"\")+' \\n \\n \\n \\n \\n \\n \\n \\n \\n\"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(28,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:152,column:12},end:{line:154,column:19}}}))?i:\"\")+\" \\n \"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"delivery_msg_count\"):e,{name:\"if\",hash:{},fn:n.program(36,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:156,column:12},end:{line:156,column:54}}}))?i:\"\")+\"\\n\"+(null!=(i=h(t,\"if\").call(s,null!=(i=null!=(i=null!=e?h(e,\"e2e_processing_latency\"):e)?h(i,\"percentiles\"):i)?h(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(32,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:157,column:12},end:{line:163,column:19}}}))?i:\"\")+\" \\n\"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(40,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:165,column:8},end:{line:184,column:15}}}))?i:\"\")+\"
 Message QueuesStatistics
NSQd HostDepthMemory + DiskIn-FlightDeferredRequeuedTimed OutMessagesConnections
Total:'+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:145,column:16},end:{line:145,column:33}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"memory_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:146,column:16},end:{line:146,column:40}}}))+\" + \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"backend_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:146,column:43},end:{line:146,column:68}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"in_flight_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:147,column:16},end:{line:147,column:43}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"deferred_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:148,column:16},end:{line:148,column:42}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"requeue_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:149,column:16},end:{line:149,column:41}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"timeout_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:150,column:16},end:{line:150,column:41}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"message_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:151,column:16},end:{line:151,column:41}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"client_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:155,column:16},end:{line:155,column:40}}}))+\"
\\n
\\n
\\n\"},4:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
\\n
\\n \\n
\\n
\\n \\n
\\n
\\n'+(null!=(a=o(t,\"if\").call(null!=e?e:n.nullContext||{},null!=e?o(e,\"paused\"):e,{name:\"if\",hash:{},fn:n.program(5,a,0),inverse:n.program(7,a,0),data:a,loc:{start:{line:38,column:8},end:{line:42,column:15}}}))?a:\"\")+\"
\\n
\\n\"},5:function(n,e,t,l,a){return' \\n'},7:function(n,e,t,l,a){return' \\n'},9:function(n,e,t,l,a){return\"6\"},11:function(n,e,t,l,a){return\"5\"},13:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' E2E Processing Latency\\n'},15:function(n,e,t,l,a){return\"Rate\"},17:function(n,e,t,l,a){return'Delivery'},19:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" \"+i((n(t,\"floatToPercent\")||e&&n(e,\"floatToPercent\")||r).call(o,null!=e?n(e,\"quantile\"):e,{name:\"floatToPercent\",hash:{},data:a,loc:{start:{line:72,column:20},end:{line:72,column:47}}}))+\"\"+i((n(t,\"percSuffix\")||e&&n(e,\"percSuffix\")||r).call(o,null!=e?n(e,\"quantile\"):e,{name:\"percSuffix\",hash:{},data:a,loc:{start:{line:72,column:52},end:{line:72,column:75}}}))+\"\\n\"},21:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=n.escapeExpression,h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" \\n \\n\"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"show_broadcast_address\"):e,{name:\"if\",hash:{},fn:n.program(22,a,0,o,r),inverse:n.program(24,a,0,o,r),data:a,loc:{start:{line:78,column:16},end:{line:82,column:23}}}))?i:\"\")+\" \"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"paused\"):e,{name:\"if\",hash:{},fn:n.program(26,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:83,column:16},end:{line:83,column:85}}}))?i:\"\")+\"\\n \\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:85,column:16},end:{line:85,column:33}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"memory_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:86,column:16},end:{line:86,column:40}}}))+\" + \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"backend_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:86,column:43},end:{line:86,column:68}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"in_flight_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:87,column:16},end:{line:87,column:43}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"deferred_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:88,column:16},end:{line:88,column:42}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"requeue_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:89,column:16},end:{line:89,column:41}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"timeout_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:90,column:16},end:{line:90,column:41}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"message_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:91,column:16},end:{line:91,column:41}}}))+\"\\n\"+(null!=(i=h(t,\"if\").call(s,null!=r[1]?h(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(28,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:92,column:12},end:{line:94,column:19}}}))?i:\"\")+\" \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"client_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:95,column:16},end:{line:95,column:40}}}))+\"\\n\"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"delivery_msg_count\"):e,{name:\"if\",hash:{},fn:n.program(30,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:96,column:12},end:{line:113,column:19}}}))?i:\"\")+(null!=(i=h(t,\"if\").call(s,null!=(i=null!=(i=null!=e?h(e,\"e2e_processing_latency\"):e)?h(i,\"percentiles\"):i)?h(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(32,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:114,column:12},end:{line:120,column:19}}}))?i:\"\")+\" \\n\"+(null!=(i=h(t,\"if\").call(s,null!=r[1]?h(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(35,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:122,column:8},end:{line:141,column:15}}}))?i:\"\")},22:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=\"function\",s=n.escapeExpression,c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" \"+s(typeof(n=null!=(n=c(t,\"hostname_port\")||(null!=e?c(e,\"hostname_port\"):e))?n:r)==i?n.call(o,{name:\"hostname_port\",hash:{},data:a,loc:{start:{line:79,column:16},end:{line:79,column:33}}}):n)+' ('+s(typeof(n=null!=(n=c(t,\"node\")||(null!=e?c(e,\"node\"):e))?n:r)==i?n.call(o,{name:\"node\",hash:{},data:a,loc:{start:{line:79,column:89},end:{line:79,column:97}}}):n)+\")\\n\"},24:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,s=\"function\",c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' '+i(typeof(n=null!=(n=c(t,\"hostname_port\")||(null!=e?c(e,\"hostname_port\"):e))?n:r)==s?n.call(o,{name:\"hostname_port\",hash:{},data:a,loc:{start:{line:81,column:70},end:{line:81,column:87}}}):n)+\"\\n\"},26:function(n,e,t,l,a){return' paused'},28:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n'},30:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=\"function\",s=n.escapeExpression,c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n
\\n
\\n
\\n '+s((c(t,\"floatToPercent\")||e&&c(e,\"floatToPercent\")||r).call(o,null!=e?c(e,\"zone_local_percentage\"):e,{name:\"floatToPercent\",hash:{},data:a,loc:{start:{line:101,column:50},end:{line:101,column:90}}}))+'%\\n
\\n
\\n '+s((c(t,\"floatToPercent\")||e&&c(e,\"floatToPercent\")||r).call(o,null!=e?c(e,\"region_local_percentage\"):e,{name:\"floatToPercent\",hash:{},data:a,loc:{start:{line:104,column:50},end:{line:104,column:92}}}))+'%\\n
\\n
\\n '+s((c(t,\"floatToPercent\")||e&&c(e,\"floatToPercent\")||r).call(o,null!=e?c(e,\"global_percentage\"):e,{name:\"floatToPercent\",hash:{},data:a,loc:{start:{line:107,column:50},end:{line:107,column:86}}}))+'%\\n
\\n
\\n '+s((c(t,\"floatToDecimalPercent\")||e&&c(e,\"floatToDecimalPercent\")||r).call(o,null!=e?c(e,\"zone_local_percentage\"):e,{name:\"floatToDecimalPercent\",hash:{},data:a,loc:{start:{line:110,column:114},end:{line:110,column:161}}}))+'% | '+s((c(t,\"floatToDecimalPercent\")||e&&c(e,\"floatToDecimalPercent\")||r).call(o,null!=e?c(e,\"region_local_percentage\"):e,{name:\"floatToDecimalPercent\",hash:{},data:a,loc:{start:{line:110,column:224},end:{line:110,column:273}}}))+'% | '+s((c(t,\"floatToDecimalPercent\")||e&&c(e,\"floatToDecimalPercent\")||r).call(o,null!=e?c(e,\"global_percentage\"):e,{name:\"floatToDecimalPercent\",hash:{},data:a,loc:{start:{line:110,column:312},end:{line:110,column:355}}}))+\"%\\n
\\n \\n\"},32:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return null!=(e=o(t,\"each\").call(null!=e?e:n.nullContext||{},null!=(e=null!=e?o(e,\"e2e_processing_latency\"):e)?o(e,\"percentiles\"):e,{name:\"each\",hash:{},fn:n.program(33,a,0),inverse:n.noop,data:a,loc:{start:{line:115,column:16},end:{line:119,column:25}}}))?e:\"\"},33:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n '+i((n(t,\"nanotohuman\")||e&&n(e,\"nanotohuman\")||r).call(o,null!=e?n(e,\"average\"):e,{name:\"nanotohuman\",hash:{},data:a,loc:{start:{line:117,column:116},end:{line:117,column:139}}}))+\"\\n \\n\"},35:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n '+(null!=(r=s(t,\"if\").call(o,null!=e?s(e,\"delivery_msg_count\"):e,{name:\"if\",hash:{},fn:n.program(36,a,0),inverse:n.noop,data:a,loc:{start:{line:134,column:12},end:{line:134,column:54}}}))?r:\"\")+\"\\n\"+(null!=(r=s(t,\"if\").call(o,null!=(r=null!=(r=null!=e?s(e,\"e2e_processing_latency\"):e)?s(r,\"percentiles\"):r)?s(r,\"length\"):r,{name:\"if\",hash:{},fn:n.program(38,a,0),inverse:n.noop,data:a,loc:{start:{line:135,column:12},end:{line:139,column:19}}}))?r:\"\")+\" \\n\"},36:function(n,e,t,l,a){return\"\"},38:function(n,e,t,l,a){var o=n.escapeExpression,r=null!=e?e:n.nullContext||{},i=n.hooks.helperMissing,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n'},40:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n '+(null!=(r=s(t,\"if\").call(o,null!=e?s(e,\"delivery_msg_count\"):e,{name:\"if\",hash:{},fn:n.program(36,a,0),inverse:n.noop,data:a,loc:{start:{line:177,column:12},end:{line:177,column:54}}}))?r:\"\")+\"\\n\"+(null!=(r=s(t,\"if\").call(o,null!=(r=null!=(r=null!=e?s(e,\"e2e_processing_latency\"):e)?s(r,\"percentiles\"):r)?s(r,\"length\"):r,{name:\"if\",hash:{},fn:n.program(41,a,0),inverse:n.noop,data:a,loc:{start:{line:178,column:12},end:{line:182,column:19}}}))?r:\"\")+\" \\n\"},41:function(n,e,t,l,a){var o=n.escapeExpression,r=null!=e?e:n.nullContext||{},i=n.hooks.helperMissing,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n'},43:function(n,e,t,l,a){return'

Notice

No clients connected to this channel
\\n'},45:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n'+(null!=(a=o(t,\"each\").call(null!=e?e:n.nullContext||{},null!=e?o(e,\"clients\"):e,{name:\"each\",hash:{},fn:n.program(46,a,0),inverse:n.noop,data:a,loc:{start:{line:210,column:12},end:{line:260,column:21}}}))?a:\"\")+\"
Client HostUser-AgentAttributesNSQd HostIn-FlightReady CountFinishedRequeuedMessagesConnected
\\n\"},46:function(n,e,t,l,a){var o,r,i=null!=e?e:n.nullContext||{},s=n.hooks.helperMissing,c=\"function\",u=n.escapeExpression,h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" \\n '+u(typeof(r=null!=(r=h(t,\"hostname_port\")||(null!=e?h(e,\"hostname_port\"):e))?r:s)==c?r.call(i,{name:\"hostname_port\",hash:{},data:a,loc:{start:{line:212,column:47},end:{line:212,column:64}}}):r)+(null!=(o=h(t,\"if\").call(i,null!=e?h(e,\"show_client_id\"):e,{name:\"if\",hash:{},fn:n.program(51,a,0),inverse:n.noop,data:a,loc:{start:{line:212,column:64},end:{line:212,column:109}}}))?o:\"\")+\"\\n \"+(null!=(o=h(t,\"if\").call(i,null!=(o=null!=e?h(e,\"user_agent\"):e)?h(o,\"length\"):o,{name:\"if\",hash:{},fn:n.program(53,a,0),inverse:n.noop,data:a,loc:{start:{line:213,column:20},end:{line:213,column:81}}}))?o:\"\")+\"\\n \\n\"+(null!=(o=h(t,\"if\").call(i,null!=e?h(e,\"sample_rate\"):e,{name:\"if\",hash:{},fn:n.program(55,a,0),inverse:n.noop,data:a,loc:{start:{line:215,column:20},end:{line:217,column:27}}}))?o:\"\")+(null!=(o=h(t,\"if\").call(i,null!=e?h(e,\"tls\"):e,{name:\"if\",hash:{},fn:n.program(57,a,0),inverse:n.noop,data:a,loc:{start:{line:218,column:20},end:{line:220,column:27}}}))?o:\"\")+(null!=(o=h(t,\"if\").call(i,null!=e?h(e,\"deflate\"):e,{name:\"if\",hash:{},fn:n.program(60,a,0),inverse:n.noop,data:a,loc:{start:{line:221,column:20},end:{line:223,column:27}}}))?o:\"\")+(null!=(o=h(t,\"if\").call(i,null!=e?h(e,\"snappy\"):e,{name:\"if\",hash:{},fn:n.program(62,a,0),inverse:n.noop,data:a,loc:{start:{line:224,column:20},end:{line:226,column:27}}}))?o:\"\")+(null!=(o=h(t,\"if\").call(i,null!=e?h(e,\"authed\"):e,{name:\"if\",hash:{},fn:n.program(64,a,0),inverse:n.noop,data:a,loc:{start:{line:227,column:20},end:{line:233,column:27}}}))?o:\"\")+(null!=(o=h(t,\"if\").call(i,null!=e?h(e,\"topology_region\"):e,{name:\"if\",hash:{},fn:n.program(71,a,0),inverse:n.noop,data:a,loc:{start:{line:234,column:20},end:{line:236,column:27}}}))?o:\"\")+(null!=(o=h(t,\"if\").call(i,null!=e?h(e,\"topology_zone\"):e,{name:\"if\",hash:{},fn:n.program(73,a,0),inverse:n.noop,data:a,loc:{start:{line:237,column:20},end:{line:239,column:27}}}))?o:\"\")+(null!=(o=h(t,\"if\").call(i,(h(t,\"and\")||e&&h(e,\"and\")||s).call(i,null!=e?h(e,\"node_topology_zone\"):e,(h(t,\"eq\")||e&&h(e,\"eq\")||s).call(i,null!=e?h(e,\"topology_zone\"):e,null!=e?h(e,\"node_topology_zone\"):e,{name:\"eq\",hash:{},data:a,loc:{start:{line:242,column:24},end:{line:242,column:61}}}),{name:\"and\",hash:{},data:a,loc:{start:{line:240,column:26},end:{line:242,column:62}}}),{name:\"if\",hash:{},fn:n.program(75,a,0),inverse:n.program(77,a,0),data:a,loc:{start:{line:240,column:20},end:{line:250,column:27}}}))?o:\"\")+' \\n '+u(typeof(r=null!=(r=h(t,\"node\")||(null!=e?h(e,\"node\"):e))?r:s)==c?r.call(i,{name:\"node\",hash:{},data:a,loc:{start:{line:252,column:74},end:{line:252,column:82}}}):r)+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||s).call(i,null!=e?h(e,\"in_flight_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:253,column:20},end:{line:253,column:47}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||s).call(i,null!=e?h(e,\"ready_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:254,column:20},end:{line:254,column:43}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||s).call(i,null!=e?h(e,\"finish_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:255,column:20},end:{line:255,column:44}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||s).call(i,null!=e?h(e,\"requeue_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:256,column:20},end:{line:256,column:45}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||s).call(i,null!=e?h(e,\"message_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:257,column:20},end:{line:257,column:45}}}))+\"\\n \"+u((h(t,\"nanotohuman\")||e&&h(e,\"nanotohuman\")||s).call(i,null!=e?h(e,\"connected\"):e,{name:\"nanotohuman\",hash:{},data:a,loc:{start:{line:258,column:20},end:{line:258,column:45}}}))+\"\\n \\n\"},47:function(n,e,t,l,a){return'\"bg-zone-local\"'},49:function(n,e,t,l,a){return'\"bg-region-local\"'},51:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" (\"+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"client_id\")||(null!=e?o(e,\"client_id\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"client_id\",hash:{},data:a,loc:{start:{line:212,column:88},end:{line:212,column:101}}}):o)+\")\"},53:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\"\"+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"user_agent\")||(null!=e?o(e,\"user_agent\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"user_agent\",hash:{},data:a,loc:{start:{line:213,column:52},end:{line:213,column:66}}}):o)+\"\"},55:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' Sampled '+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"sample_rate\")||(null!=e?o(e,\"sample_rate\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"sample_rate\",hash:{},data:a,loc:{start:{line:216,column:63},end:{line:216,column:78}}}):o)+\"%\\n\"},57:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' TLS\\n\"},58:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=\"function\",s=n.escapeExpression,c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'title=\"'+s(typeof(n=null!=(n=c(t,\"tls_version\")||(null!=e?c(e,\"tls_version\"):e))?n:r)==i?n.call(o,{name:\"tls_version\",hash:{},data:a,loc:{start:{line:219,column:84},end:{line:219,column:99}}}):n)+\" \"+s(typeof(n=null!=(n=c(t,\"tls_cipher_suite\")||(null!=e?c(e,\"tls_cipher_suite\"):e))?n:r)==i?n.call(o,{name:\"tls_cipher_suite\",hash:{},data:a,loc:{start:{line:219,column:100},end:{line:219,column:120}}}):n)+\" \"+s(typeof(n=null!=(n=c(t,\"tls_negotiated_protocol\")||(null!=e?c(e,\"tls_negotiated_protocol\"):e))?n:r)==i?n.call(o,{name:\"tls_negotiated_protocol\",hash:{},data:a,loc:{start:{line:219,column:121},end:{line:219,column:148}}}):n)+\" mutual:\"+s(typeof(n=null!=(n=c(t,\"tls_negotiated_protocol_is_mutual\")||(null!=e?c(e,\"tls_negotiated_protocol_is_mutual\"):e))?n:r)==i?n.call(o,{name:\"tls_negotiated_protocol_is_mutual\",hash:{},data:a,loc:{start:{line:219,column:156},end:{line:219,column:193}}}):n)+'\"'},60:function(n,e,t,l,a){return' Deflate\\n'},62:function(n,e,t,l,a){return' Snappy\\n'},64:function(n,e,t,l,a){var o,r=null!=e?e:n.nullContext||{},i=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n '+(null!=(o=i(t,\"if\").call(r,null!=e?i(e,\"auth_identity_url\"):e,{name:\"if\",hash:{},fn:n.program(65,a,0),inverse:n.noop,data:a,loc:{start:{line:229,column:24},end:{line:229,column:88}}}))?o:\"\")+'\\n \\n '+(null!=(o=i(t,\"if\").call(r,null!=e?i(e,\"auth_identity_url\"):e,{name:\"if\",hash:{},fn:n.program(69,a,0),inverse:n.noop,data:a,loc:{start:{line:231,column:24},end:{line:231,column:60}}}))?o:\"\")+\"\\n \\n\"},65:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},67:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" Identity:\"+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"auth_identity\")||(null!=e?o(e,\"auth_identity\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"auth_identity\",hash:{},data:a,loc:{start:{line:230,column:113},end:{line:230,column:130}}}):o)},69:function(n,e,t,l,a){return\"\"},71:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' '+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"topology_region\")||(null!=e?o(e,\"topology_region\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"topology_region\",hash:{},data:a,loc:{start:{line:235,column:58},end:{line:235,column:77}}}):o)+\"\\n\"},73:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' '+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"topology_zone\")||(null!=e?o(e,\"topology_zone\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"topology_zone\",hash:{},data:a,loc:{start:{line:238,column:58},end:{line:238,column:75}}}):o)+\"\\n\"},75:function(n,e,t,l,a){return' zoneLocal\\n'},77:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return null!=(a=i(t,\"if\").call(o,(i(t,\"and\")||e&&i(e,\"and\")||r).call(o,null!=e?i(e,\"node_topology_region\"):e,(i(t,\"eq\")||e&&i(e,\"eq\")||r).call(o,null!=e?i(e,\"topology_region\"):e,null!=e?i(e,\"node_topology_region\"):e,{name:\"eq\",hash:{},data:a,loc:{start:{line:247,column:28},end:{line:247,column:69}}}),{name:\"and\",hash:{},data:a,loc:{start:{line:245,column:30},end:{line:247,column:70}}}),{name:\"if\",hash:{},fn:n.program(78,a,0),inverse:n.noop,data:a,loc:{start:{line:245,column:24},end:{line:249,column:31}}}))?a:\"\"},78:function(n,e,t,l,a){return' regionLocal\\n'},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=n.escapeExpression,h=\"function\",p=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return(null!=(i=n.invokePartial(p(l,\"warning\"),e,{name:\"warning\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+(null!=(i=n.invokePartial(p(l,\"error\"),e,{name:\"error\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+'\\n
    \\n
  1. Streams\\n
  2. '+u(typeof(l=null!=(l=p(t,\"topic\")||(null!=e?p(e,\"topic\"):e))?l:c)==h?l.call(s,{name:\"topic\",hash:{},data:a,loc:{start:{line:6,column:72},end:{line:6,column:81}}}):l)+'\\n
  3. '+u(typeof(l=null!=(l=p(t,\"name\")||(null!=e?p(e,\"name\"):e))?l:c)==h?l.call(s,{name:\"name\",hash:{},data:a,loc:{start:{line:7,column:21},end:{line:7,column:29}}}):l)+'
    4. \\n
\\n\\n
\\n
\\n
\\n

Topic: '+u(typeof(l=null!=(l=p(t,\"topic\")||(null!=e?p(e,\"topic\"):e))?l:c)==h?l.call(s,{name:\"topic\",hash:{},data:a,loc:{start:{line:13,column:30},end:{line:13,column:39}}}):l)+\"\\n

Channel: \"+u(typeof(l=null!=(l=p(t,\"name\")||(null!=e?p(e,\"name\"):e))?l:c)==h?l.call(s,{name:\"name\",hash:{},data:a,loc:{start:{line:14,column:32},end:{line:14,column:40}}}):l)+\"\\n

\\n
\\n
\\n\\n\"+(null!=(i=p(t,\"unless\").call(s,null!=(i=null!=e?p(e,\"nodes\"):e)?p(i,\"length\"):i,{name:\"unless\",hash:{},fn:n.program(1,a,0,o,r),inverse:n.program(3,a,0,o,r),data:a,loc:{start:{line:19,column:0},end:{line:188,column:11}}}))?i:\"\")+'\\n

Client Connections

\\n\\n
\\n
\\n'+(null!=(i=p(t,\"unless\").call(s,null!=(i=null!=e?p(e,\"clients\"):e)?p(i,\"length\"):i,{name:\"unless\",hash:{},fn:n.program(43,a,0,o,r),inverse:n.program(45,a,0,o,r),data:a,loc:{start:{line:194,column:8},end:{line:262,column:19}}}))?i:\"\")+\"
\\n
\\n\"},usePartial:!0,useData:!0,useDepths:!0})},{\"hbsfy/runtime\":35}],50:[function(t,n,e){var l=t(\"jquery\");window.jQuery=l;t(\"bootstrap\");var a=t(\"bootbox\"),o=t(\"../lib/pubsub\"),r=t(\"../app_state\"),i=t(\"./base\"),s=i.extend({className:\"channel container-fluid\",template:t(\"./spinner.hbs\"),events:{\"click .channel-actions button\":\"channelAction\",\"click .popup\":\"showDeliveryBreakdown\"},initialize:function(){i.prototype.initialize.apply(this,arguments),this.listenTo(r,\"change:graph_interval\",this.render);var e=this.model.get(\"isAdmin\");this.model.fetch().done(function(n){this.template=t(\"./channel.hbs\"),this.render({message:n.message,isAdmin:e})}.bind(this)).fail(this.handleViewError.bind(this)).always(o.trigger.bind(o,\"view:ready\"))},showDeliveryBreakdown:function(n){n.preventDefault(),n.stopPropagation(),document.getElementById(l(n.currentTarget).data(\"id\")).classList.toggle(\"show\")},channelAction:function(n){n.preventDefault(),n.stopPropagation();var t=l(n.currentTarget).data(\"action\"),n=\"Are you sure you want to \"+t+\" \"+this.model.get(\"topic\")+\"/\"+this.model.get(\"name\")+\"?\";a.confirm(n,function(n){var e;!0===n&&(\"delete\"===t?(e=this.model.get(\"topic\"),l.ajax(this.model.url(),{method:\"DELETE\"}).done(function(){window.location=r.basePath(\"/topics/\"+encodeURIComponent(e))}).fail(this.handleAJAXError.bind(this))):l.post(this.model.url(),JSON.stringify({action:t})).done(function(){window.location.reload(!0)}).fail(this.handleAJAXError.bind(this)))}.bind(this))}});n.exports=s},{\"../app_state\":36,\"../lib/pubsub\":41,\"./base\":48,\"./channel.hbs\":49,\"./spinner.hbs\":62,bootbox:void 0,bootstrap:1,jquery:void 0}],51:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n'},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a){var o,r=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\"\\n\\n\"+(null!=(o=n.invokePartial(r(l,\"warning\"),e,{name:\"warning\",data:a,helpers:t,partials:l,decorators:n.decorators}))?o:\"\")+(null!=(o=n.invokePartial(r(l,\"error\"),e,{name:\"error\",data:a,helpers:t,partials:l,decorators:n.decorators}))?o:\"\")+'\\n
\\n
\\n
\\n

Messages Processed

\\n

\\n'+(null!=(o=r(t,\"if\").call(null!=e?e:n.nullContext||{},null!=e?r(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(1,a,0),inverse:n.noop,data:a,loc:{start:{line:205,column:0},end:{line:207,column:7}}}))?o:\"\")+\"
\\n\"},usePartial:!0,useData:!0})},{\"hbsfy/runtime\":35}],52:[function(n,e,t){var l=n(\"underscore\"),i=n(\"jquery\"),a=n(\"../app_state\"),o=n(\"./base\"),n=o.extend({className:\"counter container-fluid\",template:n(\"./counter.hbs\"),initialize:function(){o.prototype.initialize.apply(this,arguments),this.listenTo(a,\"change:graph_interval\",function(){clearTimeout(this.poller),clearTimeout(this.animator),this.render(),this.start()}),this.start()},remove:function(){clearTimeout(this.poller),clearTimeout(this.animator),o.prototype.remove.apply(this,arguments)},start:function(){this.poller=null,this.animator=null,this.delta=0,this.looping=!1,this.targetPollInterval=1e4,this.currentNum=-1,this.lastNum=0,this.interval=100,this.graphUrl=null,this.updateStats()},startLoop:function(n){this.interval=n,this.poller=setTimeout(this.updateStats.bind(this),n)},updateStats:function(){var n;i.get(a.apiPath(\"/counter\")).done(function(n){var e,t;this.removed||(t=l.reduce(n.stats,function(n,e){return n+e.message_count},0),-1===this.currentNum?(this.currentNum=t,this.lastNum=t,this.writeCounts(this.currentNum)):t>this.lastNum&&(e=t-this.lastNum,this.delta=e/(this.interval/1e3)/50,this.lastNum=t,this.animator||this.displayFrame()),(t=this.interval)a?((o=i(l[a])).show(),o.find(\".top\").text(r),o.find(\".bottom\").text(r)):i(e).append(''+r+''+r+\"\\n\")}i(\".numbers .number\").each(function(n,e){n>=t.length&&i(e).hide()})}});e.exports=n},{\"../app_state\":36,\"./base\":48,\"./counter.hbs\":51,jquery:void 0,underscore:void 0}],53:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a){return'style=\"display: none;\"'},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a){var o,r=null!=e?e:n.nullContext||{},i=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
\\n
\\n
\\n '+n.escapeExpression(\"function\"==typeof(e=null!=(e=i(t,\"message\")||(null!=e?i(e,\"message\"):e))?e:n.hooks.helperMissing)?e.call(r,{name:\"message\",hash:{},data:a,loc:{start:{line:4,column:12},end:{line:4,column:23}}}):e)+\"\\n
\\n
\\n
\\n\"},useData:!0})},{\"hbsfy/runtime\":35}],54:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a){var o,r=null!=e?e:n.nullContext||{},i=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
  • \\n '+n.escapeExpression(\"function\"==typeof(o=null!=(o=i(t,\"graph_interval\")||(null!=e?i(e,\"graph_interval\"):e))?o:n.hooks.helperMissing)?o.call(r,{name:\"graph_interval\",hash:{},data:a,loc:{start:{line:20,column:171},end:{line:20,column:189}}}):o)+' \\n
      \\n
    • Graph Timeframe
      • \\n'+(null!=(a=i(t,\"each\").call(r,null!=e?i(e,\"graph_intervals\"):e,{name:\"each\",hash:{},fn:n.program(2,a,0),inverse:n.noop,data:a,loc:{start:{line:23,column:20},end:{line:25,column:29}}}))?a:\"\")+\"
    \\n
    • \\n\"},2:function(n,e,t,l,a){return'
  • '+n.escapeExpression(n.lambda(e,e))+\"
    • \\n\"},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'\\n\"},useData:!0})},{\"hbsfy/runtime\":35}],55:[function(n,e,t){var l=n(\"underscore\"),a=n(\"jquery\"),o=n(\"../app_state\"),r=n(\"./base\"),n=r.extend({className:\"header\",template:n(\"./header.hbs\"),events:{\"click .dropdown-menu li\":\"onGraphIntervalClick\"},initialize:function(){r.prototype.initialize.apply(this,arguments),this.listenTo(o,\"change:graph_interval\",this.render)},getRenderCtx:function(){return l.extend(r.prototype.getRenderCtx.apply(this,arguments),{graph_intervals:[\"1h\",\"2h\",\"12h\",\"24h\",\"48h\",\"168h\",\"off\"],graph_interval:o.get(\"graph_interval\")})},onReset:function(){this.render(),this.$(\".dropdown-toggle\").dropdown()},onGraphIntervalClick:function(n){n.stopPropagation(),o.set(\"graph_interval\",a(n.target).text())}});e.exports=n},{\"../app_state\":36,\"./base\":48,\"./header.hbs\":54,jquery:void 0,underscore:void 0}],56:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a){return'
    \\n

    Notice

    nsqadmin is not configured with nsqlookupd hosts\\n
    \\n'},3:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
    \\n
    \\n \\n \\n \\n \\n'+(null!=(i=c(t,\"each\").call(s,null!=e?c(e,\"nsqlookupd\"):e,{name:\"each\",hash:{},fn:n.program(4,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:21,column:12},end:{line:23,column:21}}}))?i:\"\")+'
    nsqlookupd Host
    \\n
    \\n
    \\n\\n
    \\n
    \\n'+(null!=(i=c(t,\"if\").call(s,null!=e?c(e,\"topics\"):e,{name:\"if\",hash:{},fn:n.program(6,a,0,o,r),inverse:n.program(10,a,0,o,r),data:a,loc:{start:{line:30,column:8},end:{line:50,column:15}}}))?i:\"\")+\"
    \\n
    \\n\\n\"+(null!=(i=c(t,\"if\").call(s,null!=e?c(e,\"isAdmin\"):e,{name:\"if\",hash:{},fn:n.program(12,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:54,column:0},end:{line:72,column:7}}}))?i:\"\")},4:function(n,e,t,l,a){return\" \"+n.escapeExpression(n.lambda(e,e))+\"\\n\"},6:function(n,e,t,l,a,o,r){var i=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
    \\n Below is a tree of Topics/Channels that are currently inactive (i.e. not produced on any nsqd in the cluster but are present in the lookup data)\\n
    \\n
      \\n'+(null!=(a=i(t,\"each\").call(null!=e?e:n.nullContext||{},null!=e?i(e,\"topics\"):e,{name:\"each\",hash:{},fn:n.program(7,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:35,column:12},end:{line:46,column:21}}}))?a:\"\")+\"
    \\n\"},7:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=\"function\",h=n.escapeExpression,p=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
  • \\n '+h(typeof(i=null!=(i=p(t,\"name\")||(null!=e?p(e,\"name\"):e))?i:c)==u?i.call(s,{name:\"name\",hash:{},data:a,loc:{start:{line:37,column:186},end:{line:37,column:194}}}):i)+\"\\n
      \\n\"+(null!=(a=p(t,\"each\").call(s,null!=e?p(e,\"channels\"):e,{name:\"each\",hash:{},fn:n.program(8,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:39,column:20},end:{line:43,column:29}}}))?a:\"\")+\"
    \\n
    • \\n\"},8:function(n,e,t,l,a,o,r){var i=n.lambda,s=n.escapeExpression,c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
  • \\n '+s(i(e,e))+\"\\n
    • \\n\"},10:function(n,e,t,l,a){return'

    Notice

    No inactive Topics
    \\n'},12:function(n,e,t,l,a){return'
    \\n
    \\n
    \\n Create Topic/Channel\\n
    \\n

    This provides a way to setup a stream hierarchy\\n before services are deployed to production.\\n

    If Channel Name is empty, just the topic is created.\\n

    \\n
    \\n \\n \\n
    \\n \\n
    \\n
    \\n
    \\n'},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a,o,r){var i,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return(null!=(i=n.invokePartial(s(l,\"warning\"),e,{name:\"warning\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+(null!=(i=n.invokePartial(s(l,\"error\"),e,{name:\"error\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+'\\n
    \\n
    \\n

    Lookup

    \\n
    \\n
    \\n\\n'+(null!=(i=s(t,\"unless\").call(null!=e?e:n.nullContext||{},null!=(i=null!=e?s(e,\"nsqlookupd\"):e)?s(i,\"length\"):i,{name:\"unless\",hash:{},fn:n.program(1,a,0,o,r),inverse:n.program(3,a,0,o,r),data:a,loc:{start:{line:10,column:0},end:{line:73,column:11}}}))?i:\"\")},usePartial:!0,useData:!0,useDepths:!0})},{\"hbsfy/runtime\":35}],57:[function(t,n,e){var l=t(\"underscore\"),a=t(\"jquery\"),o=t(\"../app_state\"),r=t(\"../lib/pubsub\"),i=t(\"./base\"),s=t(\"../models/topic\"),c=t(\"../models/channel\"),u=i.extend({className:\"lookup container-fluid\",template:t(\"./spinner.hbs\"),events:{\"click .hierarchy button\":\"onCreateTopicChannel\",\"click .delete-topic-link\":\"onDeleteTopic\",\"click .delete-channel-link\":\"onDeleteChannel\"},initialize:function(){i.prototype.initialize.apply(this,arguments);var e=arguments[0].isAdmin;a.ajax(o.apiPath(\"/topics?inactive=true\")).done(function(n){this.template=t(\"./lookup.hbs\"),this.render({topics:l.map(n.topics,function(n,e){return{name:e,channels:n}}),message:n.message,isAdmin:e})}.bind(this)).fail(this.handleViewError.bind(this)).always(r.trigger.bind(r,\"view:ready\"))},onCreateTopicChannel:function(n){n.preventDefault(),n.stopPropagation();var e=a(n.target.form.elements.topic).val(),n=a(n.target.form.elements.channel).val();\"\"===e&&\"\"===n||a.post(o.apiPath(\"/topics\"),JSON.stringify({topic:e,channel:n})).done(function(){window.location.reload(!0)}).fail(this.handleAJAXError.bind(this))},onDeleteTopic:function(n){n.preventDefault(),n.stopPropagation(),new s({name:a(n.target).data(\"topic\")}).destroy({dataType:\"text\"}).done(function(){window.location.reload(!0)}).fail(this.handleAJAXError.bind(this))},onDeleteChannel:function(n){n.preventDefault(),n.stopPropagation(),new c({topic:a(n.target).data(\"topic\"),name:a(n.target).data(\"channel\")}).destroy({dataType:\"text\"}).done(function(){window.location.reload(!0)}).fail(this.handleAJAXError.bind(this))}});n.exports=u},{\"../app_state\":36,\"../lib/pubsub\":41,\"../models/channel\":43,\"../models/topic\":45,\"./base\":48,\"./lookup.hbs\":56,\"./spinner.hbs\":62,jquery:void 0,underscore:void 0}],58:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
    \\n
    \\n \\n \\n \\n \\n \\n \\n \\n
    \\n \\n
    GC Pressure
    \\n     		\\n \\n
    GC Pause Percentiles
    \\n     		\\n \\n
    GC Runs
    \\n     		\\n \\n
    Heap Objects In-Use
    \\n
    \\n
    \\n
    \\n'},3:function(n,e,t,l,a){return'
    \\n

    Notice

    No topics exist on this node.\\n
    \\n'},5:function(n,e,t,l,a,o,r){var i,s=n.escapeExpression,c=null!=e?e:n.nullContext||{},u=n.hooks.helperMissing,h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n \\n \\n \\n \\n\"+(null!=(i=h(t,\"each\").call(c,null!=e?h(e,\"topics\"):e,{name:\"each\",hash:{},fn:n.program(6,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:50,column:8},end:{line:175,column:17}}}))?i:\"\")+\"
    '+s(n.lambda(null!=(i=null!=r[1]?h(r[1],\"topics\"):r[1])?h(i,\"length\"):i,e))+' Topics'+s((h(t,\"commafy\")||e&&h(e,\"commafy\")||u).call(c,null!=e?h(e,\"total_messages\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:47,column:24},end:{line:47,column:50}}}))+\" Messages\"+s((h(t,\"commafy\")||e&&h(e,\"commafy\")||u).call(c,null!=e?h(e,\"total_clients\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:48,column:24},end:{line:48,column:49}}}))+\" Clients
    \\n\"},6:function(n,e,t,l,a,o,r){var i,s=n.escapeExpression,c=null!=e?e:n.nullContext||{},u=n.hooks.helperMissing,h=\"function\",p=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n Topic\\n Depth\\n Memory + Disk\\n \\n Messages\\n Channels\\n \\n \\n \\n '+s(typeof(i=null!=(i=p(t,\"topic_name\")||(null!=e?p(e,\"topic_name\"):e))?i:u)==h?i.call(c,{name:\"topic_name\",hash:{},data:a,loc:{start:{line:61,column:158},end:{line:61,column:172}}}):i)+\"\\n \"+(null!=(i=p(t,\"if\").call(c,null!=e?p(e,\"paused\"):e,{name:\"if\",hash:{},fn:n.program(7,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:62,column:16},end:{line:62,column:84}}}))?i:\"\")+\"\\n \\n \\n \"+(null!=(i=p(t,\"if\").call(c,null!=r[1]?p(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(9,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:65,column:16},end:{line:65,column:183}}}))?i:\"\")+\"\\n \"+s((p(t,\"commafy\")||e&&p(e,\"commafy\")||u).call(c,null!=e?p(e,\"depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:66,column:16},end:{line:66,column:33}}}))+\"\\n \\n \"+s((p(t,\"commafy\")||e&&p(e,\"commafy\")||u).call(c,null!=e?p(e,\"memory_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:68,column:16},end:{line:68,column:40}}}))+\" + \"+s((p(t,\"commafy\")||e&&p(e,\"commafy\")||u).call(c,null!=e?p(e,\"backend_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:68,column:43},end:{line:68,column:68}}}))+'\\n \\n \\n '+(null!=(i=p(t,\"if\").call(c,null!=r[1]?p(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(11,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:71,column:16},end:{line:71,column:199}}}))?i:\"\")+\"\\n \"+s((p(t,\"commafy\")||e&&p(e,\"commafy\")||u).call(c,null!=e?p(e,\"message_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:72,column:16},end:{line:72,column:41}}}))+\"\\n \\n \"+s((p(t,\"commafy\")||e&&p(e,\"commafy\")||u).call(c,null!=(i=null!=e?p(e,\"channels\"):e)?p(i,\"length\"):i,{name:\"commafy\",hash:{},data:a,loc:{start:{line:74,column:16},end:{line:74,column:43}}}))+\"\\n \\n\"+(null!=(i=p(t,\"unless\").call(c,null!=(i=null!=e?p(e,\"channels\"):e)?p(i,\"length\"):i,{name:\"unless\",hash:{},fn:n.program(13,a,0,o,r),inverse:n.program(15,a,0,o,r),data:a,loc:{start:{line:76,column:8},end:{line:174,column:19}}}))?i:\"\")},7:function(n,e,t,l,a){return'paused'},9:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},11:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},13:function(n,e,t,l,a){return' \\n \\n

    Notice

    No channels exist for this topic.
    \\n \\n \\n'},15:function(n,e,t,l,a,o,r){var i=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return null!=(a=i(t,\"each\").call(null!=e?e:n.nullContext||{},null!=e?i(e,\"channels\"):e,{name:\"each\",hash:{},fn:n.program(16,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:83,column:8},end:{line:173,column:17}}}))?a:\"\"},16:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=n.escapeExpression,h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n Channel\\n Depth\\n Memory + Disk\\n In-Flight\\n Deferred\\n Requeued\\n Timed Out\\n Messages\\n Connections\\n \\n \\n \\n \\n '+u(\"function\"==typeof(i=null!=(i=h(t,\"channel_name\")||(null!=e?h(e,\"channel_name\"):e))?i:c)?i.call(s,{name:\"channel_name\",hash:{},data:a,loc:{start:{line:99,column:16},end:{line:99,column:32}}}):i)+\"\\n \"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"paused\"):e,{name:\"if\",hash:{},fn:n.program(7,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:100,column:16},end:{line:100,column:84}}}))?i:\"\")+\"\\n \\n \\n \"+(null!=(i=h(t,\"if\").call(s,null!=r[3]?h(r[3],\"graph_active\"):r[3],{name:\"if\",hash:{},fn:n.program(17,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:103,column:16},end:{line:103,column:225}}}))?i:\"\")+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:104,column:16},end:{line:104,column:33}}}))+\"\\n \\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"memory_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:106,column:16},end:{line:106,column:40}}}))+\" + \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"backend_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:106,column:43},end:{line:106,column:68}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"in_flight_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:107,column:16},end:{line:107,column:43}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"deferred_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:108,column:16},end:{line:108,column:42}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"requeue_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:109,column:16},end:{line:109,column:41}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"timeout_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:110,column:16},end:{line:110,column:41}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"message_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:111,column:16},end:{line:111,column:41}}}))+\"\\n \\n \"+(null!=(i=h(t,\"if\").call(s,null!=r[3]?h(r[3],\"graph_active\"):r[3],{name:\"if\",hash:{},fn:n.program(19,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:113,column:16},end:{line:113,column:229}}}))?i:\"\")+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=(i=null!=e?h(e,\"clients\"):e)?h(i,\"length\"):i,{name:\"commafy\",hash:{},data:a,loc:{start:{line:114,column:16},end:{line:114,column:42}}}))+\"\\n \\n \\n\"+(null!=(i=h(t,\"unless\").call(s,null!=(i=null!=e?h(e,\"clients\"):e)?h(i,\"length\"):i,{name:\"unless\",hash:{},fn:n.program(21,a,0,o,r),inverse:n.program(23,a,0,o,r),data:a,loc:{start:{line:117,column:8},end:{line:172,column:19}}}))?i:\"\")},17:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},19:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},21:function(n,e,t,l,a){return' \\n \\n

    Notice

    No clients connected to this channel.
    \\n \\n \\n'},23:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" \\n \\n Client Host\\n User-Agent\\n \\n Attributes\\n In-Flight\\n Ready Count\\n Requeued\\n Finished\\n Messages\\n Connected\\n \\n\"+(null!=(a=o(t,\"each\").call(null!=e?e:n.nullContext||{},null!=e?o(e,\"clients\"):e,{name:\"each\",hash:{},fn:n.program(24,a,0),inverse:n.noop,data:a,loc:{start:{line:137,column:8},end:{line:171,column:17}}}))?a:\"\")},24:function(n,e,t,l,a){var o,r=null!=e?e:n.nullContext||{},i=n.hooks.helperMissing,s=\"function\",c=n.escapeExpression,u=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n '+c(typeof(o=null!=(o=u(t,\"hostname\")||(null!=e?u(e,\"hostname\"):e))?o:i)==s?o.call(r,{name:\"hostname\",hash:{},data:a,loc:{start:{line:140,column:43},end:{line:140,column:55}}}):o)+(null!=(o=u(t,\"if\").call(r,null!=e?u(e,\"show_client_id\"):e,{name:\"if\",hash:{},fn:n.program(25,a,0),inverse:n.noop,data:a,loc:{start:{line:140,column:55},end:{line:140,column:100}}}))?o:\"\")+\"\\n \"+(null!=(o=u(t,\"if\").call(r,null!=(o=null!=e?u(e,\"user_agent\"):e)?u(o,\"length\"):o,{name:\"if\",hash:{},fn:n.program(27,a,0),inverse:n.noop,data:a,loc:{start:{line:141,column:16},end:{line:141,column:77}}}))?o:\"\")+\"\\n \\n \\n\"+(null!=(o=u(t,\"if\").call(r,null!=e?u(e,\"sample_rate\"):e,{name:\"if\",hash:{},fn:n.program(29,a,0),inverse:n.noop,data:a,loc:{start:{line:144,column:16},end:{line:146,column:23}}}))?o:\"\")+(null!=(o=u(t,\"if\").call(r,null!=e?u(e,\"tls\"):e,{name:\"if\",hash:{},fn:n.program(31,a,0),inverse:n.noop,data:a,loc:{start:{line:147,column:16},end:{line:149,column:23}}}))?o:\"\")+(null!=(o=u(t,\"if\").call(r,null!=e?u(e,\"deflate\"):e,{name:\"if\",hash:{},fn:n.program(34,a,0),inverse:n.noop,data:a,loc:{start:{line:150,column:16},end:{line:152,column:23}}}))?o:\"\")+(null!=(o=u(t,\"if\").call(r,null!=e?u(e,\"snappy\"):e,{name:\"if\",hash:{},fn:n.program(36,a,0),inverse:n.noop,data:a,loc:{start:{line:153,column:16},end:{line:155,column:23}}}))?o:\"\")+(null!=(o=u(t,\"if\").call(r,null!=e?u(e,\"authed\"):e,{name:\"if\",hash:{},fn:n.program(38,a,0),inverse:n.noop,data:a,loc:{start:{line:156,column:16},end:{line:162,column:23}}}))?o:\"\")+\" \\n \"+c((u(t,\"commafy\")||e&&u(e,\"commafy\")||i).call(r,null!=e?u(e,\"in_flight_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:164,column:16},end:{line:164,column:43}}}))+\"\\n \"+c((u(t,\"commafy\")||e&&u(e,\"commafy\")||i).call(r,null!=e?u(e,\"ready_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:165,column:16},end:{line:165,column:39}}}))+\"\\n \"+c((u(t,\"commafy\")||e&&u(e,\"commafy\")||i).call(r,null!=e?u(e,\"requeue_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:166,column:16},end:{line:166,column:41}}}))+\"\\n \"+c((u(t,\"commafy\")||e&&u(e,\"commafy\")||i).call(r,null!=e?u(e,\"finish_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:167,column:16},end:{line:167,column:40}}}))+\"\\n \"+c((u(t,\"commafy\")||e&&u(e,\"commafy\")||i).call(r,null!=e?u(e,\"message_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:168,column:16},end:{line:168,column:41}}}))+\"\\n \"+c((u(t,\"nanotohuman\")||e&&u(e,\"nanotohuman\")||i).call(r,null!=e?u(e,\"connected\"):e,{name:\"nanotohuman\",hash:{},data:a,loc:{start:{line:169,column:16},end:{line:169,column:41}}}))+\"\\n \\n\"},25:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" (\"+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"client_id\")||(null!=e?o(e,\"client_id\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"client_id\",hash:{},data:a,loc:{start:{line:140,column:79},end:{line:140,column:92}}}):o)+\")\"},27:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\"\"+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"user_agent\")||(null!=e?o(e,\"user_agent\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"user_agent\",hash:{},data:a,loc:{start:{line:141,column:48},end:{line:141,column:62}}}):o)+\"\"},29:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' Sampled '+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"sample_rate\")||(null!=e?o(e,\"sample_rate\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"sample_rate\",hash:{},data:a,loc:{start:{line:145,column:59},end:{line:145,column:74}}}):o)+\"%\\n\"},31:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' TLS\\n\"},32:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=\"function\",s=n.escapeExpression,c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'title=\"'+s(typeof(n=null!=(n=c(t,\"tls_version\")||(null!=e?c(e,\"tls_version\"):e))?n:r)==i?n.call(o,{name:\"tls_version\",hash:{},data:a,loc:{start:{line:148,column:80},end:{line:148,column:95}}}):n)+\" \"+s(typeof(n=null!=(n=c(t,\"tls_cipher_suite\")||(null!=e?c(e,\"tls_cipher_suite\"):e))?n:r)==i?n.call(o,{name:\"tls_cipher_suite\",hash:{},data:a,loc:{start:{line:148,column:96},end:{line:148,column:116}}}):n)+\" \"+s(typeof(n=null!=(n=c(t,\"tls_negotiated_protocol\")||(null!=e?c(e,\"tls_negotiated_protocol\"):e))?n:r)==i?n.call(o,{name:\"tls_negotiated_protocol\",hash:{},data:a,loc:{start:{line:148,column:117},end:{line:148,column:144}}}):n)+\" mutual:\"+s(typeof(n=null!=(n=c(t,\"tls_negotiated_protocol_is_mutual\")||(null!=e?c(e,\"tls_negotiated_protocol_is_mutual\"):e))?n:r)==i?n.call(o,{name:\"tls_negotiated_protocol_is_mutual\",hash:{},data:a,loc:{start:{line:148,column:152},end:{line:148,column:189}}}):n)+'\"'},34:function(n,e,t,l,a){return' Deflate\\n'},36:function(n,e,t,l,a){return' Snappy\\n'},38:function(n,e,t,l,a){var o,r=null!=e?e:n.nullContext||{},i=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n '+(null!=(o=i(t,\"if\").call(r,null!=e?i(e,\"auth_identity_url\"):e,{name:\"if\",hash:{},fn:n.program(39,a,0),inverse:n.noop,data:a,loc:{start:{line:158,column:20},end:{line:158,column:84}}}))?o:\"\")+'\\n \\n '+(null!=(o=i(t,\"if\").call(r,null!=e?i(e,\"auth_identity_url\"):e,{name:\"if\",hash:{},fn:n.program(43,a,0),inverse:n.noop,data:a,loc:{start:{line:160,column:20},end:{line:160,column:56}}}))?o:\"\")+\"\\n \\n\"},39:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},41:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" Identity:\"+n.escapeExpression(\"function\"==typeof(o=null!=(o=o(t,\"auth_identity\")||(null!=e?o(e,\"auth_identity\"):e))?o:n.hooks.helperMissing)?o.call(null!=e?e:n.nullContext||{},{name:\"auth_identity\",hash:{},data:a,loc:{start:{line:159,column:109},end:{line:159,column:126}}}):o)},43:function(n,e,t,l,a){return\"\"},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=n.escapeExpression,h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return(null!=(i=n.invokePartial(h(l,\"warning\"),e,{name:\"warning\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+(null!=(i=n.invokePartial(h(l,\"error\"),e,{name:\"error\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+'\\n
      \\n
    1. Nodes\\n
    2. '+u(\"function\"==typeof(u=null!=(u=h(t,\"name\")||(null!=e?h(e,\"name\"):e))?u:c)?u.call(s,{name:\"name\",hash:{},data:a,loc:{start:{line:6,column:23},end:{line:6,column:31}}}):u)+\"
      3. \\n
    \\n\\n\"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(1,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:9,column:0},end:{line:34,column:7}}}))?i:\"\")+'\\n
    \\n
    \\n'+(null!=(i=h(t,\"unless\").call(s,null!=(i=null!=e?h(e,\"topics\"):e)?h(i,\"length\"):i,{name:\"unless\",hash:{},fn:n.program(3,a,0,o,r),inverse:n.program(5,a,0,o,r),data:a,loc:{start:{line:38,column:4},end:{line:177,column:15}}}))?i:\"\")+\"
    \\n\"},usePartial:!0,useData:!0,useDepths:!0})},{\"hbsfy/runtime\":35}],59:[function(e,n,t){var l=e(\"../lib/pubsub\"),a=e(\"../app_state\"),o=e(\"./base\"),r=o.extend({className:\"node container-fluid\",template:e(\"./spinner.hbs\"),initialize:function(){o.prototype.initialize.apply(this,arguments),this.listenTo(a,\"change:graph_interval\",this.render),this.model.fetch().done(function(n){this.template=e(\"./node.hbs\"),this.render({message:n.message})}.bind(this)).fail(this.handleViewError.bind(this)).always(l.trigger.bind(l,\"view:ready\"))}});n.exports=r},{\"../app_state\":36,\"../lib/pubsub\":41,\"./base\":48,\"./node.hbs\":58,\"./spinner.hbs\":62}],60:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a){return\" Lookupd Conns.\\n\"},3:function(n,e,t,l,a,o,r){var i,s,c=null!=e?e:n.nullContext||{},u=n.hooks.helperMissing,h=\"function\",p=n.escapeExpression,d=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" \\n \"+p(typeof(s=null!=(s=d(t,\"hostname\")||(null!=e?d(e,\"hostname\"):e))?s:u)==h?s.call(c,{name:\"hostname\",hash:{},data:a,loc:{start:{line:28,column:20},end:{line:28,column:32}}}):s)+'\\n '+p(typeof(s=null!=(s=d(t,\"broadcast_address\")||(null!=e?d(e,\"broadcast_address\"):e))?s:u)==h?s.call(c,{name:\"broadcast_address\",hash:{},data:a,loc:{start:{line:29,column:92},end:{line:29,column:113}}}):s)+\"\\n \"+p(typeof(s=null!=(s=d(t,\"tcp_port\")||(null!=e?d(e,\"tcp_port\"):e))?s:u)==h?s.call(c,{name:\"tcp_port\",hash:{},data:a,loc:{start:{line:30,column:20},end:{line:30,column:32}}}):s)+\"\\n \"+p(typeof(s=null!=(s=d(t,\"http_port\")||(null!=e?d(e,\"http_port\"):e))?s:u)==h?s.call(c,{name:\"http_port\",hash:{},data:a,loc:{start:{line:31,column:20},end:{line:31,column:33}}}):s)+\"\\n \"+p(typeof(s=null!=(s=d(t,\"version\")||(null!=e?d(e,\"version\"):e))?s:u)==h?s.call(c,{name:\"version\",hash:{},data:a,loc:{start:{line:32,column:20},end:{line:32,column:31}}}):s)+\"\\n \"+p(typeof(s=null!=(s=d(t,\"topology_region\")||(null!=e?d(e,\"topology_region\"):e))?s:u)==h?s.call(c,{name:\"topology_region\",hash:{},data:a,loc:{start:{line:33,column:20},end:{line:33,column:39}}}):s)+\"\\n \"+p(typeof(s=null!=(s=d(t,\"topology_zone\")||(null!=e?d(e,\"topology_zone\"):e))?s:u)==h?s.call(c,{name:\"topology_zone\",hash:{},data:a,loc:{start:{line:34,column:20},end:{line:34,column:37}}}):s)+\"\\n\"+(null!=(i=d(t,\"if\").call(c,null!=(i=null!=r[1]?d(r[1],\"nsqlookupd\"):r[1])?d(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(6,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:35,column:16},end:{line:42,column:23}}}))?i:\"\")+\" \\n\"+(null!=(i=d(t,\"if\").call(c,null!=(i=null!=e?d(e,\"topics\"):e)?d(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(11,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:44,column:16},end:{line:49,column:23}}}))?i:\"\")+\" \\n \\n\"},4:function(n,e,t,l,a){return'class=\"warning\"'},6:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n '+n.escapeExpression(n.lambda(null!=(i=null!=e?c(e,\"remote_addresses\"):e)?c(i,\"length\"):i,e))+'\\n
    \\n '+(null!=(i=c(t,\"each\").call(s,null!=e?c(e,\"remote_addresses\"):e,{name:\"each\",hash:{},fn:n.program(9,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:39,column:24},end:{line:39,column:72}}}))?i:\"\")+\"\\n
    \\n \\n\"},7:function(n,e,t,l,a){return\"btn-warning\"},9:function(n,e,t,l,a){return n.escapeExpression(n.lambda(e,e))+\"
    \"},11:function(n,e,t,l,a){var o,r=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' '+n.escapeExpression(n.lambda(null!=(o=null!=e?r(e,\"topics\"):e)?r(o,\"length\"):o,e))+\"\\n\"+(null!=(o=r(t,\"each\").call(null!=e?e:n.nullContext||{},null!=e?r(e,\"topics\"):e,{name:\"each\",hash:{},fn:n.program(12,a,0),inverse:n.noop,data:a,loc:{start:{line:46,column:20},end:{line:48,column:29}}}))?o:\"\")},12:function(n,e,t,l,a){var o,r,i=null!=e?e:n.nullContext||{},s=n.hooks.helperMissing,c=n.escapeExpression,u=\"function\",h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \"+c(typeof(r=null!=(r=h(t,\"topic\")||(null!=e?h(e,\"topic\"):e))?r:s)==u?r.call(i,{name:\"topic\",hash:{},data:a,loc:{start:{line:47,column:223},end:{line:47,column:232}}}):r)+\"\\n\"},13:function(n,e,t,l,a){return\"label-warning\"},15:function(n,e,t,l,a){return\"label-primary\"},17:function(n,e,t,l,a){return'title=\"this topic is currently tombstoned on this node\"'},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return(null!=(i=n.invokePartial(c(l,\"warning\"),e,{name:\"warning\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+(null!=(i=n.invokePartial(c(l,\"error\"),e,{name:\"error\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+'\\n
    \\n
    \\n

    NSQd Nodes ('+n.escapeExpression(n.lambda(null!=(i=null!=e?c(e,\"collection\"):e)?c(i,\"length\"):i,e))+')

    \\n
    \\n
    \\n\\n
    \\n
    \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n'+(null!=(i=c(t,\"if\").call(s,null!=(i=null!=e?c(e,\"nsqlookupd\"):e)?c(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(1,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:21,column:16},end:{line:23,column:23}}}))?i:\"\")+\" \\n \\n\"+(null!=(i=c(t,\"each\").call(s,null!=e?c(e,\"collection\"):e,{name:\"each\",hash:{},fn:n.program(3,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:26,column:12},end:{line:52,column:21}}}))?i:\"\")+\"
    HostnameBroadcast AddressTCP PortHTTP PortVersionRegionZoneTopics
    \\n
    \\n
    \\n\"},usePartial:!0,useData:!0,useDepths:!0})},{\"hbsfy/runtime\":35}],61:[function(e,n,t){var l=e(\"jquery\"),a=e(\"../lib/pubsub\"),o=e(\"../app_state\"),r=e(\"./base\"),i=e(\"../collections/nodes\"),s=r.extend({className:\"nodes container-fluid\",template:e(\"./spinner.hbs\"),events:{\"click .conn-count\":\"onClickConnCount\"},initialize:function(){r.prototype.initialize.apply(this,arguments),this.listenTo(o,\"change:graph_interval\",this.render),this.collection=new i,this.collection.fetch().done(function(n){this.template=e(\"./nodes.hbs\"),this.render({message:n.message})}.bind(this)).fail(this.handleViewError.bind(this)).always(a.trigger.bind(a,\"view:ready\"))},onClickConnCount:function(n){n.preventDefault(),l(n.target).next().toggle()}});n.exports=s},{\"../app_state\":36,\"../collections/nodes\":37,\"../lib/pubsub\":41,\"./base\":48,\"./nodes.hbs\":60,\"./spinner.hbs\":62,jquery:void 0}],62:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a){return'
    \\n \\n \\n \\n
    \\n'},useData:!0})},{\"hbsfy/runtime\":35}],63:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
    \\n
    \\n

    Topic Message Queue

    \\n
    \\n

    Notice

    No producers exist for this topic.\\n

    See Lookup for more information.\\n

    \\n
    \\n
    \\n'},3:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=n.escapeExpression,h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"isAdmin\"):e,{name:\"if\",hash:{},fn:n.program(4,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:28,column:0},end:{line:44,column:7}}}))?i:\"\")+'\\n
    \\n
    \\n

    Topic Message Queue

    \\n \\n'+(null!=(i=h(t,\"if\").call(s,null!=(i=null!=(i=null!=e?h(e,\"e2e_processing_latency\"):e)?h(i,\"percentiles\"):i)?h(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(9,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:50,column:8},end:{line:55,column:15}}}))?i:\"\")+\" \\n \\n \\n \\n \\n \"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(14,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:61,column:12},end:{line:61,column:52}}}))?i:\"\")+\"\\n \\n\"+(null!=(i=h(t,\"each\").call(s,null!=(i=null!=e?h(e,\"e2e_processing_latency\"):e)?h(i,\"percentiles\"):i,{name:\"each\",hash:{},fn:n.program(16,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:63,column:12},end:{line:65,column:21}}}))?i:\"\")+\" \\n\"+(null!=(i=h(t,\"each\").call(s,null!=e?h(e,\"nodes\"):e,{name:\"each\",hash:{},fn:n.program(18,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:67,column:8},end:{line:108,column:17}}}))?i:\"\")+' \\n \\n \\n \\n \\n \"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(33,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:114,column:12},end:{line:114,column:110}}}))?i:\"\")+\"\\n \\n\"+(null!=(i=h(t,\"if\").call(s,null!=(i=null!=(i=null!=e?h(e,\"e2e_processing_latency\"):e)?h(i,\"percentiles\"):i)?h(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(27,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:116,column:12},end:{line:122,column:19}}}))?i:\"\")+\" \\n\"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(35,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:124,column:8},end:{line:138,column:15}}}))?i:\"\")+\"
    NSQd HostDepthMemory + DiskMessagesChannels
    Total:'+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:111,column:16},end:{line:111,column:33}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"memory_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:112,column:16},end:{line:112,column:40}}}))+\" + \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"backend_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:112,column:43},end:{line:112,column:68}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"message_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:113,column:16},end:{line:113,column:41}}}))+\"\"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=(i=null!=e?h(e,\"channels\"):e)?h(i,\"length\"):i,{name:\"commafy\",hash:{},data:a,loc:{start:{line:115,column:16},end:{line:115,column:43}}}))+\"
    \\n
    \\n
    \\n\"},4:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
    \\n
    \\n \\n
    \\n
    \\n \\n
    \\n
    \\n'+(null!=(a=o(t,\"if\").call(null!=e?e:n.nullContext||{},null!=e?o(e,\"paused\"):e,{name:\"if\",hash:{},fn:n.program(5,a,0),inverse:n.program(7,a,0),data:a,loc:{start:{line:37,column:8},end:{line:41,column:15}}}))?a:\"\")+\"
    \\n
    \\n\"},5:function(n,e,t,l,a){return' \\n'},7:function(n,e,t,l,a){return' \\n'},9:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n E2E Processing Latency\\n \\n'},10:function(n,e,t,l,a){return\"6\"},12:function(n,e,t,l,a){return\"5\"},14:function(n,e,t,l,a){return\"Rate\"},16:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" \"+i((n(t,\"floatToPercent\")||e&&n(e,\"floatToPercent\")||r).call(o,null!=e?n(e,\"quantile\"):e,{name:\"floatToPercent\",hash:{},data:a,loc:{start:{line:64,column:20},end:{line:64,column:47}}}))+\"\"+i((n(t,\"percSuffix\")||e&&n(e,\"percSuffix\")||r).call(o,null!=e?n(e,\"quantile\"):e,{name:\"percSuffix\",hash:{},data:a,loc:{start:{line:64,column:52},end:{line:64,column:75}}}))+\"\\n\"},18:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=n.escapeExpression,h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n'+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"show_broadcast_address\"):e,{name:\"if\",hash:{},fn:n.program(19,a,0,o,r),inverse:n.program(21,a,0,o,r),data:a,loc:{start:{line:71,column:16},end:{line:75,column:23}}}))?i:\"\")+\" \"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"paused\"):e,{name:\"if\",hash:{},fn:n.program(23,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:76,column:16},end:{line:76,column:85}}}))?i:\"\")+\"\\n \\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:78,column:16},end:{line:78,column:33}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"memory_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:79,column:16},end:{line:79,column:40}}}))+\" + \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"backend_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:79,column:43},end:{line:79,column:68}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"message_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:80,column:16},end:{line:80,column:41}}}))+\"\\n\"+(null!=(i=h(t,\"if\").call(s,null!=r[1]?h(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(25,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:81,column:12},end:{line:83,column:19}}}))?i:\"\")+\" \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=(i=null!=e?h(e,\"channels\"):e)?h(i,\"length\"):i,{name:\"commafy\",hash:{},data:a,loc:{start:{line:84,column:16},end:{line:84,column:48}}}))+\"\\n\"+(null!=(i=h(t,\"if\").call(s,null!=(i=null!=(i=null!=e?h(e,\"e2e_processing_latency\"):e)?h(i,\"percentiles\"):i)?h(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(27,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:85,column:12},end:{line:91,column:19}}}))?i:\"\")+\" \\n\"+(null!=(i=h(t,\"if\").call(s,null!=r[1]?h(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(30,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:93,column:8},end:{line:107,column:15}}}))?i:\"\")},19:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=\"function\",s=n.escapeExpression,c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return\" \"+s(typeof(n=null!=(n=c(t,\"hostname_port\")||(null!=e?c(e,\"hostname_port\"):e))?n:r)==i?n.call(o,{name:\"hostname_port\",hash:{},data:a,loc:{start:{line:72,column:16},end:{line:72,column:33}}}):n)+' ('+s(typeof(n=null!=(n=c(t,\"node\")||(null!=e?c(e,\"node\"):e))?n:r)==i?n.call(o,{name:\"node\",hash:{},data:a,loc:{start:{line:72,column:89},end:{line:72,column:97}}}):n)+\")\\n\"},21:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,s=\"function\",c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' '+i(typeof(n=null!=(n=c(t,\"hostname_port\")||(null!=e?c(e,\"hostname_port\"):e))?n:r)==s?n.call(o,{name:\"hostname_port\",hash:{},data:a,loc:{start:{line:74,column:70},end:{line:74,column:87}}}):n)+\"\\n\"},23:function(n,e,t,l,a){return' paused'},25:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n'},27:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return null!=(e=o(t,\"each\").call(null!=e?e:n.nullContext||{},null!=(e=null!=e?o(e,\"e2e_processing_latency\"):e)?o(e,\"percentiles\"):e,{name:\"each\",hash:{},fn:n.program(28,a,0),inverse:n.noop,data:a,loc:{start:{line:86,column:16},end:{line:90,column:25}}}))?e:\"\"},28:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n '+i((n(t,\"nanotohuman\")||e&&n(e,\"nanotohuman\")||r).call(o,null!=e?n(e,\"average\"):e,{name:\"nanotohuman\",hash:{},data:a,loc:{start:{line:88,column:116},end:{line:88,column:139}}}))+\"\\n \\n\"},30:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n \\n \\n \\n \\n'+(null!=(e=s(t,\"if\").call(o,null!=(e=null!=(e=null!=e?s(e,\"e2e_processing_latency\"):e)?s(e,\"percentiles\"):e)?s(e,\"length\"):e,{name:\"if\",hash:{},fn:n.program(31,a,0),inverse:n.noop,data:a,loc:{start:{line:101,column:16},end:{line:105,column:23}}}))?e:\"\")+\" \\n\"},31:function(n,e,t,l,a){var o=n.escapeExpression,r=null!=e?e:n.nullContext||{},i=n.hooks.helperMissing,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n'},33:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},35:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n \\n \\n \\n \\n'+(null!=(e=s(t,\"if\").call(o,null!=(e=null!=(e=null!=e?s(e,\"e2e_processing_latency\"):e)?s(e,\"percentiles\"):e)?s(e,\"length\"):e,{name:\"if\",hash:{},fn:n.program(36,a,0),inverse:n.noop,data:a,loc:{start:{line:132,column:16},end:{line:136,column:23}}}))?e:\"\")+\" \\n\"},36:function(n,e,t,l,a){var o=n.escapeExpression,r=null!=e?e:n.nullContext||{},i=n.hooks.helperMissing,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n'},38:function(n,e,t,l,a){return'
    \\n

    Channel Message Queues

    \\n
    \\n

    Notice

    No channels exist for this topic.\\n

    Messages will queue at the topic until a channel is created.\\n

    \\n'},40:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
    \\n

    Channel Message Queues

    \\n \\n'+(null!=(i=c(t,\"if\").call(s,null!=(i=null!=(i=null!=e?c(e,\"e2e_processing_latency\"):e)?c(i,\"percentiles\"):i)?c(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(41,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:157,column:12},end:{line:162,column:19}}}))?i:\"\")+\" \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n\"+(null!=(i=c(t,\"each\").call(s,null!=(i=null!=e?c(e,\"e2e_processing_latency\"):e)?c(i,\"percentiles\"):i,{name:\"each\",hash:{},fn:n.program(16,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:173,column:16},end:{line:175,column:25}}}))?i:\"\")+\" \\n\\n\"+(null!=(i=c(t,\"each\").call(s,null!=e?c(e,\"channels\"):e,{name:\"each\",hash:{},fn:n.program(46,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:178,column:12},end:{line:218,column:21}}}))?i:\"\")+\"
    ChannelDepthMemory + DiskIn-FlightDeferredRequeuedTimed OutMessagesConnections
    \\n\"},41:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n E2E Processing Latency\\n \\n'},42:function(n,e,t,l,a){return\"10\"},44:function(n,e,t,l,a){return\"9\"},46:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=n.escapeExpression,h=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n '+u(\"function\"==typeof(i=null!=(i=h(t,\"channel_name\")||(null!=e?h(e,\"channel_name\"):e))?i:c)?i.call(s,{name:\"channel_name\",hash:{},data:a,loc:{start:{line:181,column:118},end:{line:181,column:134}}}):i)+\"\\n \"+(null!=(i=h(t,\"if\").call(s,null!=e?h(e,\"paused\"):e,{name:\"if\",hash:{},fn:n.program(47,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:182,column:20},end:{line:182,column:88}}}))?i:\"\")+\"\\n \\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:184,column:20},end:{line:184,column:37}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"memory_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:185,column:20},end:{line:185,column:44}}}))+\" + \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"backend_depth\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:185,column:47},end:{line:185,column:72}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"in_flight_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:186,column:20},end:{line:186,column:47}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"deferred_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:187,column:20},end:{line:187,column:46}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"requeue_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:188,column:20},end:{line:188,column:45}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"timeout_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:189,column:20},end:{line:189,column:45}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"message_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:190,column:20},end:{line:190,column:45}}}))+\"\\n \"+u((h(t,\"commafy\")||e&&h(e,\"commafy\")||c).call(s,null!=e?h(e,\"client_count\"):e,{name:\"commafy\",hash:{},data:a,loc:{start:{line:191,column:20},end:{line:191,column:44}}}))+\"\\n\"+(null!=(i=h(t,\"if\").call(s,null!=(i=null!=(i=null!=e?h(e,\"e2e_processing_latency\"):e)?h(i,\"percentiles\"):i)?h(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(49,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:192,column:16},end:{line:198,column:23}}}))?i:\"\")+\" \\n\"+(null!=(i=h(t,\"if\").call(s,null!=r[1]?h(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(52,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:200,column:12},end:{line:217,column:19}}}))?i:\"\")},47:function(n,e,t,l,a){return'paused'},49:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return null!=(e=o(t,\"each\").call(null!=e?e:n.nullContext||{},null!=(e=null!=e?o(e,\"e2e_processing_latency\"):e)?o(e,\"percentiles\"):e,{name:\"each\",hash:{},fn:n.program(50,a,0),inverse:n.noop,data:a,loc:{start:{line:193,column:20},end:{line:197,column:27}}}))?e:\"\"},50:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n '+i((n(t,\"nanotohuman\")||e&&n(e,\"nanotohuman\")||r).call(o,null!=e?n(e,\"average\"):e,{name:\"nanotohuman\",hash:{},data:a,loc:{start:{line:195,column:120},end:{line:195,column:143}}}))+\"\\n \\n\"},52:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n'+(null!=(e=s(t,\"if\").call(o,null!=(e=null!=(e=null!=e?s(e,\"e2e_processing_latency\"):e)?s(e,\"percentiles\"):e)?s(e,\"length\"):e,{name:\"if\",hash:{},fn:n.program(53,a,0),inverse:n.noop,data:a,loc:{start:{line:211,column:16},end:{line:215,column:23}}}))?e:\"\")+\" \\n\"},53:function(n,e,t,l,a){var o=n.escapeExpression,r=null!=e?e:n.nullContext||{},i=n.hooks.helperMissing,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n'},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.hooks.helperMissing,u=n.escapeExpression,h=\"function\",p=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return(null!=(i=n.invokePartial(p(l,\"warning\"),e,{name:\"warning\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+(null!=(i=n.invokePartial(p(l,\"error\"),e,{name:\"error\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+'\\n
      \\n
    1. Streams\\n
    2. '+u(typeof(l=null!=(l=p(t,\"name\")||(null!=e?p(e,\"name\"):e))?l:c)==h?l.call(s,{name:\"name\",hash:{},data:a,loc:{start:{line:6,column:21},end:{line:6,column:29}}}):l)+'
      3. \\n
    \\n\\n
    \\n
    \\n
    \\n

    Topic: '+u(typeof(l=null!=(l=p(t,\"name\")||(null!=e?p(e,\"name\"):e))?l:c)==h?l.call(s,{name:\"name\",hash:{},data:a,loc:{start:{line:12,column:30},end:{line:12,column:38}}}):l)+\"\\n

    \\n
    \\n
    \\n\\n\"+(null!=(i=p(t,\"unless\").call(s,null!=(i=null!=e?p(e,\"nodes\"):e)?p(i,\"length\"):i,{name:\"unless\",hash:{},fn:n.program(1,a,0,o,r),inverse:n.program(3,a,0,o,r),data:a,loc:{start:{line:17,column:0},end:{line:142,column:11}}}))?i:\"\")+'\\n\\n
    \\n'+(null!=(i=p(t,\"unless\").call(s,null!=(i=null!=e?p(e,\"channels\"):e)?p(i,\"length\"):i,{name:\"unless\",hash:{},fn:n.program(38,a,0,o,r),inverse:n.program(40,a,0,o,r),data:a,loc:{start:{line:146,column:4},end:{line:220,column:19}}}))?i:\"\")+\"
    \\n
    \\n\"},usePartial:!0,useData:!0,useDepths:!0})},{\"hbsfy/runtime\":35}],64:[function(t,n,e){var l=t(\"jquery\");window.jQuery=l;t(\"bootstrap\");var a=t(\"bootbox\"),o=t(\"../lib/pubsub\"),r=t(\"../app_state\"),i=t(\"./base\"),s=i.extend({className:\"topic container-fluid\",template:t(\"./spinner.hbs\"),events:{\"click .topic-actions button\":\"topicAction\"},initialize:function(){i.prototype.initialize.apply(this,arguments),this.listenTo(r,\"change:graph_interval\",this.render);var e=this.model.get(\"isAdmin\");this.model.fetch().done(function(n){this.template=t(\"./topic.hbs\"),this.render({message:n.message,isAdmin:e})}.bind(this)).fail(this.handleViewError.bind(this)).always(o.trigger.bind(o,\"view:ready\"))},topicAction:function(n){n.preventDefault(),n.stopPropagation();var e=l(n.currentTarget).data(\"action\"),n=\"Are you sure you want to \"+e+\" \"+this.model.get(\"name\")+\"?\";a.confirm(n,function(n){!0===n&&(\"delete\"===e?l.ajax(this.model.url(),{method:\"DELETE\"}).done(function(){window.location=r.basePath(\"/\")}):l.post(this.model.url(),JSON.stringify({action:e})).done(function(){window.location.reload(!0)}).fail(this.handleAJAXError.bind(this)))}.bind(this))}});n.exports=s},{\"../app_state\":36,\"../lib/pubsub\":41,\"./base\":48,\"./spinner.hbs\":62,\"./topic.hbs\":63,bootbox:void 0,bootstrap:1,jquery:void 0}],65:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a,o,r){var i,s=null!=e?e:n.nullContext||{},c=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n \\n \\n '+(null!=(i=c(t,\"if\").call(s,null!=e?c(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(2,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:16,column:16},end:{line:16,column:69}}}))?i:\"\")+\"\\n \"+(null!=(i=c(t,\"if\").call(s,null!=e?c(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(4,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:17,column:16},end:{line:17,column:72}}}))?i:\"\")+\"\\n \"+(null!=(i=c(t,\"if\").call(s,null!=e?c(e,\"graph_active\"):e,{name:\"if\",hash:{},fn:n.program(6,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:18,column:16},end:{line:18,column:68}}}))?i:\"\")+\"\\n \\n\"+(null!=(i=c(t,\"each\").call(s,null!=e?c(e,\"collection\"):e,{name:\"each\",hash:{},fn:n.program(8,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:20,column:12},end:{line:27,column:21}}}))?i:\"\")+\"
    Topic
    \\n\"},2:function(n,e,t,l,a){return'Depth'},4:function(n,e,t,l,a){return'Messages'},6:function(n,e,t,l,a){return'Rate'},8:function(n,e,t,l,a,o,r){var i=null!=e?e:n.nullContext||{},s=n.hooks.helperMissing,c=n.escapeExpression,u=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return' \\n '+c(\"function\"==typeof(e=null!=(e=u(t,\"name\")||(null!=e?u(e,\"name\"):e))?e:s)?e.call(i,{name:\"name\",hash:{},data:a,loc:{start:{line:22,column:85},end:{line:22,column:93}}}):e)+\"\\n \"+(null!=(e=u(t,\"if\").call(i,null!=r[1]?u(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(9,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:23,column:16},end:{line:23,column:200}}}))?e:\"\")+\"\\n \"+(null!=(e=u(t,\"if\").call(i,null!=r[1]?u(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(11,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:24,column:16},end:{line:24,column:208}}}))?e:\"\")+\"\\n \"+(null!=(e=u(t,\"if\").call(i,null!=r[1]?u(r[1],\"graph_active\"):r[1],{name:\"if\",hash:{},fn:n.program(13,a,0,o,r),inverse:n.noop,data:a,loc:{start:{line:25,column:16},end:{line:25,column:111}}}))?e:\"\")+\"\\n \\n\"},9:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},11:function(n,e,t,l,a){var o=null!=e?e:n.nullContext||{},r=n.hooks.helperMissing,i=n.escapeExpression,n=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},13:function(n,e,t,l,a){var o=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return''},15:function(n,e,t,l,a){return'

    Notice

    No Topics Found
    \\n'},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a,o,r){var i,s=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return(null!=(i=n.invokePartial(s(l,\"warning\"),e,{name:\"warning\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+(null!=(i=n.invokePartial(s(l,\"error\"),e,{name:\"error\",data:a,helpers:t,partials:l,decorators:n.decorators}))?i:\"\")+'\\n
    \\n
    \\n

    Topics

    \\n
    \\n
    \\n\\n
    \\n
    \\n'+(null!=(i=s(t,\"if\").call(null!=e?e:n.nullContext||{},null!=(i=null!=e?s(e,\"collection\"):e)?s(i,\"length\"):i,{name:\"if\",hash:{},fn:n.program(1,a,0,o,r),inverse:n.program(15,a,0,o,r),data:a,loc:{start:{line:12,column:4},end:{line:31,column:11}}}))?i:\"\")+\"
    \\n
    \\n\"},usePartial:!0,useData:!0,useDepths:!0})},{\"hbsfy/runtime\":35}],66:[function(e,n,t){var l=e(\"../lib/pubsub\"),a=e(\"../app_state\"),o=e(\"./base\"),r=e(\"../collections/topics\"),i=o.extend({className:\"topics container-fluid\",template:e(\"./spinner.hbs\"),initialize:function(){o.prototype.initialize.apply(this,arguments),this.listenTo(a,\"change:graph_interval\",this.render),this.collection=new r,this.collection.fetch().done(function(n){this.template=e(\"./topics.hbs\"),this.render({message:n.message})}.bind(this)).fail(this.handleViewError.bind(this)).always(l.trigger.bind(l,\"view:ready\"))}});n.exports=i},{\"../app_state\":36,\"../collections/topics\":38,\"../lib/pubsub\":41,\"./base\":48,\"./spinner.hbs\":62,\"./topics.hbs\":65}],67:[function(n,e,t){n=n(\"hbsfy/runtime\");e.exports=n.template({1:function(n,e,t,l,a){return'style=\"display: none;\"'},compiler:[8,\">= 4.3.0\"],main:function(n,e,t,l,a){var o,r=null!=e?e:n.nullContext||{},i=n.lookupProperty||function(n,e){if(Object.prototype.hasOwnProperty.call(n,e))return n[e]};return'
    \\n
    \\n
    \\n '+n.escapeExpression(\"function\"==typeof(e=null!=(e=i(t,\"message\")||(null!=e?i(e,\"message\"):e))?e:n.hooks.helperMissing)?e.call(r,{name:\"message\",hash:{},data:a,loc:{start:{line:4,column:12},end:{line:4,column:23}}}):e)+\"\\n
    \\n
    \\n
    \\n\"},useData:!0})},{\"hbsfy/runtime\":35}]},{},[42]);\n//# sourceMappingURL=main.js.map\n" }, { "path": "nsqadmin/static/build/vendor.js", "content": "require=function r(i,o,a){function s(t,e){if(!o[t]){if(!i[t]){var n=\"function\"==typeof require&&require;if(!e&&n)return n(t,!0);if(u)return u(t,!0);throw(n=new Error(\"Cannot find module '\"+t+\"'\")).code=\"MODULE_NOT_FOUND\",n}n=o[t]={exports:{}},i[t][0].call(n.exports,function(e){return s(i[t][1][e]||e)},n,n.exports,r,i,o,a)}return o[t].exports}for(var u=\"function\"==typeof require&&require,e=0;ethis.length?this.length:r)<0&&(r+=this.length+1);for(var i=[],o=[],a=[],s=[],u={},l=t.add,c=t.merge,f=t.remove,p=!1,d=this.comparator&&null==r&&!1!==t.sort,h=x.isString(this.comparator)?this.comparator:null,g=0;g
    ',header:'
    ',footer:'
    ',closeButton:'',form:'
    ',button:'',option:\"\",promptMessage:'
    ',inputs:{text:'',textarea:'',email:'',select:'',checkbox:'
    ',radio:'
    ',date:'',time:'',number:'',password:'',range:''}},h={locale:\"en\",backdrop:\"static\",animate:!0,className:null,closeButton:!0,show:!0,container:\"body\",value:\"\",inputType:\"text\",swapButtonOrder:!1,centerVertical:!1,multiple:!1,scrollable:!1,reusable:!1};function l(e,t,n){return c.extend(!0,{},e,function(e,t){var n=e.length,r={};if(n<1||2\").attr(\"label\",t.group)),n=o[t.group]);var r=c(d.option);r.attr(\"value\",t.value).text(t.text),n.append(r)}),m(o,function(e,t){u.append(t)}),u.val(i.value);break;case\"checkbox\":var s=c.isArray(i.value)?i.value:[i.value];if(!(a=i.inputOptions||[]).length)throw new Error('prompt with \"inputType\" set to \"checkbox\" requires at least one option');u=c('
    '),m(a,function(e,n){if(n.value===f||n.text===f)throw new Error('each option needs a \"value\" property and a \"text\" property');var r=c(d.inputs[i.inputType]);r.find(\"input\").attr(\"value\",n.value),r.find(\"label\").append(\"\\n\"+n.text),m(s,function(e,t){t===n.value&&r.find(\"input\").prop(\"checked\",!0)}),u.append(r)});break;case\"radio\":if(i.value!==f&&c.isArray(i.value))throw new Error('prompt with \"inputType\" set to \"radio\" requires a single, non-array value for \"value\"');if(!(a=i.inputOptions||[]).length)throw new Error('prompt with \"inputType\" set to \"radio\" requires at least one option');var u=c('
    '),l=!0;m(a,function(e,t){if(t.value===f||t.text===f)throw new Error('each option needs a \"value\" property and a \"text\" property');var n=c(d.inputs[i.inputType]);n.find(\"input\").attr(\"value\",t.value),n.find(\"label\").append(\"\\n\"+t.text),i.value!==f&&t.value===i.value&&(n.find(\"input\").prop(\"checked\",!0),l=!1),u.append(n)}),l&&u.find('input[type=\"radio\"]').first().prop(\"checked\",!0)}return r.append(u),r.on(\"submit\",function(e){e.preventDefault(),e.stopPropagation(),t.find(\".bootbox-accept\").trigger(\"click\")}),\"\"!==c.trim(i.message)&&(n=c(d.promptMessage).html(i.message),r.prepend(n)),i.message=r,(t=p.dialog(i)).off(\"shown.bs.modal\",y),t.on(\"shown.bs.modal\",function(){u.focus()}),!0===e&&t.modal(\"show\"),t},p})},{jquery:\"jquery\"}],jquery:[function(e,n,t){!function(e,t){\"use strict\";\"object\"==typeof n&&\"object\"==typeof n.exports?n.exports=e.document?t(e,!0):function(e){if(!e.document)throw new Error(\"jQuery requires a window with a document\");return t(e)}:t(e)}(\"undefined\"!=typeof window?window:this,function(C,e){\"use strict\";function g(e){return null!=e&&e===e.window}var t=[],n=Object.getPrototypeOf,s=t.slice,v=t.flat?function(e){return t.flat.call(e)}:function(e){return t.concat.apply([],e)},u=t.push,i=t.indexOf,r={},o=r.toString,m=r.hasOwnProperty,a=m.toString,l=a.call(Object),y={},b=function(e){return\"function\"==typeof e&&\"number\"!=typeof e.nodeType&&\"function\"!=typeof e.item},E=C.document,c={type:!0,src:!0,nonce:!0,noModule:!0};function x(e,t,n){var r,i,o=(n=n||E).createElement(\"script\");if(o.text=e,t)for(r in c)(i=t[r]||t.getAttribute&&t.getAttribute(r))&&o.setAttribute(r,i);n.head.appendChild(o).parentNode.removeChild(o)}function h(e){return null==e?e+\"\":\"object\"==typeof e||\"function\"==typeof e?r[o.call(e)]||\"object\":typeof e}var T=function(e,t){return new T.fn.init(e,t)};function f(e){var t=!!e&&\"length\"in e&&e.length,n=h(e);return!b(e)&&!g(e)&&(\"array\"===n||0===t||\"number\"==typeof t&&0>10|55296,1023&e|56320))}function r(){C()}var e,p,x,o,i,d,h,g,w,u,l,C,E,a,T,v,s,c,m,k=\"sizzle\"+ +new Date,y=n.document,N=0,b=0,A=ue(),O=ue(),S=ue(),j=ue(),_=function(e,t){return e===t&&(l=!0),0},L={}.hasOwnProperty,t=[],I=t.pop,D=t.push,M=t.push,q=t.slice,R=function(e,t){for(var n=0,r=e.length;n+~]|\"+P+\")\"+P+\"*\"),V=new RegExp(P+\"|>\"),X=new RegExp(K),Y=new RegExp(\"^\"+F+\"$\"),J={ID:new RegExp(\"^#(\"+F+\")\"),CLASS:new RegExp(\"^\\\\.(\"+F+\")\"),TAG:new RegExp(\"^(\"+F+\"|[*])\"),ATTR:new RegExp(\"^\"+B),PSEUDO:new RegExp(\"^\"+K),CHILD:new RegExp(\"^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\\\(\"+P+\"*(even|odd|(([+-]|)(\\\\d*)n|)\"+P+\"*(?:([+-]|)\"+P+\"*(\\\\d+)|))\"+P+\"*\\\\)|)\",\"i\"),bool:new RegExp(\"^(?:\"+H+\")$\",\"i\"),needsContext:new RegExp(\"^\"+P+\"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\\\(\"+P+\"*((?:-\\\\d)?\\\\d*)\"+P+\"*\\\\)|)(?=[^-]|$)\",\"i\")},G=/HTML$/i,Q=/^(?:input|select|textarea|button)$/i,Z=/^h\\d$/i,ee=/^[^{]+\\{\\s*\\[native \\w/,te=/^(?:#([\\w-]+)|(\\w+)|\\.([\\w-]+))$/,ne=/[+~]/,re=new RegExp(\"\\\\\\\\[\\\\da-fA-F]{1,6}\"+P+\"?|\\\\\\\\([^\\\\r\\\\n\\\\f])\",\"g\"),ie=/([\\0-\\x1f\\x7f]|^-?\\d)|^-$|[^\\0-\\x1f\\x7f-\\uFFFF\\w-]/g,oe=function(e,t){return t?\"\\0\"===e?\"�\":e.slice(0,-1)+\"\\\\\"+e.charCodeAt(e.length-1).toString(16)+\" \":\"\\\\\"+e},ae=ye(function(e){return!0===e.disabled&&\"fieldset\"===e.nodeName.toLowerCase()},{dir:\"parentNode\",next:\"legend\"});try{M.apply(t=q.call(y.childNodes),y.childNodes),t[y.childNodes.length].nodeType}catch(e){M={apply:t.length?function(e,t){D.apply(e,q.call(t))}:function(e,t){for(var n=e.length,r=0;e[n++]=t[r++];);e.length=n-1}}}function se(t,e,n,r){var i,o,a,s,u,l,c=e&&e.ownerDocument,f=e?e.nodeType:9;if(n=n||[],\"string\"!=typeof t||!t||1!==f&&9!==f&&11!==f)return n;if(!r&&(C(e),e=e||E,T)){if(11!==f&&(s=te.exec(t)))if(l=s[1]){if(9===f){if(!(o=e.getElementById(l)))return n;if(o.id===l)return n.push(o),n}else if(c&&(o=c.getElementById(l))&&m(e,o)&&o.id===l)return n.push(o),n}else{if(s[2])return M.apply(n,e.getElementsByTagName(t)),n;if((l=s[3])&&p.getElementsByClassName&&e.getElementsByClassName)return M.apply(n,e.getElementsByClassName(l)),n}if(p.qsa&&!j[t+\" \"]&&(!v||!v.test(t))&&(1!==f||\"object\"!==e.nodeName.toLowerCase())){if(l=t,c=e,1===f&&(V.test(t)||U.test(t))){for((c=ne.test(t)&&ge(e.parentNode)||e)===e&&p.scope||((a=e.getAttribute(\"id\"))?a=a.replace(ie,oe):e.setAttribute(\"id\",a=k)),i=(u=d(t)).length;i--;)u[i]=(a?\"#\"+a:\":scope\")+\" \"+me(u[i]);l=u.join(\",\")}try{return M.apply(n,c.querySelectorAll(l)),n}catch(e){j(t,!0)}finally{a===k&&e.removeAttribute(\"id\")}}}return g(t.replace(W,\"$1\"),e,n,r)}function ue(){var n=[];function r(e,t){return n.push(e+\" \")>x.cacheLength&&delete r[n.shift()],r[e+\" \"]=t}return r}function le(e){return e[k]=!0,e}function ce(e){var t=E.createElement(\"fieldset\");try{return!!e(t)}catch(e){return!1}finally{t.parentNode&&t.parentNode.removeChild(t),t=null}}function fe(e,t){for(var n=e.split(\"|\"),r=n.length;r--;)x.attrHandle[n[r]]=t}function pe(e,t){var n=t&&e,r=n&&1===e.nodeType&&1===t.nodeType&&e.sourceIndex-t.sourceIndex;if(r)return r;if(n)for(;n=n.nextSibling;)if(n===t)return-1;return e?1:-1}function de(t){return function(e){return\"form\"in e?e.parentNode&&!1===e.disabled?\"label\"in e?\"label\"in e.parentNode?e.parentNode.disabled===t:e.disabled===t:e.isDisabled===t||e.isDisabled!==!t&&ae(e)===t:e.disabled===t:\"label\"in e&&e.disabled===t}}function he(a){return le(function(o){return o=+o,le(function(e,t){for(var n,r=a([],e.length,o),i=r.length;i--;)e[n=r[i]]&&(e[n]=!(t[n]=e[n]))})})}function ge(e){return e&&void 0!==e.getElementsByTagName&&e}for(e in p=se.support={},i=se.isXML=function(e){var t=e&&e.namespaceURI,e=e&&(e.ownerDocument||e).documentElement;return!G.test(t||e&&e.nodeName||\"HTML\")},C=se.setDocument=function(e){var t,e=e?e.ownerDocument||e:y;return e!=E&&9===e.nodeType&&e.documentElement&&(a=(E=e).documentElement,T=!i(E),y!=E&&(t=E.defaultView)&&t.top!==t&&(t.addEventListener?t.addEventListener(\"unload\",r,!1):t.attachEvent&&t.attachEvent(\"onunload\",r)),p.scope=ce(function(e){return a.appendChild(e).appendChild(E.createElement(\"div\")),void 0!==e.querySelectorAll&&!e.querySelectorAll(\":scope fieldset div\").length}),p.attributes=ce(function(e){return e.className=\"i\",!e.getAttribute(\"className\")}),p.getElementsByTagName=ce(function(e){return e.appendChild(E.createComment(\"\")),!e.getElementsByTagName(\"*\").length}),p.getElementsByClassName=ee.test(E.getElementsByClassName),p.getById=ce(function(e){return a.appendChild(e).id=k,!E.getElementsByName||!E.getElementsByName(k).length}),p.getById?(x.filter.ID=function(e){var t=e.replace(re,f);return function(e){return e.getAttribute(\"id\")===t}},x.find.ID=function(e,t){if(void 0!==t.getElementById&&T){e=t.getElementById(e);return e?[e]:[]}}):(x.filter.ID=function(e){var t=e.replace(re,f);return function(e){e=void 0!==e.getAttributeNode&&e.getAttributeNode(\"id\");return e&&e.value===t}},x.find.ID=function(e,t){if(void 0!==t.getElementById&&T){var n,r,i,o=t.getElementById(e);if(o){if((n=o.getAttributeNode(\"id\"))&&n.value===e)return[o];for(i=t.getElementsByName(e),r=0;o=i[r++];)if((n=o.getAttributeNode(\"id\"))&&n.value===e)return[o]}return[]}}),x.find.TAG=p.getElementsByTagName?function(e,t){return void 0!==t.getElementsByTagName?t.getElementsByTagName(e):p.qsa?t.querySelectorAll(e):void 0}:function(e,t){var n,r=[],i=0,o=t.getElementsByTagName(e);if(\"*\"!==e)return o;for(;n=o[i++];)1===n.nodeType&&r.push(n);return r},x.find.CLASS=p.getElementsByClassName&&function(e,t){if(void 0!==t.getElementsByClassName&&T)return t.getElementsByClassName(e)},s=[],v=[],(p.qsa=ee.test(E.querySelectorAll))&&(ce(function(e){var t;a.appendChild(e).innerHTML=\"\",e.querySelectorAll(\"[msallowcapture^='']\").length&&v.push(\"[*^$]=\"+P+\"*(?:''|\\\"\\\")\"),e.querySelectorAll(\"[selected]\").length||v.push(\"\\\\[\"+P+\"*(?:value|\"+H+\")\"),e.querySelectorAll(\"[id~=\"+k+\"-]\").length||v.push(\"~=\"),(t=E.createElement(\"input\")).setAttribute(\"name\",\"\"),e.appendChild(t),e.querySelectorAll(\"[name='']\").length||v.push(\"\\\\[\"+P+\"*name\"+P+\"*=\"+P+\"*(?:''|\\\"\\\")\"),e.querySelectorAll(\":checked\").length||v.push(\":checked\"),e.querySelectorAll(\"a#\"+k+\"+*\").length||v.push(\".#.+[+~]\"),e.querySelectorAll(\"\\\\\\f\"),v.push(\"[\\\\r\\\\n\\\\f]\")}),ce(function(e){e.innerHTML=\"\";var t=E.createElement(\"input\");t.setAttribute(\"type\",\"hidden\"),e.appendChild(t).setAttribute(\"name\",\"D\"),e.querySelectorAll(\"[name=d]\").length&&v.push(\"name\"+P+\"*[*^$|!~]?=\"),2!==e.querySelectorAll(\":enabled\").length&&v.push(\":enabled\",\":disabled\"),a.appendChild(e).disabled=!0,2!==e.querySelectorAll(\":disabled\").length&&v.push(\":enabled\",\":disabled\"),e.querySelectorAll(\"*,:x\"),v.push(\",.*:\")})),(p.matchesSelector=ee.test(c=a.matches||a.webkitMatchesSelector||a.mozMatchesSelector||a.oMatchesSelector||a.msMatchesSelector))&&ce(function(e){p.disconnectedMatch=c.call(e,\"*\"),c.call(e,\"[s!='']:x\"),s.push(\"!=\",K)}),v=v.length&&new RegExp(v.join(\"|\")),s=s.length&&new RegExp(s.join(\"|\")),t=ee.test(a.compareDocumentPosition),m=t||ee.test(a.contains)?function(e,t){var n=9===e.nodeType?e.documentElement:e,t=t&&t.parentNode;return e===t||!(!t||1!==t.nodeType||!(n.contains?n.contains(t):e.compareDocumentPosition&&16&e.compareDocumentPosition(t)))}:function(e,t){if(t)for(;t=t.parentNode;)if(t===e)return!0;return!1},_=t?function(e,t){if(e===t)return l=!0,0;var n=!e.compareDocumentPosition-!t.compareDocumentPosition;return n||(1&(n=(e.ownerDocument||e)==(t.ownerDocument||t)?e.compareDocumentPosition(t):1)||!p.sortDetached&&t.compareDocumentPosition(e)===n?e==E||e.ownerDocument==y&&m(y,e)?-1:t==E||t.ownerDocument==y&&m(y,t)?1:u?R(u,e)-R(u,t):0:4&n?-1:1)}:function(e,t){if(e===t)return l=!0,0;var n,r=0,i=e.parentNode,o=t.parentNode,a=[e],s=[t];if(!i||!o)return e==E?-1:t==E?1:i?-1:o?1:u?R(u,e)-R(u,t):0;if(i===o)return pe(e,t);for(n=e;n=n.parentNode;)a.unshift(n);for(n=t;n=n.parentNode;)s.unshift(n);for(;a[r]===s[r];)r++;return r?pe(a[r],s[r]):a[r]==y?-1:s[r]==y?1:0}),E},se.matches=function(e,t){return se(e,null,null,t)},se.matchesSelector=function(e,t){if(C(e),p.matchesSelector&&T&&!j[t+\" \"]&&(!s||!s.test(t))&&(!v||!v.test(t)))try{var n=c.call(e,t);if(n||p.disconnectedMatch||e.document&&11!==e.document.nodeType)return n}catch(e){j(t,!0)}return 0\":{dir:\"parentNode\",first:!0},\" \":{dir:\"parentNode\"},\"+\":{dir:\"previousSibling\",first:!0},\"~\":{dir:\"previousSibling\"}},preFilter:{ATTR:function(e){return e[1]=e[1].replace(re,f),e[3]=(e[3]||e[4]||e[5]||\"\").replace(re,f),\"~=\"===e[2]&&(e[3]=\" \"+e[3]+\" \"),e.slice(0,4)},CHILD:function(e){return e[1]=e[1].toLowerCase(),\"nth\"===e[1].slice(0,3)?(e[3]||se.error(e[0]),e[4]=+(e[4]?e[5]+(e[6]||1):2*(\"even\"===e[3]||\"odd\"===e[3])),e[5]=+(e[7]+e[8]||\"odd\"===e[3])):e[3]&&se.error(e[0]),e},PSEUDO:function(e){var t,n=!e[6]&&e[2];return J.CHILD.test(e[0])?null:(e[3]?e[2]=e[4]||e[5]||\"\":n&&X.test(n)&&(t=d(n,!0))&&(t=n.indexOf(\")\",n.length-t)-n.length)&&(e[0]=e[0].slice(0,t),e[2]=n.slice(0,t)),e.slice(0,3))}},filter:{TAG:function(e){var t=e.replace(re,f).toLowerCase();return\"*\"===e?function(){return!0}:function(e){return e.nodeName&&e.nodeName.toLowerCase()===t}},CLASS:function(e){var t=A[e+\" \"];return t||(t=new RegExp(\"(^|\"+P+\")\"+e+\"(\"+P+\"|$)\"))&&A(e,function(e){return t.test(\"string\"==typeof e.className&&e.className||void 0!==e.getAttribute&&e.getAttribute(\"class\")||\"\")})},ATTR:function(t,n,r){return function(e){e=se.attr(e,t);return null==e?\"!=\"===n:!n||(e+=\"\",\"=\"===n?e===r:\"!=\"===n?e!==r:\"^=\"===n?r&&0===e.indexOf(r):\"*=\"===n?r&&-1\",\"#\"===e.firstChild.getAttribute(\"href\")})||fe(\"type|href|height|width\",function(e,t,n){if(!n)return e.getAttribute(t,\"type\"===t.toLowerCase()?1:2)}),p.attributes&&ce(function(e){return e.innerHTML=\"\",e.firstChild.setAttribute(\"value\",\"\"),\"\"===e.firstChild.getAttribute(\"value\")})||fe(\"value\",function(e,t,n){if(!n&&\"input\"===e.nodeName.toLowerCase())return e.defaultValue}),ce(function(e){return null==e.getAttribute(\"disabled\")})||fe(H,function(e,t,n){if(!n)return!0===e[t]?t.toLowerCase():(t=e.getAttributeNode(t))&&t.specified?t.value:null}),se}(C);T.find=p,T.expr=p.selectors,T.expr[\":\"]=T.expr.pseudos,T.uniqueSort=T.unique=p.uniqueSort,T.text=p.getText,T.isXMLDoc=p.isXML,T.contains=p.contains,T.escapeSelector=p.escape;function d(e,t,n){for(var r=[],i=void 0!==n;(e=e[t])&&9!==e.nodeType;)if(1===e.nodeType){if(i&&T(e).is(n))break;r.push(e)}return r}function w(e,t){for(var n=[];e;e=e.nextSibling)1===e.nodeType&&e!==t&&n.push(e);return n}var k=T.expr.match.needsContext;function N(e,t){return e.nodeName&&e.nodeName.toLowerCase()===t.toLowerCase()}var A=/^<([a-z][^\\/\\0>:\\x20\\t\\r\\n\\f]*)[\\x20\\t\\r\\n\\f]*\\/?>(?:<\\/\\1>|)$/i;function O(e,n,r){return b(n)?T.grep(e,function(e,t){return!!n.call(e,t,e)!==r}):n.nodeType?T.grep(e,function(e){return e===n!==r}):\"string\"!=typeof n?T.grep(e,function(e){return-1)[^>]*|#([\\w-]+))$/;(T.fn.init=function(e,t,n){if(!e)return this;if(n=n||j,\"string\"!=typeof e)return e.nodeType?(this[0]=e,this.length=1,this):b(e)?void 0!==n.ready?n.ready(e):e(T):T.makeArray(e,this);if(!(r=\"<\"===e[0]&&\">\"===e[e.length-1]&&3<=e.length?[null,e,null]:S.exec(e))||!r[1]&&t)return(!t||t.jquery?t||n:this.constructor(t)).find(e);if(r[1]){if(t=t instanceof T?t[0]:t,T.merge(this,T.parseHTML(r[1],t&&t.nodeType?t.ownerDocument||t:E,!0)),A.test(r[1])&&T.isPlainObject(t))for(var r in t)b(this[r])?this[r](t[r]):this.attr(r,t[r]);return this}return(e=E.getElementById(r[2]))&&(this[0]=e,this.length=1),this}).prototype=T.fn;var j=T(E),_=/^(?:parents|prev(?:Until|All))/,L={children:!0,contents:!0,next:!0,prev:!0};function I(e,t){for(;(e=e[t])&&1!==e.nodeType;);return e}T.fn.extend({has:function(e){var t=T(e,this),n=t.length;return this.filter(function(){for(var e=0;e\\x20\\t\\r\\n\\f]*)/i,fe=/^$|^module$|\\/(?:java|ecma)script/i;Vt=E.createDocumentFragment().appendChild(E.createElement(\"div\")),(p=E.createElement(\"input\")).setAttribute(\"type\",\"radio\"),p.setAttribute(\"checked\",\"checked\"),p.setAttribute(\"name\",\"t\"),Vt.appendChild(p),y.checkClone=Vt.cloneNode(!0).cloneNode(!0).lastChild.checked,Vt.innerHTML=\"\",y.noCloneChecked=!!Vt.cloneNode(!0).lastChild.defaultValue,Vt.innerHTML=\"\",y.option=!!Vt.lastChild;var pe={thead:[1,\"\",\"
    \"],col:[2,\"\",\"
    \"],tr:[2,\"\",\"
    \"],td:[3,\"\",\"
    \"],_default:[0,\"\",\"\"]};function de(e,t){var n=void 0!==e.getElementsByTagName?e.getElementsByTagName(t||\"*\"):void 0!==e.querySelectorAll?e.querySelectorAll(t||\"*\"):[];return void 0===t||t&&N(e,t)?T.merge([e],n):n}function he(e,t){for(var n=0,r=e.length;n\",\"\"]);var ge=/<|&#?\\w+;/;function ve(e,t,n,r,i){for(var o,a,s,u,l,c=t.createDocumentFragment(),f=[],p=0,d=e.length;p\\s*$/g;function Ne(e,t){return N(e,\"table\")&&N(11!==t.nodeType?t:t.firstChild,\"tr\")&&T(e).children(\"tbody\")[0]||e}function Ae(e){return e.type=(null!==e.getAttribute(\"type\"))+\"/\"+e.type,e}function Oe(e){return\"true/\"===(e.type||\"\").slice(0,5)?e.type=e.type.slice(5):e.removeAttribute(\"type\"),e}function Se(e,t){var n,r,i,o;if(1===t.nodeType){if(X.hasData(e)&&(o=X.get(e).events))for(i in X.remove(t,\"handle events\"),o)for(n=0,r=o[i].length;n\").attr(n.scriptAttrs||{}).prop({charset:n.scriptCharset,src:n.url}).on(\"load error\",i=function(e){r.remove(),i=null,e&&t(\"error\"===e.type?404:200,e.type)}),E.head.appendChild(r[0])},abort:function(){i&&i()}}});var Vt,Xt=[],Yt=/(=)\\?(?=&|$)|\\?\\?/;T.ajaxSetup({jsonp:\"callback\",jsonpCallback:function(){var e=Xt.pop()||T.expando+\"_\"+Nt.guid++;return this[e]=!0,e}}),T.ajaxPrefilter(\"json jsonp\",function(e,t,n){var r,i,o,a=!1!==e.jsonp&&(Yt.test(e.url)?\"url\":\"string\"==typeof e.data&&0===(e.contentType||\"\").indexOf(\"application/x-www-form-urlencoded\")&&Yt.test(e.data)&&\"data\");if(a||\"jsonp\"===e.dataTypes[0])return r=e.jsonpCallback=b(e.jsonpCallback)?e.jsonpCallback():e.jsonpCallback,a?e[a]=e[a].replace(Yt,\"$1\"+r):!1!==e.jsonp&&(e.url+=(At.test(e.url)?\"&\":\"?\")+e.jsonp+\"=\"+r),e.converters[\"script json\"]=function(){return o||T.error(r+\" was not called\"),o[0]},e.dataTypes[0]=\"json\",i=C[r],C[r]=function(){o=arguments},n.always(function(){void 0===i?T(C).removeProp(r):C[r]=i,e[r]&&(e.jsonpCallback=t.jsonpCallback,Xt.push(r)),o&&b(i)&&i(o[0]),o=i=void 0}),\"script\"}),y.createHTMLDocument=((Vt=E.implementation.createHTMLDocument(\"\").body).innerHTML=\"
    \",2===Vt.childNodes.length),T.parseHTML=function(e,t,n){return\"string\"!=typeof e?[]:(\"boolean\"==typeof t&&(n=t,t=!1),t||(y.createHTMLDocument?((r=(t=E.implementation.createHTMLDocument(\"\")).createElement(\"base\")).href=E.location.href,t.head.appendChild(r)):t=E),r=!n&&[],(n=A.exec(e))?[t.createElement(n[1])]:(n=ve([e],t,r),r&&r.length&&T(r).remove(),T.merge([],n.childNodes)));var r},T.fn.load=function(e,t,n){var r,i,o,a=this,s=e.indexOf(\" \");return-1\").append(T.parseHTML(e)).find(r):e)}).always(n&&function(e,t){a.each(function(){n.apply(this,o||[e.responseText,t,e])})}),this},T.expr.pseudos.animated=function(t){return T.grep(T.timers,function(e){return t===e.elem}).length},T.offset={setOffset:function(e,t,n){var r,i,o,a,s=T.css(e,\"position\"),u=T(e),l={};\"static\"===s&&(e.style.position=\"relative\"),o=u.offset(),r=T.css(e,\"top\"),a=T.css(e,\"left\"),a=(\"absolute\"===s||\"fixed\"===s)&&-1<(r+a).indexOf(\"auto\")?(i=(s=u.position()).top,s.left):(i=parseFloat(r)||0,parseFloat(a)||0),null!=(t=b(t)?t.call(e,n,T.extend({},o)):t).top&&(l.top=t.top-o.top+i),null!=t.left&&(l.left=t.left-o.left+a),\"using\"in t?t.using.call(e,l):u.css(l)}},T.fn.extend({offset:function(t){if(arguments.length)return void 0===t?this:this.each(function(e){T.offset.setOffset(this,t,e)});var e,n=this[0];return n?n.getClientRects().length?(e=n.getBoundingClientRect(),n=n.ownerDocument.defaultView,{top:e.top+n.pageYOffset,left:e.left+n.pageXOffset}):{top:0,left:0}:void 0},position:function(){if(this[0]){var e,t,n,r=this[0],i={top:0,left:0};if(\"fixed\"===T.css(r,\"position\"))t=r.getBoundingClientRect();else{for(t=this.offset(),n=r.ownerDocument,e=r.offsetParent||n.documentElement;e&&(e===n.body||e===n.documentElement)&&\"static\"===T.css(e,\"position\");)e=e.parentNode;e&&e!==r&&1===e.nodeType&&((i=T(e).offset()).top+=T.css(e,\"borderTopWidth\",!0),i.left+=T.css(e,\"borderLeftWidth\",!0))}return{top:t.top-i.top-T.css(r,\"marginTop\",!0),left:t.left-i.left-T.css(r,\"marginLeft\",!0)}}},offsetParent:function(){return this.map(function(){for(var e=this.offsetParent;e&&\"static\"===T.css(e,\"position\");)e=e.offsetParent;return e||ne})}}),T.each({scrollLeft:\"pageXOffset\",scrollTop:\"pageYOffset\"},function(t,i){var o=\"pageYOffset\"===i;T.fn[t]=function(e){return B(this,function(e,t,n){var r;return g(e)?r=e:9===e.nodeType&&(r=e.defaultView),void 0===n?r?r[i]:e[t]:void(r?r.scrollTo(o?r.pageXOffset:n,o?n:r.pageYOffset):e[t]=n)},t,e,arguments.length)}}),T.each([\"top\",\"left\"],function(e,n){T.cssHooks[n]=Ve(y.pixelPosition,function(e,t){if(t)return t=Ue(e,n),Be.test(t)?T(e).position()[n]+\"px\":t})}),T.each({Height:\"height\",Width:\"width\"},function(a,s){T.each({padding:\"inner\"+a,content:s,\"\":\"outer\"+a},function(r,o){T.fn[o]=function(e,t){var n=arguments.length&&(r||\"boolean\"!=typeof e),i=r||(!0===e||!0===t?\"margin\":\"border\");return B(this,function(e,t,n){var r;return g(e)?0===o.indexOf(\"outer\")?e[\"inner\"+a]:e.document.documentElement[\"client\"+a]:9===e.nodeType?(r=e.documentElement,Math.max(e.body[\"scroll\"+a],r[\"scroll\"+a],e.body[\"offset\"+a],r[\"offset\"+a],r[\"client\"+a])):void 0===n?T.css(e,t,i):T.style(e,t,n,i)},s,n?e:void 0,n)}})}),T.each([\"ajaxStart\",\"ajaxStop\",\"ajaxComplete\",\"ajaxError\",\"ajaxSuccess\",\"ajaxSend\"],function(e,t){T.fn[t]=function(e){return this.on(t,e)}}),T.fn.extend({bind:function(e,t,n){return this.on(e,null,t,n)},unbind:function(e,t){return this.off(e,null,t)},delegate:function(e,t,n,r){return this.on(t,e,n,r)},undelegate:function(e,t,n){return 1===arguments.length?this.off(e,\"**\"):this.off(t,e||\"**\",n)},hover:function(e,t){return this.mouseenter(e).mouseleave(t||e)}}),T.each(\"blur focus focusin focusout resize scroll click dblclick mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave change select submit keydown keypress keyup contextmenu\".split(\" \"),function(e,n){T.fn[n]=function(e,t){return 0\":\">\",'\"':\""\",\"'\":\"'\",\"`\":\"`\"},Ke=Fe(Be),$e=Fe(ye(Be)),We=te.templateSettings={evaluate:/<%([\\s\\S]+?)%>/g,interpolate:/<%=([\\s\\S]+?)%>/g,escape:/<%-([\\s\\S]+?)%>/g},ze=/(.)^/,Ue={\"'\":\"'\",\"\\\\\":\"\\\\\",\"\\r\":\"r\",\"\\n\":\"n\",\"\\u2028\":\"u2028\",\"\\u2029\":\"u2029\"},Ve=/\\\\|'|\\r|\\n|\\u2028|\\u2029/g;function Xe(e){return\"\\\\\"+Ue[e]}var Ye=/^\\s*(\\w|\\$)+\\s*$/;var Je=0;function Ge(e,t,n,r,i){if(!(r instanceof t))return e.apply(n,i);n=Te(e.prototype),i=e.apply(n,i);return x(i)?i:n}var Qe=b(function(i,o){var a=Qe.placeholder,s=function(){for(var e=0,t=o.length,n=Array(t),r=0;r img {\n width: 30px;\n height: 30px;\n margin-right: 5px;\n margin-top: -5px;\n display: inline;\n}\n\n.bg-zone-local {\n background-color: rgb(221,255,221);\n}\n\n.bg-region-local {\n background-color: rgb(254,254,194);\n}\n\n.bg-global {\n background-color: white;\n}\n\n.popup {\n position: relative;\n display: inline-block;\n cursor: pointer;\n}\n \n/* The actual popup (appears on top) */\n.popup .popuptext {\n visibility: hidden;\n width: 180px;\n height: 27px;\n background-color: white;\n color: #4b4b4b;\n text-align: center;\n border-radius: 6px;\n border-right: 1px solid #777;\n border-left: 1px solid #777;\n border-top: 1px solid #777;\n border-bottom: 1px solid #777;\n padding: 2px 0;\n position: absolute;\n z-index: 1;\n top: 60%;\n left: 50%;\n margin-left: -85%;\n}\n\n/* Popup arrow */\n.popup .popuptext::after {\n content: \"\";\n position: absolute;\n top: -5px;\n left: 50%;\n margin-left: -5px;\n width: 0;\n height: 0;\n border-left: 5px solid transparent;\n border-right: 5px solid transparent;\n border-bottom: 5px solid #777;\n}\n\n/* Toggle this class when clicking on the popup container (hide and show the popup) */\n.popup .show {\n visibility: visible;\n -webkit-animation: fadeIn 1s;\n animation: fadeIn 1s\n}\n\n/* Add animation (fade in the popup) */\n@-webkit-keyframes fadeIn {\n from {opacity: 0;}\n to {opacity: 1;}\n}\n\n@keyframes fadeIn {\n from {opacity: 0;}\n to {opacity:1 ;}\n}" }, { "path": "nsqadmin/static/html/index.html", "content": "\n\n\n nsqadmin\n \n \n \n \n \n\n\n \n\n
    \n\n \n \n \n\n\n" }, { "path": "nsqadmin/static/js/app_state.js", "content": "var Backbone = require('backbone');\nvar _ = require('underscore');\n\nvar AppState = Backbone.Model.extend({\n defaults: function() {\n return {\n 'VERSION': VERSION,\n 'GRAPHITE_URL': GRAPHITE_URL,\n 'GRAPH_ENABLED': GRAPH_ENABLED,\n 'STATSD_INTERVAL': STATSD_INTERVAL,\n 'STATSD_COUNTER_FORMAT': STATSD_COUNTER_FORMAT,\n 'STATSD_GAUGE_FORMAT': STATSD_GAUGE_FORMAT,\n 'STATSD_PREFIX': STATSD_PREFIX,\n 'NSQLOOKUPD': NSQLOOKUPD,\n 'graph_interval': '2h',\n 'IS_ADMIN': IS_ADMIN,\n 'BASE_PATH': BASE_PATH\n };\n },\n\n initialize: function() {\n this.on('change:graph_interval', function(model, v) {\n localStorage.setItem('graph_interval', v);\n });\n\n var qp = _.object(_.compact(_.map(window.location.search.slice(1).split('&'),\n function(item) { return item ? item.split('=') : false; })));\n\n var def = this.get('GRAPH_ENABLED') ? '2h' : 'off';\n var interval = qp['t'] || localStorage.getItem('graph_interval') || def;\n this.set('graph_interval', interval);\n },\n\n basePath: function(p) {\n // if base path is / then don't prefix\n var bp = this.get('BASE_PATH') === '/' ? '' : this.get('BASE_PATH');\n // remove trailing /, but guarantee at least /\n return (bp + p).replace(/\\/$/, '') || '/';\n },\n\n apiPath: function(p) {\n return this.basePath('/api' + p);\n }\n});\n\nvar appState = new AppState();\n\nwindow.AppState = appState;\n\nmodule.exports = appState;\n" }, { "path": "nsqadmin/static/js/collections/nodes.js", "content": "var Backbone = require('backbone');\n\nvar AppState = require('../app_state');\n\nvar NodeModel = require('../models/node');\n\nvar Nodes = Backbone.Collection.extend({\n model: NodeModel,\n\n comparator: 'id',\n\n constructor: function Nodes() {\n Backbone.Collection.prototype.constructor.apply(this, arguments);\n },\n\n url: function() {\n return AppState.apiPath('/nodes');\n },\n\n parse: function(resp) {\n resp['nodes'].forEach(function(n) {\n var jaddr = n['broadcast_address'];\n if (jaddr.includes(':')) {\n // ipv6 raw address contains ':'\n // it must be wrapped in '[ ]' when joined with port\n jaddr = '[' + jaddr + ']';\n }\n n['broadcast_address_http'] = jaddr + ':' + n['http_port'];\n });\n return resp['nodes'];\n }\n});\n\nmodule.exports = Nodes;\n" }, { "path": "nsqadmin/static/js/collections/topics.js", "content": "var _ = require('underscore');\nvar Backbone = require('backbone');\n\nvar AppState = require('../app_state');\n\nvar Topic = require('../models/topic');\n\nvar Topics = Backbone.Collection.extend({\n model: Topic,\n\n comparator: 'id',\n\n constructor: function Topics() {\n Backbone.Collection.prototype.constructor.apply(this, arguments);\n },\n\n url: function() {\n return AppState.apiPath('/topics');\n },\n\n parse: function(resp) {\n var topics = _.map(resp['topics'], function(name) {\n return {'name': name};\n });\n return topics;\n }\n});\n\nmodule.exports = Topics;\n" }, { "path": "nsqadmin/static/js/lib/ajax_setup.js", "content": "var $ = require('jquery');\nvar _ = require('underscore');\n\n// Set up some headers and options for every request.\n$.ajaxPrefilter(function(options) {\n options['headers'] = _.defaults(options['headers'] || {}, {\n 'X-UserAgent': USER_AGENT,\n 'Accept': 'application/vnd.nsq; version=1.0'\n });\n options['timeout'] = 20 * 1000;\n options['contentType'] = 'application/json';\n});\n" }, { "path": "nsqadmin/static/js/lib/handlebars_helpers.js", "content": "var $ = require('jquery');\nvar _ = require('underscore');\nvar Handlebars = require('hbsfy/runtime');\n\nvar AppState = require('../app_state');\n\nvar formatStatsdKey = function(metricType, key) {\n var fullKey = key;\n var fmt;\n\n if (metricType === 'counter') {\n fmt = AppState.get('STATSD_COUNTER_FORMAT');\n fullKey = fmt.replace(/%s/g, key);\n } else if (metricType === 'gauge') {\n fmt = AppState.get('STATSD_GAUGE_FORMAT');\n fullKey = fmt.replace(/%s/g, key);\n }\n\n return fullKey;\n};\n\nvar statsdPrefix = function(host) {\n var prefix = AppState.get('STATSD_PREFIX');\n var statsdHostKey = host.replace(/[\\.:]/g, '_');\n prefix = prefix.replace(/%s/g, statsdHostKey);\n if (prefix.substring(prefix.length, 1) !== '.') {\n prefix += '.';\n }\n return prefix;\n};\n\n/* eslint-disable key-spacing */\nvar metricType = function(key) {\n return {\n 'depth': 'gauge',\n 'in_flight_count': 'gauge',\n 'deferred_count': 'gauge',\n 'requeue_count': 'counter',\n 'timeout_count': 'counter',\n 'message_count': 'counter',\n 'clients': 'gauge',\n '*_bytes': 'gauge',\n 'gc_pause_*': 'gauge',\n 'gc_runs': 'counter',\n 'heap_objects': 'gauge',\n 'e2e_processing_latency': 'gauge'\n }[key];\n};\n/* eslint-enable key-spacing */\n\nvar genColorList = function(typ, key) {\n if (typ === 'topic' || typ === 'channel') {\n if (key === 'depth' || key === 'deferred_count') {\n return 'red';\n }\n } else if (typ === 'node') {\n return 'red,green,blue,purple';\n } else if (typ === 'counter') {\n return 'green';\n }\n return 'blue';\n};\n\n// sanitizeGraphiteKey removes special characters from a graphite key\n// this matches behavior of bitly/statsdaemon\n// eslint-disable-next-line max-len\n// https://github.com/bitly/statsdaemon/blob/fc46d9cfe29b674a0c8abc723afaa9370430cdcd/statsdaemon.go#L64-L88\nvar sanitizeGraphiteKey = function(s) {\n return s.replaceAll(' ', '_').replaceAll('/', '-').replaceAll(/[^a-zA-Z0-9-_.]/g, '');\n};\n\nvar genTargets = function(typ, node, ns1, ns2, key) {\n var targets = [];\n var prefix = statsdPrefix(node ? node : '*');\n var fullKey;\n var target;\n if (typ === 'topic') {\n fullKey = formatStatsdKey(metricType(key),\n prefix + 'topic.' + sanitizeGraphiteKey(ns1) + '.' + key);\n targets.push('sumSeries(' + fullKey + ')');\n } else if (typ === 'channel') {\n fullKey = formatStatsdKey(metricType(key),\n prefix + 'topic.' + sanitizeGraphiteKey(ns1) + '.channel.' +\n sanitizeGraphiteKey(ns2) + '.' + key);\n targets.push('sumSeries(' + fullKey + ')');\n } else if (typ === 'node') {\n target = prefix + 'mem.' + key;\n if (key === 'gc_runs') {\n target = 'movingAverage(' + target + ',45)';\n }\n targets.push(formatStatsdKey(metricType(key), target));\n } else if (typ === 'e2e') {\n targets = _.map(ns1['percentiles'], function(p) {\n var t;\n if (ns1['channel'] !== '') {\n t = prefix + 'topic.' + ns1['topic'] + '.channel.' + ns1['channel'] + '.' +\n key + '_' + (p['quantile'] * 100);\n } else {\n t = prefix + 'topic.' + ns1['topic'] + '.' + key + '_' + (p['quantile'] * 100);\n }\n if (node === '*') {\n t = 'averageSeries(' + t + ')';\n }\n return 'scale(' + formatStatsdKey(metricType(key), t) + ',0.000001)';\n });\n } else if (typ === 'counter') {\n fullKey = formatStatsdKey(metricType(key), prefix + 'topic.*.channel.*.' + key);\n targets.push('sumSeries(' + fullKey + ')');\n }\n return targets;\n};\n\nHandlebars.registerHelper('default', function(x, defaultValue) {\n return x ? x : defaultValue;\n});\n\nHandlebars.registerHelper('ifeq', function(a, b, options) {\n return (a === b) ? options.fn(this) : options.inverse(this);\n});\n\nHandlebars.registerHelper('unlesseq', function(a, b, options) {\n return (a !== b) ? options.fn(this) : options.inverse(this);\n});\n\nHandlebars.registerHelper('ifgteq', function(a, b, options) {\n return (a >= b) ? options.fn(this) : options.inverse(this);\n});\n\nHandlebars.registerHelper('iflteq', function(a, b, options) {\n return (a <= b) ? options.fn(this) : options.inverse(this);\n});\n\nHandlebars.registerHelper('length', function(xs) {\n return xs.length;\n});\n\nHandlebars.registerHelper('lowercase', function(s) {\n return s.toLowerCase();\n});\n\nHandlebars.registerHelper('uppercase', function(s) {\n return s.toUpperCase();\n});\n\n// this helper is inclusive of the top number\nHandlebars.registerHelper('for', function(from, to, incr, block) {\n var accum = '';\n for (var i = from; i <= to; i += incr) {\n accum += block.fn(i);\n }\n return accum;\n});\n\n// Logical operators as helper functions, which can be useful when used within\n// an `if` or `unless` block via the new helper composition syntax, like so:\n//\n// {{#if (or step.unlocked step.is_finished)}}\n// Step is unlocked or finished!\n// {{/if}}\n//\n// Any number of arguments may be given to either helper. NOTE: _.initial() is\n// used below because every helper takes an options hash as its last argument.\nHandlebars.registerHelper('and', function() {\n return _.all(_.initial(arguments));\n});\n\nHandlebars.registerHelper('or', function() {\n return _.any(_.initial(arguments));\n});\n\nHandlebars.registerHelper('eq', function(a, b) {\n return a === b;\n});\n\nHandlebars.registerHelper('neq', function(a, b) {\n return a !== b;\n});\n\nHandlebars.registerHelper('urlencode', function(a) {\n return encodeURIComponent(a);\n});\n\nHandlebars.registerHelper('floatToPercent', function(f) {\n return Math.floor(f * 100);\n});\n\nHandlebars.registerHelper('floatToDecimalPercent', function(f) {\n return parseFloat((f * 100).toFixed(2));\n});\n\nHandlebars.registerHelper('percSuffix', function(f) {\n var v = Math.floor(f * 100) % 10;\n if (v === 1) {\n return 'st';\n } else if (v === 2) {\n return 'nd';\n } else if (v === 3) {\n return 'rd';\n }\n return 'th';\n});\n\nHandlebars.registerHelper('commafy', function(n) {\n n = n || 0;\n return n.toString().replace(/\\B(?=(\\d{3})+(?!\\d))/g, ',');\n});\n\nfunction round(num, places) {\n var multiplier = Math.pow(10, places);\n return Math.round(num * multiplier) / multiplier;\n}\n\nHandlebars.registerHelper('nanotohuman', function(n) {\n var s = '';\n var v;\n if (n >= 3600000000000) {\n v = Math.floor(n / 3600000000000);\n n = n % 3600000000000;\n s = v + 'h';\n }\n if (n >= 60000000000) {\n v = Math.floor(n / 60000000000);\n n = n % 60000000000;\n s += v + 'm';\n }\n if (n >= 1000000000) {\n n = round(n / 1000000000, 2);\n s += n + 's';\n } else if (n >= 1000000) {\n n = round(n / 1000000, 2);\n s += n + 'ms';\n } else if (n >= 1000) {\n n = round(n / 1000, 2);\n s += n + 'us';\n } else {\n s = n + 'ns';\n }\n return s;\n});\n\nHandlebars.registerHelper('sparkline', function(typ, node, ns1, ns2, key) {\n var q = {\n 'colorList': genColorList(typ, key),\n 'height': '20',\n 'width': '120',\n 'hideGrid': 'true',\n 'hideLegend': 'true',\n 'hideAxes': 'true',\n 'bgcolor': 'ff000000', // transparent\n 'fgcolor': 'black',\n 'margin': '0',\n 'yMin': '0',\n 'lineMode': 'connected',\n 'drawNullAsZero': 'false',\n 'from': '-' + AppState.get('graph_interval'),\n 'until': '-1min'\n };\n\n var interval = AppState.get('STATSD_INTERVAL') + 'sec';\n q['target'] = _.map(genTargets(typ, node, ns1, ns2, key), function(t) {\n return 'summarize(' + t + ',\"' + interval + '\",\"avg\")';\n });\n\n return AppState.get('GRAPHITE_URL') + '/render?' + $.param(q, true);\n});\n\nHandlebars.registerHelper('large_graph', function(typ, node, ns1, ns2, key) {\n var q = {\n 'colorList': genColorList(typ, key),\n 'height': '450',\n 'width': '800',\n 'bgcolor': 'ff000000', // transparent\n 'fgcolor': '999999',\n 'yMin': '0',\n 'lineMode': 'connected',\n 'drawNullAsZero': 'false',\n 'from': '-' + AppState.get('graph_interval'),\n 'until': '-1min'\n };\n\n var interval = AppState.get('STATSD_INTERVAL') + 'sec';\n q['target'] = _.map(genTargets(typ, node, ns1, ns2, key), function(t) {\n if (metricType(key) === 'counter') {\n var scale = 1 / AppState.get('STATSD_INTERVAL');\n t = 'scale(' + t + ',' + scale + ')';\n }\n return 'summarize(' + t + ',\"' + interval + '\",\"avg\")';\n });\n\n return AppState.get('GRAPHITE_URL') + '/render?' + $.param(q, true);\n});\n\nHandlebars.registerHelper('rate', function(typ, node, ns1, ns2) {\n return genTargets(typ, node, ns1, ns2, 'message_count')[0];\n});\n\nHandlebars.registerPartial('error', require('../views/error.hbs'));\nHandlebars.registerPartial('warning', require('../views/warning.hbs'));\n\nHandlebars.registerHelper('basePath', function(p) {\n return AppState.basePath(p);\n});\n" }, { "path": "nsqadmin/static/js/lib/pubsub.js", "content": "var _ = require('underscore');\nvar Backbone = require('backbone');\n\nvar Pubsub = _.clone(Backbone.Events);\n\n// making this global to more easily trigger events from the console\nwindow.Pubsub = Pubsub;\n\nmodule.exports = Pubsub;\n" }, { "path": "nsqadmin/static/js/main.js", "content": "var $ = require('jquery');\nvar Backbone = require('backbone');\n\n// var Pubsub = require('./lib/pubsub');\nvar Router = require('./router');\n\nvar AppView = require('./views/app');\n\n// When using browserify, we need to tell Backbone what jQuery to use.\nBackbone.$ = $;\n\n// Side effects:\nrequire('./lib/ajax_setup');\nrequire('./lib/handlebars_helpers');\n\nvar start = function() {\n new AppView();\n Router.start();\n};\n\n// Pubsub.on('all', function() {\n// console.log.apply(console, arguments);\n// });\n\nstart();\n" }, { "path": "nsqadmin/static/js/models/channel.js", "content": "var _ = require('underscore');\n\nvar AppState = require('../app_state');\nvar Backbone = require('backbone');\n\nvar Channel = Backbone.Model.extend({\n idAttribute: 'name',\n\n constructor: function Channel() {\n Backbone.Model.prototype.constructor.apply(this, arguments);\n },\n\n url: function() {\n return AppState.apiPath('/topics/' +\n encodeURIComponent(this.get('topic')) + '/' +\n encodeURIComponent(this.get('name')));\n },\n\n parse: function(response) {\n response['nodes'] = _.map(response['nodes'] || [], function(node) {\n var nodeParts = node['node'].split(':');\n var port = nodeParts.pop();\n var address = nodeParts.join(':');\n var hostname = node['hostname'];\n var zonecount = node['zone_local_msg_count'];\n var deliverycount = node['delivery_msg_count'];\n var regioncount = node['region_local_msg_count'];\n var globalcount = node['global_msg_count'];\n node['show_broadcast_address'] = hostname.toLowerCase() !== address.toLowerCase();\n node['hostname_port'] = hostname + ':' + port;\n node['zone_local_percentage'] = zonecount / deliverycount;\n node['region_local_percentage'] = regioncount / deliverycount;\n node['global_percentage'] = globalcount / deliverycount;\n if (isNaN(node['zone_local_percentage'])) {\n node['zone_local_percentage'] = 0;\n }\n if (isNaN(node['region_local_percentage'])) {\n node['region_local_percentage'] = 0;\n }\n if (isNaN(node['global_percentage'])) {\n node['global_percentage'] = 0;\n }\n return node;\n });\n\n response['clients'] = _.map(response['clients'] || [], function(client) {\n var clientId = client['client_id'];\n var hostname = client['hostname'];\n var shortHostname = hostname.split('.')[0];\n\n // ignore client_id if it's duplicative\n client['show_client_id'] = (clientId.toLowerCase() !== shortHostname.toLowerCase()\n && clientId.toLowerCase() !== hostname.toLowerCase());\n\n var port = client['remote_address'].split(':').pop();\n client['hostname_port'] = hostname + ':' + port;\n\n return client;\n });\n\n return response;\n }\n});\n\nmodule.exports = Channel;\n" }, { "path": "nsqadmin/static/js/models/node.js", "content": "var AppState = require('../app_state');\nvar Backbone = require('backbone');\n\nvar NodeModel = Backbone.Model.extend({\n idAttribute: 'name',\n\n constructor: function Node() {\n Backbone.Model.prototype.constructor.apply(this, arguments);\n },\n\n urlRoot: function() {\n return AppState.apiPath('/nodes');\n },\n\n tombstoneTopic: function(topic) {\n return this.destroy({\n 'data': JSON.stringify({'topic': topic}),\n 'dataType': 'text'\n });\n }\n});\n\nmodule.exports = NodeModel;\n" }, { "path": "nsqadmin/static/js/models/topic.js", "content": "var _ = require('underscore');\n\nvar AppState = require('../app_state');\nvar Backbone = require('backbone');\n\nvar Topic = Backbone.Model.extend({\n idAttribute: 'name',\n\n constructor: function Topic() {\n Backbone.Model.prototype.constructor.apply(this, arguments);\n },\n\n url: function() {\n return AppState.apiPath('/topics/' + encodeURIComponent(this.get('name')));\n },\n\n parse: function(response) {\n response['nodes'] = _.map(response['nodes'] || [], function(node) {\n var nodeParts = node['node'].split(':');\n var port = nodeParts.pop();\n var address = nodeParts.join(':');\n var hostname = node['hostname'];\n node['show_broadcast_address'] = hostname.toLowerCase() !== address.toLowerCase();\n node['hostname_port'] = hostname + ':' + port;\n return node;\n });\n return response;\n }\n});\n\nmodule.exports = Topic;\n" }, { "path": "nsqadmin/static/js/router.js", "content": "var Backbone = require('backbone');\n\nvar AppState = require('./app_state');\nvar Pubsub = require('./lib/pubsub');\n\n\nvar Router = Backbone.Router.extend({\n initialize: function() {\n var bp = function(p) {\n // remove leading slash\n return AppState.basePath(p).substring(1);\n };\n this.route(bp('/'), 'topics');\n this.route(bp('/topics/(:topic)(/:channel)'), 'topic');\n this.route(bp('/lookup'), 'lookup');\n this.route(bp('/nodes(/:node)'), 'nodes');\n this.route(bp('/counter'), 'counter');\n // this.listenTo(this, 'route', function(route, params) {\n // console.log('Route: %o; params: %o', route, params);\n // });\n },\n\n start: function() {\n Backbone.history.start({\n 'pushState': true\n });\n },\n\n topics: function() {\n Pubsub.trigger('topics:show');\n },\n\n topic: function(topic, channel) {\n if (channel !== null) {\n Pubsub.trigger('channel:show', topic, channel);\n return;\n }\n Pubsub.trigger('topic:show', topic);\n },\n\n lookup: function() {\n Pubsub.trigger('lookup:show');\n },\n\n nodes: function(node) {\n if (node !== null) {\n Pubsub.trigger('node:show', node);\n return;\n }\n Pubsub.trigger('nodes:show');\n },\n\n counter: function() {\n Pubsub.trigger('counter:show');\n }\n});\n\n\nmodule.exports = new Router();\n" }, { "path": "nsqadmin/static/js/views/app.js", "content": "var $ = require('jquery');\n\nwindow.jQuery = $;\nvar bootstrap = require('bootstrap'); //eslint-disable-line no-unused-vars\nvar bootbox = require('bootbox');\n\nvar AppState = require('../app_state');\nvar Pubsub = require('../lib/pubsub');\nvar Router = require('../router');\n\nvar BaseView = require('./base');\nvar HeaderView = require('./header');\nvar TopicsView = require('./topics');\nvar TopicView = require('./topic');\nvar ChannelView = require('./channel');\nvar LookupView = require('./lookup');\nvar NodesView = require('./nodes');\nvar NodeView = require('./node');\nvar CounterView = require('./counter');\n\nvar NodeModel = require('../models/node');\nvar TopicModel = require('../models/topic');\nvar ChannelModel = require('../models/channel');\n\nvar AppView = BaseView.extend({\n // not a fan of setting a view's el to an existing element on the page\n // for the top-level AppView, it seems appropriate, however\n el: '#container',\n\n events: {\n 'click .link': 'onLinkClick',\n 'click .tombstone-link': 'onTombstoneClick'\n },\n\n initialize: function() {\n BaseView.prototype.initialize.apply(this, arguments);\n\n this.listenTo(Pubsub, 'topics:show', this.showTopics);\n this.listenTo(Pubsub, 'topic:show', this.showTopic);\n this.listenTo(Pubsub, 'channel:show', this.showChannel);\n this.listenTo(Pubsub, 'lookup:show', this.showLookup);\n this.listenTo(Pubsub, 'nodes:show', this.showNodes);\n this.listenTo(Pubsub, 'node:show', this.showNode);\n this.listenTo(Pubsub, 'counter:show', this.showCounter);\n\n this.listenTo(Pubsub, 'view:ready', function() {\n $('.rate').each(function(i, el) {\n var $el = $(el);\n var interval = AppState.get('STATSD_INTERVAL');\n var q = {\n 'target': $el.attr('target'),\n 'from': '-' + (2 * interval) + 'sec',\n 'until': '-' + interval + 'sec',\n 'format': 'json',\n };\n var formatRate = function(data) {\n if (data[0] === null ||\n data[0]['datapoints'][0] === null ||\n data[0]['datapoints'][0][0] < 0) {\n return 'N/A';\n } else {\n return (data[0]['datapoints'][0][0] / interval).toFixed(2);\n }\n };\n $.ajax({\n url: AppState.get('GRAPHITE_URL') + '/render',\n data: q,\n dataType: 'jsonp',\n jsonp: 'jsonp'\n })\n .done(function(data) { $el.html(formatRate(data)); })\n .fail(function() { $el.html('ERROR'); });\n });\n });\n this.render();\n },\n\n postRender: function() {\n this.appendSubview(new HeaderView());\n },\n\n showView: function(f) {\n window.scrollTo(0, 0);\n if (this.currentView) {\n this.currentView.remove();\n }\n this.currentView = f();\n this.appendSubview(this.currentView);\n },\n\n showTopics: function() {\n this.showView(function() {\n return new TopicsView();\n });\n },\n\n showTopic: function(topic) {\n this.showView(function() {\n var model = new TopicModel({'name': topic, 'isAdmin': AppState.get('IS_ADMIN')});\n return new TopicView({'model': model});\n });\n },\n\n showChannel: function(topic, channel) {\n this.showView(function() {\n var model = new ChannelModel({\n 'topic': topic,\n 'name': channel,\n 'isAdmin': AppState.get('IS_ADMIN')\n });\n return new ChannelView({'model': model});\n });\n },\n\n showLookup: function() {\n this.showView(function() {\n return new LookupView({'isAdmin': AppState.get('IS_ADMIN')});\n });\n },\n\n showNodes: function() {\n this.showView(function() {\n return new NodesView();\n });\n },\n\n showNode: function(node) {\n this.showView(function() {\n var model = new NodeModel({'name': node});\n return new NodeView({'model': model});\n });\n },\n\n showCounter: function() {\n this.showView(function() {\n return new CounterView();\n });\n },\n\n onLinkClick: function(e) {\n if (e.ctrlKey || e.metaKey) {\n // allow ctrl+click to open in a new tab\n return;\n }\n e.preventDefault();\n e.stopPropagation();\n Router.navigate($(e.currentTarget).attr('href'), {'trigger': true});\n },\n\n onTombstoneClick: function(e) {\n e.preventDefault();\n e.stopPropagation();\n var nodeName = $(e.target).data('node');\n var topicName = $(e.target).data('topic');\n var txt = 'Are you sure you want to tombstone ' + nodeName + '?';\n bootbox.confirm(txt, function(result) {\n if (result !== true) {\n return;\n }\n var node = new NodeModel({\n 'name': nodeName\n });\n node.tombstoneTopic(topicName)\n .done(function() { window.location.reload(true); })\n .fail(this.handleAJAXError.bind(this));\n }.bind(this));\n }\n});\n\nmodule.exports = AppView;\n" }, { "path": "nsqadmin/static/js/views/base.js", "content": "var $ = require('jquery');\nvar _ = require('underscore');\nvar Backbone = require('backbone');\n\nvar AppState = require('../app_state');\n\nvar errorTemplate = require('./error.hbs');\n\n\nvar BaseView = Backbone.View.extend({\n constructor: function(options) {\n // As of 1.10, Backbone no longer automatically attaches options passed\n // to the constructor as this.options, but that's often useful in some\n // cases, like a className function, that happen before initialize()\n // would have a chance to attach the same options.\n this.options = options || {};\n return Backbone.View.prototype.constructor.apply(this, arguments);\n },\n\n initialize: function() {\n this.subviews = [];\n this.rendered = false;\n },\n\n template: function() {},\n\n skippedRender: function() {},\n\n render: function(data) {\n if (this.renderOnce && this.rendered) {\n this.skippedRender();\n return this;\n }\n this.removeSubviews();\n var ctx = this.getRenderCtx(data);\n // console.log('render ctx: %o', ctx);\n var html = this.template(ctx);\n if (!this.removed) {\n this.$el.empty();\n this.$el.append(html);\n this.postRender(ctx);\n }\n this.rendered = true;\n return this;\n },\n\n getRenderCtx: function(data) {\n var ctx = {\n 'graph_enabled': AppState.get('GRAPH_ENABLED'),\n 'graph_interval': AppState.get('graph_interval'),\n 'graph_active': AppState.get('GRAPH_ENABLED') &&\n AppState.get('graph_interval') !== 'off',\n 'nsqlookupd': AppState.get('NSQLOOKUPD'),\n 'version': AppState.get('VERSION')\n };\n if (this.model) {\n ctx = _.extend(ctx, this.model.toJSON());\n } else if (this.collection) {\n ctx = _.extend(ctx, {'collection': this.collection.toJSON()});\n }\n if (data) {\n ctx = _.extend(ctx, data);\n }\n return ctx;\n },\n\n postRender: function() {},\n\n appendSubview: function(subview, selector) {\n return this.appendSubviews([subview], selector);\n },\n\n appendSubviews: function(subviews, selector) {\n this.subviews.push.apply(this.subviews, subviews);\n var $el = selector ? this.$(selector) : this.$el;\n $el.append(subviews.map(function(subview) {\n return subview.render().delegateEvents().el;\n }));\n },\n\n removeSubviews: function() {\n while (this.subviews.length) {\n this.subviews.pop().remove();\n }\n },\n\n remove: function() {\n this.removed = true;\n this.removeSubviews();\n Backbone.View.prototype.remove.apply(this, arguments);\n },\n\n parseErrorMessage: function(jqXHR) {\n var msg = 'ERROR: failed to connect to nsqadmin';\n if (jqXHR.readyState === 4) {\n try {\n var parsed = JSON.parse(jqXHR.responseText);\n msg = parsed['message'];\n } catch (err) {\n msg = 'ERROR: failed to decode JSON - ' + err.message;\n }\n }\n return msg;\n },\n\n handleAJAXError: function(jqXHR) {\n $('#warning, #error').hide();\n $('#error .alert').text(this.parseErrorMessage(jqXHR));\n $('#error').show();\n },\n\n handleViewError: function(jqXHR) {\n this.removeSubviews();\n this.$el.html(errorTemplate({'message': this.parseErrorMessage(jqXHR)}));\n }\n});\n\nmodule.exports = BaseView;\n" }, { "path": "nsqadmin/static/js/views/channel.hbs", "content": "{{> warning}}\n{{> error}}\n\n
      \n
    1. Streams\n
    2. {{topic}}\n
    3. {{name}}
      4. \n
    \n\n
    \n
    \n
    \n

    Topic: {{topic}}\n

    Channel: {{name}}\n

    \n
    \n
    \n\n{{#unless nodes.length}}\n
    \n
    \n
    \n

    Notice

    No producers exist for this topic/channel.\n

    See Lookup for more information.\n

    \n
    \n
    \n{{else}}\n{{#if isAdmin}}\n
    \n
    \n \n
    \n
    \n \n
    \n
    \n {{#if paused}}\n \n {{else}}\n \n {{/if}}\n
    \n
    \n{{/if}}\n\n
    \n
    \n

    Channel

    \n \n \n \n \n \n {{#if e2e_processing_latency.percentiles.length}}\n \n {{/if}}\n \n \n \n \n \n \n \n \n \n \n {{#if graph_active}}{{/if}}\n \n {{#if delivery_msg_count}}{{/if}}\n {{#each e2e_processing_latency.percentiles}}\n \n {{/each}}\n \n {{#each nodes}}\n \n \n \n \n \n \n \n \n \n {{#if ../graph_active}}\n \n {{/if}}\n \n {{#if delivery_msg_count}}\n \n {{/if}}\n {{#if e2e_processing_latency.percentiles.length}}\n {{#each e2e_processing_latency.percentiles}}\n \n {{/each}}\n {{/if}}\n \n {{#if ../graph_active}}\n \n \n \n \n \n \n \n \n \n \n \n {{#if delivery_msg_count}}{{/if}}\n {{#if e2e_processing_latency.percentiles.length}}\n \n {{/if}}\n \n {{/if}}\n {{/each}}\n \n \n \n \n \n \n \n \n \n {{#if graph_active}}\n \n {{/if}}\n \n {{#if delivery_msg_count}}{{/if}}\n {{#if e2e_processing_latency.percentiles.length}}\n {{#each e2e_processing_latency.percentiles}}\n \n {{/each}}\n {{/if}}\n \n {{#if graph_active}}\n \n \n \n \n \n \n \n \n \n \n \n {{#if delivery_msg_count}}{{/if}}\n {{#if e2e_processing_latency.percentiles.length}}\n \n {{/if}}\n \n {{/if}}\n
     Message QueuesStatisticsE2E Processing Latency
    NSQd HostDepthMemory + DiskIn-FlightDeferredRequeuedTimed OutMessagesRateConnectionsDelivery{{floatToPercent quantile}}{{percSuffix quantile}}
    \n {{#if show_broadcast_address}}\n {{hostname_port}} ({{node}})\n {{else}}\n {{hostname_port}}\n {{/if}}\n {{#if paused}} paused{{/if}}\n {{commafy depth}}{{commafy memory_depth}} + {{commafy backend_depth}}{{commafy in_flight_count}}{{commafy deferred_count}}{{commafy requeue_count}}{{commafy timeout_count}}{{commafy message_count}}{{commafy client_count}}\n
    \n
    \n
    \n {{floatToPercent zone_local_percentage}}%\n
    \n
    \n {{floatToPercent region_local_percentage}}%\n
    \n
    \n {{floatToPercent global_percentage}}%\n
    \n
    \n {{floatToDecimalPercent zone_local_percentage}}% | {{floatToDecimalPercent region_local_percentage}}% | {{floatToDecimalPercent global_percentage}}%\n
    \n     		\n {{nanotohuman average}}\n
    \n \n
    Total:{{commafy depth}}{{commafy memory_depth}} + {{commafy backend_depth}}{{commafy in_flight_count}}{{commafy deferred_count}}{{commafy requeue_count}}{{commafy timeout_count}}{{commafy message_count}}{{commafy client_count}}\n {{nanotohuman average}}\n
    \n \n
    \n
    \n
    \n{{/unless}}\n\n

    Client Connections

    \n\n
    \n
    \n {{#unless clients.length}}\n

    Notice

    No clients connected to this channel
    \n {{else}}\n \n \n \n \n \n \n \n \n \n \n \n \n \n {{#each clients}}\n \n \n \n \n \n \n \n \n \n \n \n \n {{/each}}\n
    Client HostUser-AgentAttributesNSQd HostIn-FlightReady CountFinishedRequeuedMessagesConnected
    {{hostname_port}}{{#if show_client_id}} ({{client_id}}){{/if}}{{#if user_agent.length}}{{user_agent}}{{/if}}\n {{#if sample_rate}}\n Sampled {{sample_rate}}%\n {{/if}}\n {{#if tls}}\n TLS\n {{/if}}\n {{#if deflate}}\n Deflate\n {{/if}}\n {{#if snappy}}\n Snappy\n {{/if}}\n {{#if authed}}\n \n {{#if auth_identity_url}}{{/if}}\n \n {{#if auth_identity_url}}{{/if}}\n \n {{/if}}\n {{#if topology_region}}\n {{topology_region}}\n {{/if}}\n {{#if topology_zone}}\n {{topology_zone}}\n {{/if}}\n {{#if (and\n node_topology_zone\n (eq topology_zone node_topology_zone))}}\n zoneLocal\n {{else}}\n {{#if (and\n node_topology_region\n (eq topology_region node_topology_region))}}\n regionLocal\n {{/if}}\n {{/if}}\n {{node}}{{commafy in_flight_count}}{{commafy ready_count}}{{commafy finish_count}}{{commafy requeue_count}}{{commafy message_count}}{{nanotohuman connected}}
    \n {{/unless}}\n
    \n
    \n" }, { "path": "nsqadmin/static/js/views/channel.js", "content": "var $ = require('jquery');\n\nwindow.jQuery = $;\nvar bootstrap = require('bootstrap'); //eslint-disable-line no-unused-vars\nvar bootbox = require('bootbox');\n\nvar Pubsub = require('../lib/pubsub');\nvar AppState = require('../app_state');\n\nvar BaseView = require('./base');\n\nvar ChannelView = BaseView.extend({\n className: 'channel container-fluid',\n\n template: require('./spinner.hbs'),\n\n events: {\n 'click .channel-actions button': 'channelAction',\n 'click .popup': 'showDeliveryBreakdown'\n },\n\n initialize: function() {\n BaseView.prototype.initialize.apply(this, arguments);\n this.listenTo(AppState, 'change:graph_interval', this.render);\n var isAdmin = this.model.get('isAdmin');\n this.model.fetch()\n .done(function(data) {\n this.template = require('./channel.hbs');\n this.render({'message': data['message'], 'isAdmin': isAdmin});\n }.bind(this))\n .fail(this.handleViewError.bind(this))\n .always(Pubsub.trigger.bind(Pubsub, 'view:ready'));\n },\n\n showDeliveryBreakdown: function(e) {\n e.preventDefault();\n e.stopPropagation();\n var popup = document.getElementById($(e.currentTarget).data('id'));\n popup.classList.toggle(\"show\");\n },\n\n channelAction: function(e) {\n e.preventDefault();\n e.stopPropagation();\n var action = $(e.currentTarget).data('action');\n var txt = 'Are you sure you want to ' +\n action + ' ' + this.model.get('topic') +\n '/' + this.model.get('name') + '?';\n bootbox.confirm(txt, function(result) {\n if (result !== true) {\n return;\n }\n if (action === 'delete') {\n var topic = this.model.get('topic');\n $.ajax(this.model.url(), {'method': 'DELETE'})\n .done(function() {\n window.location = AppState.basePath('/topics/' +\n encodeURIComponent(topic));\n })\n .fail(this.handleAJAXError.bind(this));\n } else {\n $.post(this.model.url(), JSON.stringify({'action': action}))\n .done(function() { window.location.reload(true); })\n .fail(this.handleAJAXError.bind(this));\n }\n }.bind(this));\n }\n});\n\nmodule.exports = ChannelView;\n" }, { "path": "nsqadmin/static/js/views/counter.hbs", "content": "\n\n{{> warning}}\n{{> error}}\n\n
    \n
    \n
    \n

    Messages Processed

    \n

    \n{{#if graph_active}}\n \n{{/if}}\n
    \n" }, { "path": "nsqadmin/static/js/views/counter.js", "content": "var _ = require('underscore');\nvar $ = require('jquery');\n\nvar AppState = require('../app_state');\n\nvar BaseView = require('./base');\n\nvar CounterView = BaseView.extend({\n className: 'counter container-fluid',\n\n template: require('./counter.hbs'),\n\n initialize: function() {\n BaseView.prototype.initialize.apply(this, arguments);\n this.listenTo(AppState, 'change:graph_interval', function() {\n clearTimeout(this.poller);\n clearTimeout(this.animator);\n this.render();\n this.start();\n });\n this.start();\n },\n\n remove: function() {\n clearTimeout(this.poller);\n clearTimeout(this.animator);\n BaseView.prototype.remove.apply(this, arguments);\n },\n\n start: function() {\n this.poller = null;\n this.animator = null;\n this.delta = 0;\n this.looping = false;\n this.targetPollInterval = 10000;\n this.currentNum = -1;\n this.lastNum = 0;\n this.interval = 100;\n this.graphUrl = null;\n this.updateStats();\n },\n\n startLoop: function(i) {\n this.interval = i;\n this.poller = setTimeout(this.updateStats.bind(this), i);\n },\n\n updateStats: function() {\n $.get(AppState.apiPath('/counter')).done(function(data) {\n if (this.removed) {\n return;\n }\n\n var num = _.reduce(data['stats'], function(n, v) {\n return n + v['message_count'];\n }, 0);\n\n if (this.currentNum === -1) {\n // seed the display\n this.currentNum = num;\n this.lastNum = num;\n this.writeCounts(this.currentNum);\n } else if (num > this.lastNum) {\n var delta = num - this.lastNum;\n this.delta = (delta / (this.interval / 1000)) / 50;\n this.lastNum = num;\n\n if (!this.animator) {\n this.displayFrame();\n }\n }\n\n var newInterval = this.interval;\n if (newInterval < this.targetPollInterval) {\n newInterval = this.interval + 1000;\n }\n this.startLoop(newInterval);\n\n $('#warning, #error').hide();\n if (data['message'] !== '') {\n $('#warning .alert').text(data['message']);\n $('#warning').show();\n }\n }.bind(this)).fail(function(jqXHR) {\n if (this.removed) {\n return;\n }\n\n clearTimeout(this.animator);\n this.animator = null;\n\n this.startLoop(10000);\n\n this.handleAJAXError(jqXHR);\n }.bind(this));\n\n if ($('#big_graph').length) {\n if (!this.graphUrl) {\n this.graphUrl = $('#big_graph').attr('src');\n }\n var uniq = Math.floor(Math.random() * 1000000);\n $('#big_graph').attr('src', this.graphUrl + '&_uniq=' + uniq);\n }\n },\n\n displayFrame: function() {\n this.currentNum = Math.min(this.currentNum + this.delta, this.lastNum);\n this.writeCounts(this.currentNum);\n if (this.currentNum < this.lastNum) {\n this.animator = setTimeout(this.displayFrame.bind(this), 1000 / 60);\n } else {\n this.animator = null;\n }\n },\n\n writeCounts: function(c) {\n var text = parseInt(c, 10).toString();\n var node = $('.numbers')[0];\n var n = $('.numbers .number');\n for (var i = 0; i < text.length; i++) {\n var v = text.charAt(i);\n if (n.length > i) {\n var el = $(n[i]);\n el.show();\n el.find('.top').text(v);\n el.find('.bottom').text(v);\n } else {\n $(node).append('' + v +\n '' + v + '\\n');\n }\n }\n $('.numbers .number').each(function(ii, vv) {\n if (ii >= text.length) {\n $(vv).hide();\n }\n });\n }\n});\n\nmodule.exports = CounterView;\n" }, { "path": "nsqadmin/static/js/views/error.hbs", "content": "
    \n
    \n
    \n {{message}}\n
    \n
    \n
    \n" }, { "path": "nsqadmin/static/js/views/header.hbs", "content": "\n" }, { "path": "nsqadmin/static/js/views/header.js", "content": "var _ = require('underscore');\nvar $ = require('jquery');\n\nvar AppState = require('../app_state');\n\nvar BaseView = require('./base');\n\nvar HeaderView = BaseView.extend({\n className: 'header',\n\n template: require('./header.hbs'),\n\n events: {\n 'click .dropdown-menu li': 'onGraphIntervalClick'\n },\n\n initialize: function() {\n BaseView.prototype.initialize.apply(this, arguments);\n this.listenTo(AppState, 'change:graph_interval', this.render);\n },\n\n getRenderCtx: function() {\n return _.extend(BaseView.prototype.getRenderCtx.apply(this, arguments), {\n 'graph_intervals': ['1h', '2h', '12h', '24h', '48h', '168h', 'off'],\n 'graph_interval': AppState.get('graph_interval')\n });\n },\n\n onReset: function() {\n this.render();\n this.$('.dropdown-toggle').dropdown();\n },\n\n onGraphIntervalClick: function(e) {\n e.stopPropagation();\n AppState.set('graph_interval', $(e.target).text());\n }\n});\n\nmodule.exports = HeaderView;\n" }, { "path": "nsqadmin/static/js/views/lookup.hbs", "content": "{{> warning}}\n{{> error}}\n\n
    \n
    \n

    Lookup

    \n
    \n
    \n\n{{#unless nsqlookupd.length}}\n
    \n

    Notice

    nsqadmin is not configured with nsqlookupd hosts\n
    \n{{else}}\n
    \n
    \n \n \n \n \n {{#each nsqlookupd}}\n \n {{/each}}\n
    nsqlookupd Host
    {{this}}
    \n
    \n
    \n\n
    \n
    \n {{#if topics}}\n
    \n Below is a tree of Topics/Channels that are currently inactive (i.e. not produced on any nsqd in the cluster but are present in the lookup data)\n
    \n
      \n {{#each topics}}\n
    • \n {{name}}\n
        \n {{#each channels}}\n
      • \n {{this}}\n
        • \n {{/each}}\n
      \n
      • \n {{/each}}\n
    \n {{else}}\n

    Notice

    No inactive Topics
    \n {{/if}}\n
    \n
    \n\n{{#if isAdmin}}\n
    \n
    \n
    \n Create Topic/Channel\n
    \n

    This provides a way to setup a stream hierarchy\n before services are deployed to production.\n

    If Channel Name is empty, just the topic is created.\n

    \n
    \n \n \n
    \n \n
    \n
    \n
    \n{{/if}}\n{{/unless}}\n" }, { "path": "nsqadmin/static/js/views/lookup.js", "content": "var _ = require('underscore');\nvar $ = require('jquery');\n\nvar AppState = require('../app_state');\nvar Pubsub = require('../lib/pubsub');\nvar BaseView = require('./base');\n\nvar Topic = require('../models/topic');\nvar Channel = require('../models/channel');\n\nvar LookupView = BaseView.extend({\n className: 'lookup container-fluid',\n\n template: require('./spinner.hbs'),\n\n events: {\n 'click .hierarchy button': 'onCreateTopicChannel',\n 'click .delete-topic-link': 'onDeleteTopic',\n 'click .delete-channel-link': 'onDeleteChannel'\n },\n\n initialize: function() {\n BaseView.prototype.initialize.apply(this, arguments);\n var isAdmin = arguments[0]['isAdmin'];\n $.ajax(AppState.apiPath('/topics?inactive=true'))\n .done(function(data) {\n this.template = require('./lookup.hbs');\n this.render({\n 'topics': _.map(data['topics'], function(v, k) {\n return {'name': k, 'channels': v};\n }),\n 'message': data['message'],\n 'isAdmin': isAdmin\n });\n }.bind(this))\n .fail(this.handleViewError.bind(this))\n .always(Pubsub.trigger.bind(Pubsub, 'view:ready'));\n },\n\n onCreateTopicChannel: function(e) {\n e.preventDefault();\n e.stopPropagation();\n var topic = $(e.target.form.elements['topic']).val();\n var channel = $(e.target.form.elements['channel']).val();\n if (topic === '' && channel === '') {\n return;\n }\n $.post(AppState.apiPath('/topics'), JSON.stringify({\n 'topic': topic,\n 'channel': channel\n }))\n .done(function() { window.location.reload(true); })\n .fail(this.handleAJAXError.bind(this));\n },\n\n onDeleteTopic: function(e) {\n e.preventDefault();\n e.stopPropagation();\n var topic = new Topic({\n 'name': $(e.target).data('topic')\n });\n topic.destroy({'dataType': 'text'})\n .done(function() { window.location.reload(true); })\n .fail(this.handleAJAXError.bind(this));\n },\n\n onDeleteChannel: function(e) {\n e.preventDefault();\n e.stopPropagation();\n var channel = new Channel({\n 'topic': $(e.target).data('topic'),\n 'name': $(e.target).data('channel')\n });\n channel.destroy({'dataType': 'text'})\n .done(function() { window.location.reload(true); })\n .fail(this.handleAJAXError.bind(this));\n }\n});\n\nmodule.exports = LookupView;\n" }, { "path": "nsqadmin/static/js/views/node.hbs", "content": "{{> warning}}\n{{> error}}\n\n
      \n
    1. Nodes\n
    2. {{name}}
      3. \n
    \n\n{{#if graph_active}}\n
    \n
    \n \n \n \n \n \n \n \n
    \n \n
    GC Pressure
    \n     		\n \n
    GC Pause Percentiles
    \n     		\n \n
    GC Runs
    \n     		\n \n
    Heap Objects In-Use
    \n
    \n
    \n
    \n{{/if}}\n\n
    \n
    \n {{#unless topics.length}}\n
    \n

    Notice

    No topics exist on this node.\n
    \n {{else}}\n \n \n \n \n \n \n \n {{#each topics}}\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n {{#unless channels.length}}\n \n \n \n {{else}}\n {{#each channels}}\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n {{#unless clients.length}}\n \n \n \n {{else}}\n \n \n \n \n \n \n \n \n \n \n \n \n \n {{#each clients}}\n \n \n \n \n \n \n \n \n \n \n \n \n \n {{/each}}\n {{/unless}}\n {{/each}}\n {{/unless}}\n {{/each}}\n
    {{../topics.length}} Topics{{commafy total_messages}} Messages{{commafy total_clients}} Clients
    TopicDepthMemory + DiskMessagesChannels
    \n {{topic_name}}\n {{#if paused}}paused{{/if}}\n \n {{#if ../graph_active}}{{/if}}\n {{commafy depth}}\n {{commafy memory_depth}} + {{commafy backend_depth}}\n {{#if ../graph_active}}{{/if}}\n {{commafy message_count}}\n {{commafy channels.length}}
    \n

    Notice

    No channels exist for this topic.
    \n
    ChannelDepthMemory + DiskIn-FlightDeferredRequeuedTimed OutMessagesConnections
    \n {{channel_name}}\n {{#if paused}}paused{{/if}}\n \n {{#if ../../../graph_active}}{{/if}}\n {{commafy depth}}\n {{commafy memory_depth}} + {{commafy backend_depth}}{{commafy in_flight_count}}{{commafy deferred_count}}{{commafy requeue_count}}{{commafy timeout_count}}{{commafy message_count}}\n {{#if ../../../graph_active}}{{/if}}\n {{commafy clients.length}}\n
    \n

    Notice

    No clients connected to this channel.
    \n
    Client HostUser-AgentAttributesIn-FlightReady CountRequeuedFinishedMessagesConnected
    {{hostname}}{{#if show_client_id}} ({{client_id}}){{/if}}{{#if user_agent.length}}{{user_agent}}{{/if}}\n {{#if sample_rate}}\n Sampled {{sample_rate}}%\n {{/if}}\n {{#if tls}}\n TLS\n {{/if}}\n {{#if deflate}}\n Deflate\n {{/if}}\n {{#if snappy}}\n Snappy\n {{/if}}\n {{#if authed}}\n \n {{#if auth_identity_url}}{{/if}}\n \n {{#if auth_identity_url}}{{/if}}\n \n {{/if}}\n {{commafy in_flight_count}}{{commafy ready_count}}{{commafy requeue_count}}{{commafy finish_count}}{{commafy message_count}}{{nanotohuman connected}}
    \n {{/unless}}\n
    \n" }, { "path": "nsqadmin/static/js/views/node.js", "content": "var Pubsub = require('../lib/pubsub');\nvar AppState = require('../app_state');\n\nvar BaseView = require('./base');\n\nvar NodeView = BaseView.extend({\n className: 'node container-fluid',\n\n template: require('./spinner.hbs'),\n\n initialize: function() {\n BaseView.prototype.initialize.apply(this, arguments);\n this.listenTo(AppState, 'change:graph_interval', this.render);\n this.model.fetch()\n .done(function(data) {\n this.template = require('./node.hbs');\n this.render({'message': data['message']});\n }.bind(this))\n .fail(this.handleViewError.bind(this))\n .always(Pubsub.trigger.bind(Pubsub, 'view:ready'));\n }\n});\n\nmodule.exports = NodeView;\n" }, { "path": "nsqadmin/static/js/views/nodes.hbs", "content": "{{> warning}}\n{{> error}}\n\n
    \n
    \n

    NSQd Nodes ({{collection.length}})

    \n
    \n
    \n\n
    \n
    \n \n \n \n \n \n \n \n \n \n {{#if nsqlookupd.length}}\n \n {{/if}}\n \n \n {{#each collection}}\n \n \n \n \n \n \n \n \n {{#if ../nsqlookupd.length}}\n \n {{/if}}\n \n \n {{/each}}\n
    HostnameBroadcast AddressTCP PortHTTP PortVersionRegionZoneLookupd Conns.Topics
    {{hostname}}{{broadcast_address}}{{tcp_port}}{{http_port}}{{version}}{{topology_region}}{{topology_zone}}\n {{remote_addresses.length}}\n
    \n {{#each remote_addresses}}{{this}}
    {{/each}}\n
    \n     		\n {{#if topics.length}}\n {{topics.length}}\n {{#each topics}}\n {{topic}}\n {{/each}}\n {{/if}}\n
    \n
    \n
    \n" }, { "path": "nsqadmin/static/js/views/nodes.js", "content": "var $ = require('jquery');\n\nvar Pubsub = require('../lib/pubsub');\nvar AppState = require('../app_state');\n\nvar BaseView = require('./base');\n\nvar Nodes = require('../collections/nodes');\n\nvar NodesView = BaseView.extend({\n className: 'nodes container-fluid',\n\n template: require('./spinner.hbs'),\n\n events: {\n 'click .conn-count': 'onClickConnCount'\n },\n\n initialize: function() {\n BaseView.prototype.initialize.apply(this, arguments);\n this.listenTo(AppState, 'change:graph_interval', this.render);\n this.collection = new Nodes();\n this.collection.fetch()\n .done(function(data) {\n this.template = require('./nodes.hbs');\n this.render({'message': data['message']});\n }.bind(this))\n .fail(this.handleViewError.bind(this))\n .always(Pubsub.trigger.bind(Pubsub, 'view:ready'));\n },\n\n onClickConnCount: function(e) {\n e.preventDefault();\n $(e.target).next().toggle();\n }\n});\n\nmodule.exports = NodesView;\n" }, { "path": "nsqadmin/static/js/views/spinner.hbs", "content": "
    \n \n \n \n
    \n" }, { "path": "nsqadmin/static/js/views/topic.hbs", "content": "{{> warning}}\n{{> error}}\n\n
      \n
    1. Streams\n
    2. {{name}}
      3. \n
    \n\n
    \n
    \n
    \n

    Topic: {{name}}\n

    \n
    \n
    \n\n{{#unless nodes.length}}\n
    \n
    \n

    Topic Message Queue

    \n
    \n

    Notice

    No producers exist for this topic.\n

    See Lookup for more information.\n

    \n
    \n
    \n{{else}}\n{{#if isAdmin}}\n
    \n
    \n \n
    \n
    \n \n
    \n
    \n {{#if paused}}\n \n {{else}}\n \n {{/if}}\n
    \n
    \n{{/if}}\n\n
    \n
    \n

    Topic Message Queue

    \n \n {{#if e2e_processing_latency.percentiles.length}}\n \n \n \n \n {{/if}}\n \n \n \n \n \n {{#if graph_active}}{{/if}}\n \n {{#each e2e_processing_latency.percentiles}}\n \n {{/each}}\n \n {{#each nodes}}\n \n \n \n \n \n {{#if ../graph_active}}\n \n {{/if}}\n \n {{#if e2e_processing_latency.percentiles.length}}\n {{#each e2e_processing_latency.percentiles}}\n \n {{/each}}\n {{/if}}\n \n {{#if ../graph_active}}\n \n \n \n \n \n \n \n {{#if e2e_processing_latency.percentiles.length}}\n \n {{/if}}\n \n {{/if}}\n {{/each}}\n \n \n \n \n \n {{#if graph_active}}{{/if}}\n \n {{#if e2e_processing_latency.percentiles.length}}\n {{#each e2e_processing_latency.percentiles}}\n \n {{/each}}\n {{/if}}\n \n {{#if graph_active}}\n \n \n \n \n \n \n \n {{#if e2e_processing_latency.percentiles.length}}\n \n {{/if}}\n \n {{/if}}\n
    E2E Processing Latency
    NSQd HostDepthMemory + DiskMessagesRateChannels{{floatToPercent quantile}}{{percSuffix quantile}}
    \n \n {{#if show_broadcast_address}}\n {{hostname_port}} ({{node}})\n {{else}}\n {{hostname_port}}\n {{/if}}\n {{#if paused}} paused{{/if}}\n {{commafy depth}}{{commafy memory_depth}} + {{commafy backend_depth}}{{commafy message_count}}{{commafy this/channels.length}}\n {{nanotohuman average}}\n
    \n \n
    Total:{{commafy depth}}{{commafy memory_depth}} + {{commafy backend_depth}}{{commafy message_count}}{{commafy channels.length}}\n {{nanotohuman average}}\n
    \n \n
    \n
    \n
    \n{{/unless}}\n\n\n
    \n {{#unless channels.length}}\n
    \n

    Channel Message Queues

    \n
    \n

    Notice

    No channels exist for this topic.\n

    Messages will queue at the topic until a channel is created.\n

    \n {{else}}\n
    \n

    Channel Message Queues

    \n \n {{#if e2e_processing_latency.percentiles.length}}\n \n \n \n \n {{/if}}\n \n \n \n \n \n \n \n \n \n \n {{#each e2e_processing_latency.percentiles}}\n \n {{/each}}\n \n\n {{#each channels}}\n \n \n \n \n \n \n \n \n \n \n {{#if e2e_processing_latency.percentiles.length}}\n {{#each e2e_processing_latency.percentiles}}\n \n {{/each}}\n {{/if}}\n \n {{#if ../graph_active}}\n \n \n \n \n \n \n \n \n \n \n {{#if e2e_processing_latency.percentiles.length}}\n \n {{/if}}\n \n {{/if}}\n {{/each}}\n
    E2E Processing Latency
    ChannelDepthMemory + DiskIn-FlightDeferredRequeuedTimed OutMessagesConnections{{floatToPercent quantile}}{{percSuffix quantile}}
    \n {{channel_name}}\n {{#if paused}}paused{{/if}}\n {{commafy depth}}{{commafy memory_depth}} + {{commafy backend_depth}}{{commafy in_flight_count}}{{commafy deferred_count}}{{commafy requeue_count}}{{commafy timeout_count}}{{commafy message_count}}{{commafy client_count}}\n {{nanotohuman average}}\n
    \n \n
    \n {{/unless}}\n
    \n
    \n" }, { "path": "nsqadmin/static/js/views/topic.js", "content": "var $ = require('jquery');\n\nwindow.jQuery = $;\nvar bootstrap = require('bootstrap'); //eslint-disable-line no-unused-vars\nvar bootbox = require('bootbox');\n\nvar Pubsub = require('../lib/pubsub');\nvar AppState = require('../app_state');\n\nvar BaseView = require('./base');\n\nvar TopicView = BaseView.extend({\n className: 'topic container-fluid',\n\n template: require('./spinner.hbs'),\n\n events: {\n 'click .topic-actions button': 'topicAction'\n },\n\n initialize: function() {\n BaseView.prototype.initialize.apply(this, arguments);\n this.listenTo(AppState, 'change:graph_interval', this.render);\n var isAdmin = this.model.get('isAdmin');\n this.model.fetch()\n .done(function(data) {\n this.template = require('./topic.hbs');\n this.render({'message': data['message'], 'isAdmin': isAdmin});\n }.bind(this))\n .fail(this.handleViewError.bind(this))\n .always(Pubsub.trigger.bind(Pubsub, 'view:ready'));\n },\n\n topicAction: function(e) {\n e.preventDefault();\n e.stopPropagation();\n var action = $(e.currentTarget).data('action');\n var txt = 'Are you sure you want to ' +\n action + ' ' + this.model.get('name') + '?';\n bootbox.confirm(txt, function(result) {\n if (result !== true) {\n return;\n }\n if (action === 'delete') {\n $.ajax(this.model.url(), {'method': 'DELETE'})\n .done(function() { window.location = AppState.basePath('/'); });\n } else {\n $.post(this.model.url(), JSON.stringify({'action': action}))\n .done(function() { window.location.reload(true); })\n .fail(this.handleAJAXError.bind(this));\n }\n }.bind(this));\n }\n});\n\nmodule.exports = TopicView;\n" }, { "path": "nsqadmin/static/js/views/topics.hbs", "content": "{{> warning}}\n{{> error}}\n\n
    \n
    \n

    Topics

    \n
    \n
    \n\n
    \n
    \n {{#if collection.length}}\n \n \n \n {{#if graph_active}}{{/if}}\n {{#if graph_active}}{{/if}}\n {{#if graph_active}}{{/if}}\n \n {{#each collection}}\n \n \n {{#if ../graph_active}}{{/if}}\n {{#if ../graph_active}}{{/if}}\n {{#if ../graph_active}}{{/if}}\n \n {{/each}}\n
    TopicDepthMessagesRate
    {{name}}
    \n {{else}}\n

    Notice

    No Topics Found
    \n {{/if}}\n
    \n
    \n" }, { "path": "nsqadmin/static/js/views/topics.js", "content": "var Pubsub = require('../lib/pubsub');\nvar AppState = require('../app_state');\n\nvar BaseView = require('./base');\nvar Topics = require('../collections/topics');\n\nvar TopicsView = BaseView.extend({\n className: 'topics container-fluid',\n\n template: require('./spinner.hbs'),\n\n initialize: function() {\n BaseView.prototype.initialize.apply(this, arguments);\n this.listenTo(AppState, 'change:graph_interval', this.render);\n this.collection = new Topics();\n this.collection.fetch()\n .done(function(data) {\n this.template = require('./topics.hbs');\n this.render({'message': data['message']});\n }.bind(this))\n .fail(this.handleViewError.bind(this))\n .always(Pubsub.trigger.bind(Pubsub, 'view:ready'));\n }\n});\n\nmodule.exports = TopicsView;\n" }, { "path": "nsqadmin/static/js/views/warning.hbs", "content": "
    \n
    \n
    \n {{message}}\n
    \n
    \n
    \n" }, { "path": "nsqadmin/static.go", "content": "package nsqadmin\n\nimport (\n\t\"embed\"\n)\n\n//go:embed static/build\nvar static embed.FS\n\nfunc staticAsset(name string) ([]byte, error) {\n\treturn static.ReadFile(\"static/build/\" + name)\n}\n" }, { "path": "nsqadmin/test/ca.key", "content": "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC,D6208487450BDF1C\n\ntucU/4j6agGQlW60D8V3Zr/QHcLhhyFagres1gGdWfGIqluNZb7omki/XidHXJSG\neB9vV2Xb12/8umc31e7Mnmn9hd34230v/KlAnJ4ukDpJpbmjnEx3F9uiqYFi/yxQ\navSsfF6Tsh3XOh3Oe27I/xfYx37g6Agd+EQEJ1hvWvygMIJMTDMP5ZaFoZANtFLy\nhDEZ6woJSn9avF/L+1GW8jl2aI1QbdKkK0jDHgFAwUI4sjWeXvEQNNYY3trTIoMo\nwab3vi+4XziFONbS4OZrZUYfZPB5YOFbtT2whzggp2HdSTiu48/Ld3N8SjuMrKfm\nuR+nd+ovQ5kVWHInzWAIXSyPhgR9ZY8eyXaHNJJfzNu3HY72lfzD/NtZfacMRBr6\nM3Wg/OKPS7ZrtqCWkY9P3KK9Cul8Jzy229fSqHo8Rg4=\n-----END RSA PRIVATE KEY-----\n" }, { "path": "nsqadmin/test/ca.pem", "content": "-----BEGIN CERTIFICATE-----\nMIIBVDCB/6ADAgECAgkAvHG4Z/7nX/gwDQYJKoZIhvcNAQEFBQAwDTELMAkGA1UE\nAwwCY2EwHhcNMTcwOTE2MTc0NDE0WhcNMjcwOTE0MTc0NDE0WjANMQswCQYDVQQD\nDAJjYTBcMA0GCSqGSIb3DQEBAQUAA0sAMEgCQQDJAM3Tr1BoxlLJtTy0oRcp93dT\n9hhHwms8P1V3k2FpXYRS4deUo+uwcAM9KGDt9VMXVBEchtI4VYTvLgatBPUBAgMB\nAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFMyLx7rjKBe/xZQLnVzI\nuqNNVzxRMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAANBAJiN1XgPPlQ2\nz7PhtzXStaz/BJVqhD7g9fsZsmoPX4ifDsTfzUsRB56Aq/NTsKiIYQkFPHH0donG\n++a5ZVWjgYk=\n-----END CERTIFICATE-----\n" }, { "path": "nsqadmin/test/ca.srl", "content": "91418D04995922E7\n" }, { "path": "nsqadmin/test/cert.pem", "content": "-----BEGIN CERTIFICATE-----\nMIIEbjCCA1agAwIBAgIJAK6x7y6AwBmLMA0GCSqGSIb3DQEBBQUAMIGAMQswCQYD\nVQQGEwJVUzERMA8GA1UECBMITmV3IFlvcmsxFjAUBgNVBAcTDU5ldyBZb3JrIENp\ndHkxDDAKBgNVBAoTA05TUTETMBEGA1UEAxMKdGVzdC5sb2NhbDEjMCEGCSqGSIb3\nDQEJARYUbXJlaWZlcnNvbkBnbWFpbC5jb20wHhcNMTMwNjI4MDA0MzQ4WhcNMTYw\nNDE3MDA0MzQ4WjCBgDELMAkGA1UEBhMCVVMxETAPBgNVBAgTCE5ldyBZb3JrMRYw\nFAYDVQQHEw1OZXcgWW9yayBDaXR5MQwwCgYDVQQKEwNOU1ExEzARBgNVBAMTCnRl\nc3QubG9jYWwxIzAhBgkqhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwuY29tMIIB\nIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnX0KB+svwy+yHU2qggz/EaGg\ncraKShagKo+9M9y5HLM852ngk5c+t+tJJbx3N954Wr1FXBuGIv1ltU05rU4zhvBS\n25tVP1UIEnT5pBt2TeetLkl199Y7fxh1hKmnwJMG3fy3VZdNXEndBombXMmtXpQY\nshuEJHKeUNDbQKz5X+GjEdkTPO/HY/VMHsxS23pbSimQozMg3hvLIdgv0aS3QECz\nydZBgTPThy3uDtHIuCpxCwXd/vDF68ATlYgo3h3lh2vxNwM/pjklIUhzMh4XaKQF\n7m3/0KbtUcXfy0QHueeuMr11E9MAFNyRN4xf9Fk1yB97KJ3PJBTC5WD/m1nW+QID\nAQABo4HoMIHlMB0GA1UdDgQWBBR3HMBws4lmYYSIgwoZsfW+bbgaMjCBtQYDVR0j\nBIGtMIGqgBR3HMBws4lmYYSIgwoZsfW+bbgaMqGBhqSBgzCBgDELMAkGA1UEBhMC\nVVMxETAPBgNVBAgTCE5ldyBZb3JrMRYwFAYDVQQHEw1OZXcgWW9yayBDaXR5MQww\nCgYDVQQKEwNOU1ExEzARBgNVBAMTCnRlc3QubG9jYWwxIzAhBgkqhkiG9w0BCQEW\nFG1yZWlmZXJzb25AZ21haWwuY29tggkArrHvLoDAGYswDAYDVR0TBAUwAwEB/zAN\nBgkqhkiG9w0BAQUFAAOCAQEANOYTbanW2iyV1v4oYpcM/y3TWcQKzSME8D2SGFZb\ndbMYU81hH3TTlQdvyeh3FAcdjhKE8Xi/RfNNjEslTBscdKXePGpZg6eXRNJzPP5K\nKZPf5u6tcpAeUOKrMqbGwbE+h2QixxG1EoVQtE421szsU2P7nHRTdHzKFRnOerfl\nPhm3NocR0P40Rv7WKdxpOvqc+XKf0onTruoVYoPWGpwcLixCG0zu4ZQ23/L/Dy18\n4u70Hbq6O/6kq9FBFaDNp3IhiEdu2Cq6ZplU6bL9XDF27KIEErHwtuqBHVlMG+zB\noH/k9vZvwH7OwAjHdKp+1yeZFLYC8K5hjFIHqcdwpZCNIg==\n-----END CERTIFICATE-----\n" }, { "path": "nsqadmin/test/client.key", "content": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAuMz5IJ01Wqaj2qJAU7o2aoc2iP1BRhTba+cWrsjbZA+6kXAk\n19Dlz/nNK8gxsBMl3rzTGWXKUm/rKBDp680zJkj8nzcGApGlzbDoqXym8ixOniOC\naH/RODPC8OtD2UozeXlUOQWK+ufkMvmVBc/IeJyFkcm7jhX+LHVFn+3iWLRNbzGm\nhbf8oNHvybS7HSuV3OYiHDJxA0d6IgozqVOYVSqdDabQ2FN/8nKzpMHjwMpfT008\n5hUhr72E5XuJ3biPUwoOrRETSa1/EQcAEH0wSwHU1AGdnGh3/5ynVupfef0qkyqw\nNseYs9U++IUFXqA1IuEmAlJn5QczAHjPQe8xBQIDAQABAoIBAGx4CyZEgCOUOgrD\nP2SloPkIIk9n7x82cNA11I+E35kszkI9g7KVL77SDcZL/DYwFwNU68c1gvq+LFXZ\nD6RTTlmDb5v4TPPHD33a/8UzoD33GbIif5HcrC4D28FTJgDtV6dOOsw5X6kD4WK2\nMe02V6HLpW677PVqHUV1FAfaNgf/1SIhJCechYSO+3pKPtli8DnsmOd336wNtWZR\nQ3Ctm63Eq7AiLmGLOkLYtll9Kr6WfCSFHBmqJg9MsQEfINXMCOB54Z+xkoijjJLO\nzNSk/3XvppB1JpmAusbo9Mq03Ci9JdinYSKxwVOG6e+5cosrOHpDQyZIWa87ZTpk\nkjLCgEECgYEA33JCGTEwaqfAuYOspCZvi2Mqz7t6QiwvwxgDf4JFLgDQArJfzKlC\nbAzaSU8bT+SN96CF9LU4XekN5/paGZAFjWxraPkK0oL28kQrtzfEQv2XL6GFoNAh\nWsQtoaGddL9VPWlCiHeueYERIDNrZCaVeAx6OV6W3CEwGOkRiQyntlUCgYEA07lf\nLpxF8aAgMToq3++D5RGAXaT2PYxhZZFb5Y2tS7a4T0ulQwXQi4H2dObGJ32zCjFG\nls1H91mNKGBbyKnF1jT/dt/acqymvipASm1xq6uysIllA4xp56sDnJgG9bnXjYFY\nm2yVNIoIQGZ7jp7e6t1KQ9San6V8asWlvBLXX/ECgYEAuhF9dVj+xnH3DQTXSMIw\n9NOZnO6zelMtWrqufwnN7ecDUJuVJupzw2JYi99yEO90QRbNNd+KlrkxuVFCojLK\nTOBR+VIZbv9cAJZACQxJRLfDpAhPLIDkpZ7jmMrqQYPqyX7TxqxTAB84UaY/8WAn\n65YIWamo2ppQYQ4Eaim9pxkCgYAMwfW/TEFWruxhqvycY8VRzz0p51/DE6tmwFyG\nN4RCtK7kcE1z/Wy0i087ehBknslkCtYTDimQ+P9teGjvbXNzVdwy4Ig8MrUVblxT\nX8birkTlKFJC5XoYMJDWJb79nYYki6+4JdHTyaF3p/U4AdCy3ES2U6BBkGov0NsM\nuyHpMQKBgGc9wIGHwAGLg3iVHGXbXELdpzVztLRC7D12TPfdtGBLTD5NH5jAx8K/\nw61+l4tzFcK3jfWzwdme277cBgfq/aw2DDt07vkOvPzhUwVZiE39bfddfIvyed99\nXzEMf8THh5wdm6mLsOVgdfOfFcDi5ReIv1/7+eOd/MQi8TxIVfAu\n-----END RSA PRIVATE KEY-----\n" }, { "path": "nsqadmin/test/client.pem", "content": "-----BEGIN CERTIFICATE-----\nMIICyzCCAnWgAwIBAgIJAJFBjQSZWSLnMA0GCSqGSIb3DQEBCwUAMA0xCzAJBgNV\nBAMMAmNhMB4XDTE3MDkxNjE3NDQxNFoXDTI3MDkxNDE3NDQxNFowgYcxIzAhBgkq\nhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwuY29tMQswCQYDVQQGEwJERTEMMAoG\nA1UECBMDTlJXMQ4wDAYDVQQHEwVFYXJ0aDEXMBUGA1UEChMOUmFuZG9tIENvbXBh\nbnkxCzAJBgNVBAsTAklUMQ8wDQYDVQQDEwZuc3EuaW8wggEiMA0GCSqGSIb3DQEB\nAQUAA4IBDwAwggEKAoIBAQC4zPkgnTVapqPaokBTujZqhzaI/UFGFNtr5xauyNtk\nD7qRcCTX0OXP+c0ryDGwEyXevNMZZcpSb+soEOnrzTMmSPyfNwYCkaXNsOipfKby\nLE6eI4Jof9E4M8Lw60PZSjN5eVQ5BYr65+Qy+ZUFz8h4nIWRybuOFf4sdUWf7eJY\ntE1vMaaFt/yg0e/JtLsdK5Xc5iIcMnEDR3oiCjOpU5hVKp0NptDYU3/ycrOkwePA\nyl9PTTzmFSGvvYTle4nduI9TCg6tERNJrX8RBwAQfTBLAdTUAZ2caHf/nKdW6l95\n/SqTKrA2x5iz1T74hQVeoDUi4SYCUmflBzMAeM9B7zEFAgMBAAGjdTBzMAwGA1Ud\nEwEB/wQCMAAwHQYDVR0OBBYEFMzYXiTD7moi0oXqSQ5D2LsO51GXMB8GA1UdIwQY\nMBaAFMyLx7rjKBe/xZQLnVzIuqNNVzxRMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\nDDAKBggrBgEFBQcDAjANBgkqhkiG9w0BAQsFAANBAAujxas6toRhMl/+kZrly0G/\nAvjbA3WY5cLLIdffGdQ5bsS3aOP23nj98ut7unNUNsCo+eUwpJgvabnFnL+NFZA=\n-----END CERTIFICATE-----\n" }, { "path": "nsqadmin/test/client.req", "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIICzTCCAbUCAQAwgYcxIzAhBgkqhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwu\nY29tMQswCQYDVQQGEwJERTEMMAoGA1UECBMDTlJXMQ4wDAYDVQQHEwVFYXJ0aDEX\nMBUGA1UEChMOUmFuZG9tIENvbXBhbnkxCzAJBgNVBAsTAklUMQ8wDQYDVQQDEwZu\nc3EuaW8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC4zPkgnTVapqPa\nokBTujZqhzaI/UFGFNtr5xauyNtkD7qRcCTX0OXP+c0ryDGwEyXevNMZZcpSb+so\nEOnrzTMmSPyfNwYCkaXNsOipfKbyLE6eI4Jof9E4M8Lw60PZSjN5eVQ5BYr65+Qy\n+ZUFz8h4nIWRybuOFf4sdUWf7eJYtE1vMaaFt/yg0e/JtLsdK5Xc5iIcMnEDR3oi\nCjOpU5hVKp0NptDYU3/ycrOkwePAyl9PTTzmFSGvvYTle4nduI9TCg6tERNJrX8R\nBwAQfTBLAdTUAZ2caHf/nKdW6l95/SqTKrA2x5iz1T74hQVeoDUi4SYCUmflBzMA\neM9B7zEFAgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEALD1VjbBjaeX7l7JR4IuL\nwrHVrFDiRYsgvqyw39j2MC95VwrGwzf4cXCE+RuqE/DhbV9UI7sWKaJfFs9Usq+g\nVSoKnHSEylt34y6ABSc5eAik+GnheJZbJ6UDjxcvNd0UpFGMrHbsXVyQd0Y1XAu7\nnxCYIa82kNA+Opb+ra03hkLC7wvRLbXTOB2g6JLkyhYR6S5GkOFTNnz2AJerN7zt\nNEL7owlRcjyIsL5tjDpDH1944NNtzhgmrUeIjB08reayuot9RKznMVGwBfY6DIHM\nQ4uNN3CMOOoAHr1UzvBf/qfvb6ltPTMKSV1OncLlC3C59NoO8vhKIHCN18Ya2OMu\nrw==\n-----END CERTIFICATE REQUEST-----\n" }, { "path": "nsqadmin/test/key.pem", "content": "-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEAnX0KB+svwy+yHU2qggz/EaGgcraKShagKo+9M9y5HLM852ng\nk5c+t+tJJbx3N954Wr1FXBuGIv1ltU05rU4zhvBS25tVP1UIEnT5pBt2TeetLkl1\n99Y7fxh1hKmnwJMG3fy3VZdNXEndBombXMmtXpQYshuEJHKeUNDbQKz5X+GjEdkT\nPO/HY/VMHsxS23pbSimQozMg3hvLIdgv0aS3QECzydZBgTPThy3uDtHIuCpxCwXd\n/vDF68ATlYgo3h3lh2vxNwM/pjklIUhzMh4XaKQF7m3/0KbtUcXfy0QHueeuMr11\nE9MAFNyRN4xf9Fk1yB97KJ3PJBTC5WD/m1nW+QIDAQABAoIBACvtfKbIywG+hAf4\nad7skRjx5DcbA2e29+XnQfb9UgTXWd2SgrmoLi5OypBkCTzkKN3mfTo70yZfV8dC\nSxwz+9tfnTz0DssjhKThS+CiaFVCkeOfSfBfKSlCQUVHrSrh18CDhP+yvDlJwQTZ\nzSQMfPcsh9bmJe2kqtQP7ZgUp1o+vaB8Sju8YYrO6FllxbdLRGm4pfvvrHIRRmXa\noVHn0ei0JpwoTY9kHYht4LNeJnbP/MCWdmcuv3Gnel7jAlhaKab5aNIGr0Xe7aIQ\niX6mpZ0/Rnt8o/XcTOg8l3ruIdVuySX6SYn08JMnfFkXdNYRVhoV1tC5ElWkaZLf\nhPmj2yECgYEAyts0R0b8cZ6HTAyuLm3ilw0s0v0/MM9ZtaqMRilr2WEtAhF0GpHG\nTzmGnii0WcTNXD7NTsNcECR/0ZpXPRleMczsL2Juwd4FkQ37h7hdKPseJNrfyHRg\nVolOFBX9H14C3wMB9cwdsG4Egw7fE27WCoreEquHgwFxl1zBrXKH088CgYEAxr8w\nBKZs0bF7LRrFT5pH8hpMLYHMYk8ZIOfgmEGVBKDQCOERPR9a9kqUss7wl/98LVNK\nRnFlyWD6Z0/QcQsLL4LjBeZJ25qEMc6JXm9VGAzhXA1ZkUofVoYCnG+f6KUn8CuJ\n/AcV2ZDFsEP10IiQG0hKsceXiwFEvEr8306tMrcCgYBLgnscSR0xAeyk71dq6vZc\necgEpcX+2kAvclOSzlpZ6WVCjtKkDT0/Qk+M0eQIQkybGLl9pxS+4Yc+s2/jy2yX\npwsHvGE0AvwZeZX2eDcdSRR4bYy9ZixyKdwJeAHnyivRbaIuJ5Opl9pQGpoI9snv\n1K9DTdw8dK4exKVHdgl/WwKBgDkmLsuXg4EEtPOyV/xc08VVNIR9Z2T5c7NXmeiO\nKyiKiWeUOF3ID2L07S9BfENozq9F3PzGjMtMXJSqibiHwW6nB1rh7mj8VHjx9+Q0\nxVZGFeNfX1r84mgB3uxW2LeQDhzsmB/lda37CC14TU3qhu2hawEV8IijE73FHlOk\nDv+fAoGAI4/XO5o5tNn5Djo8gHmGMCbinUE9+VySxl7wd7PK8w2VSofO88ofixDk\nNX94yBYhg5WZcLdPm45RyUnq+WVQYz9IKUrdxLFTH+wxyzUqZCW7jgXCvWV+071q\nvqm9C+kndq+18/1VKuCSGWnF7Ay4lbsgPXY2s4VKRxcb3QpZSPU=\n-----END RSA PRIVATE KEY-----\n" }, { "path": "nsqadmin/test/server.key", "content": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAuYs7NerYpXWq/IPaBGriHobhIs/i4AJHra7QhZ5N/EfBCb0N\n6K3tI4GRHMZ4lB0HwINWmxaPs6D2yDCxpGgnru6qCu0/91+Lq+GEZj+Fz1E2yfCq\ntecTySvWf9en+SdTKFZTGGIAu3zFcozuX2fdhkjJ6ibZhnrXKlPU4IN0STkkRfFX\n+yb/6P88mGh7PZKBb67Qw1bppzXE/SWgHVqDqu++whT1quwxnRhh/SHgwugN+6iK\nuTDcJ1jQ1QWe+xna7Te6nNBUvL6dV49jpoDOjHsv+UaAvzNeHCHqypTC0wwYHXyg\nTTAU+E20c2NOCC7P0PAid1liNeuuEWtM2bRrFQIDAQABAoIBABPzc7d1fDw2bd9f\nMic9cvkDWdwLbILX2+tCG+vyPMJ+2LP6Xy+A3DnwKbFlafvLL1U1Ci/8+hC/oymd\nisx54qJ9yU0Je9JWtMcTpc/0zqefPPvz4/dRVKBSFWuDve0dnGR++8poZ1nBrd2G\nZ+9cVMamtwd1i/hY5yAHCaHmoK9qxXuSjtbPkqDqruloGO5iT+tcICjAeWyJIx1P\n5VJjuKDx4AhU4B8DpBSTtA02BKltvr2K6D4zhJ255oI4iLUkYk4IwxB7yZH+6Y++\nmDa6e8iM3ut+qMkCxto/CfMCqYfoRzmykiazZ5qFXfGwv3OkWUEOBXU//Vpklvnh\nOQWYL/UCgYEA7mkKXbCrCNJEdly4LVLPKPFjVUPhMdXcqdnoKZ6BiVYFMqTuG4+w\no/Rf6ZpqZ3Rf656/ypd1atVvK2g4zfq3j1W6lDnpLjKaLhc8a+ZH4NWXBORIzlUW\naR2xVcPpAokor37FRAWbduiwVSCHlDtg48i+rvukyqmeRBKjrIgX3lsCgYEAxzut\n5GiMZ9TkjqYXdlUDIDKWMHMfLfRY7+Q/cBdr7AfiMVw4PIWJG/IuxCGwGp77gRg6\nBeBsi8Htj5/EGbRiK4kjFyf8LqYGOeYLZ3I8+olr+tNmF4/yKwxku3VylpmrpCAV\n/8emso1rAWo6Y98MTNSGgvLqhDU6OD7tTSv3908CgYA4+aVeio/1RbrSxonFWxri\n3/0rLVOuAzv+43KWL6kpVwNa/QtiTs6aABbDzwFKxAcAWinfkp6e727n4rpgj2A6\nwvQZ5FUTk0hBZ5ArAReAZcr3gk7b8H2wlUYCBxWyY3Dzr8oY3XYvzqAFWAbOp/oZ\ntanMS5swS6TlA8dVvhhmLQKBgGZwcBPOEctdcntKOSwVv/qxJ/oXZ0O4rHYENP4M\nfOgqkYnxsdSkkH/3AUbFT4gQkJ6q90KIRyeA+gXsDudskUFzTMCeRZMyuGbSurBg\n06u6NvQL+CVLVSf/QlgEpnt63f8QpF8Up8iM4CUlGoq5Z9ilOdhg0GZT+/BpopgY\ncHIPAoGBAKyLumgeG+gMFK5x/Fi6zjBT5MK8Tw0VMkahW01Jx2laYibS/a8AEFmn\nySdrmLPkOmmWgXCk3m2m5PkvM5qH/KNugOA0+WX2CTvwt4ZgwWUalQ5R43wSIeDC\nMXVfwC8uE66PmpYgmsu4H0vnCGfacOCQhfdq01SLobgBiQSrm/6D\n-----END RSA PRIVATE KEY-----\n" }, { "path": "nsqadmin/test/server.pem", "content": "-----BEGIN CERTIFICATE-----\nMIIC3jCCAoigAwIBAgIJAJFBjQSZWSLmMA0GCSqGSIb3DQEBCwUAMA0xCzAJBgNV\nBAMMAmNhMB4XDTE3MDkxNjE3NDQxNFoXDTI3MDkxNDE3NDQxNFowgYcxIzAhBgkq\nhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwuY29tMQswCQYDVQQGEwJERTEMMAoG\nA1UECBMDTlJXMQ4wDAYDVQQHEwVFYXJ0aDEXMBUGA1UEChMOUmFuZG9tIENvbXBh\nbnkxCzAJBgNVBAsTAklUMQ8wDQYDVQQDEwZuc3EuaW8wggEiMA0GCSqGSIb3DQEB\nAQUAA4IBDwAwggEKAoIBAQC5izs16tildar8g9oEauIehuEiz+LgAketrtCFnk38\nR8EJvQ3ore0jgZEcxniUHQfAg1abFo+zoPbIMLGkaCeu7qoK7T/3X4ur4YRmP4XP\nUTbJ8Kq15xPJK9Z/16f5J1MoVlMYYgC7fMVyjO5fZ92GSMnqJtmGetcqU9Tgg3RJ\nOSRF8Vf7Jv/o/zyYaHs9koFvrtDDVumnNcT9JaAdWoOq777CFPWq7DGdGGH9IeDC\n6A37qIq5MNwnWNDVBZ77GdrtN7qc0FS8vp1Xj2OmgM6Mey/5RoC/M14cIerKlMLT\nDBgdfKBNMBT4TbRzY04ILs/Q8CJ3WWI1664Ra0zZtGsVAgMBAAGjgYcwgYQwDAYD\nVR0TAQH/BAIwADAdBgNVHQ4EFgQUt6UXJ8BQB++I/cQG5ZgaiUyqOE8wHwYDVR0j\nBBgwFoAUzIvHuuMoF7/FlAudXMi6o01XPFEwDgYDVR0PAQH/BAQDAgWgMBMGA1Ud\nJQQMMAoGCCsGAQUFBwMBMA8GA1UdEQQIMAaHBH8AAAEwDQYJKoZIhvcNAQELBQAD\nQQCCwm0F2eTmtIcKDxrXGZ7q9y9mfsROfYaCnH+vKUDw2vmqQkInzhLeEElDGQcR\nww0IKCnDHEruNb2tKyQM/70L\n-----END CERTIFICATE-----\n" }, { "path": "nsqadmin/test/server.req", "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIICzTCCAbUCAQAwgYcxIzAhBgkqhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwu\nY29tMQswCQYDVQQGEwJERTEMMAoGA1UECBMDTlJXMQ4wDAYDVQQHEwVFYXJ0aDEX\nMBUGA1UEChMOUmFuZG9tIENvbXBhbnkxCzAJBgNVBAsTAklUMQ8wDQYDVQQDEwZu\nc3EuaW8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC5izs16tildar8\ng9oEauIehuEiz+LgAketrtCFnk38R8EJvQ3ore0jgZEcxniUHQfAg1abFo+zoPbI\nMLGkaCeu7qoK7T/3X4ur4YRmP4XPUTbJ8Kq15xPJK9Z/16f5J1MoVlMYYgC7fMVy\njO5fZ92GSMnqJtmGetcqU9Tgg3RJOSRF8Vf7Jv/o/zyYaHs9koFvrtDDVumnNcT9\nJaAdWoOq777CFPWq7DGdGGH9IeDC6A37qIq5MNwnWNDVBZ77GdrtN7qc0FS8vp1X\nj2OmgM6Mey/5RoC/M14cIerKlMLTDBgdfKBNMBT4TbRzY04ILs/Q8CJ3WWI1664R\na0zZtGsVAgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEASNjcoZbyNcwMQjMmNoil\nS/7pCRn4aYzZIVjrVtOHQ9GHC23MSep5um2gIcMFPiuYyu9Byl8CSVtc1op2fAKS\nvrugoZaCrp/A76hqOfNxgh7VmgTux8bG5Qcjaija1BNWpbyaWARdBxN/WgS5CpCj\nu2yzv8mrzzFNrDMlsmiEMvtkMzdhiZ4YY8zm6CdrbIR5z1eqf4e+rs4oJtTKNNAD\nhewk8CGiUW1hOx2jpjcIVMRy+ofVHRX2xQ6Sw8qxCNsiv8IPAAivgAbFJO76ZSbH\neQ7uKWszmBEroyFvZ0rfmFLXuopU125pyBDl5FUKYAZzCBx9tr5dROCbw/rXDhke\nig==\n-----END CERTIFICATE REQUEST-----\n" }, { "path": "nsqd/README.md", "content": "## nsqd\n\n`nsqd` is the daemon that receives, queues, and delivers messages to clients.\n\nRead the [docs](https://nsq.io/components/nsqd.html)\n" }, { "path": "nsqd/backend_queue.go", "content": "package nsqd\n\n// BackendQueue represents the behavior for the secondary message\n// storage system\ntype BackendQueue interface {\n\tPut([]byte) error\n\tReadChan() <-chan []byte // this is expected to be an *unbuffered* channel\n\tClose() error\n\tDelete() error\n\tDepth() int64\n\tEmpty() error\n}\n" }, { "path": "nsqd/buffer_pool.go", "content": "package nsqd\n\nimport (\n\t\"bytes\"\n\t\"sync\"\n)\n\nvar bp sync.Pool\n\nfunc init() {\n\tbp.New = func() interface{} {\n\t\treturn &bytes.Buffer{}\n\t}\n}\n\nfunc bufferPoolGet() *bytes.Buffer {\n\treturn bp.Get().(*bytes.Buffer)\n}\n\nfunc bufferPoolPut(b *bytes.Buffer) {\n\tb.Reset()\n\tbp.Put(b)\n}\n" }, { "path": "nsqd/channel.go", "content": "package nsqd\n\nimport (\n\t\"container/heap\"\n\t\"errors\"\n\t\"fmt\"\n\t\"math\"\n\t\"strings\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-diskqueue\"\n\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/pqueue\"\n\t\"github.com/nsqio/nsq/internal/quantile\"\n)\n\ntype Consumer interface {\n\tUnPause()\n\tPause()\n\tClose() error\n\tTimedOutMessage()\n\tStats(string) ClientStats\n\tEmpty()\n}\n\n// Channel represents the concrete type for a NSQ channel (and also\n// implements the Queue interface)\n//\n// There can be multiple channels per topic, each with there own unique set\n// of subscribers (clients).\n//\n// Channels maintain all client and message metadata, orchestrating in-flight\n// messages, timeouts, requeuing, etc.\ntype Channel struct {\n\t// 64bit atomic vars need to be first for proper alignment on 32bit platforms\n\trequeueCount uint64\n\tmessageCount uint64\n\tzoneLocalMsgCount uint64\n\tregionLocalMsgCount uint64\n\tglobalMsgCount uint64\n\ttimeoutCount uint64\n\n\tsync.RWMutex\n\n\ttopicName string\n\tname string\n\tnsqd *NSQD\n\n\tbackend BackendQueue\n\n\ttopologyAwareConsumption bool\n\tzoneLocalMsgChan chan *Message\n\tregionLocalMsgChan chan *Message\n\tmemoryMsgChan chan *Message\n\texitFlag int32\n\texitMutex sync.RWMutex\n\n\t// state tracking\n\tclients map[int64]Consumer\n\tpaused int32\n\tephemeral bool\n\tdeleteCallback func(*Channel)\n\tdeleter sync.Once\n\n\t// Stats tracking\n\te2eProcessingLatencyStream *quantile.Quantile\n\n\t// TODO: these can be DRYd up\n\tdeferredMessages map[MessageID]*pqueue.Item\n\tdeferredPQ pqueue.PriorityQueue\n\tdeferredMutex sync.Mutex\n\tinFlightMessages map[MessageID]*Message\n\tinFlightPQ inFlightPqueue\n\tinFlightMutex sync.Mutex\n}\n\n// NewChannel creates a new instance of the Channel type and returns a pointer\nfunc NewChannel(topicName string, channelName string, nsqd *NSQD,\n\tdeleteCallback func(*Channel)) *Channel {\n\n\tc := &Channel{\n\t\ttopicName: topicName,\n\t\tname: channelName,\n\t\tmemoryMsgChan: nil,\n\t\tclients: make(map[int64]Consumer),\n\t\tdeleteCallback: deleteCallback,\n\t\tnsqd: nsqd,\n\t\tephemeral: strings.HasSuffix(channelName, \"#ephemeral\"),\n\t\ttopologyAwareConsumption: nsqd.getOpts().HasExperiment(TopologyAwareConsumption),\n\t}\n\n\tif nsqd.getOpts().TopologyRegion != \"\" {\n\t\tc.regionLocalMsgChan = make(chan *Message)\n\t}\n\tif nsqd.getOpts().TopologyZone != \"\" {\n\t\tc.zoneLocalMsgChan = make(chan *Message)\n\t}\n\n\t// avoid mem-queue if size == 0 for more consistent ordering\n\tif nsqd.getOpts().MemQueueSize > 0 || c.ephemeral {\n\t\tc.memoryMsgChan = make(chan *Message, nsqd.getOpts().MemQueueSize)\n\t}\n\tif len(nsqd.getOpts().E2EProcessingLatencyPercentiles) > 0 {\n\t\tc.e2eProcessingLatencyStream = quantile.New(\n\t\t\tnsqd.getOpts().E2EProcessingLatencyWindowTime,\n\t\t\tnsqd.getOpts().E2EProcessingLatencyPercentiles,\n\t\t)\n\t}\n\n\tc.initPQ()\n\n\tif c.ephemeral {\n\t\tc.backend = newDummyBackendQueue()\n\t} else {\n\t\tdqLogf := func(level diskqueue.LogLevel, f string, args ...interface{}) {\n\t\t\topts := nsqd.getOpts()\n\t\t\tlg.Logf(opts.Logger, opts.LogLevel, lg.LogLevel(level), f, args...)\n\t\t}\n\t\t// backend names, for uniqueness, automatically include the topic...\n\t\tbackendName := getBackendName(topicName, channelName)\n\t\tc.backend = diskqueue.New(\n\t\t\tbackendName,\n\t\t\tnsqd.getOpts().DataPath,\n\t\t\tnsqd.getOpts().MaxBytesPerFile,\n\t\t\tint32(minValidMsgLength),\n\t\t\tint32(nsqd.getOpts().MaxMsgSize)+minValidMsgLength,\n\t\t\tnsqd.getOpts().SyncEvery,\n\t\t\tnsqd.getOpts().SyncTimeout,\n\t\t\tdqLogf,\n\t\t)\n\t}\n\n\tc.nsqd.Notify(c, !c.ephemeral)\n\n\treturn c\n}\n\nfunc (c *Channel) initPQ() {\n\tpqSize := int(math.Max(1, float64(c.nsqd.getOpts().MemQueueSize)/10))\n\n\tc.inFlightMutex.Lock()\n\tc.inFlightMessages = make(map[MessageID]*Message)\n\tc.inFlightPQ = newInFlightPqueue(pqSize)\n\tc.inFlightMutex.Unlock()\n\n\tc.deferredMutex.Lock()\n\tc.deferredMessages = make(map[MessageID]*pqueue.Item)\n\tc.deferredPQ = pqueue.New(pqSize)\n\tc.deferredMutex.Unlock()\n}\n\n// Exiting returns a boolean indicating if this channel is closed/exiting\nfunc (c *Channel) Exiting() bool {\n\treturn atomic.LoadInt32(&c.exitFlag) == 1\n}\n\n// Delete empties the channel and closes\nfunc (c *Channel) Delete() error {\n\treturn c.exit(true)\n}\n\n// Close cleanly closes the Channel\nfunc (c *Channel) Close() error {\n\treturn c.exit(false)\n}\n\nfunc (c *Channel) exit(deleted bool) error {\n\tc.exitMutex.Lock()\n\tdefer c.exitMutex.Unlock()\n\n\tif !atomic.CompareAndSwapInt32(&c.exitFlag, 0, 1) {\n\t\treturn errors.New(\"exiting\")\n\t}\n\n\tif deleted {\n\t\tc.nsqd.logf(LOG_INFO, \"CHANNEL(%s): deleting\", c.name)\n\n\t\t// since we are explicitly deleting a channel (not just at system exit time)\n\t\t// de-register this from the lookupd\n\t\tc.nsqd.Notify(c, !c.ephemeral)\n\t} else {\n\t\tc.nsqd.logf(LOG_INFO, \"CHANNEL(%s): closing\", c.name)\n\t}\n\n\t// this forceably closes client connections\n\tc.RLock()\n\tfor _, client := range c.clients {\n\t\tclient.Close()\n\t}\n\tc.RUnlock()\n\n\tif deleted {\n\t\t// empty the queue (deletes the backend files, too)\n\t\tc.Empty()\n\t\treturn c.backend.Delete()\n\t}\n\n\t// write anything leftover to disk\n\tc.flush()\n\treturn c.backend.Close()\n}\n\nfunc (c *Channel) Empty() error {\n\tc.Lock()\n\tdefer c.Unlock()\n\n\tc.initPQ()\n\tfor _, client := range c.clients {\n\t\tclient.Empty()\n\t}\n\n\tfor {\n\t\tselect {\n\t\tcase <-c.zoneLocalMsgChan:\n\t\tcase <-c.regionLocalMsgChan:\n\t\tcase <-c.memoryMsgChan:\n\t\tdefault:\n\t\t\tgoto finish\n\t\t}\n\t}\n\nfinish:\n\treturn c.backend.Empty()\n}\n\n// flush persists all the messages in internal memory buffers to the backend\n// it does not drain inflight/deferred because it is only called in Close()\nfunc (c *Channel) flush() error {\n\tif len(c.zoneLocalMsgChan) > 0 || len(c.regionLocalMsgChan) > 0 || len(c.memoryMsgChan) > 0 || len(c.inFlightMessages) > 0 || len(c.deferredMessages) > 0 {\n\t\tc.nsqd.logf(LOG_INFO, \"CHANNEL(%s): flushing %d memory %d in-flight %d deferred messages to backend\",\n\t\t\tc.name, len(c.memoryMsgChan)+len(c.zoneLocalMsgChan)+len(c.regionLocalMsgChan), len(c.inFlightMessages), len(c.deferredMessages))\n\t}\n\n\tfor {\n\t\tselect {\n\t\tcase msg := <-c.zoneLocalMsgChan:\n\t\t\terr := writeMessageToBackend(msg, c.backend)\n\t\t\tif err != nil {\n\t\t\t\tc.nsqd.logf(LOG_ERROR, \"failed to write message to backend - %s\", err)\n\t\t\t}\n\t\tcase msg := <-c.regionLocalMsgChan:\n\t\t\terr := writeMessageToBackend(msg, c.backend)\n\t\t\tif err != nil {\n\t\t\t\tc.nsqd.logf(LOG_ERROR, \"failed to write message to backend - %s\", err)\n\t\t\t}\n\t\tcase msg := <-c.memoryMsgChan:\n\t\t\terr := writeMessageToBackend(msg, c.backend)\n\t\t\tif err != nil {\n\t\t\t\tc.nsqd.logf(LOG_ERROR, \"failed to write message to backend - %s\", err)\n\t\t\t}\n\t\tdefault:\n\t\t\tgoto finish\n\t\t}\n\t}\n\nfinish:\n\tc.inFlightMutex.Lock()\n\tfor _, msg := range c.inFlightMessages {\n\t\terr := writeMessageToBackend(msg, c.backend)\n\t\tif err != nil {\n\t\t\tc.nsqd.logf(LOG_ERROR, \"failed to write message to backend - %s\", err)\n\t\t}\n\t}\n\tc.inFlightMutex.Unlock()\n\n\tc.deferredMutex.Lock()\n\tfor _, item := range c.deferredMessages {\n\t\tmsg := item.Value.(*Message)\n\t\terr := writeMessageToBackend(msg, c.backend)\n\t\tif err != nil {\n\t\t\tc.nsqd.logf(LOG_ERROR, \"failed to write message to backend - %s\", err)\n\t\t}\n\t}\n\tc.deferredMutex.Unlock()\n\n\treturn nil\n}\n\nfunc (c *Channel) Depth() int64 {\n\treturn int64(len(c.memoryMsgChan)) + int64(len(c.zoneLocalMsgChan)) + int64(len(c.regionLocalMsgChan)) + c.backend.Depth()\n}\n\nfunc (c *Channel) Pause() error {\n\treturn c.doPause(true)\n}\n\nfunc (c *Channel) UnPause() error {\n\treturn c.doPause(false)\n}\n\nfunc (c *Channel) doPause(pause bool) error {\n\tif pause {\n\t\tatomic.StoreInt32(&c.paused, 1)\n\t} else {\n\t\tatomic.StoreInt32(&c.paused, 0)\n\t}\n\n\tc.RLock()\n\tfor _, client := range c.clients {\n\t\tif pause {\n\t\t\tclient.Pause()\n\t\t} else {\n\t\t\tclient.UnPause()\n\t\t}\n\t}\n\tc.RUnlock()\n\treturn nil\n}\n\nfunc (c *Channel) IsPaused() bool {\n\treturn atomic.LoadInt32(&c.paused) == 1\n}\n\n// PutMessage writes a Message to the queue\nfunc (c *Channel) PutMessage(m *Message) error {\n\tc.exitMutex.RLock()\n\tdefer c.exitMutex.RUnlock()\n\tif c.Exiting() {\n\t\treturn errors.New(\"exiting\")\n\t}\n\terr := c.put(m)\n\tif err != nil {\n\t\treturn err\n\t}\n\tatomic.AddUint64(&c.messageCount, 1)\n\treturn nil\n}\n\nfunc (c *Channel) put(m *Message) error {\n\tif c.topologyAwareConsumption {\n\t\t// Attempt zone local, region local and finally the memory channel\n\t\t// we do this to ensure that we preferentially deliver messages based on toplogy\n\t\t//\n\t\t// Because messagePump is intermittently unavailable while writing a msg to a client\n\t\t// we continue to add lower priority channels in the select loop, this means at each\n\t\t// attempt a higher priority channel can still win\n\t\tselect {\n\t\tcase c.zoneLocalMsgChan <- m:\n\t\t\treturn nil\n\t\tdefault:\n\t\t}\n\t\tselect {\n\t\tcase c.zoneLocalMsgChan <- m:\n\t\t\treturn nil\n\t\tcase c.regionLocalMsgChan <- m:\n\t\t\treturn nil\n\t\tdefault:\n\t\t}\n\n\t\tselect {\n\t\tcase c.zoneLocalMsgChan <- m:\n\t\t\treturn nil\n\t\tcase c.regionLocalMsgChan <- m:\n\t\t\treturn nil\n\t\tcase c.memoryMsgChan <- m:\n\t\t\treturn nil\n\t\tdefault:\n\t\t}\n\n\t} else {\n\n\t\tselect {\n\t\tcase c.memoryMsgChan <- m:\n\t\t\treturn nil\n\t\tdefault:\n\t\t}\n\t}\n\n\terr := writeMessageToBackend(m, c.backend)\n\tc.nsqd.SetHealth(err)\n\tif err != nil {\n\t\tc.nsqd.logf(LOG_ERROR, \"CHANNEL(%s): failed to write message to backend - %s\",\n\t\t\tc.name, err)\n\t\treturn err\n\t}\n\treturn nil\n}\n\nfunc (c *Channel) PutMessageDeferred(msg *Message, timeout time.Duration) {\n\tatomic.AddUint64(&c.messageCount, 1)\n\tc.StartDeferredTimeout(msg, timeout)\n}\n\n// TouchMessage resets the timeout for an in-flight message\nfunc (c *Channel) TouchMessage(clientID int64, id MessageID, clientMsgTimeout time.Duration) error {\n\tmsg, err := c.popInFlightMessage(clientID, id)\n\tif err != nil {\n\t\treturn err\n\t}\n\tc.removeFromInFlightPQ(msg)\n\n\tnewTimeout := time.Now().Add(clientMsgTimeout)\n\tif newTimeout.Sub(msg.deliveryTS) >=\n\t\tc.nsqd.getOpts().MaxMsgTimeout {\n\t\t// we would have gone over, set to the max\n\t\tnewTimeout = msg.deliveryTS.Add(c.nsqd.getOpts().MaxMsgTimeout)\n\t}\n\n\tmsg.pri = newTimeout.UnixNano()\n\terr = c.pushInFlightMessage(msg)\n\tif err != nil {\n\t\treturn err\n\t}\n\tc.addToInFlightPQ(msg)\n\treturn nil\n}\n\n// FinishMessage successfully discards an in-flight message\nfunc (c *Channel) FinishMessage(clientID int64, id MessageID) error {\n\tmsg, err := c.popInFlightMessage(clientID, id)\n\tif err != nil {\n\t\treturn err\n\t}\n\tc.removeFromInFlightPQ(msg)\n\tif c.e2eProcessingLatencyStream != nil {\n\t\tc.e2eProcessingLatencyStream.Insert(msg.Timestamp)\n\t}\n\treturn nil\n}\n\n// RequeueMessage requeues a message based on `time.Duration`, ie:\n//\n// `timeoutMs` == 0 - requeue a message immediately\n// `timeoutMs` > 0 - asynchronously wait for the specified timeout\n//\n//\tand requeue a message (aka \"deferred requeue\")\nfunc (c *Channel) RequeueMessage(clientID int64, id MessageID, timeout time.Duration) error {\n\t// remove from inflight first\n\tmsg, err := c.popInFlightMessage(clientID, id)\n\tif err != nil {\n\t\treturn err\n\t}\n\tc.removeFromInFlightPQ(msg)\n\tatomic.AddUint64(&c.requeueCount, 1)\n\n\tif timeout == 0 {\n\t\tc.exitMutex.RLock()\n\t\tif c.Exiting() {\n\t\t\tc.exitMutex.RUnlock()\n\t\t\treturn errors.New(\"exiting\")\n\t\t}\n\t\terr := c.put(msg)\n\t\tc.exitMutex.RUnlock()\n\t\treturn err\n\t}\n\n\t// deferred requeue\n\treturn c.StartDeferredTimeout(msg, timeout)\n}\n\n// AddClient adds a client to the Channel's client list\nfunc (c *Channel) AddClient(clientID int64, client Consumer) error {\n\tc.exitMutex.RLock()\n\tdefer c.exitMutex.RUnlock()\n\n\tif c.Exiting() {\n\t\treturn errors.New(\"exiting\")\n\t}\n\n\tc.RLock()\n\t_, ok := c.clients[clientID]\n\tnumClients := len(c.clients)\n\tc.RUnlock()\n\tif ok {\n\t\treturn nil\n\t}\n\n\tmaxChannelConsumers := c.nsqd.getOpts().MaxChannelConsumers\n\tif maxChannelConsumers != 0 && numClients >= maxChannelConsumers {\n\t\treturn fmt.Errorf(\"consumers for %s:%s exceeds limit of %d\",\n\t\t\tc.topicName, c.name, maxChannelConsumers)\n\t}\n\n\tc.Lock()\n\tc.clients[clientID] = client\n\tc.Unlock()\n\treturn nil\n}\n\n// RemoveClient removes a client from the Channel's client list\nfunc (c *Channel) RemoveClient(clientID int64) {\n\tc.exitMutex.RLock()\n\tdefer c.exitMutex.RUnlock()\n\n\tif c.Exiting() {\n\t\treturn\n\t}\n\n\tc.RLock()\n\t_, ok := c.clients[clientID]\n\tc.RUnlock()\n\tif !ok {\n\t\treturn\n\t}\n\n\tc.Lock()\n\tdelete(c.clients, clientID)\n\tnumClients := len(c.clients)\n\tc.Unlock()\n\n\tif numClients == 0 && c.ephemeral {\n\t\tgo c.deleter.Do(func() { c.deleteCallback(c) })\n\t}\n}\n\nfunc (c *Channel) StartInFlightTimeout(msg *Message, clientID int64, timeout time.Duration) error {\n\tnow := time.Now()\n\tmsg.clientID = clientID\n\tmsg.deliveryTS = now\n\tmsg.pri = now.Add(timeout).UnixNano()\n\terr := c.pushInFlightMessage(msg)\n\tif err != nil {\n\t\treturn err\n\t}\n\tc.addToInFlightPQ(msg)\n\treturn nil\n}\n\nfunc (c *Channel) StartDeferredTimeout(msg *Message, timeout time.Duration) error {\n\tabsTs := time.Now().Add(timeout).UnixNano()\n\titem := &pqueue.Item{Value: msg, Priority: absTs}\n\terr := c.pushDeferredMessage(item)\n\tif err != nil {\n\t\treturn err\n\t}\n\tc.addToDeferredPQ(item)\n\treturn nil\n}\n\n// pushInFlightMessage atomically adds a message to the in-flight dictionary\nfunc (c *Channel) pushInFlightMessage(msg *Message) error {\n\tc.inFlightMutex.Lock()\n\t_, ok := c.inFlightMessages[msg.ID]\n\tif ok {\n\t\tc.inFlightMutex.Unlock()\n\t\treturn errors.New(\"ID already in flight\")\n\t}\n\tc.inFlightMessages[msg.ID] = msg\n\tc.inFlightMutex.Unlock()\n\treturn nil\n}\n\n// popInFlightMessage atomically removes a message from the in-flight dictionary\nfunc (c *Channel) popInFlightMessage(clientID int64, id MessageID) (*Message, error) {\n\tc.inFlightMutex.Lock()\n\tmsg, ok := c.inFlightMessages[id]\n\tif !ok {\n\t\tc.inFlightMutex.Unlock()\n\t\treturn nil, errors.New(\"ID not in flight\")\n\t}\n\tif msg.clientID != clientID {\n\t\tc.inFlightMutex.Unlock()\n\t\treturn nil, errors.New(\"client does not own message\")\n\t}\n\tdelete(c.inFlightMessages, id)\n\tc.inFlightMutex.Unlock()\n\treturn msg, nil\n}\n\nfunc (c *Channel) addToInFlightPQ(msg *Message) {\n\tc.inFlightMutex.Lock()\n\tc.inFlightPQ.Push(msg)\n\tc.inFlightMutex.Unlock()\n}\n\nfunc (c *Channel) removeFromInFlightPQ(msg *Message) {\n\tc.inFlightMutex.Lock()\n\tif msg.index == -1 {\n\t\t// this item has already been popped off the pqueue\n\t\tc.inFlightMutex.Unlock()\n\t\treturn\n\t}\n\tc.inFlightPQ.Remove(msg.index)\n\tc.inFlightMutex.Unlock()\n}\n\nfunc (c *Channel) pushDeferredMessage(item *pqueue.Item) error {\n\tc.deferredMutex.Lock()\n\t// TODO: these map lookups are costly\n\tid := item.Value.(*Message).ID\n\t_, ok := c.deferredMessages[id]\n\tif ok {\n\t\tc.deferredMutex.Unlock()\n\t\treturn errors.New(\"ID already deferred\")\n\t}\n\tc.deferredMessages[id] = item\n\tc.deferredMutex.Unlock()\n\treturn nil\n}\n\nfunc (c *Channel) popDeferredMessage(id MessageID) (*pqueue.Item, error) {\n\tc.deferredMutex.Lock()\n\t// TODO: these map lookups are costly\n\titem, ok := c.deferredMessages[id]\n\tif !ok {\n\t\tc.deferredMutex.Unlock()\n\t\treturn nil, errors.New(\"ID not deferred\")\n\t}\n\tdelete(c.deferredMessages, id)\n\tc.deferredMutex.Unlock()\n\treturn item, nil\n}\n\nfunc (c *Channel) addToDeferredPQ(item *pqueue.Item) {\n\tc.deferredMutex.Lock()\n\theap.Push(&c.deferredPQ, item)\n\tc.deferredMutex.Unlock()\n}\n\nfunc (c *Channel) processDeferredQueue(t int64) bool {\n\tc.exitMutex.RLock()\n\tdefer c.exitMutex.RUnlock()\n\n\tif c.Exiting() {\n\t\treturn false\n\t}\n\n\tdirty := false\n\tfor {\n\t\tc.deferredMutex.Lock()\n\t\titem, _ := c.deferredPQ.PeekAndShift(t)\n\t\tc.deferredMutex.Unlock()\n\n\t\tif item == nil {\n\t\t\tgoto exit\n\t\t}\n\t\tdirty = true\n\n\t\tmsg := item.Value.(*Message)\n\t\t_, err := c.popDeferredMessage(msg.ID)\n\t\tif err != nil {\n\t\t\tgoto exit\n\t\t}\n\t\tc.put(msg)\n\t}\n\nexit:\n\treturn dirty\n}\n\nfunc (c *Channel) processInFlightQueue(t int64) bool {\n\tc.exitMutex.RLock()\n\tdefer c.exitMutex.RUnlock()\n\n\tif c.Exiting() {\n\t\treturn false\n\t}\n\n\tdirty := false\n\tfor {\n\t\tc.inFlightMutex.Lock()\n\t\tmsg, _ := c.inFlightPQ.PeekAndShift(t)\n\t\tc.inFlightMutex.Unlock()\n\n\t\tif msg == nil {\n\t\t\tgoto exit\n\t\t}\n\t\tdirty = true\n\n\t\t_, err := c.popInFlightMessage(msg.clientID, msg.ID)\n\t\tif err != nil {\n\t\t\tgoto exit\n\t\t}\n\t\tatomic.AddUint64(&c.timeoutCount, 1)\n\t\tc.RLock()\n\t\tclient, ok := c.clients[msg.clientID]\n\t\tc.RUnlock()\n\t\tif ok {\n\t\t\tclient.TimedOutMessage()\n\t\t}\n\t\tc.put(msg)\n\t}\n\nexit:\n\treturn dirty\n}\n" }, { "path": "nsqd/channel_test.go", "content": "package nsqd\n\nimport (\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"os\"\n\t\"strconv\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\n// ensure that we can push a message through a topic and get it out of a channel\nfunc TestPutMessage(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_put_message\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel1 := topic.GetChannel(\"ch\")\n\n\tvar id MessageID\n\tmsg := NewMessage(id, []byte(\"test\"))\n\ttopic.PutMessage(msg)\n\n\toutputMsg := <-channel1.memoryMsgChan\n\ttest.Equal(t, msg.ID, outputMsg.ID)\n\ttest.Equal(t, msg.Body, outputMsg.Body)\n}\n\n// ensure that both channels get the same message\nfunc TestPutMessage2Chan(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_put_message_2chan\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel1 := topic.GetChannel(\"ch1\")\n\tchannel2 := topic.GetChannel(\"ch2\")\n\n\tvar id MessageID\n\tmsg := NewMessage(id, []byte(\"test\"))\n\ttopic.PutMessage(msg)\n\n\toutputMsg1 := <-channel1.memoryMsgChan\n\ttest.Equal(t, msg.ID, outputMsg1.ID)\n\ttest.Equal(t, msg.Body, outputMsg1.Body)\n\n\toutputMsg2 := <-channel2.memoryMsgChan\n\ttest.Equal(t, msg.ID, outputMsg2.ID)\n\ttest.Equal(t, msg.Body, outputMsg2.Body)\n}\n\nfunc TestInFlightWorker(t *testing.T) {\n\tcount := 250\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MsgTimeout = 100 * time.Millisecond\n\topts.QueueScanRefreshInterval = 100 * time.Millisecond\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_in_flight_worker\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"channel\")\n\n\tfor i := 0; i < count; i++ {\n\t\tmsg := NewMessage(topic.GenerateID(), []byte(\"test\"))\n\t\tchannel.StartInFlightTimeout(msg, 0, opts.MsgTimeout)\n\t}\n\n\tchannel.Lock()\n\tinFlightMsgs := len(channel.inFlightMessages)\n\tchannel.Unlock()\n\ttest.Equal(t, count, inFlightMsgs)\n\n\tchannel.inFlightMutex.Lock()\n\tinFlightPQMsgs := len(channel.inFlightPQ)\n\tchannel.inFlightMutex.Unlock()\n\ttest.Equal(t, count, inFlightPQMsgs)\n\n\t// the in flight worker has a resolution of 100ms so we need to wait\n\t// at least that much longer than our msgTimeout (in worst case)\n\ttime.Sleep(4 * opts.MsgTimeout)\n\n\tchannel.Lock()\n\tinFlightMsgs = len(channel.inFlightMessages)\n\tchannel.Unlock()\n\ttest.Equal(t, 0, inFlightMsgs)\n\n\tchannel.inFlightMutex.Lock()\n\tinFlightPQMsgs = len(channel.inFlightPQ)\n\tchannel.inFlightMutex.Unlock()\n\ttest.Equal(t, 0, inFlightPQMsgs)\n}\n\nfunc TestChannelEmpty(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_channel_empty\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"channel\")\n\n\tmsgs := make([]*Message, 0, 25)\n\tfor i := 0; i < 25; i++ {\n\t\tmsg := NewMessage(topic.GenerateID(), []byte(\"test\"))\n\t\tchannel.StartInFlightTimeout(msg, 0, opts.MsgTimeout)\n\t\tmsgs = append(msgs, msg)\n\t}\n\n\tchannel.RequeueMessage(0, msgs[len(msgs)-1].ID, 100*time.Millisecond)\n\ttest.Equal(t, 24, len(channel.inFlightMessages))\n\ttest.Equal(t, 24, len(channel.inFlightPQ))\n\ttest.Equal(t, 1, len(channel.deferredMessages))\n\ttest.Equal(t, 1, len(channel.deferredPQ))\n\n\tchannel.Empty()\n\n\ttest.Equal(t, 0, len(channel.inFlightMessages))\n\ttest.Equal(t, 0, len(channel.inFlightPQ))\n\ttest.Equal(t, 0, len(channel.deferredMessages))\n\ttest.Equal(t, 0, len(channel.deferredPQ))\n\ttest.Equal(t, int64(0), channel.Depth())\n}\n\nfunc TestChannelEmptyConsumer(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, _ := mustConnectNSQD(tcpAddr)\n\tdefer conn.Close()\n\n\ttopicName := \"test_channel_empty\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"channel\")\n\tclient := newClientV2(0, conn, nsqd)\n\tclient.SetReadyCount(25)\n\terr := channel.AddClient(client.ID, client)\n\ttest.Equal(t, err, nil)\n\n\tfor i := 0; i < 25; i++ {\n\t\tmsg := NewMessage(topic.GenerateID(), []byte(\"test\"))\n\t\tchannel.StartInFlightTimeout(msg, 0, opts.MsgTimeout)\n\t\tclient.SendingMessage()\n\t}\n\n\tfor _, cl := range channel.clients {\n\t\tstats := cl.Stats(\"\").(ClientV2Stats)\n\t\ttest.Equal(t, int64(25), stats.InFlightCount)\n\t}\n\n\tchannel.Empty()\n\n\tfor _, cl := range channel.clients {\n\t\tstats := cl.Stats(\"\").(ClientV2Stats)\n\t\ttest.Equal(t, int64(0), stats.InFlightCount)\n\t}\n}\n\nfunc TestMaxChannelConsumers(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MaxChannelConsumers = 1\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, _ := mustConnectNSQD(tcpAddr)\n\tdefer conn.Close()\n\n\ttopicName := \"test_max_channel_consumers\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"channel\")\n\n\tclient1 := newClientV2(1, conn, nsqd)\n\tclient1.SetReadyCount(25)\n\terr := channel.AddClient(client1.ID, client1)\n\ttest.Equal(t, err, nil)\n\n\tclient2 := newClientV2(2, conn, nsqd)\n\tclient2.SetReadyCount(25)\n\terr = channel.AddClient(client2.ID, client2)\n\ttest.NotEqual(t, err, nil)\n}\n\nfunc TestChannelHealth(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MemQueueSize = 2\n\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopic := nsqd.GetTopic(\"test\")\n\n\tchannel := topic.GetChannel(\"channel\")\n\n\tchannel.backend = &errorBackendQueue{}\n\n\tmsg := NewMessage(topic.GenerateID(), make([]byte, 100))\n\terr := channel.PutMessage(msg)\n\ttest.Nil(t, err)\n\n\tmsg = NewMessage(topic.GenerateID(), make([]byte, 100))\n\terr = channel.PutMessage(msg)\n\ttest.Nil(t, err)\n\n\tmsg = NewMessage(topic.GenerateID(), make([]byte, 100))\n\terr = channel.PutMessage(msg)\n\ttest.NotNil(t, err)\n\n\turl := fmt.Sprintf(\"http://%s/ping\", httpAddr)\n\tresp, err := http.Get(url)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 500, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"NOK - never gonna happen\", string(body))\n\n\tchannel.backend = &errorRecoveredBackendQueue{}\n\n\tmsg = NewMessage(topic.GenerateID(), make([]byte, 100))\n\terr = channel.PutMessage(msg)\n\ttest.Nil(t, err)\n\n\tresp, err = http.Get(url)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"OK\", string(body))\n}\n" }, { "path": "nsqd/client_v2.go", "content": "package nsqd\n\nimport (\n\t\"bufio\"\n\t\"compress/flate\"\n\t\"crypto/tls\"\n\t\"fmt\"\n\t\"net\"\n\t\"strings\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\t\"github.com/golang/snappy\"\n\t\"github.com/nsqio/nsq/internal/auth\"\n)\n\nconst defaultBufferSize = 16 * 1024\n\nconst (\n\tstateInit = iota\n\tstateDisconnected\n\tstateConnected\n\tstateSubscribed\n\tstateClosing\n)\n\ntype identifyDataV2 struct {\n\tClientID string `json:\"client_id\"`\n\tHostname string `json:\"hostname\"`\n\tHeartbeatInterval int `json:\"heartbeat_interval\"`\n\tOutputBufferSize int `json:\"output_buffer_size\"`\n\tOutputBufferTimeout int `json:\"output_buffer_timeout\"`\n\tFeatureNegotiation bool `json:\"feature_negotiation\"`\n\tTLSv1 bool `json:\"tls_v1\"`\n\tDeflate bool `json:\"deflate\"`\n\tDeflateLevel int `json:\"deflate_level\"`\n\tSnappy bool `json:\"snappy\"`\n\tSampleRate int32 `json:\"sample_rate\"`\n\tUserAgent string `json:\"user_agent\"`\n\tMsgTimeout int `json:\"msg_timeout\"`\n\tTopologyRegion string `json:\"topology_region\"`\n\tTopologyZone string `json:\"topology_zone\"`\n}\n\ntype identifyEvent struct {\n\tOutputBufferTimeout time.Duration\n\tHeartbeatInterval time.Duration\n\tSampleRate int32\n\tMsgTimeout time.Duration\n\tTopologyRegion string\n\tTopologyZone string\n}\n\ntype PubCount struct {\n\tTopic string `json:\"topic\"`\n\tCount uint64 `json:\"count\"`\n}\n\ntype ClientV2Stats struct {\n\tClientID string `json:\"client_id\"`\n\tHostname string `json:\"hostname\"`\n\tVersion string `json:\"version\"`\n\tRemoteAddress string `json:\"remote_address\"`\n\tState int32 `json:\"state\"`\n\tReadyCount int64 `json:\"ready_count\"`\n\tInFlightCount int64 `json:\"in_flight_count\"`\n\tMessageCount uint64 `json:\"message_count\"`\n\tZoneLocalMsgCount uint64 `json:\"zone_local_msg_count,omitempty\"`\n\tRegionLocalMsgCount uint64 `json:\"region_local_msg_count,omitempty\"`\n\tGlobalMsgCount uint64 `json:\"global_msg_count,omitempty\"`\n\tFinishCount uint64 `json:\"finish_count\"`\n\tRequeueCount uint64 `json:\"requeue_count\"`\n\tConnectTime int64 `json:\"connect_ts\"`\n\tSampleRate int32 `json:\"sample_rate\"`\n\tDeflate bool `json:\"deflate\"`\n\tSnappy bool `json:\"snappy\"`\n\tUserAgent string `json:\"user_agent\"`\n\tAuthed bool `json:\"authed,omitempty\"`\n\tAuthIdentity string `json:\"auth_identity,omitempty\"`\n\tAuthIdentityURL string `json:\"auth_identity_url,omitempty\"`\n\tTopologyZone string `json:\"topology_zone\"`\n\tTopologyRegion string `json:\"topology_region\"`\n\n\tPubCounts []PubCount `json:\"pub_counts,omitempty\"`\n\n\tTLS bool `json:\"tls\"`\n\tCipherSuite string `json:\"tls_cipher_suite\"`\n\tTLSVersion string `json:\"tls_version\"`\n\tTLSNegotiatedProtocol string `json:\"tls_negotiated_protocol\"`\n\tTLSNegotiatedProtocolIsMutual bool `json:\"tls_negotiated_protocol_is_mutual\"`\n}\n\nfunc (s ClientV2Stats) String() string {\n\tconnectTime := time.Unix(s.ConnectTime, 0)\n\tduration := time.Since(connectTime).Truncate(time.Second)\n\n\t_, port, _ := net.SplitHostPort(s.RemoteAddress)\n\tid := fmt.Sprintf(\"%s:%s %s\", s.Hostname, port, s.UserAgent)\n\n\t// producer\n\tif len(s.PubCounts) > 0 {\n\t\tvar total uint64\n\t\tvar topicOut []string\n\t\tfor _, v := range s.PubCounts {\n\t\t\ttotal += v.Count\n\t\t\ttopicOut = append(topicOut, fmt.Sprintf(\"%s=%d\", v.Topic, v.Count))\n\t\t}\n\t\treturn fmt.Sprintf(\"[%s %-21s] msgs: %-8d topics: %s connected: %s\",\n\t\t\ts.Version,\n\t\t\tid,\n\t\t\ttotal,\n\t\t\tstrings.Join(topicOut, \",\"),\n\t\t\tduration,\n\t\t)\n\t}\n\n\t// consumer\n\treturn fmt.Sprintf(\"[%s %-21s] state: %d inflt: %-4d rdy: %-4d fin: %-8d re-q: %-8d msgs: %-8d connected: %s\",\n\t\ts.Version,\n\t\tid,\n\t\ts.State,\n\t\ts.InFlightCount,\n\t\ts.ReadyCount,\n\t\ts.FinishCount,\n\t\ts.RequeueCount,\n\t\ts.MessageCount,\n\t\tduration,\n\t)\n}\n\ntype clientV2 struct {\n\t// 64bit atomic vars need to be first for proper alignment on 32bit platforms\n\tReadyCount int64\n\tInFlightCount int64\n\tMessageCount uint64\n\tZoneLocalMsgCount uint64\n\tRegionLocalMsgCount uint64\n\tGlobalMsgCount uint64\n\tFinishCount uint64\n\tRequeueCount uint64\n\n\tpubCounts map[string]uint64\n\n\twriteLock sync.RWMutex\n\tmetaLock sync.RWMutex\n\n\tID int64\n\tnsqd *NSQD\n\tUserAgent string\n\n\t// original connection\n\tnet.Conn\n\n\t// connections based on negotiated features\n\ttlsConn *tls.Conn\n\tflateWriter *flate.Writer\n\n\t// reading/writing interfaces\n\tReader *bufio.Reader\n\tWriter *bufio.Writer\n\n\tOutputBufferSize int\n\tOutputBufferTimeout time.Duration\n\n\tHeartbeatInterval time.Duration\n\n\tMsgTimeout time.Duration\n\n\tState int32\n\tConnectTime time.Time\n\tChannel *Channel\n\tReadyStateChan chan int\n\tExitChan chan int\n\n\tClientID string\n\tHostname string\n\tTopologyRegion string\n\tTopologyZone string\n\n\tSampleRate int32\n\n\tIdentifyEventChan chan identifyEvent\n\tSubEventChan chan *Channel\n\n\tTLS int32\n\tSnappy int32\n\tDeflate int32\n\n\t// re-usable buffer for reading the 4-byte lengths off the wire\n\tlenBuf [4]byte\n\tlenSlice []byte\n\n\tAuthSecret string\n\tAuthState *auth.State\n}\n\nfunc newClientV2(id int64, conn net.Conn, nsqd *NSQD) *clientV2 {\n\tvar identifier string\n\tif conn != nil {\n\t\tidentifier, _, _ = net.SplitHostPort(conn.RemoteAddr().String())\n\t}\n\n\tc := &clientV2{\n\t\tID: id,\n\t\tnsqd: nsqd,\n\n\t\tConn: conn,\n\n\t\tReader: bufio.NewReaderSize(conn, defaultBufferSize),\n\t\tWriter: bufio.NewWriterSize(conn, defaultBufferSize),\n\n\t\tOutputBufferSize: defaultBufferSize,\n\t\tOutputBufferTimeout: nsqd.getOpts().OutputBufferTimeout,\n\n\t\tMsgTimeout: nsqd.getOpts().MsgTimeout,\n\n\t\t// ReadyStateChan has a buffer of 1 to guarantee that in the event\n\t\t// there is a race the state update is not lost\n\t\tReadyStateChan: make(chan int, 1),\n\t\tExitChan: make(chan int),\n\t\tConnectTime: time.Now(),\n\t\tState: stateInit,\n\n\t\tClientID: identifier,\n\t\tHostname: identifier,\n\n\t\tSubEventChan: make(chan *Channel, 1),\n\t\tIdentifyEventChan: make(chan identifyEvent, 1),\n\n\t\t// heartbeats are client configurable but default to 30s\n\t\tHeartbeatInterval: nsqd.getOpts().ClientTimeout / 2,\n\n\t\tpubCounts: make(map[string]uint64),\n\t}\n\tc.lenSlice = c.lenBuf[:]\n\treturn c\n}\n\nfunc (c *clientV2) String() string {\n\treturn c.RemoteAddr().String()\n}\n\nfunc (c *clientV2) Type() int {\n\tc.metaLock.RLock()\n\thasPublished := len(c.pubCounts) > 0\n\tc.metaLock.RUnlock()\n\tif hasPublished {\n\t\treturn typeProducer\n\t}\n\treturn typeConsumer\n}\n\nfunc (c *clientV2) Identify(data identifyDataV2) error {\n\tc.nsqd.logf(LOG_INFO, \"[%s] IDENTIFY: %+v\", c, data)\n\n\tc.metaLock.Lock()\n\tc.ClientID = data.ClientID\n\tc.Hostname = data.Hostname\n\tc.UserAgent = data.UserAgent\n\tc.TopologyRegion = data.TopologyRegion\n\tc.TopologyZone = data.TopologyZone\n\tc.metaLock.Unlock()\n\n\terr := c.SetHeartbeatInterval(data.HeartbeatInterval)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = c.SetOutputBuffer(data.OutputBufferSize, data.OutputBufferTimeout)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = c.SetSampleRate(data.SampleRate)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = c.SetMsgTimeout(data.MsgTimeout)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tie := identifyEvent{\n\t\tOutputBufferTimeout: c.OutputBufferTimeout,\n\t\tHeartbeatInterval: c.HeartbeatInterval,\n\t\tSampleRate: c.SampleRate,\n\t\tMsgTimeout: c.MsgTimeout,\n\t\tTopologyRegion: c.TopologyRegion,\n\t\tTopologyZone: c.TopologyZone,\n\t}\n\n\t// update the client's message pump\n\tselect {\n\tcase c.IdentifyEventChan <- ie:\n\tdefault:\n\t}\n\n\treturn nil\n}\n\nfunc (c *clientV2) Stats(topicName string) ClientStats {\n\tc.metaLock.RLock()\n\tclientID := c.ClientID\n\thostname := c.Hostname\n\tuserAgent := c.UserAgent\n\ttopologyZone := c.TopologyZone\n\ttopologyRegion := c.TopologyRegion\n\tvar identity string\n\tvar identityURL string\n\tif c.AuthState != nil {\n\t\tidentity = c.AuthState.Identity\n\t\tidentityURL = c.AuthState.IdentityURL\n\t}\n\tpubCounts := make([]PubCount, 0, len(c.pubCounts))\n\tfor topic, count := range c.pubCounts {\n\t\tif len(topicName) > 0 && topic != topicName {\n\t\t\tcontinue\n\t\t}\n\t\tpubCounts = append(pubCounts, PubCount{\n\t\t\tTopic: topic,\n\t\t\tCount: count,\n\t\t})\n\t\tbreak\n\t}\n\tc.metaLock.RUnlock()\n\tstats := ClientV2Stats{\n\t\tVersion: \"V2\",\n\t\tRemoteAddress: c.RemoteAddr().String(),\n\t\tClientID: clientID,\n\t\tHostname: hostname,\n\t\tUserAgent: userAgent,\n\t\tState: atomic.LoadInt32(&c.State),\n\t\tReadyCount: atomic.LoadInt64(&c.ReadyCount),\n\t\tInFlightCount: atomic.LoadInt64(&c.InFlightCount),\n\t\tMessageCount: atomic.LoadUint64(&c.MessageCount),\n\t\tZoneLocalMsgCount: atomic.LoadUint64(&c.ZoneLocalMsgCount),\n\t\tRegionLocalMsgCount: atomic.LoadUint64(&c.RegionLocalMsgCount),\n\t\tGlobalMsgCount: atomic.LoadUint64(&c.GlobalMsgCount),\n\t\tFinishCount: atomic.LoadUint64(&c.FinishCount),\n\t\tRequeueCount: atomic.LoadUint64(&c.RequeueCount),\n\t\tConnectTime: c.ConnectTime.Unix(),\n\t\tSampleRate: atomic.LoadInt32(&c.SampleRate),\n\t\tTLS: atomic.LoadInt32(&c.TLS) == 1,\n\t\tDeflate: atomic.LoadInt32(&c.Deflate) == 1,\n\t\tSnappy: atomic.LoadInt32(&c.Snappy) == 1,\n\t\tAuthed: c.HasAuthorizations(),\n\t\tAuthIdentity: identity,\n\t\tAuthIdentityURL: identityURL,\n\t\tPubCounts: pubCounts,\n\t\tTopologyZone: topologyZone,\n\t\tTopologyRegion: topologyRegion,\n\t}\n\tif stats.TLS {\n\t\tp := prettyConnectionState{c.tlsConn.ConnectionState()}\n\t\tstats.CipherSuite = p.GetCipherSuite()\n\t\tstats.TLSVersion = p.GetVersion()\n\t\tstats.TLSNegotiatedProtocol = p.NegotiatedProtocol\n\t\tstats.TLSNegotiatedProtocolIsMutual = p.NegotiatedProtocolIsMutual\n\t}\n\treturn stats\n}\n\n// struct to convert from integers to the human readable strings\ntype prettyConnectionState struct {\n\ttls.ConnectionState\n}\n\nfunc (p *prettyConnectionState) GetCipherSuite() string {\n\tswitch p.CipherSuite {\n\tcase tls.TLS_RSA_WITH_RC4_128_SHA:\n\t\treturn \"TLS_RSA_WITH_RC4_128_SHA\"\n\tcase tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA:\n\t\treturn \"TLS_RSA_WITH_3DES_EDE_CBC_SHA\"\n\tcase tls.TLS_RSA_WITH_AES_128_CBC_SHA:\n\t\treturn \"TLS_RSA_WITH_AES_128_CBC_SHA\"\n\tcase tls.TLS_RSA_WITH_AES_256_CBC_SHA:\n\t\treturn \"TLS_RSA_WITH_AES_256_CBC_SHA\"\n\tcase tls.TLS_ECDHE_ECDSA_WITH_RC4_128_SHA:\n\t\treturn \"TLS_ECDHE_ECDSA_WITH_RC4_128_SHA\"\n\tcase tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA:\n\t\treturn \"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA\"\n\tcase tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA:\n\t\treturn \"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA\"\n\tcase tls.TLS_ECDHE_RSA_WITH_RC4_128_SHA:\n\t\treturn \"TLS_ECDHE_RSA_WITH_RC4_128_SHA\"\n\tcase tls.TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA:\n\t\treturn \"TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA\"\n\tcase tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA:\n\t\treturn \"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA\"\n\tcase tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA:\n\t\treturn \"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA\"\n\tcase tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:\n\t\treturn \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\"\n\tcase tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256:\n\t\treturn \"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\"\n\t}\n\treturn fmt.Sprintf(\"Unknown %d\", p.CipherSuite)\n}\n\nfunc (p *prettyConnectionState) GetVersion() string {\n\tswitch p.Version {\n\tcase tls.VersionTLS10:\n\t\treturn \"TLS1.0\"\n\tcase tls.VersionTLS11:\n\t\treturn \"TLS1.1\"\n\tcase tls.VersionTLS12:\n\t\treturn \"TLS1.2\"\n\tcase tls.VersionTLS13:\n\t\treturn \"TLS1.3\"\n\tdefault:\n\t\treturn fmt.Sprintf(\"Unknown %d\", p.Version)\n\t}\n}\n\nfunc (c *clientV2) IsReadyForMessages() bool {\n\tif c.Channel.IsPaused() {\n\t\treturn false\n\t}\n\n\treadyCount := atomic.LoadInt64(&c.ReadyCount)\n\tinFlightCount := atomic.LoadInt64(&c.InFlightCount)\n\n\tc.nsqd.logf(LOG_DEBUG, \"[%s] state rdy: %4d inflt: %4d\", c, readyCount, inFlightCount)\n\n\tif inFlightCount >= readyCount || readyCount <= 0 {\n\t\treturn false\n\t}\n\n\treturn true\n}\n\nfunc (c *clientV2) SetReadyCount(count int64) {\n\toldCount := atomic.SwapInt64(&c.ReadyCount, count)\n\n\tif oldCount != count {\n\t\tc.tryUpdateReadyState()\n\t}\n}\n\nfunc (c *clientV2) tryUpdateReadyState() {\n\t// you can always *try* to write to ReadyStateChan because in the cases\n\t// where you cannot the message pump loop would have iterated anyway.\n\t// the atomic integer operations guarantee correctness of the value.\n\tselect {\n\tcase c.ReadyStateChan <- 1:\n\tdefault:\n\t}\n}\n\nfunc (c *clientV2) FinishedMessage() {\n\tatomic.AddUint64(&c.FinishCount, 1)\n\tatomic.AddInt64(&c.InFlightCount, -1)\n\tc.tryUpdateReadyState()\n}\n\nfunc (c *clientV2) Empty() {\n\tatomic.StoreInt64(&c.InFlightCount, 0)\n\tc.tryUpdateReadyState()\n}\n\nfunc (c *clientV2) SendingMessage() {\n\tatomic.AddInt64(&c.InFlightCount, 1)\n\tatomic.AddUint64(&c.MessageCount, 1)\n}\n\nfunc (c *clientV2) PublishedMessage(topic string, count uint64) {\n\tc.metaLock.Lock()\n\tc.pubCounts[topic] += count\n\tc.metaLock.Unlock()\n}\n\nfunc (c *clientV2) TimedOutMessage() {\n\tatomic.AddInt64(&c.InFlightCount, -1)\n\tc.tryUpdateReadyState()\n}\n\nfunc (c *clientV2) RequeuedMessage() {\n\tatomic.AddUint64(&c.RequeueCount, 1)\n\tatomic.AddInt64(&c.InFlightCount, -1)\n\tc.tryUpdateReadyState()\n}\n\nfunc (c *clientV2) StartClose() {\n\t// Force the client into ready 0\n\tc.SetReadyCount(0)\n\t// mark this client as closing\n\tatomic.StoreInt32(&c.State, stateClosing)\n}\n\nfunc (c *clientV2) Pause() {\n\tc.tryUpdateReadyState()\n}\n\nfunc (c *clientV2) UnPause() {\n\tc.tryUpdateReadyState()\n}\n\nfunc (c *clientV2) SetHeartbeatInterval(desiredInterval int) error {\n\tc.writeLock.Lock()\n\tdefer c.writeLock.Unlock()\n\n\tswitch {\n\tcase desiredInterval == -1:\n\t\tc.HeartbeatInterval = 0\n\tcase desiredInterval == 0:\n\t\t// do nothing (use default)\n\tcase desiredInterval >= 1000 &&\n\t\tdesiredInterval <= int(c.nsqd.getOpts().MaxHeartbeatInterval/time.Millisecond):\n\t\tc.HeartbeatInterval = time.Duration(desiredInterval) * time.Millisecond\n\tdefault:\n\t\treturn fmt.Errorf(\"heartbeat interval (%d) is invalid\", desiredInterval)\n\t}\n\n\treturn nil\n}\n\nfunc (c *clientV2) SetOutputBuffer(desiredSize int, desiredTimeout int) error {\n\tc.writeLock.Lock()\n\tdefer c.writeLock.Unlock()\n\n\tswitch {\n\tcase desiredTimeout == -1:\n\t\tc.OutputBufferTimeout = 0\n\tcase desiredTimeout == 0:\n\t\t// do nothing (use default)\n\tcase true &&\n\t\tdesiredTimeout >= int(c.nsqd.getOpts().MinOutputBufferTimeout/time.Millisecond) &&\n\t\tdesiredTimeout <= int(c.nsqd.getOpts().MaxOutputBufferTimeout/time.Millisecond):\n\n\t\tc.OutputBufferTimeout = time.Duration(desiredTimeout) * time.Millisecond\n\tdefault:\n\t\treturn fmt.Errorf(\"output buffer timeout (%d) is invalid\", desiredTimeout)\n\t}\n\n\tswitch {\n\tcase desiredSize == -1:\n\t\t// effectively no buffer (every write will go directly to the wrapped net.Conn)\n\t\tc.OutputBufferSize = 1\n\t\tc.OutputBufferTimeout = 0\n\tcase desiredSize == 0:\n\t\t// do nothing (use default)\n\tcase desiredSize >= 64 && desiredSize <= int(c.nsqd.getOpts().MaxOutputBufferSize):\n\t\tc.OutputBufferSize = desiredSize\n\tdefault:\n\t\treturn fmt.Errorf(\"output buffer size (%d) is invalid\", desiredSize)\n\t}\n\n\tif desiredSize != 0 {\n\t\terr := c.Writer.Flush()\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tc.Writer = bufio.NewWriterSize(c.Conn, c.OutputBufferSize)\n\t}\n\n\treturn nil\n}\n\nfunc (c *clientV2) SetSampleRate(sampleRate int32) error {\n\tif sampleRate < 0 || sampleRate > 99 {\n\t\treturn fmt.Errorf(\"sample rate (%d) is invalid\", sampleRate)\n\t}\n\tatomic.StoreInt32(&c.SampleRate, sampleRate)\n\treturn nil\n}\n\nfunc (c *clientV2) SetMsgTimeout(msgTimeout int) error {\n\tc.writeLock.Lock()\n\tdefer c.writeLock.Unlock()\n\n\tswitch {\n\tcase msgTimeout == 0:\n\t\t// do nothing (use default)\n\tcase msgTimeout >= 1000 &&\n\t\tmsgTimeout <= int(c.nsqd.getOpts().MaxMsgTimeout/time.Millisecond):\n\t\tc.MsgTimeout = time.Duration(msgTimeout) * time.Millisecond\n\tdefault:\n\t\treturn fmt.Errorf(\"msg timeout (%d) is invalid\", msgTimeout)\n\t}\n\n\treturn nil\n}\n\nfunc (c *clientV2) UpgradeTLS() error {\n\tc.writeLock.Lock()\n\tdefer c.writeLock.Unlock()\n\n\ttlsConn := tls.Server(c.Conn, c.nsqd.tlsConfig)\n\ttlsConn.SetDeadline(time.Now().Add(5 * time.Second))\n\terr := tlsConn.Handshake()\n\tif err != nil {\n\t\treturn err\n\t}\n\tc.tlsConn = tlsConn\n\n\tc.Reader = bufio.NewReaderSize(c.tlsConn, defaultBufferSize)\n\tc.Writer = bufio.NewWriterSize(c.tlsConn, c.OutputBufferSize)\n\n\tatomic.StoreInt32(&c.TLS, 1)\n\n\treturn nil\n}\n\nfunc (c *clientV2) UpgradeDeflate(level int) error {\n\tc.writeLock.Lock()\n\tdefer c.writeLock.Unlock()\n\n\tconn := c.Conn\n\tif c.tlsConn != nil {\n\t\tconn = c.tlsConn\n\t}\n\n\tc.Reader = bufio.NewReaderSize(flate.NewReader(conn), defaultBufferSize)\n\n\tfw, _ := flate.NewWriter(conn, level)\n\tc.flateWriter = fw\n\tc.Writer = bufio.NewWriterSize(fw, c.OutputBufferSize)\n\n\tatomic.StoreInt32(&c.Deflate, 1)\n\n\treturn nil\n}\n\nfunc (c *clientV2) UpgradeSnappy() error {\n\tc.writeLock.Lock()\n\tdefer c.writeLock.Unlock()\n\n\tconn := c.Conn\n\tif c.tlsConn != nil {\n\t\tconn = c.tlsConn\n\t}\n\n\tc.Reader = bufio.NewReaderSize(snappy.NewReader(conn), defaultBufferSize)\n\t//lint:ignore SA1019 NewWriter is deprecated by NewBufferedWriter, but we're doing our own buffering\n\tc.Writer = bufio.NewWriterSize(snappy.NewWriter(conn), c.OutputBufferSize)\n\n\tatomic.StoreInt32(&c.Snappy, 1)\n\n\treturn nil\n}\n\nfunc (c *clientV2) Flush() error {\n\tvar zeroTime time.Time\n\tif c.HeartbeatInterval > 0 {\n\t\tc.SetWriteDeadline(time.Now().Add(c.HeartbeatInterval))\n\t} else {\n\t\tc.SetWriteDeadline(zeroTime)\n\t}\n\n\terr := c.Writer.Flush()\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tif c.flateWriter != nil {\n\t\treturn c.flateWriter.Flush()\n\t}\n\n\treturn nil\n}\n\nfunc (c *clientV2) QueryAuthd() error {\n\tremoteIP := \"\"\n\tif c.RemoteAddr().Network() == \"tcp\" {\n\t\tip, _, err := net.SplitHostPort(c.String())\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\tremoteIP = ip\n\t}\n\n\ttlsEnabled := atomic.LoadInt32(&c.TLS) == 1\n\tcommonName := \"\"\n\tif tlsEnabled {\n\t\ttlsConnState := c.tlsConn.ConnectionState()\n\t\tif len(tlsConnState.PeerCertificates) > 0 {\n\t\t\tcommonName = tlsConnState.PeerCertificates[0].Subject.CommonName\n\t\t}\n\t}\n\n\tauthState, err := auth.QueryAnyAuthd(c.nsqd.getOpts().AuthHTTPAddresses,\n\t\tremoteIP, tlsEnabled, commonName, c.AuthSecret,\n\t\tc.nsqd.clientTLSConfig,\n\t\tc.nsqd.getOpts().HTTPClientConnectTimeout,\n\t\tc.nsqd.getOpts().HTTPClientRequestTimeout,\n\t\tc.nsqd.getOpts().AuthHTTPRequestMethod,\n\t)\n\tif err != nil {\n\t\treturn err\n\t}\n\tc.AuthState = authState\n\treturn nil\n}\n\nfunc (c *clientV2) Auth(secret string) error {\n\tc.AuthSecret = secret\n\treturn c.QueryAuthd()\n}\n\nfunc (c *clientV2) IsAuthorized(topic, channel string) (bool, error) {\n\tif c.AuthState == nil {\n\t\treturn false, nil\n\t}\n\tif c.AuthState.IsExpired() {\n\t\terr := c.QueryAuthd()\n\t\tif err != nil {\n\t\t\treturn false, err\n\t\t}\n\t}\n\tif c.AuthState.IsAllowed(topic, channel) {\n\t\treturn true, nil\n\t}\n\treturn false, nil\n}\n\nfunc (c *clientV2) HasAuthorizations() bool {\n\tif c.AuthState != nil {\n\t\treturn len(c.AuthState.Authorizations) != 0\n\t}\n\treturn false\n}\n" }, { "path": "nsqd/dqname.go", "content": "//go:build !windows\n// +build !windows\n\npackage nsqd\n\nfunc getBackendName(topicName, channelName string) string {\n\t// backend names, for uniqueness, automatically include the topic... :\n\tbackendName := topicName + \":\" + channelName\n\treturn backendName\n}\n" }, { "path": "nsqd/dqname_windows.go", "content": "//go:build windows\n// +build windows\n\npackage nsqd\n\n// On Windows, file names cannot contain colons.\nfunc getBackendName(topicName, channelName string) string {\n\t// backend names, for uniqueness, automatically include the topic... ;\n\tbackendName := topicName + \";\" + channelName\n\treturn backendName\n}\n" }, { "path": "nsqd/dummy_backend_queue.go", "content": "package nsqd\n\ntype dummyBackendQueue struct {\n\treadChan chan []byte\n}\n\nfunc newDummyBackendQueue() BackendQueue {\n\treturn &dummyBackendQueue{readChan: make(chan []byte)}\n}\n\nfunc (d *dummyBackendQueue) Put([]byte) error {\n\treturn nil\n}\n\nfunc (d *dummyBackendQueue) ReadChan() <-chan []byte {\n\treturn d.readChan\n}\n\nfunc (d *dummyBackendQueue) Close() error {\n\treturn nil\n}\n\nfunc (d *dummyBackendQueue) Delete() error {\n\treturn nil\n}\n\nfunc (d *dummyBackendQueue) Depth() int64 {\n\treturn int64(0)\n}\n\nfunc (d *dummyBackendQueue) Empty() error {\n\treturn nil\n}\n" }, { "path": "nsqd/guid.go", "content": "package nsqd\n\n// the core algorithm here was borrowed from:\n// Blake Mizerany's `noeqd` https://github.com/bmizerany/noeqd\n// and indirectly:\n// Twitter's `snowflake` https://github.com/twitter/snowflake\n\n// only minor cleanup and changes to introduce a type, combine the concept\n// of workerID + datacenterId into a single identifier, and modify the\n// behavior when sequences rollover for our specific implementation needs\n\nimport (\n\t\"encoding/hex\"\n\t\"errors\"\n\t\"sync\"\n\t\"time\"\n)\n\nconst (\n\tnodeIDBits = uint64(10)\n\tsequenceBits = uint64(12)\n\tnodeIDShift = sequenceBits\n\ttimestampShift = sequenceBits + nodeIDBits\n\tsequenceMask = int64(-1) ^ (int64(-1) << sequenceBits)\n\n\t// ( 2012-10-28 16:23:42 UTC ).UnixNano() >> 20\n\ttwepoch = int64(1288834974288)\n)\n\nvar ErrTimeBackwards = errors.New(\"time has gone backwards\")\nvar ErrSequenceExpired = errors.New(\"sequence expired\")\nvar ErrIDBackwards = errors.New(\"ID went backward\")\n\ntype guid int64\n\ntype guidFactory struct {\n\tsync.Mutex\n\n\tnodeID int64\n\tsequence int64\n\tlastTimestamp int64\n\tlastID guid\n}\n\nfunc NewGUIDFactory(nodeID int64) *guidFactory {\n\treturn &guidFactory{\n\t\tnodeID: nodeID,\n\t}\n}\n\nfunc (f *guidFactory) NewGUID() (guid, error) {\n\tf.Lock()\n\n\t// divide by 1048576, giving pseudo-milliseconds\n\tts := time.Now().UnixNano() >> 20\n\n\tif ts < f.lastTimestamp {\n\t\tf.Unlock()\n\t\treturn 0, ErrTimeBackwards\n\t}\n\n\tif f.lastTimestamp == ts {\n\t\tf.sequence = (f.sequence + 1) & sequenceMask\n\t\tif f.sequence == 0 {\n\t\t\tf.Unlock()\n\t\t\treturn 0, ErrSequenceExpired\n\t\t}\n\t} else {\n\t\tf.sequence = 0\n\t}\n\n\tf.lastTimestamp = ts\n\n\tid := guid(((ts - twepoch) << timestampShift) |\n\t\t(f.nodeID << nodeIDShift) |\n\t\tf.sequence)\n\n\tif id <= f.lastID {\n\t\tf.Unlock()\n\t\treturn 0, ErrIDBackwards\n\t}\n\n\tf.lastID = id\n\n\tf.Unlock()\n\n\treturn id, nil\n}\n\nfunc (g guid) Hex() MessageID {\n\tvar h MessageID\n\tvar b [8]byte\n\n\tb[0] = byte(g >> 56)\n\tb[1] = byte(g >> 48)\n\tb[2] = byte(g >> 40)\n\tb[3] = byte(g >> 32)\n\tb[4] = byte(g >> 24)\n\tb[5] = byte(g >> 16)\n\tb[6] = byte(g >> 8)\n\tb[7] = byte(g)\n\n\thex.Encode(h[:], b[:])\n\treturn h\n}\n" }, { "path": "nsqd/guid_test.go", "content": "package nsqd\n\nimport (\n\t\"testing\"\n\t\"unsafe\"\n)\n\nfunc BenchmarkGUIDCopy(b *testing.B) {\n\tsource := make([]byte, 16)\n\tvar dest MessageID\n\tfor i := 0; i < b.N; i++ {\n\t\tcopy(dest[:], source)\n\t}\n}\n\nfunc BenchmarkGUIDUnsafe(b *testing.B) {\n\tsource := make([]byte, 16)\n\tvar dest MessageID\n\tfor i := 0; i < b.N; i++ {\n\t\tdest = *(*MessageID)(unsafe.Pointer(&source[0]))\n\t}\n\t_ = dest\n}\n\nfunc BenchmarkGUID(b *testing.B) {\n\tvar okays, errors, fails int64\n\tvar previd guid\n\tfactory := &guidFactory{}\n\tfor i := 0; i < b.N; i++ {\n\t\tid, err := factory.NewGUID()\n\t\tif err != nil {\n\t\t\terrors++\n\t\t} else if id == previd {\n\t\t\tfails++\n\t\t\tb.Fail()\n\t\t} else {\n\t\t\tokays++\n\t\t}\n\t\tid.Hex()\n\t}\n\tb.Logf(\"okays=%d errors=%d bads=%d\", okays, errors, fails)\n}\n" }, { "path": "nsqd/http.go", "content": "package nsqd\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net\"\n\t\"net/http\"\n\t\"net/http/pprof\"\n\t\"net/url\"\n\t\"os\"\n\t\"reflect\"\n\t\"runtime\"\n\t\"runtime/debug\"\n\t\"strconv\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/julienschmidt/httprouter\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nvar boolParams = map[string]bool{\n\t\"true\": true,\n\t\"1\": true,\n\t\"false\": false,\n\t\"0\": false,\n}\n\ntype httpServer struct {\n\tnsqd *NSQD\n\ttlsEnabled bool\n\ttlsRequired bool\n\trouter http.Handler\n}\n\nfunc newHTTPServer(nsqd *NSQD, tlsEnabled bool, tlsRequired bool) *httpServer {\n\tlog := http_api.Log(nsqd.logf)\n\n\trouter := httprouter.New()\n\trouter.HandleMethodNotAllowed = true\n\trouter.PanicHandler = http_api.LogPanicHandler(nsqd.logf)\n\trouter.NotFound = http_api.LogNotFoundHandler(nsqd.logf)\n\trouter.MethodNotAllowed = http_api.LogMethodNotAllowedHandler(nsqd.logf)\n\ts := &httpServer{\n\t\tnsqd: nsqd,\n\t\ttlsEnabled: tlsEnabled,\n\t\ttlsRequired: tlsRequired,\n\t\trouter: router,\n\t}\n\n\trouter.Handle(\"GET\", \"/ping\", http_api.Decorate(s.pingHandler, log, http_api.PlainText))\n\trouter.Handle(\"GET\", \"/info\", http_api.Decorate(s.doInfo, log, http_api.V1))\n\n\t// v1 negotiate\n\trouter.Handle(\"POST\", \"/pub\", http_api.Decorate(s.doPUB, http_api.V1))\n\trouter.Handle(\"POST\", \"/mpub\", http_api.Decorate(s.doMPUB, http_api.V1))\n\trouter.Handle(\"GET\", \"/stats\", http_api.Decorate(s.doStats, log, http_api.V1))\n\n\t// only v1\n\trouter.Handle(\"POST\", \"/topic/create\", http_api.Decorate(s.doCreateTopic, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/topic/delete\", http_api.Decorate(s.doDeleteTopic, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/topic/empty\", http_api.Decorate(s.doEmptyTopic, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/topic/pause\", http_api.Decorate(s.doPauseTopic, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/topic/unpause\", http_api.Decorate(s.doPauseTopic, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/channel/create\", http_api.Decorate(s.doCreateChannel, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/channel/delete\", http_api.Decorate(s.doDeleteChannel, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/channel/empty\", http_api.Decorate(s.doEmptyChannel, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/channel/pause\", http_api.Decorate(s.doPauseChannel, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/channel/unpause\", http_api.Decorate(s.doPauseChannel, log, http_api.V1))\n\trouter.Handle(\"GET\", \"/config/:opt\", http_api.Decorate(s.doConfig, log, http_api.V1))\n\trouter.Handle(\"PUT\", \"/config/:opt\", http_api.Decorate(s.doConfig, log, http_api.V1))\n\n\t// debug\n\trouter.HandlerFunc(\"GET\", \"/debug/pprof/\", pprof.Index)\n\trouter.HandlerFunc(\"GET\", \"/debug/pprof/cmdline\", pprof.Cmdline)\n\trouter.HandlerFunc(\"GET\", \"/debug/pprof/symbol\", pprof.Symbol)\n\trouter.HandlerFunc(\"POST\", \"/debug/pprof/symbol\", pprof.Symbol)\n\trouter.HandlerFunc(\"GET\", \"/debug/pprof/profile\", pprof.Profile)\n\trouter.Handler(\"GET\", \"/debug/pprof/heap\", pprof.Handler(\"heap\"))\n\trouter.Handler(\"GET\", \"/debug/pprof/goroutine\", pprof.Handler(\"goroutine\"))\n\trouter.Handler(\"GET\", \"/debug/pprof/block\", pprof.Handler(\"block\"))\n\trouter.Handle(\"PUT\", \"/debug/setblockrate\", http_api.Decorate(setBlockRateHandler, log, http_api.PlainText))\n\trouter.Handle(\"POST\", \"/debug/freememory\", http_api.Decorate(freeMemory, log, http_api.PlainText))\n\trouter.Handler(\"GET\", \"/debug/pprof/threadcreate\", pprof.Handler(\"threadcreate\"))\n\n\treturn s\n}\n\nfunc setBlockRateHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\trate, err := strconv.Atoi(req.FormValue(\"rate\"))\n\tif err != nil {\n\t\treturn nil, http_api.Err{http.StatusBadRequest, fmt.Sprintf(\"invalid block rate : %s\", err.Error())}\n\t}\n\truntime.SetBlockProfileRate(rate)\n\treturn nil, nil\n}\n\nfunc freeMemory(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tdebug.FreeOSMemory()\n\treturn nil, nil\n}\n\nfunc (s *httpServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {\n\tif !s.tlsEnabled && s.tlsRequired {\n\t\tresp := fmt.Sprintf(`{\"message\": \"TLS_REQUIRED\", \"https_port\": %d}`,\n\t\t\ts.nsqd.RealHTTPSAddr().Port)\n\t\tw.Header().Set(\"X-NSQ-Content-Type\", \"nsq; version=1.0\")\n\t\tw.Header().Set(\"Content-Type\", \"application/json; charset=utf-8\")\n\t\tw.WriteHeader(403)\n\t\tio.WriteString(w, resp)\n\t\treturn\n\t}\n\ts.router.ServeHTTP(w, req)\n}\n\nfunc (s *httpServer) pingHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\thealth := s.nsqd.GetHealth()\n\tif !s.nsqd.IsHealthy() {\n\t\treturn nil, http_api.Err{500, health}\n\t}\n\treturn health, nil\n}\n\nfunc (s *httpServer) doInfo(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\thostname, err := os.Hostname()\n\tif err != nil {\n\t\treturn nil, http_api.Err{500, err.Error()}\n\t}\n\ttcpPort := -1 // in case of unix socket\n\tif s.nsqd.RealTCPAddr().Network() == \"tcp\" {\n\t\ttcpPort = s.nsqd.RealTCPAddr().(*net.TCPAddr).Port\n\t}\n\thttpPort := -1 // in case of unix socket\n\tif s.nsqd.RealHTTPAddr().Network() == \"tcp\" {\n\t\thttpPort = s.nsqd.RealHTTPAddr().(*net.TCPAddr).Port\n\t}\n\n\treturn struct {\n\t\tVersion string `json:\"version\"`\n\t\tBroadcastAddress string `json:\"broadcast_address\"`\n\t\tHostname string `json:\"hostname\"`\n\t\tHTTPPort int `json:\"http_port\"`\n\t\tTCPPort int `json:\"tcp_port\"`\n\t\tStartTime int64 `json:\"start_time\"`\n\t\tMaxHeartBeatInterval time.Duration `json:\"max_heartbeat_interval\"`\n\t\tMaxOutBufferSize int64 `json:\"max_output_buffer_size\"`\n\t\tMaxOutBufferTimeout time.Duration `json:\"max_output_buffer_timeout\"`\n\t\tMaxDeflateLevel int `json:\"max_deflate_level\"`\n\t\tTopologyZone string `json:\"topology_zone\"`\n\t\tTopologyRegion string `json:\"topology_region\"`\n\t}{\n\t\tVersion: version.Binary,\n\t\tBroadcastAddress: s.nsqd.getOpts().BroadcastAddress,\n\t\tHostname: hostname,\n\t\tTCPPort: tcpPort,\n\t\tHTTPPort: httpPort,\n\t\tStartTime: s.nsqd.GetStartTime().Unix(),\n\t\tMaxHeartBeatInterval: s.nsqd.getOpts().MaxHeartbeatInterval,\n\t\tMaxOutBufferSize: s.nsqd.getOpts().MaxOutputBufferSize,\n\t\tMaxOutBufferTimeout: s.nsqd.getOpts().MaxOutputBufferTimeout,\n\t\tMaxDeflateLevel: s.nsqd.getOpts().MaxDeflateLevel,\n\t\tTopologyZone: s.nsqd.getOpts().TopologyZone,\n\t\tTopologyRegion: s.nsqd.getOpts().TopologyRegion,\n\t}, nil\n}\n\nfunc (s *httpServer) getExistingTopicFromQuery(req *http.Request) (*http_api.ReqParams, *Topic, string, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\ts.nsqd.logf(LOG_ERROR, \"failed to parse request params - %s\", err)\n\t\treturn nil, nil, \"\", http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, channelName, err := http_api.GetTopicChannelArgs(reqParams)\n\tif err != nil {\n\t\treturn nil, nil, \"\", http_api.Err{400, err.Error()}\n\t}\n\n\ttopic, err := s.nsqd.GetExistingTopic(topicName)\n\tif err != nil {\n\t\treturn nil, nil, \"\", http_api.Err{404, \"TOPIC_NOT_FOUND\"}\n\t}\n\n\treturn reqParams, topic, channelName, err\n}\n\nfunc (s *httpServer) getTopicFromQuery(req *http.Request) (url.Values, *Topic, error) {\n\treqParams, err := url.ParseQuery(req.URL.RawQuery)\n\tif err != nil {\n\t\ts.nsqd.logf(LOG_ERROR, \"failed to parse request params - %s\", err)\n\t\treturn nil, nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicNames, ok := reqParams[\"topic\"]\n\tif !ok {\n\t\treturn nil, nil, http_api.Err{400, \"MISSING_ARG_TOPIC\"}\n\t}\n\ttopicName := topicNames[0]\n\n\tif !protocol.IsValidTopicName(topicName) {\n\t\treturn nil, nil, http_api.Err{400, \"INVALID_TOPIC\"}\n\t}\n\n\treturn reqParams, s.nsqd.GetTopic(topicName), nil\n}\n\nfunc (s *httpServer) doPUB(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t// TODO: one day I'd really like to just error on chunked requests\n\t// to be able to fail \"too big\" requests before we even read\n\n\tif req.ContentLength > s.nsqd.getOpts().MaxMsgSize {\n\t\treturn nil, http_api.Err{413, \"MSG_TOO_BIG\"}\n\t}\n\n\t// add 1 so that it's greater than our max when we test for it\n\t// (LimitReader returns a \"fake\" EOF)\n\treadMax := s.nsqd.getOpts().MaxMsgSize + 1\n\tbody, err := io.ReadAll(io.LimitReader(req.Body, readMax))\n\tif err != nil {\n\t\treturn nil, http_api.Err{500, \"INTERNAL_ERROR\"}\n\t}\n\tif int64(len(body)) == readMax {\n\t\treturn nil, http_api.Err{413, \"MSG_TOO_BIG\"}\n\t}\n\tif len(body) == 0 {\n\t\treturn nil, http_api.Err{400, \"MSG_EMPTY\"}\n\t}\n\n\treqParams, topic, err := s.getTopicFromQuery(req)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tvar deferred time.Duration\n\tif ds, ok := reqParams[\"defer\"]; ok {\n\t\tvar di int64\n\t\tdi, err = strconv.ParseInt(ds[0], 10, 64)\n\t\tif err != nil {\n\t\t\treturn nil, http_api.Err{400, \"INVALID_DEFER\"}\n\t\t}\n\t\tdeferred = time.Duration(di) * time.Millisecond\n\t\tif deferred < 0 || deferred > s.nsqd.getOpts().MaxDeferTimeout {\n\t\t\treturn nil, http_api.Err{400, \"INVALID_DEFER\"}\n\t\t}\n\t}\n\n\tmsg := NewMessage(topic.GenerateID(), body)\n\tmsg.deferred = deferred\n\terr = topic.PutMessage(msg)\n\tif err != nil {\n\t\treturn nil, http_api.Err{503, \"EXITING\"}\n\t}\n\n\treturn \"OK\", nil\n}\n\nfunc (s *httpServer) doMPUB(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\tvar msgs []*Message\n\tvar exit bool\n\n\t// TODO: one day I'd really like to just error on chunked requests\n\t// to be able to fail \"too big\" requests before we even read\n\n\tif req.ContentLength > s.nsqd.getOpts().MaxBodySize {\n\t\treturn nil, http_api.Err{413, \"BODY_TOO_BIG\"}\n\t}\n\n\treqParams, topic, err := s.getTopicFromQuery(req)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// text mode is default, but unrecognized binary opt considered true\n\tbinaryMode := false\n\tif vals, ok := reqParams[\"binary\"]; ok {\n\t\tif binaryMode, ok = boolParams[vals[0]]; !ok {\n\t\t\tbinaryMode = true\n\t\t\ts.nsqd.logf(LOG_WARN, \"deprecated value '%s' used for /mpub binary param\", vals[0])\n\t\t}\n\t}\n\tif binaryMode {\n\t\ttmp := make([]byte, 4)\n\t\tmsgs, err = readMPUB(req.Body, tmp, topic,\n\t\t\ts.nsqd.getOpts().MaxMsgSize, s.nsqd.getOpts().MaxBodySize)\n\t\tif err != nil {\n\t\t\treturn nil, http_api.Err{413, err.(*protocol.FatalClientErr).Code[2:]}\n\t\t}\n\t} else {\n\t\t// add 1 so that it's greater than our max when we test for it\n\t\t// (LimitReader returns a \"fake\" EOF)\n\t\treadMax := s.nsqd.getOpts().MaxBodySize + 1\n\t\trdr := bufio.NewReader(io.LimitReader(req.Body, readMax))\n\t\ttotal := 0\n\t\tfor !exit {\n\t\t\tvar block []byte\n\t\t\tblock, err = rdr.ReadBytes('\\n')\n\t\t\tif err != nil {\n\t\t\t\tif err != io.EOF {\n\t\t\t\t\treturn nil, http_api.Err{500, \"INTERNAL_ERROR\"}\n\t\t\t\t}\n\t\t\t\texit = true\n\t\t\t}\n\t\t\ttotal += len(block)\n\t\t\tif int64(total) == readMax {\n\t\t\t\treturn nil, http_api.Err{413, \"BODY_TOO_BIG\"}\n\t\t\t}\n\n\t\t\tif len(block) > 0 && block[len(block)-1] == '\\n' {\n\t\t\t\tblock = block[:len(block)-1]\n\t\t\t}\n\n\t\t\t// silently discard 0 length messages\n\t\t\t// this maintains the behavior pre 0.2.22\n\t\t\tif len(block) == 0 {\n\t\t\t\tcontinue\n\t\t\t}\n\n\t\t\tif int64(len(block)) > s.nsqd.getOpts().MaxMsgSize {\n\t\t\t\treturn nil, http_api.Err{413, \"MSG_TOO_BIG\"}\n\t\t\t}\n\n\t\t\tmsg := NewMessage(topic.GenerateID(), block)\n\t\t\tmsgs = append(msgs, msg)\n\t\t}\n\t}\n\n\terr = topic.PutMessages(msgs)\n\tif err != nil {\n\t\treturn nil, http_api.Err{503, \"EXITING\"}\n\t}\n\n\treturn \"OK\", nil\n}\n\nfunc (s *httpServer) doCreateTopic(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t_, _, err := s.getTopicFromQuery(req)\n\treturn nil, err\n}\n\nfunc (s *httpServer) doEmptyTopic(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\ts.nsqd.logf(LOG_ERROR, \"failed to parse request params - %s\", err)\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, err := reqParams.Get(\"topic\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"MISSING_ARG_TOPIC\"}\n\t}\n\n\tif !protocol.IsValidTopicName(topicName) {\n\t\treturn nil, http_api.Err{400, \"INVALID_TOPIC\"}\n\t}\n\n\ttopic, err := s.nsqd.GetExistingTopic(topicName)\n\tif err != nil {\n\t\treturn nil, http_api.Err{404, \"TOPIC_NOT_FOUND\"}\n\t}\n\n\terr = topic.Empty()\n\tif err != nil {\n\t\treturn nil, http_api.Err{500, \"INTERNAL_ERROR\"}\n\t}\n\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doDeleteTopic(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\ts.nsqd.logf(LOG_ERROR, \"failed to parse request params - %s\", err)\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, err := reqParams.Get(\"topic\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"MISSING_ARG_TOPIC\"}\n\t}\n\n\terr = s.nsqd.DeleteExistingTopic(topicName)\n\tif err != nil {\n\t\treturn nil, http_api.Err{404, \"TOPIC_NOT_FOUND\"}\n\t}\n\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doPauseTopic(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\ts.nsqd.logf(LOG_ERROR, \"failed to parse request params - %s\", err)\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, err := reqParams.Get(\"topic\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"MISSING_ARG_TOPIC\"}\n\t}\n\n\ttopic, err := s.nsqd.GetExistingTopic(topicName)\n\tif err != nil {\n\t\treturn nil, http_api.Err{404, \"TOPIC_NOT_FOUND\"}\n\t}\n\n\tif strings.Contains(req.URL.Path, \"unpause\") {\n\t\terr = topic.UnPause()\n\t} else {\n\t\terr = topic.Pause()\n\t}\n\tif err != nil {\n\t\ts.nsqd.logf(LOG_ERROR, \"failure in %s - %s\", req.URL.Path, err)\n\t\treturn nil, http_api.Err{500, \"INTERNAL_ERROR\"}\n\t}\n\n\t// pro-actively persist metadata so in case of process failure\n\t// nsqd won't suddenly (un)pause a topic\n\ts.nsqd.Lock()\n\ts.nsqd.PersistMetadata()\n\ts.nsqd.Unlock()\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doCreateChannel(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t_, topic, channelName, err := s.getExistingTopicFromQuery(req)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\ttopic.GetChannel(channelName)\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doEmptyChannel(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t_, topic, channelName, err := s.getExistingTopicFromQuery(req)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tchannel, err := topic.GetExistingChannel(channelName)\n\tif err != nil {\n\t\treturn nil, http_api.Err{404, \"CHANNEL_NOT_FOUND\"}\n\t}\n\n\terr = channel.Empty()\n\tif err != nil {\n\t\treturn nil, http_api.Err{500, \"INTERNAL_ERROR\"}\n\t}\n\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doDeleteChannel(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t_, topic, channelName, err := s.getExistingTopicFromQuery(req)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\terr = topic.DeleteExistingChannel(channelName)\n\tif err != nil {\n\t\treturn nil, http_api.Err{404, \"CHANNEL_NOT_FOUND\"}\n\t}\n\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doPauseChannel(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t_, topic, channelName, err := s.getExistingTopicFromQuery(req)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tchannel, err := topic.GetExistingChannel(channelName)\n\tif err != nil {\n\t\treturn nil, http_api.Err{404, \"CHANNEL_NOT_FOUND\"}\n\t}\n\n\tif strings.Contains(req.URL.Path, \"unpause\") {\n\t\terr = channel.UnPause()\n\t} else {\n\t\terr = channel.Pause()\n\t}\n\tif err != nil {\n\t\ts.nsqd.logf(LOG_ERROR, \"failure in %s - %s\", req.URL.Path, err)\n\t\treturn nil, http_api.Err{500, \"INTERNAL_ERROR\"}\n\t}\n\n\t// pro-actively persist metadata so in case of process failure\n\t// nsqd won't suddenly (un)pause a channel\n\ts.nsqd.Lock()\n\ts.nsqd.PersistMetadata()\n\ts.nsqd.Unlock()\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doStats(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\ts.nsqd.logf(LOG_ERROR, \"failed to parse request params - %s\", err)\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\tformatString, _ := reqParams.Get(\"format\")\n\ttopicName, _ := reqParams.Get(\"topic\")\n\tchannelName, _ := reqParams.Get(\"channel\")\n\tincludeClientsParam, _ := reqParams.Get(\"include_clients\")\n\tincludeMemParam, _ := reqParams.Get(\"include_mem\")\n\tjsonFormat := formatString == \"json\"\n\n\tincludeClients, ok := boolParams[includeClientsParam]\n\tif !ok {\n\t\tincludeClients = true\n\t}\n\tincludeMem, ok := boolParams[includeMemParam]\n\tif !ok {\n\t\tincludeMem = true\n\t}\n\n\tstats := s.nsqd.GetStats(topicName, channelName, includeClients)\n\thealth := s.nsqd.GetHealth()\n\tstartTime := s.nsqd.GetStartTime()\n\tuptime := time.Since(startTime)\n\n\tvar ms *memStats\n\tif includeMem {\n\t\tm := getMemStats()\n\t\tms = &m\n\t}\n\tif !jsonFormat {\n\t\treturn s.printStats(stats, ms, health, startTime, uptime), nil\n\t}\n\n\t// TODO: should producer stats be hung off topics?\n\treturn struct {\n\t\tVersion string `json:\"version\"`\n\t\tHealth string `json:\"health\"`\n\t\tStartTime int64 `json:\"start_time\"`\n\t\tTopics []TopicStats `json:\"topics\"`\n\t\tMemory *memStats `json:\"memory,omitempty\"`\n\t\tProducers []ClientStats `json:\"producers\"`\n\t}{version.Binary, health, startTime.Unix(), stats.Topics, ms, stats.Producers}, nil\n}\n\nfunc (s *httpServer) printStats(stats Stats, ms *memStats, health string, startTime time.Time, uptime time.Duration) []byte {\n\tvar buf bytes.Buffer\n\tw := &buf\n\n\tfmt.Fprintf(w, \"%s\\n\", version.String(\"nsqd\"))\n\tfmt.Fprintf(w, \"start_time %v\\n\", startTime.Format(time.RFC3339))\n\tfmt.Fprintf(w, \"uptime %s\\n\", uptime)\n\n\tfmt.Fprintf(w, \"\\nHealth: %s\\n\", health)\n\n\tif ms != nil {\n\t\tfmt.Fprintf(w, \"\\nMemory:\\n\")\n\t\tfmt.Fprintf(w, \" %-25s\\t%d\\n\", \"heap_objects\", ms.HeapObjects)\n\t\tfmt.Fprintf(w, \" %-25s\\t%d\\n\", \"heap_idle_bytes\", ms.HeapIdleBytes)\n\t\tfmt.Fprintf(w, \" %-25s\\t%d\\n\", \"heap_in_use_bytes\", ms.HeapInUseBytes)\n\t\tfmt.Fprintf(w, \" %-25s\\t%d\\n\", \"heap_released_bytes\", ms.HeapReleasedBytes)\n\t\tfmt.Fprintf(w, \" %-25s\\t%d\\n\", \"gc_pause_usec_100\", ms.GCPauseUsec100)\n\t\tfmt.Fprintf(w, \" %-25s\\t%d\\n\", \"gc_pause_usec_99\", ms.GCPauseUsec99)\n\t\tfmt.Fprintf(w, \" %-25s\\t%d\\n\", \"gc_pause_usec_95\", ms.GCPauseUsec95)\n\t\tfmt.Fprintf(w, \" %-25s\\t%d\\n\", \"next_gc_bytes\", ms.NextGCBytes)\n\t\tfmt.Fprintf(w, \" %-25s\\t%d\\n\", \"gc_total_runs\", ms.GCTotalRuns)\n\t}\n\n\tif len(stats.Topics) == 0 {\n\t\tfmt.Fprintf(w, \"\\nTopics: None\\n\")\n\t} else {\n\t\tfmt.Fprintf(w, \"\\nTopics:\")\n\t}\n\n\tfor _, t := range stats.Topics {\n\t\tvar pausedPrefix string\n\t\tif t.Paused {\n\t\t\tpausedPrefix = \"*P \"\n\t\t} else {\n\t\t\tpausedPrefix = \" \"\n\t\t}\n\t\tfmt.Fprintf(w, \"\\n%s[%-15s] depth: %-5d be-depth: %-5d msgs: %-8d e2e%%: %s\\n\",\n\t\t\tpausedPrefix,\n\t\t\tt.TopicName,\n\t\t\tt.Depth,\n\t\t\tt.BackendDepth,\n\t\t\tt.MessageCount,\n\t\t\tt.E2eProcessingLatency,\n\t\t)\n\t\tfor _, c := range t.Channels {\n\t\t\tif c.Paused {\n\t\t\t\tpausedPrefix = \" *P \"\n\t\t\t} else {\n\t\t\t\tpausedPrefix = \" \"\n\t\t\t}\n\t\t\tfmt.Fprintf(w, \"%s[%-25s] depth: %-5d be-depth: %-5d inflt: %-4d def: %-4d re-q: %-5d timeout: %-5d msgs: %-8d e2e%%: %s\\n\",\n\t\t\t\tpausedPrefix,\n\t\t\t\tc.ChannelName,\n\t\t\t\tc.Depth,\n\t\t\t\tc.BackendDepth,\n\t\t\t\tc.InFlightCount,\n\t\t\t\tc.DeferredCount,\n\t\t\t\tc.RequeueCount,\n\t\t\t\tc.TimeoutCount,\n\t\t\t\tc.MessageCount,\n\t\t\t\tc.E2eProcessingLatency,\n\t\t\t)\n\t\t\tfor _, client := range c.Clients {\n\t\t\t\tfmt.Fprintf(w, \" %s\\n\", client)\n\t\t\t}\n\t\t}\n\t}\n\n\tif len(stats.Producers) == 0 {\n\t\tfmt.Fprintf(w, \"\\nProducers: None\\n\")\n\t} else {\n\t\tfmt.Fprintf(w, \"\\nProducers:\\n\")\n\t\tfor _, client := range stats.Producers {\n\t\t\tfmt.Fprintf(w, \" %s\\n\", client)\n\t\t}\n\t}\n\n\treturn buf.Bytes()\n}\n\nfunc (s *httpServer) doConfig(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\topt := ps.ByName(\"opt\")\n\n\tif req.Method == \"PUT\" {\n\t\t// add 1 so that it's greater than our max when we test for it\n\t\t// (LimitReader returns a \"fake\" EOF)\n\t\treadMax := s.nsqd.getOpts().MaxMsgSize + 1\n\t\tbody, err := io.ReadAll(io.LimitReader(req.Body, readMax))\n\t\tif err != nil {\n\t\t\treturn nil, http_api.Err{500, \"INTERNAL_ERROR\"}\n\t\t}\n\t\tif int64(len(body)) == readMax || len(body) == 0 {\n\t\t\treturn nil, http_api.Err{413, \"INVALID_VALUE\"}\n\t\t}\n\n\t\topts := *s.nsqd.getOpts()\n\t\tswitch opt {\n\t\tcase \"nsqlookupd_tcp_addresses\":\n\t\t\terr := json.Unmarshal(body, &opts.NSQLookupdTCPAddresses)\n\t\t\tif err != nil {\n\t\t\t\treturn nil, http_api.Err{400, \"INVALID_VALUE\"}\n\t\t\t}\n\t\tcase \"log_level\":\n\t\t\tlogLevelStr := string(body)\n\t\t\tlogLevel, err := lg.ParseLogLevel(logLevelStr)\n\t\t\tif err != nil {\n\t\t\t\treturn nil, http_api.Err{400, \"INVALID_VALUE\"}\n\t\t\t}\n\t\t\topts.LogLevel = logLevel\n\t\tdefault:\n\t\t\treturn nil, http_api.Err{400, \"INVALID_OPTION\"}\n\t\t}\n\t\ts.nsqd.swapOpts(&opts)\n\t\ts.nsqd.triggerOptsNotification()\n\t}\n\n\tv, ok := getOptByCfgName(s.nsqd.getOpts(), opt)\n\tif !ok {\n\t\treturn nil, http_api.Err{400, \"INVALID_OPTION\"}\n\t}\n\n\treturn v, nil\n}\n\nfunc getOptByCfgName(opts interface{}, name string) (interface{}, bool) {\n\tval := reflect.ValueOf(opts).Elem()\n\ttyp := val.Type()\n\tfor i := 0; i < typ.NumField(); i++ {\n\t\tfield := typ.Field(i)\n\t\tflagName := field.Tag.Get(\"flag\")\n\t\tcfgName := field.Tag.Get(\"cfg\")\n\t\tif flagName == \"\" {\n\t\t\tcontinue\n\t\t}\n\t\tif cfgName == \"\" {\n\t\t\tcfgName = strings.Replace(flagName, \"-\", \"_\", -1)\n\t\t}\n\t\tif name != cfgName {\n\t\t\tcontinue\n\t\t}\n\t\treturn val.FieldByName(field.Name).Interface(), true\n\t}\n\treturn nil, false\n}\n" }, { "path": "nsqd/http_test.go", "content": "package nsqd\n\nimport (\n\t\"bytes\"\n\t\"crypto/tls\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net\"\n\t\"net/http\"\n\t\"os\"\n\t\"runtime\"\n\t\"strconv\"\n\t\"strings\"\n\t\"sync\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/test\"\n\t\"github.com/nsqio/nsq/internal/version\"\n\t\"github.com/nsqio/nsq/nsqlookupd\"\n)\n\ntype ErrMessage struct {\n\tMessage string `json:\"message\"`\n}\n\ntype InfoDoc struct {\n\tVersion string `json:\"version\"`\n\tBroadcastAddress string `json:\"broadcast_address\"`\n\tHostname string `json:\"hostname\"`\n\tHTTPPort int `json:\"http_port\"`\n\tTCPPort int `json:\"tcp_port\"`\n\tStartTime int64 `json:\"start_time\"`\n}\n\nfunc TestHTTPpub(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_pub\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\n\tbuf := bytes.NewBuffer([]byte(\"test message\"))\n\turl := fmt.Sprintf(\"http://%s/pub?topic=%s\", httpAddr, topicName)\n\tresp, err := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, \"OK\", string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\ttest.Equal(t, int64(1), topic.Depth())\n}\n\nfunc TestHTTPpubEmpty(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_pub_empty\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\n\tbuf := bytes.NewBuffer([]byte(\"\"))\n\turl := fmt.Sprintf(\"http://%s/pub?topic=%s\", httpAddr, topicName)\n\tresp, err := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, `{\"message\":\"MSG_EMPTY\"}`, string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\ttest.Equal(t, int64(0), topic.Depth())\n}\n\nfunc TestHTTPmpub(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_mpub\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\n\tmsg := []byte(\"test message\")\n\tmsgs := make([][]byte, 4)\n\tfor i := range msgs {\n\t\tmsgs[i] = msg\n\t}\n\tbuf := bytes.NewBuffer(bytes.Join(msgs, []byte(\"\\n\")))\n\n\turl := fmt.Sprintf(\"http://%s/mpub?topic=%s\", httpAddr, topicName)\n\tresp, err := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, \"OK\", string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\ttest.Equal(t, int64(4), topic.Depth())\n}\n\nfunc TestHTTPmpubEmpty(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_mpub_empty\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\n\tmsg := []byte(\"test message\")\n\tmsgs := make([][]byte, 4)\n\tfor i := range msgs {\n\t\tmsgs[i] = msg\n\t}\n\tbuf := bytes.NewBuffer(bytes.Join(msgs, []byte(\"\\n\")))\n\t_, err := buf.Write([]byte(\"\\n\"))\n\ttest.Nil(t, err)\n\n\turl := fmt.Sprintf(\"http://%s/mpub?topic=%s\", httpAddr, topicName)\n\tresp, err := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, \"OK\", string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\ttest.Equal(t, int64(4), topic.Depth())\n}\n\nfunc TestHTTPmpubBinary(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_mpub_bin\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\n\tmpub := make([][]byte, 5)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 100)\n\t}\n\tcmd, _ := nsq.MultiPublish(topicName, mpub)\n\tbuf := bytes.NewBuffer(cmd.Body)\n\n\turl := fmt.Sprintf(\"http://%s/mpub?topic=%s&binary=true\", httpAddr, topicName)\n\tresp, err := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, \"OK\", string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\ttest.Equal(t, int64(5), topic.Depth())\n}\n\nfunc TestHTTPmpubForNonNormalizedBinaryParam(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_mpub_bin\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\n\tmpub := make([][]byte, 5)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 100)\n\t}\n\tcmd, _ := nsq.MultiPublish(topicName, mpub)\n\tbuf := bytes.NewBuffer(cmd.Body)\n\n\turl := fmt.Sprintf(\"http://%s/mpub?topic=%s&binary=non_normalized_binary_param\", httpAddr, topicName)\n\tresp, err := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, \"OK\", string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\ttest.Equal(t, int64(5), topic.Depth())\n}\n\nfunc TestHTTPpubDefer(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_pub_defer\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tch := topic.GetChannel(\"ch\")\n\n\tbuf := bytes.NewBuffer([]byte(\"test message\"))\n\turl := fmt.Sprintf(\"http://%s/pub?topic=%s&defer=%d\", httpAddr, topicName, 1000)\n\tresp, err := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, \"OK\", string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\tch.deferredMutex.Lock()\n\tnumDef := len(ch.deferredMessages)\n\tch.deferredMutex.Unlock()\n\ttest.Equal(t, 1, numDef)\n}\n\nfunc TestHTTPSRequire(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\topts.TLSClientAuthPolicy = \"require\"\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_pub_req\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\n\tbuf := bytes.NewBuffer([]byte(\"test message\"))\n\turl := fmt.Sprintf(\"http://%s/pub?topic=%s\", httpAddr, topicName)\n\tresp, _ := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Equal(t, 403, resp.StatusCode)\n\n\thttpsAddr := nsqd.httpsListener.Addr().(*net.TCPAddr)\n\tcert, err := tls.LoadX509KeyPair(\"./test/certs/cert.pem\", \"./test/certs/key.pem\")\n\ttest.Nil(t, err)\n\ttlsConfig := &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tInsecureSkipVerify: true,\n\t\tMinVersion: 0,\n\t}\n\ttransport := &http.Transport{\n\t\tTLSClientConfig: tlsConfig,\n\t}\n\tclient := &http.Client{Transport: transport}\n\n\tbuf = bytes.NewBuffer([]byte(\"test message\"))\n\turl = fmt.Sprintf(\"https://%s/pub?topic=%s\", httpsAddr, topicName)\n\tresp, err = client.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, \"OK\", string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\ttest.Equal(t, int64(1), topic.Depth())\n}\n\nfunc TestHTTPSRequireVerify(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\topts.TLSRootCAFile = \"./test/certs/ca.pem\"\n\topts.TLSClientAuthPolicy = \"require-verify\"\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\thttpsAddr := nsqd.httpsListener.Addr().(*net.TCPAddr)\n\ttopicName := \"test_http_pub_req_verf\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\n\t// no cert\n\tbuf := bytes.NewBuffer([]byte(\"test message\"))\n\turl := fmt.Sprintf(\"http://%s/pub?topic=%s\", httpAddr, topicName)\n\tresp, _ := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Equal(t, 403, resp.StatusCode)\n\n\t// unsigned cert\n\tcert, err := tls.LoadX509KeyPair(\"./test/certs/cert.pem\", \"./test/certs/key.pem\")\n\ttest.Nil(t, err)\n\ttlsConfig := &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tInsecureSkipVerify: true,\n\t}\n\ttransport := &http.Transport{\n\t\tTLSClientConfig: tlsConfig,\n\t}\n\tclient := &http.Client{Transport: transport}\n\n\tbuf = bytes.NewBuffer([]byte(\"test message\"))\n\turl = fmt.Sprintf(\"https://%s/pub?topic=%s\", httpsAddr, topicName)\n\t_, err = client.Post(url, \"application/octet-stream\", buf)\n\ttest.NotNil(t, err)\n\n\t// signed cert\n\tcert, err = tls.LoadX509KeyPair(\"./test/certs/client.pem\", \"./test/certs/client.key\")\n\ttest.Nil(t, err)\n\ttlsConfig = &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tInsecureSkipVerify: true,\n\t}\n\ttransport = &http.Transport{\n\t\tTLSClientConfig: tlsConfig,\n\t}\n\tclient = &http.Client{Transport: transport}\n\n\tbuf = bytes.NewBuffer([]byte(\"test message\"))\n\turl = fmt.Sprintf(\"https://%s/pub?topic=%s\", httpsAddr, topicName)\n\tresp, err = client.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, \"OK\", string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\ttest.Equal(t, int64(1), topic.Depth())\n}\n\nfunc TestTLSRequireVerifyExceptHTTP(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\topts.TLSRootCAFile = \"./test/certs/ca.pem\"\n\topts.TLSClientAuthPolicy = \"require-verify\"\n\topts.TLSRequired = TLSRequiredExceptHTTP\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_req_verf_except_http\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\n\t// no cert\n\tbuf := bytes.NewBuffer([]byte(\"test message\"))\n\turl := fmt.Sprintf(\"http://%s/pub?topic=%s\", httpAddr, topicName)\n\tresp, err := http.Post(url, \"application/octet-stream\", buf)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, \"OK\", string(body))\n\n\ttime.Sleep(5 * time.Millisecond)\n\n\ttest.Equal(t, int64(1), topic.Depth())\n}\n\nfunc TestHTTPV1TopicChannel(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_http_topic_channel2\" + strconv.Itoa(int(time.Now().Unix()))\n\tchannelName := \"ch2\"\n\n\turl := fmt.Sprintf(\"http://%s/topic/create?topic=%s\", httpAddr, topicName)\n\tresp, err := http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"\", string(body))\n\ttest.Equal(t, \"nsq; version=1.0\", resp.Header.Get(\"X-NSQ-Content-Type\"))\n\n\turl = fmt.Sprintf(\"http://%s/channel/create?topic=%s&channel=%s\", httpAddr, topicName, channelName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"\", string(body))\n\ttest.Equal(t, \"nsq; version=1.0\", resp.Header.Get(\"X-NSQ-Content-Type\"))\n\n\ttopic, err := nsqd.GetExistingTopic(topicName)\n\ttest.Nil(t, err)\n\ttest.NotNil(t, topic)\n\n\tchannel, err := topic.GetExistingChannel(channelName)\n\ttest.Nil(t, err)\n\ttest.NotNil(t, channel)\n\n\tem := ErrMessage{}\n\n\turl = fmt.Sprintf(\"http://%s/topic/pause\", httpAddr)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_TOPIC\", em.Message)\n\n\turl = fmt.Sprintf(\"http://%s/topic/pause?topic=%s\", httpAddr, topicName+\"abc\")\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 404, resp.StatusCode)\n\ttest.Equal(t, \"Not Found\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"TOPIC_NOT_FOUND\", em.Message)\n\n\turl = fmt.Sprintf(\"http://%s/topic/pause?topic=%s\", httpAddr, topicName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"\", string(body))\n\ttest.Equal(t, \"nsq; version=1.0\", resp.Header.Get(\"X-NSQ-Content-Type\"))\n\n\ttest.Equal(t, true, topic.IsPaused())\n\n\turl = fmt.Sprintf(\"http://%s/topic/unpause?topic=%s\", httpAddr, topicName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"\", string(body))\n\ttest.Equal(t, \"nsq; version=1.0\", resp.Header.Get(\"X-NSQ-Content-Type\"))\n\n\ttest.Equal(t, false, topic.IsPaused())\n\n\turl = fmt.Sprintf(\"http://%s/channel/pause?topic=%s&channel=%s\", httpAddr, topicName, channelName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"\", string(body))\n\ttest.Equal(t, \"nsq; version=1.0\", resp.Header.Get(\"X-NSQ-Content-Type\"))\n\n\ttest.Equal(t, true, channel.IsPaused())\n\n\turl = fmt.Sprintf(\"http://%s/channel/unpause?topic=%s&channel=%s\", httpAddr, topicName, channelName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"\", string(body))\n\ttest.Equal(t, \"nsq; version=1.0\", resp.Header.Get(\"X-NSQ-Content-Type\"))\n\n\ttest.Equal(t, false, channel.IsPaused())\n\n\turl = fmt.Sprintf(\"http://%s/channel/delete?topic=%s&channel=%s\", httpAddr, topicName, channelName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"\", string(body))\n\ttest.Equal(t, \"nsq; version=1.0\", resp.Header.Get(\"X-NSQ-Content-Type\"))\n\n\t_, err = topic.GetExistingChannel(channelName)\n\ttest.NotNil(t, err)\n\n\turl = fmt.Sprintf(\"http://%s/topic/delete?topic=%s\", httpAddr, topicName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"\", string(body))\n\ttest.Equal(t, \"nsq; version=1.0\", resp.Header.Get(\"X-NSQ-Content-Type\"))\n\n\t_, err = nsqd.GetExistingTopic(topicName)\n\ttest.NotNil(t, err)\n}\n\nfunc TestHTTPClientStats(t *testing.T) {\n\ttopicName := \"test_http_client_stats\" + strconv.Itoa(int(time.Now().Unix()))\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\ttcpAddr, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\tvar d struct {\n\t\tTopics []struct {\n\t\t\tChannels []struct {\n\t\t\t\tClientCount int `json:\"client_count\"`\n\t\t\t\tClients []struct {\n\t\t\t\t} `json:\"clients\"`\n\t\t\t} `json:\"channels\"`\n\t\t} `json:\"topics\"`\n\t\tMemory *struct{} `json:\"memory,omitempty\"`\n\t}\n\n\tendpoint := fmt.Sprintf(\"http://%s/stats?format=json\", httpAddr)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &d)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, 1, len(d.Topics[0].Channels[0].Clients))\n\ttest.Equal(t, 1, d.Topics[0].Channels[0].ClientCount)\n\ttest.NotNil(t, d.Memory)\n\n\tendpoint = fmt.Sprintf(\"http://%s/stats?format=json&include_clients=true\", httpAddr)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &d)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, 1, len(d.Topics[0].Channels[0].Clients))\n\ttest.Equal(t, 1, d.Topics[0].Channels[0].ClientCount)\n\n\tendpoint = fmt.Sprintf(\"http://%s/stats?format=json&include_clients=false\", httpAddr)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &d)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, 0, len(d.Topics[0].Channels[0].Clients))\n\ttest.Equal(t, 1, d.Topics[0].Channels[0].ClientCount)\n\n\tendpoint = fmt.Sprintf(\"http://%s/stats?format=json&include_mem=true\", httpAddr)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &d)\n\ttest.Nil(t, err)\n\n\ttest.NotNil(t, d.Memory)\n\n\td.Memory = nil\n\tendpoint = fmt.Sprintf(\"http://%s/stats?format=json&include_mem=false\", httpAddr)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &d)\n\ttest.Nil(t, err)\n\n\ttest.Nil(t, d.Memory)\n}\n\nfunc TestHTTPgetStatusJSON(t *testing.T) {\n\ttestTime := time.Now()\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tnsqd.startTime = testTime\n\texpectedJSON := fmt.Sprintf(`{\"version\":\"%v\",\"health\":\"OK\",\"start_time\":%v,\"topics\":[],\"memory\":{`, version.Binary, testTime.Unix())\n\n\turl := fmt.Sprintf(\"http://%s/stats?format=json\", httpAddr)\n\tresp, err := http.Get(url)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\ttest.Equal(t, true, strings.HasPrefix(string(body), expectedJSON))\n}\n\nfunc TestHTTPgetStatusText(t *testing.T) {\n\ttestTime := time.Now()\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tnsqd.startTime = testTime\n\n\turl := fmt.Sprintf(\"http://%s/stats?format=text\", httpAddr)\n\tresp, err := http.Get(url)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\ttest.NotNil(t, body)\n}\n\nfunc TestHTTPconfig(t *testing.T) {\n\tlopts := nsqlookupd.NewOptions()\n\tlopts.Logger = test.NewTestLogger(t)\n\n\tlopts1 := *lopts\n\t_, _, lookupd1 := mustStartNSQLookupd(&lopts1)\n\tdefer lookupd1.Exit()\n\tlopts2 := *lopts\n\t_, _, lookupd2 := mustStartNSQLookupd(&lopts2)\n\tdefer lookupd2.Exit()\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\turl := fmt.Sprintf(\"http://%s/config/nsqlookupd_tcp_addresses\", httpAddr)\n\tresp, err := http.Get(url)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\ttest.Equal(t, \"[]\", string(body))\n\n\tclient := http.Client{}\n\taddrs := fmt.Sprintf(`[\"%s\",\"%s\"]`, lookupd1.RealTCPAddr().String(), lookupd2.RealTCPAddr().String())\n\turl = fmt.Sprintf(\"http://%s/config/nsqlookupd_tcp_addresses\", httpAddr)\n\treq, err := http.NewRequest(\"PUT\", url, bytes.NewBuffer([]byte(addrs)))\n\ttest.Nil(t, err)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\ttest.Equal(t, addrs, string(body))\n\n\turl = fmt.Sprintf(\"http://%s/config/log_level\", httpAddr)\n\treq, err = http.NewRequest(\"PUT\", url, bytes.NewBuffer([]byte(`fatal`)))\n\ttest.Nil(t, err)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\t_, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 200, resp.StatusCode)\n\ttest.Equal(t, LOG_FATAL, nsqd.getOpts().LogLevel)\n\n\turl = fmt.Sprintf(\"http://%s/config/log_level\", httpAddr)\n\treq, err = http.NewRequest(\"PUT\", url, bytes.NewBuffer([]byte(`bad`)))\n\ttest.Nil(t, err)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\t_, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 400, resp.StatusCode)\n}\n\nfunc TestHTTPerrors(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\turl := fmt.Sprintf(\"http://%s/stats\", httpAddr)\n\tresp, err := http.Post(url, \"text/plain\", nil)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ := io.ReadAll(resp.Body)\n\ttest.Equal(t, 405, resp.StatusCode)\n\ttest.Equal(t, `{\"message\":\"METHOD_NOT_ALLOWED\"}`, string(body))\n\n\turl = fmt.Sprintf(\"http://%s/not_found\", httpAddr)\n\tresp, err = http.Get(url)\n\ttest.Nil(t, err)\n\tdefer resp.Body.Close()\n\tbody, _ = io.ReadAll(resp.Body)\n\ttest.Equal(t, 404, resp.StatusCode)\n\ttest.Equal(t, `{\"message\":\"NOT_FOUND\"}`, string(body))\n}\n\nfunc TestDeleteTopic(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tem := ErrMessage{}\n\n\turl := fmt.Sprintf(\"http://%s/topic/delete\", httpAddr)\n\tresp, err := http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_TOPIC\", em.Message)\n\n\ttopicName := \"test_http_delete_topic\" + strconv.Itoa(int(time.Now().Unix()))\n\n\turl = fmt.Sprintf(\"http://%s/topic/delete?topic=%s\", httpAddr, topicName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 404, resp.StatusCode)\n\ttest.Equal(t, \"Not Found\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"TOPIC_NOT_FOUND\", em.Message)\n\n\tnsqd.GetTopic(topicName)\n\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\ttest.Equal(t, []byte(\"\"), body)\n}\n\nfunc TestEmptyTopic(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tem := ErrMessage{}\n\n\turl := fmt.Sprintf(\"http://%s/topic/empty\", httpAddr)\n\tresp, err := http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_TOPIC\", em.Message)\n\n\ttopicName := \"test_http_empty_topic\" + strconv.Itoa(int(time.Now().Unix()))\n\n\turl = fmt.Sprintf(\"http://%s/topic/empty?topic=%s\", httpAddr, topicName+\"$\")\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"INVALID_TOPIC\", em.Message)\n\n\turl = fmt.Sprintf(\"http://%s/topic/empty?topic=%s\", httpAddr, topicName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 404, resp.StatusCode)\n\ttest.Equal(t, \"Not Found\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"TOPIC_NOT_FOUND\", em.Message)\n\n\tnsqd.GetTopic(topicName)\n\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\ttest.Equal(t, []byte(\"\"), body)\n}\n\nfunc TestEmptyChannel(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tem := ErrMessage{}\n\n\turl := fmt.Sprintf(\"http://%s/channel/empty\", httpAddr)\n\tresp, err := http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_TOPIC\", em.Message)\n\n\ttopicName := \"test_http_empty_channel\" + strconv.Itoa(int(time.Now().Unix()))\n\n\turl = fmt.Sprintf(\"http://%s/channel/empty?topic=%s\", httpAddr, topicName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_CHANNEL\", em.Message)\n\n\tchannelName := \"ch\"\n\n\turl = fmt.Sprintf(\"http://%s/channel/empty?topic=%s&channel=%s\", httpAddr, topicName, channelName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 404, resp.StatusCode)\n\ttest.Equal(t, \"Not Found\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"TOPIC_NOT_FOUND\", em.Message)\n\n\ttopic := nsqd.GetTopic(topicName)\n\n\turl = fmt.Sprintf(\"http://%s/channel/empty?topic=%s&channel=%s\", httpAddr, topicName, channelName)\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 404, resp.StatusCode)\n\ttest.Equal(t, \"Not Found\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"CHANNEL_NOT_FOUND\", em.Message)\n\n\ttopic.GetChannel(channelName)\n\n\tresp, err = http.Post(url, \"application/json\", nil)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\ttest.Equal(t, []byte(\"\"), body)\n}\n\nfunc TestInfo(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tinfo := InfoDoc{}\n\n\turl := fmt.Sprintf(\"http://%s/info\", httpAddr)\n\tresp, err := http.Get(url)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &info)\n\ttest.Nil(t, err)\n\ttest.Equal(t, version.Binary, info.Version)\n}\n\nfunc BenchmarkHTTPpub(b *testing.B) {\n\tvar wg sync.WaitGroup\n\tb.StopTimer()\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tmsg := make([]byte, 256)\n\ttopicName := \"bench_http_pub\" + strconv.Itoa(int(time.Now().Unix()))\n\turl := fmt.Sprintf(\"http://%s/pub?topic=%s\", httpAddr, topicName)\n\tclient := &http.Client{}\n\tb.SetBytes(int64(len(msg)))\n\tb.StartTimer()\n\n\tfor j := 0; j < runtime.GOMAXPROCS(0); j++ {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tnum := b.N / runtime.GOMAXPROCS(0)\n\t\t\tfor i := 0; i < num; i++ {\n\t\t\t\tbuf := bytes.NewBuffer(msg)\n\t\t\t\treq, _ := http.NewRequest(\"POST\", url, buf)\n\t\t\t\tresp, err := client.Do(req)\n\t\t\t\tif err != nil {\n\t\t\t\t\tpanic(err.Error())\n\t\t\t\t}\n\t\t\t\tbody, _ := io.ReadAll(resp.Body)\n\t\t\t\tif !bytes.Equal(body, []byte(\"OK\")) {\n\t\t\t\t\tpanic(\"bad response\")\n\t\t\t\t}\n\t\t\t\tresp.Body.Close()\n\t\t\t}\n\t\t\twg.Done()\n\t\t}()\n\t}\n\n\twg.Wait()\n\n\tb.StopTimer()\n\tnsqd.Exit()\n}\n" }, { "path": "nsqd/in_flight_pqueue.go", "content": "package nsqd\n\ntype inFlightPqueue []*Message\n\nfunc newInFlightPqueue(capacity int) inFlightPqueue {\n\treturn make(inFlightPqueue, 0, capacity)\n}\n\nfunc (pq inFlightPqueue) Swap(i, j int) {\n\tpq[i], pq[j] = pq[j], pq[i]\n\tpq[i].index = i\n\tpq[j].index = j\n}\n\nfunc (pq *inFlightPqueue) Push(x *Message) {\n\tn := len(*pq)\n\tc := cap(*pq)\n\tif n+1 > c {\n\t\tnpq := make(inFlightPqueue, n, c*2)\n\t\tcopy(npq, *pq)\n\t\t*pq = npq\n\t}\n\t*pq = (*pq)[0 : n+1]\n\tx.index = n\n\t(*pq)[n] = x\n\tpq.up(n)\n}\n\nfunc (pq *inFlightPqueue) Pop() *Message {\n\tn := len(*pq)\n\tc := cap(*pq)\n\tpq.Swap(0, n-1)\n\tpq.down(0, n-1)\n\tif n < (c/2) && c > 25 {\n\t\tnpq := make(inFlightPqueue, n, c/2)\n\t\tcopy(npq, *pq)\n\t\t*pq = npq\n\t}\n\tx := (*pq)[n-1]\n\tx.index = -1\n\t*pq = (*pq)[0 : n-1]\n\treturn x\n}\n\nfunc (pq *inFlightPqueue) Remove(i int) *Message {\n\tn := len(*pq)\n\tif n-1 != i {\n\t\tpq.Swap(i, n-1)\n\t\tpq.down(i, n-1)\n\t\tpq.up(i)\n\t}\n\tx := (*pq)[n-1]\n\tx.index = -1\n\t*pq = (*pq)[0 : n-1]\n\treturn x\n}\n\nfunc (pq *inFlightPqueue) PeekAndShift(max int64) (*Message, int64) {\n\tif len(*pq) == 0 {\n\t\treturn nil, 0\n\t}\n\n\tx := (*pq)[0]\n\tif x.pri > max {\n\t\treturn nil, x.pri - max\n\t}\n\tpq.Pop()\n\n\treturn x, 0\n}\n\nfunc (pq *inFlightPqueue) up(j int) {\n\tfor {\n\t\ti := (j - 1) / 2 // parent\n\t\tif i == j || (*pq)[j].pri >= (*pq)[i].pri {\n\t\t\tbreak\n\t\t}\n\t\tpq.Swap(i, j)\n\t\tj = i\n\t}\n}\n\nfunc (pq *inFlightPqueue) down(i, n int) {\n\tfor {\n\t\tj1 := 2*i + 1\n\t\tif j1 >= n || j1 < 0 { // j1 < 0 after int overflow\n\t\t\tbreak\n\t\t}\n\t\tj := j1 // left child\n\t\tif j2 := j1 + 1; j2 < n && (*pq)[j1].pri >= (*pq)[j2].pri {\n\t\t\tj = j2 // = 2*i + 2 // right child\n\t\t}\n\t\tif (*pq)[j].pri >= (*pq)[i].pri {\n\t\t\tbreak\n\t\t}\n\t\tpq.Swap(i, j)\n\t\ti = j\n\t}\n}\n" }, { "path": "nsqd/in_flight_pqueue_test.go", "content": "package nsqd\n\nimport (\n\t\"fmt\"\n\t\"math/rand\"\n\t\"sort\"\n\t\"testing\"\n\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\nfunc TestPriorityQueue(t *testing.T) {\n\tc := 100\n\tpq := newInFlightPqueue(c)\n\n\tfor i := 0; i < c+1; i++ {\n\t\tpq.Push(&Message{clientID: int64(i), pri: int64(i)})\n\t}\n\ttest.Equal(t, c+1, len(pq))\n\ttest.Equal(t, c*2, cap(pq))\n\n\tfor i := 0; i < c+1; i++ {\n\t\tmsg := pq.Pop()\n\t\ttest.Equal(t, int64(i), msg.clientID)\n\t}\n\ttest.Equal(t, c/4, cap(pq))\n}\n\nfunc TestUnsortedInsert(t *testing.T) {\n\tc := 100\n\tpq := newInFlightPqueue(c)\n\tints := make([]int, 0, c)\n\n\tfor i := 0; i < c; i++ {\n\t\tv := rand.Int()\n\t\tints = append(ints, v)\n\t\tpq.Push(&Message{pri: int64(v)})\n\t}\n\ttest.Equal(t, c, len(pq))\n\ttest.Equal(t, c, cap(pq))\n\n\tsort.Ints(ints)\n\n\tfor i := 0; i < c; i++ {\n\t\tmsg, _ := pq.PeekAndShift(int64(ints[len(ints)-1]))\n\t\ttest.Equal(t, int64(ints[i]), msg.pri)\n\t}\n}\n\nfunc TestRemove(t *testing.T) {\n\tc := 100\n\tpq := newInFlightPqueue(c)\n\n\tmsgs := make(map[MessageID]*Message)\n\tfor i := 0; i < c; i++ {\n\t\tm := &Message{pri: int64(rand.Intn(100000000))}\n\t\tcopy(m.ID[:], fmt.Sprintf(\"%016d\", m.pri))\n\t\tmsgs[m.ID] = m\n\t\tpq.Push(m)\n\t}\n\n\tfor i := 0; i < 10; i++ {\n\t\tidx := rand.Intn((c - 1) - i)\n\t\tvar fm *Message\n\t\tfor _, m := range msgs {\n\t\t\tif m.index == idx {\n\t\t\t\tfm = m\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t\trm := pq.Remove(idx)\n\t\ttest.Equal(t, fmt.Sprintf(\"%s\", fm.ID), fmt.Sprintf(\"%s\", rm.ID))\n\t}\n\n\tlastPriority := pq.Pop().pri\n\tfor i := 0; i < (c - 10 - 1); i++ {\n\t\tmsg := pq.Pop()\n\t\ttest.Equal(t, true, lastPriority <= msg.pri)\n\t\tlastPriority = msg.pri\n\t}\n}\n" }, { "path": "nsqd/logger.go", "content": "package nsqd\n\nimport (\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype Logger lg.Logger\n\nconst (\n\tLOG_DEBUG = lg.DEBUG\n\tLOG_INFO = lg.INFO\n\tLOG_WARN = lg.WARN\n\tLOG_ERROR = lg.ERROR\n\tLOG_FATAL = lg.FATAL\n)\n\nfunc (n *NSQD) logf(level lg.LogLevel, f string, args ...interface{}) {\n\topts := n.getOpts()\n\tlg.Logf(opts.Logger, opts.LogLevel, level, f, args...)\n}\n" }, { "path": "nsqd/lookup.go", "content": "package nsqd\n\nimport (\n\t\"bytes\"\n\t\"encoding/json\"\n\t\"net\"\n\t\"os\"\n\t\"strconv\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nfunc connectCallback(n *NSQD, hostname string) func(*lookupPeer) {\n\treturn func(lp *lookupPeer) {\n\t\tci := make(map[string]interface{})\n\t\tci[\"version\"] = version.Binary\n\t\tci[\"tcp_port\"] = n.getOpts().BroadcastTCPPort\n\t\tci[\"http_port\"] = n.getOpts().BroadcastHTTPPort\n\t\tci[\"hostname\"] = hostname\n\t\tci[\"broadcast_address\"] = n.getOpts().BroadcastAddress\n\t\tci[\"topology_zone\"] = n.getOpts().TopologyZone\n\t\tci[\"topology_region\"] = n.getOpts().TopologyRegion\n\n\t\tcmd, err := nsq.Identify(ci)\n\t\tif err != nil {\n\t\t\tlp.Close()\n\t\t\treturn\n\t\t}\n\n\t\tresp, err := lp.Command(cmd)\n\t\tif err != nil {\n\t\t\tn.logf(LOG_ERROR, \"LOOKUPD(%s): %s - %s\", lp, cmd, err)\n\t\t\treturn\n\t\t} else if bytes.Equal(resp, []byte(\"E_INVALID\")) {\n\t\t\tn.logf(LOG_INFO, \"LOOKUPD(%s): lookupd returned %s\", lp, resp)\n\t\t\tlp.Close()\n\t\t\treturn\n\t\t}\n\n\t\terr = json.Unmarshal(resp, &lp.Info)\n\t\tif err != nil {\n\t\t\tn.logf(LOG_ERROR, \"LOOKUPD(%s): parsing response - %s\", lp, resp)\n\t\t\tlp.Close()\n\t\t\treturn\n\t\t}\n\t\tn.logf(LOG_INFO, \"LOOKUPD(%s): peer info %+v\", lp, lp.Info)\n\t\tif lp.Info.BroadcastAddress == \"\" {\n\t\t\tn.logf(LOG_ERROR, \"LOOKUPD(%s): no broadcast address\", lp)\n\t\t}\n\n\t\t// build all the commands first so we exit the lock(s) as fast as possible\n\t\tvar commands []*nsq.Command\n\t\tn.RLock()\n\t\tfor _, topic := range n.topicMap {\n\t\t\ttopic.RLock()\n\t\t\tif len(topic.channelMap) == 0 {\n\t\t\t\tcommands = append(commands, nsq.Register(topic.name, \"\"))\n\t\t\t} else {\n\t\t\t\tfor _, channel := range topic.channelMap {\n\t\t\t\t\tcommands = append(commands, nsq.Register(channel.topicName, channel.name))\n\t\t\t\t}\n\t\t\t}\n\t\t\ttopic.RUnlock()\n\t\t}\n\t\tn.RUnlock()\n\n\t\tfor _, cmd := range commands {\n\t\t\tn.logf(LOG_INFO, \"LOOKUPD(%s): %s\", lp, cmd)\n\t\t\t_, err := lp.Command(cmd)\n\t\t\tif err != nil {\n\t\t\t\tn.logf(LOG_ERROR, \"LOOKUPD(%s): %s - %s\", lp, cmd, err)\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t}\n}\n\nfunc (n *NSQD) lookupLoop() {\n\tvar lookupPeers []*lookupPeer\n\tvar lookupAddrs []string\n\tconnect := true\n\n\thostname, err := os.Hostname()\n\tif err != nil {\n\t\tn.logf(LOG_FATAL, \"failed to get hostname - %s\", err)\n\t\tos.Exit(1)\n\t}\n\n\t// for announcements, lookupd determines the host automatically\n\tticker := time.NewTicker(15 * time.Second)\n\tdefer ticker.Stop()\n\tfor {\n\t\tif connect {\n\t\t\tfor _, host := range n.getOpts().NSQLookupdTCPAddresses {\n\t\t\t\tif in(host, lookupAddrs) {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tn.logf(LOG_INFO, \"LOOKUP(%s): adding peer\", host)\n\t\t\t\tlookupPeer := newLookupPeer(host, n.getOpts().MaxBodySize, n.logf,\n\t\t\t\t\tconnectCallback(n, hostname))\n\t\t\t\tlookupPeer.Command(nil) // start the connection\n\t\t\t\tlookupPeers = append(lookupPeers, lookupPeer)\n\t\t\t\tlookupAddrs = append(lookupAddrs, host)\n\t\t\t}\n\t\t\tn.lookupPeers.Store(lookupPeers)\n\t\t\tconnect = false\n\t\t}\n\n\t\tselect {\n\t\tcase <-ticker.C:\n\t\t\t// send a heartbeat and read a response (read detects closed conns)\n\t\t\tfor _, lookupPeer := range lookupPeers {\n\t\t\t\tn.logf(LOG_DEBUG, \"LOOKUPD(%s): sending heartbeat\", lookupPeer)\n\t\t\t\tcmd := nsq.Ping()\n\t\t\t\t_, err := lookupPeer.Command(cmd)\n\t\t\t\tif err != nil {\n\t\t\t\t\tn.logf(LOG_ERROR, \"LOOKUPD(%s): %s - %s\", lookupPeer, cmd, err)\n\t\t\t\t}\n\t\t\t}\n\t\tcase val := <-n.notifyChan:\n\t\t\tvar cmd *nsq.Command\n\t\t\tvar branch string\n\n\t\t\tswitch val := val.(type) {\n\t\t\tcase *Channel:\n\t\t\t\t// notify all nsqlookupds that a new channel exists, or that it's removed\n\t\t\t\tbranch = \"channel\"\n\t\t\t\tchannel := val\n\t\t\t\tif channel.Exiting() {\n\t\t\t\t\tcmd = nsq.UnRegister(channel.topicName, channel.name)\n\t\t\t\t} else {\n\t\t\t\t\tcmd = nsq.Register(channel.topicName, channel.name)\n\t\t\t\t}\n\t\t\tcase *Topic:\n\t\t\t\t// notify all nsqlookupds that a new topic exists, or that it's removed\n\t\t\t\tbranch = \"topic\"\n\t\t\t\ttopic := val\n\t\t\t\tif topic.Exiting() {\n\t\t\t\t\tcmd = nsq.UnRegister(topic.name, \"\")\n\t\t\t\t} else {\n\t\t\t\t\tcmd = nsq.Register(topic.name, \"\")\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tfor _, lookupPeer := range lookupPeers {\n\t\t\t\tn.logf(LOG_INFO, \"LOOKUPD(%s): %s %s\", lookupPeer, branch, cmd)\n\t\t\t\t_, err := lookupPeer.Command(cmd)\n\t\t\t\tif err != nil {\n\t\t\t\t\tn.logf(LOG_ERROR, \"LOOKUPD(%s): %s - %s\", lookupPeer, cmd, err)\n\t\t\t\t}\n\t\t\t}\n\t\tcase <-n.optsNotificationChan:\n\t\t\tvar tmpPeers []*lookupPeer\n\t\t\tvar tmpAddrs []string\n\t\t\tfor _, lp := range lookupPeers {\n\t\t\t\tif in(lp.addr, n.getOpts().NSQLookupdTCPAddresses) {\n\t\t\t\t\ttmpPeers = append(tmpPeers, lp)\n\t\t\t\t\ttmpAddrs = append(tmpAddrs, lp.addr)\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tn.logf(LOG_INFO, \"LOOKUP(%s): removing peer\", lp)\n\t\t\t\tlp.Close()\n\t\t\t}\n\t\t\tlookupPeers = tmpPeers\n\t\t\tlookupAddrs = tmpAddrs\n\t\t\tconnect = true\n\t\tcase <-n.exitChan:\n\t\t\tgoto exit\n\t\t}\n\t}\n\nexit:\n\tn.logf(LOG_INFO, \"LOOKUP: closing\")\n}\n\nfunc in(s string, lst []string) bool {\n\tfor _, v := range lst {\n\t\tif s == v {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc (n *NSQD) lookupdHTTPAddrs() []string {\n\tvar lookupHTTPAddrs []string\n\tlookupPeers := n.lookupPeers.Load()\n\tif lookupPeers == nil {\n\t\treturn nil\n\t}\n\tfor _, lp := range lookupPeers.([]*lookupPeer) {\n\t\tif len(lp.Info.BroadcastAddress) <= 0 {\n\t\t\tcontinue\n\t\t}\n\t\taddr := net.JoinHostPort(lp.Info.BroadcastAddress, strconv.Itoa(lp.Info.HTTPPort))\n\t\tlookupHTTPAddrs = append(lookupHTTPAddrs, addr)\n\t}\n\treturn lookupHTTPAddrs\n}\n" }, { "path": "nsqd/lookup_peer.go", "content": "package nsqd\n\nimport (\n\t\"encoding/binary\"\n\t\"fmt\"\n\t\"io\"\n\t\"net\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\n// lookupPeer is a low-level type for connecting/reading/writing to nsqlookupd\n//\n// A lookupPeer instance is designed to connect lazily to nsqlookupd and reconnect\n// gracefully (i.e. it is all handled by the library). Clients can simply use the\n// Command interface to perform a round-trip.\ntype lookupPeer struct {\n\tlogf lg.AppLogFunc\n\taddr string\n\tconn net.Conn\n\tstate int32\n\tconnectCallback func(*lookupPeer)\n\tmaxBodySize int64\n\tInfo peerInfo\n}\n\n// peerInfo contains metadata for a lookupPeer instance (and is JSON marshalable)\ntype peerInfo struct {\n\tTCPPort int `json:\"tcp_port\"`\n\tHTTPPort int `json:\"http_port\"`\n\tVersion string `json:\"version\"`\n\tBroadcastAddress string `json:\"broadcast_address\"`\n}\n\n// newLookupPeer creates a new lookupPeer instance connecting to the supplied address.\n//\n// The supplied connectCallback will be called *every* time the instance connects.\nfunc newLookupPeer(addr string, maxBodySize int64, l lg.AppLogFunc, connectCallback func(*lookupPeer)) *lookupPeer {\n\treturn &lookupPeer{\n\t\tlogf: l,\n\t\taddr: addr,\n\t\tstate: stateDisconnected,\n\t\tmaxBodySize: maxBodySize,\n\t\tconnectCallback: connectCallback,\n\t}\n}\n\n// Connect will Dial the specified address, with timeouts\nfunc (lp *lookupPeer) Connect() error {\n\tlp.logf(lg.INFO, \"LOOKUP connecting to %s\", lp.addr)\n\tconn, err := net.DialTimeout(\"tcp\", lp.addr, time.Second)\n\tif err != nil {\n\t\treturn err\n\t}\n\tlp.conn = conn\n\treturn nil\n}\n\n// String returns the specified address\nfunc (lp *lookupPeer) String() string {\n\treturn lp.addr\n}\n\n// Read implements the io.Reader interface, adding deadlines\nfunc (lp *lookupPeer) Read(data []byte) (int, error) {\n\tlp.conn.SetReadDeadline(time.Now().Add(time.Second))\n\treturn lp.conn.Read(data)\n}\n\n// Write implements the io.Writer interface, adding deadlines\nfunc (lp *lookupPeer) Write(data []byte) (int, error) {\n\tlp.conn.SetWriteDeadline(time.Now().Add(time.Second))\n\treturn lp.conn.Write(data)\n}\n\n// Close implements the io.Closer interface\nfunc (lp *lookupPeer) Close() error {\n\tlp.state = stateDisconnected\n\tif lp.conn != nil {\n\t\treturn lp.conn.Close()\n\t}\n\treturn nil\n}\n\n// Command performs a round-trip for the specified Command.\n//\n// It will lazily connect to nsqlookupd and gracefully handle\n// reconnecting in the event of a failure.\n//\n// It returns the response from nsqlookupd as []byte\nfunc (lp *lookupPeer) Command(cmd *nsq.Command) ([]byte, error) {\n\tinitialState := lp.state\n\tif lp.state != stateConnected {\n\t\terr := lp.Connect()\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\tlp.state = stateConnected\n\t\t_, err = lp.Write(nsq.MagicV1)\n\t\tif err != nil {\n\t\t\tlp.Close()\n\t\t\treturn nil, err\n\t\t}\n\t\tif initialState == stateDisconnected {\n\t\t\tlp.connectCallback(lp)\n\t\t}\n\t\tif lp.state != stateConnected {\n\t\t\treturn nil, fmt.Errorf(\"lookupPeer connectCallback() failed\")\n\t\t}\n\t}\n\tif cmd == nil {\n\t\treturn nil, nil\n\t}\n\t_, err := cmd.WriteTo(lp)\n\tif err != nil {\n\t\tlp.Close()\n\t\treturn nil, err\n\t}\n\tresp, err := readResponseBounded(lp, lp.maxBodySize)\n\tif err != nil {\n\t\tlp.Close()\n\t\treturn nil, err\n\t}\n\treturn resp, nil\n}\n\nfunc readResponseBounded(r io.Reader, limit int64) ([]byte, error) {\n\tvar msgSize int32\n\n\t// message size\n\terr := binary.Read(r, binary.BigEndian, &msgSize)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif int64(msgSize) > limit {\n\t\treturn nil, fmt.Errorf(\"response body size (%d) is greater than limit (%d)\",\n\t\t\tmsgSize, limit)\n\t}\n\n\t// message binary data\n\tbuf := make([]byte, msgSize)\n\t_, err = io.ReadFull(r, buf)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn buf, nil\n}\n" }, { "path": "nsqd/message.go", "content": "package nsqd\n\nimport (\n\t\"encoding/binary\"\n\t\"fmt\"\n\t\"io\"\n\t\"time\"\n)\n\nconst (\n\tMsgIDLength = 16\n\tminValidMsgLength = MsgIDLength + 8 + 2 // Timestamp + Attempts\n)\n\ntype MessageID [MsgIDLength]byte\n\ntype Message struct {\n\tID MessageID\n\tBody []byte\n\tTimestamp int64\n\tAttempts uint16\n\n\t// for in-flight handling\n\tdeliveryTS time.Time\n\tclientID int64\n\tpri int64\n\tindex int\n\tdeferred time.Duration\n}\n\nfunc NewMessage(id MessageID, body []byte) *Message {\n\treturn &Message{\n\t\tID: id,\n\t\tBody: body,\n\t\tTimestamp: time.Now().UnixNano(),\n\t}\n}\n\nfunc (m *Message) WriteTo(w io.Writer) (int64, error) {\n\tvar buf [10]byte\n\tvar total int64\n\n\tbinary.BigEndian.PutUint64(buf[:8], uint64(m.Timestamp))\n\tbinary.BigEndian.PutUint16(buf[8:10], uint16(m.Attempts))\n\n\tn, err := w.Write(buf[:])\n\ttotal += int64(n)\n\tif err != nil {\n\t\treturn total, err\n\t}\n\n\tn, err = w.Write(m.ID[:])\n\ttotal += int64(n)\n\tif err != nil {\n\t\treturn total, err\n\t}\n\n\tn, err = w.Write(m.Body)\n\ttotal += int64(n)\n\tif err != nil {\n\t\treturn total, err\n\t}\n\n\treturn total, nil\n}\n\n// decodeMessage deserializes data (as []byte) and creates a new Message\n//\n//\t[x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x][x]...\n//\t| (int64) || || (hex string encoded in ASCII) || (binary)\n//\t| 8-byte || || 16-byte || N-byte\n//\t------------------------------------------------------------------------------------------...\n//\t nanosecond timestamp ^^ message ID message body\n//\t (uint16)\n//\t 2-byte\n//\t attempts\nfunc decodeMessage(b []byte) (*Message, error) {\n\tvar msg Message\n\n\tif len(b) < minValidMsgLength {\n\t\treturn nil, fmt.Errorf(\"invalid message buffer size (%d)\", len(b))\n\t}\n\n\tmsg.Timestamp = int64(binary.BigEndian.Uint64(b[:8]))\n\tmsg.Attempts = binary.BigEndian.Uint16(b[8:10])\n\tcopy(msg.ID[:], b[10:10+MsgIDLength])\n\tmsg.Body = b[10+MsgIDLength:]\n\n\treturn &msg, nil\n}\n\nfunc writeMessageToBackend(msg *Message, bq BackendQueue) error {\n\tbuf := bufferPoolGet()\n\tdefer bufferPoolPut(buf)\n\t_, err := msg.WriteTo(buf)\n\tif err != nil {\n\t\treturn err\n\t}\n\treturn bq.Put(buf.Bytes())\n}\n" }, { "path": "nsqd/nsqd.go", "content": "package nsqd\n\nimport (\n\t\"context\"\n\t\"crypto/tls\"\n\t\"crypto/x509\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"log\"\n\t\"math/rand\"\n\t\"net\"\n\t\"os\"\n\t\"path\"\n\t\"strings\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/clusterinfo\"\n\t\"github.com/nsqio/nsq/internal/dirlock\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/statsd\"\n\t\"github.com/nsqio/nsq/internal/util\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nconst (\n\tTLSNotRequired = iota\n\tTLSRequiredExceptHTTP\n\tTLSRequired\n)\n\ntype errStore struct {\n\terr error\n}\n\ntype NSQD struct {\n\t// 64bit atomic vars need to be first for proper alignment on 32bit platforms\n\tclientIDSequence int64\n\n\tsync.RWMutex\n\tctx context.Context\n\t// ctxCancel cancels a context that main() is waiting on\n\tctxCancel context.CancelFunc\n\n\topts atomic.Value\n\n\tdl *dirlock.DirLock\n\tisLoading int32\n\tisExiting int32\n\terrValue atomic.Value\n\tstartTime time.Time\n\n\ttopicMap map[string]*Topic\n\n\tlookupPeers atomic.Value\n\n\ttcpServer *tcpServer\n\ttcpListener net.Listener\n\thttpListener net.Listener\n\thttpsListener net.Listener\n\ttlsConfig *tls.Config\n\tclientTLSConfig *tls.Config\n\n\tpoolSize int\n\n\tnotifyChan chan interface{}\n\toptsNotificationChan chan struct{}\n\texitChan chan int\n\twaitGroup util.WaitGroupWrapper\n\n\tci *clusterinfo.ClusterInfo\n}\n\nfunc New(opts *Options) (*NSQD, error) {\n\tvar err error\n\n\tdataPath := opts.DataPath\n\tif opts.DataPath == \"\" {\n\t\tcwd, _ := os.Getwd()\n\t\tdataPath = cwd\n\t}\n\tif opts.Logger == nil {\n\t\topts.Logger = log.New(os.Stderr, opts.LogPrefix, log.Ldate|log.Ltime|log.Lmicroseconds)\n\t}\n\n\tn := &NSQD{\n\t\tstartTime: time.Now(),\n\t\ttopicMap: make(map[string]*Topic),\n\t\texitChan: make(chan int),\n\t\tnotifyChan: make(chan interface{}),\n\t\toptsNotificationChan: make(chan struct{}, 1),\n\t\tdl: dirlock.New(dataPath),\n\t}\n\tn.ctx, n.ctxCancel = context.WithCancel(context.Background())\n\thttpcli := http_api.NewClient(nil, opts.HTTPClientConnectTimeout, opts.HTTPClientRequestTimeout)\n\tn.ci = clusterinfo.New(n.logf, httpcli)\n\n\tn.lookupPeers.Store([]*lookupPeer{})\n\n\tn.swapOpts(opts)\n\tn.errValue.Store(errStore{})\n\n\terr = n.dl.Lock()\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to lock data-path: %v\", err)\n\t}\n\n\tif opts.MaxDeflateLevel < 1 || opts.MaxDeflateLevel > 9 {\n\t\treturn nil, errors.New(\"--max-deflate-level must be [1,9]\")\n\t}\n\n\tif opts.ID < 0 || opts.ID >= 1024 {\n\t\treturn nil, errors.New(\"--node-id must be [0,1024)\")\n\t}\n\n\tif opts.TLSClientAuthPolicy != \"\" && opts.TLSRequired == TLSNotRequired {\n\t\topts.TLSRequired = TLSRequired\n\t}\n\n\ttlsConfig, err := buildTLSConfig(opts)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to build TLS config - %s\", err)\n\t}\n\tif tlsConfig == nil && opts.TLSRequired != TLSNotRequired {\n\t\treturn nil, errors.New(\"cannot require TLS client connections without TLS key and cert\")\n\t}\n\tn.tlsConfig = tlsConfig\n\n\tclientTLSConfig, err := buildClientTLSConfig(opts)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"failed to build client TLS config - %s\", err)\n\t}\n\tn.clientTLSConfig = clientTLSConfig\n\n\tif opts.AuthHTTPRequestMethod != \"post\" && opts.AuthHTTPRequestMethod != \"get\" {\n\t\treturn nil, errors.New(\"--auth-http-request-method must be post or get\")\n\t}\n\n\tfor _, v := range opts.E2EProcessingLatencyPercentiles {\n\t\tif v <= 0 || v > 1 {\n\t\t\treturn nil, fmt.Errorf(\"invalid E2E processing latency percentile: %v\", v)\n\t\t}\n\t}\n\n\tn.logf(LOG_INFO, version.String(\"nsqd\"))\n\tn.logf(LOG_INFO, \"ID: %d\", opts.ID)\n\n\tn.tcpServer = &tcpServer{nsqd: n}\n\tn.tcpListener, err = net.Listen(util.TypeOfAddr(opts.TCPAddress), opts.TCPAddress)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"listen (%s) failed - %s\", opts.TCPAddress, err)\n\t}\n\tif opts.HTTPAddress != \"\" {\n\t\tn.httpListener, err = net.Listen(util.TypeOfAddr(opts.HTTPAddress), opts.HTTPAddress)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"listen (%s) failed - %s\", opts.HTTPAddress, err)\n\t\t}\n\t}\n\tif n.tlsConfig != nil && opts.HTTPSAddress != \"\" {\n\t\tn.httpsListener, err = tls.Listen(\"tcp\", opts.HTTPSAddress, n.tlsConfig)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"listen (%s) failed - %s\", opts.HTTPSAddress, err)\n\t\t}\n\t}\n\tif opts.BroadcastHTTPPort == 0 {\n\t\ttcpAddr, ok := n.RealHTTPAddr().(*net.TCPAddr)\n\t\tif ok {\n\t\t\topts.BroadcastHTTPPort = tcpAddr.Port\n\t\t}\n\t}\n\n\tif opts.BroadcastTCPPort == 0 {\n\t\ttcpAddr, ok := n.RealTCPAddr().(*net.TCPAddr)\n\t\tif ok {\n\t\t\topts.BroadcastTCPPort = tcpAddr.Port\n\t\t}\n\t}\n\n\tif opts.StatsdPrefix != \"\" {\n\t\tvar port string = fmt.Sprint(opts.BroadcastHTTPPort)\n\t\tstatsdHostKey := statsd.HostKey(net.JoinHostPort(opts.BroadcastAddress, port))\n\t\tprefixWithHost := strings.Replace(opts.StatsdPrefix, \"%s\", statsdHostKey, -1)\n\t\tif prefixWithHost[len(prefixWithHost)-1] != '.' {\n\t\t\tprefixWithHost += \".\"\n\t\t}\n\t\topts.StatsdPrefix = prefixWithHost\n\t}\n\n\treturn n, nil\n}\n\nfunc (n *NSQD) getOpts() *Options {\n\treturn n.opts.Load().(*Options)\n}\n\nfunc (n *NSQD) swapOpts(opts *Options) {\n\tn.opts.Store(opts)\n}\n\nfunc (n *NSQD) triggerOptsNotification() {\n\tselect {\n\tcase n.optsNotificationChan <- struct{}{}:\n\tdefault:\n\t}\n}\n\nfunc (n *NSQD) RealTCPAddr() net.Addr {\n\tif n.tcpListener == nil {\n\t\treturn &net.TCPAddr{}\n\t}\n\treturn n.tcpListener.Addr()\n\n}\n\nfunc (n *NSQD) RealHTTPAddr() net.Addr {\n\tif n.httpListener == nil {\n\t\treturn &net.TCPAddr{}\n\t}\n\treturn n.httpListener.Addr()\n}\n\nfunc (n *NSQD) RealHTTPSAddr() *net.TCPAddr {\n\tif n.httpsListener == nil {\n\t\treturn &net.TCPAddr{}\n\t}\n\treturn n.httpsListener.Addr().(*net.TCPAddr)\n}\n\nfunc (n *NSQD) SetHealth(err error) {\n\tn.errValue.Store(errStore{err: err})\n}\n\nfunc (n *NSQD) IsHealthy() bool {\n\treturn n.GetError() == nil\n}\n\nfunc (n *NSQD) GetError() error {\n\terrValue := n.errValue.Load()\n\treturn errValue.(errStore).err\n}\n\nfunc (n *NSQD) GetHealth() string {\n\terr := n.GetError()\n\tif err != nil {\n\t\treturn fmt.Sprintf(\"NOK - %s\", err)\n\t}\n\treturn \"OK\"\n}\n\nfunc (n *NSQD) GetStartTime() time.Time {\n\treturn n.startTime\n}\n\nfunc (n *NSQD) Main() error {\n\texitCh := make(chan error)\n\tvar once sync.Once\n\texitFunc := func(err error) {\n\t\tonce.Do(func() {\n\t\t\tif err != nil {\n\t\t\t\tn.logf(LOG_FATAL, \"%s\", err)\n\t\t\t}\n\t\t\texitCh <- err\n\t\t})\n\t}\n\n\tn.waitGroup.Wrap(func() {\n\t\texitFunc(protocol.TCPServer(n.tcpListener, n.tcpServer, n.logf))\n\t})\n\tif n.httpListener != nil {\n\t\thttpServer := newHTTPServer(n, false, n.getOpts().TLSRequired == TLSRequired)\n\t\tn.waitGroup.Wrap(func() {\n\t\t\texitFunc(http_api.Serve(n.httpListener, httpServer, \"HTTP\", n.logf))\n\t\t})\n\t}\n\tif n.httpsListener != nil {\n\t\thttpsServer := newHTTPServer(n, true, true)\n\t\tn.waitGroup.Wrap(func() {\n\t\t\texitFunc(http_api.Serve(n.httpsListener, httpsServer, \"HTTPS\", n.logf))\n\t\t})\n\t}\n\n\tn.waitGroup.Wrap(n.queueScanLoop)\n\tn.waitGroup.Wrap(n.lookupLoop)\n\tif n.getOpts().StatsdAddress != \"\" {\n\t\tn.waitGroup.Wrap(n.statsdLoop)\n\t}\n\n\terr := <-exitCh\n\treturn err\n}\n\n// Metadata is the collection of persistent information about the current NSQD.\ntype Metadata struct {\n\tTopics []TopicMetadata `json:\"topics\"`\n\tVersion string `json:\"version\"`\n}\n\n// TopicMetadata is the collection of persistent information about a topic.\ntype TopicMetadata struct {\n\tName string `json:\"name\"`\n\tPaused bool `json:\"paused\"`\n\tChannels []ChannelMetadata `json:\"channels\"`\n}\n\n// ChannelMetadata is the collection of persistent information about a channel.\ntype ChannelMetadata struct {\n\tName string `json:\"name\"`\n\tPaused bool `json:\"paused\"`\n}\n\nfunc newMetadataFile(opts *Options) string {\n\treturn path.Join(opts.DataPath, \"nsqd.dat\")\n}\n\nfunc readOrEmpty(fn string) ([]byte, error) {\n\tdata, err := os.ReadFile(fn)\n\tif err != nil {\n\t\tif !os.IsNotExist(err) {\n\t\t\treturn nil, fmt.Errorf(\"failed to read metadata from %s - %s\", fn, err)\n\t\t}\n\t}\n\treturn data, nil\n}\n\nfunc writeSyncFile(fn string, data []byte) error {\n\tf, err := os.OpenFile(fn, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\t_, err = f.Write(data)\n\tif err == nil {\n\t\terr = f.Sync()\n\t}\n\tf.Close()\n\treturn err\n}\n\nfunc (n *NSQD) LoadMetadata() error {\n\tatomic.StoreInt32(&n.isLoading, 1)\n\tdefer atomic.StoreInt32(&n.isLoading, 0)\n\n\tfn := newMetadataFile(n.getOpts())\n\n\tdata, err := readOrEmpty(fn)\n\tif err != nil {\n\t\treturn err\n\t}\n\tif data == nil {\n\t\treturn nil // fresh start\n\t}\n\n\tvar m Metadata\n\terr = json.Unmarshal(data, &m)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to parse metadata in %s - %s\", fn, err)\n\t}\n\n\tfor _, t := range m.Topics {\n\t\tif !protocol.IsValidTopicName(t.Name) {\n\t\t\tn.logf(LOG_WARN, \"skipping creation of invalid topic %s\", t.Name)\n\t\t\tcontinue\n\t\t}\n\t\ttopic := n.GetTopic(t.Name)\n\t\tif t.Paused {\n\t\t\ttopic.Pause()\n\t\t}\n\t\tfor _, c := range t.Channels {\n\t\t\tif !protocol.IsValidChannelName(c.Name) {\n\t\t\t\tn.logf(LOG_WARN, \"skipping creation of invalid channel %s\", c.Name)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tchannel := topic.GetChannel(c.Name)\n\t\t\tif c.Paused {\n\t\t\t\tchannel.Pause()\n\t\t\t}\n\t\t}\n\t\ttopic.Start()\n\t}\n\treturn nil\n}\n\n// GetMetadata retrieves the current topic and channel set of the NSQ daemon. If\n// the ephemeral flag is set, ephemeral topics are also returned even though these\n// are not saved to disk.\nfunc (n *NSQD) GetMetadata(ephemeral bool) *Metadata {\n\tmeta := &Metadata{\n\t\tVersion: version.Binary,\n\t}\n\tfor _, topic := range n.topicMap {\n\t\tif topic.ephemeral && !ephemeral {\n\t\t\tcontinue\n\t\t}\n\t\ttopicData := TopicMetadata{\n\t\t\tName: topic.name,\n\t\t\tPaused: topic.IsPaused(),\n\t\t}\n\t\ttopic.Lock()\n\t\tfor _, channel := range topic.channelMap {\n\t\t\tif channel.ephemeral {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\ttopicData.Channels = append(topicData.Channels, ChannelMetadata{\n\t\t\t\tName: channel.name,\n\t\t\t\tPaused: channel.IsPaused(),\n\t\t\t})\n\t\t}\n\t\ttopic.Unlock()\n\t\tmeta.Topics = append(meta.Topics, topicData)\n\t}\n\treturn meta\n}\n\nfunc (n *NSQD) PersistMetadata() error {\n\t// persist metadata about what topics/channels we have, across restarts\n\tfileName := newMetadataFile(n.getOpts())\n\n\tn.logf(LOG_INFO, \"NSQ: persisting topic/channel metadata to %s\", fileName)\n\n\tdata, err := json.Marshal(n.GetMetadata(false))\n\tif err != nil {\n\t\treturn err\n\t}\n\ttmpFileName := fmt.Sprintf(\"%s.%d.tmp\", fileName, rand.Int())\n\n\terr = writeSyncFile(tmpFileName, data)\n\tif err != nil {\n\t\treturn err\n\t}\n\terr = os.Rename(tmpFileName, fileName)\n\tif err != nil {\n\t\treturn err\n\t}\n\t// technically should fsync DataPath here\n\n\treturn nil\n}\n\nfunc (n *NSQD) Exit() {\n\tif !atomic.CompareAndSwapInt32(&n.isExiting, 0, 1) {\n\t\t// avoid double call\n\t\treturn\n\t}\n\tif n.tcpListener != nil {\n\t\tn.tcpListener.Close()\n\t}\n\n\tif n.tcpServer != nil {\n\t\tn.tcpServer.Close()\n\t}\n\n\tif n.httpListener != nil {\n\t\tn.httpListener.Close()\n\t}\n\n\tif n.httpsListener != nil {\n\t\tn.httpsListener.Close()\n\t}\n\n\tn.Lock()\n\terr := n.PersistMetadata()\n\tif err != nil {\n\t\tn.logf(LOG_ERROR, \"failed to persist metadata - %s\", err)\n\t}\n\tn.logf(LOG_INFO, \"NSQ: closing topics\")\n\tfor _, topic := range n.topicMap {\n\t\ttopic.Close()\n\t}\n\tn.Unlock()\n\n\tn.logf(LOG_INFO, \"NSQ: stopping subsystems\")\n\tclose(n.exitChan)\n\tn.waitGroup.Wait()\n\tn.dl.Unlock()\n\tn.logf(LOG_INFO, \"NSQ: bye\")\n\tn.ctxCancel()\n}\n\n// GetTopic performs a thread safe operation\n// to return a pointer to a Topic object (potentially new)\nfunc (n *NSQD) GetTopic(topicName string) *Topic {\n\t// most likely we already have this topic, so try read lock first\n\tn.RLock()\n\tt, ok := n.topicMap[topicName]\n\tn.RUnlock()\n\tif ok {\n\t\treturn t\n\t}\n\n\tn.Lock()\n\n\tt, ok = n.topicMap[topicName]\n\tif ok {\n\t\tn.Unlock()\n\t\treturn t\n\t}\n\tdeleteCallback := func(t *Topic) {\n\t\tn.DeleteExistingTopic(t.name)\n\t}\n\tt = NewTopic(topicName, n, deleteCallback)\n\tn.topicMap[topicName] = t\n\n\tn.Unlock()\n\n\tn.logf(LOG_INFO, \"TOPIC(%s): created\", t.name)\n\t// topic is created but messagePump not yet started\n\n\t// if this topic was created while loading metadata at startup don't do any further initialization\n\t// (topic will be \"started\" after loading completes)\n\tif atomic.LoadInt32(&n.isLoading) == 1 {\n\t\treturn t\n\t}\n\n\t// if using lookupd, make a blocking call to get channels and immediately create them\n\t// to ensure that all channels receive published messages\n\tlookupdHTTPAddrs := n.lookupdHTTPAddrs()\n\tif len(lookupdHTTPAddrs) > 0 {\n\t\tchannelNames, err := n.ci.GetLookupdTopicChannels(t.name, lookupdHTTPAddrs)\n\t\tif err != nil {\n\t\t\tn.logf(LOG_WARN, \"failed to query nsqlookupd for channels to pre-create for topic %s - %s\", t.name, err)\n\t\t}\n\t\tfor _, channelName := range channelNames {\n\t\t\tif strings.HasSuffix(channelName, \"#ephemeral\") {\n\t\t\t\tcontinue // do not create ephemeral channel with no consumer client\n\t\t\t}\n\t\t\tt.GetChannel(channelName)\n\t\t}\n\t} else if len(n.getOpts().NSQLookupdTCPAddresses) > 0 {\n\t\tn.logf(LOG_ERROR, \"no available nsqlookupd to query for channels to pre-create for topic %s\", t.name)\n\t}\n\n\t// now that all channels are added, start topic messagePump\n\tt.Start()\n\treturn t\n}\n\n// GetExistingTopic gets a topic only if it exists\nfunc (n *NSQD) GetExistingTopic(topicName string) (*Topic, error) {\n\tn.RLock()\n\tdefer n.RUnlock()\n\ttopic, ok := n.topicMap[topicName]\n\tif !ok {\n\t\treturn nil, errors.New(\"topic does not exist\")\n\t}\n\treturn topic, nil\n}\n\n// DeleteExistingTopic removes a topic only if it exists\nfunc (n *NSQD) DeleteExistingTopic(topicName string) error {\n\tn.RLock()\n\ttopic, ok := n.topicMap[topicName]\n\tif !ok {\n\t\tn.RUnlock()\n\t\treturn errors.New(\"topic does not exist\")\n\t}\n\tn.RUnlock()\n\n\t// delete empties all channels and the topic itself before closing\n\t// (so that we dont leave any messages around)\n\t//\n\t// we do this before removing the topic from map below (with no lock)\n\t// so that any incoming writes will error and not create a new topic\n\t// to enforce ordering\n\ttopic.Delete()\n\n\tn.Lock()\n\tdelete(n.topicMap, topicName)\n\tn.Unlock()\n\n\treturn nil\n}\n\nfunc (n *NSQD) Notify(v interface{}, persist bool) {\n\t// since the in-memory metadata is incomplete,\n\t// should not persist metadata while loading it.\n\t// nsqd will call `PersistMetadata` it after loading\n\tloading := atomic.LoadInt32(&n.isLoading) == 1\n\tn.waitGroup.Wrap(func() {\n\t\t// by selecting on exitChan we guarantee that\n\t\t// we do not block exit, see issue #123\n\t\tselect {\n\t\tcase <-n.exitChan:\n\t\tcase n.notifyChan <- v:\n\t\t\tif loading || !persist {\n\t\t\t\treturn\n\t\t\t}\n\t\t\tn.Lock()\n\t\t\terr := n.PersistMetadata()\n\t\t\tif err != nil {\n\t\t\t\tn.logf(LOG_ERROR, \"failed to persist metadata - %s\", err)\n\t\t\t}\n\t\t\tn.Unlock()\n\t\t}\n\t})\n}\n\n// channels returns a flat slice of all channels in all topics\nfunc (n *NSQD) channels() []*Channel {\n\tvar channels []*Channel\n\tn.RLock()\n\tfor _, t := range n.topicMap {\n\t\tt.RLock()\n\t\tfor _, c := range t.channelMap {\n\t\t\tchannels = append(channels, c)\n\t\t}\n\t\tt.RUnlock()\n\t}\n\tn.RUnlock()\n\treturn channels\n}\n\n// resizePool adjusts the size of the pool of queueScanWorker goroutines\n//\n//\t1 <= pool <= min(num * 0.25, QueueScanWorkerPoolMax)\nfunc (n *NSQD) resizePool(num int, workCh chan *Channel, responseCh chan bool, closeCh chan int) {\n\tidealPoolSize := int(float64(num) * 0.25)\n\tif idealPoolSize < 1 {\n\t\tidealPoolSize = 1\n\t} else if idealPoolSize > n.getOpts().QueueScanWorkerPoolMax {\n\t\tidealPoolSize = n.getOpts().QueueScanWorkerPoolMax\n\t}\n\tfor {\n\t\tif idealPoolSize == n.poolSize {\n\t\t\tbreak\n\t\t} else if idealPoolSize < n.poolSize {\n\t\t\t// contract\n\t\t\tcloseCh <- 1\n\t\t\tn.poolSize--\n\t\t} else {\n\t\t\t// expand\n\t\t\tn.waitGroup.Wrap(func() {\n\t\t\t\tn.queueScanWorker(workCh, responseCh, closeCh)\n\t\t\t})\n\t\t\tn.poolSize++\n\t\t}\n\t}\n}\n\n// queueScanWorker receives work (in the form of a channel) from queueScanLoop\n// and processes the deferred and in-flight queues\nfunc (n *NSQD) queueScanWorker(workCh chan *Channel, responseCh chan bool, closeCh chan int) {\n\tfor {\n\t\tselect {\n\t\tcase c := <-workCh:\n\t\t\tnow := time.Now().UnixNano()\n\t\t\tdirty := false\n\t\t\tif c.processInFlightQueue(now) {\n\t\t\t\tdirty = true\n\t\t\t}\n\t\t\tif c.processDeferredQueue(now) {\n\t\t\t\tdirty = true\n\t\t\t}\n\t\t\tresponseCh <- dirty\n\t\tcase <-closeCh:\n\t\t\treturn\n\t\t}\n\t}\n}\n\n// queueScanLoop runs in a single goroutine to process in-flight and deferred\n// priority queues. It manages a pool of queueScanWorker (configurable max of\n// QueueScanWorkerPoolMax (default: 4)) that process channels concurrently.\n//\n// It copies Redis's probabilistic expiration algorithm: it wakes up every\n// QueueScanInterval (default: 100ms) to select a random QueueScanSelectionCount\n// (default: 20) channels from a locally cached list (refreshed every\n// QueueScanRefreshInterval (default: 5s)).\n//\n// If either of the queues had work to do the channel is considered \"dirty\".\n//\n// If QueueScanDirtyPercent (default: 25%) of the selected channels were dirty,\n// the loop continues without sleep.\nfunc (n *NSQD) queueScanLoop() {\n\tworkCh := make(chan *Channel, n.getOpts().QueueScanSelectionCount)\n\tresponseCh := make(chan bool, n.getOpts().QueueScanSelectionCount)\n\tcloseCh := make(chan int)\n\n\tworkTicker := time.NewTicker(n.getOpts().QueueScanInterval)\n\trefreshTicker := time.NewTicker(n.getOpts().QueueScanRefreshInterval)\n\n\tchannels := n.channels()\n\tn.resizePool(len(channels), workCh, responseCh, closeCh)\n\n\tfor {\n\t\tselect {\n\t\tcase <-workTicker.C:\n\t\t\tif len(channels) == 0 {\n\t\t\t\tcontinue\n\t\t\t}\n\t\tcase <-refreshTicker.C:\n\t\t\tchannels = n.channels()\n\t\t\tn.resizePool(len(channels), workCh, responseCh, closeCh)\n\t\t\tcontinue\n\t\tcase <-n.exitChan:\n\t\t\tgoto exit\n\t\t}\n\n\t\tnum := n.getOpts().QueueScanSelectionCount\n\t\tif num > len(channels) {\n\t\t\tnum = len(channels)\n\t\t}\n\n\tloop:\n\t\tfor _, i := range util.UniqRands(num, len(channels)) {\n\t\t\tworkCh <- channels[i]\n\t\t}\n\n\t\tnumDirty := 0\n\t\tfor i := 0; i < num; i++ {\n\t\t\tif <-responseCh {\n\t\t\t\tnumDirty++\n\t\t\t}\n\t\t}\n\n\t\tif float64(numDirty)/float64(num) > n.getOpts().QueueScanDirtyPercent {\n\t\t\tgoto loop\n\t\t}\n\t}\n\nexit:\n\tn.logf(LOG_INFO, \"QUEUESCAN: closing\")\n\tclose(closeCh)\n\tworkTicker.Stop()\n\trefreshTicker.Stop()\n}\n\nfunc buildTLSConfig(opts *Options) (*tls.Config, error) {\n\tvar tlsConfig *tls.Config\n\n\tif opts.TLSCert == \"\" && opts.TLSKey == \"\" {\n\t\treturn nil, nil\n\t}\n\n\ttlsClientAuthPolicy := tls.VerifyClientCertIfGiven\n\n\tcert, err := tls.LoadX509KeyPair(opts.TLSCert, opts.TLSKey)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tswitch opts.TLSClientAuthPolicy {\n\tcase \"require\":\n\t\ttlsClientAuthPolicy = tls.RequireAnyClientCert\n\tcase \"require-verify\":\n\t\ttlsClientAuthPolicy = tls.RequireAndVerifyClientCert\n\tdefault:\n\t\ttlsClientAuthPolicy = tls.NoClientCert\n\t}\n\n\ttlsConfig = &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tClientAuth: tlsClientAuthPolicy,\n\t\tMinVersion: opts.TLSMinVersion,\n\t}\n\n\tif opts.TLSRootCAFile != \"\" {\n\t\ttlsCertPool := x509.NewCertPool()\n\t\tcaCertFile, err := os.ReadFile(opts.TLSRootCAFile)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\tif !tlsCertPool.AppendCertsFromPEM(caCertFile) {\n\t\t\treturn nil, errors.New(\"failed to append certificate to pool\")\n\t\t}\n\t\ttlsConfig.ClientCAs = tlsCertPool\n\t}\n\n\treturn tlsConfig, nil\n}\n\nfunc buildClientTLSConfig(opts *Options) (*tls.Config, error) {\n\ttlsConfig := &tls.Config{\n\t\tMinVersion: opts.TLSMinVersion,\n\t}\n\n\tif opts.TLSRootCAFile != \"\" {\n\t\ttlsCertPool := x509.NewCertPool()\n\t\tcaCertFile, err := os.ReadFile(opts.TLSRootCAFile)\n\t\tif err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t\tif !tlsCertPool.AppendCertsFromPEM(caCertFile) {\n\t\t\treturn nil, errors.New(\"failed to append certificate to pool\")\n\t\t}\n\t\ttlsConfig.RootCAs = tlsCertPool\n\t}\n\n\treturn tlsConfig, nil\n}\n\nfunc (n *NSQD) IsAuthEnabled() bool {\n\treturn len(n.getOpts().AuthHTTPAddresses) != 0\n}\n\n// Context returns a context that will be canceled when nsqd initiates the shutdown\nfunc (n *NSQD) Context() context.Context {\n\treturn n.ctx\n}\n" }, { "path": "nsqd/nsqd_test.go", "content": "package nsqd\n\nimport (\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io/fs\"\n\t\"net\"\n\t\"os\"\n\t\"strconv\"\n\t\"sync/atomic\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/test\"\n\t\"github.com/nsqio/nsq/nsqlookupd\"\n)\n\nconst (\n\tConnectTimeout = 2 * time.Second\n\tRequestTimeout = 5 * time.Second\n)\n\nfunc getMetadata(n *NSQD) (*Metadata, error) {\n\tfn := newMetadataFile(n.getOpts())\n\tdata, err := os.ReadFile(fn)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tvar m Metadata\n\terr = json.Unmarshal(data, &m)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn &m, nil\n}\n\nfunc TestStartup(t *testing.T) {\n\tvar msg *Message\n\n\titerations := 300\n\tdoneExitChan := make(chan int)\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MemQueueSize = 100\n\topts.MaxBytesPerFile = 10240\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\n\torigDataPath := opts.DataPath\n\n\ttopicName := \"nsqd_test\" + strconv.Itoa(int(time.Now().Unix()))\n\n\texitChan := make(chan int)\n\tgo func() {\n\t\t<-exitChan\n\t\tnsqd.Exit()\n\t\tdoneExitChan <- 1\n\t}()\n\n\t// verify nsqd metadata shows no topics\n\terr := nsqd.PersistMetadata()\n\ttest.Nil(t, err)\n\tatomic.StoreInt32(&nsqd.isLoading, 1)\n\tnsqd.GetTopic(topicName) // will not persist if `flagLoading`\n\tm, err := getMetadata(nsqd)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 0, len(m.Topics))\n\tnsqd.DeleteExistingTopic(topicName)\n\tatomic.StoreInt32(&nsqd.isLoading, 0)\n\n\tbody := make([]byte, 256)\n\ttopic := nsqd.GetTopic(topicName)\n\tfor i := 0; i < iterations; i++ {\n\t\tmsg := NewMessage(topic.GenerateID(), body)\n\t\ttopic.PutMessage(msg)\n\t}\n\n\tt.Logf(\"pulling from channel\")\n\tchannel1 := topic.GetChannel(\"ch1\")\n\n\tt.Logf(\"read %d msgs\", iterations/2)\n\tfor i := 0; i < iterations/2; i++ {\n\t\tselect {\n\t\tcase msg = <-channel1.memoryMsgChan:\n\t\tcase b := <-channel1.backend.ReadChan():\n\t\t\tmsg, _ = decodeMessage(b)\n\t\t}\n\t\tt.Logf(\"read message %d\", i+1)\n\t\ttest.Equal(t, body, msg.Body)\n\t}\n\n\tfor {\n\t\tif channel1.Depth() == int64(iterations/2) {\n\t\t\tbreak\n\t\t}\n\t\ttime.Sleep(50 * time.Millisecond)\n\t}\n\n\t// make sure metadata shows the topic\n\tm, err = getMetadata(nsqd)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 1, len(m.Topics))\n\ttest.Equal(t, topicName, m.Topics[0].Name)\n\n\texitChan <- 1\n\t<-doneExitChan\n\n\t// start up a new nsqd w/ the same folder\n\n\topts = NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MemQueueSize = 100\n\topts.MaxBytesPerFile = 10240\n\topts.DataPath = origDataPath\n\t_, _, nsqd = mustStartNSQD(opts)\n\n\tgo func() {\n\t\t<-exitChan\n\t\tnsqd.Exit()\n\t\tdoneExitChan <- 1\n\t}()\n\n\ttopic = nsqd.GetTopic(topicName)\n\t// should be empty; channel should have drained everything\n\tcount := topic.Depth()\n\ttest.Equal(t, int64(0), count)\n\n\tchannel1 = topic.GetChannel(\"ch1\")\n\n\tfor {\n\t\tif channel1.Depth() == int64(iterations/2) {\n\t\t\tbreak\n\t\t}\n\t\ttime.Sleep(50 * time.Millisecond)\n\t}\n\n\t// read the other half of the messages\n\tfor i := 0; i < iterations/2; i++ {\n\t\tselect {\n\t\tcase msg = <-channel1.memoryMsgChan:\n\t\tcase b := <-channel1.backend.ReadChan():\n\t\t\tmsg, _ = decodeMessage(b)\n\t\t}\n\t\tt.Logf(\"read message %d\", i+1)\n\t\ttest.Equal(t, body, msg.Body)\n\t}\n\n\t// verify we drained things\n\ttest.Equal(t, 0, len(topic.memoryMsgChan))\n\ttest.Equal(t, int64(0), topic.backend.Depth())\n\n\texitChan <- 1\n\t<-doneExitChan\n}\n\nfunc TestEphemeralTopicsAndChannels(t *testing.T) {\n\t// ephemeral topics/channels are lazily removed after the last channel/client is removed\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MemQueueSize = 100\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\n\ttopicName := \"ephemeral_topic\" + strconv.Itoa(int(time.Now().Unix())) + \"#ephemeral\"\n\tdoneExitChan := make(chan int)\n\n\texitChan := make(chan int)\n\tgo func() {\n\t\t<-exitChan\n\t\tnsqd.Exit()\n\t\tdoneExitChan <- 1\n\t}()\n\n\tbody := []byte(\"an_ephemeral_message\")\n\ttopic := nsqd.GetTopic(topicName)\n\tephemeralChannel := topic.GetChannel(\"ch1#ephemeral\")\n\tclient := newClientV2(0, nil, nsqd)\n\terr := ephemeralChannel.AddClient(client.ID, client)\n\ttest.Equal(t, err, nil)\n\n\tmsg := NewMessage(topic.GenerateID(), body)\n\ttopic.PutMessage(msg)\n\tmsg = <-ephemeralChannel.memoryMsgChan\n\ttest.Equal(t, body, msg.Body)\n\n\tephemeralChannel.RemoveClient(client.ID)\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\ttopic.Lock()\n\tnumChannels := len(topic.channelMap)\n\ttopic.Unlock()\n\ttest.Equal(t, 0, numChannels)\n\n\tnsqd.Lock()\n\tnumTopics := len(nsqd.topicMap)\n\tnsqd.Unlock()\n\ttest.Equal(t, 0, numTopics)\n\n\texitChan <- 1\n\t<-doneExitChan\n}\n\nfunc TestPauseMetadata(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\t// avoid concurrency issue of async PersistMetadata() calls\n\tatomic.StoreInt32(&nsqd.isLoading, 1)\n\ttopicName := \"pause_metadata\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"ch\")\n\tatomic.StoreInt32(&nsqd.isLoading, 0)\n\tnsqd.PersistMetadata()\n\n\tvar isPaused = func(n *NSQD, topicIndex int, channelIndex int) bool {\n\t\tm, _ := getMetadata(n)\n\t\treturn m.Topics[topicIndex].Channels[channelIndex].Paused\n\t}\n\n\ttest.Equal(t, false, isPaused(nsqd, 0, 0))\n\n\tchannel.Pause()\n\ttest.Equal(t, false, isPaused(nsqd, 0, 0))\n\n\tnsqd.PersistMetadata()\n\ttest.Equal(t, true, isPaused(nsqd, 0, 0))\n\n\tchannel.UnPause()\n\ttest.Equal(t, true, isPaused(nsqd, 0, 0))\n\n\tnsqd.PersistMetadata()\n\ttest.Equal(t, false, isPaused(nsqd, 0, 0))\n}\n\nfunc mustStartNSQLookupd(opts *nsqlookupd.Options) (net.Addr, net.Addr, *nsqlookupd.NSQLookupd) {\n\topts.TCPAddress = \"127.0.0.1:0\"\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\tlookupd, err := nsqlookupd.New(opts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := lookupd.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\treturn lookupd.RealTCPAddr(), lookupd.RealHTTPAddr(), lookupd\n}\n\nfunc TestReconfigure(t *testing.T) {\n\tlopts := nsqlookupd.NewOptions()\n\tlopts.Logger = test.NewTestLogger(t)\n\n\tlopts1 := *lopts\n\t_, _, lookupd1 := mustStartNSQLookupd(&lopts1)\n\tdefer lookupd1.Exit()\n\tlopts2 := *lopts\n\t_, _, lookupd2 := mustStartNSQLookupd(&lopts2)\n\tdefer lookupd2.Exit()\n\tlopts3 := *lopts\n\t_, _, lookupd3 := mustStartNSQLookupd(&lopts3)\n\tdefer lookupd3.Exit()\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tnewOpts := NewOptions()\n\tnewOpts.Logger = opts.Logger\n\tnewOpts.NSQLookupdTCPAddresses = []string{lookupd1.RealTCPAddr().String()}\n\tnsqd.swapOpts(newOpts)\n\tnsqd.triggerOptsNotification()\n\ttest.Equal(t, 1, len(nsqd.getOpts().NSQLookupdTCPAddresses))\n\n\tvar numLookupPeers int\n\tfor i := 0; i < 100; i++ {\n\t\tnumLookupPeers = len(nsqd.lookupPeers.Load().([]*lookupPeer))\n\t\tif numLookupPeers == 1 {\n\t\t\tbreak\n\t\t}\n\t\ttime.Sleep(10 * time.Millisecond)\n\t}\n\ttest.Equal(t, 1, numLookupPeers)\n\n\tnewOpts = NewOptions()\n\tnewOpts.Logger = opts.Logger\n\tnewOpts.NSQLookupdTCPAddresses = []string{lookupd2.RealTCPAddr().String(), lookupd3.RealTCPAddr().String()}\n\tnsqd.swapOpts(newOpts)\n\tnsqd.triggerOptsNotification()\n\ttest.Equal(t, 2, len(nsqd.getOpts().NSQLookupdTCPAddresses))\n\n\tfor i := 0; i < 100; i++ {\n\t\tnumLookupPeers = len(nsqd.lookupPeers.Load().([]*lookupPeer))\n\t\tif numLookupPeers == 2 {\n\t\t\tbreak\n\t\t}\n\t\ttime.Sleep(10 * time.Millisecond)\n\t}\n\ttest.Equal(t, 2, numLookupPeers)\n\n\tvar lookupPeers []string\n\tfor _, lp := range nsqd.lookupPeers.Load().([]*lookupPeer) {\n\t\tlookupPeers = append(lookupPeers, lp.addr)\n\t}\n\ttest.Equal(t, newOpts.NSQLookupdTCPAddresses, lookupPeers)\n}\n\nfunc TestCluster(t *testing.T) {\n\tlopts := nsqlookupd.NewOptions()\n\tlopts.Logger = test.NewTestLogger(t)\n\tlopts.BroadcastAddress = \"127.0.0.1\"\n\t_, _, lookupd := mustStartNSQLookupd(lopts)\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.NSQLookupdTCPAddresses = []string{lookupd.RealTCPAddr().String()}\n\topts.BroadcastAddress = \"127.0.0.1\"\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"cluster_test\" + strconv.Itoa(int(time.Now().Unix()))\n\n\thostname, err := os.Hostname()\n\ttest.Nil(t, err)\n\n\turl := fmt.Sprintf(\"http://%s/topic/create?topic=%s\", nsqd.RealHTTPAddr(), topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).POSTV1(url, nil, nil)\n\ttest.Nil(t, err)\n\n\turl = fmt.Sprintf(\"http://%s/channel/create?topic=%s&channel=ch\", nsqd.RealHTTPAddr(), topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).POSTV1(url, nil, nil)\n\ttest.Nil(t, err)\n\n\t// allow some time for nsqd to push info to nsqlookupd\n\ttime.Sleep(350 * time.Millisecond)\n\n\tvar d map[string][]struct {\n\t\tHostname string `json:\"hostname\"`\n\t\tBroadcastAddress string `json:\"broadcast_address\"`\n\t\tTCPPort int `json:\"tcp_port\"`\n\t\tTombstoned bool `json:\"tombstoned\"`\n\t}\n\n\tendpoint := fmt.Sprintf(\"http://%s/debug\", lookupd.RealHTTPAddr())\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &d)\n\ttest.Nil(t, err)\n\n\ttopicData := d[\"topic:\"+topicName+\":\"]\n\ttest.Equal(t, 1, len(topicData))\n\n\ttest.Equal(t, hostname, topicData[0].Hostname)\n\ttest.Equal(t, \"127.0.0.1\", topicData[0].BroadcastAddress)\n\ttest.Equal(t, nsqd.RealTCPAddr().(*net.TCPAddr).Port, topicData[0].TCPPort)\n\ttest.Equal(t, false, topicData[0].Tombstoned)\n\n\tchannelData := d[\"channel:\"+topicName+\":ch\"]\n\ttest.Equal(t, 1, len(channelData))\n\n\ttest.Equal(t, hostname, channelData[0].Hostname)\n\ttest.Equal(t, \"127.0.0.1\", channelData[0].BroadcastAddress)\n\ttest.Equal(t, nsqd.RealTCPAddr().(*net.TCPAddr).Port, channelData[0].TCPPort)\n\ttest.Equal(t, false, channelData[0].Tombstoned)\n\n\tvar lr struct {\n\t\tProducers []struct {\n\t\t\tHostname string `json:\"hostname\"`\n\t\t\tBroadcastAddress string `json:\"broadcast_address\"`\n\t\t\tTCPPort int `json:\"tcp_port\"`\n\t\t} `json:\"producers\"`\n\t\tChannels []string `json:\"channels\"`\n\t}\n\n\tendpoint = fmt.Sprintf(\"http://%s/lookup?topic=%s\", lookupd.RealHTTPAddr(), topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &lr)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, 1, len(lr.Producers))\n\ttest.Equal(t, hostname, lr.Producers[0].Hostname)\n\ttest.Equal(t, \"127.0.0.1\", lr.Producers[0].BroadcastAddress)\n\ttest.Equal(t, nsqd.RealTCPAddr().(*net.TCPAddr).Port, lr.Producers[0].TCPPort)\n\ttest.Equal(t, 1, len(lr.Channels))\n\ttest.Equal(t, \"ch\", lr.Channels[0])\n\n\turl = fmt.Sprintf(\"http://%s/topic/delete?topic=%s\", nsqd.RealHTTPAddr(), topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).POSTV1(url, nil, nil)\n\ttest.Nil(t, err)\n\n\t// allow some time for nsqd to push info to nsqlookupd\n\ttime.Sleep(350 * time.Millisecond)\n\n\tendpoint = fmt.Sprintf(\"http://%s/lookup?topic=%s\", lookupd.RealHTTPAddr(), topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &lr)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, 0, len(lr.Producers))\n\n\tvar dd map[string][]interface{}\n\tendpoint = fmt.Sprintf(\"http://%s/debug\", lookupd.RealHTTPAddr())\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &dd)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, 0, len(dd[\"topic:\"+topicName+\":\"]))\n\ttest.Equal(t, 0, len(dd[\"channel:\"+topicName+\":ch\"]))\n}\n\nfunc TestSetHealth(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\tnsqd, err := New(opts)\n\ttest.Nil(t, err)\n\tdefer nsqd.Exit()\n\n\ttest.Nil(t, nsqd.GetError())\n\ttest.Equal(t, true, nsqd.IsHealthy())\n\n\tnsqd.SetHealth(nil)\n\ttest.Nil(t, nsqd.GetError())\n\ttest.Equal(t, true, nsqd.IsHealthy())\n\n\tnsqd.SetHealth(errors.New(\"health error\"))\n\ttest.NotNil(t, nsqd.GetError())\n\ttest.Equal(t, \"NOK - health error\", nsqd.GetHealth())\n\ttest.Equal(t, false, nsqd.IsHealthy())\n\n\tnsqd.SetHealth(nil)\n\ttest.Nil(t, nsqd.GetError())\n\ttest.Equal(t, \"OK\", nsqd.GetHealth())\n\ttest.Equal(t, true, nsqd.IsHealthy())\n}\n\nfunc TestUnixSocketStartup(t *testing.T) {\n\tisSocket := func(path string) bool {\n\t\tfileInfo, err := os.Stat(path)\n\t\tif err != nil {\n\t\t\treturn false\n\t\t}\n\t\treturn fileInfo.Mode().Type() == fs.ModeSocket\n\t}\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\n\t_, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttest.Equal(t, isSocket(opts.TCPAddress), true)\n\ttest.Equal(t, isSocket(opts.HTTPAddress), true)\n}\n" }, { "path": "nsqd/options.go", "content": "package nsqd\n\nimport (\n\t\"crypto/md5\"\n\t\"crypto/tls\"\n\t\"hash/crc32\"\n\t\"io\"\n\t\"log\"\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype Options struct {\n\t// basic options\n\tID int64 `flag:\"node-id\" cfg:\"id\"`\n\tLogLevel lg.LogLevel `flag:\"log-level\"`\n\tLogPrefix string `flag:\"log-prefix\"`\n\tLogger Logger\n\n\tTCPAddress string `flag:\"tcp-address\"`\n\tHTTPAddress string `flag:\"http-address\"`\n\tHTTPSAddress string `flag:\"https-address\"`\n\tBroadcastAddress string `flag:\"broadcast-address\"`\n\tBroadcastTCPPort int `flag:\"broadcast-tcp-port\"`\n\tBroadcastHTTPPort int `flag:\"broadcast-http-port\"`\n\tNSQLookupdTCPAddresses []string `flag:\"lookupd-tcp-address\" cfg:\"nsqlookupd_tcp_addresses\"`\n\tAuthHTTPAddresses []string `flag:\"auth-http-address\" cfg:\"auth_http_addresses\"`\n\tAuthHTTPRequestMethod string `flag:\"auth-http-request-method\" cfg:\"auth_http_request_method\"`\n\tHTTPClientConnectTimeout time.Duration `flag:\"http-client-connect-timeout\" cfg:\"http_client_connect_timeout\"`\n\tHTTPClientRequestTimeout time.Duration `flag:\"http-client-request-timeout\" cfg:\"http_client_request_timeout\"`\n\tTopologyRegion string `flag:\"topology-region\"`\n\tTopologyZone string `flag:\"topology-zone\"`\n\n\t// diskqueue options\n\tDataPath string `flag:\"data-path\"`\n\tMemQueueSize int64 `flag:\"mem-queue-size\"`\n\tMaxBytesPerFile int64 `flag:\"max-bytes-per-file\"`\n\tSyncEvery int64 `flag:\"sync-every\"`\n\tSyncTimeout time.Duration `flag:\"sync-timeout\"`\n\n\tQueueScanInterval time.Duration\n\tQueueScanRefreshInterval time.Duration\n\tQueueScanSelectionCount int `flag:\"queue-scan-selection-count\"`\n\tQueueScanWorkerPoolMax int `flag:\"queue-scan-worker-pool-max\"`\n\tQueueScanDirtyPercent float64\n\n\t// msg and command options\n\tMsgTimeout time.Duration `flag:\"msg-timeout\"`\n\tMaxMsgTimeout time.Duration `flag:\"max-msg-timeout\"`\n\tMaxMsgSize int64 `flag:\"max-msg-size\"`\n\tMaxBodySize int64 `flag:\"max-body-size\"`\n\tMaxReqTimeout time.Duration `flag:\"max-req-timeout\"`\n\tClientTimeout time.Duration\n\tMaxDeferTimeout time.Duration `flag:\"max-defer-timeout\"`\n\n\t// client overridable configuration options\n\tMaxHeartbeatInterval time.Duration `flag:\"max-heartbeat-interval\"`\n\tMaxRdyCount int64 `flag:\"max-rdy-count\"`\n\tMaxOutputBufferSize int64 `flag:\"max-output-buffer-size\"`\n\tMaxOutputBufferTimeout time.Duration `flag:\"max-output-buffer-timeout\"`\n\tMinOutputBufferTimeout time.Duration `flag:\"min-output-buffer-timeout\"`\n\tOutputBufferTimeout time.Duration `flag:\"output-buffer-timeout\"`\n\tMaxChannelConsumers int `flag:\"max-channel-consumers\"`\n\n\t// statsd integration\n\tStatsdAddress string `flag:\"statsd-address\"`\n\tStatsdPrefix string `flag:\"statsd-prefix\"`\n\tStatsdInterval time.Duration `flag:\"statsd-interval\"`\n\tStatsdMemStats bool `flag:\"statsd-mem-stats\"`\n\tStatsdUDPPacketSize int `flag:\"statsd-udp-packet-size\"`\n\tStatsdExcludeEphemeral bool `flag:\"statsd-exclude-ephemeral\"`\n\n\t// e2e message latency\n\tE2EProcessingLatencyWindowTime time.Duration `flag:\"e2e-processing-latency-window-time\"`\n\tE2EProcessingLatencyPercentiles []float64 `flag:\"e2e-processing-latency-percentile\" cfg:\"e2e_processing_latency_percentiles\"`\n\n\t// TLS config\n\tTLSCert string `flag:\"tls-cert\"`\n\tTLSKey string `flag:\"tls-key\"`\n\tTLSClientAuthPolicy string `flag:\"tls-client-auth-policy\"`\n\tTLSRootCAFile string `flag:\"tls-root-ca-file\"`\n\tTLSRequired int `flag:\"tls-required\"`\n\tTLSMinVersion uint16 `flag:\"tls-min-version\"`\n\n\t// compression\n\tDeflateEnabled bool `flag:\"deflate\"`\n\tMaxDeflateLevel int `flag:\"max-deflate-level\"`\n\tSnappyEnabled bool `flag:\"snappy\"`\n\n\t// experimental features\n\tExperiments []string `flag:\"enable-experiment\" cfg:\"enable_experiment\"`\n}\n\ntype Experiment string\n\nconst (\n\tTopologyAwareConsumption Experiment = \"topology-aware-consumption\"\n)\n\nvar AllExperiments = []Experiment{\n\tTopologyAwareConsumption,\n}\n\nfunc (o Options) HasExperiment(e Experiment) bool {\n\tfor _, v := range o.Experiments {\n\t\tif string(e) == v {\n\t\t\treturn true\n\t\t}\n\t}\n\treturn false\n}\n\nfunc NewOptions() *Options {\n\thostname, err := os.Hostname()\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\th := md5.New()\n\tio.WriteString(h, hostname)\n\tdefaultID := int64(crc32.ChecksumIEEE(h.Sum(nil)) % 1024)\n\n\treturn &Options{\n\t\tID: defaultID,\n\t\tLogPrefix: \"[nsqd] \",\n\t\tLogLevel: lg.INFO,\n\n\t\tTCPAddress: \"0.0.0.0:4150\",\n\t\tHTTPAddress: \"0.0.0.0:4151\",\n\t\tHTTPSAddress: \"0.0.0.0:4152\",\n\t\tBroadcastAddress: hostname,\n\t\tBroadcastTCPPort: 0,\n\t\tBroadcastHTTPPort: 0,\n\n\t\tNSQLookupdTCPAddresses: make([]string, 0),\n\t\tAuthHTTPAddresses: make([]string, 0),\n\t\tAuthHTTPRequestMethod: \"get\",\n\n\t\tHTTPClientConnectTimeout: 2 * time.Second,\n\t\tHTTPClientRequestTimeout: 5 * time.Second,\n\n\t\tMemQueueSize: 10000,\n\t\tMaxBytesPerFile: 100 * 1024 * 1024,\n\t\tSyncEvery: 2500,\n\t\tSyncTimeout: 2 * time.Second,\n\n\t\tQueueScanInterval: 100 * time.Millisecond,\n\t\tQueueScanRefreshInterval: 5 * time.Second,\n\t\tQueueScanSelectionCount: 20,\n\t\tQueueScanWorkerPoolMax: 4,\n\t\tQueueScanDirtyPercent: 0.25,\n\n\t\tMsgTimeout: 60 * time.Second,\n\t\tMaxMsgTimeout: 15 * time.Minute,\n\t\tMaxMsgSize: 1024 * 1024,\n\t\tMaxBodySize: 5 * 1024 * 1024,\n\t\tMaxReqTimeout: 1 * time.Hour,\n\t\tClientTimeout: 60 * time.Second,\n\t\tMaxDeferTimeout: 1 * time.Hour,\n\n\t\tMaxHeartbeatInterval: 60 * time.Second,\n\t\tMaxRdyCount: 2500,\n\t\tMaxOutputBufferSize: 64 * 1024,\n\t\tMaxOutputBufferTimeout: 30 * time.Second,\n\t\tMinOutputBufferTimeout: 25 * time.Millisecond,\n\t\tOutputBufferTimeout: 250 * time.Millisecond,\n\t\tMaxChannelConsumers: 0,\n\n\t\tStatsdPrefix: \"nsq.%s\",\n\t\tStatsdInterval: 60 * time.Second,\n\t\tStatsdMemStats: true,\n\t\tStatsdUDPPacketSize: 508,\n\n\t\tE2EProcessingLatencyWindowTime: time.Duration(10 * time.Minute),\n\n\t\tDeflateEnabled: true,\n\t\tMaxDeflateLevel: 6,\n\t\tSnappyEnabled: true,\n\n\t\tTLSMinVersion: tls.VersionTLS10,\n\t}\n}\n" }, { "path": "nsqd/protocol_v2.go", "content": "package nsqd\n\nimport (\n\t\"bytes\"\n\t\"encoding/binary\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"math/rand\"\n\t\"net\"\n\t\"sync/atomic\"\n\t\"time\"\n\t\"unsafe\"\n\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\nconst (\n\tframeTypeResponse int32 = 0\n\tframeTypeError int32 = 1\n\tframeTypeMessage int32 = 2\n)\n\nvar separatorBytes = []byte(\" \")\nvar heartbeatBytes = []byte(\"_heartbeat_\")\nvar okBytes = []byte(\"OK\")\n\ntype protocolV2 struct {\n\tnsqd *NSQD\n}\n\nfunc (p *protocolV2) NewClient(conn net.Conn) protocol.Client {\n\tclientID := atomic.AddInt64(&p.nsqd.clientIDSequence, 1)\n\treturn newClientV2(clientID, conn, p.nsqd)\n}\n\nfunc (p *protocolV2) IOLoop(c protocol.Client) error {\n\tvar err error\n\tvar line []byte\n\tvar zeroTime time.Time\n\n\tclient := c.(*clientV2)\n\n\t// synchronize the startup of messagePump in order\n\t// to guarantee that it gets a chance to initialize\n\t// goroutine local state derived from client attributes\n\t// and avoid a potential race with IDENTIFY (where a client\n\t// could have changed or disabled said attributes)\n\tmessagePumpStartedChan := make(chan bool)\n\tgo p.messagePump(client, messagePumpStartedChan)\n\t<-messagePumpStartedChan\n\n\tfor {\n\t\tif client.HeartbeatInterval > 0 {\n\t\t\tclient.SetReadDeadline(time.Now().Add(client.HeartbeatInterval * 2))\n\t\t} else {\n\t\t\tclient.SetReadDeadline(zeroTime)\n\t\t}\n\n\t\t// ReadSlice does not allocate new space for the data each request\n\t\t// ie. the returned slice is only valid until the next call to it\n\t\tline, err = client.Reader.ReadSlice('\\n')\n\t\tif err != nil {\n\t\t\tif err == io.EOF {\n\t\t\t\terr = nil\n\t\t\t} else {\n\t\t\t\terr = fmt.Errorf(\"failed to read command - %s\", err)\n\t\t\t}\n\t\t\tbreak\n\t\t}\n\n\t\t// trim the '\\n'\n\t\tline = line[:len(line)-1]\n\t\t// optionally trim the '\\r'\n\t\tif len(line) > 0 && line[len(line)-1] == '\\r' {\n\t\t\tline = line[:len(line)-1]\n\t\t}\n\t\tparams := bytes.Split(line, separatorBytes)\n\n\t\tp.nsqd.logf(LOG_DEBUG, \"PROTOCOL(V2): [%s] %s\", client, params)\n\n\t\tvar response []byte\n\t\tresponse, err = p.Exec(client, params)\n\t\tif err != nil {\n\t\t\tctx := \"\"\n\t\t\tif parentErr := err.(protocol.ChildErr).Parent(); parentErr != nil {\n\t\t\t\tctx = \" - \" + parentErr.Error()\n\t\t\t}\n\t\t\tp.nsqd.logf(LOG_ERROR, \"[%s] - %s%s\", client, err, ctx)\n\n\t\t\tsendErr := p.Send(client, frameTypeError, []byte(err.Error()))\n\t\t\tif sendErr != nil {\n\t\t\t\tp.nsqd.logf(LOG_ERROR, \"[%s] - %s%s\", client, sendErr, ctx)\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\t// errors of type FatalClientErr should forceably close the connection\n\t\t\tif _, ok := err.(*protocol.FatalClientErr); ok {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tif response != nil {\n\t\t\terr = p.Send(client, frameTypeResponse, response)\n\t\t\tif err != nil {\n\t\t\t\terr = fmt.Errorf(\"failed to send response - %s\", err)\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}\n\n\tp.nsqd.logf(LOG_INFO, \"PROTOCOL(V2): [%s] exiting ioloop\", client)\n\tclose(client.ExitChan)\n\tif client.Channel != nil {\n\t\tclient.Channel.RemoveClient(client.ID)\n\t}\n\n\treturn err\n}\n\nfunc (p *protocolV2) SendMessage(client *clientV2, msg *Message) error {\n\tp.nsqd.logf(LOG_DEBUG, \"PROTOCOL(V2): writing msg(%s) to client(%s) - %s\", msg.ID, client, msg.Body)\n\n\tbuf := bufferPoolGet()\n\tdefer bufferPoolPut(buf)\n\n\t_, err := msg.WriteTo(buf)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\terr = p.Send(client, frameTypeMessage, buf.Bytes())\n\tif err != nil {\n\t\treturn err\n\t}\n\n\treturn nil\n}\n\nfunc (p *protocolV2) Send(client *clientV2, frameType int32, data []byte) error {\n\tclient.writeLock.Lock()\n\n\tvar zeroTime time.Time\n\tif client.HeartbeatInterval > 0 {\n\t\tclient.SetWriteDeadline(time.Now().Add(client.HeartbeatInterval))\n\t} else {\n\t\tclient.SetWriteDeadline(zeroTime)\n\t}\n\n\t_, err := protocol.SendFramedResponse(client.Writer, frameType, data)\n\tif err != nil {\n\t\tclient.writeLock.Unlock()\n\t\treturn err\n\t}\n\n\tif frameType != frameTypeMessage {\n\t\terr = client.Flush()\n\t}\n\n\tclient.writeLock.Unlock()\n\n\treturn err\n}\n\nfunc (p *protocolV2) Exec(client *clientV2, params [][]byte) ([]byte, error) {\n\tif bytes.Equal(params[0], []byte(\"IDENTIFY\")) {\n\t\treturn p.IDENTIFY(client, params)\n\t}\n\terr := enforceTLSPolicy(client, p, params[0])\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tswitch {\n\tcase bytes.Equal(params[0], []byte(\"FIN\")):\n\t\treturn p.FIN(client, params)\n\tcase bytes.Equal(params[0], []byte(\"RDY\")):\n\t\treturn p.RDY(client, params)\n\tcase bytes.Equal(params[0], []byte(\"REQ\")):\n\t\treturn p.REQ(client, params)\n\tcase bytes.Equal(params[0], []byte(\"PUB\")):\n\t\treturn p.PUB(client, params)\n\tcase bytes.Equal(params[0], []byte(\"MPUB\")):\n\t\treturn p.MPUB(client, params)\n\tcase bytes.Equal(params[0], []byte(\"DPUB\")):\n\t\treturn p.DPUB(client, params)\n\tcase bytes.Equal(params[0], []byte(\"NOP\")):\n\t\treturn p.NOP(client, params)\n\tcase bytes.Equal(params[0], []byte(\"TOUCH\")):\n\t\treturn p.TOUCH(client, params)\n\tcase bytes.Equal(params[0], []byte(\"SUB\")):\n\t\treturn p.SUB(client, params)\n\tcase bytes.Equal(params[0], []byte(\"CLS\")):\n\t\treturn p.CLS(client, params)\n\tcase bytes.Equal(params[0], []byte(\"AUTH\")):\n\t\treturn p.AUTH(client, params)\n\t}\n\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", fmt.Sprintf(\"invalid command %s\", params[0]))\n}\n\nfunc (p *protocolV2) messagePump(client *clientV2, startedChan chan bool) {\n\tvar err error\n\tvar zoneMsgChan, regionMsgChan, memoryMsgChan chan *Message\n\tvar backendMsgChan <-chan []byte\n\tvar subChannel *Channel\n\t// NOTE: `flusherChan` is used to bound message latency for\n\t// the pathological case of a channel on a low volume topic\n\t// with >1 clients having >1 RDY counts\n\tvar flusherChan <-chan time.Time\n\tvar sampleRate int32\n\tvar regionLocal, zoneLocal bool\n\n\tsubEventChan := client.SubEventChan\n\tidentifyEventChan := client.IdentifyEventChan\n\toutputBufferTicker := time.NewTicker(client.OutputBufferTimeout)\n\theartbeatTicker := time.NewTicker(client.HeartbeatInterval)\n\theartbeatChan := heartbeatTicker.C\n\tmsgTimeout := client.MsgTimeout\n\n\t// v2 opportunistically buffers data to clients to reduce write system calls\n\t// we force flush in two cases:\n\t// 1. when the client is not ready to receive messages\n\t// 2. we're buffered and the channel has nothing left to send us\n\t// (ie. we would block in this loop anyway)\n\t//\n\tflushed := true\n\n\t// signal to the goroutine that started the messagePump\n\t// that we've started up\n\tclose(startedChan)\n\n\tfor {\n\t\tvar b []byte\n\t\tvar msg *Message\n\t\tif subChannel == nil || !client.IsReadyForMessages() {\n\t\t\t// the client is not ready to receive messages...\n\t\t\tmemoryMsgChan = nil\n\t\t\tregionMsgChan = nil\n\t\t\tzoneMsgChan = nil\n\t\t\tbackendMsgChan = nil\n\t\t\tflusherChan = nil\n\t\t\t// force flush\n\t\t\tclient.writeLock.Lock()\n\t\t\terr = client.Flush()\n\t\t\tclient.writeLock.Unlock()\n\t\t\tif err != nil {\n\t\t\t\tgoto exit\n\t\t\t}\n\t\t\tflushed = true\n\t\t} else if flushed {\n\t\t\t// last iteration we flushed...\n\t\t\t// do not select on the flusher ticker channel\n\t\t\tmemoryMsgChan = subChannel.memoryMsgChan\n\t\t\tif zoneLocal {\n\t\t\t\tzoneMsgChan = subChannel.zoneLocalMsgChan\n\t\t\t}\n\t\t\tif regionLocal {\n\t\t\t\tregionMsgChan = subChannel.regionLocalMsgChan\n\t\t\t}\n\t\t\tbackendMsgChan = subChannel.backend.ReadChan()\n\t\t\tflusherChan = nil\n\t\t} else {\n\t\t\t// we're buffered (if there isn't any more data we should flush)...\n\t\t\t// select on the flusher ticker channel, too\n\t\t\tmemoryMsgChan = subChannel.memoryMsgChan\n\t\t\tif zoneLocal {\n\t\t\t\tzoneMsgChan = subChannel.zoneLocalMsgChan\n\t\t\t}\n\t\t\tif regionLocal {\n\t\t\t\tregionMsgChan = subChannel.regionLocalMsgChan\n\t\t\t}\n\t\t\tbackendMsgChan = subChannel.backend.ReadChan()\n\t\t\tflusherChan = outputBufferTicker.C\n\t\t}\n\n\t\tselect {\n\t\tcase <-flusherChan:\n\t\t\t// if this case wins, we're either starved\n\t\t\t// or we won the race between other channels...\n\t\t\t// in either case, force flush\n\t\t\tclient.writeLock.Lock()\n\t\t\terr = client.Flush()\n\t\t\tclient.writeLock.Unlock()\n\t\t\tif err != nil {\n\t\t\t\tgoto exit\n\t\t\t}\n\t\t\tflushed = true\n\t\tcase <-client.ReadyStateChan:\n\t\tcase subChannel = <-subEventChan:\n\t\t\t// you can't SUB anymore\n\t\t\tsubEventChan = nil\n\t\tcase identifyData := <-identifyEventChan:\n\t\t\t// you can't IDENTIFY anymore\n\t\t\tidentifyEventChan = nil\n\n\t\t\toutputBufferTicker.Stop()\n\t\t\tif identifyData.OutputBufferTimeout > 0 {\n\t\t\t\toutputBufferTicker = time.NewTicker(identifyData.OutputBufferTimeout)\n\t\t\t}\n\n\t\t\theartbeatTicker.Stop()\n\t\t\theartbeatChan = nil\n\t\t\tif identifyData.HeartbeatInterval > 0 {\n\t\t\t\theartbeatTicker = time.NewTicker(identifyData.HeartbeatInterval)\n\t\t\t\theartbeatChan = heartbeatTicker.C\n\t\t\t}\n\n\t\t\tif identifyData.SampleRate > 0 {\n\t\t\t\tsampleRate = identifyData.SampleRate\n\t\t\t}\n\n\t\t\tmsgTimeout = identifyData.MsgTimeout\n\t\t\tisToplogyAware := p.nsqd.getOpts().HasExperiment(TopologyAwareConsumption)\n\t\t\tif identifyData.TopologyZone == p.nsqd.getOpts().TopologyZone && isToplogyAware {\n\t\t\t\tzoneLocal = true\n\t\t\t}\n\t\t\tif identifyData.TopologyRegion == p.nsqd.getOpts().TopologyRegion && isToplogyAware {\n\t\t\t\tregionLocal = true\n\t\t\t}\n\t\tcase <-heartbeatChan:\n\t\t\terr = p.Send(client, frameTypeResponse, heartbeatBytes)\n\t\t\tif err != nil {\n\t\t\t\tgoto exit\n\t\t\t}\n\t\tcase b = <-backendMsgChan:\n\t\t\t// decodeMessage then handle 'msg'\n\t\tcase msg = <-zoneMsgChan:\n\t\t\tatomic.AddUint64(&client.Channel.zoneLocalMsgCount, 1)\n\t\tcase msg = <-regionMsgChan:\n\t\t\tif zoneLocal {\n\t\t\t\tatomic.AddUint64(&client.Channel.zoneLocalMsgCount, 1)\n\t\t\t} else {\n\t\t\t\tatomic.AddUint64(&client.Channel.regionLocalMsgCount, 1)\n\t\t\t}\n\t\tcase msg = <-memoryMsgChan:\n\t\t\tif zoneLocal {\n\t\t\t\tatomic.AddUint64(&client.Channel.zoneLocalMsgCount, 1)\n\t\t\t} else if regionLocal {\n\t\t\t\tatomic.AddUint64(&client.Channel.regionLocalMsgCount, 1)\n\t\t\t} else {\n\t\t\t\tatomic.AddUint64(&client.Channel.globalMsgCount, 1)\n\t\t\t}\n\t\tcase <-client.ExitChan:\n\t\t\tgoto exit\n\t\t}\n\t\tif len(b) != 0 {\n\t\t\tmsg, err = decodeMessage(b)\n\t\t\tif err != nil {\n\t\t\t\tp.nsqd.logf(LOG_ERROR, \"failed to decode message - %s\", err)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t}\n\t\tif msg != nil {\n\t\t\tif sampleRate > 0 && rand.Int31n(100) > sampleRate {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tmsg.Attempts++\n\t\t\tsubChannel.StartInFlightTimeout(msg, client.ID, msgTimeout)\n\t\t\tclient.SendingMessage()\n\t\t\terr = p.SendMessage(client, msg)\n\t\t\tif err != nil {\n\t\t\t\tgoto exit\n\t\t\t}\n\t\t\tflushed = false\n\t\t}\n\n\t}\n\nexit:\n\tp.nsqd.logf(LOG_INFO, \"PROTOCOL(V2): [%s] exiting messagePump\", client)\n\theartbeatTicker.Stop()\n\toutputBufferTicker.Stop()\n\tif err != nil {\n\t\tp.nsqd.logf(LOG_ERROR, \"PROTOCOL(V2): [%s] messagePump error - %s\", client, err)\n\t}\n}\n\nfunc (p *protocolV2) IDENTIFY(client *clientV2, params [][]byte) ([]byte, error) {\n\tvar err error\n\n\tif atomic.LoadInt32(&client.State) != stateInit {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"cannot IDENTIFY in current state\")\n\t}\n\n\tbodyLen, err := readLen(client.Reader, client.lenSlice)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"IDENTIFY failed to read body size\")\n\t}\n\n\tif int64(bodyLen) > p.nsqd.getOpts().MaxBodySize {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_BODY\",\n\t\t\tfmt.Sprintf(\"IDENTIFY body too big %d > %d\", bodyLen, p.nsqd.getOpts().MaxBodySize))\n\t}\n\n\tif bodyLen <= 0 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_BODY\",\n\t\t\tfmt.Sprintf(\"IDENTIFY invalid body size %d\", bodyLen))\n\t}\n\n\tbody := make([]byte, bodyLen)\n\t_, err = io.ReadFull(client.Reader, body)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"IDENTIFY failed to read body\")\n\t}\n\n\t// body is a json structure with producer information\n\tvar identifyData identifyDataV2\n\terr = json.Unmarshal(body, &identifyData)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"IDENTIFY failed to decode JSON body\")\n\t}\n\n\tp.nsqd.logf(LOG_DEBUG, \"PROTOCOL(V2): [%s] %+v\", client, identifyData)\n\n\terr = client.Identify(identifyData)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"IDENTIFY \"+err.Error())\n\t}\n\n\t// bail out early if we're not negotiating features\n\tif !identifyData.FeatureNegotiation {\n\t\treturn okBytes, nil\n\t}\n\n\ttlsv1 := p.nsqd.tlsConfig != nil && identifyData.TLSv1\n\tdeflate := p.nsqd.getOpts().DeflateEnabled && identifyData.Deflate\n\tdeflateLevel := 6\n\tif deflate && identifyData.DeflateLevel > 0 {\n\t\tdeflateLevel = identifyData.DeflateLevel\n\t}\n\tif max := p.nsqd.getOpts().MaxDeflateLevel; max < deflateLevel {\n\t\tdeflateLevel = max\n\t}\n\tsnappy := p.nsqd.getOpts().SnappyEnabled && identifyData.Snappy\n\n\tif deflate && snappy {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_IDENTIFY_FAILED\", \"cannot enable both deflate and snappy compression\")\n\t}\n\n\tresp, err := json.Marshal(struct {\n\t\tMaxRdyCount int64 `json:\"max_rdy_count\"`\n\t\tVersion string `json:\"version\"`\n\t\tMaxMsgTimeout int64 `json:\"max_msg_timeout\"`\n\t\tMsgTimeout int64 `json:\"msg_timeout\"`\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t\tDeflate bool `json:\"deflate\"`\n\t\tDeflateLevel int `json:\"deflate_level\"`\n\t\tMaxDeflateLevel int `json:\"max_deflate_level\"`\n\t\tSnappy bool `json:\"snappy\"`\n\t\tSampleRate int32 `json:\"sample_rate\"`\n\t\tAuthRequired bool `json:\"auth_required\"`\n\t\tOutputBufferSize int `json:\"output_buffer_size\"`\n\t\tOutputBufferTimeout int64 `json:\"output_buffer_timeout\"`\n\t\tTopologyRegion string `json:\"topology_region\"`\n\t\tTopologyZone string `json:\"topology_zone\"`\n\t}{\n\t\tMaxRdyCount: p.nsqd.getOpts().MaxRdyCount,\n\t\tVersion: version.Binary,\n\t\tMaxMsgTimeout: int64(p.nsqd.getOpts().MaxMsgTimeout / time.Millisecond),\n\t\tMsgTimeout: int64(client.MsgTimeout / time.Millisecond),\n\t\tTLSv1: tlsv1,\n\t\tDeflate: deflate,\n\t\tDeflateLevel: deflateLevel,\n\t\tMaxDeflateLevel: p.nsqd.getOpts().MaxDeflateLevel,\n\t\tSnappy: snappy,\n\t\tSampleRate: client.SampleRate,\n\t\tAuthRequired: p.nsqd.IsAuthEnabled(),\n\t\tOutputBufferSize: client.OutputBufferSize,\n\t\tOutputBufferTimeout: int64(client.OutputBufferTimeout / time.Millisecond),\n\t\tTopologyRegion: p.nsqd.getOpts().TopologyRegion,\n\t\tTopologyZone: p.nsqd.getOpts().TopologyZone,\n\t})\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_IDENTIFY_FAILED\", \"IDENTIFY failed \"+err.Error())\n\t}\n\n\terr = p.Send(client, frameTypeResponse, resp)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_IDENTIFY_FAILED\", \"IDENTIFY failed \"+err.Error())\n\t}\n\n\tif tlsv1 {\n\t\tp.nsqd.logf(LOG_INFO, \"PROTOCOL(V2): [%s] upgrading connection to TLS\", client)\n\t\terr = client.UpgradeTLS()\n\t\tif err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_IDENTIFY_FAILED\", \"IDENTIFY failed \"+err.Error())\n\t\t}\n\n\t\terr = p.Send(client, frameTypeResponse, okBytes)\n\t\tif err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_IDENTIFY_FAILED\", \"IDENTIFY failed \"+err.Error())\n\t\t}\n\t}\n\n\tif snappy {\n\t\tp.nsqd.logf(LOG_INFO, \"PROTOCOL(V2): [%s] upgrading connection to snappy\", client)\n\t\terr = client.UpgradeSnappy()\n\t\tif err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_IDENTIFY_FAILED\", \"IDENTIFY failed \"+err.Error())\n\t\t}\n\n\t\terr = p.Send(client, frameTypeResponse, okBytes)\n\t\tif err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_IDENTIFY_FAILED\", \"IDENTIFY failed \"+err.Error())\n\t\t}\n\t}\n\n\tif deflate {\n\t\tp.nsqd.logf(LOG_INFO, \"PROTOCOL(V2): [%s] upgrading connection to deflate (level %d)\", client, deflateLevel)\n\t\terr = client.UpgradeDeflate(deflateLevel)\n\t\tif err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_IDENTIFY_FAILED\", \"IDENTIFY failed \"+err.Error())\n\t\t}\n\n\t\terr = p.Send(client, frameTypeResponse, okBytes)\n\t\tif err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_IDENTIFY_FAILED\", \"IDENTIFY failed \"+err.Error())\n\t\t}\n\t}\n\n\treturn nil, nil\n}\n\nfunc (p *protocolV2) AUTH(client *clientV2, params [][]byte) ([]byte, error) {\n\tif atomic.LoadInt32(&client.State) != stateInit {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"cannot AUTH in current state\")\n\t}\n\n\tif len(params) != 1 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"AUTH invalid number of parameters\")\n\t}\n\n\tbodyLen, err := readLen(client.Reader, client.lenSlice)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"AUTH failed to read body size\")\n\t}\n\n\tif int64(bodyLen) > p.nsqd.getOpts().MaxBodySize {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_BODY\",\n\t\t\tfmt.Sprintf(\"AUTH body too big %d > %d\", bodyLen, p.nsqd.getOpts().MaxBodySize))\n\t}\n\n\tif bodyLen <= 0 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_BODY\",\n\t\t\tfmt.Sprintf(\"AUTH invalid body size %d\", bodyLen))\n\t}\n\n\tbody := make([]byte, bodyLen)\n\t_, err = io.ReadFull(client.Reader, body)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"AUTH failed to read body\")\n\t}\n\n\tif client.HasAuthorizations() {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"AUTH already set\")\n\t}\n\n\tif !client.nsqd.IsAuthEnabled() {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_AUTH_DISABLED\", \"AUTH disabled\")\n\t}\n\n\tif err := client.Auth(string(body)); err != nil {\n\t\t// we don't want to leak errors contacting the auth server to untrusted clients\n\t\tp.nsqd.logf(LOG_WARN, \"PROTOCOL(V2): [%s] AUTH failed %s\", client, err)\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_AUTH_FAILED\", \"AUTH failed\")\n\t}\n\n\tif !client.HasAuthorizations() {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_UNAUTHORIZED\", \"AUTH no authorizations found\")\n\t}\n\n\tresp, err := json.Marshal(struct {\n\t\tIdentity string `json:\"identity\"`\n\t\tIdentityURL string `json:\"identity_url\"`\n\t\tPermissionCount int `json:\"permission_count\"`\n\t}{\n\t\tIdentity: client.AuthState.Identity,\n\t\tIdentityURL: client.AuthState.IdentityURL,\n\t\tPermissionCount: len(client.AuthState.Authorizations),\n\t})\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_AUTH_ERROR\", \"AUTH error \"+err.Error())\n\t}\n\n\terr = p.Send(client, frameTypeResponse, resp)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_AUTH_ERROR\", \"AUTH error \"+err.Error())\n\t}\n\n\treturn nil, nil\n\n}\n\nfunc (p *protocolV2) CheckAuth(client *clientV2, cmd, topicName, channelName string) error {\n\t// if auth is enabled, the client must have authorized already\n\t// compare topic/channel against cached authorization data (refetching if expired)\n\tif client.nsqd.IsAuthEnabled() {\n\t\tif !client.HasAuthorizations() {\n\t\t\treturn protocol.NewFatalClientErr(nil, \"E_AUTH_FIRST\",\n\t\t\t\tfmt.Sprintf(\"AUTH required before %s\", cmd))\n\t\t}\n\t\tok, err := client.IsAuthorized(topicName, channelName)\n\t\tif err != nil {\n\t\t\t// we don't want to leak errors contacting the auth server to untrusted clients\n\t\t\tp.nsqd.logf(LOG_WARN, \"PROTOCOL(V2): [%s] AUTH failed %s\", client, err)\n\t\t\treturn protocol.NewFatalClientErr(nil, \"E_AUTH_FAILED\", \"AUTH failed\")\n\t\t}\n\t\tif !ok {\n\t\t\treturn protocol.NewFatalClientErr(nil, \"E_UNAUTHORIZED\",\n\t\t\t\tfmt.Sprintf(\"AUTH failed for %s on %q %q\", cmd, topicName, channelName))\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc (p *protocolV2) SUB(client *clientV2, params [][]byte) ([]byte, error) {\n\tif atomic.LoadInt32(&client.State) != stateInit {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"cannot SUB in current state\")\n\t}\n\n\tif client.HeartbeatInterval <= 0 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"cannot SUB with heartbeats disabled\")\n\t}\n\n\tif len(params) < 3 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"SUB insufficient number of parameters\")\n\t}\n\n\ttopicName := string(params[1])\n\tif !protocol.IsValidTopicName(topicName) {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_TOPIC\",\n\t\t\tfmt.Sprintf(\"SUB topic name %q is not valid\", topicName))\n\t}\n\n\tchannelName := string(params[2])\n\tif !protocol.IsValidChannelName(channelName) {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_CHANNEL\",\n\t\t\tfmt.Sprintf(\"SUB channel name %q is not valid\", channelName))\n\t}\n\n\tif err := p.CheckAuth(client, \"SUB\", topicName, channelName); err != nil {\n\t\treturn nil, err\n\t}\n\n\t// This retry-loop is a work-around for a race condition, where the\n\t// last client can leave the channel between GetChannel() and AddClient().\n\t// Avoid adding a client to an ephemeral channel / topic which has started exiting.\n\tvar channel *Channel\n\tfor i := 1; ; i++ {\n\t\ttopic := p.nsqd.GetTopic(topicName)\n\t\tchannel = topic.GetChannel(channelName)\n\t\tif err := channel.AddClient(client.ID, client); err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_SUB_FAILED\", \"SUB failed \"+err.Error())\n\t\t}\n\n\t\tif (channel.ephemeral && channel.Exiting()) || (topic.ephemeral && topic.Exiting()) {\n\t\t\tchannel.RemoveClient(client.ID)\n\t\t\tif i < 2 {\n\t\t\t\ttime.Sleep(100 * time.Millisecond)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_SUB_FAILED\", \"SUB failed to deleted topic/channel\")\n\t\t}\n\t\tbreak\n\t}\n\tatomic.StoreInt32(&client.State, stateSubscribed)\n\tclient.Channel = channel\n\t// update message pump\n\tclient.SubEventChan <- channel\n\n\treturn okBytes, nil\n}\n\nfunc (p *protocolV2) RDY(client *clientV2, params [][]byte) ([]byte, error) {\n\tstate := atomic.LoadInt32(&client.State)\n\n\tif state == stateClosing {\n\t\t// just ignore ready changes on a closing channel\n\t\tp.nsqd.logf(LOG_INFO,\n\t\t\t\"PROTOCOL(V2): [%s] ignoring RDY after CLS in state ClientStateV2Closing\",\n\t\t\tclient)\n\t\treturn nil, nil\n\t}\n\n\tif state != stateSubscribed {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"cannot RDY in current state\")\n\t}\n\n\tcount := int64(1)\n\tif len(params) > 1 {\n\t\tb10, err := protocol.ByteToBase10(params[1])\n\t\tif err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_INVALID\",\n\t\t\t\tfmt.Sprintf(\"RDY could not parse count %s\", params[1]))\n\t\t}\n\t\tcount = int64(b10)\n\t}\n\n\tif count < 0 || count > p.nsqd.getOpts().MaxRdyCount {\n\t\t// this needs to be a fatal error otherwise clients would have\n\t\t// inconsistent state\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\",\n\t\t\tfmt.Sprintf(\"RDY count %d out of range 0-%d\", count, p.nsqd.getOpts().MaxRdyCount))\n\t}\n\n\tclient.SetReadyCount(count)\n\n\treturn nil, nil\n}\n\nfunc (p *protocolV2) FIN(client *clientV2, params [][]byte) ([]byte, error) {\n\tstate := atomic.LoadInt32(&client.State)\n\tif state != stateSubscribed && state != stateClosing {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"cannot FIN in current state\")\n\t}\n\n\tif len(params) < 2 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"FIN insufficient number of params\")\n\t}\n\n\tid, err := getMessageID(params[1])\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", err.Error())\n\t}\n\n\terr = client.Channel.FinishMessage(client.ID, *id)\n\tif err != nil {\n\t\treturn nil, protocol.NewClientErr(err, \"E_FIN_FAILED\",\n\t\t\tfmt.Sprintf(\"FIN %s failed %s\", *id, err.Error()))\n\t}\n\n\tclient.FinishedMessage()\n\n\treturn nil, nil\n}\n\nfunc (p *protocolV2) REQ(client *clientV2, params [][]byte) ([]byte, error) {\n\tstate := atomic.LoadInt32(&client.State)\n\tif state != stateSubscribed && state != stateClosing {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"cannot REQ in current state\")\n\t}\n\n\tif len(params) < 3 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"REQ insufficient number of params\")\n\t}\n\n\tid, err := getMessageID(params[1])\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", err.Error())\n\t}\n\n\ttimeoutMs, err := protocol.ByteToBase10(params[2])\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_INVALID\",\n\t\t\tfmt.Sprintf(\"REQ could not parse timeout %s\", params[2]))\n\t}\n\ttimeoutDuration := time.Duration(timeoutMs) * time.Millisecond\n\n\tmaxReqTimeout := p.nsqd.getOpts().MaxReqTimeout\n\tclampedTimeout := timeoutDuration\n\n\tif timeoutDuration < 0 {\n\t\tclampedTimeout = 0\n\t} else if timeoutDuration > maxReqTimeout {\n\t\tclampedTimeout = maxReqTimeout\n\t}\n\tif clampedTimeout != timeoutDuration {\n\t\tp.nsqd.logf(LOG_INFO, \"PROTOCOL(V2): [%s] REQ timeout %d out of range 0-%d. Setting to %d\",\n\t\t\tclient, timeoutDuration, maxReqTimeout, clampedTimeout)\n\t\ttimeoutDuration = clampedTimeout\n\t}\n\n\terr = client.Channel.RequeueMessage(client.ID, *id, timeoutDuration)\n\tif err != nil {\n\t\treturn nil, protocol.NewClientErr(err, \"E_REQ_FAILED\",\n\t\t\tfmt.Sprintf(\"REQ %s failed %s\", *id, err.Error()))\n\t}\n\n\tclient.RequeuedMessage()\n\n\treturn nil, nil\n}\n\nfunc (p *protocolV2) CLS(client *clientV2, params [][]byte) ([]byte, error) {\n\tif atomic.LoadInt32(&client.State) != stateSubscribed {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"cannot CLS in current state\")\n\t}\n\n\tclient.StartClose()\n\n\treturn []byte(\"CLOSE_WAIT\"), nil\n}\n\nfunc (p *protocolV2) NOP(client *clientV2, params [][]byte) ([]byte, error) {\n\treturn nil, nil\n}\n\nfunc (p *protocolV2) PUB(client *clientV2, params [][]byte) ([]byte, error) {\n\tvar err error\n\n\tif len(params) < 2 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"PUB insufficient number of parameters\")\n\t}\n\n\ttopicName := string(params[1])\n\tif !protocol.IsValidTopicName(topicName) {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_TOPIC\",\n\t\t\tfmt.Sprintf(\"PUB topic name %q is not valid\", topicName))\n\t}\n\n\tbodyLen, err := readLen(client.Reader, client.lenSlice)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_MESSAGE\", \"PUB failed to read message body size\")\n\t}\n\n\tif bodyLen <= 0 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_MESSAGE\",\n\t\t\tfmt.Sprintf(\"PUB invalid message body size %d\", bodyLen))\n\t}\n\n\tif int64(bodyLen) > p.nsqd.getOpts().MaxMsgSize {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_MESSAGE\",\n\t\t\tfmt.Sprintf(\"PUB message too big %d > %d\", bodyLen, p.nsqd.getOpts().MaxMsgSize))\n\t}\n\n\tmessageBody := make([]byte, bodyLen)\n\t_, err = io.ReadFull(client.Reader, messageBody)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_MESSAGE\", \"PUB failed to read message body\")\n\t}\n\n\tif err := p.CheckAuth(client, \"PUB\", topicName, \"\"); err != nil {\n\t\treturn nil, err\n\t}\n\n\ttopic := p.nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), messageBody)\n\terr = topic.PutMessage(msg)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_PUB_FAILED\", \"PUB failed \"+err.Error())\n\t}\n\n\tclient.PublishedMessage(topicName, 1)\n\n\treturn okBytes, nil\n}\n\nfunc (p *protocolV2) MPUB(client *clientV2, params [][]byte) ([]byte, error) {\n\tvar err error\n\n\tif len(params) < 2 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"MPUB insufficient number of parameters\")\n\t}\n\n\ttopicName := string(params[1])\n\tif !protocol.IsValidTopicName(topicName) {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_TOPIC\",\n\t\t\tfmt.Sprintf(\"E_BAD_TOPIC MPUB topic name %q is not valid\", topicName))\n\t}\n\n\tif err := p.CheckAuth(client, \"MPUB\", topicName, \"\"); err != nil {\n\t\treturn nil, err\n\t}\n\n\ttopic := p.nsqd.GetTopic(topicName)\n\n\tbodyLen, err := readLen(client.Reader, client.lenSlice)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"MPUB failed to read body size\")\n\t}\n\n\tif bodyLen <= 0 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_BODY\",\n\t\t\tfmt.Sprintf(\"MPUB invalid body size %d\", bodyLen))\n\t}\n\n\tif int64(bodyLen) > p.nsqd.getOpts().MaxBodySize {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_BODY\",\n\t\t\tfmt.Sprintf(\"MPUB body too big %d > %d\", bodyLen, p.nsqd.getOpts().MaxBodySize))\n\t}\n\n\tmessages, err := readMPUB(client.Reader, client.lenSlice, topic,\n\t\tp.nsqd.getOpts().MaxMsgSize, p.nsqd.getOpts().MaxBodySize)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// if we've made it this far we've validated all the input,\n\t// the only possible error is that the topic is exiting during\n\t// this next call (and no messages will be queued in that case)\n\terr = topic.PutMessages(messages)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_MPUB_FAILED\", \"MPUB failed \"+err.Error())\n\t}\n\n\tclient.PublishedMessage(topicName, uint64(len(messages)))\n\n\treturn okBytes, nil\n}\n\nfunc (p *protocolV2) DPUB(client *clientV2, params [][]byte) ([]byte, error) {\n\tvar err error\n\n\tif len(params) < 3 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"DPUB insufficient number of parameters\")\n\t}\n\n\ttopicName := string(params[1])\n\tif !protocol.IsValidTopicName(topicName) {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_TOPIC\",\n\t\t\tfmt.Sprintf(\"DPUB topic name %q is not valid\", topicName))\n\t}\n\n\ttimeoutMs, err := protocol.ByteToBase10(params[2])\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_INVALID\",\n\t\t\tfmt.Sprintf(\"DPUB could not parse defer timeout %s\", params[2]))\n\t}\n\ttimeoutDuration := time.Duration(timeoutMs) * time.Millisecond\n\n\tif timeoutDuration < 0 || timeoutDuration > p.nsqd.getOpts().MaxDeferTimeout {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\",\n\t\t\tfmt.Sprintf(\"DPUB defer timeout %d out of range 0-%d\",\n\t\t\t\ttimeoutMs, p.nsqd.getOpts().MaxDeferTimeout/time.Millisecond))\n\t}\n\n\tbodyLen, err := readLen(client.Reader, client.lenSlice)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_MESSAGE\", \"DPUB failed to read message body size\")\n\t}\n\n\tif bodyLen <= 0 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_MESSAGE\",\n\t\t\tfmt.Sprintf(\"DPUB invalid message body size %d\", bodyLen))\n\t}\n\n\tif int64(bodyLen) > p.nsqd.getOpts().MaxMsgSize {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_MESSAGE\",\n\t\t\tfmt.Sprintf(\"DPUB message too big %d > %d\", bodyLen, p.nsqd.getOpts().MaxMsgSize))\n\t}\n\n\tmessageBody := make([]byte, bodyLen)\n\t_, err = io.ReadFull(client.Reader, messageBody)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_MESSAGE\", \"DPUB failed to read message body\")\n\t}\n\n\tif err := p.CheckAuth(client, \"DPUB\", topicName, \"\"); err != nil {\n\t\treturn nil, err\n\t}\n\n\ttopic := p.nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), messageBody)\n\tmsg.deferred = timeoutDuration\n\terr = topic.PutMessage(msg)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_DPUB_FAILED\", \"DPUB failed \"+err.Error())\n\t}\n\n\tclient.PublishedMessage(topicName, 1)\n\n\treturn okBytes, nil\n}\n\nfunc (p *protocolV2) TOUCH(client *clientV2, params [][]byte) ([]byte, error) {\n\tstate := atomic.LoadInt32(&client.State)\n\tif state != stateSubscribed && state != stateClosing {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"cannot TOUCH in current state\")\n\t}\n\n\tif len(params) < 2 {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"TOUCH insufficient number of params\")\n\t}\n\n\tid, err := getMessageID(params[1])\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", err.Error())\n\t}\n\n\tclient.writeLock.RLock()\n\tmsgTimeout := client.MsgTimeout\n\tclient.writeLock.RUnlock()\n\terr = client.Channel.TouchMessage(client.ID, *id, msgTimeout)\n\tif err != nil {\n\t\treturn nil, protocol.NewClientErr(err, \"E_TOUCH_FAILED\",\n\t\t\tfmt.Sprintf(\"TOUCH %s failed %s\", *id, err.Error()))\n\t}\n\n\treturn nil, nil\n}\n\nfunc readMPUB(r io.Reader, tmp []byte, topic *Topic, maxMessageSize int64, maxBodySize int64) ([]*Message, error) {\n\tnumMessages, err := readLen(r, tmp)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"MPUB failed to read message count\")\n\t}\n\n\t// 4 == total num, 5 == length + min 1\n\tmaxMessages := (maxBodySize - 4) / 5\n\tif numMessages <= 0 || int64(numMessages) > maxMessages {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\",\n\t\t\tfmt.Sprintf(\"MPUB invalid message count %d\", numMessages))\n\t}\n\n\tmessages := make([]*Message, 0, numMessages)\n\tfor i := int32(0); i < numMessages; i++ {\n\t\tmessageSize, err := readLen(r, tmp)\n\t\tif err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_MESSAGE\",\n\t\t\t\tfmt.Sprintf(\"MPUB failed to read message(%d) body size\", i))\n\t\t}\n\n\t\tif messageSize <= 0 {\n\t\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_MESSAGE\",\n\t\t\t\tfmt.Sprintf(\"MPUB invalid message(%d) body size %d\", i, messageSize))\n\t\t}\n\n\t\tif int64(messageSize) > maxMessageSize {\n\t\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_MESSAGE\",\n\t\t\t\tfmt.Sprintf(\"MPUB message too big %d > %d\", messageSize, maxMessageSize))\n\t\t}\n\n\t\tmsgBody := make([]byte, messageSize)\n\t\t_, err = io.ReadFull(r, msgBody)\n\t\tif err != nil {\n\t\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_MESSAGE\", \"MPUB failed to read message body\")\n\t\t}\n\n\t\tmessages = append(messages, NewMessage(topic.GenerateID(), msgBody))\n\t}\n\n\treturn messages, nil\n}\n\n// validate and cast the bytes on the wire to a message ID\nfunc getMessageID(p []byte) (*MessageID, error) {\n\tif len(p) != MsgIDLength {\n\t\treturn nil, errors.New(\"invalid message ID\")\n\t}\n\treturn (*MessageID)(unsafe.Pointer(&p[0])), nil\n}\n\nfunc readLen(r io.Reader, tmp []byte) (int32, error) {\n\t_, err := io.ReadFull(r, tmp)\n\tif err != nil {\n\t\treturn 0, err\n\t}\n\treturn int32(binary.BigEndian.Uint32(tmp)), nil\n}\n\nfunc enforceTLSPolicy(client *clientV2, p *protocolV2, command []byte) error {\n\tif p.nsqd.getOpts().TLSRequired != TLSNotRequired && atomic.LoadInt32(&client.TLS) != 1 {\n\t\treturn protocol.NewFatalClientErr(nil, \"E_INVALID\",\n\t\t\tfmt.Sprintf(\"cannot %s in current state (TLS required)\", command))\n\t}\n\treturn nil\n}\n" }, { "path": "nsqd/protocol_v2_test.go", "content": "package nsqd\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"compress/flate\"\n\t\"crypto/tls\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"math\"\n\t\"math/rand\"\n\t\"net\"\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"net/url\"\n\t\"os\"\n\t\"runtime\"\n\t\"strconv\"\n\t\"strings\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/golang/snappy\"\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\nfunc mustStartNSQD(opts *Options) (net.Addr, net.Addr, *NSQD) {\n\topts.TCPAddress = \"127.0.0.1:0\"\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\topts.HTTPSAddress = \"127.0.0.1:0\"\n\tif opts.DataPath == \"\" {\n\t\ttmpDir, err := os.MkdirTemp(\"\", \"nsq-test-\")\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t\topts.DataPath = tmpDir\n\t}\n\tnsqd, err := New(opts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := nsqd.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\treturn nsqd.RealTCPAddr(), nsqd.RealHTTPAddr(), nsqd\n}\n\nfunc mustConnectNSQD(tcpAddr net.Addr) (net.Conn, error) {\n\tconn, err := net.DialTimeout(\"tcp\", tcpAddr.String(), time.Second)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tconn.Write(nsq.MagicV2)\n\treturn conn, nil\n}\n\nfunc identify(t *testing.T, conn io.ReadWriter, extra map[string]interface{}, f int32) []byte {\n\tci := make(map[string]interface{})\n\tci[\"client_id\"] = \"test\"\n\tci[\"feature_negotiation\"] = true\n\tfor k, v := range extra {\n\t\tci[k] = v\n\t}\n\tcmd, _ := nsq.Identify(ci)\n\t_, err := cmd.WriteTo(conn)\n\ttest.Nil(t, err)\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, err := nsq.UnpackResponse(resp)\n\ttest.Nil(t, err)\n\ttest.Equal(t, frameType, f)\n\treturn data\n}\n\nfunc sub(t *testing.T, conn io.ReadWriter, topicName string, channelName string) {\n\t_, err := nsq.Subscribe(topicName, channelName).WriteTo(conn)\n\ttest.Nil(t, err)\n\treadValidate(t, conn, frameTypeResponse, \"OK\")\n}\n\nfunc authCmd(t *testing.T, conn io.ReadWriter, authSecret string, expectSuccess string) {\n\tauth, _ := nsq.Auth(authSecret)\n\t_, err := auth.WriteTo(conn)\n\ttest.Nil(t, err)\n\tif expectSuccess != \"\" {\n\t\treadValidate(t, conn, nsq.FrameTypeResponse, expectSuccess)\n\t}\n}\n\nfunc subFail(t *testing.T, conn io.ReadWriter, topicName string, channelName string) {\n\t_, err := nsq.Subscribe(topicName, channelName).WriteTo(conn)\n\ttest.Nil(t, err)\n\tresp, _ := nsq.ReadResponse(conn)\n\tframeType, _, _ := nsq.UnpackResponse(resp)\n\ttest.Equal(t, frameTypeError, frameType)\n}\n\nfunc readValidate(t *testing.T, conn io.Reader, f int32, d string) []byte {\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, err := nsq.UnpackResponse(resp)\n\ttest.Nil(t, err)\n\ttest.Equal(t, f, frameType)\n\ttest.Equal(t, d, string(data))\n\treturn data\n}\n\n// test channel/topic names\nfunc TestChannelTopicNames(t *testing.T) {\n\ttest.Equal(t, true, protocol.IsValidChannelName(\"test\"))\n\ttest.Equal(t, true, protocol.IsValidChannelName(\"test-with_period.\"))\n\ttest.Equal(t, true, protocol.IsValidChannelName(\"test#ephemeral\"))\n\ttest.Equal(t, true, protocol.IsValidTopicName(\"test\"))\n\ttest.Equal(t, true, protocol.IsValidTopicName(\"test-with_period.\"))\n\ttest.Equal(t, true, protocol.IsValidTopicName(\"test#ephemeral\"))\n\ttest.Equal(t, false, protocol.IsValidTopicName(\"test:ephemeral\"))\n}\n\n// exercise the basic operations of the V2 protocol\nfunc TestBasicV2(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 60 * time.Second\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.PutMessage(msg)\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n\ttest.Equal(t, uint16(1), msgOut.Attempts)\n}\n\nfunc TestMultipleConsumerV2(t *testing.T) {\n\tmsgChan := make(chan *Message)\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 60 * time.Second\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_multiple_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.GetChannel(\"ch1\")\n\ttopic.GetChannel(\"ch2\")\n\ttopic.PutMessage(msg)\n\n\tfor _, i := range []string{\"1\", \"2\"} {\n\t\tconn, err := mustConnectNSQD(tcpAddr)\n\t\ttest.Nil(t, err)\n\t\tdefer conn.Close()\n\n\t\tidentify(t, conn, nil, frameTypeResponse)\n\t\tsub(t, conn, topicName, \"ch\"+i)\n\n\t\t_, err = nsq.Ready(1).WriteTo(conn)\n\t\ttest.Nil(t, err)\n\n\t\tgo func(c net.Conn) {\n\t\t\tresp, err := nsq.ReadResponse(c)\n\t\t\ttest.Nil(t, err)\n\t\t\t_, data, err := nsq.UnpackResponse(resp)\n\t\t\ttest.Nil(t, err)\n\t\t\tmsg, err := decodeMessage(data)\n\t\t\ttest.Nil(t, err)\n\t\t\tmsgChan <- msg\n\t\t}(conn)\n\t}\n\n\tmsgOut := <-msgChan\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n\ttest.Equal(t, uint16(1), msgOut.Attempts)\n\tmsgOut = <-msgChan\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n\ttest.Equal(t, uint16(1), msgOut.Attempts)\n}\n\n// TestSameZoneConsumerV2 tests that a published message goes to same-zone consumer first\n// if it's message pump is waiting\nfunc TestSameZoneConsumerV2(t *testing.T) {\n\topts := NewOptions()\n\topts.Experiments = []string{string(TopologyAwareConsumption)}\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 60 * time.Second\n\topts.TopologyRegion = \"region\"\n\topts.TopologyZone = \"zone\"\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_zone_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\ttopic.GetChannel(\"ch\")\n\n\tvar sameZone, diffZone int64\n\tvar exiting int32\n\tdone := make(chan bool, 21)\n\tfor _, zone := range []string{\"zone\", \"zone\", \"zone2\", \"zone2\"} {\n\t\tzone := zone\n\t\tconn, err := mustConnectNSQD(tcpAddr)\n\t\ttest.Nil(t, err)\n\t\tdefer conn.Close()\n\n\t\tidentify(t, conn, map[string]interface{}{\"topology_zone\": zone}, frameTypeResponse)\n\t\tsub(t, conn, topicName, \"ch\")\n\n\t\t_, err = nsq.Ready(10).WriteTo(conn)\n\t\ttest.Nil(t, err)\n\n\t\tgo func(c net.Conn, zone string) {\n\t\t\tfor {\n\t\t\t\tresp, err := nsq.ReadResponse(c)\n\t\t\t\tif atomic.LoadInt32(&exiting) == 1 {\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\ttest.Nil(t, err)\n\t\t\t\t_, data, err := nsq.UnpackResponse(resp)\n\t\t\t\ttest.Nil(t, err)\n\t\t\t\t_, err = decodeMessage(data)\n\t\t\t\ttest.Nil(t, err)\n\t\t\t\tif zone == \"zone\" {\n\t\t\t\t\tatomic.AddInt64(&sameZone, 1)\n\t\t\t\t} else {\n\t\t\t\t\tatomic.AddInt64(&diffZone, 1)\n\t\t\t\t}\n\t\t\t\tdone <- true\n\t\t\t}\n\t\t}(conn, zone)\n\t}\n\n\t// first 20 messages go to same zone (each has RDY 10)\n\t// next message goes to global memoryChan (All consumers)\n\tfor i := 0; i < 21; i++ {\n\t\ttopic.PutMessage(NewMessage(topic.GenerateID(), make([]byte, 100)))\n\t\tif i%2 == 0 {\n\t\t\t// sleep long enough for messagePump to wait again\n\t\t\ttime.Sleep(time.Millisecond)\n\t\t}\n\t}\n\tvar doneCount int64\n\tfor range done {\n\t\tdoneCount += 1\n\t\tif doneCount == 21 {\n\t\t\tbreak\n\t\t}\n\t}\n\tt.Logf(\"got same zone %d diffZone %d\", sameZone, diffZone)\n\tatomic.StoreInt32(&exiting, 1)\n\ttest.Equal(t, int64(20), sameZone)\n\ttest.Equal(t, int64(1), diffZone)\n}\n\nfunc TestClientTimeout(t *testing.T) {\n\ttopicName := \"test_client_timeout_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 150 * time.Millisecond\n\topts.LogLevel = LOG_DEBUG\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\ttime.Sleep(150 * time.Millisecond)\n\n\t// depending on timing there may be 1 or 2 hearbeats sent\n\t// just read until we get an error\n\ttimer := time.After(100 * time.Millisecond)\n\tfor {\n\t\tselect {\n\t\tcase <-timer:\n\t\t\tt.Fatalf(\"test timed out\")\n\t\tdefault:\n\t\t\t_, err := nsq.ReadResponse(conn)\n\t\t\tif err != nil {\n\t\t\t\tgoto done\n\t\t\t}\n\t\t}\n\t}\ndone:\n}\n\nfunc TestClientHeartbeat(t *testing.T) {\n\ttopicName := \"test_hb_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 200 * time.Millisecond\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(conn)\n\t_, data, _ := nsq.UnpackResponse(resp)\n\ttest.Equal(t, []byte(\"_heartbeat_\"), data)\n\n\ttime.Sleep(20 * time.Millisecond)\n\n\t_, err = nsq.Nop().WriteTo(conn)\n\ttest.Nil(t, err)\n\n\t// wait long enough that would have timed out (had we not sent the above cmd)\n\ttime.Sleep(100 * time.Millisecond)\n\n\t_, err = nsq.Nop().WriteTo(conn)\n\ttest.Nil(t, err)\n}\n\nfunc TestClientHeartbeatDisableSUB(t *testing.T) {\n\ttopicName := \"test_hb_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 200 * time.Millisecond\n\topts.LogLevel = LOG_DEBUG\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"heartbeat_interval\": -1,\n\t}, frameTypeResponse)\n\tsubFail(t, conn, topicName, \"ch\")\n}\n\nfunc TestClientHeartbeatDisable(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 100 * time.Millisecond\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"heartbeat_interval\": -1,\n\t}, frameTypeResponse)\n\n\ttime.Sleep(150 * time.Millisecond)\n\n\t_, err = nsq.Nop().WriteTo(conn)\n\ttest.Nil(t, err)\n}\n\nfunc TestMaxHeartbeatIntervalValid(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MaxHeartbeatInterval = 300 * time.Second\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\thbi := int(opts.MaxHeartbeatInterval / time.Millisecond)\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"heartbeat_interval\": hbi,\n\t}, frameTypeResponse)\n}\n\nfunc TestMaxHeartbeatIntervalInvalid(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MaxHeartbeatInterval = 300 * time.Second\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\thbi := int(opts.MaxHeartbeatInterval/time.Millisecond + 1)\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"heartbeat_interval\": hbi,\n\t}, frameTypeError)\n\ttest.Equal(t, \"E_BAD_BODY IDENTIFY heartbeat interval (300001) is invalid\", string(data))\n}\n\nfunc TestPausing(t *testing.T) {\n\ttopicName := \"test_pause_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\tchannel := topic.GetChannel(\"ch\")\n\ttopic.PutMessage(msg)\n\n\t// receive the first message via the client, finish it, and send new RDY\n\tresp, _ := nsq.ReadResponse(conn)\n\t_, data, _ := nsq.UnpackResponse(resp)\n\tmsg, _ = decodeMessage(data)\n\ttest.Equal(t, []byte(\"test body\"), msg.Body)\n\n\t_, err = nsq.Finish(nsq.MessageID(msg.ID)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\t// sleep to allow the RDY state to take effect\n\ttime.Sleep(50 * time.Millisecond)\n\n\t// pause the channel... the client shouldn't receive any more messages\n\tchannel.Pause()\n\n\t// sleep to allow the paused state to take effect\n\ttime.Sleep(50 * time.Millisecond)\n\n\tmsg = NewMessage(topic.GenerateID(), []byte(\"test body2\"))\n\ttopic.PutMessage(msg)\n\n\t// allow the client to possibly get a message, the test would hang indefinitely\n\t// if pausing was not working\n\ttime.Sleep(50 * time.Millisecond)\n\tmsg = <-channel.memoryMsgChan\n\ttest.Equal(t, []byte(\"test body2\"), msg.Body)\n\n\t// unpause the channel... the client should now be pushed a message\n\tchannel.UnPause()\n\n\tmsg = NewMessage(topic.GenerateID(), []byte(\"test body3\"))\n\ttopic.PutMessage(msg)\n\n\tresp, _ = nsq.ReadResponse(conn)\n\t_, data, _ = nsq.UnpackResponse(resp)\n\tmsg, _ = decodeMessage(data)\n\ttest.Equal(t, []byte(\"test body3\"), msg.Body)\n}\n\nfunc TestEmptyCommand(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t_, err = conn.Write([]byte(\"\\n\\n\"))\n\ttest.Nil(t, err)\n\n\t// if we didn't panic here we're good, see issue #120\n}\n\nfunc TestSizeLimits(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxMsgSize = 100\n\topts.MaxBodySize = 1000\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\ttopicName := \"test_limits_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t// PUB that's valid\n\tnsq.Publish(topicName, make([]byte, 95)).WriteTo(conn)\n\tresp, _ := nsq.ReadResponse(conn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\t// PUB that's invalid (too big)\n\tnsq.Publish(topicName, make([]byte, 105)).WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_MESSAGE PUB message too big 105 > 100\", string(data))\n\n\t// need to reconnect\n\tconn, err = mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t// PUB thats empty\n\tnsq.Publish(topicName, []byte{}).WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_MESSAGE PUB invalid message body size 0\", string(data))\n\n\t// need to reconnect\n\tconn, err = mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t// MPUB body that's valid\n\tmpub := make([][]byte, 5)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 100)\n\t}\n\tcmd, _ := nsq.MultiPublish(topicName, mpub)\n\tcmd.WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\t// MPUB body that's invalid (body too big)\n\tmpub = make([][]byte, 11)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 100)\n\t}\n\tcmd, _ = nsq.MultiPublish(topicName, mpub)\n\tcmd.WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_BODY MPUB body too big 1148 > 1000\", string(data))\n\n\t// need to reconnect\n\tconn, err = mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t// MPUB that's invalid (one message empty)\n\tmpub = make([][]byte, 5)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 100)\n\t}\n\tmpub = append(mpub, []byte{})\n\tcmd, _ = nsq.MultiPublish(topicName, mpub)\n\tcmd.WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_MESSAGE MPUB invalid message(5) body size 0\", string(data))\n\n\t// need to reconnect\n\tconn, err = mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t// MPUB body that's invalid (one of the messages is too big)\n\tmpub = make([][]byte, 5)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 101)\n\t}\n\tcmd, _ = nsq.MultiPublish(topicName, mpub)\n\tcmd.WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_MESSAGE MPUB message too big 101 > 100\", string(data))\n}\n\nfunc TestDPUB(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\ttopicName := \"test_dpub_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t// valid\n\tnsq.DeferredPublish(topicName, time.Second, make([]byte, 100)).WriteTo(conn)\n\tresp, _ := nsq.ReadResponse(conn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\ttime.Sleep(25 * time.Millisecond)\n\n\tch := nsqd.GetTopic(topicName).GetChannel(\"ch\")\n\tch.deferredMutex.Lock()\n\tnumDef := len(ch.deferredMessages)\n\tch.deferredMutex.Unlock()\n\ttest.Equal(t, 1, numDef)\n\ttest.Equal(t, 1, int(atomic.LoadUint64(&ch.messageCount)))\n\n\t// duration out of range\n\tnsq.DeferredPublish(topicName, opts.MaxDeferTimeout+100*time.Millisecond, make([]byte, 100)).WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_INVALID DPUB defer timeout 3600100 out of range 0-3600000\", string(data))\n}\n\nfunc TestTouch(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MsgTimeout = 150 * time.Millisecond\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_touch\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"ch\")\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.PutMessage(msg)\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\n\ttime.Sleep(75 * time.Millisecond)\n\n\t_, err = nsq.Touch(nsq.MessageID(msg.ID)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttime.Sleep(75 * time.Millisecond)\n\n\t_, err = nsq.Finish(nsq.MessageID(msg.ID)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, uint64(0), channel.timeoutCount)\n}\n\nfunc TestMaxRdyCount(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxRdyCount = 50\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_max_rdy_count\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.PutMessage(msg)\n\n\tdata := identify(t, conn, nil, frameTypeResponse)\n\tr := struct {\n\t\tMaxRdyCount int64 `json:\"max_rdy_count\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, int64(50), r.MaxRdyCount)\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(int(opts.MaxRdyCount)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\n\t_, err = nsq.Ready(int(opts.MaxRdyCount) + 1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err = nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\ttest.Equal(t, int32(1), frameType)\n\ttest.Equal(t, \"E_INVALID RDY count 51 out of range 0-50\", string(data))\n}\n\nfunc TestFatalError(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t_, err = conn.Write([]byte(\"ASDF\\n\"))\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\ttest.Equal(t, int32(1), frameType)\n\ttest.Equal(t, \"E_INVALID invalid command ASDF\", string(data))\n\n\t_, err = nsq.ReadResponse(conn)\n\ttest.NotNil(t, err)\n}\n\nfunc TestOutputBuffering(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxOutputBufferSize = 512 * 1024\n\topts.MaxOutputBufferTimeout = time.Second\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_output_buffering\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\toutputBufferSize := 256 * 1024\n\toutputBufferTimeout := 500\n\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), make([]byte, outputBufferSize-1024))\n\ttopic.PutMessage(msg)\n\n\tstart := time.Now()\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": outputBufferSize,\n\t\t\"output_buffer_timeout\": outputBufferTimeout,\n\t}, frameTypeResponse)\n\tvar decoded map[string]interface{}\n\tjson.Unmarshal(data, &decoded)\n\tv, ok := decoded[\"output_buffer_size\"]\n\ttest.Equal(t, true, ok)\n\ttest.Equal(t, outputBufferSize, int(v.(float64)))\n\tv = decoded[\"output_buffer_timeout\"]\n\ttest.Equal(t, outputBufferTimeout, int(v.(float64)))\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(10).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tend := time.Now()\n\n\ttest.Equal(t, true, int(end.Sub(start)/time.Millisecond) >= outputBufferTimeout)\n\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n}\n\nfunc TestOutputBufferingValidity(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxOutputBufferSize = 512 * 1024\n\topts.MaxOutputBufferTimeout = time.Second\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": 512 * 1024,\n\t\t\"output_buffer_timeout\": 1000,\n\t}, frameTypeResponse)\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": -1,\n\t\t\"output_buffer_timeout\": -1,\n\t}, frameTypeResponse)\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": 0,\n\t\t\"output_buffer_timeout\": 0,\n\t}, frameTypeResponse)\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": 512*1024 + 1,\n\t\t\"output_buffer_timeout\": 0,\n\t}, frameTypeError)\n\ttest.Equal(t, fmt.Sprintf(\"E_BAD_BODY IDENTIFY output buffer size (%d) is invalid\", 512*1024+1), string(data))\n\n\tconn, err = mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata = identify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": 0,\n\t\t\"output_buffer_timeout\": 1001,\n\t}, frameTypeError)\n\ttest.Equal(t, \"E_BAD_BODY IDENTIFY output buffer timeout (1001) is invalid\", string(data))\n}\n\nfunc TestTLS(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestTLSRequired(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\topts.TLSRequired = TLSRequiredExceptHTTP\n\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_tls_required\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tsubFail(t, conn, topicName, \"ch\")\n\n\tconn, err = mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestTLSAuthRequire(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\topts.TLSClientAuthPolicy = \"require\"\n\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\t// No Certs\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\t_, err = nsq.ReadResponse(tlsConn)\n\ttest.NotNil(t, err)\n\n\t// With Unsigned Cert\n\tconn, err = mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata = identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr = struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\n\tcert, err := tls.LoadX509KeyPair(\"./test/certs/cert.pem\", \"./test/certs/key.pem\")\n\ttest.Nil(t, err)\n\ttlsConfig = &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn = tls.Client(conn, tlsConfig)\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n}\n\nfunc TestTLSAuthRequireVerify(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\topts.TLSRootCAFile = \"./test/certs/ca.pem\"\n\topts.TLSClientAuthPolicy = \"require-verify\"\n\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\t// with no cert\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\t_, err = nsq.ReadResponse(tlsConn)\n\ttest.NotNil(t, err)\n\n\t// with invalid cert\n\tconn, err = mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata = identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr = struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\tcert, err := tls.LoadX509KeyPair(\"./test/certs/cert.pem\", \"./test/certs/key.pem\")\n\ttest.Nil(t, err)\n\ttlsConfig = &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn = tls.Client(conn, tlsConfig)\n\t_, err = nsq.ReadResponse(tlsConn)\n\ttest.NotNil(t, err)\n\n\t// with valid cert\n\tconn, err = mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata = identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr = struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\tcert, err = tls.LoadX509KeyPair(\"./test/certs/client.pem\", \"./test/certs/client.key\")\n\ttest.Nil(t, err)\n\ttlsConfig = &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn = tls.Client(conn, tlsConfig)\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestDeflate(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.DeflateEnabled = true\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"deflate\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tDeflate bool `json:\"deflate\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.Deflate)\n\n\tcompressConn := flate.NewReader(conn)\n\tresp, _ := nsq.ReadResponse(compressConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\ntype readWriter struct {\n\tio.Reader\n\tio.Writer\n}\n\nfunc TestSnappy(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.SnappyEnabled = true\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"snappy\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tSnappy bool `json:\"snappy\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.Snappy)\n\n\tcompressConn := snappy.NewReader(conn)\n\tresp, _ := nsq.ReadResponse(compressConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\tmsgBody := make([]byte, 128000)\n\t//lint:ignore SA1019 NewWriter is deprecated by NewBufferedWriter, but we don't want to buffer\n\tw := snappy.NewWriter(conn)\n\n\trw := readWriter{compressConn, w}\n\n\ttopicName := \"test_snappy\" + strconv.Itoa(int(time.Now().Unix()))\n\tsub(t, rw, topicName, \"ch\")\n\n\t_, err = nsq.Ready(1).WriteTo(rw)\n\ttest.Nil(t, err)\n\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), msgBody)\n\ttopic.PutMessage(msg)\n\n\tresp, _ = nsq.ReadResponse(compressConn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n}\n\nfunc TestTLSDeflate(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.DeflateEnabled = true\n\topts.TLSCert = \"./test/certs/cert.pem\"\n\topts.TLSKey = \"./test/certs/key.pem\"\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t\t\"deflate\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t\tDeflate bool `json:\"deflate\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\ttest.Equal(t, true, r.Deflate)\n\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\tcompressConn := flate.NewReader(tlsConn)\n\n\tresp, _ = nsq.ReadResponse(compressConn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestSampling(t *testing.T) {\n\trand.Seed(time.Now().UTC().UnixNano())\n\n\tnum := 10000\n\tsampleRate := 42\n\tslack := 5\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxRdyCount = int64(num)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"sample_rate\": int32(sampleRate),\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tSampleRate int32 `json:\"sample_rate\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, int32(sampleRate), r.SampleRate)\n\n\ttopicName := \"test_sampling\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tfor i := 0; i < num; i++ {\n\t\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\t\ttopic.PutMessage(msg)\n\t}\n\tchannel := topic.GetChannel(\"ch\")\n\n\t// let the topic drain into the channel\n\ttime.Sleep(50 * time.Millisecond)\n\n\tsub(t, conn, topicName, \"ch\")\n\t_, err = nsq.Ready(num).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tgo func() {\n\t\tfor {\n\t\t\t_, err := nsq.ReadResponse(conn)\n\t\t\tif err != nil {\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t}()\n\n\tdoneChan := make(chan int)\n\tgo func() {\n\t\tfor {\n\t\t\tif channel.Depth() == 0 {\n\t\t\t\tclose(doneChan)\n\t\t\t\treturn\n\t\t\t}\n\t\t\ttime.Sleep(5 * time.Millisecond)\n\t\t}\n\t}()\n\t<-doneChan\n\n\tchannel.inFlightMutex.Lock()\n\tnumInFlight := len(channel.inFlightMessages)\n\tchannel.inFlightMutex.Unlock()\n\n\ttest.Equal(t, true, numInFlight <= int(float64(num)*float64(sampleRate+slack)/100.0))\n\ttest.Equal(t, true, numInFlight >= int(float64(num)*float64(sampleRate-slack)/100.0))\n}\n\nfunc TestTLSSnappy(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.SnappyEnabled = true\n\topts.TLSCert = \"./test/certs/cert.pem\"\n\topts.TLSKey = \"./test/certs/key.pem\"\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t\t\"snappy\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t\tSnappy bool `json:\"snappy\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\ttest.Equal(t, true, r.Snappy)\n\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\tcompressConn := snappy.NewReader(tlsConn)\n\n\tresp, _ = nsq.ReadResponse(compressConn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestClientMsgTimeout(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.QueueScanRefreshInterval = 100 * time.Millisecond\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_cmsg_timeout\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tch := topic.GetChannel(\"ch\")\n\tmsg := NewMessage(topic.GenerateID(), make([]byte, 100))\n\ttopic.PutMessage(msg)\n\n\t// without this the race detector thinks there's a write\n\t// to msg.Attempts that races with the read in the protocol's messagePump...\n\t// it does not reflect a realistically possible condition\n\ttopic.PutMessage(NewMessage(topic.GenerateID(), make([]byte, 100)))\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"msg_timeout\": 1000,\n\t}, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\ttest.Equal(t, 0, int(atomic.LoadUint64(&ch.timeoutCount)))\n\ttest.Equal(t, 0, int(atomic.LoadUint64(&ch.requeueCount)))\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(conn)\n\t_, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n\n\t_, err = nsq.Ready(0).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttime.Sleep(1150 * time.Millisecond)\n\n\ttest.Equal(t, 1, int(atomic.LoadUint64(&ch.timeoutCount)))\n\ttest.Equal(t, 0, int(atomic.LoadUint64(&ch.requeueCount)))\n\n\t_, err = nsq.Finish(nsq.MessageID(msgOut.ID)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, fmt.Sprintf(\"E_FIN_FAILED FIN %s failed ID not in flight\", msgOut.ID),\n\t\tstring(data))\n}\n\nfunc TestBadFin(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{}, frameTypeResponse)\n\tsub(t, conn, \"test_fin\", \"ch\")\n\n\tfin := nsq.Finish(nsq.MessageID{})\n\tfin.Params[0] = []byte(\"\")\n\t_, err = fin.WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(conn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_INVALID invalid message ID\", string(data))\n}\n\nfunc TestReqTimeoutRange(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxReqTimeout = 1 * time.Minute\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_req\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"ch\")\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.PutMessage(msg)\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\n\t_, err = nsq.Requeue(nsq.MessageID(msg.ID), -1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\t// It should be immediately available for another attempt\n\tresp, err = nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tmsgOut, _ = decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\n\t// The priority (processing time) should be >= this\n\tminTs := time.Now().Add(opts.MaxReqTimeout).UnixNano()\n\n\t_, err = nsq.Requeue(nsq.MessageID(msg.ID), opts.MaxReqTimeout*2).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\tchannel.deferredMutex.Lock()\n\tpqItem := channel.deferredMessages[msg.ID]\n\tchannel.deferredMutex.Unlock()\n\n\ttest.NotNil(t, pqItem)\n\ttest.Equal(t, true, pqItem.Priority >= minTs)\n}\n\nfunc TestClientAuth(t *testing.T) {\n\tauthResponse := `{\"ttl\":1, \"authorizations\":[]}`\n\tauthSecret := \"testsecret\"\n\tauthError := \"E_UNAUTHORIZED AUTH no authorizations found\"\n\tauthSuccess := \"\"\n\ttlsEnabled := false\n\tcommonName := \"\"\n\thttpAuthRequestMethod := \"get\"\n\trunAuthTest(t, authResponse, authSecret, authError, authSuccess, tlsEnabled, commonName, httpAuthRequestMethod)\n\n\t// now one that will succeed\n\tauthResponse = `{\"ttl\":10, \"authorizations\":\n\t\t[{\"topic\":\"test\", \"channels\":[\".*\"], \"permissions\":[\"subscribe\",\"publish\"]}]\n\t}`\n\tauthError = \"\"\n\tauthSuccess = `{\"identity\":\"\",\"identity_url\":\"\",\"permission_count\":1}`\n\trunAuthTest(t, authResponse, authSecret, authError, authSuccess, tlsEnabled, commonName, httpAuthRequestMethod)\n\n\t// one with TLS enabled\n\ttlsEnabled = true\n\tcommonName = \"test.local\"\n\trunAuthTest(t, authResponse, authSecret, authError, authSuccess, tlsEnabled, commonName, httpAuthRequestMethod)\n\n\t// test POST based authentication\n\thttpAuthRequestMethod = \"post\"\n\trunAuthTest(t, authResponse, authSecret, authError, authSuccess, tlsEnabled, commonName, httpAuthRequestMethod)\n\n}\n\nfunc runAuthTest(t *testing.T, authResponse string, authSecret string, authError string,\n\tauthSuccess string, tlsEnabled bool, commonName string, httpAuthRequestMethod string) {\n\tvar err error\n\tvar expectedRemoteIP string\n\texpectedTLS := \"false\"\n\tif tlsEnabled {\n\t\texpectedTLS = \"true\"\n\t}\n\n\tauthd := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {\n\t\tt.Logf(\"in test auth handler %s\", r.RequestURI)\n\t\ttest.Equal(t, httpAuthRequestMethod, strings.ToLower(r.Method))\n\n\t\tvar values url.Values\n\n\t\tif r.Method == \"POST\" {\n\t\t\terr = json.NewDecoder(r.Body).Decode(&values)\n\t\t\tif err != nil {\n\t\t\t\tt.Error(err)\n\t\t\t}\n\t\t} else {\n\t\t\tr.ParseForm()\n\t\t\tvalues = r.Form\n\t\t}\n\t\ttest.Equal(t, expectedRemoteIP, values.Get(\"remote_ip\"))\n\t\ttest.Equal(t, expectedTLS, values.Get(\"tls\"))\n\t\ttest.Equal(t, commonName, values.Get(\"common_name\"))\n\t\ttest.Equal(t, authSecret, values.Get(\"secret\"))\n\t\tfmt.Fprint(w, authResponse)\n\t}))\n\tdefer authd.Close()\n\n\taddr, err := url.Parse(authd.URL)\n\ttest.Nil(t, err)\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.AuthHTTPAddresses = []string{addr.Host}\n\topts.AuthHTTPRequestMethod = httpAuthRequestMethod\n\tif tlsEnabled {\n\t\topts.TLSCert = \"./test/certs/server.pem\"\n\t\topts.TLSKey = \"./test/certs/server.key\"\n\t\topts.TLSClientAuthPolicy = \"require\"\n\t}\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": tlsEnabled,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, tlsEnabled, r.TLSv1)\n\n\tvar c io.ReadWriter\n\tvar tlsConn *tls.Conn\n\tc = conn\n\tif tlsEnabled {\n\t\tcert, err := tls.LoadX509KeyPair(\"./test/certs/cert.pem\", \"./test/certs/key.pem\")\n\t\ttest.Nil(t, err)\n\t\ttlsConfig := &tls.Config{\n\t\t\tCertificates: []tls.Certificate{cert},\n\t\t\tInsecureSkipVerify: true,\n\t\t}\n\t\ttlsConn = tls.Client(conn, tlsConfig)\n\t\terr = tlsConn.Handshake()\n\t\ttest.Nil(t, err)\n\t\tc = tlsConn\n\n\t\tresp, _ := nsq.ReadResponse(tlsConn)\n\t\tframeType, data, _ := nsq.UnpackResponse(resp)\n\t\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\t\ttest.Equal(t, frameTypeResponse, frameType)\n\t\ttest.Equal(t, []byte(\"OK\"), data)\n\t}\n\n\texpectedRemoteIP, _, _ = net.SplitHostPort(conn.LocalAddr().String())\n\n\tauthCmd(t, c, authSecret, authSuccess)\n\tif authError != \"\" {\n\t\treadValidate(t, c, frameTypeError, authError)\n\t} else {\n\t\tsub(t, c, \"test\", \"ch\")\n\t}\n}\n\nfunc TestIOLoopReturnsClientErrWhenSendFails(t *testing.T) {\n\tfakeConn := test.NewFakeNetConn()\n\tfakeConn.WriteFunc = func(b []byte) (int, error) {\n\t\treturn 0, errors.New(\"write error\")\n\t}\n\n\ttestIOLoopReturnsClientErr(t, fakeConn)\n}\n\nfunc TestIOLoopReturnsClientErrWhenSendSucceeds(t *testing.T) {\n\tfakeConn := test.NewFakeNetConn()\n\tfakeConn.WriteFunc = func(b []byte) (int, error) {\n\t\treturn len(b), nil\n\t}\n\n\ttestIOLoopReturnsClientErr(t, fakeConn)\n}\n\nfunc testIOLoopReturnsClientErr(t *testing.T, fakeConn test.FakeNetConn) {\n\tfakeConn.ReadFunc = func(b []byte) (int, error) {\n\t\treturn copy(b, []byte(\"INVALID_COMMAND\\n\")), nil\n\t}\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\n\tnsqd, err := New(opts)\n\ttest.Nil(t, err)\n\tprot := &protocolV2{nsqd: nsqd}\n\tdefer prot.nsqd.Exit()\n\n\tclient := prot.NewClient(fakeConn)\n\terr = prot.IOLoop(client)\n\ttest.NotNil(t, err)\n\ttest.Equal(t, \"E_INVALID invalid command INVALID_COMMAND\", err.Error())\n\ttest.NotNil(t, err.(*protocol.FatalClientErr))\n}\n\nfunc BenchmarkProtocolV2Exec(b *testing.B) {\n\tb.StopTimer()\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(b)\n\tnsqd, _ := New(opts)\n\tp := &protocolV2{nsqd}\n\tc := newClientV2(0, nil, nsqd)\n\tparams := [][]byte{[]byte(\"NOP\")}\n\tb.StartTimer()\n\n\tfor i := 0; i < b.N; i++ {\n\t\tp.Exec(c, params)\n\t}\n}\n\nfunc benchmarkProtocolV2PubMultiTopic(b *testing.B, numTopics int) {\n\tvar wg sync.WaitGroup\n\tb.StopTimer()\n\topts := NewOptions()\n\tsize := 200\n\tbatchSize := int(opts.MaxBodySize) / (size + 4)\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tmsg := make([]byte, size)\n\tbatch := make([][]byte, batchSize)\n\tfor i := range batch {\n\t\tbatch[i] = msg\n\t}\n\tb.SetBytes(int64(len(msg)))\n\tb.StartTimer()\n\n\tfor j := 0; j < numTopics; j++ {\n\t\ttopicName := fmt.Sprintf(\"bench_v2_pub_multi_topic_%d_%d\", j, time.Now().Unix())\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tconn, err := mustConnectNSQD(tcpAddr)\n\t\t\tif err != nil {\n\t\t\t\tpanic(err.Error())\n\t\t\t}\n\t\t\trw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))\n\n\t\t\tnum := b.N / numTopics / batchSize\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tfor i := 0; i < num; i++ {\n\t\t\t\t\tcmd, _ := nsq.MultiPublish(topicName, batch)\n\t\t\t\t\t_, err := cmd.WriteTo(rw)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t\terr = rw.Flush()\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tfor i := 0; i < num; i++ {\n\t\t\t\t\tresp, err := nsq.ReadResponse(rw)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t\t_, data, _ := nsq.UnpackResponse(resp)\n\t\t\t\t\tif !bytes.Equal(data, []byte(\"OK\")) {\n\t\t\t\t\t\tpanic(\"invalid response\")\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\twg.Done()\n\t\t}()\n\t}\n\n\twg.Wait()\n\n\tb.StopTimer()\n\tnsqd.Exit()\n}\n\nfunc BenchmarkProtocolV2PubMultiTopic1(b *testing.B) { benchmarkProtocolV2PubMultiTopic(b, 1) }\nfunc BenchmarkProtocolV2PubMultiTopic2(b *testing.B) { benchmarkProtocolV2PubMultiTopic(b, 2) }\nfunc BenchmarkProtocolV2PubMultiTopic4(b *testing.B) { benchmarkProtocolV2PubMultiTopic(b, 4) }\nfunc BenchmarkProtocolV2PubMultiTopic8(b *testing.B) { benchmarkProtocolV2PubMultiTopic(b, 8) }\nfunc BenchmarkProtocolV2PubMultiTopic16(b *testing.B) { benchmarkProtocolV2PubMultiTopic(b, 16) }\nfunc BenchmarkProtocolV2PubMultiTopic32(b *testing.B) { benchmarkProtocolV2PubMultiTopic(b, 32) }\n\nfunc benchmarkProtocolV2Pub(b *testing.B, size int) {\n\tvar wg sync.WaitGroup\n\tb.StopTimer()\n\topts := NewOptions()\n\tbatchSize := int(opts.MaxBodySize) / (size + 4)\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tmsg := make([]byte, size)\n\tbatch := make([][]byte, batchSize)\n\tfor i := range batch {\n\t\tbatch[i] = msg\n\t}\n\ttopicName := \"bench_v2_pub\" + strconv.Itoa(int(time.Now().Unix()))\n\tb.SetBytes(int64(len(msg)))\n\tb.StartTimer()\n\n\tfor j := 0; j < runtime.GOMAXPROCS(0); j++ {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tconn, err := mustConnectNSQD(tcpAddr)\n\t\t\tif err != nil {\n\t\t\t\tpanic(err.Error())\n\t\t\t}\n\t\t\trw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))\n\n\t\t\tnum := b.N / runtime.GOMAXPROCS(0) / batchSize\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tfor i := 0; i < num; i++ {\n\t\t\t\t\tcmd, _ := nsq.MultiPublish(topicName, batch)\n\t\t\t\t\t_, err := cmd.WriteTo(rw)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t\terr = rw.Flush()\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tfor i := 0; i < num; i++ {\n\t\t\t\t\tresp, err := nsq.ReadResponse(rw)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t\t_, data, _ := nsq.UnpackResponse(resp)\n\t\t\t\t\tif !bytes.Equal(data, []byte(\"OK\")) {\n\t\t\t\t\t\tpanic(\"invalid response\")\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\twg.Done()\n\t\t}()\n\t}\n\n\twg.Wait()\n\n\tb.StopTimer()\n\tnsqd.Exit()\n}\n\nfunc BenchmarkProtocolV2Pub256(b *testing.B) { benchmarkProtocolV2Pub(b, 256) }\nfunc BenchmarkProtocolV2Pub512(b *testing.B) { benchmarkProtocolV2Pub(b, 512) }\nfunc BenchmarkProtocolV2Pub1k(b *testing.B) { benchmarkProtocolV2Pub(b, 1024) }\nfunc BenchmarkProtocolV2Pub2k(b *testing.B) { benchmarkProtocolV2Pub(b, 2*1024) }\nfunc BenchmarkProtocolV2Pub4k(b *testing.B) { benchmarkProtocolV2Pub(b, 4*1024) }\nfunc BenchmarkProtocolV2Pub8k(b *testing.B) { benchmarkProtocolV2Pub(b, 8*1024) }\nfunc BenchmarkProtocolV2Pub16k(b *testing.B) { benchmarkProtocolV2Pub(b, 16*1024) }\nfunc BenchmarkProtocolV2Pub32k(b *testing.B) { benchmarkProtocolV2Pub(b, 32*1024) }\nfunc BenchmarkProtocolV2Pub64k(b *testing.B) { benchmarkProtocolV2Pub(b, 64*1024) }\nfunc BenchmarkProtocolV2Pub128k(b *testing.B) { benchmarkProtocolV2Pub(b, 128*1024) }\nfunc BenchmarkProtocolV2Pub256k(b *testing.B) { benchmarkProtocolV2Pub(b, 256*1024) }\nfunc BenchmarkProtocolV2Pub512k(b *testing.B) { benchmarkProtocolV2Pub(b, 512*1024) }\nfunc BenchmarkProtocolV2Pub1m(b *testing.B) { benchmarkProtocolV2Pub(b, 1024*1024) }\n\nfunc benchmarkProtocolV2Sub(b *testing.B, size int) {\n\tvar wg sync.WaitGroup\n\tb.StopTimer()\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tmsg := make([]byte, size)\n\ttopicName := \"bench_v2_sub\" + strconv.Itoa(b.N) + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tfor i := 0; i < b.N; i++ {\n\t\tmsg := NewMessage(topic.GenerateID(), msg)\n\t\ttopic.PutMessage(msg)\n\t}\n\ttopic.GetChannel(\"ch\")\n\tb.SetBytes(int64(len(msg)))\n\tgoChan := make(chan int)\n\trdyChan := make(chan int)\n\tworkers := runtime.GOMAXPROCS(0)\n\tfor j := 0; j < workers; j++ {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tsubWorker(b.N, workers, tcpAddr, topicName, rdyChan, goChan)\n\t\t\twg.Done()\n\t\t}()\n\t\t<-rdyChan\n\t}\n\tb.StartTimer()\n\n\tclose(goChan)\n\twg.Wait()\n\n\tb.StopTimer()\n\tnsqd.Exit()\n}\n\nfunc subWorker(n int, workers int, tcpAddr net.Addr, topicName string, rdyChan chan int, goChan chan int) {\n\tconn, err := mustConnectNSQD(tcpAddr)\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\trw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriterSize(conn, 65536))\n\n\tidentify(nil, conn, nil, frameTypeResponse)\n\tsub(nil, conn, topicName, \"ch\")\n\n\trdyCount := int(math.Min(math.Max(float64(n/workers), 1), 2500))\n\trdyChan <- 1\n\t<-goChan\n\tnsq.Ready(rdyCount).WriteTo(rw)\n\trw.Flush()\n\tnum := n / workers\n\tfor i := 0; i < num; i++ {\n\t\tresp, err := nsq.ReadResponse(rw)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tframeType, data, err := nsq.UnpackResponse(resp)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tif frameType != frameTypeMessage {\n\t\t\tpanic(\"got something else\")\n\t\t}\n\t\tmsg, err := decodeMessage(data)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tnsq.Finish(nsq.MessageID(msg.ID)).WriteTo(rw)\n\t\tif (i+1)%rdyCount == 0 || i+1 == num {\n\t\t\tif i+1 == num {\n\t\t\t\tnsq.Ready(0).WriteTo(conn)\n\t\t\t}\n\t\t\trw.Flush()\n\t\t}\n\t}\n}\n\nfunc BenchmarkProtocolV2Sub256(b *testing.B) { benchmarkProtocolV2Sub(b, 256) }\nfunc BenchmarkProtocolV2Sub512(b *testing.B) { benchmarkProtocolV2Sub(b, 512) }\nfunc BenchmarkProtocolV2Sub1k(b *testing.B) { benchmarkProtocolV2Sub(b, 1024) }\nfunc BenchmarkProtocolV2Sub2k(b *testing.B) { benchmarkProtocolV2Sub(b, 2*1024) }\nfunc BenchmarkProtocolV2Sub4k(b *testing.B) { benchmarkProtocolV2Sub(b, 4*1024) }\nfunc BenchmarkProtocolV2Sub8k(b *testing.B) { benchmarkProtocolV2Sub(b, 8*1024) }\nfunc BenchmarkProtocolV2Sub16k(b *testing.B) { benchmarkProtocolV2Sub(b, 16*1024) }\nfunc BenchmarkProtocolV2Sub32k(b *testing.B) { benchmarkProtocolV2Sub(b, 32*1024) }\nfunc BenchmarkProtocolV2Sub64k(b *testing.B) { benchmarkProtocolV2Sub(b, 64*1024) }\nfunc BenchmarkProtocolV2Sub128k(b *testing.B) { benchmarkProtocolV2Sub(b, 128*1024) }\nfunc BenchmarkProtocolV2Sub256k(b *testing.B) { benchmarkProtocolV2Sub(b, 256*1024) }\nfunc BenchmarkProtocolV2Sub512k(b *testing.B) { benchmarkProtocolV2Sub(b, 512*1024) }\nfunc BenchmarkProtocolV2Sub1m(b *testing.B) { benchmarkProtocolV2Sub(b, 1024*1024) }\n\nfunc benchmarkProtocolV2MultiSub(b *testing.B, num int) {\n\tvar wg sync.WaitGroup\n\tb.StopTimer()\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tmsg := make([]byte, 256)\n\tb.SetBytes(int64(len(msg) * num))\n\n\tgoChan := make(chan int)\n\trdyChan := make(chan int)\n\tworkers := runtime.GOMAXPROCS(0)\n\tfor i := 0; i < num; i++ {\n\t\ttopicName := \"bench_v2\" + strconv.Itoa(b.N) + \"_\" + strconv.Itoa(i) + \"_\" + strconv.Itoa(int(time.Now().Unix()))\n\t\ttopic := nsqd.GetTopic(topicName)\n\t\tfor i := 0; i < b.N; i++ {\n\t\t\tmsg := NewMessage(topic.GenerateID(), msg)\n\t\t\ttopic.PutMessage(msg)\n\t\t}\n\t\ttopic.GetChannel(\"ch\")\n\n\t\tfor j := 0; j < workers; j++ {\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tsubWorker(b.N, workers, tcpAddr, topicName, rdyChan, goChan)\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\t<-rdyChan\n\t\t}\n\t}\n\tb.StartTimer()\n\n\tclose(goChan)\n\twg.Wait()\n\n\tb.StopTimer()\n\tnsqd.Exit()\n}\n\nfunc BenchmarkProtocolV2MultiSub1(b *testing.B) { benchmarkProtocolV2MultiSub(b, 1) }\nfunc BenchmarkProtocolV2MultiSub2(b *testing.B) { benchmarkProtocolV2MultiSub(b, 2) }\nfunc BenchmarkProtocolV2MultiSub4(b *testing.B) { benchmarkProtocolV2MultiSub(b, 4) }\nfunc BenchmarkProtocolV2MultiSub8(b *testing.B) { benchmarkProtocolV2MultiSub(b, 8) }\nfunc BenchmarkProtocolV2MultiSub16(b *testing.B) { benchmarkProtocolV2MultiSub(b, 16) }\n" }, { "path": "nsqd/protocol_v2_unixsocket_test.go", "content": "package nsqd\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"compress/flate\"\n\t\"crypto/tls\"\n\t\"encoding/json\"\n\t\"errors\"\n\t\"fmt\"\n\t\"math\"\n\t\"math/rand\"\n\t\"net\"\n\t\"os\"\n\t\"path\"\n\t\"runtime\"\n\t\"strconv\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/golang/snappy\"\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\nfunc mustUnixSocketStartNSQD(opts *Options) (net.Addr, net.Addr, *NSQD) {\n\ttmpDir := os.TempDir()\n\topts.TCPAddress = path.Join(tmpDir, fmt.Sprintf(\"nsqd-%d.sock\", rand.Int()))\n\topts.HTTPAddress = path.Join(tmpDir, fmt.Sprintf(\"nsqd-%d.sock\", rand.Int()))\n\n\tif opts.DataPath == \"\" {\n\t\ttmpDir, err := os.MkdirTemp(\"\", \"nsq-test-\")\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t\topts.DataPath = tmpDir\n\t}\n\tnsqd, err := New(opts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := nsqd.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\treturn nsqd.RealTCPAddr(), nsqd.RealHTTPAddr(), nsqd\n}\n\nfunc mustUnixSocketConnectNSQD(addr net.Addr) (net.Conn, error) {\n\tconn, err := net.DialTimeout(\"unix\", addr.String(), time.Second)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tconn.Write(nsq.MagicV2)\n\treturn conn, nil\n}\n\n// exercise the basic operations of the V2 protocol\nfunc TestUnixSocketBasicV2(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 60 * time.Second\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.PutMessage(msg)\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n\ttest.Equal(t, uint16(1), msgOut.Attempts)\n}\n\nfunc TestUnixSocketMultipleConsumerV2(t *testing.T) {\n\tmsgChan := make(chan *Message)\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 60 * time.Second\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_multiple_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.GetChannel(\"ch1\")\n\ttopic.GetChannel(\"ch2\")\n\ttopic.PutMessage(msg)\n\n\tfor _, i := range []string{\"1\", \"2\"} {\n\t\tconn, err := mustUnixSocketConnectNSQD(addr)\n\t\ttest.Nil(t, err)\n\t\tdefer conn.Close()\n\n\t\tidentify(t, conn, nil, frameTypeResponse)\n\t\tsub(t, conn, topicName, \"ch\"+i)\n\n\t\t_, err = nsq.Ready(1).WriteTo(conn)\n\t\ttest.Nil(t, err)\n\n\t\tgo func(c net.Conn) {\n\t\t\tresp, err := nsq.ReadResponse(c)\n\t\t\ttest.Nil(t, err)\n\t\t\t_, data, err := nsq.UnpackResponse(resp)\n\t\t\ttest.Nil(t, err)\n\t\t\tmsg, err := decodeMessage(data)\n\t\t\ttest.Nil(t, err)\n\t\t\tmsgChan <- msg\n\t\t}(conn)\n\t}\n\n\tmsgOut := <-msgChan\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n\ttest.Equal(t, uint16(1), msgOut.Attempts)\n\tmsgOut = <-msgChan\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n\ttest.Equal(t, uint16(1), msgOut.Attempts)\n}\n\nfunc TestUnixSocketClientTimeout(t *testing.T) {\n\ttopicName := \"test_client_timeout_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 150 * time.Millisecond\n\topts.LogLevel = LOG_DEBUG\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\ttime.Sleep(150 * time.Millisecond)\n\n\t// depending on timing there may be 1 or 2 hearbeats sent\n\t// just read until we get an error\n\ttimer := time.After(100 * time.Millisecond)\n\tfor {\n\t\tselect {\n\t\tcase <-timer:\n\t\t\tt.Fatalf(\"test timed out\")\n\t\tdefault:\n\t\t\t_, err := nsq.ReadResponse(conn)\n\t\t\tif err != nil {\n\t\t\t\tgoto done\n\t\t\t}\n\t\t}\n\t}\ndone:\n}\n\nfunc TestUnixSocketClientHeartbeat(t *testing.T) {\n\ttopicName := \"test_hb_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 200 * time.Millisecond\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(conn)\n\t_, data, _ := nsq.UnpackResponse(resp)\n\ttest.Equal(t, []byte(\"_heartbeat_\"), data)\n\n\ttime.Sleep(20 * time.Millisecond)\n\n\t_, err = nsq.Nop().WriteTo(conn)\n\ttest.Nil(t, err)\n\n\t// wait long enough that would have timed out (had we not sent the above cmd)\n\ttime.Sleep(100 * time.Millisecond)\n\n\t_, err = nsq.Nop().WriteTo(conn)\n\ttest.Nil(t, err)\n}\n\nfunc TestUnixSocketClientHeartbeatDisableSUB(t *testing.T) {\n\ttopicName := \"test_hb_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 200 * time.Millisecond\n\topts.LogLevel = LOG_DEBUG\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"heartbeat_interval\": -1,\n\t}, frameTypeResponse)\n\tsubFail(t, conn, topicName, \"ch\")\n}\n\nfunc TestUnixSocketClientHeartbeatDisable(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.ClientTimeout = 100 * time.Millisecond\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"heartbeat_interval\": -1,\n\t}, frameTypeResponse)\n\n\ttime.Sleep(150 * time.Millisecond)\n\n\t_, err = nsq.Nop().WriteTo(conn)\n\ttest.Nil(t, err)\n}\n\nfunc TestUnixSocketMaxHeartbeatIntervalValid(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MaxHeartbeatInterval = 300 * time.Second\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\thbi := int(opts.MaxHeartbeatInterval / time.Millisecond)\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"heartbeat_interval\": hbi,\n\t}, frameTypeResponse)\n}\n\nfunc TestUnixSocketMaxHeartbeatIntervalInvalid(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MaxHeartbeatInterval = 300 * time.Second\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\thbi := int(opts.MaxHeartbeatInterval/time.Millisecond + 1)\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"heartbeat_interval\": hbi,\n\t}, frameTypeError)\n\ttest.Equal(t, \"E_BAD_BODY IDENTIFY heartbeat interval (300001) is invalid\", string(data))\n}\n\nfunc TestUnixSocketPausing(t *testing.T) {\n\ttopicName := \"test_pause_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\tchannel := topic.GetChannel(\"ch\")\n\ttopic.PutMessage(msg)\n\n\t// receive the first message via the client, finish it, and send new RDY\n\tresp, _ := nsq.ReadResponse(conn)\n\t_, data, _ := nsq.UnpackResponse(resp)\n\tmsg, _ = decodeMessage(data)\n\ttest.Equal(t, []byte(\"test body\"), msg.Body)\n\n\t_, err = nsq.Finish(nsq.MessageID(msg.ID)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\t// sleep to allow the RDY state to take effect\n\ttime.Sleep(50 * time.Millisecond)\n\n\t// pause the channel... the client shouldn't receive any more messages\n\tchannel.Pause()\n\n\t// sleep to allow the paused state to take effect\n\ttime.Sleep(50 * time.Millisecond)\n\n\tmsg = NewMessage(topic.GenerateID(), []byte(\"test body2\"))\n\ttopic.PutMessage(msg)\n\n\t// allow the client to possibly get a message, the test would hang indefinitely\n\t// if pausing was not working\n\ttime.Sleep(50 * time.Millisecond)\n\tmsg = <-channel.memoryMsgChan\n\ttest.Equal(t, []byte(\"test body2\"), msg.Body)\n\n\t// unpause the channel... the client should now be pushed a message\n\tchannel.UnPause()\n\n\tmsg = NewMessage(topic.GenerateID(), []byte(\"test body3\"))\n\ttopic.PutMessage(msg)\n\n\tresp, _ = nsq.ReadResponse(conn)\n\t_, data, _ = nsq.UnpackResponse(resp)\n\tmsg, _ = decodeMessage(data)\n\ttest.Equal(t, []byte(\"test body3\"), msg.Body)\n}\n\nfunc TestUnixSocketEmptyCommand(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t_, err = conn.Write([]byte(\"\\n\\n\"))\n\ttest.Nil(t, err)\n\n\t// if we didn't panic here we're good, see issue #120\n}\n\nfunc TestUnixSocketSizeLimits(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxMsgSize = 100\n\topts.MaxBodySize = 1000\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\ttopicName := \"test_limits_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t// PUB that's valid\n\tnsq.Publish(topicName, make([]byte, 95)).WriteTo(conn)\n\tresp, _ := nsq.ReadResponse(conn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\t// PUB that's invalid (too big)\n\tnsq.Publish(topicName, make([]byte, 105)).WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_MESSAGE PUB message too big 105 > 100\", string(data))\n\n\t// need to reconnect\n\tconn, err = mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t// PUB thats empty\n\tnsq.Publish(topicName, []byte{}).WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_MESSAGE PUB invalid message body size 0\", string(data))\n\n\t// need to reconnect\n\tconn, err = mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t// MPUB body that's valid\n\tmpub := make([][]byte, 5)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 100)\n\t}\n\tcmd, _ := nsq.MultiPublish(topicName, mpub)\n\tcmd.WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\t// MPUB body that's invalid (body too big)\n\tmpub = make([][]byte, 11)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 100)\n\t}\n\tcmd, _ = nsq.MultiPublish(topicName, mpub)\n\tcmd.WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_BODY MPUB body too big 1148 > 1000\", string(data))\n\n\t// need to reconnect\n\tconn, err = mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t// MPUB that's invalid (one message empty)\n\tmpub = make([][]byte, 5)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 100)\n\t}\n\tmpub = append(mpub, []byte{})\n\tcmd, _ = nsq.MultiPublish(topicName, mpub)\n\tcmd.WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_MESSAGE MPUB invalid message(5) body size 0\", string(data))\n\n\t// need to reconnect\n\tconn, err = mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t// MPUB body that's invalid (one of the messages is too big)\n\tmpub = make([][]byte, 5)\n\tfor i := range mpub {\n\t\tmpub[i] = make([]byte, 101)\n\t}\n\tcmd, _ = nsq.MultiPublish(topicName, mpub)\n\tcmd.WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_BAD_MESSAGE MPUB message too big 101 > 100\", string(data))\n}\n\nfunc TestUnixSocketDPUB(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\ttopicName := \"test_dpub_v2\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\t// valid\n\tnsq.DeferredPublish(topicName, time.Second, make([]byte, 100)).WriteTo(conn)\n\tresp, _ := nsq.ReadResponse(conn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\ttime.Sleep(25 * time.Millisecond)\n\n\tch := nsqd.GetTopic(topicName).GetChannel(\"ch\")\n\tch.deferredMutex.Lock()\n\tnumDef := len(ch.deferredMessages)\n\tch.deferredMutex.Unlock()\n\ttest.Equal(t, 1, numDef)\n\ttest.Equal(t, 1, int(atomic.LoadUint64(&ch.messageCount)))\n\n\t// duration out of range\n\tnsq.DeferredPublish(topicName, opts.MaxDeferTimeout+100*time.Millisecond, make([]byte, 100)).WriteTo(conn)\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_INVALID DPUB defer timeout 3600100 out of range 0-3600000\", string(data))\n}\n\nfunc TestUnixSocketTouch(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MsgTimeout = 150 * time.Millisecond\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_touch\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"ch\")\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.PutMessage(msg)\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\n\ttime.Sleep(75 * time.Millisecond)\n\n\t_, err = nsq.Touch(nsq.MessageID(msg.ID)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttime.Sleep(75 * time.Millisecond)\n\n\t_, err = nsq.Finish(nsq.MessageID(msg.ID)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, uint64(0), channel.timeoutCount)\n}\n\nfunc TestUnixSocketMaxRdyCount(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxRdyCount = 50\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_max_rdy_count\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.PutMessage(msg)\n\n\tdata := identify(t, conn, nil, frameTypeResponse)\n\tr := struct {\n\t\tMaxRdyCount int64 `json:\"max_rdy_count\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, int64(50), r.MaxRdyCount)\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(int(opts.MaxRdyCount)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\n\t_, err = nsq.Ready(int(opts.MaxRdyCount) + 1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err = nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\ttest.Equal(t, int32(1), frameType)\n\ttest.Equal(t, \"E_INVALID RDY count 51 out of range 0-50\", string(data))\n}\n\nfunc TestUnixSocketFatalError(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\t_, err = conn.Write([]byte(\"ASDF\\n\"))\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\ttest.Equal(t, int32(1), frameType)\n\ttest.Equal(t, \"E_INVALID invalid command ASDF\", string(data))\n\n\t_, err = nsq.ReadResponse(conn)\n\ttest.NotNil(t, err)\n}\n\nfunc TestUnixSocketOutputBuffering(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxOutputBufferSize = 512 * 1024\n\topts.MaxOutputBufferTimeout = time.Second\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_output_buffering\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\toutputBufferSize := 256 * 1024\n\toutputBufferTimeout := 500\n\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), make([]byte, outputBufferSize-1024))\n\ttopic.PutMessage(msg)\n\n\tstart := time.Now()\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": outputBufferSize,\n\t\t\"output_buffer_timeout\": outputBufferTimeout,\n\t}, frameTypeResponse)\n\tvar decoded map[string]interface{}\n\tjson.Unmarshal(data, &decoded)\n\tv, ok := decoded[\"output_buffer_size\"]\n\ttest.Equal(t, true, ok)\n\ttest.Equal(t, outputBufferSize, int(v.(float64)))\n\tv = decoded[\"output_buffer_timeout\"]\n\ttest.Equal(t, outputBufferTimeout, int(v.(float64)))\n\tsub(t, conn, topicName, \"ch\")\n\n\t_, err = nsq.Ready(10).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tend := time.Now()\n\n\ttest.Equal(t, true, int(end.Sub(start)/time.Millisecond) >= outputBufferTimeout)\n\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n}\n\nfunc TestUnixSocketOutputBufferingValidity(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxOutputBufferSize = 512 * 1024\n\topts.MaxOutputBufferTimeout = time.Second\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": 512 * 1024,\n\t\t\"output_buffer_timeout\": 1000,\n\t}, frameTypeResponse)\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": -1,\n\t\t\"output_buffer_timeout\": -1,\n\t}, frameTypeResponse)\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": 0,\n\t\t\"output_buffer_timeout\": 0,\n\t}, frameTypeResponse)\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": 512*1024 + 1,\n\t\t\"output_buffer_timeout\": 0,\n\t}, frameTypeError)\n\ttest.Equal(t, fmt.Sprintf(\"E_BAD_BODY IDENTIFY output buffer size (%d) is invalid\", 512*1024+1), string(data))\n\n\tconn, err = mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata = identify(t, conn, map[string]interface{}{\n\t\t\"output_buffer_size\": 0,\n\t\t\"output_buffer_timeout\": 1001,\n\t}, frameTypeError)\n\ttest.Equal(t, \"E_BAD_BODY IDENTIFY output buffer timeout (1001) is invalid\", string(data))\n}\n\nfunc TestUnixSocketTLS(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestUnixSocketTLSRequired(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\topts.TLSRequired = TLSRequiredExceptHTTP\n\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_tls_required\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tsubFail(t, conn, topicName, \"ch\")\n\n\tconn, err = mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestUnixSocketTLSAuthRequire(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\topts.TLSClientAuthPolicy = \"require\"\n\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\t// No Certs\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\t_, err = nsq.ReadResponse(tlsConn)\n\ttest.NotNil(t, err)\n\n\t// With Unsigned Cert\n\tconn, err = mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata = identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr = struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\n\tcert, err := tls.LoadX509KeyPair(\"./test/certs/cert.pem\", \"./test/certs/key.pem\")\n\ttest.Nil(t, err)\n\ttlsConfig = &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn = tls.Client(conn, tlsConfig)\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n}\n\nfunc TestUnixSocketTLSAuthRequireVerify(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TLSCert = \"./test/certs/server.pem\"\n\topts.TLSKey = \"./test/certs/server.key\"\n\topts.TLSRootCAFile = \"./test/certs/ca.pem\"\n\topts.TLSClientAuthPolicy = \"require-verify\"\n\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\t// with no cert\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\t_, err = nsq.ReadResponse(tlsConn)\n\ttest.NotNil(t, err)\n\n\t// with invalid cert\n\tconn, err = mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata = identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr = struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\tcert, err := tls.LoadX509KeyPair(\"./test/certs/cert.pem\", \"./test/certs/key.pem\")\n\ttest.Nil(t, err)\n\ttlsConfig = &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn = tls.Client(conn, tlsConfig)\n\t_, err = nsq.ReadResponse(tlsConn)\n\ttest.NotNil(t, err)\n\n\t// with valid cert\n\tconn, err = mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata = identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t}, frameTypeResponse)\n\tr = struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\tcert, err = tls.LoadX509KeyPair(\"./test/certs/client.pem\", \"./test/certs/client.key\")\n\ttest.Nil(t, err)\n\ttlsConfig = &tls.Config{\n\t\tCertificates: []tls.Certificate{cert},\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn = tls.Client(conn, tlsConfig)\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestUnixSocketDeflate(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.DeflateEnabled = true\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"deflate\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tDeflate bool `json:\"deflate\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.Deflate)\n\n\tcompressConn := flate.NewReader(conn)\n\tresp, _ := nsq.ReadResponse(compressConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestUnixSocketSnappy(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.SnappyEnabled = true\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"snappy\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tSnappy bool `json:\"snappy\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.Snappy)\n\n\tcompressConn := snappy.NewReader(conn)\n\tresp, _ := nsq.ReadResponse(compressConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\tmsgBody := make([]byte, 128000)\n\t//lint:ignore SA1019 NewWriter is deprecated by NewBufferedWriter, but we don't want to buffer\n\tw := snappy.NewWriter(conn)\n\n\trw := readWriter{compressConn, w}\n\n\ttopicName := \"test_snappy\" + strconv.Itoa(int(time.Now().Unix()))\n\tsub(t, rw, topicName, \"ch\")\n\n\t_, err = nsq.Ready(1).WriteTo(rw)\n\ttest.Nil(t, err)\n\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), msgBody)\n\ttopic.PutMessage(msg)\n\n\tresp, _ = nsq.ReadResponse(compressConn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n}\n\nfunc TestUnixSocketTLSDeflate(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.DeflateEnabled = true\n\topts.TLSCert = \"./test/certs/cert.pem\"\n\topts.TLSKey = \"./test/certs/key.pem\"\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t\t\"deflate\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t\tDeflate bool `json:\"deflate\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\ttest.Equal(t, true, r.Deflate)\n\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\tcompressConn := flate.NewReader(tlsConn)\n\n\tresp, _ = nsq.ReadResponse(compressConn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestUnixSocketSampling(t *testing.T) {\n\trand.Seed(time.Now().UTC().UnixNano())\n\n\tnum := 10000\n\tsampleRate := 42\n\tslack := 5\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxRdyCount = int64(num)\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"sample_rate\": int32(sampleRate),\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tSampleRate int32 `json:\"sample_rate\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, int32(sampleRate), r.SampleRate)\n\n\ttopicName := \"test_sampling\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tfor i := 0; i < num; i++ {\n\t\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\t\ttopic.PutMessage(msg)\n\t}\n\tchannel := topic.GetChannel(\"ch\")\n\n\t// let the topic drain into the channel\n\ttime.Sleep(50 * time.Millisecond)\n\n\tsub(t, conn, topicName, \"ch\")\n\t_, err = nsq.Ready(num).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tgo func() {\n\t\tfor {\n\t\t\t_, err := nsq.ReadResponse(conn)\n\t\t\tif err != nil {\n\t\t\t\treturn\n\t\t\t}\n\t\t}\n\t}()\n\n\tdoneChan := make(chan int)\n\tgo func() {\n\t\tfor {\n\t\t\tif channel.Depth() == 0 {\n\t\t\t\tclose(doneChan)\n\t\t\t\treturn\n\t\t\t}\n\t\t\ttime.Sleep(5 * time.Millisecond)\n\t\t}\n\t}()\n\t<-doneChan\n\n\tchannel.inFlightMutex.Lock()\n\tnumInFlight := len(channel.inFlightMessages)\n\tchannel.inFlightMutex.Unlock()\n\n\ttest.Equal(t, true, numInFlight <= int(float64(num)*float64(sampleRate+slack)/100.0))\n\ttest.Equal(t, true, numInFlight >= int(float64(num)*float64(sampleRate-slack)/100.0))\n}\n\nfunc TestUnixSocketTLSSnappy(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.SnappyEnabled = true\n\topts.TLSCert = \"./test/certs/cert.pem\"\n\topts.TLSKey = \"./test/certs/key.pem\"\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"tls_v1\": true,\n\t\t\"snappy\": true,\n\t}, frameTypeResponse)\n\tr := struct {\n\t\tTLSv1 bool `json:\"tls_v1\"`\n\t\tSnappy bool `json:\"snappy\"`\n\t}{}\n\terr = json.Unmarshal(data, &r)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, r.TLSv1)\n\ttest.Equal(t, true, r.Snappy)\n\n\ttlsConfig := &tls.Config{\n\t\tInsecureSkipVerify: true,\n\t}\n\ttlsConn := tls.Client(conn, tlsConfig)\n\n\terr = tlsConn.Handshake()\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(tlsConn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n\n\tcompressConn := snappy.NewReader(tlsConn)\n\n\tresp, _ = nsq.ReadResponse(compressConn)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tt.Logf(\"frameType: %d, data: %s\", frameType, data)\n\ttest.Equal(t, frameTypeResponse, frameType)\n\ttest.Equal(t, []byte(\"OK\"), data)\n}\n\nfunc TestUnixSocketClientMsgTimeout(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.QueueScanRefreshInterval = 100 * time.Millisecond\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_cmsg_timeout\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tch := topic.GetChannel(\"ch\")\n\tmsg := NewMessage(topic.GenerateID(), make([]byte, 100))\n\ttopic.PutMessage(msg)\n\n\t// without this the race detector thinks there's a write\n\t// to msg.Attempts that races with the read in the protocol's messagePump...\n\t// it does not reflect a realistically possible condition\n\ttopic.PutMessage(NewMessage(topic.GenerateID(), make([]byte, 100)))\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{\n\t\t\"msg_timeout\": 1000,\n\t}, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\ttest.Equal(t, 0, int(atomic.LoadUint64(&ch.timeoutCount)))\n\ttest.Equal(t, 0, int(atomic.LoadUint64(&ch.requeueCount)))\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(conn)\n\t_, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\ttest.Equal(t, msg.Body, msgOut.Body)\n\n\t_, err = nsq.Ready(0).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttime.Sleep(1150 * time.Millisecond)\n\n\ttest.Equal(t, 1, int(atomic.LoadUint64(&ch.timeoutCount)))\n\ttest.Equal(t, 0, int(atomic.LoadUint64(&ch.requeueCount)))\n\n\t_, err = nsq.Finish(nsq.MessageID(msgOut.ID)).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, _ = nsq.ReadResponse(conn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, fmt.Sprintf(\"E_FIN_FAILED FIN %s failed ID not in flight\", msgOut.ID),\n\t\tstring(data))\n}\n\nfunc TestUnixSocketBadFin(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, map[string]interface{}{}, frameTypeResponse)\n\tsub(t, conn, \"test_fin\", \"ch\")\n\n\tfin := nsq.Finish(nsq.MessageID{})\n\tfin.Params[0] = []byte(\"\")\n\t_, err = fin.WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, _ := nsq.ReadResponse(conn)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\ttest.Equal(t, frameTypeError, frameType)\n\ttest.Equal(t, \"E_INVALID invalid message ID\", string(data))\n}\n\nfunc TestUnixSocketReqTimeoutRange(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.MaxReqTimeout = 1 * time.Minute\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_req\" + strconv.Itoa(int(time.Now().Unix()))\n\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"ch\")\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.PutMessage(msg)\n\n\t_, err = nsq.Ready(1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\tresp, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ := nsq.UnpackResponse(resp)\n\tmsgOut, _ := decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\n\t_, err = nsq.Requeue(nsq.MessageID(msg.ID), -1).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\t// It should be immediately available for another attempt\n\tresp, err = nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\tframeType, data, _ = nsq.UnpackResponse(resp)\n\tmsgOut, _ = decodeMessage(data)\n\ttest.Equal(t, frameTypeMessage, frameType)\n\ttest.Equal(t, msg.ID, msgOut.ID)\n\n\t// The priority (processing time) should be >= this\n\tminTs := time.Now().Add(opts.MaxReqTimeout).UnixNano()\n\n\t_, err = nsq.Requeue(nsq.MessageID(msg.ID), opts.MaxReqTimeout*2).WriteTo(conn)\n\ttest.Nil(t, err)\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\tchannel.deferredMutex.Lock()\n\tpqItem := channel.deferredMessages[msg.ID]\n\tchannel.deferredMutex.Unlock()\n\n\ttest.NotNil(t, pqItem)\n\ttest.Equal(t, true, pqItem.Priority >= minTs)\n}\n\nfunc TestUnixSocketIOLoopReturnsClientErrWhenSendFails(t *testing.T) {\n\tfakeConn := test.NewFakeNetConn()\n\tfakeConn.WriteFunc = func(b []byte) (int, error) {\n\t\treturn 0, errors.New(\"write error\")\n\t}\n\n\ttestUnixSocketIOLoopReturnsClientErr(t, fakeConn)\n}\n\nfunc TestUnixSocketIOLoopReturnsClientErrWhenSendSucceeds(t *testing.T) {\n\tfakeConn := test.NewFakeNetConn()\n\tfakeConn.WriteFunc = func(b []byte) (int, error) {\n\t\treturn len(b), nil\n\t}\n\n\ttestUnixSocketIOLoopReturnsClientErr(t, fakeConn)\n}\n\nfunc testUnixSocketIOLoopReturnsClientErr(t *testing.T, fakeConn test.FakeNetConn) {\n\tfakeConn.ReadFunc = func(b []byte) (int, error) {\n\t\treturn copy(b, []byte(\"INVALID_COMMAND\\n\")), nil\n\t}\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\n\tnsqd, err := New(opts)\n\ttest.Nil(t, err)\n\tprot := &protocolV2{nsqd: nsqd}\n\tdefer prot.nsqd.Exit()\n\n\tclient := prot.NewClient(fakeConn)\n\terr = prot.IOLoop(client)\n\ttest.NotNil(t, err)\n\ttest.Equal(t, \"E_INVALID invalid command INVALID_COMMAND\", err.Error())\n\ttest.NotNil(t, err.(*protocol.FatalClientErr))\n}\n\nfunc BenchmarkUnixSocketProtocolV2Exec(b *testing.B) {\n\tb.StopTimer()\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(b)\n\tnsqd, _ := New(opts)\n\tp := &protocolV2{nsqd}\n\tc := newClientV2(0, nil, nsqd)\n\tparams := [][]byte{[]byte(\"NOP\")}\n\tb.StartTimer()\n\n\tfor i := 0; i < b.N; i++ {\n\t\tp.Exec(c, params)\n\t}\n}\n\nfunc benchmarkUnixSocketProtocolV2PubMultiTopic(b *testing.B, numTopics int) {\n\tvar wg sync.WaitGroup\n\tb.StopTimer()\n\topts := NewOptions()\n\tsize := 200\n\tbatchSize := int(opts.MaxBodySize) / (size + 4)\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tmsg := make([]byte, size)\n\tbatch := make([][]byte, batchSize)\n\tfor i := range batch {\n\t\tbatch[i] = msg\n\t}\n\tb.SetBytes(int64(len(msg)))\n\tb.StartTimer()\n\n\tfor j := 0; j < numTopics; j++ {\n\t\ttopicName := fmt.Sprintf(\"bench_v2_pub_multi_topic_%d_%d\", j, time.Now().Unix())\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tconn, err := mustUnixSocketConnectNSQD(addr)\n\t\t\tif err != nil {\n\t\t\t\tpanic(err.Error())\n\t\t\t}\n\t\t\trw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))\n\n\t\t\tnum := b.N / numTopics / batchSize\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tfor i := 0; i < num; i++ {\n\t\t\t\t\tcmd, _ := nsq.MultiPublish(topicName, batch)\n\t\t\t\t\t_, err := cmd.WriteTo(rw)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t\terr = rw.Flush()\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tfor i := 0; i < num; i++ {\n\t\t\t\t\tresp, err := nsq.ReadResponse(rw)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t\t_, data, _ := nsq.UnpackResponse(resp)\n\t\t\t\t\tif !bytes.Equal(data, []byte(\"OK\")) {\n\t\t\t\t\t\tpanic(\"invalid response\")\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\twg.Done()\n\t\t}()\n\t}\n\n\twg.Wait()\n\n\tb.StopTimer()\n\tnsqd.Exit()\n}\n\nfunc BenchmarkUnixSocketProtocolV2PubMultiTopic1(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2PubMultiTopic(b, 1)\n}\nfunc BenchmarkUnixSocketkProtocolV2PubMultiTopic2(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2PubMultiTopic(b, 2)\n}\nfunc BenchmarkUnixSocketProtocolV2PubMultiTopic4(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2PubMultiTopic(b, 4)\n}\nfunc BenchmarkUnixSocketProtocolV2PubMultiTopic8(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2PubMultiTopic(b, 8)\n}\nfunc BenchmarkUnixSocketProtocolV2PubMultiTopic16(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2PubMultiTopic(b, 16)\n}\nfunc BenchmarkUnixSocketProtocolV2PubMultiTopic32(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2PubMultiTopic(b, 32)\n}\n\nfunc benchmarkUnixSocketProtocolV2Pub(b *testing.B, size int) {\n\tvar wg sync.WaitGroup\n\tb.StopTimer()\n\topts := NewOptions()\n\tbatchSize := int(opts.MaxBodySize) / (size + 4)\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tmsg := make([]byte, size)\n\tbatch := make([][]byte, batchSize)\n\tfor i := range batch {\n\t\tbatch[i] = msg\n\t}\n\ttopicName := \"bench_v2_pub\" + strconv.Itoa(int(time.Now().Unix()))\n\tb.SetBytes(int64(len(msg)))\n\tb.StartTimer()\n\n\tfor j := 0; j < runtime.GOMAXPROCS(0); j++ {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tconn, err := mustUnixSocketConnectNSQD(addr)\n\t\t\tif err != nil {\n\t\t\t\tpanic(err.Error())\n\t\t\t}\n\t\t\trw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))\n\n\t\t\tnum := b.N / runtime.GOMAXPROCS(0) / batchSize\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tfor i := 0; i < num; i++ {\n\t\t\t\t\tcmd, _ := nsq.MultiPublish(topicName, batch)\n\t\t\t\t\t_, err := cmd.WriteTo(rw)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t\terr = rw.Flush()\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tfor i := 0; i < num; i++ {\n\t\t\t\t\tresp, err := nsq.ReadResponse(rw)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\tpanic(err.Error())\n\t\t\t\t\t}\n\t\t\t\t\t_, data, _ := nsq.UnpackResponse(resp)\n\t\t\t\t\tif !bytes.Equal(data, []byte(\"OK\")) {\n\t\t\t\t\t\tpanic(\"invalid response\")\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\twg.Done()\n\t\t}()\n\t}\n\n\twg.Wait()\n\n\tb.StopTimer()\n\tnsqd.Exit()\n}\n\nfunc BenchmarkUnixSocketProtocolV2Pub256(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 256) }\nfunc BenchmarkUnixSocketProtocolV2Pub512(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 512) }\nfunc BenchmarkUnixSocketProtocolV2Pub1k(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 1024) }\nfunc BenchmarkUnixSocketProtocolV2Pub2k(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 2*1024) }\nfunc BenchmarkUnixSocketProtocolV2Pub4k(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 4*1024) }\nfunc BenchmarkUnixSocketProtocolV2Pub8k(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 8*1024) }\nfunc BenchmarkUnixSocketProtocolV2Pub16k(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 16*1024) }\nfunc BenchmarkUnixSocketProtocolV2Pub32k(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 32*1024) }\nfunc BenchmarkUnixSocketProtocolV2Pub64k(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 64*1024) }\nfunc BenchmarkUnixSocketProtocolV2Pub128k(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2Pub(b, 128*1024)\n}\nfunc BenchmarkUnixSocketProtocolV2Pub256k(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2Pub(b, 256*1024)\n}\nfunc BenchmarkUnixSocketProtocolV2Pub512k(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2Pub(b, 512*1024)\n}\nfunc BenchmarkUnixSocketProtocolV2Pub1m(b *testing.B) { benchmarkUnixSocketProtocolV2Pub(b, 1024*1024) }\n\nfunc benchmarkUnixSocketProtocolV2Sub(b *testing.B, size int) {\n\tvar wg sync.WaitGroup\n\tb.StopTimer()\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tmsg := make([]byte, size)\n\ttopicName := \"bench_v2_sub\" + strconv.Itoa(b.N) + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tfor i := 0; i < b.N; i++ {\n\t\tmsg := NewMessage(topic.GenerateID(), msg)\n\t\ttopic.PutMessage(msg)\n\t}\n\ttopic.GetChannel(\"ch\")\n\tb.SetBytes(int64(len(msg)))\n\tgoChan := make(chan int)\n\trdyChan := make(chan int)\n\tworkers := runtime.GOMAXPROCS(0)\n\tfor j := 0; j < workers; j++ {\n\t\twg.Add(1)\n\t\tgo func() {\n\t\t\tsubWorker(b.N, workers, addr, topicName, rdyChan, goChan)\n\t\t\twg.Done()\n\t\t}()\n\t\t<-rdyChan\n\t}\n\tb.StartTimer()\n\n\tclose(goChan)\n\twg.Wait()\n\n\tb.StopTimer()\n\tnsqd.Exit()\n}\n\nfunc subUnixSocketWorker(n int, workers int, addr net.Addr, topicName string, rdyChan chan int, goChan chan int) {\n\tconn, err := mustUnixSocketConnectNSQD(addr)\n\tif err != nil {\n\t\tpanic(err.Error())\n\t}\n\trw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriterSize(conn, 65536))\n\n\tidentify(nil, conn, nil, frameTypeResponse)\n\tsub(nil, conn, topicName, \"ch\")\n\n\trdyCount := int(math.Min(math.Max(float64(n/workers), 1), 2500))\n\trdyChan <- 1\n\t<-goChan\n\tnsq.Ready(rdyCount).WriteTo(rw)\n\trw.Flush()\n\tnum := n / workers\n\tfor i := 0; i < num; i++ {\n\t\tresp, err := nsq.ReadResponse(rw)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tframeType, data, err := nsq.UnpackResponse(resp)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tif frameType != frameTypeMessage {\n\t\t\tpanic(\"got something else\")\n\t\t}\n\t\tmsg, err := decodeMessage(data)\n\t\tif err != nil {\n\t\t\tpanic(err.Error())\n\t\t}\n\t\tnsq.Finish(nsq.MessageID(msg.ID)).WriteTo(rw)\n\t\tif (i+1)%rdyCount == 0 || i+1 == num {\n\t\t\tif i+1 == num {\n\t\t\t\tnsq.Ready(0).WriteTo(conn)\n\t\t\t}\n\t\t\trw.Flush()\n\t\t}\n\t}\n}\n\nfunc BenchmarkUnixSocketProtocolV2Sub256(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 256) }\nfunc BenchmarkUnixSocketProtocolV2Sub512(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 512) }\nfunc BenchmarkUnixSocketProtocolV2Sub1k(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 1024) }\nfunc BenchmarkUnixSocketProtocolV2Sub2k(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 2*1024) }\nfunc BenchmarkUnixSocketProtocolV2Sub4k(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 4*1024) }\nfunc BenchmarkUnixSocketProtocolV2Sub8k(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 8*1024) }\nfunc BenchmarkUnixSocketProtocolV2Sub16k(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 16*1024) }\nfunc BenchmarkUnixSocketProtocolV2Sub32k(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 32*1024) }\nfunc BenchmarkUnixSocketProtocolV2Sub64k(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 64*1024) }\nfunc BenchmarkUnixSocketProtocolV2Sub128k(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2Sub(b, 128*1024)\n}\nfunc BenchmarkUnixSocketProtocolV2Sub256k(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2Sub(b, 256*1024)\n}\nfunc BenchmarkUnixSocketProtocolV2Sub512k(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2Sub(b, 512*1024)\n}\nfunc BenchmarkUnixSocketProtocolV2Sub1m(b *testing.B) { benchmarkUnixSocketProtocolV2Sub(b, 1024*1024) }\n\nfunc benchmarkUnixSocketProtocolV2MultiSub(b *testing.B, num int) {\n\tvar wg sync.WaitGroup\n\tb.StopTimer()\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\taddr, _, nsqd := mustUnixSocketStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tmsg := make([]byte, 256)\n\tb.SetBytes(int64(len(msg) * num))\n\n\tgoChan := make(chan int)\n\trdyChan := make(chan int)\n\tworkers := runtime.GOMAXPROCS(0)\n\tfor i := 0; i < num; i++ {\n\t\ttopicName := \"bench_v2\" + strconv.Itoa(b.N) + \"_\" + strconv.Itoa(i) + \"_\" + strconv.Itoa(int(time.Now().Unix()))\n\t\ttopic := nsqd.GetTopic(topicName)\n\t\tfor i := 0; i < b.N; i++ {\n\t\t\tmsg := NewMessage(topic.GenerateID(), msg)\n\t\t\ttopic.PutMessage(msg)\n\t\t}\n\t\ttopic.GetChannel(\"ch\")\n\n\t\tfor j := 0; j < workers; j++ {\n\t\t\twg.Add(1)\n\t\t\tgo func() {\n\t\t\t\tsubUnixSocketWorker(b.N, workers, addr, topicName, rdyChan, goChan)\n\t\t\t\twg.Done()\n\t\t\t}()\n\t\t\t<-rdyChan\n\t\t}\n\t}\n\tb.StartTimer()\n\n\tclose(goChan)\n\twg.Wait()\n\n\tb.StopTimer()\n\tnsqd.Exit()\n}\n\nfunc BenchmarkUnixSocketProtocolV2MultiSub2(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2MultiSub(b, 2)\n}\nfunc BenchmarkUnixSocketProtocolV2MultiSub1(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2MultiSub(b, 1)\n}\nfunc BenchmarkUnixSocketProtocolV2MultiSub4(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2MultiSub(b, 4)\n}\nfunc BenchmarkUnixSocketProtocolV2MultiSub8(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2MultiSub(b, 8)\n}\nfunc BenchmarkUnixSocketProtocolV2MultiSub16(b *testing.B) {\n\tbenchmarkUnixSocketProtocolV2MultiSub(b, 16)\n}\n" }, { "path": "nsqd/stats.go", "content": "package nsqd\n\nimport (\n\t\"runtime\"\n\t\"sort\"\n\t\"sync/atomic\"\n\n\t\"github.com/nsqio/nsq/internal/quantile\"\n)\n\ntype Stats struct {\n\tTopics []TopicStats\n\tProducers []ClientStats\n}\n\ntype ClientStats interface {\n\tString() string\n}\n\ntype TopicStats struct {\n\tTopicName string `json:\"topic_name\"`\n\tChannels []ChannelStats `json:\"channels\"`\n\tDepth int64 `json:\"depth\"`\n\tBackendDepth int64 `json:\"backend_depth\"`\n\tMessageCount uint64 `json:\"message_count\"`\n\tMessageBytes uint64 `json:\"message_bytes\"`\n\tPaused bool `json:\"paused\"`\n\n\tE2eProcessingLatency *quantile.Result `json:\"e2e_processing_latency\"`\n}\n\nfunc NewTopicStats(t *Topic, channels []ChannelStats) TopicStats {\n\treturn TopicStats{\n\t\tTopicName: t.name,\n\t\tChannels: channels,\n\t\tDepth: t.Depth(),\n\t\tBackendDepth: t.backend.Depth(),\n\t\tMessageCount: atomic.LoadUint64(&t.messageCount),\n\t\tMessageBytes: atomic.LoadUint64(&t.messageBytes),\n\t\tPaused: t.IsPaused(),\n\n\t\tE2eProcessingLatency: t.AggregateChannelE2eProcessingLatency().Result(),\n\t}\n}\n\ntype ChannelStats struct {\n\tChannelName string `json:\"channel_name\"`\n\tDepth int64 `json:\"depth\"`\n\tBackendDepth int64 `json:\"backend_depth\"`\n\tInFlightCount int `json:\"in_flight_count\"`\n\tDeferredCount int `json:\"deferred_count\"`\n\tMessageCount uint64 `json:\"message_count\"`\n\tZoneLocalMsgCount uint64 `json:\"zone_local_msg_count,omitempty\"`\n\tRegionLocalMsgCount uint64 `json:\"region_local_msg_count,omitempty\"`\n\tGlobalMsgCount uint64 `json:\"global_msg_count,omitempty\"`\n\tRequeueCount uint64 `json:\"requeue_count\"`\n\tTimeoutCount uint64 `json:\"timeout_count\"`\n\tClientCount int `json:\"client_count\"`\n\tClients []ClientStats `json:\"clients\"`\n\tPaused bool `json:\"paused\"`\n\n\tE2eProcessingLatency *quantile.Result `json:\"e2e_processing_latency\"`\n}\n\nfunc NewChannelStats(c *Channel, clients []ClientStats, clientCount int) ChannelStats {\n\tc.inFlightMutex.Lock()\n\tinflight := len(c.inFlightMessages)\n\tc.inFlightMutex.Unlock()\n\tc.deferredMutex.Lock()\n\tdeferred := len(c.deferredMessages)\n\tc.deferredMutex.Unlock()\n\n\treturn ChannelStats{\n\t\tChannelName: c.name,\n\t\tDepth: c.Depth(),\n\t\tBackendDepth: c.backend.Depth(),\n\t\tInFlightCount: inflight,\n\t\tDeferredCount: deferred,\n\t\tMessageCount: atomic.LoadUint64(&c.messageCount),\n\t\tZoneLocalMsgCount: atomic.LoadUint64(&c.zoneLocalMsgCount),\n\t\tRegionLocalMsgCount: atomic.LoadUint64(&c.regionLocalMsgCount),\n\t\tGlobalMsgCount: atomic.LoadUint64(&c.globalMsgCount),\n\t\tRequeueCount: atomic.LoadUint64(&c.requeueCount),\n\t\tTimeoutCount: atomic.LoadUint64(&c.timeoutCount),\n\t\tClientCount: clientCount,\n\t\tClients: clients,\n\t\tPaused: c.IsPaused(),\n\n\t\tE2eProcessingLatency: c.e2eProcessingLatencyStream.Result(),\n\t}\n}\n\ntype Topics []*Topic\n\nfunc (t Topics) Len() int { return len(t) }\nfunc (t Topics) Swap(i, j int) { t[i], t[j] = t[j], t[i] }\n\ntype TopicsByName struct {\n\tTopics\n}\n\nfunc (t TopicsByName) Less(i, j int) bool { return t.Topics[i].name < t.Topics[j].name }\n\ntype Channels []*Channel\n\nfunc (c Channels) Len() int { return len(c) }\nfunc (c Channels) Swap(i, j int) { c[i], c[j] = c[j], c[i] }\n\ntype ChannelsByName struct {\n\tChannels\n}\n\nfunc (c ChannelsByName) Less(i, j int) bool { return c.Channels[i].name < c.Channels[j].name }\n\nfunc (n *NSQD) GetStats(topic string, channel string, includeClients bool) Stats {\n\tvar stats Stats\n\n\tn.RLock()\n\tvar realTopics []*Topic\n\tif topic == \"\" {\n\t\trealTopics = make([]*Topic, 0, len(n.topicMap))\n\t\tfor _, t := range n.topicMap {\n\t\t\trealTopics = append(realTopics, t)\n\t\t}\n\t} else if val, exists := n.topicMap[topic]; exists {\n\t\trealTopics = []*Topic{val}\n\t} else {\n\t\tn.RUnlock()\n\t\treturn stats\n\t}\n\tn.RUnlock()\n\tsort.Sort(TopicsByName{realTopics})\n\n\ttopics := make([]TopicStats, 0, len(realTopics))\n\n\tfor _, t := range realTopics {\n\t\tt.RLock()\n\t\tvar realChannels []*Channel\n\t\tif channel == \"\" {\n\t\t\trealChannels = make([]*Channel, 0, len(t.channelMap))\n\t\t\tfor _, c := range t.channelMap {\n\t\t\t\trealChannels = append(realChannels, c)\n\t\t\t}\n\t\t} else if val, exists := t.channelMap[channel]; exists {\n\t\t\trealChannels = []*Channel{val}\n\t\t} else {\n\t\t\tt.RUnlock()\n\t\t\tcontinue\n\t\t}\n\t\tt.RUnlock()\n\t\tsort.Sort(ChannelsByName{realChannels})\n\t\tchannels := make([]ChannelStats, 0, len(realChannels))\n\t\tfor _, c := range realChannels {\n\t\t\tvar clients []ClientStats\n\t\t\tvar clientCount int\n\t\t\tc.RLock()\n\t\t\tif includeClients {\n\t\t\t\tclients = make([]ClientStats, 0, len(c.clients))\n\t\t\t\tfor _, client := range c.clients {\n\t\t\t\t\tclients = append(clients, client.Stats(topic))\n\t\t\t\t}\n\t\t\t}\n\t\t\tclientCount = len(c.clients)\n\t\t\tc.RUnlock()\n\t\t\tchannels = append(channels, NewChannelStats(c, clients, clientCount))\n\t\t}\n\t\ttopics = append(topics, NewTopicStats(t, channels))\n\t}\n\tstats.Topics = topics\n\n\tif includeClients {\n\t\tvar producerStats []ClientStats\n\t\tn.tcpServer.conns.Range(func(k, v interface{}) bool {\n\t\t\tc := v.(Client)\n\t\t\tif c.Type() == typeProducer {\n\t\t\t\tproducerStats = append(producerStats, c.Stats(topic))\n\t\t\t}\n\t\t\treturn true\n\t\t})\n\t\tstats.Producers = producerStats\n\t}\n\n\treturn stats\n}\n\ntype memStats struct {\n\tHeapObjects uint64 `json:\"heap_objects\"`\n\tHeapIdleBytes uint64 `json:\"heap_idle_bytes\"`\n\tHeapInUseBytes uint64 `json:\"heap_in_use_bytes\"`\n\tHeapReleasedBytes uint64 `json:\"heap_released_bytes\"`\n\tGCPauseUsec100 uint64 `json:\"gc_pause_usec_100\"`\n\tGCPauseUsec99 uint64 `json:\"gc_pause_usec_99\"`\n\tGCPauseUsec95 uint64 `json:\"gc_pause_usec_95\"`\n\tNextGCBytes uint64 `json:\"next_gc_bytes\"`\n\tGCTotalRuns uint32 `json:\"gc_total_runs\"`\n}\n\nfunc getMemStats() memStats {\n\tvar ms runtime.MemStats\n\truntime.ReadMemStats(&ms)\n\n\t// sort the GC pause array\n\tlength := len(ms.PauseNs)\n\tif int(ms.NumGC) < length {\n\t\tlength = int(ms.NumGC)\n\t}\n\tgcPauses := make(Uint64Slice, length)\n\tcopy(gcPauses, ms.PauseNs[:length])\n\tsort.Sort(gcPauses)\n\n\treturn memStats{\n\t\tms.HeapObjects,\n\t\tms.HeapIdle,\n\t\tms.HeapInuse,\n\t\tms.HeapReleased,\n\t\tpercentile(100.0, gcPauses, len(gcPauses)) / 1000,\n\t\tpercentile(99.0, gcPauses, len(gcPauses)) / 1000,\n\t\tpercentile(95.0, gcPauses, len(gcPauses)) / 1000,\n\t\tms.NextGC,\n\t\tms.NumGC,\n\t}\n\n}\n" }, { "path": "nsqd/stats_test.go", "content": "package nsqd\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"os\"\n\t\"strconv\"\n\t\"sync\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/golang/snappy\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\nfunc TestStats(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\ttcpAddr, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_stats\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"test body\"))\n\ttopic.PutMessage(msg)\n\n\taccompanyTopicName := \"accompany_test_stats\" + strconv.Itoa(int(time.Now().Unix()))\n\taccompanyTopic := nsqd.GetTopic(accompanyTopicName)\n\tmsg = NewMessage(accompanyTopic.GenerateID(), []byte(\"accompany test body\"))\n\taccompanyTopic.PutMessage(msg)\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tidentify(t, conn, nil, frameTypeResponse)\n\tsub(t, conn, topicName, \"ch\")\n\n\tstats := nsqd.GetStats(topicName, \"ch\", true).Topics\n\tt.Logf(\"stats: %+v\", stats)\n\n\ttest.Equal(t, 1, len(stats))\n\ttest.Equal(t, 1, len(stats[0].Channels))\n\ttest.Equal(t, 1, len(stats[0].Channels[0].Clients))\n\ttest.Equal(t, 1, stats[0].Channels[0].ClientCount)\n\n\tstats = nsqd.GetStats(topicName, \"ch\", false).Topics\n\tt.Logf(\"stats: %+v\", stats)\n\n\ttest.Equal(t, 1, len(stats))\n\ttest.Equal(t, 1, len(stats[0].Channels))\n\ttest.Equal(t, 0, len(stats[0].Channels[0].Clients))\n\ttest.Equal(t, 1, stats[0].Channels[0].ClientCount)\n\n\tstats = nsqd.GetStats(topicName, \"none_exist_channel\", false).Topics\n\tt.Logf(\"stats: %+v\", stats)\n\n\ttest.Equal(t, 0, len(stats))\n\n\tstats = nsqd.GetStats(\"none_exist_topic\", \"none_exist_channel\", false).Topics\n\tt.Logf(\"stats: %+v\", stats)\n\n\ttest.Equal(t, 0, len(stats))\n}\n\nfunc TestClientAttributes(t *testing.T) {\n\tuserAgent := \"Test User Agent\"\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.SnappyEnabled = true\n\ttcpAddr, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\tconn, err := mustConnectNSQD(tcpAddr)\n\ttest.Nil(t, err)\n\tdefer conn.Close()\n\n\tdata := identify(t, conn, map[string]interface{}{\n\t\t\"snappy\": true,\n\t\t\"user_agent\": userAgent,\n\t}, frameTypeResponse)\n\tresp := struct {\n\t\tSnappy bool `json:\"snappy\"`\n\t\tUserAgent string `json:\"user_agent\"`\n\t}{}\n\terr = json.Unmarshal(data, &resp)\n\ttest.Nil(t, err)\n\ttest.Equal(t, true, resp.Snappy)\n\n\tr := snappy.NewReader(conn)\n\t//lint:ignore SA1019 NewWriter is deprecated by NewBufferedWriter, but we don't want to buffer\n\tw := snappy.NewWriter(conn)\n\treadValidate(t, r, frameTypeResponse, \"OK\")\n\n\ttopicName := \"test_client_attributes\" + strconv.Itoa(int(time.Now().Unix()))\n\tsub(t, readWriter{r, w}, topicName, \"ch\")\n\n\tvar d struct {\n\t\tTopics []struct {\n\t\t\tChannels []struct {\n\t\t\t\tClients []struct {\n\t\t\t\t\tUserAgent string `json:\"user_agent\"`\n\t\t\t\t\tSnappy bool `json:\"snappy\"`\n\t\t\t\t} `json:\"clients\"`\n\t\t\t} `json:\"channels\"`\n\t\t} `json:\"topics\"`\n\t}\n\n\tendpoint := fmt.Sprintf(\"http://%s/stats?format=json\", httpAddr)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &d)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, userAgent, d.Topics[0].Channels[0].Clients[0].UserAgent)\n\ttest.Equal(t, true, d.Topics[0].Channels[0].Clients[0].Snappy)\n}\n\nfunc TestStatsChannelLocking(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_channel_empty\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\tchannel := topic.GetChannel(\"channel\")\n\n\tvar wg sync.WaitGroup\n\n\twg.Add(2)\n\tgo func() {\n\t\tfor i := 0; i < 25; i++ {\n\t\t\tmsg := NewMessage(topic.GenerateID(), []byte(\"test\"))\n\t\t\ttopic.PutMessage(msg)\n\t\t\tchannel.StartInFlightTimeout(msg, 0, opts.MsgTimeout)\n\t\t}\n\t\twg.Done()\n\t}()\n\n\tgo func() {\n\t\tfor i := 0; i < 25; i++ {\n\t\t\tnsqd.GetStats(\"\", \"\", true)\n\t\t}\n\t\twg.Done()\n\t}()\n\n\twg.Wait()\n\n\tstats := nsqd.GetStats(topicName, \"channel\", false).Topics\n\tt.Logf(\"stats: %+v\", stats)\n\n\ttest.Equal(t, 1, len(stats))\n\ttest.Equal(t, 1, len(stats[0].Channels))\n\ttest.Equal(t, 25, stats[0].Channels[0].InFlightCount)\n}\n" }, { "path": "nsqd/statsd.go", "content": "package nsqd\n\nimport (\n\t\"fmt\"\n\t\"math\"\n\t\"net\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/statsd\"\n\t\"github.com/nsqio/nsq/internal/writers\"\n)\n\ntype Uint64Slice []uint64\n\nfunc (s Uint64Slice) Len() int {\n\treturn len(s)\n}\n\nfunc (s Uint64Slice) Swap(i, j int) {\n\ts[i], s[j] = s[j], s[i]\n}\n\nfunc (s Uint64Slice) Less(i, j int) bool {\n\treturn s[i] < s[j]\n}\n\nfunc (n *NSQD) statsdLoop() {\n\tvar lastMemStats memStats\n\tvar lastStats Stats\n\tinterval := n.getOpts().StatsdInterval\n\tticker := time.NewTicker(interval)\n\tfor {\n\t\tselect {\n\t\tcase <-n.exitChan:\n\t\t\tgoto exit\n\t\tcase <-ticker.C:\n\t\t\taddr := n.getOpts().StatsdAddress\n\t\t\tprefix := n.getOpts().StatsdPrefix\n\t\t\texcludeEphemeral := n.getOpts().StatsdExcludeEphemeral\n\t\t\tconn, err := net.DialTimeout(\"udp\", addr, time.Second)\n\t\t\tif err != nil {\n\t\t\t\tn.logf(LOG_ERROR, \"failed to create UDP socket to statsd(%s)\", addr)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tsw := writers.NewSpreadWriter(conn, interval-time.Second, n.exitChan)\n\t\t\tbw := writers.NewBoundaryBufferedWriter(sw, n.getOpts().StatsdUDPPacketSize)\n\t\t\tclient := statsd.NewClient(bw, prefix)\n\n\t\t\tn.logf(LOG_INFO, \"STATSD: pushing stats to %s\", addr)\n\n\t\t\tstats := n.GetStats(\"\", \"\", false)\n\t\t\tfor _, topic := range stats.Topics {\n\t\t\t\tif excludeEphemeral && strings.HasSuffix(topic.TopicName, \"#ephemeral\") {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\t// try to find the topic in the last collection\n\t\t\t\tlastTopic := TopicStats{}\n\t\t\t\tfor _, checkTopic := range lastStats.Topics {\n\t\t\t\t\tif topic.TopicName == checkTopic.TopicName {\n\t\t\t\t\t\tlastTopic = checkTopic\n\t\t\t\t\t\tbreak\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tdiff := topic.MessageCount - lastTopic.MessageCount\n\t\t\t\tstat := fmt.Sprintf(\"topic.%s.message_count\", topic.TopicName)\n\t\t\t\tclient.Incr(stat, int64(diff))\n\n\t\t\t\tdiff = topic.MessageBytes - lastTopic.MessageBytes\n\t\t\t\tstat = fmt.Sprintf(\"topic.%s.message_bytes\", topic.TopicName)\n\t\t\t\tclient.Incr(stat, int64(diff))\n\n\t\t\t\tstat = fmt.Sprintf(\"topic.%s.depth\", topic.TopicName)\n\t\t\t\tclient.Gauge(stat, topic.Depth)\n\n\t\t\t\tstat = fmt.Sprintf(\"topic.%s.backend_depth\", topic.TopicName)\n\t\t\t\tclient.Gauge(stat, topic.BackendDepth)\n\n\t\t\t\tfor _, item := range topic.E2eProcessingLatency.Percentiles {\n\t\t\t\t\tstat = fmt.Sprintf(\"topic.%s.e2e_processing_latency_%.0f\", topic.TopicName, item[\"quantile\"]*100.0)\n\t\t\t\t\t// We can cast the value to int64 since a value of 1 is the\n\t\t\t\t\t// minimum resolution we will have, so there is no loss of\n\t\t\t\t\t// accuracy\n\t\t\t\t\tclient.Gauge(stat, int64(item[\"value\"]))\n\t\t\t\t}\n\n\t\t\t\tfor _, channel := range topic.Channels {\n\t\t\t\t\tif excludeEphemeral && strings.HasSuffix(channel.ChannelName, \"#ephemeral\") {\n\t\t\t\t\t\tcontinue\n\t\t\t\t\t}\n\n\t\t\t\t\t// try to find the channel in the last collection\n\t\t\t\t\tlastChannel := ChannelStats{}\n\t\t\t\t\tfor _, checkChannel := range lastTopic.Channels {\n\t\t\t\t\t\tif channel.ChannelName == checkChannel.ChannelName {\n\t\t\t\t\t\t\tlastChannel = checkChannel\n\t\t\t\t\t\t\tbreak\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t\tdiff := channel.MessageCount - lastChannel.MessageCount\n\t\t\t\t\tstat := fmt.Sprintf(\"topic.%s.channel.%s.message_count\", topic.TopicName, channel.ChannelName)\n\t\t\t\t\tclient.Incr(stat, int64(diff))\n\n\t\t\t\t\tstat = fmt.Sprintf(\"topic.%s.channel.%s.depth\", topic.TopicName, channel.ChannelName)\n\t\t\t\t\tclient.Gauge(stat, channel.Depth)\n\n\t\t\t\t\tstat = fmt.Sprintf(\"topic.%s.channel.%s.backend_depth\", topic.TopicName, channel.ChannelName)\n\t\t\t\t\tclient.Gauge(stat, channel.BackendDepth)\n\n\t\t\t\t\tstat = fmt.Sprintf(\"topic.%s.channel.%s.in_flight_count\", topic.TopicName, channel.ChannelName)\n\t\t\t\t\tclient.Gauge(stat, int64(channel.InFlightCount))\n\n\t\t\t\t\tstat = fmt.Sprintf(\"topic.%s.channel.%s.deferred_count\", topic.TopicName, channel.ChannelName)\n\t\t\t\t\tclient.Gauge(stat, int64(channel.DeferredCount))\n\n\t\t\t\t\tdiff = channel.RequeueCount - lastChannel.RequeueCount\n\t\t\t\t\tstat = fmt.Sprintf(\"topic.%s.channel.%s.requeue_count\", topic.TopicName, channel.ChannelName)\n\t\t\t\t\tclient.Incr(stat, int64(diff))\n\n\t\t\t\t\tdiff = channel.TimeoutCount - lastChannel.TimeoutCount\n\t\t\t\t\tstat = fmt.Sprintf(\"topic.%s.channel.%s.timeout_count\", topic.TopicName, channel.ChannelName)\n\t\t\t\t\tclient.Incr(stat, int64(diff))\n\n\t\t\t\t\tstat = fmt.Sprintf(\"topic.%s.channel.%s.clients\", topic.TopicName, channel.ChannelName)\n\t\t\t\t\tclient.Gauge(stat, int64(channel.ClientCount))\n\n\t\t\t\t\tfor _, item := range channel.E2eProcessingLatency.Percentiles {\n\t\t\t\t\t\tstat = fmt.Sprintf(\"topic.%s.channel.%s.e2e_processing_latency_%.0f\", topic.TopicName, channel.ChannelName, item[\"quantile\"]*100.0)\n\t\t\t\t\t\tclient.Gauge(stat, int64(item[\"value\"]))\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\tlastStats = stats\n\n\t\t\tif n.getOpts().StatsdMemStats {\n\t\t\t\tms := getMemStats()\n\n\t\t\t\tclient.Gauge(\"mem.heap_objects\", int64(ms.HeapObjects))\n\t\t\t\tclient.Gauge(\"mem.heap_idle_bytes\", int64(ms.HeapIdleBytes))\n\t\t\t\tclient.Gauge(\"mem.heap_in_use_bytes\", int64(ms.HeapInUseBytes))\n\t\t\t\tclient.Gauge(\"mem.heap_released_bytes\", int64(ms.HeapReleasedBytes))\n\t\t\t\tclient.Gauge(\"mem.gc_pause_usec_100\", int64(ms.GCPauseUsec100))\n\t\t\t\tclient.Gauge(\"mem.gc_pause_usec_99\", int64(ms.GCPauseUsec99))\n\t\t\t\tclient.Gauge(\"mem.gc_pause_usec_95\", int64(ms.GCPauseUsec95))\n\t\t\t\tclient.Gauge(\"mem.next_gc_bytes\", int64(ms.NextGCBytes))\n\t\t\t\tclient.Incr(\"mem.gc_runs\", int64(ms.GCTotalRuns-lastMemStats.GCTotalRuns))\n\n\t\t\t\tlastMemStats = ms\n\t\t\t}\n\n\t\t\tbw.Flush()\n\t\t\tsw.Flush()\n\t\t\tconn.Close()\n\t\t}\n\t}\n\nexit:\n\tticker.Stop()\n\tn.logf(LOG_INFO, \"STATSD: closing\")\n}\n\nfunc percentile(perc float64, arr []uint64, length int) uint64 {\n\tif length == 0 {\n\t\treturn 0\n\t}\n\tindexOfPerc := int(math.Floor(((perc / 100.0) * float64(length)) + 0.5))\n\tif indexOfPerc >= length {\n\t\tindexOfPerc = length - 1\n\t}\n\treturn arr[indexOfPerc]\n}\n" }, { "path": "nsqd/tcp.go", "content": "package nsqd\n\nimport (\n\t\"io\"\n\t\"net\"\n\t\"sync\"\n\n\t\"github.com/nsqio/nsq/internal/protocol\"\n)\n\nconst (\n\ttypeConsumer = iota\n\ttypeProducer\n)\n\ntype Client interface {\n\tType() int\n\tStats(string) ClientStats\n}\n\ntype tcpServer struct {\n\tnsqd *NSQD\n\tconns sync.Map\n}\n\nfunc (p *tcpServer) Handle(conn net.Conn) {\n\tp.nsqd.logf(LOG_INFO, \"TCP: new client(%s)\", conn.RemoteAddr())\n\n\t// The client should initialize itself by sending a 4 byte sequence indicating\n\t// the version of the protocol that it intends to communicate, this will allow us\n\t// to gracefully upgrade the protocol away from text/line oriented to whatever...\n\tbuf := make([]byte, 4)\n\t_, err := io.ReadFull(conn, buf)\n\tif err != nil {\n\t\tp.nsqd.logf(LOG_ERROR, \"failed to read protocol version - %s\", err)\n\t\tconn.Close()\n\t\treturn\n\t}\n\tprotocolMagic := string(buf)\n\n\tp.nsqd.logf(LOG_INFO, \"CLIENT(%s): desired protocol magic '%s'\",\n\t\tconn.RemoteAddr(), protocolMagic)\n\n\tvar prot protocol.Protocol\n\tswitch protocolMagic {\n\tcase \" V2\":\n\t\tprot = &protocolV2{nsqd: p.nsqd}\n\tdefault:\n\t\tprotocol.SendFramedResponse(conn, frameTypeError, []byte(\"E_BAD_PROTOCOL\"))\n\t\tconn.Close()\n\t\tp.nsqd.logf(LOG_ERROR, \"client(%s) bad protocol magic '%s'\",\n\t\t\tconn.RemoteAddr(), protocolMagic)\n\t\treturn\n\t}\n\n\tclient := prot.NewClient(conn)\n\tp.conns.Store(conn.RemoteAddr(), client)\n\n\terr = prot.IOLoop(client)\n\tif err != nil {\n\t\tp.nsqd.logf(LOG_ERROR, \"client(%s) - %s\", conn.RemoteAddr(), err)\n\t}\n\n\tp.conns.Delete(conn.RemoteAddr())\n\tclient.Close()\n}\n\nfunc (p *tcpServer) Close() {\n\tp.conns.Range(func(k, v interface{}) bool {\n\t\tv.(protocol.Client).Close()\n\t\treturn true\n\t})\n}\n" }, { "path": "nsqd/test/cert.sh", "content": "#!/bin/bash\n# ./cert.sh foo@foo.com 127.0.0.1\n# Found: https://gist.github.com/ncw/9253562#file-makecert-sh\n\nif [ \"$1\" == \"\" ]; then\n echo \"Need email as argument\"\n exit 1\nfi\n\nif [ \"$2\" == \"\" ]; then\n echo \"Need CN as argument\"\n exit 1\nfi\n\nPRIVKEY=\"test\"\nEMAIL=$1\nCN=$2\n\nrm -rf tmp\nmkdir tmp\ncd tmp\n\necho \"make CA\"\nopenssl req -new -x509 -days 3650 -keyout ca.key -out ca.pem \\\n -config ../openssl.conf -extensions ca \\\n -subj \"/CN=ca\" \\\n -passout pass:$PRIVKEY\n\necho \"make server cert\"\nopenssl genrsa -out server.key 2048\nopenssl req -new -sha256 -key server.key -out server.req \\\n -subj \"/emailAddress=${EMAIL}/C=DE/ST=NRW/L=Earth/O=Random Company/OU=IT/CN=${CN}\"\nopenssl x509 -req -days 3650 -sha256 -in server.req -CA ca.pem -CAkey ca.key -CAcreateserial -passin pass:$PRIVKEY -out server.pem \\\n -extfile ../openssl.conf -extensions server\n \n\necho \"make client cert\"\nopenssl genrsa -out client.key 2048\nopenssl req -new -sha256 -key client.key -out client.req \\\n -subj \"/emailAddress=${EMAIL}/C=DE/ST=NRW/L=Earth/O=Random Company/OU=IT/CN=${CN}\"\nopenssl x509 -req -days 3650 -sha256 -in client.req -CA ca.pem -CAkey ca.key -CAserial ca.srl -passin pass:$PRIVKEY -out client.pem \\\n -extfile ../openssl.conf -extensions client\n\ncd ..\nmv tmp/* certs\nrm -rf tmp\n" }, { "path": "nsqd/test/certs/ca.key", "content": "-----BEGIN RSA PRIVATE KEY-----\nProc-Type: 4,ENCRYPTED\nDEK-Info: DES-EDE3-CBC,D6208487450BDF1C\n\ntucU/4j6agGQlW60D8V3Zr/QHcLhhyFagres1gGdWfGIqluNZb7omki/XidHXJSG\neB9vV2Xb12/8umc31e7Mnmn9hd34230v/KlAnJ4ukDpJpbmjnEx3F9uiqYFi/yxQ\navSsfF6Tsh3XOh3Oe27I/xfYx37g6Agd+EQEJ1hvWvygMIJMTDMP5ZaFoZANtFLy\nhDEZ6woJSn9avF/L+1GW8jl2aI1QbdKkK0jDHgFAwUI4sjWeXvEQNNYY3trTIoMo\nwab3vi+4XziFONbS4OZrZUYfZPB5YOFbtT2whzggp2HdSTiu48/Ld3N8SjuMrKfm\nuR+nd+ovQ5kVWHInzWAIXSyPhgR9ZY8eyXaHNJJfzNu3HY72lfzD/NtZfacMRBr6\nM3Wg/OKPS7ZrtqCWkY9P3KK9Cul8Jzy229fSqHo8Rg4=\n-----END RSA PRIVATE KEY-----\n" }, { "path": "nsqd/test/certs/ca.pem", "content": "-----BEGIN CERTIFICATE-----\nMIIBVDCB/6ADAgECAgkAvHG4Z/7nX/gwDQYJKoZIhvcNAQEFBQAwDTELMAkGA1UE\nAwwCY2EwHhcNMTcwOTE2MTc0NDE0WhcNMjcwOTE0MTc0NDE0WjANMQswCQYDVQQD\nDAJjYTBcMA0GCSqGSIb3DQEBAQUAA0sAMEgCQQDJAM3Tr1BoxlLJtTy0oRcp93dT\n9hhHwms8P1V3k2FpXYRS4deUo+uwcAM9KGDt9VMXVBEchtI4VYTvLgatBPUBAgMB\nAAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFMyLx7rjKBe/xZQLnVzI\nuqNNVzxRMA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAANBAJiN1XgPPlQ2\nz7PhtzXStaz/BJVqhD7g9fsZsmoPX4ifDsTfzUsRB56Aq/NTsKiIYQkFPHH0donG\n++a5ZVWjgYk=\n-----END CERTIFICATE-----\n" }, { "path": "nsqd/test/certs/ca.srl", "content": "91418D04995922E7\n" }, { "path": "nsqd/test/certs/cert.pem", "content": "-----BEGIN CERTIFICATE-----\nMIIEbjCCA1agAwIBAgIJAK6x7y6AwBmLMA0GCSqGSIb3DQEBBQUAMIGAMQswCQYD\nVQQGEwJVUzERMA8GA1UECBMITmV3IFlvcmsxFjAUBgNVBAcTDU5ldyBZb3JrIENp\ndHkxDDAKBgNVBAoTA05TUTETMBEGA1UEAxMKdGVzdC5sb2NhbDEjMCEGCSqGSIb3\nDQEJARYUbXJlaWZlcnNvbkBnbWFpbC5jb20wHhcNMTMwNjI4MDA0MzQ4WhcNMTYw\nNDE3MDA0MzQ4WjCBgDELMAkGA1UEBhMCVVMxETAPBgNVBAgTCE5ldyBZb3JrMRYw\nFAYDVQQHEw1OZXcgWW9yayBDaXR5MQwwCgYDVQQKEwNOU1ExEzARBgNVBAMTCnRl\nc3QubG9jYWwxIzAhBgkqhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwuY29tMIIB\nIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnX0KB+svwy+yHU2qggz/EaGg\ncraKShagKo+9M9y5HLM852ngk5c+t+tJJbx3N954Wr1FXBuGIv1ltU05rU4zhvBS\n25tVP1UIEnT5pBt2TeetLkl199Y7fxh1hKmnwJMG3fy3VZdNXEndBombXMmtXpQY\nshuEJHKeUNDbQKz5X+GjEdkTPO/HY/VMHsxS23pbSimQozMg3hvLIdgv0aS3QECz\nydZBgTPThy3uDtHIuCpxCwXd/vDF68ATlYgo3h3lh2vxNwM/pjklIUhzMh4XaKQF\n7m3/0KbtUcXfy0QHueeuMr11E9MAFNyRN4xf9Fk1yB97KJ3PJBTC5WD/m1nW+QID\nAQABo4HoMIHlMB0GA1UdDgQWBBR3HMBws4lmYYSIgwoZsfW+bbgaMjCBtQYDVR0j\nBIGtMIGqgBR3HMBws4lmYYSIgwoZsfW+bbgaMqGBhqSBgzCBgDELMAkGA1UEBhMC\nVVMxETAPBgNVBAgTCE5ldyBZb3JrMRYwFAYDVQQHEw1OZXcgWW9yayBDaXR5MQww\nCgYDVQQKEwNOU1ExEzARBgNVBAMTCnRlc3QubG9jYWwxIzAhBgkqhkiG9w0BCQEW\nFG1yZWlmZXJzb25AZ21haWwuY29tggkArrHvLoDAGYswDAYDVR0TBAUwAwEB/zAN\nBgkqhkiG9w0BAQUFAAOCAQEANOYTbanW2iyV1v4oYpcM/y3TWcQKzSME8D2SGFZb\ndbMYU81hH3TTlQdvyeh3FAcdjhKE8Xi/RfNNjEslTBscdKXePGpZg6eXRNJzPP5K\nKZPf5u6tcpAeUOKrMqbGwbE+h2QixxG1EoVQtE421szsU2P7nHRTdHzKFRnOerfl\nPhm3NocR0P40Rv7WKdxpOvqc+XKf0onTruoVYoPWGpwcLixCG0zu4ZQ23/L/Dy18\n4u70Hbq6O/6kq9FBFaDNp3IhiEdu2Cq6ZplU6bL9XDF27KIEErHwtuqBHVlMG+zB\noH/k9vZvwH7OwAjHdKp+1yeZFLYC8K5hjFIHqcdwpZCNIg==\n-----END CERTIFICATE-----\n" }, { "path": "nsqd/test/certs/client.key", "content": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAuMz5IJ01Wqaj2qJAU7o2aoc2iP1BRhTba+cWrsjbZA+6kXAk\n19Dlz/nNK8gxsBMl3rzTGWXKUm/rKBDp680zJkj8nzcGApGlzbDoqXym8ixOniOC\naH/RODPC8OtD2UozeXlUOQWK+ufkMvmVBc/IeJyFkcm7jhX+LHVFn+3iWLRNbzGm\nhbf8oNHvybS7HSuV3OYiHDJxA0d6IgozqVOYVSqdDabQ2FN/8nKzpMHjwMpfT008\n5hUhr72E5XuJ3biPUwoOrRETSa1/EQcAEH0wSwHU1AGdnGh3/5ynVupfef0qkyqw\nNseYs9U++IUFXqA1IuEmAlJn5QczAHjPQe8xBQIDAQABAoIBAGx4CyZEgCOUOgrD\nP2SloPkIIk9n7x82cNA11I+E35kszkI9g7KVL77SDcZL/DYwFwNU68c1gvq+LFXZ\nD6RTTlmDb5v4TPPHD33a/8UzoD33GbIif5HcrC4D28FTJgDtV6dOOsw5X6kD4WK2\nMe02V6HLpW677PVqHUV1FAfaNgf/1SIhJCechYSO+3pKPtli8DnsmOd336wNtWZR\nQ3Ctm63Eq7AiLmGLOkLYtll9Kr6WfCSFHBmqJg9MsQEfINXMCOB54Z+xkoijjJLO\nzNSk/3XvppB1JpmAusbo9Mq03Ci9JdinYSKxwVOG6e+5cosrOHpDQyZIWa87ZTpk\nkjLCgEECgYEA33JCGTEwaqfAuYOspCZvi2Mqz7t6QiwvwxgDf4JFLgDQArJfzKlC\nbAzaSU8bT+SN96CF9LU4XekN5/paGZAFjWxraPkK0oL28kQrtzfEQv2XL6GFoNAh\nWsQtoaGddL9VPWlCiHeueYERIDNrZCaVeAx6OV6W3CEwGOkRiQyntlUCgYEA07lf\nLpxF8aAgMToq3++D5RGAXaT2PYxhZZFb5Y2tS7a4T0ulQwXQi4H2dObGJ32zCjFG\nls1H91mNKGBbyKnF1jT/dt/acqymvipASm1xq6uysIllA4xp56sDnJgG9bnXjYFY\nm2yVNIoIQGZ7jp7e6t1KQ9San6V8asWlvBLXX/ECgYEAuhF9dVj+xnH3DQTXSMIw\n9NOZnO6zelMtWrqufwnN7ecDUJuVJupzw2JYi99yEO90QRbNNd+KlrkxuVFCojLK\nTOBR+VIZbv9cAJZACQxJRLfDpAhPLIDkpZ7jmMrqQYPqyX7TxqxTAB84UaY/8WAn\n65YIWamo2ppQYQ4Eaim9pxkCgYAMwfW/TEFWruxhqvycY8VRzz0p51/DE6tmwFyG\nN4RCtK7kcE1z/Wy0i087ehBknslkCtYTDimQ+P9teGjvbXNzVdwy4Ig8MrUVblxT\nX8birkTlKFJC5XoYMJDWJb79nYYki6+4JdHTyaF3p/U4AdCy3ES2U6BBkGov0NsM\nuyHpMQKBgGc9wIGHwAGLg3iVHGXbXELdpzVztLRC7D12TPfdtGBLTD5NH5jAx8K/\nw61+l4tzFcK3jfWzwdme277cBgfq/aw2DDt07vkOvPzhUwVZiE39bfddfIvyed99\nXzEMf8THh5wdm6mLsOVgdfOfFcDi5ReIv1/7+eOd/MQi8TxIVfAu\n-----END RSA PRIVATE KEY-----\n" }, { "path": "nsqd/test/certs/client.pem", "content": "-----BEGIN CERTIFICATE-----\nMIICyzCCAnWgAwIBAgIJAJFBjQSZWSLnMA0GCSqGSIb3DQEBCwUAMA0xCzAJBgNV\nBAMMAmNhMB4XDTE3MDkxNjE3NDQxNFoXDTI3MDkxNDE3NDQxNFowgYcxIzAhBgkq\nhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwuY29tMQswCQYDVQQGEwJERTEMMAoG\nA1UECBMDTlJXMQ4wDAYDVQQHEwVFYXJ0aDEXMBUGA1UEChMOUmFuZG9tIENvbXBh\nbnkxCzAJBgNVBAsTAklUMQ8wDQYDVQQDEwZuc3EuaW8wggEiMA0GCSqGSIb3DQEB\nAQUAA4IBDwAwggEKAoIBAQC4zPkgnTVapqPaokBTujZqhzaI/UFGFNtr5xauyNtk\nD7qRcCTX0OXP+c0ryDGwEyXevNMZZcpSb+soEOnrzTMmSPyfNwYCkaXNsOipfKby\nLE6eI4Jof9E4M8Lw60PZSjN5eVQ5BYr65+Qy+ZUFz8h4nIWRybuOFf4sdUWf7eJY\ntE1vMaaFt/yg0e/JtLsdK5Xc5iIcMnEDR3oiCjOpU5hVKp0NptDYU3/ycrOkwePA\nyl9PTTzmFSGvvYTle4nduI9TCg6tERNJrX8RBwAQfTBLAdTUAZ2caHf/nKdW6l95\n/SqTKrA2x5iz1T74hQVeoDUi4SYCUmflBzMAeM9B7zEFAgMBAAGjdTBzMAwGA1Ud\nEwEB/wQCMAAwHQYDVR0OBBYEFMzYXiTD7moi0oXqSQ5D2LsO51GXMB8GA1UdIwQY\nMBaAFMyLx7rjKBe/xZQLnVzIuqNNVzxRMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUE\nDDAKBggrBgEFBQcDAjANBgkqhkiG9w0BAQsFAANBAAujxas6toRhMl/+kZrly0G/\nAvjbA3WY5cLLIdffGdQ5bsS3aOP23nj98ut7unNUNsCo+eUwpJgvabnFnL+NFZA=\n-----END CERTIFICATE-----\n" }, { "path": "nsqd/test/certs/client.req", "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIICzTCCAbUCAQAwgYcxIzAhBgkqhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwu\nY29tMQswCQYDVQQGEwJERTEMMAoGA1UECBMDTlJXMQ4wDAYDVQQHEwVFYXJ0aDEX\nMBUGA1UEChMOUmFuZG9tIENvbXBhbnkxCzAJBgNVBAsTAklUMQ8wDQYDVQQDEwZu\nc3EuaW8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC4zPkgnTVapqPa\nokBTujZqhzaI/UFGFNtr5xauyNtkD7qRcCTX0OXP+c0ryDGwEyXevNMZZcpSb+so\nEOnrzTMmSPyfNwYCkaXNsOipfKbyLE6eI4Jof9E4M8Lw60PZSjN5eVQ5BYr65+Qy\n+ZUFz8h4nIWRybuOFf4sdUWf7eJYtE1vMaaFt/yg0e/JtLsdK5Xc5iIcMnEDR3oi\nCjOpU5hVKp0NptDYU3/ycrOkwePAyl9PTTzmFSGvvYTle4nduI9TCg6tERNJrX8R\nBwAQfTBLAdTUAZ2caHf/nKdW6l95/SqTKrA2x5iz1T74hQVeoDUi4SYCUmflBzMA\neM9B7zEFAgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEALD1VjbBjaeX7l7JR4IuL\nwrHVrFDiRYsgvqyw39j2MC95VwrGwzf4cXCE+RuqE/DhbV9UI7sWKaJfFs9Usq+g\nVSoKnHSEylt34y6ABSc5eAik+GnheJZbJ6UDjxcvNd0UpFGMrHbsXVyQd0Y1XAu7\nnxCYIa82kNA+Opb+ra03hkLC7wvRLbXTOB2g6JLkyhYR6S5GkOFTNnz2AJerN7zt\nNEL7owlRcjyIsL5tjDpDH1944NNtzhgmrUeIjB08reayuot9RKznMVGwBfY6DIHM\nQ4uNN3CMOOoAHr1UzvBf/qfvb6ltPTMKSV1OncLlC3C59NoO8vhKIHCN18Ya2OMu\nrw==\n-----END CERTIFICATE REQUEST-----\n" }, { "path": "nsqd/test/certs/key.pem", "content": "-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEAnX0KB+svwy+yHU2qggz/EaGgcraKShagKo+9M9y5HLM852ng\nk5c+t+tJJbx3N954Wr1FXBuGIv1ltU05rU4zhvBS25tVP1UIEnT5pBt2TeetLkl1\n99Y7fxh1hKmnwJMG3fy3VZdNXEndBombXMmtXpQYshuEJHKeUNDbQKz5X+GjEdkT\nPO/HY/VMHsxS23pbSimQozMg3hvLIdgv0aS3QECzydZBgTPThy3uDtHIuCpxCwXd\n/vDF68ATlYgo3h3lh2vxNwM/pjklIUhzMh4XaKQF7m3/0KbtUcXfy0QHueeuMr11\nE9MAFNyRN4xf9Fk1yB97KJ3PJBTC5WD/m1nW+QIDAQABAoIBACvtfKbIywG+hAf4\nad7skRjx5DcbA2e29+XnQfb9UgTXWd2SgrmoLi5OypBkCTzkKN3mfTo70yZfV8dC\nSxwz+9tfnTz0DssjhKThS+CiaFVCkeOfSfBfKSlCQUVHrSrh18CDhP+yvDlJwQTZ\nzSQMfPcsh9bmJe2kqtQP7ZgUp1o+vaB8Sju8YYrO6FllxbdLRGm4pfvvrHIRRmXa\noVHn0ei0JpwoTY9kHYht4LNeJnbP/MCWdmcuv3Gnel7jAlhaKab5aNIGr0Xe7aIQ\niX6mpZ0/Rnt8o/XcTOg8l3ruIdVuySX6SYn08JMnfFkXdNYRVhoV1tC5ElWkaZLf\nhPmj2yECgYEAyts0R0b8cZ6HTAyuLm3ilw0s0v0/MM9ZtaqMRilr2WEtAhF0GpHG\nTzmGnii0WcTNXD7NTsNcECR/0ZpXPRleMczsL2Juwd4FkQ37h7hdKPseJNrfyHRg\nVolOFBX9H14C3wMB9cwdsG4Egw7fE27WCoreEquHgwFxl1zBrXKH088CgYEAxr8w\nBKZs0bF7LRrFT5pH8hpMLYHMYk8ZIOfgmEGVBKDQCOERPR9a9kqUss7wl/98LVNK\nRnFlyWD6Z0/QcQsLL4LjBeZJ25qEMc6JXm9VGAzhXA1ZkUofVoYCnG+f6KUn8CuJ\n/AcV2ZDFsEP10IiQG0hKsceXiwFEvEr8306tMrcCgYBLgnscSR0xAeyk71dq6vZc\necgEpcX+2kAvclOSzlpZ6WVCjtKkDT0/Qk+M0eQIQkybGLl9pxS+4Yc+s2/jy2yX\npwsHvGE0AvwZeZX2eDcdSRR4bYy9ZixyKdwJeAHnyivRbaIuJ5Opl9pQGpoI9snv\n1K9DTdw8dK4exKVHdgl/WwKBgDkmLsuXg4EEtPOyV/xc08VVNIR9Z2T5c7NXmeiO\nKyiKiWeUOF3ID2L07S9BfENozq9F3PzGjMtMXJSqibiHwW6nB1rh7mj8VHjx9+Q0\nxVZGFeNfX1r84mgB3uxW2LeQDhzsmB/lda37CC14TU3qhu2hawEV8IijE73FHlOk\nDv+fAoGAI4/XO5o5tNn5Djo8gHmGMCbinUE9+VySxl7wd7PK8w2VSofO88ofixDk\nNX94yBYhg5WZcLdPm45RyUnq+WVQYz9IKUrdxLFTH+wxyzUqZCW7jgXCvWV+071q\nvqm9C+kndq+18/1VKuCSGWnF7Ay4lbsgPXY2s4VKRxcb3QpZSPU=\n-----END RSA PRIVATE KEY-----\n" }, { "path": "nsqd/test/certs/server.key", "content": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAuYs7NerYpXWq/IPaBGriHobhIs/i4AJHra7QhZ5N/EfBCb0N\n6K3tI4GRHMZ4lB0HwINWmxaPs6D2yDCxpGgnru6qCu0/91+Lq+GEZj+Fz1E2yfCq\ntecTySvWf9en+SdTKFZTGGIAu3zFcozuX2fdhkjJ6ibZhnrXKlPU4IN0STkkRfFX\n+yb/6P88mGh7PZKBb67Qw1bppzXE/SWgHVqDqu++whT1quwxnRhh/SHgwugN+6iK\nuTDcJ1jQ1QWe+xna7Te6nNBUvL6dV49jpoDOjHsv+UaAvzNeHCHqypTC0wwYHXyg\nTTAU+E20c2NOCC7P0PAid1liNeuuEWtM2bRrFQIDAQABAoIBABPzc7d1fDw2bd9f\nMic9cvkDWdwLbILX2+tCG+vyPMJ+2LP6Xy+A3DnwKbFlafvLL1U1Ci/8+hC/oymd\nisx54qJ9yU0Je9JWtMcTpc/0zqefPPvz4/dRVKBSFWuDve0dnGR++8poZ1nBrd2G\nZ+9cVMamtwd1i/hY5yAHCaHmoK9qxXuSjtbPkqDqruloGO5iT+tcICjAeWyJIx1P\n5VJjuKDx4AhU4B8DpBSTtA02BKltvr2K6D4zhJ255oI4iLUkYk4IwxB7yZH+6Y++\nmDa6e8iM3ut+qMkCxto/CfMCqYfoRzmykiazZ5qFXfGwv3OkWUEOBXU//Vpklvnh\nOQWYL/UCgYEA7mkKXbCrCNJEdly4LVLPKPFjVUPhMdXcqdnoKZ6BiVYFMqTuG4+w\no/Rf6ZpqZ3Rf656/ypd1atVvK2g4zfq3j1W6lDnpLjKaLhc8a+ZH4NWXBORIzlUW\naR2xVcPpAokor37FRAWbduiwVSCHlDtg48i+rvukyqmeRBKjrIgX3lsCgYEAxzut\n5GiMZ9TkjqYXdlUDIDKWMHMfLfRY7+Q/cBdr7AfiMVw4PIWJG/IuxCGwGp77gRg6\nBeBsi8Htj5/EGbRiK4kjFyf8LqYGOeYLZ3I8+olr+tNmF4/yKwxku3VylpmrpCAV\n/8emso1rAWo6Y98MTNSGgvLqhDU6OD7tTSv3908CgYA4+aVeio/1RbrSxonFWxri\n3/0rLVOuAzv+43KWL6kpVwNa/QtiTs6aABbDzwFKxAcAWinfkp6e727n4rpgj2A6\nwvQZ5FUTk0hBZ5ArAReAZcr3gk7b8H2wlUYCBxWyY3Dzr8oY3XYvzqAFWAbOp/oZ\ntanMS5swS6TlA8dVvhhmLQKBgGZwcBPOEctdcntKOSwVv/qxJ/oXZ0O4rHYENP4M\nfOgqkYnxsdSkkH/3AUbFT4gQkJ6q90KIRyeA+gXsDudskUFzTMCeRZMyuGbSurBg\n06u6NvQL+CVLVSf/QlgEpnt63f8QpF8Up8iM4CUlGoq5Z9ilOdhg0GZT+/BpopgY\ncHIPAoGBAKyLumgeG+gMFK5x/Fi6zjBT5MK8Tw0VMkahW01Jx2laYibS/a8AEFmn\nySdrmLPkOmmWgXCk3m2m5PkvM5qH/KNugOA0+WX2CTvwt4ZgwWUalQ5R43wSIeDC\nMXVfwC8uE66PmpYgmsu4H0vnCGfacOCQhfdq01SLobgBiQSrm/6D\n-----END RSA PRIVATE KEY-----\n" }, { "path": "nsqd/test/certs/server.pem", "content": "-----BEGIN CERTIFICATE-----\nMIIC3jCCAoigAwIBAgIJAJFBjQSZWSLmMA0GCSqGSIb3DQEBCwUAMA0xCzAJBgNV\nBAMMAmNhMB4XDTE3MDkxNjE3NDQxNFoXDTI3MDkxNDE3NDQxNFowgYcxIzAhBgkq\nhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwuY29tMQswCQYDVQQGEwJERTEMMAoG\nA1UECBMDTlJXMQ4wDAYDVQQHEwVFYXJ0aDEXMBUGA1UEChMOUmFuZG9tIENvbXBh\nbnkxCzAJBgNVBAsTAklUMQ8wDQYDVQQDEwZuc3EuaW8wggEiMA0GCSqGSIb3DQEB\nAQUAA4IBDwAwggEKAoIBAQC5izs16tildar8g9oEauIehuEiz+LgAketrtCFnk38\nR8EJvQ3ore0jgZEcxniUHQfAg1abFo+zoPbIMLGkaCeu7qoK7T/3X4ur4YRmP4XP\nUTbJ8Kq15xPJK9Z/16f5J1MoVlMYYgC7fMVyjO5fZ92GSMnqJtmGetcqU9Tgg3RJ\nOSRF8Vf7Jv/o/zyYaHs9koFvrtDDVumnNcT9JaAdWoOq777CFPWq7DGdGGH9IeDC\n6A37qIq5MNwnWNDVBZ77GdrtN7qc0FS8vp1Xj2OmgM6Mey/5RoC/M14cIerKlMLT\nDBgdfKBNMBT4TbRzY04ILs/Q8CJ3WWI1664Ra0zZtGsVAgMBAAGjgYcwgYQwDAYD\nVR0TAQH/BAIwADAdBgNVHQ4EFgQUt6UXJ8BQB++I/cQG5ZgaiUyqOE8wHwYDVR0j\nBBgwFoAUzIvHuuMoF7/FlAudXMi6o01XPFEwDgYDVR0PAQH/BAQDAgWgMBMGA1Ud\nJQQMMAoGCCsGAQUFBwMBMA8GA1UdEQQIMAaHBH8AAAEwDQYJKoZIhvcNAQELBQAD\nQQCCwm0F2eTmtIcKDxrXGZ7q9y9mfsROfYaCnH+vKUDw2vmqQkInzhLeEElDGQcR\nww0IKCnDHEruNb2tKyQM/70L\n-----END CERTIFICATE-----\n" }, { "path": "nsqd/test/certs/server.req", "content": "-----BEGIN CERTIFICATE REQUEST-----\nMIICzTCCAbUCAQAwgYcxIzAhBgkqhkiG9w0BCQEWFG1yZWlmZXJzb25AZ21haWwu\nY29tMQswCQYDVQQGEwJERTEMMAoGA1UECBMDTlJXMQ4wDAYDVQQHEwVFYXJ0aDEX\nMBUGA1UEChMOUmFuZG9tIENvbXBhbnkxCzAJBgNVBAsTAklUMQ8wDQYDVQQDEwZu\nc3EuaW8wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC5izs16tildar8\ng9oEauIehuEiz+LgAketrtCFnk38R8EJvQ3ore0jgZEcxniUHQfAg1abFo+zoPbI\nMLGkaCeu7qoK7T/3X4ur4YRmP4XPUTbJ8Kq15xPJK9Z/16f5J1MoVlMYYgC7fMVy\njO5fZ92GSMnqJtmGetcqU9Tgg3RJOSRF8Vf7Jv/o/zyYaHs9koFvrtDDVumnNcT9\nJaAdWoOq777CFPWq7DGdGGH9IeDC6A37qIq5MNwnWNDVBZ77GdrtN7qc0FS8vp1X\nj2OmgM6Mey/5RoC/M14cIerKlMLTDBgdfKBNMBT4TbRzY04ILs/Q8CJ3WWI1664R\na0zZtGsVAgMBAAGgADANBgkqhkiG9w0BAQsFAAOCAQEASNjcoZbyNcwMQjMmNoil\nS/7pCRn4aYzZIVjrVtOHQ9GHC23MSep5um2gIcMFPiuYyu9Byl8CSVtc1op2fAKS\nvrugoZaCrp/A76hqOfNxgh7VmgTux8bG5Qcjaija1BNWpbyaWARdBxN/WgS5CpCj\nu2yzv8mrzzFNrDMlsmiEMvtkMzdhiZ4YY8zm6CdrbIR5z1eqf4e+rs4oJtTKNNAD\nhewk8CGiUW1hOx2jpjcIVMRy+ofVHRX2xQ6Sw8qxCNsiv8IPAAivgAbFJO76ZSbH\neQ7uKWszmBEroyFvZ0rfmFLXuopU125pyBDl5FUKYAZzCBx9tr5dROCbw/rXDhke\nig==\n-----END CERTIFICATE REQUEST-----\n" }, { "path": "nsqd/test/openssl.conf", "content": "[req]\ndistinguished_name = req_distinguished_name\n\n[req_distinguished_name]\n\n[ca]\nbasicConstraints = critical, CA:true\nsubjectKeyIdentifier = hash\nkeyUsage = critical, cRLSign, keyCertSign\n\n[client]\nbasicConstraints = critical, CA:FALSE\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer\nkeyUsage = critical, digitalSignature, keyEncipherment\nextendedKeyUsage = clientAuth\n\n[server]\nbasicConstraints = critical, CA:FALSE\nsubjectKeyIdentifier = hash\nauthorityKeyIdentifier = keyid,issuer\nkeyUsage = critical, digitalSignature, keyEncipherment\nextendedKeyUsage = serverAuth\nsubjectAltName = IP:127.0.0.1\n" }, { "path": "nsqd/topic.go", "content": "package nsqd\n\nimport (\n\t\"errors\"\n\t\"strings\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-diskqueue\"\n\t\"github.com/nsqio/nsq/internal/lg\"\n\t\"github.com/nsqio/nsq/internal/quantile\"\n\t\"github.com/nsqio/nsq/internal/util\"\n)\n\ntype Topic struct {\n\t// 64bit atomic vars need to be first for proper alignment on 32bit platforms\n\tmessageCount uint64\n\tmessageBytes uint64\n\n\tsync.RWMutex\n\n\tname string\n\tchannelMap map[string]*Channel\n\tbackend BackendQueue\n\tmemoryMsgChan chan *Message\n\tstartChan chan int\n\texitChan chan int\n\tchannelUpdateChan chan int\n\twaitGroup util.WaitGroupWrapper\n\texitFlag int32\n\tidFactory *guidFactory\n\n\tephemeral bool\n\tdeleteCallback func(*Topic)\n\tdeleter sync.Once\n\n\tpaused int32\n\tpauseChan chan int\n\n\tnsqd *NSQD\n}\n\n// Topic constructor\nfunc NewTopic(topicName string, nsqd *NSQD, deleteCallback func(*Topic)) *Topic {\n\tt := &Topic{\n\t\tname: topicName,\n\t\tchannelMap: make(map[string]*Channel),\n\t\tmemoryMsgChan: make(chan *Message, nsqd.getOpts().MemQueueSize),\n\t\tstartChan: make(chan int, 1),\n\t\texitChan: make(chan int),\n\t\tchannelUpdateChan: make(chan int),\n\t\tnsqd: nsqd,\n\t\tpaused: 0,\n\t\tpauseChan: make(chan int),\n\t\tdeleteCallback: deleteCallback,\n\t\tidFactory: NewGUIDFactory(nsqd.getOpts().ID),\n\t}\n\tif strings.HasSuffix(topicName, \"#ephemeral\") {\n\t\tt.ephemeral = true\n\t\tt.backend = newDummyBackendQueue()\n\t} else {\n\t\tdqLogf := func(level diskqueue.LogLevel, f string, args ...interface{}) {\n\t\t\topts := nsqd.getOpts()\n\t\t\tlg.Logf(opts.Logger, opts.LogLevel, lg.LogLevel(level), f, args...)\n\t\t}\n\t\tt.backend = diskqueue.New(\n\t\t\ttopicName,\n\t\t\tnsqd.getOpts().DataPath,\n\t\t\tnsqd.getOpts().MaxBytesPerFile,\n\t\t\tint32(minValidMsgLength),\n\t\t\tint32(nsqd.getOpts().MaxMsgSize)+minValidMsgLength,\n\t\t\tnsqd.getOpts().SyncEvery,\n\t\t\tnsqd.getOpts().SyncTimeout,\n\t\t\tdqLogf,\n\t\t)\n\t}\n\n\tt.waitGroup.Wrap(t.messagePump)\n\n\tt.nsqd.Notify(t, !t.ephemeral)\n\n\treturn t\n}\n\nfunc (t *Topic) Start() {\n\tselect {\n\tcase t.startChan <- 1:\n\tdefault:\n\t}\n}\n\n// Exiting returns a boolean indicating if this topic is closed/exiting\nfunc (t *Topic) Exiting() bool {\n\treturn atomic.LoadInt32(&t.exitFlag) == 1\n}\n\n// GetChannel performs a thread safe operation\n// to return a pointer to a Channel object (potentially new)\n// for the given Topic\nfunc (t *Topic) GetChannel(channelName string) *Channel {\n\tt.Lock()\n\tchannel, isNew := t.getOrCreateChannel(channelName)\n\tt.Unlock()\n\n\tif isNew {\n\t\t// update messagePump state\n\t\tselect {\n\t\tcase t.channelUpdateChan <- 1:\n\t\tcase <-t.exitChan:\n\t\t}\n\t}\n\n\treturn channel\n}\n\n// this expects the caller to handle locking\nfunc (t *Topic) getOrCreateChannel(channelName string) (*Channel, bool) {\n\tchannel, ok := t.channelMap[channelName]\n\tif !ok {\n\t\tdeleteCallback := func(c *Channel) {\n\t\t\tt.DeleteExistingChannel(c.name)\n\t\t}\n\t\tchannel = NewChannel(t.name, channelName, t.nsqd, deleteCallback)\n\t\tt.channelMap[channelName] = channel\n\t\tt.nsqd.logf(LOG_INFO, \"TOPIC(%s): new channel(%s)\", t.name, channel.name)\n\t\treturn channel, true\n\t}\n\treturn channel, false\n}\n\nfunc (t *Topic) GetExistingChannel(channelName string) (*Channel, error) {\n\tt.RLock()\n\tdefer t.RUnlock()\n\tchannel, ok := t.channelMap[channelName]\n\tif !ok {\n\t\treturn nil, errors.New(\"channel does not exist\")\n\t}\n\treturn channel, nil\n}\n\n// DeleteExistingChannel removes a channel from the topic only if it exists\nfunc (t *Topic) DeleteExistingChannel(channelName string) error {\n\tt.RLock()\n\tchannel, ok := t.channelMap[channelName]\n\tt.RUnlock()\n\tif !ok {\n\t\treturn errors.New(\"channel does not exist\")\n\t}\n\n\tt.nsqd.logf(LOG_INFO, \"TOPIC(%s): deleting channel %s\", t.name, channel.name)\n\n\t// delete empties the channel before closing\n\t// (so that we dont leave any messages around)\n\t//\n\t// we do this before removing the channel from map below (with no lock)\n\t// so that any incoming subs will error and not create a new channel\n\t// to enforce ordering\n\tchannel.Delete()\n\n\tt.Lock()\n\tdelete(t.channelMap, channelName)\n\tnumChannels := len(t.channelMap)\n\tt.Unlock()\n\n\t// update messagePump state\n\tselect {\n\tcase t.channelUpdateChan <- 1:\n\tcase <-t.exitChan:\n\t}\n\n\tif numChannels == 0 && t.ephemeral {\n\t\tgo t.deleter.Do(func() { t.deleteCallback(t) })\n\t}\n\n\treturn nil\n}\n\n// PutMessage writes a Message to the queue\nfunc (t *Topic) PutMessage(m *Message) error {\n\tt.RLock()\n\tdefer t.RUnlock()\n\tif atomic.LoadInt32(&t.exitFlag) == 1 {\n\t\treturn errors.New(\"exiting\")\n\t}\n\terr := t.put(m)\n\tif err != nil {\n\t\treturn err\n\t}\n\tatomic.AddUint64(&t.messageCount, 1)\n\tatomic.AddUint64(&t.messageBytes, uint64(len(m.Body)))\n\treturn nil\n}\n\n// PutMessages writes multiple Messages to the queue\nfunc (t *Topic) PutMessages(msgs []*Message) error {\n\tt.RLock()\n\tdefer t.RUnlock()\n\tif atomic.LoadInt32(&t.exitFlag) == 1 {\n\t\treturn errors.New(\"exiting\")\n\t}\n\n\tmessageTotalBytes := 0\n\n\tfor i, m := range msgs {\n\t\terr := t.put(m)\n\t\tif err != nil {\n\t\t\tatomic.AddUint64(&t.messageCount, uint64(i))\n\t\t\tatomic.AddUint64(&t.messageBytes, uint64(messageTotalBytes))\n\t\t\treturn err\n\t\t}\n\t\tmessageTotalBytes += len(m.Body)\n\t}\n\n\tatomic.AddUint64(&t.messageBytes, uint64(messageTotalBytes))\n\tatomic.AddUint64(&t.messageCount, uint64(len(msgs)))\n\treturn nil\n}\n\nfunc (t *Topic) put(m *Message) error {\n\t// If mem-queue-size == 0, avoid memory chan, for more consistent ordering,\n\t// but try to use memory chan for deferred messages (they lose deferred timer\n\t// in backend queue) or if topic is ephemeral (there is no backend queue).\n\tif cap(t.memoryMsgChan) > 0 || t.ephemeral || m.deferred != 0 {\n\t\tselect {\n\t\tcase t.memoryMsgChan <- m:\n\t\t\treturn nil\n\t\tdefault:\n\t\t\tbreak // write to backend\n\t\t}\n\t}\n\terr := writeMessageToBackend(m, t.backend)\n\tt.nsqd.SetHealth(err)\n\tif err != nil {\n\t\tt.nsqd.logf(LOG_ERROR,\n\t\t\t\"TOPIC(%s) ERROR: failed to write message to backend - %s\",\n\t\t\tt.name, err)\n\t\treturn err\n\t}\n\treturn nil\n}\n\nfunc (t *Topic) Depth() int64 {\n\treturn int64(len(t.memoryMsgChan)) + t.backend.Depth()\n}\n\n// messagePump selects over the in-memory and backend queue and\n// writes messages to every channel for this topic\nfunc (t *Topic) messagePump() {\n\tvar msg *Message\n\tvar buf []byte\n\tvar err error\n\tvar chans []*Channel\n\tvar memoryMsgChan chan *Message\n\tvar backendChan <-chan []byte\n\n\t// do not pass messages before Start(), but avoid blocking Pause() or GetChannel()\n\tfor {\n\t\tselect {\n\t\tcase <-t.channelUpdateChan:\n\t\t\tcontinue\n\t\tcase <-t.pauseChan:\n\t\t\tcontinue\n\t\tcase <-t.exitChan:\n\t\t\tgoto exit\n\t\tcase <-t.startChan:\n\t\t}\n\t\tbreak\n\t}\n\tt.RLock()\n\tfor _, c := range t.channelMap {\n\t\tchans = append(chans, c)\n\t}\n\tt.RUnlock()\n\tif len(chans) > 0 && !t.IsPaused() {\n\t\tmemoryMsgChan = t.memoryMsgChan\n\t\tbackendChan = t.backend.ReadChan()\n\t}\n\n\t// main message loop\n\tfor {\n\t\tselect {\n\t\tcase msg = <-memoryMsgChan:\n\t\tcase buf = <-backendChan:\n\t\t\tmsg, err = decodeMessage(buf)\n\t\t\tif err != nil {\n\t\t\t\tt.nsqd.logf(LOG_ERROR, \"failed to decode message - %s\", err)\n\t\t\t\tcontinue\n\t\t\t}\n\t\tcase <-t.channelUpdateChan:\n\t\t\tchans = chans[:0]\n\t\t\tt.RLock()\n\t\t\tfor _, c := range t.channelMap {\n\t\t\t\tchans = append(chans, c)\n\t\t\t}\n\t\t\tt.RUnlock()\n\t\t\tif len(chans) == 0 || t.IsPaused() {\n\t\t\t\tmemoryMsgChan = nil\n\t\t\t\tbackendChan = nil\n\t\t\t} else {\n\t\t\t\tmemoryMsgChan = t.memoryMsgChan\n\t\t\t\tbackendChan = t.backend.ReadChan()\n\t\t\t}\n\t\t\tcontinue\n\t\tcase <-t.pauseChan:\n\t\t\tif len(chans) == 0 || t.IsPaused() {\n\t\t\t\tmemoryMsgChan = nil\n\t\t\t\tbackendChan = nil\n\t\t\t} else {\n\t\t\t\tmemoryMsgChan = t.memoryMsgChan\n\t\t\t\tbackendChan = t.backend.ReadChan()\n\t\t\t}\n\t\t\tcontinue\n\t\tcase <-t.exitChan:\n\t\t\tgoto exit\n\t\t}\n\n\t\tfor i, channel := range chans {\n\t\t\tchanMsg := msg\n\t\t\t// copy the message because each channel\n\t\t\t// needs a unique instance but...\n\t\t\t// fastpath to avoid copy if its the first channel\n\t\t\t// (the topic already created the first copy)\n\t\t\tif i > 0 {\n\t\t\t\tchanMsg = NewMessage(msg.ID, msg.Body)\n\t\t\t\tchanMsg.Timestamp = msg.Timestamp\n\t\t\t\tchanMsg.deferred = msg.deferred\n\t\t\t}\n\t\t\tif chanMsg.deferred != 0 {\n\t\t\t\tchannel.PutMessageDeferred(chanMsg, chanMsg.deferred)\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\terr := channel.PutMessage(chanMsg)\n\t\t\tif err != nil {\n\t\t\t\tt.nsqd.logf(LOG_ERROR,\n\t\t\t\t\t\"TOPIC(%s) ERROR: failed to put msg(%s) to channel(%s) - %s\",\n\t\t\t\t\tt.name, msg.ID, channel.name, err)\n\t\t\t}\n\t\t}\n\t}\n\nexit:\n\tt.nsqd.logf(LOG_INFO, \"TOPIC(%s): closing ... messagePump\", t.name)\n}\n\n// Delete empties the topic and all its channels and closes\nfunc (t *Topic) Delete() error {\n\treturn t.exit(true)\n}\n\n// Close persists all outstanding topic data and closes all its channels\nfunc (t *Topic) Close() error {\n\treturn t.exit(false)\n}\n\nfunc (t *Topic) exit(deleted bool) error {\n\tif !atomic.CompareAndSwapInt32(&t.exitFlag, 0, 1) {\n\t\treturn errors.New(\"exiting\")\n\t}\n\n\tif deleted {\n\t\tt.nsqd.logf(LOG_INFO, \"TOPIC(%s): deleting\", t.name)\n\n\t\t// since we are explicitly deleting a topic (not just at system exit time)\n\t\t// de-register this from the lookupd\n\t\tt.nsqd.Notify(t, !t.ephemeral)\n\t} else {\n\t\tt.nsqd.logf(LOG_INFO, \"TOPIC(%s): closing\", t.name)\n\t}\n\n\tclose(t.exitChan)\n\n\t// synchronize the close of messagePump()\n\tt.waitGroup.Wait()\n\n\tif deleted {\n\t\tt.Lock()\n\t\tfor _, channel := range t.channelMap {\n\t\t\tdelete(t.channelMap, channel.name)\n\t\t\tchannel.Delete()\n\t\t}\n\t\tt.Unlock()\n\n\t\t// empty the queue (deletes the backend files, too)\n\t\tt.Empty()\n\t\treturn t.backend.Delete()\n\t}\n\n\t// close all the channels\n\tt.RLock()\n\tfor _, channel := range t.channelMap {\n\t\terr := channel.Close()\n\t\tif err != nil {\n\t\t\t// we need to continue regardless of error to close all the channels\n\t\t\tt.nsqd.logf(LOG_ERROR, \"channel(%s) close - %s\", channel.name, err)\n\t\t}\n\t}\n\tt.RUnlock()\n\n\t// write anything leftover to disk\n\tt.flush()\n\treturn t.backend.Close()\n}\n\nfunc (t *Topic) Empty() error {\n\tfor {\n\t\tselect {\n\t\tcase <-t.memoryMsgChan:\n\t\tdefault:\n\t\t\tgoto finish\n\t\t}\n\t}\n\nfinish:\n\treturn t.backend.Empty()\n}\n\nfunc (t *Topic) flush() error {\n\tif len(t.memoryMsgChan) > 0 {\n\t\tt.nsqd.logf(LOG_INFO,\n\t\t\t\"TOPIC(%s): flushing %d memory messages to backend\",\n\t\t\tt.name, len(t.memoryMsgChan))\n\t}\n\n\tfor {\n\t\tselect {\n\t\tcase msg := <-t.memoryMsgChan:\n\t\t\terr := writeMessageToBackend(msg, t.backend)\n\t\t\tif err != nil {\n\t\t\t\tt.nsqd.logf(LOG_ERROR,\n\t\t\t\t\t\"ERROR: failed to write message to backend - %s\", err)\n\t\t\t}\n\t\tdefault:\n\t\t\tgoto finish\n\t\t}\n\t}\n\nfinish:\n\treturn nil\n}\n\nfunc (t *Topic) AggregateChannelE2eProcessingLatency() *quantile.Quantile {\n\tvar latencyStream *quantile.Quantile\n\tt.RLock()\n\trealChannels := make([]*Channel, 0, len(t.channelMap))\n\tfor _, c := range t.channelMap {\n\t\trealChannels = append(realChannels, c)\n\t}\n\tt.RUnlock()\n\tfor _, c := range realChannels {\n\t\tif c.e2eProcessingLatencyStream == nil {\n\t\t\tcontinue\n\t\t}\n\t\tif latencyStream == nil {\n\t\t\tlatencyStream = quantile.New(\n\t\t\t\tt.nsqd.getOpts().E2EProcessingLatencyWindowTime,\n\t\t\t\tt.nsqd.getOpts().E2EProcessingLatencyPercentiles)\n\t\t}\n\t\tlatencyStream.Merge(c.e2eProcessingLatencyStream)\n\t}\n\treturn latencyStream\n}\n\nfunc (t *Topic) Pause() error {\n\treturn t.doPause(true)\n}\n\nfunc (t *Topic) UnPause() error {\n\treturn t.doPause(false)\n}\n\nfunc (t *Topic) doPause(pause bool) error {\n\tif pause {\n\t\tatomic.StoreInt32(&t.paused, 1)\n\t} else {\n\t\tatomic.StoreInt32(&t.paused, 0)\n\t}\n\n\tselect {\n\tcase t.pauseChan <- 1:\n\tcase <-t.exitChan:\n\t}\n\n\treturn nil\n}\n\nfunc (t *Topic) IsPaused() bool {\n\treturn atomic.LoadInt32(&t.paused) == 1\n}\n\nfunc (t *Topic) GenerateID() MessageID {\n\tvar i int64 = 0\n\tfor {\n\t\tid, err := t.idFactory.NewGUID()\n\t\tif err == nil {\n\t\t\treturn id.Hex()\n\t\t}\n\t\tif i%10000 == 0 {\n\t\t\tt.nsqd.logf(LOG_ERROR, \"TOPIC(%s): failed to create guid - %s\", t.name, err)\n\t\t}\n\t\ttime.Sleep(time.Millisecond)\n\t\ti++\n\t}\n}\n" }, { "path": "nsqd/topic_test.go", "content": "package nsqd\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"os\"\n\t\"runtime\"\n\t\"strconv\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\nfunc TestGetTopic(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopic1 := nsqd.GetTopic(\"test\")\n\ttest.NotNil(t, topic1)\n\ttest.Equal(t, \"test\", topic1.name)\n\n\ttopic2 := nsqd.GetTopic(\"test\")\n\ttest.Equal(t, topic1, topic2)\n\n\ttopic3 := nsqd.GetTopic(\"test2\")\n\ttest.Equal(t, \"test2\", topic3.name)\n\ttest.NotEqual(t, topic2, topic3)\n}\n\nfunc TestGetChannel(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopic := nsqd.GetTopic(\"test\")\n\n\tchannel1 := topic.GetChannel(\"ch1\")\n\ttest.NotNil(t, channel1)\n\ttest.Equal(t, \"ch1\", channel1.name)\n\n\tchannel2 := topic.GetChannel(\"ch2\")\n\n\ttest.Equal(t, channel1, topic.channelMap[\"ch1\"])\n\ttest.Equal(t, channel2, topic.channelMap[\"ch2\"])\n}\n\ntype errorBackendQueue struct{}\n\nfunc (d *errorBackendQueue) Put([]byte) error { return errors.New(\"never gonna happen\") }\nfunc (d *errorBackendQueue) ReadChan() <-chan []byte { return nil }\nfunc (d *errorBackendQueue) Close() error { return nil }\nfunc (d *errorBackendQueue) Delete() error { return nil }\nfunc (d *errorBackendQueue) Depth() int64 { return 0 }\nfunc (d *errorBackendQueue) Empty() error { return nil }\n\ntype errorRecoveredBackendQueue struct{ errorBackendQueue }\n\nfunc (d *errorRecoveredBackendQueue) Put([]byte) error { return nil }\n\nfunc TestHealth(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.MemQueueSize = 2\n\t_, httpAddr, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopic := nsqd.GetTopic(\"test\")\n\ttopic.backend = &errorBackendQueue{}\n\n\tmsg := NewMessage(topic.GenerateID(), make([]byte, 100))\n\terr := topic.PutMessage(msg)\n\ttest.Nil(t, err)\n\n\tmsg = NewMessage(topic.GenerateID(), make([]byte, 100))\n\terr = topic.PutMessages([]*Message{msg})\n\ttest.Nil(t, err)\n\n\tmsg = NewMessage(topic.GenerateID(), make([]byte, 100))\n\terr = topic.PutMessage(msg)\n\ttest.NotNil(t, err)\n\n\tmsg = NewMessage(topic.GenerateID(), make([]byte, 100))\n\terr = topic.PutMessages([]*Message{msg})\n\ttest.NotNil(t, err)\n\n\turl := fmt.Sprintf(\"http://%s/ping\", httpAddr)\n\tresp, err := http.Get(url)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 500, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"NOK - never gonna happen\", string(body))\n\n\ttopic.backend = &errorRecoveredBackendQueue{}\n\n\tmsg = NewMessage(topic.GenerateID(), make([]byte, 100))\n\terr = topic.PutMessages([]*Message{msg})\n\ttest.Nil(t, err)\n\n\tresp, err = http.Get(url)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\ttest.Equal(t, \"OK\", string(body))\n}\n\nfunc TestDeletes(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopic := nsqd.GetTopic(\"test\")\n\n\tchannel1 := topic.GetChannel(\"ch1\")\n\ttest.NotNil(t, channel1)\n\n\terr := topic.DeleteExistingChannel(\"ch1\")\n\ttest.Nil(t, err)\n\ttest.Equal(t, 0, len(topic.channelMap))\n\n\tchannel2 := topic.GetChannel(\"ch2\")\n\ttest.NotNil(t, channel2)\n\n\terr = nsqd.DeleteExistingTopic(\"test\")\n\ttest.Nil(t, err)\n\ttest.Equal(t, 0, len(topic.channelMap))\n\ttest.Equal(t, 0, len(nsqd.topicMap))\n}\n\nfunc TestDeleteLast(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopic := nsqd.GetTopic(\"test\")\n\n\tchannel1 := topic.GetChannel(\"ch1\")\n\ttest.NotNil(t, channel1)\n\n\terr := topic.DeleteExistingChannel(\"ch1\")\n\ttest.Nil(t, err)\n\ttest.Equal(t, 0, len(topic.channelMap))\n\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"aaaaaaaaaaaaaaaaaaaaaaaaaaa\"))\n\terr = topic.PutMessage(msg)\n\ttime.Sleep(100 * time.Millisecond)\n\ttest.Nil(t, err)\n\ttest.Equal(t, int64(1), topic.Depth())\n}\n\nfunc TestPause(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\n\ttopicName := \"test_topic_pause\" + strconv.Itoa(int(time.Now().Unix()))\n\ttopic := nsqd.GetTopic(topicName)\n\terr := topic.Pause()\n\ttest.Nil(t, err)\n\n\tchannel := topic.GetChannel(\"ch1\")\n\ttest.NotNil(t, channel)\n\n\tmsg := NewMessage(topic.GenerateID(), []byte(\"aaaaaaaaaaaaaaaaaaaaaaaaaaa\"))\n\terr = topic.PutMessage(msg)\n\ttest.Nil(t, err)\n\n\ttime.Sleep(15 * time.Millisecond)\n\n\ttest.Equal(t, int64(1), topic.Depth())\n\ttest.Equal(t, int64(0), channel.Depth())\n\n\terr = topic.UnPause()\n\ttest.Nil(t, err)\n\n\ttime.Sleep(15 * time.Millisecond)\n\n\ttest.Equal(t, int64(0), topic.Depth())\n\ttest.Equal(t, int64(1), channel.Depth())\n}\n\nfunc BenchmarkTopicPut(b *testing.B) {\n\tb.StopTimer()\n\ttopicName := \"bench_topic_put\" + strconv.Itoa(b.N)\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\tb.StartTimer()\n\n\tfor i := 0; i <= b.N; i++ {\n\t\ttopic := nsqd.GetTopic(topicName)\n\t\tmsg := NewMessage(topic.GenerateID(), []byte(\"aaaaaaaaaaaaaaaaaaaaaaaaaaa\"))\n\t\ttopic.PutMessage(msg)\n\t}\n}\n\nfunc BenchmarkTopicToChannelPut(b *testing.B) {\n\tb.StopTimer()\n\ttopicName := \"bench_topic_to_channel_put\" + strconv.Itoa(b.N)\n\tchannelName := \"bench\"\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(b)\n\topts.MemQueueSize = int64(b.N)\n\t_, _, nsqd := mustStartNSQD(opts)\n\tdefer os.RemoveAll(opts.DataPath)\n\tdefer nsqd.Exit()\n\tchannel := nsqd.GetTopic(topicName).GetChannel(channelName)\n\tb.StartTimer()\n\n\tfor i := 0; i <= b.N; i++ {\n\t\ttopic := nsqd.GetTopic(topicName)\n\t\tmsg := NewMessage(topic.GenerateID(), []byte(\"aaaaaaaaaaaaaaaaaaaaaaaaaaa\"))\n\t\ttopic.PutMessage(msg)\n\t}\n\n\tfor {\n\t\tif len(channel.memoryMsgChan) == b.N {\n\t\t\tbreak\n\t\t}\n\t\truntime.Gosched()\n\t}\n}\n" }, { "path": "nsqlookupd/README.md", "content": "## nsqlookupd\n\n`nsqlookupd` is the daemon that manages topology metadata and serves client requests to\ndiscover the location of topics at runtime.\n\nRead the [docs](https://nsq.io/components/nsqlookupd.html)\n" }, { "path": "nsqlookupd/client_v1.go", "content": "package nsqlookupd\n\nimport (\n\t\"net\"\n)\n\ntype ClientV1 struct {\n\tnet.Conn\n\tpeerInfo *PeerInfo\n}\n\nfunc NewClientV1(conn net.Conn) *ClientV1 {\n\treturn &ClientV1{\n\t\tConn: conn,\n\t}\n}\n\nfunc (c *ClientV1) String() string {\n\treturn c.RemoteAddr().String()\n}\n" }, { "path": "nsqlookupd/http.go", "content": "package nsqlookupd\n\nimport (\n\t\"fmt\"\n\t\"net/http\"\n\t\"net/http/pprof\"\n\t\"sync/atomic\"\n\n\t\"github.com/julienschmidt/httprouter\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\ntype httpServer struct {\n\tnsqlookupd *NSQLookupd\n\trouter http.Handler\n}\n\nfunc newHTTPServer(l *NSQLookupd) *httpServer {\n\tlog := http_api.Log(l.logf)\n\n\trouter := httprouter.New()\n\trouter.HandleMethodNotAllowed = true\n\trouter.PanicHandler = http_api.LogPanicHandler(l.logf)\n\trouter.NotFound = http_api.LogNotFoundHandler(l.logf)\n\trouter.MethodNotAllowed = http_api.LogMethodNotAllowedHandler(l.logf)\n\ts := &httpServer{\n\t\tnsqlookupd: l,\n\t\trouter: router,\n\t}\n\n\trouter.Handle(\"GET\", \"/ping\", http_api.Decorate(s.pingHandler, log, http_api.PlainText))\n\trouter.Handle(\"GET\", \"/info\", http_api.Decorate(s.doInfo, log, http_api.V1))\n\n\t// v1 negotiate\n\trouter.Handle(\"GET\", \"/debug\", http_api.Decorate(s.doDebug, log, http_api.V1))\n\trouter.Handle(\"GET\", \"/lookup\", http_api.Decorate(s.doLookup, log, http_api.V1))\n\trouter.Handle(\"GET\", \"/topics\", http_api.Decorate(s.doTopics, log, http_api.V1))\n\trouter.Handle(\"GET\", \"/channels\", http_api.Decorate(s.doChannels, log, http_api.V1))\n\trouter.Handle(\"GET\", \"/nodes\", http_api.Decorate(s.doNodes, log, http_api.V1))\n\n\t// only v1\n\trouter.Handle(\"POST\", \"/topic/create\", http_api.Decorate(s.doCreateTopic, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/topic/delete\", http_api.Decorate(s.doDeleteTopic, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/channel/create\", http_api.Decorate(s.doCreateChannel, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/channel/delete\", http_api.Decorate(s.doDeleteChannel, log, http_api.V1))\n\trouter.Handle(\"POST\", \"/topic/tombstone\", http_api.Decorate(s.doTombstoneTopicProducer, log, http_api.V1))\n\n\t// debug\n\trouter.HandlerFunc(\"GET\", \"/debug/pprof\", pprof.Index)\n\trouter.HandlerFunc(\"GET\", \"/debug/pprof/cmdline\", pprof.Cmdline)\n\trouter.HandlerFunc(\"GET\", \"/debug/pprof/symbol\", pprof.Symbol)\n\trouter.HandlerFunc(\"POST\", \"/debug/pprof/symbol\", pprof.Symbol)\n\trouter.HandlerFunc(\"GET\", \"/debug/pprof/profile\", pprof.Profile)\n\trouter.Handler(\"GET\", \"/debug/pprof/heap\", pprof.Handler(\"heap\"))\n\trouter.Handler(\"GET\", \"/debug/pprof/goroutine\", pprof.Handler(\"goroutine\"))\n\trouter.Handler(\"GET\", \"/debug/pprof/block\", pprof.Handler(\"block\"))\n\trouter.Handler(\"GET\", \"/debug/pprof/threadcreate\", pprof.Handler(\"threadcreate\"))\n\n\treturn s\n}\n\nfunc (s *httpServer) ServeHTTP(w http.ResponseWriter, req *http.Request) {\n\ts.router.ServeHTTP(w, req)\n}\n\nfunc (s *httpServer) pingHandler(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treturn \"OK\", nil\n}\n\nfunc (s *httpServer) doInfo(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treturn struct {\n\t\tVersion string `json:\"version\"`\n\t}{\n\t\tVersion: version.Binary,\n\t}, nil\n}\n\nfunc (s *httpServer) doTopics(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\ttopics := s.nsqlookupd.DB.FindRegistrations(\"topic\", \"*\", \"\").Keys()\n\treturn map[string]interface{}{\n\t\t\"topics\": topics,\n\t}, nil\n}\n\nfunc (s *httpServer) doChannels(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, err := reqParams.Get(\"topic\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"MISSING_ARG_TOPIC\"}\n\t}\n\n\tchannels := s.nsqlookupd.DB.FindRegistrations(\"channel\", topicName, \"*\").SubKeys()\n\treturn map[string]interface{}{\n\t\t\"channels\": channels,\n\t}, nil\n}\n\nfunc (s *httpServer) doLookup(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, err := reqParams.Get(\"topic\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"MISSING_ARG_TOPIC\"}\n\t}\n\n\tregistration := s.nsqlookupd.DB.FindRegistrations(\"topic\", topicName, \"\")\n\tif len(registration) == 0 {\n\t\treturn nil, http_api.Err{404, \"TOPIC_NOT_FOUND\"}\n\t}\n\n\tchannels := s.nsqlookupd.DB.FindRegistrations(\"channel\", topicName, \"*\").SubKeys()\n\tproducers := s.nsqlookupd.DB.FindProducers(\"topic\", topicName, \"\")\n\tproducers = producers.FilterByActive(s.nsqlookupd.opts.InactiveProducerTimeout,\n\t\ts.nsqlookupd.opts.TombstoneLifetime)\n\treturn map[string]interface{}{\n\t\t\"channels\": channels,\n\t\t\"producers\": producers.PeerInfo(),\n\t}, nil\n}\n\nfunc (s *httpServer) doCreateTopic(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, err := reqParams.Get(\"topic\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"MISSING_ARG_TOPIC\"}\n\t}\n\n\tif !protocol.IsValidTopicName(topicName) {\n\t\treturn nil, http_api.Err{400, \"INVALID_ARG_TOPIC\"}\n\t}\n\n\ts.nsqlookupd.logf(LOG_INFO, \"DB: adding topic(%s)\", topicName)\n\tkey := Registration{\"topic\", topicName, \"\"}\n\ts.nsqlookupd.DB.AddRegistration(key)\n\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doDeleteTopic(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, err := reqParams.Get(\"topic\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"MISSING_ARG_TOPIC\"}\n\t}\n\n\tregistrations := s.nsqlookupd.DB.FindRegistrations(\"channel\", topicName, \"*\")\n\tfor _, registration := range registrations {\n\t\ts.nsqlookupd.logf(LOG_INFO, \"DB: removing channel(%s) from topic(%s)\", registration.SubKey, topicName)\n\t\ts.nsqlookupd.DB.RemoveRegistration(registration)\n\t}\n\n\tregistrations = s.nsqlookupd.DB.FindRegistrations(\"topic\", topicName, \"\")\n\tfor _, registration := range registrations {\n\t\ts.nsqlookupd.logf(LOG_INFO, \"DB: removing topic(%s)\", topicName)\n\t\ts.nsqlookupd.DB.RemoveRegistration(registration)\n\t}\n\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doTombstoneTopicProducer(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, err := reqParams.Get(\"topic\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"MISSING_ARG_TOPIC\"}\n\t}\n\n\tnode, err := reqParams.Get(\"node\")\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"MISSING_ARG_NODE\"}\n\t}\n\n\ts.nsqlookupd.logf(LOG_INFO, \"DB: setting tombstone for producer@%s of topic(%s)\", node, topicName)\n\tproducers := s.nsqlookupd.DB.FindProducers(\"topic\", topicName, \"\")\n\tfor _, p := range producers {\n\t\tthisNode := fmt.Sprintf(\"%s:%d\", p.peerInfo.BroadcastAddress, p.peerInfo.HTTPPort)\n\t\tif thisNode == node {\n\t\t\tp.Tombstone()\n\t\t}\n\t}\n\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doCreateChannel(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, channelName, err := http_api.GetTopicChannelArgs(reqParams)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, err.Error()}\n\t}\n\n\ts.nsqlookupd.logf(LOG_INFO, \"DB: adding channel(%s) in topic(%s)\", channelName, topicName)\n\tkey := Registration{\"channel\", topicName, channelName}\n\ts.nsqlookupd.DB.AddRegistration(key)\n\n\ts.nsqlookupd.logf(LOG_INFO, \"DB: adding topic(%s)\", topicName)\n\tkey = Registration{\"topic\", topicName, \"\"}\n\ts.nsqlookupd.DB.AddRegistration(key)\n\n\treturn nil, nil\n}\n\nfunc (s *httpServer) doDeleteChannel(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\treqParams, err := http_api.NewReqParams(req)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, \"INVALID_REQUEST\"}\n\t}\n\n\ttopicName, channelName, err := http_api.GetTopicChannelArgs(reqParams)\n\tif err != nil {\n\t\treturn nil, http_api.Err{400, err.Error()}\n\t}\n\n\tregistrations := s.nsqlookupd.DB.FindRegistrations(\"channel\", topicName, channelName)\n\tif len(registrations) == 0 {\n\t\treturn nil, http_api.Err{404, \"CHANNEL_NOT_FOUND\"}\n\t}\n\n\ts.nsqlookupd.logf(LOG_INFO, \"DB: removing channel(%s) from topic(%s)\", channelName, topicName)\n\tfor _, registration := range registrations {\n\t\ts.nsqlookupd.DB.RemoveRegistration(registration)\n\t}\n\n\treturn nil, nil\n}\n\ntype node struct {\n\tRemoteAddress string `json:\"remote_address\"`\n\tHostname string `json:\"hostname\"`\n\tBroadcastAddress string `json:\"broadcast_address\"`\n\tTCPPort int `json:\"tcp_port\"`\n\tHTTPPort int `json:\"http_port\"`\n\tVersion string `json:\"version\"`\n\tToplogyZone string `json:\"topology_zone\"`\n\tToplogyRegion string `json:\"topology_region\"`\n\tTombstones []bool `json:\"tombstones\"`\n\tTopics []string `json:\"topics\"`\n}\n\nfunc (s *httpServer) doNodes(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\t// dont filter out tombstoned nodes\n\tproducers := s.nsqlookupd.DB.FindProducers(\"client\", \"\", \"\").FilterByActive(\n\t\ts.nsqlookupd.opts.InactiveProducerTimeout, 0)\n\tnodes := make([]*node, len(producers))\n\ttopicProducersMap := make(map[string]Producers)\n\tfor i, p := range producers {\n\t\ttopics := s.nsqlookupd.DB.LookupRegistrations(p.peerInfo.id).Filter(\"topic\", \"*\", \"\").Keys()\n\n\t\t// for each topic find the producer that matches this peer\n\t\t// to add tombstone information\n\t\ttombstones := make([]bool, len(topics))\n\t\tfor j, t := range topics {\n\t\t\tif _, exists := topicProducersMap[t]; !exists {\n\t\t\t\ttopicProducersMap[t] = s.nsqlookupd.DB.FindProducers(\"topic\", t, \"\")\n\t\t\t}\n\n\t\t\ttopicProducers := topicProducersMap[t]\n\t\t\tfor _, tp := range topicProducers {\n\t\t\t\tif tp.peerInfo == p.peerInfo {\n\t\t\t\t\ttombstones[j] = tp.IsTombstoned(s.nsqlookupd.opts.TombstoneLifetime)\n\t\t\t\t\tbreak\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\tnodes[i] = &node{\n\t\t\tRemoteAddress: p.peerInfo.RemoteAddress,\n\t\t\tHostname: p.peerInfo.Hostname,\n\t\t\tBroadcastAddress: p.peerInfo.BroadcastAddress,\n\t\t\tTCPPort: p.peerInfo.TCPPort,\n\t\t\tHTTPPort: p.peerInfo.HTTPPort,\n\t\t\tVersion: p.peerInfo.Version,\n\t\t\tToplogyZone: p.peerInfo.TopologyZone,\n\t\t\tToplogyRegion: p.peerInfo.TopologyRegion,\n\t\t\tTombstones: tombstones,\n\t\t\tTopics: topics,\n\t\t}\n\t}\n\n\treturn map[string]interface{}{\n\t\t\"producers\": nodes,\n\t}, nil\n}\n\nfunc (s *httpServer) doDebug(w http.ResponseWriter, req *http.Request, ps httprouter.Params) (interface{}, error) {\n\ts.nsqlookupd.DB.RLock()\n\tdefer s.nsqlookupd.DB.RUnlock()\n\n\tdata := make(map[string][]map[string]interface{})\n\tfor r, producers := range s.nsqlookupd.DB.registrationMap {\n\t\tkey := r.Category + \":\" + r.Key + \":\" + r.SubKey\n\t\tfor _, p := range producers {\n\t\t\tm := map[string]interface{}{\n\t\t\t\t\"id\": p.peerInfo.id,\n\t\t\t\t\"hostname\": p.peerInfo.Hostname,\n\t\t\t\t\"broadcast_address\": p.peerInfo.BroadcastAddress,\n\t\t\t\t\"tcp_port\": p.peerInfo.TCPPort,\n\t\t\t\t\"http_port\": p.peerInfo.HTTPPort,\n\t\t\t\t\"version\": p.peerInfo.Version,\n\t\t\t\t\"topology_zone\": p.peerInfo.TopologyZone,\n\t\t\t\t\"topology_region\": p.peerInfo.TopologyRegion,\n\t\t\t\t\"last_update\": atomic.LoadInt64(&p.peerInfo.lastUpdate),\n\t\t\t\t\"tombstoned\": p.tombstoned,\n\t\t\t\t\"tombstoned_at\": p.tombstonedAt.UnixNano(),\n\t\t\t}\n\t\t\tdata[key] = append(data[key], m)\n\t\t}\n\t}\n\n\treturn data, nil\n}\n" }, { "path": "nsqlookupd/http_test.go", "content": "package nsqlookupd\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"os\"\n\t\"strconv\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/test\"\n\t\"github.com/nsqio/nsq/internal/version\"\n\t\"github.com/nsqio/nsq/nsqd\"\n)\n\ntype InfoDoc struct {\n\tVersion string `json:\"version\"`\n}\n\ntype ChannelsDoc struct {\n\tChannels []interface{} `json:\"channels\"`\n}\n\ntype ErrMessage struct {\n\tMessage string `json:\"message\"`\n}\n\nfunc bootstrapNSQCluster(t *testing.T) (string, []*nsqd.NSQD, *NSQLookupd) {\n\tlgr := test.NewTestLogger(t)\n\n\tnsqlookupdOpts := NewOptions()\n\tnsqlookupdOpts.TCPAddress = \"127.0.0.1:0\"\n\tnsqlookupdOpts.HTTPAddress = \"127.0.0.1:0\"\n\tnsqlookupdOpts.BroadcastAddress = \"127.0.0.1\"\n\tnsqlookupdOpts.Logger = lgr\n\tnsqlookupd1, err := New(nsqlookupdOpts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := nsqlookupd1.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\tnsqdOpts := nsqd.NewOptions()\n\tnsqdOpts.TCPAddress = \"127.0.0.1:0\"\n\tnsqdOpts.HTTPAddress = \"127.0.0.1:0\"\n\tnsqdOpts.BroadcastAddress = \"127.0.0.1\"\n\tnsqdOpts.NSQLookupdTCPAddresses = []string{nsqlookupd1.RealTCPAddr().String()}\n\tnsqdOpts.Logger = lgr\n\ttmpDir, err := os.MkdirTemp(\"\", fmt.Sprintf(\"nsq-test-%d\", time.Now().UnixNano()))\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tnsqdOpts.DataPath = tmpDir\n\tnsqd1, err := nsqd.New(nsqdOpts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := nsqd1.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\n\ttime.Sleep(100 * time.Millisecond)\n\n\treturn tmpDir, []*nsqd.NSQD{nsqd1}, nsqlookupd1\n}\n\nfunc makeTopic(nsqlookupd *NSQLookupd, topicName string) {\n\tkey := Registration{\"topic\", topicName, \"\"}\n\tnsqlookupd.DB.AddRegistration(key)\n}\n\nfunc makeChannel(nsqlookupd *NSQLookupd, topicName string, channelName string) {\n\tkey := Registration{\"channel\", topicName, channelName}\n\tnsqlookupd.DB.AddRegistration(key)\n\tmakeTopic(nsqlookupd, topicName)\n}\n\nfunc TestPing(t *testing.T) {\n\tdataPath, nsqds, nsqlookupd1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupd1.Exit()\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/ping\", nsqlookupd1.RealHTTPAddr())\n\treq, _ := http.NewRequest(\"GET\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\ttest.Equal(t, []byte(\"OK\"), body)\n}\n\nfunc TestInfo(t *testing.T) {\n\tdataPath, nsqds, nsqlookupd1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupd1.Exit()\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/info\", nsqlookupd1.RealHTTPAddr())\n\treq, _ := http.NewRequest(\"GET\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\tinfo := InfoDoc{}\n\terr = json.Unmarshal(body, &info)\n\ttest.Nil(t, err)\n\ttest.Equal(t, version.Binary, info.Version)\n}\n\nfunc TestCreateTopic(t *testing.T) {\n\tdataPath, nsqds, nsqlookupd1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupd1.Exit()\n\n\tem := ErrMessage{}\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/topic/create\", nsqlookupd1.RealHTTPAddr())\n\n\treq, _ := http.NewRequest(\"POST\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_TOPIC\", em.Message)\n\n\ttopicName := \"sampletopicA\" + strconv.Itoa(int(time.Now().Unix())) + \"$\"\n\turl = fmt.Sprintf(\"http://%s/topic/create?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"INVALID_ARG_TOPIC\", em.Message)\n\n\ttopicName = \"sampletopicA\" + strconv.Itoa(int(time.Now().Unix()))\n\turl = fmt.Sprintf(\"http://%s/topic/create?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\ttest.Equal(t, []byte(\"\"), body)\n}\n\nfunc TestDeleteTopic(t *testing.T) {\n\tdataPath, nsqds, nsqlookupd1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupd1.Exit()\n\n\tem := ErrMessage{}\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/topic/delete\", nsqlookupd1.RealHTTPAddr())\n\n\treq, _ := http.NewRequest(\"POST\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_TOPIC\", em.Message)\n\n\ttopicName := \"sampletopicA\" + strconv.Itoa(int(time.Now().Unix()))\n\tmakeTopic(nsqlookupd1, topicName)\n\n\turl = fmt.Sprintf(\"http://%s/topic/delete?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\ttest.Equal(t, []byte(\"\"), body)\n\n\ttopicName = \"sampletopicB\" + strconv.Itoa(int(time.Now().Unix()))\n\tchannelName := \"foobar\" + strconv.Itoa(int(time.Now().Unix()))\n\tmakeChannel(nsqlookupd1, topicName, channelName)\n\n\turl = fmt.Sprintf(\"http://%s/topic/delete?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\ttest.Equal(t, []byte(\"\"), body)\n}\n\nfunc TestGetChannels(t *testing.T) {\n\tdataPath, nsqds, nsqlookupd1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupd1.Exit()\n\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/channels\", nsqlookupd1.RealHTTPAddr())\n\n\tem := ErrMessage{}\n\treq, _ := http.NewRequest(\"GET\", url, nil)\n\treq.Header.Add(\"Accept\", \"application/vnd.nsq; version=1.0\")\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_TOPIC\", em.Message)\n\n\tch := ChannelsDoc{}\n\ttopicName := \"sampletopicA\" + strconv.Itoa(int(time.Now().Unix()))\n\tmakeTopic(nsqlookupd1, topicName)\n\n\turl = fmt.Sprintf(\"http://%s/channels?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"GET\", url, nil)\n\treq.Header.Add(\"Accept\", \"application/vnd.nsq; version=1.0\")\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &ch)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 0, len(ch.Channels))\n\n\ttopicName = \"sampletopicB\" + strconv.Itoa(int(time.Now().Unix()))\n\tchannelName := \"foobar\" + strconv.Itoa(int(time.Now().Unix()))\n\tmakeChannel(nsqlookupd1, topicName, channelName)\n\n\turl = fmt.Sprintf(\"http://%s/channels?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"GET\", url, nil)\n\treq.Header.Add(\"Accept\", \"application/vnd.nsq; version=1.0\")\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &ch)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 1, len(ch.Channels))\n\ttest.Equal(t, channelName, ch.Channels[0])\n}\n\nfunc TestCreateChannel(t *testing.T) {\n\tdataPath, nsqds, nsqlookupd1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupd1.Exit()\n\n\tem := ErrMessage{}\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/channel/create\", nsqlookupd1.RealHTTPAddr())\n\n\treq, _ := http.NewRequest(\"POST\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_TOPIC\", em.Message)\n\n\ttopicName := \"sampletopicB\" + strconv.Itoa(int(time.Now().Unix())) + \"$\"\n\turl = fmt.Sprintf(\"http://%s/channel/create?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"INVALID_ARG_TOPIC\", em.Message)\n\n\ttopicName = \"sampletopicB\" + strconv.Itoa(int(time.Now().Unix()))\n\turl = fmt.Sprintf(\"http://%s/channel/create?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_CHANNEL\", em.Message)\n\n\tchannelName := \"foobar\" + strconv.Itoa(int(time.Now().Unix())) + \"$\"\n\turl = fmt.Sprintf(\"http://%s/channel/create?topic=%s&channel=%s\", nsqlookupd1.RealHTTPAddr(), topicName, channelName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"INVALID_ARG_CHANNEL\", em.Message)\n\n\tchannelName = \"foobar\" + strconv.Itoa(int(time.Now().Unix()))\n\turl = fmt.Sprintf(\"http://%s/channel/create?topic=%s&channel=%s\", nsqlookupd1.RealHTTPAddr(), topicName, channelName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\ttest.Equal(t, []byte(\"\"), body)\n}\n\nfunc TestDeleteChannel(t *testing.T) {\n\tdataPath, nsqds, nsqlookupd1 := bootstrapNSQCluster(t)\n\tdefer os.RemoveAll(dataPath)\n\tdefer nsqds[0].Exit()\n\tdefer nsqlookupd1.Exit()\n\n\tem := ErrMessage{}\n\tclient := http.Client{}\n\turl := fmt.Sprintf(\"http://%s/channel/delete\", nsqlookupd1.RealHTTPAddr())\n\n\treq, _ := http.NewRequest(\"POST\", url, nil)\n\tresp, err := client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ := io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_TOPIC\", em.Message)\n\n\ttopicName := \"sampletopicB\" + strconv.Itoa(int(time.Now().Unix())) + \"$\"\n\turl = fmt.Sprintf(\"http://%s/channel/delete?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"INVALID_ARG_TOPIC\", em.Message)\n\n\ttopicName = \"sampletopicB\" + strconv.Itoa(int(time.Now().Unix()))\n\turl = fmt.Sprintf(\"http://%s/channel/delete?topic=%s\", nsqlookupd1.RealHTTPAddr(), topicName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"MISSING_ARG_CHANNEL\", em.Message)\n\n\tchannelName := \"foobar\" + strconv.Itoa(int(time.Now().Unix())) + \"$\"\n\turl = fmt.Sprintf(\"http://%s/channel/delete?topic=%s&channel=%s\", nsqlookupd1.RealHTTPAddr(), topicName, channelName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 400, resp.StatusCode)\n\ttest.Equal(t, \"Bad Request\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"INVALID_ARG_CHANNEL\", em.Message)\n\n\tchannelName = \"foobar\" + strconv.Itoa(int(time.Now().Unix()))\n\turl = fmt.Sprintf(\"http://%s/channel/delete?topic=%s&channel=%s\", nsqlookupd1.RealHTTPAddr(), topicName, channelName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 404, resp.StatusCode)\n\ttest.Equal(t, \"Not Found\", http.StatusText(resp.StatusCode))\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\terr = json.Unmarshal(body, &em)\n\ttest.Nil(t, err)\n\ttest.Equal(t, \"CHANNEL_NOT_FOUND\", em.Message)\n\n\tmakeChannel(nsqlookupd1, topicName, channelName)\n\n\treq, _ = http.NewRequest(\"POST\", url, nil)\n\tresp, err = client.Do(req)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 200, resp.StatusCode)\n\tbody, _ = io.ReadAll(resp.Body)\n\tresp.Body.Close()\n\n\tt.Logf(\"%s\", body)\n\ttest.Equal(t, []byte(\"\"), body)\n}\n" }, { "path": "nsqlookupd/logger.go", "content": "package nsqlookupd\n\nimport (\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype Logger lg.Logger\n\nconst (\n\tLOG_DEBUG = lg.DEBUG\n\tLOG_INFO = lg.INFO\n\tLOG_WARN = lg.WARN\n\tLOG_ERROR = lg.ERROR\n\tLOG_FATAL = lg.FATAL\n)\n\nfunc (n *NSQLookupd) logf(level lg.LogLevel, f string, args ...interface{}) {\n\tlg.Logf(n.opts.Logger, n.opts.LogLevel, level, f, args...)\n}\n" }, { "path": "nsqlookupd/lookup_protocol_v1.go", "content": "package nsqlookupd\n\nimport (\n\t\"bufio\"\n\t\"encoding/binary\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"log\"\n\t\"net\"\n\t\"os\"\n\t\"strings\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\ntype LookupProtocolV1 struct {\n\tnsqlookupd *NSQLookupd\n}\n\nfunc (p *LookupProtocolV1) NewClient(conn net.Conn) protocol.Client {\n\treturn NewClientV1(conn)\n}\n\nfunc (p *LookupProtocolV1) IOLoop(c protocol.Client) error {\n\tvar err error\n\tvar line string\n\n\tclient := c.(*ClientV1)\n\n\treader := bufio.NewReader(client)\n\tfor {\n\t\tline, err = reader.ReadString('\\n')\n\t\tif err != nil {\n\t\t\tbreak\n\t\t}\n\n\t\tline = strings.TrimSpace(line)\n\t\tparams := strings.Split(line, \" \")\n\n\t\tvar response []byte\n\t\tresponse, err = p.Exec(client, reader, params)\n\t\tif err != nil {\n\t\t\tctx := \"\"\n\t\t\tif parentErr := err.(protocol.ChildErr).Parent(); parentErr != nil {\n\t\t\t\tctx = \" - \" + parentErr.Error()\n\t\t\t}\n\t\t\tp.nsqlookupd.logf(LOG_ERROR, \"[%s] - %s%s\", client, err, ctx)\n\n\t\t\t_, sendErr := protocol.SendResponse(client, []byte(err.Error()))\n\t\t\tif sendErr != nil {\n\t\t\t\tp.nsqlookupd.logf(LOG_ERROR, \"[%s] - %s%s\", client, sendErr, ctx)\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\t// errors of type FatalClientErr should forceably close the connection\n\t\t\tif _, ok := err.(*protocol.FatalClientErr); ok {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tcontinue\n\t\t}\n\n\t\tif response != nil {\n\t\t\t_, err = protocol.SendResponse(client, response)\n\t\t\tif err != nil {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}\n\n\tp.nsqlookupd.logf(LOG_INFO, \"PROTOCOL(V1): [%s] exiting ioloop\", client)\n\n\tif client.peerInfo != nil {\n\t\tregistrations := p.nsqlookupd.DB.LookupRegistrations(client.peerInfo.id)\n\t\tfor _, r := range registrations {\n\t\t\tif removed, _ := p.nsqlookupd.DB.RemoveProducer(r, client.peerInfo.id); removed {\n\t\t\t\tp.nsqlookupd.logf(LOG_INFO, \"DB: client(%s) UNREGISTER category:%s key:%s subkey:%s\",\n\t\t\t\t\tclient, r.Category, r.Key, r.SubKey)\n\t\t\t}\n\t\t}\n\t}\n\n\treturn err\n}\n\nfunc (p *LookupProtocolV1) Exec(client *ClientV1, reader *bufio.Reader, params []string) ([]byte, error) {\n\tswitch params[0] {\n\tcase \"PING\":\n\t\treturn p.PING(client, params)\n\tcase \"IDENTIFY\":\n\t\treturn p.IDENTIFY(client, reader, params[1:])\n\tcase \"REGISTER\":\n\t\treturn p.REGISTER(client, reader, params[1:])\n\tcase \"UNREGISTER\":\n\t\treturn p.UNREGISTER(client, reader, params[1:])\n\t}\n\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", fmt.Sprintf(\"invalid command %s\", params[0]))\n}\n\nfunc getTopicChan(command string, params []string) (string, string, error) {\n\tif len(params) == 0 {\n\t\treturn \"\", \"\", protocol.NewFatalClientErr(nil, \"E_INVALID\", fmt.Sprintf(\"%s insufficient number of params\", command))\n\t}\n\n\ttopicName := params[0]\n\tvar channelName string\n\tif len(params) >= 2 {\n\t\tchannelName = params[1]\n\t}\n\n\tif !protocol.IsValidTopicName(topicName) {\n\t\treturn \"\", \"\", protocol.NewFatalClientErr(nil, \"E_BAD_TOPIC\", fmt.Sprintf(\"%s topic name '%s' is not valid\", command, topicName))\n\t}\n\n\tif channelName != \"\" && !protocol.IsValidChannelName(channelName) {\n\t\treturn \"\", \"\", protocol.NewFatalClientErr(nil, \"E_BAD_CHANNEL\", fmt.Sprintf(\"%s channel name '%s' is not valid\", command, channelName))\n\t}\n\n\treturn topicName, channelName, nil\n}\n\nfunc (p *LookupProtocolV1) REGISTER(client *ClientV1, reader *bufio.Reader, params []string) ([]byte, error) {\n\tif client.peerInfo == nil {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"client must IDENTIFY\")\n\t}\n\n\ttopic, channel, err := getTopicChan(\"REGISTER\", params)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif channel != \"\" {\n\t\tkey := Registration{\"channel\", topic, channel}\n\t\tif p.nsqlookupd.DB.AddProducer(key, &Producer{peerInfo: client.peerInfo}) {\n\t\t\tp.nsqlookupd.logf(LOG_INFO, \"DB: client(%s) REGISTER category:%s key:%s subkey:%s\",\n\t\t\t\tclient, \"channel\", topic, channel)\n\t\t}\n\t}\n\tkey := Registration{\"topic\", topic, \"\"}\n\tif p.nsqlookupd.DB.AddProducer(key, &Producer{peerInfo: client.peerInfo}) {\n\t\tp.nsqlookupd.logf(LOG_INFO, \"DB: client(%s) REGISTER category:%s key:%s subkey:%s\",\n\t\t\tclient, \"topic\", topic, \"\")\n\t}\n\n\treturn []byte(\"OK\"), nil\n}\n\nfunc (p *LookupProtocolV1) UNREGISTER(client *ClientV1, reader *bufio.Reader, params []string) ([]byte, error) {\n\tif client.peerInfo == nil {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_INVALID\", \"client must IDENTIFY\")\n\t}\n\n\ttopic, channel, err := getTopicChan(\"UNREGISTER\", params)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif channel != \"\" {\n\t\tkey := Registration{\"channel\", topic, channel}\n\t\tremoved, left := p.nsqlookupd.DB.RemoveProducer(key, client.peerInfo.id)\n\t\tif removed {\n\t\t\tp.nsqlookupd.logf(LOG_INFO, \"DB: client(%s) UNREGISTER category:%s key:%s subkey:%s\",\n\t\t\t\tclient, \"channel\", topic, channel)\n\t\t}\n\t\t// for ephemeral channels, remove the channel as well if it has no producers\n\t\tif left == 0 && strings.HasSuffix(channel, \"#ephemeral\") {\n\t\t\tp.nsqlookupd.DB.RemoveRegistration(key)\n\t\t}\n\t} else {\n\t\t// no channel was specified so this is a topic unregistration\n\t\t// remove all of the channel registrations...\n\t\t// normally this shouldn't happen which is why we print a warning message\n\t\t// if anything is actually removed\n\t\tregistrations := p.nsqlookupd.DB.FindRegistrations(\"channel\", topic, \"*\")\n\t\tfor _, r := range registrations {\n\t\t\tremoved, _ := p.nsqlookupd.DB.RemoveProducer(r, client.peerInfo.id)\n\t\t\tif removed {\n\t\t\t\tp.nsqlookupd.logf(LOG_WARN, \"client(%s) unexpected UNREGISTER category:%s key:%s subkey:%s\",\n\t\t\t\t\tclient, \"channel\", topic, r.SubKey)\n\t\t\t}\n\t\t}\n\n\t\tkey := Registration{\"topic\", topic, \"\"}\n\t\tremoved, left := p.nsqlookupd.DB.RemoveProducer(key, client.peerInfo.id)\n\t\tif removed {\n\t\t\tp.nsqlookupd.logf(LOG_INFO, \"DB: client(%s) UNREGISTER category:%s key:%s subkey:%s\",\n\t\t\t\tclient, \"topic\", topic, \"\")\n\t\t}\n\t\tif left == 0 && strings.HasSuffix(topic, \"#ephemeral\") {\n\t\t\tp.nsqlookupd.DB.RemoveRegistration(key)\n\t\t}\n\t}\n\n\treturn []byte(\"OK\"), nil\n}\n\nfunc (p *LookupProtocolV1) IDENTIFY(client *ClientV1, reader *bufio.Reader, params []string) ([]byte, error) {\n\tvar err error\n\n\tif client.peerInfo != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_INVALID\", \"cannot IDENTIFY again\")\n\t}\n\n\tvar bodyLen int32\n\terr = binary.Read(reader, binary.BigEndian, &bodyLen)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"IDENTIFY failed to read body size\")\n\t}\n\n\tbody := make([]byte, bodyLen)\n\t_, err = io.ReadFull(reader, body)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"IDENTIFY failed to read body\")\n\t}\n\n\t// body is a json structure with producer information\n\tpeerInfo := PeerInfo{id: client.RemoteAddr().String()}\n\terr = json.Unmarshal(body, &peerInfo)\n\tif err != nil {\n\t\treturn nil, protocol.NewFatalClientErr(err, \"E_BAD_BODY\", \"IDENTIFY failed to decode JSON body\")\n\t}\n\n\tpeerInfo.RemoteAddress = client.RemoteAddr().String()\n\n\t// require all fields\n\tif peerInfo.BroadcastAddress == \"\" || peerInfo.TCPPort == 0 || peerInfo.HTTPPort == 0 || peerInfo.Version == \"\" {\n\t\treturn nil, protocol.NewFatalClientErr(nil, \"E_BAD_BODY\", \"IDENTIFY missing fields\")\n\t}\n\n\tatomic.StoreInt64(&peerInfo.lastUpdate, time.Now().UnixNano())\n\n\tp.nsqlookupd.logf(LOG_INFO, \"CLIENT(%s): IDENTIFY Address:%s TCP:%d HTTP:%d Version:%s\",\n\t\tclient, peerInfo.BroadcastAddress, peerInfo.TCPPort, peerInfo.HTTPPort, peerInfo.Version)\n\n\tclient.peerInfo = &peerInfo\n\tif p.nsqlookupd.DB.AddProducer(Registration{\"client\", \"\", \"\"}, &Producer{peerInfo: client.peerInfo}) {\n\t\tp.nsqlookupd.logf(LOG_INFO, \"DB: client(%s) REGISTER category:%s key:%s subkey:%s\", client, \"client\", \"\", \"\")\n\t}\n\n\t// build a response\n\tdata := make(map[string]interface{})\n\tdata[\"tcp_port\"] = p.nsqlookupd.RealTCPAddr().Port\n\tdata[\"http_port\"] = p.nsqlookupd.RealHTTPAddr().Port\n\tdata[\"version\"] = version.Binary\n\thostname, err := os.Hostname()\n\tif err != nil {\n\t\tlog.Fatalf(\"ERROR: unable to get hostname %s\", err)\n\t}\n\tdata[\"broadcast_address\"] = p.nsqlookupd.opts.BroadcastAddress\n\tdata[\"hostname\"] = hostname\n\n\tresponse, err := json.Marshal(data)\n\tif err != nil {\n\t\tp.nsqlookupd.logf(LOG_ERROR, \"marshaling %v\", data)\n\t\treturn []byte(\"OK\"), nil\n\t}\n\treturn response, nil\n}\n\nfunc (p *LookupProtocolV1) PING(client *ClientV1, params []string) ([]byte, error) {\n\tif client.peerInfo != nil {\n\t\t// we could get a PING before other commands on the same client connection\n\t\tcur := time.Unix(0, atomic.LoadInt64(&client.peerInfo.lastUpdate))\n\t\tnow := time.Now()\n\t\tp.nsqlookupd.logf(LOG_INFO, \"CLIENT(%s): pinged (last ping %s)\", client.peerInfo.id,\n\t\t\tnow.Sub(cur))\n\t\tatomic.StoreInt64(&client.peerInfo.lastUpdate, now.UnixNano())\n\t}\n\treturn []byte(\"OK\"), nil\n}\n" }, { "path": "nsqlookupd/lookup_protocol_v1_test.go", "content": "package nsqlookupd\n\nimport (\n\t\"errors\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\nfunc TestIOLoopReturnsClientErrWhenSendFails(t *testing.T) {\n\tfakeConn := test.NewFakeNetConn()\n\tfakeConn.WriteFunc = func(b []byte) (int, error) {\n\t\treturn 0, errors.New(\"write error\")\n\t}\n\n\ttestIOLoopReturnsClientErr(t, fakeConn)\n}\n\nfunc TestIOLoopReturnsClientErrWhenSendSucceeds(t *testing.T) {\n\tfakeConn := test.NewFakeNetConn()\n\tfakeConn.WriteFunc = func(b []byte) (int, error) {\n\t\treturn len(b), nil\n\t}\n\n\ttestIOLoopReturnsClientErr(t, fakeConn)\n}\n\nfunc testIOLoopReturnsClientErr(t *testing.T, fakeConn test.FakeNetConn) {\n\tfakeConn.ReadFunc = func(b []byte) (int, error) {\n\t\treturn copy(b, []byte(\"INVALID_COMMAND\\n\")), nil\n\t}\n\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.LogLevel = LOG_DEBUG\n\topts.TCPAddress = \"127.0.0.1:0\"\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\n\tnsqlookupd, err := New(opts)\n\ttest.Nil(t, err)\n\tprot := &LookupProtocolV1{nsqlookupd: nsqlookupd}\n\n\tnsqlookupd.tcpServer = &tcpServer{nsqlookupd: prot.nsqlookupd}\n\n\terrChan := make(chan error)\n\ttestIOLoop := func() {\n\t\tclient := prot.NewClient(fakeConn)\n\t\terrChan <- prot.IOLoop(client)\n\t\tdefer prot.nsqlookupd.Exit()\n\t}\n\tgo testIOLoop()\n\n\tvar timeout bool\n\n\tselect {\n\tcase err = <-errChan:\n\tcase <-time.After(2 * time.Second):\n\t\ttimeout = true\n\t}\n\n\ttest.Equal(t, false, timeout)\n\n\ttest.NotNil(t, err)\n\ttest.Equal(t, \"E_INVALID invalid command INVALID_COMMAND\", err.Error())\n\ttest.NotNil(t, err.(*protocol.FatalClientErr))\n}\n" }, { "path": "nsqlookupd/nsqlookupd.go", "content": "package nsqlookupd\n\nimport (\n\t\"fmt\"\n\t\"log\"\n\t\"net\"\n\t\"os\"\n\t\"sync\"\n\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/protocol\"\n\t\"github.com/nsqio/nsq/internal/util\"\n\t\"github.com/nsqio/nsq/internal/version\"\n)\n\ntype NSQLookupd struct {\n\tsync.RWMutex\n\topts *Options\n\ttcpListener net.Listener\n\thttpListener net.Listener\n\ttcpServer *tcpServer\n\twaitGroup util.WaitGroupWrapper\n\tDB *RegistrationDB\n}\n\nfunc New(opts *Options) (*NSQLookupd, error) {\n\tvar err error\n\n\tif opts.Logger == nil {\n\t\topts.Logger = log.New(os.Stderr, opts.LogPrefix, log.Ldate|log.Ltime|log.Lmicroseconds)\n\t}\n\tl := &NSQLookupd{\n\t\topts: opts,\n\t\tDB: NewRegistrationDB(),\n\t}\n\n\tl.logf(LOG_INFO, version.String(\"nsqlookupd\"))\n\n\tl.tcpServer = &tcpServer{nsqlookupd: l}\n\tl.tcpListener, err = net.Listen(\"tcp\", opts.TCPAddress)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"listen (%s) failed - %s\", opts.TCPAddress, err)\n\t}\n\tl.httpListener, err = net.Listen(\"tcp\", opts.HTTPAddress)\n\tif err != nil {\n\t\treturn nil, fmt.Errorf(\"listen (%s) failed - %s\", opts.HTTPAddress, err)\n\t}\n\n\treturn l, nil\n}\n\n// Main starts an instance of nsqlookupd and returns an\n// error if there was a problem starting up.\nfunc (l *NSQLookupd) Main() error {\n\texitCh := make(chan error)\n\tvar once sync.Once\n\texitFunc := func(err error) {\n\t\tonce.Do(func() {\n\t\t\tif err != nil {\n\t\t\t\tl.logf(LOG_FATAL, \"%s\", err)\n\t\t\t}\n\t\t\texitCh <- err\n\t\t})\n\t}\n\n\tl.waitGroup.Wrap(func() {\n\t\texitFunc(protocol.TCPServer(l.tcpListener, l.tcpServer, l.logf))\n\t})\n\thttpServer := newHTTPServer(l)\n\tl.waitGroup.Wrap(func() {\n\t\texitFunc(http_api.Serve(l.httpListener, httpServer, \"HTTP\", l.logf))\n\t})\n\n\terr := <-exitCh\n\treturn err\n}\n\nfunc (l *NSQLookupd) RealTCPAddr() *net.TCPAddr {\n\treturn l.tcpListener.Addr().(*net.TCPAddr)\n}\n\nfunc (l *NSQLookupd) RealHTTPAddr() *net.TCPAddr {\n\treturn l.httpListener.Addr().(*net.TCPAddr)\n}\n\nfunc (l *NSQLookupd) Exit() {\n\tif l.tcpListener != nil {\n\t\tl.tcpListener.Close()\n\t}\n\n\tif l.tcpServer != nil {\n\t\tl.tcpServer.Close()\n\t}\n\n\tif l.httpListener != nil {\n\t\tl.httpListener.Close()\n\t}\n\tl.waitGroup.Wait()\n}\n" }, { "path": "nsqlookupd/nsqlookupd_test.go", "content": "package nsqlookupd\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/nsqio/go-nsq\"\n\t\"github.com/nsqio/nsq/internal/clusterinfo\"\n\t\"github.com/nsqio/nsq/internal/http_api\"\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\nconst (\n\tConnectTimeout = 2 * time.Second\n\tRequestTimeout = 5 * time.Second\n\tTCPPort = 5000\n\tHTTPPort = 5555\n\tHostAddr = \"ip.address\"\n\tNSQDVersion = \"fake-version\"\n)\n\ntype ProducersDoc struct {\n\tProducers []interface{} `json:\"producers\"`\n}\n\ntype TopicsDoc struct {\n\tTopics []interface{} `json:\"topics\"`\n}\n\ntype LookupDoc struct {\n\tChannels []interface{} `json:\"channels\"`\n\tProducers []*PeerInfo `json:\"producers\"`\n}\n\nfunc mustStartLookupd(opts *Options) (*net.TCPAddr, *net.TCPAddr, *NSQLookupd) {\n\topts.TCPAddress = \"127.0.0.1:0\"\n\topts.HTTPAddress = \"127.0.0.1:0\"\n\tnsqlookupd, err := New(opts)\n\tif err != nil {\n\t\tpanic(err)\n\t}\n\tgo func() {\n\t\terr := nsqlookupd.Main()\n\t\tif err != nil {\n\t\t\tpanic(err)\n\t\t}\n\t}()\n\treturn nsqlookupd.RealTCPAddr(), nsqlookupd.RealHTTPAddr(), nsqlookupd\n}\n\nfunc mustConnectLookupd(t *testing.T, tcpAddr *net.TCPAddr) net.Conn {\n\tconn, err := net.DialTimeout(\"tcp\", tcpAddr.String(), time.Second)\n\tif err != nil {\n\t\tt.Fatal(\"failed to connect to lookupd\")\n\t}\n\tconn.Write(nsq.MagicV1)\n\treturn conn\n}\n\nfunc identify(t *testing.T, conn net.Conn) {\n\tci := make(map[string]interface{})\n\tci[\"tcp_port\"] = TCPPort\n\tci[\"http_port\"] = HTTPPort\n\tci[\"broadcast_address\"] = HostAddr\n\tci[\"hostname\"] = HostAddr\n\tci[\"version\"] = NSQDVersion\n\tcmd, _ := nsq.Identify(ci)\n\t_, err := cmd.WriteTo(conn)\n\ttest.Nil(t, err)\n\t_, err = nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n}\n\nfunc TestBasicLookupd(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\ttcpAddr, httpAddr, nsqlookupd := mustStartLookupd(opts)\n\tdefer nsqlookupd.Exit()\n\n\ttopics := nsqlookupd.DB.FindRegistrations(\"topic\", \"*\", \"*\")\n\ttest.Equal(t, 0, len(topics))\n\n\ttopicName := \"connectmsg\"\n\n\tconn := mustConnectLookupd(t, tcpAddr)\n\n\tidentify(t, conn)\n\n\tnsq.Register(topicName, \"channel1\").WriteTo(conn)\n\tv, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\ttest.Equal(t, []byte(\"OK\"), v)\n\n\tpr := ProducersDoc{}\n\tendpoint := fmt.Sprintf(\"http://%s/nodes\", httpAddr)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &pr)\n\ttest.Nil(t, err)\n\n\tt.Logf(\"got %v\", pr)\n\ttest.Equal(t, 1, len(pr.Producers))\n\n\ttopics = nsqlookupd.DB.FindRegistrations(\"topic\", topicName, \"\")\n\ttest.Equal(t, 1, len(topics))\n\n\tproducers := nsqlookupd.DB.FindProducers(\"topic\", topicName, \"\")\n\ttest.Equal(t, 1, len(producers))\n\tproducer := producers[0]\n\n\ttest.Equal(t, HostAddr, producer.peerInfo.BroadcastAddress)\n\ttest.Equal(t, HostAddr, producer.peerInfo.Hostname)\n\ttest.Equal(t, TCPPort, producer.peerInfo.TCPPort)\n\ttest.Equal(t, HTTPPort, producer.peerInfo.HTTPPort)\n\n\ttr := TopicsDoc{}\n\tendpoint = fmt.Sprintf(\"http://%s/topics\", httpAddr)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &tr)\n\ttest.Nil(t, err)\n\n\tt.Logf(\"got %v\", tr)\n\ttest.Equal(t, 1, len(tr.Topics))\n\n\tlr := LookupDoc{}\n\tendpoint = fmt.Sprintf(\"http://%s/lookup?topic=%s\", httpAddr, topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &lr)\n\ttest.Nil(t, err)\n\n\tt.Logf(\"got %v\", lr)\n\ttest.Equal(t, 1, len(lr.Channels))\n\ttest.Equal(t, 1, len(lr.Producers))\n\tfor _, p := range lr.Producers {\n\t\ttest.Equal(t, TCPPort, p.TCPPort)\n\t\ttest.Equal(t, HTTPPort, p.HTTPPort)\n\t\ttest.Equal(t, HostAddr, p.BroadcastAddress)\n\t\ttest.Equal(t, NSQDVersion, p.Version)\n\t}\n\n\tconn.Close()\n\ttime.Sleep(10 * time.Millisecond)\n\n\t// now there should be no producers, but still topic/channel entries\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &lr)\n\ttest.Nil(t, err)\n\n\ttest.Equal(t, 1, len(lr.Channels))\n\ttest.Equal(t, 0, len(lr.Producers))\n}\n\nfunc TestChannelUnregister(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\ttcpAddr, httpAddr, nsqlookupd := mustStartLookupd(opts)\n\tdefer nsqlookupd.Exit()\n\n\ttopics := nsqlookupd.DB.FindRegistrations(\"topic\", \"*\", \"*\")\n\ttest.Equal(t, 0, len(topics))\n\n\ttopicName := \"channel_unregister\"\n\n\tconn := mustConnectLookupd(t, tcpAddr)\n\tdefer conn.Close()\n\n\tidentify(t, conn)\n\n\tnsq.Register(topicName, \"ch1\").WriteTo(conn)\n\tv, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\ttest.Equal(t, []byte(\"OK\"), v)\n\n\ttopics = nsqlookupd.DB.FindRegistrations(\"topic\", topicName, \"\")\n\ttest.Equal(t, 1, len(topics))\n\n\tchannels := nsqlookupd.DB.FindRegistrations(\"channel\", topicName, \"*\")\n\ttest.Equal(t, 1, len(channels))\n\n\tnsq.UnRegister(topicName, \"ch1\").WriteTo(conn)\n\tv, err = nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\ttest.Equal(t, []byte(\"OK\"), v)\n\n\ttopics = nsqlookupd.DB.FindRegistrations(\"topic\", topicName, \"\")\n\ttest.Equal(t, 1, len(topics))\n\n\t// we should still have mention of the topic even though there is no producer\n\t// (ie. we haven't *deleted* the channel, just unregistered as a producer)\n\tchannels = nsqlookupd.DB.FindRegistrations(\"channel\", topicName, \"*\")\n\ttest.Equal(t, 1, len(channels))\n\n\tpr := ProducersDoc{}\n\tendpoint := fmt.Sprintf(\"http://%s/lookup?topic=%s\", httpAddr, topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &pr)\n\ttest.Nil(t, err)\n\tt.Logf(\"got %v\", pr)\n\ttest.Equal(t, 1, len(pr.Producers))\n}\n\nfunc TestTombstoneRecover(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.TombstoneLifetime = 50 * time.Millisecond\n\ttcpAddr, httpAddr, nsqlookupd := mustStartLookupd(opts)\n\tdefer nsqlookupd.Exit()\n\n\ttopicName := \"tombstone_recover\"\n\ttopicName2 := topicName + \"2\"\n\n\tconn := mustConnectLookupd(t, tcpAddr)\n\tdefer conn.Close()\n\n\tidentify(t, conn)\n\n\tnsq.Register(topicName, \"channel1\").WriteTo(conn)\n\t_, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\n\tnsq.Register(topicName2, \"channel2\").WriteTo(conn)\n\t_, err = nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\n\tendpoint := fmt.Sprintf(\"http://%s/topic/tombstone?topic=%s&node=%s:%d\",\n\t\thttpAddr, topicName, HostAddr, HTTPPort)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).POSTV1(endpoint, nil, nil)\n\ttest.Nil(t, err)\n\n\tpr := ProducersDoc{}\n\n\tendpoint = fmt.Sprintf(\"http://%s/lookup?topic=%s\", httpAddr, topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &pr)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 0, len(pr.Producers))\n\n\tendpoint = fmt.Sprintf(\"http://%s/lookup?topic=%s\", httpAddr, topicName2)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &pr)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 1, len(pr.Producers))\n\n\ttime.Sleep(75 * time.Millisecond)\n\n\tendpoint = fmt.Sprintf(\"http://%s/lookup?topic=%s\", httpAddr, topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &pr)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 1, len(pr.Producers))\n}\n\nfunc TestTombstoneUnregister(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.TombstoneLifetime = 50 * time.Millisecond\n\ttcpAddr, httpAddr, nsqlookupd := mustStartLookupd(opts)\n\tdefer nsqlookupd.Exit()\n\n\ttopicName := \"tombstone_unregister\"\n\n\tconn := mustConnectLookupd(t, tcpAddr)\n\tdefer conn.Close()\n\n\tidentify(t, conn)\n\n\tnsq.Register(topicName, \"channel1\").WriteTo(conn)\n\t_, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\n\tendpoint := fmt.Sprintf(\"http://%s/topic/tombstone?topic=%s&node=%s:%d\",\n\t\thttpAddr, topicName, HostAddr, HTTPPort)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).POSTV1(endpoint, nil, nil)\n\ttest.Nil(t, err)\n\n\tpr := ProducersDoc{}\n\n\tendpoint = fmt.Sprintf(\"http://%s/lookup?topic=%s\", httpAddr, topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &pr)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 0, len(pr.Producers))\n\n\tnsq.UnRegister(topicName, \"\").WriteTo(conn)\n\t_, err = nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\n\ttime.Sleep(55 * time.Millisecond)\n\n\tendpoint = fmt.Sprintf(\"http://%s/lookup?topic=%s\", httpAddr, topicName)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).GETV1(endpoint, &pr)\n\ttest.Nil(t, err)\n\ttest.Equal(t, 0, len(pr.Producers))\n}\n\nfunc TestInactiveNodes(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\topts.InactiveProducerTimeout = 200 * time.Millisecond\n\ttcpAddr, httpAddr, nsqlookupd := mustStartLookupd(opts)\n\tdefer nsqlookupd.Exit()\n\n\tlookupdHTTPAddrs := []string{httpAddr.String()}\n\n\ttopicName := \"inactive_nodes\"\n\n\tconn := mustConnectLookupd(t, tcpAddr)\n\tdefer conn.Close()\n\n\tidentify(t, conn)\n\n\tnsq.Register(topicName, \"channel1\").WriteTo(conn)\n\t_, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\n\tci := clusterinfo.New(nil, http_api.NewClient(nil, ConnectTimeout, RequestTimeout))\n\n\tproducers, _ := ci.GetLookupdProducers(lookupdHTTPAddrs)\n\ttest.Equal(t, 1, len(producers))\n\ttest.Equal(t, 1, len(producers[0].Topics))\n\ttest.Equal(t, topicName, producers[0].Topics[0].Topic)\n\ttest.Equal(t, false, producers[0].Topics[0].Tombstoned)\n\n\ttime.Sleep(250 * time.Millisecond)\n\n\tproducers, _ = ci.GetLookupdProducers(lookupdHTTPAddrs)\n\ttest.Equal(t, 0, len(producers))\n}\n\nfunc TestTombstonedNodes(t *testing.T) {\n\topts := NewOptions()\n\topts.Logger = test.NewTestLogger(t)\n\ttcpAddr, httpAddr, nsqlookupd := mustStartLookupd(opts)\n\tdefer nsqlookupd.Exit()\n\n\tlookupdHTTPAddrs := []string{httpAddr.String()}\n\n\ttopicName := \"inactive_nodes\"\n\n\tconn := mustConnectLookupd(t, tcpAddr)\n\tdefer conn.Close()\n\n\tidentify(t, conn)\n\n\tnsq.Register(topicName, \"channel1\").WriteTo(conn)\n\t_, err := nsq.ReadResponse(conn)\n\ttest.Nil(t, err)\n\n\tci := clusterinfo.New(nil, http_api.NewClient(nil, ConnectTimeout, RequestTimeout))\n\n\tproducers, _ := ci.GetLookupdProducers(lookupdHTTPAddrs)\n\ttest.Equal(t, 1, len(producers))\n\ttest.Equal(t, 1, len(producers[0].Topics))\n\ttest.Equal(t, topicName, producers[0].Topics[0].Topic)\n\ttest.Equal(t, false, producers[0].Topics[0].Tombstoned)\n\n\tendpoint := fmt.Sprintf(\"http://%s/topic/tombstone?topic=%s&node=%s:%d\",\n\t\thttpAddr, topicName, HostAddr, HTTPPort)\n\terr = http_api.NewClient(nil, ConnectTimeout, RequestTimeout).POSTV1(endpoint, nil, nil)\n\ttest.Nil(t, err)\n\n\tproducers, _ = ci.GetLookupdProducers(lookupdHTTPAddrs)\n\ttest.Equal(t, 1, len(producers))\n\ttest.Equal(t, 1, len(producers[0].Topics))\n\ttest.Equal(t, topicName, producers[0].Topics[0].Topic)\n\ttest.Equal(t, true, producers[0].Topics[0].Tombstoned)\n}\n" }, { "path": "nsqlookupd/options.go", "content": "package nsqlookupd\n\nimport (\n\t\"log\"\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/lg\"\n)\n\ntype Options struct {\n\tLogLevel lg.LogLevel `flag:\"log-level\"`\n\tLogPrefix string `flag:\"log-prefix\"`\n\tLogger Logger\n\n\tTCPAddress string `flag:\"tcp-address\"`\n\tHTTPAddress string `flag:\"http-address\"`\n\tBroadcastAddress string `flag:\"broadcast-address\"`\n\n\tInactiveProducerTimeout time.Duration `flag:\"inactive-producer-timeout\"`\n\tTombstoneLifetime time.Duration `flag:\"tombstone-lifetime\"`\n}\n\nfunc NewOptions() *Options {\n\thostname, err := os.Hostname()\n\tif err != nil {\n\t\tlog.Fatal(err)\n\t}\n\n\treturn &Options{\n\t\tLogPrefix: \"[nsqlookupd] \",\n\t\tLogLevel: lg.INFO,\n\t\tTCPAddress: \"0.0.0.0:4160\",\n\t\tHTTPAddress: \"0.0.0.0:4161\",\n\t\tBroadcastAddress: hostname,\n\n\t\tInactiveProducerTimeout: 300 * time.Second,\n\t\tTombstoneLifetime: 45 * time.Second,\n\t}\n}\n" }, { "path": "nsqlookupd/registration_db.go", "content": "package nsqlookupd\n\nimport (\n\t\"fmt\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n)\n\ntype RegistrationDB struct {\n\tsync.RWMutex\n\tregistrationMap map[Registration]ProducerMap\n}\n\ntype Registration struct {\n\tCategory string\n\tKey string\n\tSubKey string\n}\ntype Registrations []Registration\n\ntype PeerInfo struct {\n\tlastUpdate int64\n\tid string\n\tRemoteAddress string `json:\"remote_address\"`\n\tHostname string `json:\"hostname\"`\n\tBroadcastAddress string `json:\"broadcast_address\"`\n\tTCPPort int `json:\"tcp_port\"`\n\tHTTPPort int `json:\"http_port\"`\n\tVersion string `json:\"version\"`\n\tTopologyZone string `json:\"topology_zone\"`\n\tTopologyRegion string `json:\"topology_region\"`\n}\n\ntype Producer struct {\n\tpeerInfo *PeerInfo\n\ttombstoned bool\n\ttombstonedAt time.Time\n}\n\ntype Producers []*Producer\ntype ProducerMap map[string]*Producer\n\nfunc (p *Producer) String() string {\n\treturn fmt.Sprintf(\"%s [%d, %d]\", p.peerInfo.BroadcastAddress, p.peerInfo.TCPPort, p.peerInfo.HTTPPort)\n}\n\nfunc (p *Producer) Tombstone() {\n\tp.tombstoned = true\n\tp.tombstonedAt = time.Now()\n}\n\nfunc (p *Producer) IsTombstoned(lifetime time.Duration) bool {\n\treturn p.tombstoned && time.Since(p.tombstonedAt) < lifetime\n}\n\nfunc NewRegistrationDB() *RegistrationDB {\n\treturn &RegistrationDB{\n\t\tregistrationMap: make(map[Registration]ProducerMap),\n\t}\n}\n\n// add a registration key\nfunc (r *RegistrationDB) AddRegistration(k Registration) {\n\tr.Lock()\n\tdefer r.Unlock()\n\t_, ok := r.registrationMap[k]\n\tif !ok {\n\t\tr.registrationMap[k] = make(map[string]*Producer)\n\t}\n}\n\n// add a producer to a registration\nfunc (r *RegistrationDB) AddProducer(k Registration, p *Producer) bool {\n\tr.Lock()\n\tdefer r.Unlock()\n\t_, ok := r.registrationMap[k]\n\tif !ok {\n\t\tr.registrationMap[k] = make(map[string]*Producer)\n\t}\n\tproducers := r.registrationMap[k]\n\t_, found := producers[p.peerInfo.id]\n\tif !found {\n\t\tproducers[p.peerInfo.id] = p\n\t}\n\treturn !found\n}\n\n// remove a producer from a registration\nfunc (r *RegistrationDB) RemoveProducer(k Registration, id string) (bool, int) {\n\tr.Lock()\n\tdefer r.Unlock()\n\tproducers, ok := r.registrationMap[k]\n\tif !ok {\n\t\treturn false, 0\n\t}\n\tremoved := false\n\tif _, exists := producers[id]; exists {\n\t\tremoved = true\n\t}\n\n\t// Note: this leaves keys in the DB even if they have empty lists\n\tdelete(producers, id)\n\treturn removed, len(producers)\n}\n\n// remove a Registration and all it's producers\nfunc (r *RegistrationDB) RemoveRegistration(k Registration) {\n\tr.Lock()\n\tdefer r.Unlock()\n\tdelete(r.registrationMap, k)\n}\n\nfunc (r *RegistrationDB) needFilter(key string, subkey string) bool {\n\treturn key == \"*\" || subkey == \"*\"\n}\n\nfunc (r *RegistrationDB) FindRegistrations(category string, key string, subkey string) Registrations {\n\tr.RLock()\n\tdefer r.RUnlock()\n\tif !r.needFilter(key, subkey) {\n\t\tk := Registration{category, key, subkey}\n\t\tif _, ok := r.registrationMap[k]; ok {\n\t\t\treturn Registrations{k}\n\t\t}\n\t\treturn Registrations{}\n\t}\n\tresults := Registrations{}\n\tfor k := range r.registrationMap {\n\t\tif !k.IsMatch(category, key, subkey) {\n\t\t\tcontinue\n\t\t}\n\t\tresults = append(results, k)\n\t}\n\treturn results\n}\n\nfunc (r *RegistrationDB) FindProducers(category string, key string, subkey string) Producers {\n\tr.RLock()\n\tdefer r.RUnlock()\n\tif !r.needFilter(key, subkey) {\n\t\tk := Registration{category, key, subkey}\n\t\treturn ProducerMap2Slice(r.registrationMap[k])\n\t}\n\n\tresults := make(map[string]struct{})\n\tvar retProducers Producers\n\tfor k, producers := range r.registrationMap {\n\t\tif !k.IsMatch(category, key, subkey) {\n\t\t\tcontinue\n\t\t}\n\t\tfor _, producer := range producers {\n\t\t\t_, found := results[producer.peerInfo.id]\n\t\t\tif !found {\n\t\t\t\tresults[producer.peerInfo.id] = struct{}{}\n\t\t\t\tretProducers = append(retProducers, producer)\n\t\t\t}\n\t\t}\n\t}\n\treturn retProducers\n}\n\nfunc (r *RegistrationDB) LookupRegistrations(id string) Registrations {\n\tr.RLock()\n\tdefer r.RUnlock()\n\tresults := Registrations{}\n\tfor k, producers := range r.registrationMap {\n\t\tif _, exists := producers[id]; exists {\n\t\t\tresults = append(results, k)\n\t\t}\n\t}\n\treturn results\n}\n\nfunc (k Registration) IsMatch(category string, key string, subkey string) bool {\n\tif category != k.Category {\n\t\treturn false\n\t}\n\tif key != \"*\" && k.Key != key {\n\t\treturn false\n\t}\n\tif subkey != \"*\" && k.SubKey != subkey {\n\t\treturn false\n\t}\n\treturn true\n}\n\nfunc (rr Registrations) Filter(category string, key string, subkey string) Registrations {\n\toutput := Registrations{}\n\tfor _, k := range rr {\n\t\tif k.IsMatch(category, key, subkey) {\n\t\t\toutput = append(output, k)\n\t\t}\n\t}\n\treturn output\n}\n\nfunc (rr Registrations) Keys() []string {\n\tkeys := make([]string, len(rr))\n\tfor i, k := range rr {\n\t\tkeys[i] = k.Key\n\t}\n\treturn keys\n}\n\nfunc (rr Registrations) SubKeys() []string {\n\tsubkeys := make([]string, len(rr))\n\tfor i, k := range rr {\n\t\tsubkeys[i] = k.SubKey\n\t}\n\treturn subkeys\n}\n\nfunc (pp Producers) FilterByActive(inactivityTimeout time.Duration, tombstoneLifetime time.Duration) Producers {\n\tnow := time.Now()\n\tresults := Producers{}\n\tfor _, p := range pp {\n\t\tcur := time.Unix(0, atomic.LoadInt64(&p.peerInfo.lastUpdate))\n\t\tif now.Sub(cur) > inactivityTimeout || p.IsTombstoned(tombstoneLifetime) {\n\t\t\tcontinue\n\t\t}\n\t\tresults = append(results, p)\n\t}\n\treturn results\n}\n\nfunc (pp Producers) PeerInfo() []*PeerInfo {\n\tresults := []*PeerInfo{}\n\tfor _, p := range pp {\n\t\tresults = append(results, p.peerInfo)\n\t}\n\treturn results\n}\n\nfunc ProducerMap2Slice(pm ProducerMap) Producers {\n\tvar producers Producers\n\tfor _, producer := range pm {\n\t\tproducers = append(producers, producer)\n\t}\n\n\treturn producers\n}\n" }, { "path": "nsqlookupd/registration_db_test.go", "content": "package nsqlookupd\n\nimport (\n\t\"math/rand\"\n\t\"strconv\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/nsqio/nsq/internal/test\"\n)\n\nfunc TestRegistrationDB(t *testing.T) {\n\tsec30 := 30 * time.Second\n\tbeginningOfTime := time.Unix(1348797047, 0)\n\tpi1 := &PeerInfo{beginningOfTime.UnixNano(), \"1\", \"remote_addr:1\", \"host\", \"b_addr\", 1, 2, \"v1\", \"\", \"\"}\n\tpi2 := &PeerInfo{beginningOfTime.UnixNano(), \"2\", \"remote_addr:2\", \"host\", \"b_addr\", 2, 3, \"v1\", \"\", \"\"}\n\tpi3 := &PeerInfo{beginningOfTime.UnixNano(), \"3\", \"remote_addr:3\", \"host\", \"b_addr\", 3, 4, \"v1\", \"\", \"\"}\n\tp1 := &Producer{pi1, false, beginningOfTime}\n\tp2 := &Producer{pi2, false, beginningOfTime}\n\tp3 := &Producer{pi3, false, beginningOfTime}\n\tp4 := &Producer{pi1, false, beginningOfTime}\n\n\tdb := NewRegistrationDB()\n\n\t// add producers\n\tdb.AddProducer(Registration{\"c\", \"a\", \"\"}, p1)\n\tdb.AddProducer(Registration{\"c\", \"a\", \"\"}, p2)\n\tdb.AddProducer(Registration{\"c\", \"a\", \"b\"}, p2)\n\tdb.AddProducer(Registration{\"d\", \"a\", \"\"}, p3)\n\tdb.AddProducer(Registration{\"t\", \"a\", \"\"}, p4)\n\n\t// find producers\n\tr := db.FindRegistrations(\"c\", \"*\", \"\").Keys()\n\ttest.Equal(t, 1, len(r))\n\ttest.Equal(t, \"a\", r[0])\n\n\tp := db.FindProducers(\"t\", \"*\", \"\")\n\tt.Logf(\"%s\", p)\n\ttest.Equal(t, 1, len(p))\n\tp = db.FindProducers(\"c\", \"*\", \"\")\n\tt.Logf(\"%s\", p)\n\ttest.Equal(t, 2, len(p))\n\tp = db.FindProducers(\"c\", \"a\", \"\")\n\tt.Logf(\"%s\", p)\n\ttest.Equal(t, 2, len(p))\n\tp = db.FindProducers(\"c\", \"*\", \"b\")\n\tt.Logf(\"%s\", p)\n\ttest.Equal(t, 1, len(p))\n\ttest.Equal(t, p2.peerInfo.id, p[0].peerInfo.id)\n\n\t// filter by active\n\ttest.Equal(t, 0, len(p.FilterByActive(sec30, sec30)))\n\tp2.peerInfo.lastUpdate = time.Now().UnixNano()\n\ttest.Equal(t, 1, len(p.FilterByActive(sec30, sec30)))\n\tp = db.FindProducers(\"c\", \"*\", \"\")\n\tt.Logf(\"%s\", p)\n\ttest.Equal(t, 1, len(p.FilterByActive(sec30, sec30)))\n\n\t// tombstoning\n\tfewSecAgo := time.Now().Add(-5 * time.Second).UnixNano()\n\tp1.peerInfo.lastUpdate = fewSecAgo\n\tp2.peerInfo.lastUpdate = fewSecAgo\n\ttest.Equal(t, 2, len(p.FilterByActive(sec30, sec30)))\n\tp1.Tombstone()\n\ttest.Equal(t, 1, len(p.FilterByActive(sec30, sec30)))\n\ttime.Sleep(10 * time.Millisecond)\n\ttest.Equal(t, 2, len(p.FilterByActive(sec30, 5*time.Millisecond)))\n\t// make sure we can still retrieve p1 from another registration see #148\n\ttest.Equal(t, 1, len(db.FindProducers(\"t\", \"*\", \"\").FilterByActive(sec30, sec30)))\n\n\t// keys and subkeys\n\tk := db.FindRegistrations(\"c\", \"b\", \"\").Keys()\n\ttest.Equal(t, 0, len(k))\n\tk = db.FindRegistrations(\"c\", \"a\", \"\").Keys()\n\ttest.Equal(t, 1, len(k))\n\ttest.Equal(t, \"a\", k[0])\n\tk = db.FindRegistrations(\"c\", \"*\", \"b\").SubKeys()\n\ttest.Equal(t, 1, len(k))\n\ttest.Equal(t, \"b\", k[0])\n\n\t// removing producers\n\tdb.RemoveProducer(Registration{\"c\", \"a\", \"\"}, p1.peerInfo.id)\n\tp = db.FindProducers(\"c\", \"*\", \"*\")\n\tt.Logf(\"%s\", p)\n\ttest.Equal(t, 1, len(p))\n\n\tdb.RemoveProducer(Registration{\"c\", \"a\", \"\"}, p2.peerInfo.id)\n\tdb.RemoveProducer(Registration{\"c\", \"a\", \"b\"}, p2.peerInfo.id)\n\tp = db.FindProducers(\"c\", \"*\", \"*\")\n\tt.Logf(\"%s\", p)\n\ttest.Equal(t, 0, len(p))\n\n\t// do some key removals\n\tk = db.FindRegistrations(\"c\", \"*\", \"*\").Keys()\n\ttest.Equal(t, 2, len(k))\n\tdb.RemoveRegistration(Registration{\"c\", \"a\", \"\"})\n\tdb.RemoveRegistration(Registration{\"c\", \"a\", \"b\"})\n\tk = db.FindRegistrations(\"c\", \"*\", \"*\").Keys()\n\ttest.Equal(t, 0, len(k))\n}\n\nfunc fillRegDB(registrations int, producers int) *RegistrationDB {\n\tregDB := NewRegistrationDB()\n\tfor i := 0; i < registrations; i++ {\n\t\tregT := Registration{\"topic\", \"t\" + strconv.Itoa(i), \"\"}\n\t\tregCa := Registration{\"channel\", \"t\" + strconv.Itoa(i), \"ca\" + strconv.Itoa(i)}\n\t\tregCb := Registration{\"channel\", \"t\" + strconv.Itoa(i), \"cb\" + strconv.Itoa(i)}\n\t\tfor j := 0; j < producers; j++ {\n\t\t\tp := Producer{\n\t\t\t\tpeerInfo: &PeerInfo{\n\t\t\t\t\tid: \"p\" + strconv.Itoa(j),\n\t\t\t\t},\n\t\t\t}\n\t\t\tregDB.AddProducer(regT, &p)\n\t\t\tregDB.AddProducer(regCa, &p)\n\t\t\tregDB.AddProducer(regCb, &p)\n\t\t}\n\t}\n\treturn regDB\n}\n\nfunc benchmarkLookupRegistrations(b *testing.B, registrations int, producers int) {\n\tregDB := fillRegDB(registrations, producers)\n\tb.ResetTimer()\n\tfor i := 0; i < b.N; i++ {\n\t\tj := strconv.Itoa(rand.Intn(producers))\n\t\t_ = regDB.LookupRegistrations(\"p\" + j)\n\t}\n}\n\nfunc benchmarkRegister(b *testing.B, registrations int, producers int) {\n\tfor i := 0; i < b.N; i++ {\n\t\t_ = fillRegDB(registrations, producers)\n\t}\n}\n\nfunc benchmarkDoLookup(b *testing.B, registrations int, producers int) {\n\tregDB := fillRegDB(registrations, producers)\n\tb.ResetTimer()\n\tfor i := 0; i < b.N; i++ {\n\t\ttopic := \"t\" + strconv.Itoa(rand.Intn(registrations))\n\t\t_ = regDB.FindRegistrations(\"topic\", topic, \"\")\n\t\t_ = regDB.FindRegistrations(\"channel\", topic, \"*\").SubKeys()\n\t\t_ = regDB.FindProducers(\"topic\", topic, \"\")\n\t}\n}\n\nfunc BenchmarkLookupRegistrations8x8(b *testing.B) {\n\tbenchmarkLookupRegistrations(b, 8, 8)\n}\n\nfunc BenchmarkLookupRegistrations8x64(b *testing.B) {\n\tbenchmarkLookupRegistrations(b, 8, 64)\n}\n\nfunc BenchmarkLookupRegistrations64x64(b *testing.B) {\n\tbenchmarkLookupRegistrations(b, 64, 64)\n}\n\nfunc BenchmarkLookupRegistrations64x512(b *testing.B) {\n\tbenchmarkLookupRegistrations(b, 64, 512)\n}\n\nfunc BenchmarkLookupRegistrations512x512(b *testing.B) {\n\tbenchmarkLookupRegistrations(b, 512, 512)\n}\n\nfunc BenchmarkLookupRegistrations512x2048(b *testing.B) {\n\tbenchmarkLookupRegistrations(b, 512, 2048)\n}\n\nfunc BenchmarkRegister8x8(b *testing.B) {\n\tbenchmarkRegister(b, 8, 8)\n}\n\nfunc BenchmarkRegister8x64(b *testing.B) {\n\tbenchmarkRegister(b, 8, 64)\n}\n\nfunc BenchmarkRegister64x64(b *testing.B) {\n\tbenchmarkRegister(b, 64, 64)\n}\n\nfunc BenchmarkRegister64x512(b *testing.B) {\n\tbenchmarkRegister(b, 64, 512)\n}\n\nfunc BenchmarkRegister512x512(b *testing.B) {\n\tbenchmarkRegister(b, 512, 512)\n}\n\nfunc BenchmarkRegister512x2048(b *testing.B) {\n\tbenchmarkRegister(b, 512, 2048)\n}\n\nfunc BenchmarkDoLookup8x8(b *testing.B) {\n\tbenchmarkDoLookup(b, 8, 8)\n}\n\nfunc BenchmarkDoLookup8x64(b *testing.B) {\n\tbenchmarkDoLookup(b, 8, 64)\n}\n\nfunc BenchmarkDoLookup64x64(b *testing.B) {\n\tbenchmarkDoLookup(b, 64, 64)\n}\n\nfunc BenchmarkDoLookup64x512(b *testing.B) {\n\tbenchmarkDoLookup(b, 64, 512)\n}\n\nfunc BenchmarkDoLookup512x512(b *testing.B) {\n\tbenchmarkDoLookup(b, 512, 512)\n}\n\nfunc BenchmarkDoLookup512x2048(b *testing.B) {\n\tbenchmarkDoLookup(b, 512, 2048)\n}\n" }, { "path": "nsqlookupd/tcp.go", "content": "package nsqlookupd\n\nimport (\n\t\"io\"\n\t\"net\"\n\t\"sync\"\n\n\t\"github.com/nsqio/nsq/internal/protocol\"\n)\n\ntype tcpServer struct {\n\tnsqlookupd *NSQLookupd\n\tconns sync.Map\n}\n\nfunc (p *tcpServer) Handle(conn net.Conn) {\n\tp.nsqlookupd.logf(LOG_INFO, \"TCP: new client(%s)\", conn.RemoteAddr())\n\n\t// The client should initialize itself by sending a 4 byte sequence indicating\n\t// the version of the protocol that it intends to communicate, this will allow us\n\t// to gracefully upgrade the protocol away from text/line oriented to whatever...\n\tbuf := make([]byte, 4)\n\t_, err := io.ReadFull(conn, buf)\n\tif err != nil {\n\t\tp.nsqlookupd.logf(LOG_ERROR, \"failed to read protocol version - %s\", err)\n\t\tconn.Close()\n\t\treturn\n\t}\n\tprotocolMagic := string(buf)\n\n\tp.nsqlookupd.logf(LOG_INFO, \"CLIENT(%s): desired protocol magic '%s'\",\n\t\tconn.RemoteAddr(), protocolMagic)\n\n\tvar prot protocol.Protocol\n\tswitch protocolMagic {\n\tcase \" V1\":\n\t\tprot = &LookupProtocolV1{nsqlookupd: p.nsqlookupd}\n\tdefault:\n\t\tprotocol.SendResponse(conn, []byte(\"E_BAD_PROTOCOL\"))\n\t\tconn.Close()\n\t\tp.nsqlookupd.logf(LOG_ERROR, \"client(%s) bad protocol magic '%s'\",\n\t\t\tconn.RemoteAddr(), protocolMagic)\n\t\treturn\n\t}\n\n\tclient := prot.NewClient(conn)\n\tp.conns.Store(conn.RemoteAddr(), client)\n\n\terr = prot.IOLoop(client)\n\tif err != nil {\n\t\tp.nsqlookupd.logf(LOG_ERROR, \"client(%s) - %s\", conn.RemoteAddr(), err)\n\t}\n\n\tp.conns.Delete(conn.RemoteAddr())\n\tclient.Close()\n}\n\nfunc (p *tcpServer) Close() {\n\tp.conns.Range(func(k, v interface{}) bool {\n\t\tv.(protocol.Client).Close()\n\t\treturn true\n\t})\n}\n" }, { "path": "test.sh", "content": "#!/bin/sh\nset -e\n\nGOMAXPROCS=1 go test -timeout 90s ./...\n\nif [ \"$GOARCH\" = \"amd64\" ] || [ \"$GOARCH\" = \"arm64\" ]; then\n # go test: -race is only supported on linux/amd64, linux/ppc64le,\n # linux/arm64, freebsd/amd64, netbsd/amd64, darwin/amd64 and windows/amd64\n GOMAXPROCS=4 go test -timeout 90s -race ./...\nfi\n\n# no tests, but a build is something\nfor dir in apps/*/ bench/*/; do\n dir=${dir%/}\n if grep -q '^package main$' $dir/*.go 2>/dev/null; then\n echo \"building $dir\"\n go build -o $dir/$(basename $dir) ./$dir\n else\n echo \"(skipped $dir)\"\n fi\ndone\n\n# disable \"composite literal uses unkeyed fields\"\ngo vet -composites=false ./...\n\nFMTDIFF=\"$(find apps internal nsqd nsqlookupd -name '*.go' -exec gofmt -d '{}' ';')\"\nif [ -n \"$FMTDIFF\" ]; then\n printf '%s\\n' \"$FMTDIFF\"\n exit 1\nfi\n" } ]