Showing preview only (1,018K chars total). Download the full file or copy to clipboard to get everything.
Repository: twitter/twemproxy
Branch: master
Commit: 60aaf855ba2e
Files: 123
Total size: 975.9 KB
Directory structure:
gitextract_s05hoke1/
├── .clang-format.example
├── .dockerignore
├── .editorconfig
├── .github/
│ ├── ISSUE_TEMPLATE.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ └── workflows/
│ └── main.yml
├── .gitignore
├── ChangeLog
├── Dockerfile
├── LICENSE
├── Makefile.am
├── NOTICE
├── README.md
├── ci/
│ ├── Dockerfile
│ └── build-nutcracker.sh
├── conf/
│ ├── nutcracker.leaf.yml
│ ├── nutcracker.root.yml
│ └── nutcracker.yml
├── configure.ac
├── contrib/
│ ├── Makefile.am
│ └── yaml-0.2.5/
│ └── .gitignore
├── m4/
│ └── .gitignore
├── man/
│ └── nutcracker.8
├── notes/
│ ├── c-styleguide.txt
│ ├── debug.txt
│ ├── memcache.md
│ ├── recommendation.md
│ ├── redis.md
│ └── socket.txt
├── scripts/
│ ├── benchmark-mget.py
│ ├── extract_redis_commands_argcounts.php
│ ├── makesrpm.sh
│ ├── multi_get.sh
│ ├── nutcracker.init
│ ├── nutcracker.init.debian
│ ├── nutcracker.spec
│ ├── pipelined_read.sh
│ ├── pipelined_write.sh
│ ├── populate_memcached.sh
│ ├── redis-check.py
│ └── redis-check.sh
├── src/
│ ├── Makefile.am
│ ├── event/
│ │ ├── Makefile.am
│ │ ├── nc_epoll.c
│ │ ├── nc_event.h
│ │ ├── nc_evport.c
│ │ └── nc_kqueue.c
│ ├── hashkit/
│ │ ├── Makefile.am
│ │ ├── nc_crc16.c
│ │ ├── nc_crc32.c
│ │ ├── nc_fnv.c
│ │ ├── nc_hashkit.h
│ │ ├── nc_hsieh.c
│ │ ├── nc_jenkins.c
│ │ ├── nc_ketama.c
│ │ ├── nc_md5.c
│ │ ├── nc_modula.c
│ │ ├── nc_murmur.c
│ │ ├── nc_one_at_a_time.c
│ │ └── nc_random.c
│ ├── nc.c
│ ├── nc_array.c
│ ├── nc_array.h
│ ├── nc_client.c
│ ├── nc_client.h
│ ├── nc_conf.c
│ ├── nc_conf.h
│ ├── nc_connection.c
│ ├── nc_connection.h
│ ├── nc_core.c
│ ├── nc_core.h
│ ├── nc_log.c
│ ├── nc_log.h
│ ├── nc_mbuf.c
│ ├── nc_mbuf.h
│ ├── nc_message.c
│ ├── nc_message.h
│ ├── nc_proxy.c
│ ├── nc_proxy.h
│ ├── nc_queue.h
│ ├── nc_rbtree.c
│ ├── nc_rbtree.h
│ ├── nc_request.c
│ ├── nc_response.c
│ ├── nc_server.c
│ ├── nc_server.h
│ ├── nc_signal.c
│ ├── nc_signal.h
│ ├── nc_stats.c
│ ├── nc_stats.h
│ ├── nc_string.c
│ ├── nc_string.h
│ ├── nc_util.c
│ ├── nc_util.h
│ ├── proto/
│ │ ├── Makefile.am
│ │ ├── nc_memcache.c
│ │ ├── nc_proto.h
│ │ └── nc_redis.c
│ └── test_all.c
├── test_in_docker.sh
└── tests/
├── .gitignore
├── README.rst
├── conf/
│ ├── conf.py
│ ├── control.sh
│ ├── redis.conf
│ └── sentinel.conf
├── lib/
│ ├── server_modules.py
│ └── utils.py
├── log/
│ └── .gitignore
├── nosetests_verbose.sh
├── test_memcache/
│ ├── __init__.py
│ └── test_gets.py
├── test_redis/
│ ├── __init__.py
│ ├── common.py
│ ├── test_auth.py
│ ├── test_basic.py
│ ├── test_commands.py
│ ├── test_mget_large_binary.py
│ ├── test_mget_mset.py
│ ├── test_pipeline.py
│ └── test_protocol.py
└── test_system/
├── __init__.py
└── test_reload.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .clang-format.example
================================================
# This may be useful with https://clang.llvm.org/docs/ClangFormat.html#script-for-patch-reformatting
# but some of clang's reformattings may conflict with notes/c-style-guide.txt, which takes precedence.
#
# This is deliberately named .clang-format.example instead of .clang-format to avoid editors
# unexpectedly using this to reformatting entire files because of these shortcomings.
# It can be copied into .clang-format to use this for local development selectively.
#
# Note: this clang format file has many shortcomings.
# Attempting to apply this automatically to everything may make code less readable.
# However, this may be useful for spot checking new code,
# or if you're not certain of indentation style or general spacing/wrapping rules
#
# Known shortcomings:
# - Some places exceed the 80 column limit deliberately for readability, e.g. help strings or error messages or function prototypes. (BreakStringLiterals helps preserve some of those)
# - Some places deliberately put blocks on a single line when there are a lot of similar blocks.
# AllowShortBlocksOnASingleLine is not useful.
# - Some places deliberately put blocks on a single line when there are a lot of similar blocks.
# - No good way to eliminate space before and after PRIu64 and other macros for adjacent string literal concatenation
# - clang-format is not aware of macros, some of which have different styles from functions.
# - Function declarations are not typically aligned
# - Some variable declarations are aligned and others aren't on a case by case basis
# - The choice of function argument grouping should depends on which function arguments are semantically related,
# not just on fitting within 80 columns.
AllowShortBlocksOnASingleLine: true
BasedOnStyle: LLVM
AlwaysBreakAfterDefinitionReturnType: All
UseTab: Never
IndentWidth: 4
TabWidth: 4
ColumnLimit: 80
BreakBeforeBraces: Linux
SortIncludes: false
BreakStringLiterals: false
# BitFieldColonSpacing is too new to work in clang-format 11
# https://releases.llvm.org/11.0.0/tools/clang/docs/ClangFormatStyleOptions.html
# Latest: https://clang.llvm.org/docs/ClangFormatStyleOptions.html
#
# BitFieldColonSpacing: None
#
# XXX no way to treat the `*` indicating a value is a pointer as part of the aligned name for the declaration
# XXX function declarations are not typically aligned
AlignConsecutiveDeclarations: true
AlignConsecutiveMacros: true
================================================
FILE: .dockerignore
================================================
Dockerfile*
**/Dockerfile*
*.gz
src/nutredis
src/nutcracker
tests/_binaries/*
### Entries copied from .gitignore
# pyc
*.pyc
# Compiled Object files
*.lo
*.o
# Compiled Dynamic libraries
*.so
# Compiled Static libraries
*.la
*.a
# Compiled misc
*.dep
*.gcda
*.gcno
*.gcov
# Packages
*.tar.gz
*.tar.bz2
# Logs
*.log
# Temporary
*.swp
*.~
*.project
*.cproject
# Core and executable
core*
nutcracker
# extracted yaml
!/contrib/yaml-0.2.5.tar.gz
# Autotools
.deps
.libs
/aclocal.m4
/autom4te.cache
/stamp-h1
/autoscan.log
/libtool
================================================
FILE: .editorconfig
================================================
# https://editorconfig.org/
root = true
[*]
trim_trailing_whitespace = true
insert_final_newline = true
end_of_line = lf
charset = utf-8
# See twemproxy/notes/c-styleguide.txt
[*.c,*.h]
tab_width = 4
indent_size = 4
indent_style = space
# indent_brace_style depends on function vs conditional
max_line_length = 80
spaces_around_brackets = none
spaces_around_operators = true
================================================
FILE: .github/ISSUE_TEMPLATE.md
================================================
One line summary of the issue here.
### Expected behavior
As concisely as possible, describe the expected behavior.
### Actual behavior
As concisely as possible, describe the observed behavior.
### Steps to reproduce the behavior
Please list all relevant steps to reproduce the observed behavior.
================================================
FILE: .github/PULL_REQUEST_TEMPLATE.md
================================================
Problem
Explain the context and why you're making that change. What is the
problem you're trying to solve? In some cases there is not a problem
and this can be thought of being the motivation for your change.
Solution
Describe the modifications you've done.
Result
What will change as a result of your pull request? Note that sometimes
this section is unnecessary because it is self-explanatory based on
the solution.
================================================
FILE: .github/workflows/main.yml
================================================
# This is a basic workflow to help you get started with Actions
name: CI
# Controls when the action will run.
on:
# Triggers the workflow on push or pull request events but only for the master branch
push:
branches: [ master ]
pull_request:
branches: [ master ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# See https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#using-environment-variables-in-a-matrix
strategy:
# Run to completion even if one redis version has failures
fail-fast: false
matrix:
include:
- REDIS_VER: 3.0.7
- REDIS_VER: 3.2.13
- REDIS_VER: 4.0.14
- REDIS_VER: 5.0.12
- REDIS_VER: 6.2.4
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v2
# Runs a single command using the runners shell
- name: Build and test in docker
run: bash ./test_in_docker.sh ${{ matrix.REDIS_VER }}
================================================
FILE: .gitignore
================================================
# pyc
*.pyc
# Compiled Object files
*.lo
*.o
# Compiled Dynamic libraries
*.so
# Compiled Static libraries
*.la
*.a
# Compiled misc
*.dep
*.gcda
*.gcno
*.gcov
# Packages
*.tar.gz
*.tar.bz2
# Logs
*.log
# Temporary
*.swp
*.~
*.project
*.cproject
# Core and executable
core*
nutcracker
# extracted yaml
!/contrib/yaml-0.2.5.tar.gz
# Autotools
.deps
.libs
/aclocal.m4
/autom4te.cache
/stamp-h1
/autoscan.log
/libtool
/config/config.guess
/config/config.sub
/config/depcomp
/config/install-sh
/config/ltmain.sh
/config/missing
/config
/config.h
/config.h.in
/config.h.in~
/config.log
/config.status
/configure.scan
/configure
Makefile
Makefile.in
# The .clang-format.example file may be copied here for use with -style=file
.clang-format
test_all
*.trs
tags
================================================
FILE: ChangeLog
================================================
2021-13-07 Tyson Andre <tysonandre775@hotmail.com>
* twemproxy: version 0.5.0 release
Same as 0.5.0-RC1
2021-06-07 Tyson Andre <tysonandre775@hotmail.com>
* twemproxy: version 0.5.0-RC1 release
Add 'tcpkeepalive' pool boolean config flag setting
to enable tcp keepalive (charsyam, manju)
Support redis bitpos command (clark kang)
Fix parsing of redis error response for error type with no space,
add tests (tyson, tom dalton)
Update integration tests, add C unit test suite for 'make check' (tyson)
Increase the maximum host length+port+identifier to 273
in ketama_update (李广博)
Always initialize file permissions field when listening on a unix domain
socket (tyson)
Use number of servers instead of number of points on the continuum when
sharding requests to backend services to improve sharding performance
and fix potential invalid memory access when all hosts were ejected
from a pool. (tyson)
Optimize performance of deletion of single redis keys (vincentve)
Don't fragment memcache/redis get commands when they only have a single
key (improves performance and error handling of single key case) (tyson)
Don't let requests hang when there is a dns error when processing a
fragmented request (e.g. multiget with multiple keys) (tyson)
Allow extra parameters for redis spop (charsyam)
Update documentation and README (various)
Fix memory leak bug for redis mset (deep011)
Support arbitrarily deep nested redis multi-bulk
responses (nested arrays) (qingping209, tyson)
Upgrade from libyaml 0.1.4 to 0.2.5 (tyson)
Fix compiler warnings about wrong conversion specifiers in format
strings for logging (tyson)
Log the async backend used and any debug options in the
'--help'/'--version' output.
Add support for many more new redis commands and updates to existing
redis commands (tyson)
Optimization: Skip hashing and choosing server index when a pool has
exactly one server (tyson)
Support memcache 'version' requests by proxying the request to a single
backend memcache server to fetch the server version. (tyson)
Make error messages for creating the stats server during startup clearer. (tyson)
2015-22-06 Manju Rajashekhar <manj@cs.stanford.edu>
* twemproxy: version 0.4.1 release
backend server hostnames are resolved lazily
redis_auth is only valid for a redis pool
getaddrinfo returns non-zero +ve value on error
fix-hang-when-command-only (charsyam)
fix bug crash when get command without key and whitespace (charsyam)
mark server as failed on protocol level transiet failures like -OOM, -LOADING, etc
implemented support for parsing fine grained redis error response
remove redundant conditional judgement in rbtree deletion (leo ma)
fix bug mset has invalid pair (charsyam)
temp fix a core on kqueue (idning)
support "touch" command for memcached (panmiaocai)
fix redis parse rsp bug (charsyam)
SORT command can take multiple arguments. So it should be part of redis_argn() and not redis_arg0()
remove incorrect assert because client could send data after sending a quit request which must be discarded
allow file permissions to be set for UNIX domain listening socket (ori liveneh)
return error if formatted is greater than mbuf size by using nc_vsnprintf() in msg_prepend_format()
fix req_make_reply on msg_get, mark it as response (idning)
redis database select upon connect (arne claus)
redis_auth (charsyam)
allow null key(empty key) (idning)
fix core on invalid mset like "mset a a a" (idning)
2014-18-10 idning <idning@gmail.com>
* twemproxy: version 0.4.0 release
mget improve (idning)
many new commands supported: LEX, PFADD, PFMERGE, SORT, PING, QUIT, SCAN... (mattrobenolt, areina, idning)
handle max open file limit(allenlz)
add notice-log and use ms time in log(idning)
fix bug in string_compare (andyqzb)
fix deadlock in sighandler (idning)
2013-20-12 Manju Rajashekhar <manj@cs.stanford.edu>
* twemproxy: version 0.3.0 release
SRANDMEMBER support for the optional count argument (mkhq)
Handle case where server responds while the request is still being sent (jdi-tagged)
event ports (solaris/smartos) support
add timestamp when the server was ejected
support for set ex/px/nx/xx for redis 2.6.12 and up (ypocat)
kqueue (bsd) support (ferenyx)
fix parsing redis response to accept integer reply (charsyam)
2013-23-04 Manju Rajashekhar <manj@cs.stanford.edu>
* twemproxy: version 0.2.4 release
redis keys must be less than mbuf_data_size() in length (fifsky)
Adds support for DUMP/RESTORE commands in Redis (remotezygote)
Use of the weight value in the modula distribution (mezzatto)
Add support to unix socket connections to servers (mezzatto)
only check for duplicate server name and not 'host:port:weight' when 'name' is configured
crc16 hash support added (mezzatto)
2013-31-01 Manju Rajashekhar <manj@twitter.com>
* twemproxy: version 0.2.3 release
RPOPLPUSH, SDIFF, SDIFFSTORE, SINTER, SINTERSTORE, SMOVE, SUNION, SUNIONSTORE, ZINTERSTORE, and ZUNIONSTORE support (dcartoon)
EVAL and EVALSHA support (ferenyx)
exit 1 if configuration file is invalid (cofyc)
return non-zero exit status when nutcracker cannot start for some reason
use server names in stats (charsyam)
Fix failure to resolve long FQDN name resolve (conmame)
add support for hash tags
2012-18-10 Manju Rajashekhar <manj@twitter.com>
* twemproxy: version 0.2.2 release
fix the off-by-one error when calculating redis key length
2012-12-10 Manju Rajashekhar <manj@twitter.com>
* twemproxy: version 0.2.1 release
don't use buf in conf_add_server
allow an optional instance name for consistent hashing (charsyam)
add --stats-addr=S option
add stats-bind-any -a option (charsyam)
2012-12-03 Manju Rajashekhar <manj@twitter.com>
* twemproxy: version 0.2.0 release
add -D or --describe-stats command-line argument to print stats description
redis support in twemproxy
setup pre/post splitcopy and pre/post coalesce handlers in msg struct
memcache pre_splitcopy, post_splitcopy, pre_coalesce and post_coalesce handlers
every fragment of a msg vector keeps track of the first/last fragment, number of fragments and fragment owner
set up msg parser handler for memcache connections
refactor parsing code and create header file nc_proto.h
stats_listen should use st->addr as the listening address string
delete stats tracking memcache requests and responses; stats module no longer tracks protocol related stats
2012-10-27 Manju Rajashekhar <manj@twitter.com>
* twemproxy: version 0.1.20 release
on msg_repair, msg->pos should point to nbuf->pos and not nbuf->last
refactor memcache parsing code into proto directory
add redis option to configuration file
fix macro definition strXcmp error for big endian
fix log_hexdump and loga_hexdump
2012-07-31 Manju Rajashekhar <manj@twitter.com>
* twemproxy: version 0.1.19 release
close server connection on a stray response (yashh, bmatheny)
2012-06-19 Manju Rajashekhar <manj@twitter.com>
* twemproxy: version 0.1.18 release
command line option to set mbuf chunk size
2012-05-09 Manju Rajashekhar <manj@twitter.com>
* twemproxy: version 0.1.17 release
use _exit(0) instead of exit(0) when daemonizing
use loga instead of log_stderr in nc_stacktrace
2012-02-09 Manju Rajashekhar <manj@twitter.com>
* twemproxy: version 0.1.16 release
twemproxy (aka nutcracker) is a fast and lightweight proxy for memcached protocol.
================================================
FILE: Dockerfile
================================================
FROM gcc
COPY . /usr/src/twemproxy
WORKDIR /usr/src/twemproxy
RUN \
autoreconf -h && \
autoreconf -fvi && \
./configure && \
make && \
make install
ENTRYPOINT [ "nutcracker" ]
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
================================================
FILE: Makefile.am
================================================
MAINTAINERCLEANFILES = Makefile.in aclocal.m4 configure config.h.in config.h.in~ stamp-h.in
ACLOCAL_AMFLAGS = -I m4
SUBDIRS = contrib src
dist_man_MANS = man/nutcracker.8
EXTRA_DIST = README.md NOTICE LICENSE ChangeLog conf scripts notes
================================================
FILE: NOTICE
================================================
twemproxy is a fast and lightweight proxy for memcached protocol
Copyright (C) 2012 Twitter, Inc.
Portions of twemproxy were inspired from nginx: http://nginx.org/
The implementation of generic array (nc_array.[ch]) and red black tree
(nc_rbtree.[ch]) also comes from nginx-0.8.55.
/*
* Copyright (C) 2002-2010 Igor Sysoev
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
The generic queue implementation comes from BSD <sys/queue.h>
/*
* Copyright (c) 1991, 1993
* The Regents of the University of California. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* 3. All advertising materials mentioning features or use of this software
* must display the following acknowledgement:
* This product includes software developed by the University of
* California, Berkeley and its contributors.
* 4. Neither the name of the University nor the names of its contributors
* may be used to endorse or promote products derived from this software
* without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
The implementation of consistent hashing and individual hash algorithms were
borrowed from libmemcached.
Copyright (c) 2011, Data Differential (http://datadifferential.com/)
Copyright (c) 2007-2010, TangentOrg (Brian Aker)
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of TangentOrg nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The source also includes libyaml (yaml-0.2.5) in contrib/ directory
Copyright (c) 2006 Kirill Simonov
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# twemproxy (nutcracker) [](https://github.com/twitter/twemproxy/actions/workflows/main.yml?query=branch%3Amaster)
**twemproxy** (pronounced "two-em-proxy"), aka **nutcracker** is a fast and lightweight proxy for [memcached](http://www.memcached.org/) and [redis](http://redis.io/) protocol. It was built primarily to reduce the number of connections to the caching servers on the backend. This, together with protocol pipelining and sharding enables you to horizontally scale your distributed caching architecture.
## Build
To build twemproxy 0.5.0+ from [distribution tarball](https://github.com/twitter/twemproxy/releases):
$ ./configure
$ make
$ sudo make install
To build twemproxy 0.5.0+ from [distribution tarball](https://github.com/twitter/twemproxy/releases) in _debug mode_:
$ CFLAGS="-ggdb3 -O0" ./configure --enable-debug=full
$ make
$ sudo make install
To build twemproxy from source with _debug logs enabled_ and _assertions enabled_:
$ git clone git@github.com:twitter/twemproxy.git
$ cd twemproxy
$ autoreconf -fvi
$ ./configure --enable-debug=full
$ make
$ src/nutcracker -h
A quick checklist:
+ Use newer version of gcc (older version of gcc has problems)
+ Use CFLAGS="-O1" ./configure && make
+ Use CFLAGS="-O3 -fno-strict-aliasing" ./configure && make
+ `autoreconf -fvi && ./configure` needs `automake` and `libtool` to be installed
`make check` will run unit tests.
### Older Releases
Distribution tarballs for older twemproxy releases (<= 0.4.1) can be found on [Google Drive](https://drive.google.com/open?id=0B6pVMMV5F5dfMUdJV25abllhUWM&authuser=0).
The build steps are the same (`./configure; make; sudo make install`).
## Features
+ Fast.
+ Lightweight.
+ Maintains persistent server connections.
+ Keeps connection count on the backend caching servers low.
+ Enables pipelining of requests and responses.
+ Supports proxying to multiple servers.
+ Supports multiple server pools simultaneously.
+ Shard data automatically across multiple servers.
+ Implements the complete [memcached ascii](notes/memcache.md) and [redis](notes/redis.md) protocol.
+ Easy configuration of server pools through a YAML file.
+ Supports multiple hashing modes including consistent hashing and distribution.
+ Can be configured to disable nodes on failures.
+ Observability via stats exposed on the stats monitoring port.
+ Works with Linux, *BSD, OS X and SmartOS (Solaris)
## Help
Usage: nutcracker [-?hVdDt] [-v verbosity level] [-o output file]
[-c conf file] [-s stats port] [-a stats addr]
[-i stats interval] [-p pid file] [-m mbuf size]
Options:
-h, --help : this help
-V, --version : show version and exit
-t, --test-conf : test configuration for syntax errors and exit
-d, --daemonize : run as a daemon
-D, --describe-stats : print stats description and exit
-v, --verbose=N : set logging level (default: 5, min: 0, max: 11)
-o, --output=S : set logging file (default: stderr)
-c, --conf-file=S : set configuration file (default: conf/nutcracker.yml)
-s, --stats-port=N : set stats monitoring port (default: 22222)
-a, --stats-addr=S : set stats monitoring ip (default: 0.0.0.0)
-i, --stats-interval=N : set stats aggregation interval in msec (default: 30000 msec)
-p, --pid-file=S : set pid file (default: off)
-m, --mbuf-size=N : set size of mbuf chunk in bytes (default: 16384 bytes)
## Zero Copy
In twemproxy, all the memory for incoming requests and outgoing responses is allocated in mbuf. Mbuf enables zero-copy because the same buffer on which a request was received from the client is used for forwarding it to the server. Similarly the same mbuf on which a response was received from the server is used for forwarding it to the client.
Furthermore, memory for mbufs is managed using a reuse pool. This means that once mbuf is allocated, it is not deallocated, but just put back into the reuse pool. By default each mbuf chunk is set to 16K bytes in size. There is a trade-off between the mbuf size and number of concurrent connections twemproxy can support. A large mbuf size reduces the number of read syscalls made by twemproxy when reading requests or responses. However, with a large mbuf size, every active connection would use up 16K bytes of buffer which might be an issue when twemproxy is handling large number of concurrent connections from clients. When twemproxy is meant to handle a large number of concurrent client connections, you should set chunk size to a small value like 512 bytes using the -m or --mbuf-size=N argument.
## Configuration
Twemproxy can be configured through a YAML file specified by the -c or --conf-file command-line argument on process start. The configuration file is used to specify the server pools and the servers within each pool that twemproxy manages. The configuration files parses and understands the following keys:
+ **listen**: The listening address and port (name:port or ip:port) or an absolute path to sock file (e.g. /var/run/nutcracker.sock) for this server pool.
+ **client_connections**: The maximum number of connections allowed from redis clients. Unlimited by default, though OS-imposed limitations will still apply.
+ **hash**: The name of the hash function. Possible values are:
+ one_at_a_time
+ md5
+ crc16
+ crc32 (crc32 implementation compatible with [libmemcached](http://libmemcached.org/))
+ crc32a (correct crc32 implementation as per the spec)
+ fnv1_64
+ fnv1a_64 (default)
+ fnv1_32
+ fnv1a_32
+ hsieh
+ murmur
+ jenkins
+ **hash_tag**: A two character string that specifies the part of the key used for hashing. Eg "{}" or "$$". [Hash tag](notes/recommendation.md#hash-tags) enable mapping different keys to the same server as long as the part of the key within the tag is the same.
+ **distribution**: The key distribution mode for choosing backend servers based on the computed hash value. Possible values are:
+ ketama (default, recommended. An implementation of https://en.wikipedia.org/wiki/Consistent_hashing)
+ modula (use hash modulo number of servers to choose the backend)
+ random (choose a random backend for each key of each request)
+ **timeout**: The timeout value in msec that we wait for to establish a connection to the server or receive a response from a server. By default, we wait indefinitely.
+ **backlog**: The TCP backlog argument. Defaults to 512.
+ **tcpkeepalive**: A boolean value that controls if tcp keepalive is enabled for connections to servers. Defaults to false.
+ **preconnect**: A boolean value that controls if twemproxy should preconnect to all the servers in this pool on process start. Defaults to false.
+ **redis**: A boolean value that controls if a server pool speaks redis or memcached protocol. Defaults to false.
+ **redis_auth**: Authenticate to the Redis server on connect.
+ **redis_db**: The DB number to use on the pool servers. Defaults to 0. Note: Twemproxy will always present itself to clients as DB 0.
+ **server_connections**: The maximum number of connections that can be opened to each server. By default, we open at most 1 server connection.
+ **auto_eject_hosts**: A boolean value that controls if server should be ejected temporarily when it fails consecutively server_failure_limit times. See [liveness recommendations](notes/recommendation.md#liveness) for information. Defaults to false.
+ **server_retry_timeout**: The timeout value in msec to wait for before retrying on a temporarily ejected server, when auto_eject_hosts is set to true. Defaults to 30000 msec.
+ **server_failure_limit**: The number of consecutive failures on a server that would lead to it being temporarily ejected when auto_eject_hosts is set to true. Defaults to 2.
+ **servers**: A list of server address, port and weight (name:port:weight or ip:port:weight) for this server pool.
For example, the configuration file in [conf/nutcracker.yml](conf/nutcracker.yml), also shown below, configures 5 server pools with names - _alpha_, _beta_, _gamma_, _delta_ and omega. Clients that intend to send requests to one of the 10 servers in pool delta connect to port 22124 on 127.0.0.1. Clients that intend to send request to one of 2 servers in pool omega connect to unix path /tmp/gamma. Requests sent to pool alpha and omega have no timeout and might require timeout functionality to be implemented on the client side. On the other hand, requests sent to pool beta, gamma and delta timeout after 400 msec, 400 msec and 100 msec respectively when no response is received from the server. Of the 5 server pools, only pools alpha, gamma and delta are configured to use server ejection and hence are resilient to server failures. All the 5 server pools use ketama consistent hashing for key distribution with the key hasher for pools alpha, beta, gamma and delta set to fnv1a_64 while that for pool omega set to hsieh. Also only pool beta uses [nodes names](notes/recommendation.md#node-names-for-consistent-hashing) for consistent hashing, while pool alpha, gamma, delta and omega use 'host:port:weight' for consistent hashing. Finally, only pool alpha and beta can speak the redis protocol, while pool gamma, delta and omega speak memcached protocol.
alpha:
listen: 127.0.0.1:22121
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: true
redis: true
server_retry_timeout: 2000
server_failure_limit: 1
servers:
- 127.0.0.1:6379:1
beta:
listen: 127.0.0.1:22122
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
auto_eject_hosts: false
timeout: 400
redis: true
servers:
- 127.0.0.1:6380:1 server1
- 127.0.0.1:6381:1 server2
- 127.0.0.1:6382:1 server3
- 127.0.0.1:6383:1 server4
gamma:
listen: 127.0.0.1:22123
hash: fnv1a_64
distribution: ketama
timeout: 400
backlog: 1024
preconnect: true
auto_eject_hosts: true
server_retry_timeout: 2000
server_failure_limit: 3
servers:
- 127.0.0.1:11212:1
- 127.0.0.1:11213:1
delta:
listen: 127.0.0.1:22124
hash: fnv1a_64
distribution: ketama
timeout: 100
auto_eject_hosts: true
server_retry_timeout: 2000
server_failure_limit: 1
servers:
- 127.0.0.1:11214:1
- 127.0.0.1:11215:1
- 127.0.0.1:11216:1
- 127.0.0.1:11217:1
- 127.0.0.1:11218:1
- 127.0.0.1:11219:1
- 127.0.0.1:11220:1
- 127.0.0.1:11221:1
- 127.0.0.1:11222:1
- 127.0.0.1:11223:1
omega:
listen: /tmp/gamma 0666
hash: hsieh
distribution: ketama
auto_eject_hosts: false
servers:
- 127.0.0.1:11214:100000
- 127.0.0.1:11215:1
Finally, to make writing a syntactically correct configuration file easier, twemproxy provides a command-line argument `-t` or `--test-conf` that can be used to test the YAML configuration file for any syntax error.
## Observability
Observability in twemproxy is through logs and stats.
Twemproxy exposes stats at the granularity of server pool and servers per pool through the stats monitoring port by responding with the raw data over TCP. The stats are essentially JSON formatted key-value pairs, with the keys corresponding to counter names. By default stats are exposed on port 22222 and aggregated every 30 seconds. Both these values can be configured on program start using the `-c` or `--conf-file` and `-i` or `--stats-interval` command-line arguments respectively. You can print the description of all stats exported by using the `-D` or `--describe-stats` command-line argument.
$ nutcracker --describe-stats
pool stats:
client_eof "# eof on client connections"
client_err "# errors on client connections"
client_connections "# active client connections"
server_ejects "# times backend server was ejected"
forward_error "# times we encountered a forwarding error"
fragments "# fragments created from a multi-vector request"
server stats:
server_eof "# eof on server connections"
server_err "# errors on server connections"
server_timedout "# timeouts on server connections"
server_connections "# active server connections"
requests "# requests"
request_bytes "total request bytes"
responses "# responses"
response_bytes "total response bytes"
in_queue "# requests in incoming queue"
in_queue_bytes "current request bytes in incoming queue"
out_queue "# requests in outgoing queue"
out_queue_bytes "current request bytes in outgoing queue"
See [`notes/debug.txt`](notes/debug.txt) for examples of how to read the stats from the stats port.
Logging in twemproxy is only available when twemproxy is built with logging enabled. By default logs are written to stderr. Twemproxy can also be configured to write logs to a specific file through the `-o` or `--output` command-line argument. On a running twemproxy, we can turn log levels up and down by sending it SIGTTIN and SIGTTOU signals respectively and reopen log files by sending it SIGHUP signal.
## Pipelining
Twemproxy enables proxying multiple client connections onto one or few server connections. This architectural setup makes it ideal for pipelining requests and responses and hence saving on the round trip time.
For example, if twemproxy is proxying three client connections onto a single server and we get requests - `get key\r\n`, `set key 0 0 3\r\nval\r\n` and `delete key\r\n` on these three connections respectively, twemproxy would try to batch these requests and send them as a single message onto the server connection as `get key\r\nset key 0 0 3\r\nval\r\ndelete key\r\n`.
Pipelining is the reason why twemproxy ends up doing better in terms of throughput even though it introduces an extra hop between the client and server.
## Deployment
If you are deploying twemproxy in production, you might consider reading through the [recommendation document](notes/recommendation.md) to understand the parameters you could tune in twemproxy to run it efficiently in the production environment.
## Utils
+ [collectd-plugin](https://github.com/bewie/collectd-twemproxy)
+ [munin-plugin](https://github.com/eveiga/contrib/tree/nutcracker/plugins/nutcracker)
+ [twemproxy-ganglia-module](https://github.com/ganglia/gmond_python_modules/tree/master/twemproxy)
+ [nagios checks](https://github.com/wanelo/nagios-checks/blob/master/check_twemproxy)
+ [circonus](https://github.com/wanelo-chef/nad-checks/blob/master/recipes/twemproxy.rb)
+ [puppet module](https://github.com/wuakitv/puppet-twemproxy)
+ [nutcracker-web](https://github.com/kontera-technologies/nutcracker-web)
+ [redis-twemproxy agent](https://github.com/Stono/redis-twemproxy-agent)
+ [sensu-metrics](https://github.com/sensu-plugins/sensu-plugins-twemproxy/blob/master/bin/metrics-twemproxy.rb)
+ [redis-mgr](https://github.com/idning/redis-mgr)
+ [smitty for twemproxy failover](https://github.com/areina/smitty)
+ [Beholder, a Python agent for twemproxy failover](https://github.com/Serekh/beholder)
+ [chef cookbook](https://supermarket.getchef.com/cookbooks/twemproxy)
+ [twemsentinel](https://github.com/yak0/twemsentinel)
## Companies using Twemproxy in Production
+ [Twitter](https://twitter.com/)
+ [Wikimedia](https://www.wikimedia.org/)
+ [Pinterest](http://pinterest.com/)
+ [Snapchat](http://www.snapchat.com/)
+ [Flickr](https://www.flickr.com)
+ [Yahoo!](https://www.yahoo.com)
+ [Tumblr](https://www.tumblr.com/)
+ [Vine](http://vine.co/)
+ [Wayfair](http://www.wayfair.com/)
+ [Kiip](http://www.kiip.me/)
+ [Wuaki.tv](https://wuaki.tv/)
+ [Wanelo](http://wanelo.com/)
+ [Kontera](http://kontera.com/)
+ [Bright](http://www.bright.com/)
+ [56.com](http://www.56.com/)
+ [Digg](http://digg.com/)
+ [Gawkermedia](http://advertising.gawker.com/)
+ [3scale.net](http://3scale.net)
+ [Ooyala](http://www.ooyala.com)
+ [Twitch](http://twitch.tv)
+ [Socrata](http://www.socrata.com/)
+ [Hootsuite](http://hootsuite.com/)
+ [Trivago](http://www.trivago.com/)
+ [Machinezone](http://www.machinezone.com)
+ [Path](https://path.com)
+ [AOL](http://engineering.aol.com/)
+ [Soysuper](https://soysuper.com/)
+ [Vinted](http://vinted.com/)
+ [Poshmark](https://poshmark.com/)
+ [FanDuel](https://www.fanduel.com/)
+ [Bloomreach](http://bloomreach.com/)
+ [Hootsuite](https://hootsuite.com)
+ [Tradesy](https://www.tradesy.com/)
+ [Uber](http://uber.com) ([details](http://highscalability.com/blog/2015/9/14/how-uber-scales-their-real-time-market-platform.html))
+ [Greta](https://greta.io/)
## Issues and Support
Have a bug or a question? Please create an issue here on GitHub!
https://github.com/twitter/twemproxy/issues
## Committers
* Manju Rajashekhar ([@manju](https://twitter.com/manju))
* Lin Yang ([@idning](https://github.com/idning))
* Tyson Andre ([@TysonAndre](https://github.com/TysonAndre))
Thank you to all of our [contributors](https://github.com/twitter/twemproxy/graphs/contributors)!
## License
Copyright 2012 Twitter, Inc.
Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0
================================================
FILE: ci/Dockerfile
================================================
# Dockerfile to create a slower debug build of nutcracker with assertions enabled
# for continuous integration checks.
# ARGS: REDIS_VER
# Also see test_in_docker.sh
FROM centos:7
ENV LAST_MODIFIED_DATE 2021-04-09
RUN yum install -y \
https://repo.ius.io/ius-release-el7.rpm \
https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# socat Allow tests to open log files
# which (used below)
# python-setuptools for pip
RUN yum install -y \
tar git gcc make tcl \
autoconf automake libtool wget \
memcached \
socat \
which \
python36u python36u-pip && \
yum clean all
# Install nosetest dependencies
RUN pip3.6 install nose && \
pip3.6 install 'git+https://github.com/andymccurdy/redis-py.git@3.5.3' && \
pip3.6 install 'git+https://github.com/linsomniac/python-memcached.git@1.58'
# I can't install redis or redis-server in centos:7 (didn't add package), adding to centos 6
# Install redis and redis sentinel, needed for unit tests
# RUN yum install -y redis redis-sentinel
ARG REDIS_VER=3.2.11
RUN wget https://github.com/redis/redis/archive/$REDIS_VER.tar.gz && \
tar zxvf $REDIS_VER.tar.gz && \
pushd redis-$REDIS_VER && \
make install && \
popd && \
rm -r redis-*
# This will build twemproxy with compilation flags for running unit tests, integration tests.
# Annoyingly, this can't add multiple directories at once.
ADD conf /usr/src/twemproxy/conf
ADD contrib /usr/src/twemproxy/contrib
ADD man /usr/src/twemproxy/man
ADD m4 /usr/src/twemproxy/m4
ADD notes /usr/src/twemproxy/notes
ADD scripts /usr/src/twemproxy/scripts
ADD src /usr/src/twemproxy/src
ADD ChangeLog configure.ac Makefile.am LICENSE NOTICE README.md /usr/src/twemproxy/
WORKDIR /usr/src/twemproxy
ADD ci/build-nutcracker.sh /usr/local/bin/build-nutcracker.sh
RUN /usr/local/bin/build-nutcracker.sh
# Add the tests after adding source files, which makes it easy to quickly test changes to unit tests.
ADD tests /usr/src/twemproxy/tests
# Not installing redis utilities, since we're not running those tests.
RUN mkdir tests/_binaries -p
RUN ln -nsf $PWD/src/nutcracker tests/_binaries/nutcracker && \
cp `which redis-server` tests/_binaries/redis-server && \
cp `which redis-server` tests/_binaries/redis-sentinel && \
cp `which memcached` tests/_binaries/memcached && \
cp `which redis-cli` tests/_binaries/redis-cli
WORKDIR /usr/src/twemproxy/tests
# Allow tests to open log files
RUN chmod -R a+w log/
RUN cat /etc/passwd
RUN chown -R daemon:daemon /usr/src/twemproxy
USER daemon
================================================
FILE: ci/build-nutcracker.sh
================================================
#!/usr/bin/env bash
set -xeu
function cleanup {
rm -f *.gz
}
trap cleanup EXIT
trap cleanup INT
cleanup
export CFLAGS="-O3 -fno-strict-aliasing -I/usr/lib/x86_64-redhat-linux6E/include -B /usr/lib/x86_64-redhat-linux6E/lib64"
# TODO: Figure out how to make this apply only to the contrib/ directory. Maybe override the yaml directory's Makefile.am after extracting it.
# https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119
CFLAGS+=" -Werror -Wall -Wno-pointer-sign -Wno-sign-conversion -Wno-missing-braces -Wno-unused-value -Wno-builtin-declaration-mismatch -Wno-maybe-uninitialized"
export LDFLAGS="-lc_nonshared"
cd /usr/src/twemproxy
autoreconf -fvi
./configure --enable-debug=yes --prefix=/usr/src/twemproxy/work/usr
make -j5
make install
================================================
FILE: conf/nutcracker.leaf.yml
================================================
leaf:
listen: 127.0.0.1:22121
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: true
server_retry_timeout: 2000
server_failure_limit: 1
servers:
- 127.0.0.1:11212:1
- 127.0.0.1:11213:1
================================================
FILE: conf/nutcracker.root.yml
================================================
root:
listen: 127.0.0.1:22120
hash: fnv1a_64
distribution: ketama
preconnect: true
auto_eject_hosts: false
servers:
- 127.0.0.1:22121:1
================================================
FILE: conf/nutcracker.yml
================================================
alpha:
listen: 127.0.0.1:22121
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: true
redis: true
server_retry_timeout: 2000
server_failure_limit: 1
servers:
- 127.0.0.1:6379:1
beta:
listen: 127.0.0.1:22122
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
auto_eject_hosts: false
timeout: 400
redis: true
servers:
- 127.0.0.1:6380:1 server1
- 127.0.0.1:6381:1 server2
- 127.0.0.1:6382:1 server3
- 127.0.0.1:6383:1 server4
gamma:
listen: 127.0.0.1:22123
hash: fnv1a_64
distribution: ketama
timeout: 400
backlog: 1024
preconnect: true
auto_eject_hosts: true
server_retry_timeout: 2000
server_failure_limit: 3
servers:
- 127.0.0.1:11212:1
- 127.0.0.1:11213:1
delta:
listen: 127.0.0.1:22124
hash: fnv1a_64
distribution: ketama
timeout: 100
auto_eject_hosts: true
server_retry_timeout: 2000
server_failure_limit: 1
servers:
- 127.0.0.1:11214:1
- 127.0.0.1:11215:1
- 127.0.0.1:11216:1
- 127.0.0.1:11217:1
- 127.0.0.1:11218:1
- 127.0.0.1:11219:1
- 127.0.0.1:11220:1
- 127.0.0.1:11221:1
- 127.0.0.1:11222:1
- 127.0.0.1:11223:1
omega:
listen: /tmp/gamma
hash: hsieh
distribution: ketama
auto_eject_hosts: false
servers:
- 127.0.0.1:11214:100000
- 127.0.0.1:11215:1
================================================
FILE: configure.ac
================================================
# Define the package version numbers and the bug reporting address
m4_define([NC_MAJOR], 0)
m4_define([NC_MINOR], 5)
m4_define([NC_PATCH], 0)
m4_define([NC_BUGS], [https://github.com/twitter/twemproxy/issues])
# Initialize autoconf
AC_PREREQ([2.64])
AC_INIT([nutcracker], [NC_MAJOR.NC_MINOR.NC_PATCH], [NC_BUGS])
AC_CONFIG_SRCDIR([src/nc.c])
AC_CONFIG_AUX_DIR([config])
AC_CONFIG_HEADERS([config.h:config.h.in])
AC_CONFIG_MACRO_DIR([m4])
# Initialize automake
AM_INIT_AUTOMAKE([1.9 foreign])
# Define macro variables for the package version numbers
AC_DEFINE(NC_VERSION_MAJOR, NC_MAJOR, [Define the major version number])
AC_DEFINE(NC_VERSION_MINOR, NC_MINOR, [Define the minor version number])
AC_DEFINE(NC_VERSION_PATCH, NC_PATCH, [Define the patch version number])
AC_DEFINE(NC_VERSION_STRING, "NC_MAJOR.NC_MINOR.NC_PATCH", [Define the version string])
# Checks for language
AC_LANG([C])
# Checks for programs
AC_PROG_AWK
AC_PROG_CC
AC_PROG_CPP
AC_PROG_CXX
AC_PROG_INSTALL
AC_PROG_LN_S
AC_PROG_MAKE_SET
AC_PROG_RANLIB
AC_PROG_LIBTOOL
# Checks for typedefs, structures, and compiler characteristics
AC_C_INLINE
AC_TYPE_INT8_T
AC_TYPE_INT16_T
AC_TYPE_INT32_T
AC_TYPE_INT64_T
AC_TYPE_INTMAX_T
AC_TYPE_INTPTR_T
AC_TYPE_UINT8_T
AC_TYPE_UINT16_T
AC_TYPE_UINT32_T
AC_TYPE_UINT64_T
AC_TYPE_UINTMAX_T
AC_TYPE_UINTPTR_T
AC_TYPE_OFF_T
AC_TYPE_PID_T
AC_TYPE_SIZE_T
AC_TYPE_SSIZE_T
AC_C_BIGENDIAN(
[],
[AC_DEFINE(HAVE_LITTLE_ENDIAN, 1, [Define to 1 if machine is little endian])],
[AC_MSG_ERROR([endianess of this machine is unknown])],
[AC_MSG_ERROR([universial endianess not supported])]
)
# Checks for header files
AC_HEADER_STDBOOL
AC_CHECK_HEADERS([fcntl.h float.h limits.h stddef.h stdlib.h string.h unistd.h])
AC_CHECK_HEADERS([inttypes.h stdint.h])
AC_CHECK_HEADERS([sys/ioctl.h sys/time.h sys/uio.h])
AC_CHECK_HEADERS([sys/socket.h sys/un.h netinet/in.h arpa/inet.h netdb.h])
AC_CHECK_HEADERS([execinfo.h],
[AC_DEFINE(HAVE_BACKTRACE, [1], [Define to 1 if backtrace is supported])], [])
AC_CHECK_HEADERS([sys/epoll.h], [], [])
AC_CHECK_HEADERS([sys/event.h], [], [])
# Checks for libraries
AC_CHECK_LIB([m], [pow])
AC_CHECK_LIB([pthread], [pthread_create])
# Checks for library functions
AC_FUNC_FORK
AC_FUNC_MALLOC
AC_FUNC_REALLOC
AC_CHECK_FUNCS([dup2 gethostname gettimeofday strerror])
AC_CHECK_FUNCS([socket])
AC_CHECK_FUNCS([memchr memmove memset])
AC_CHECK_FUNCS([strchr strndup strtoul])
AC_CHECK_FUNCS([strdup])
AC_CACHE_CHECK([if epoll works], [ac_cv_epoll_works],
AC_TRY_RUN([
#include <stdio.h>
#include <stdlib.h>
#include <sys/epoll.h>
int
main(int argc, char **argv)
{
int fd;
fd = epoll_create(256);
if (fd < 0) {
perror("epoll_create:");
exit(1);
}
exit(0);
}
], [ac_cv_epoll_works=yes], [ac_cv_epoll_works=no]))
AS_IF([test "x$ac_cv_epoll_works" = "xyes"],
[AC_DEFINE([HAVE_EPOLL], [1], [Define to 1 if epoll is supported])], [])
AC_CACHE_CHECK([if kqueue works], [ac_cv_kqueue_works],
AC_TRY_RUN([
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/event.h>
#include <sys/time.h>
int
main(int argc, char **argv)
{
int fd;
fd = kqueue();
if (fd < 0) {
perror("kqueue:");
exit(1);
}
exit(0);
}
], [ac_cv_kqueue_works=yes], [ac_cv_kqueue_works=no]))
AS_IF([test "x$ac_cv_kqueue_works" = "xyes"],
[AC_DEFINE([HAVE_KQUEUE], [1], [Define to 1 if kqueue is supported])], [])
AC_CACHE_CHECK([if event ports works], [ac_cv_evports_works],
AC_TRY_RUN([
#include <stdio.h>
#include <stdlib.h>
#include <port.h>
int
main(int argc, char **argv)
{
int fd;
fd = port_create();
if (fd < 0) {
perror("port_create:");
exit(1);
}
exit(0);
}
], [ac_cv_evports_works=yes], [ac_cv_evports_works=no]))
AS_IF([test "x$ac_cv_evports_works" = "xyes"],
[AC_DEFINE([HAVE_EVENT_PORTS], [1], [Define to 1 if event ports is supported])], [])
AS_IF([test "x$ac_cv_epoll_works" = "xno" &&
test "x$ac_cv_kqueue_works" = "xno" &&
test "x$ac_cv_evports_works" = "xno"],
[AC_MSG_ERROR([either epoll or kqueue or event ports support is required])], [])
AM_CONDITIONAL([OS_LINUX], [test "x$ac_cv_epoll_works" = "xyes"])
AM_CONDITIONAL([OS_BSD], [test "x$ac_cv_kqueue_works" = "xyes"])
AM_CONDITIONAL([OS_SOLARIS], [test "x$ac_cv_evports_works" = "xyes"])
AM_CONDITIONAL([OS_FREEBSD], [test "$(uname -v | cut -c 1-10)" == "FreeBSD 10"])
# Package options
AC_MSG_CHECKING([whether to enable debug logs and asserts])
AC_ARG_ENABLE([debug],
[AS_HELP_STRING(
[--enable-debug=@<:@full|yes|log|no@:>@],
[enable debug logs and asserts @<:@default=no@:>@])
],
[],
[enable_debug=no])
AS_CASE([x$enable_debug],
[xfull], [AC_DEFINE([HAVE_ASSERT_PANIC], [1],
[Define to 1 if panic on an assert is enabled])
AC_DEFINE([HAVE_DEBUG_LOG], [1], [Define to 1 if debug log is enabled])
],
[xyes], [AC_DEFINE([HAVE_ASSERT_LOG], [1],
[Define to 1 if log on an assert is enabled])
AC_DEFINE([HAVE_DEBUG_LOG], [1], [Define to 1 if debug log is enabled])
],
[xlog], [AC_DEFINE([HAVE_DEBUG_LOG], [1], [Define to 1 if debug log is enabled])],
[xno], [],
[AC_MSG_FAILURE([invalid value ${enable_debug} for --enable-debug])])
AC_MSG_RESULT($enable_debug)
AC_MSG_CHECKING([whether to disable stats])
AC_ARG_ENABLE([stats],
[AS_HELP_STRING(
[--disable-stats],
[disable stats])
],
[disable_stats=yes],
[disable_stats=no])
AS_IF([test "x$disable_stats" = xyes],
[],
[AC_DEFINE([HAVE_STATS], [1], [Define to 1 if stats is not disabled])])
AC_MSG_RESULT($disable_stats)
# Untar the yaml-0.2.5 in contrib/ before config.status is rerun
AC_CONFIG_COMMANDS_PRE([tar xvfz contrib/yaml-0.2.5.tar.gz -C contrib])
# Call yaml-0.2.5 ./configure recursively
AC_CONFIG_SUBDIRS([contrib/yaml-0.2.5])
# Define Makefiles
AC_CONFIG_FILES([Makefile
contrib/Makefile
src/Makefile
src/hashkit/Makefile
src/proto/Makefile
src/event/Makefile])
# Generate the "configure" script
AC_OUTPUT
================================================
FILE: contrib/Makefile.am
================================================
SUBDIRS = yaml-0.2.5
EXTRA_DIST = yaml-0.2.5.tar.gz
================================================
FILE: contrib/yaml-0.2.5/.gitignore
================================================
# Ignore everything
*
# Except me
!.gitignore
================================================
FILE: m4/.gitignore
================================================
# Ignore everything
*
# Except me
!.gitignore
================================================
FILE: man/nutcracker.8
================================================
.TH NUTCRACKER 8 "June 13, 2013"
.SH NAME
nutcracker \- Fast, light-weight proxy for memcached and Redis
.SH SYNOPSIS
.B nutcracker
.RI [ options ]
.SH DESCRIPTION
\fBnutcracker\fP, also known as \fBtwemproxy\fP (pronounced "two-em-proxy"), is
a fast and lightweight proxy for the memcached and Redis protocols.
.PP
It was primarily built to reduce the connection count on backend caching
servers, but it has a number of features, such as:
.IP \[bu]
Maintains persistent server connections to backend servers.
.IP \[bu]
Enables pipelining of requests and responses.
.IP \[bu]
Supports multiple server pools simultaneously.
.IP \[bu]
Shard data automatically across multiple servers.
.IP \[bu]
Supports multiple hashing modes including consistent hashing and
distribution.
.IP \[bu]
High-availability by disabling nodes on failures.
.IP \[bu]
Observability through stats exposed on stats monitoring port.
.SH OPTIONS
.TP
.BR \-h ", " \-\-help
Show usage information and exit.
.TP
.BR \-V ", " \-\-version
Show version and exit.
.TP
.BR \-t ", " \-\-test-conf
Test configuration for syntax errors and exit.
.TP
.BR \-D ", " \-\-describe-stats
Print stats description and exit.
.TP
.BR \-v ", " \-\-verbose=\fIN\fP
Set logging level to \fIN\fP. (default: 5, min: 0, max: 11)
.TP
.BR \-o ", " \-\-output=\fIfilename\fP
Set logging file to \fIfilename\fP.
.TP
.BR \-c ", " \-\-conf-file=\fIfilename\fP
Set configuration file to \fIfilename\fP.
.TP
.BR \-s ", " \-\-stats-port=\fIport\fP
Set stats monitoring port to \fIport\fP.
(default: 22222)
.TP
.BR \-a ", " \-\-stats-addr=\fIaddress\fP
Set stats monitoring IP to \fIaddress\fP.
(default: 0.0.0.0)
.TP
.BR \-i ", " \-\-stats-interval=\fIinterval\fP
Set stats aggregation interval in msec to \fIinterval\fP.
(default: 30000 msec)
.TP
.BR \-m ", " \-\-mbuf-size=\fIsize\fP
Set size of mbuf chunk in bytes to \fIsize\fP. (default: 16384 bytes)
.TP
.BR \-d ", " \-\-daemonize
Run as a daemon.
.TP
.BR \-p ", " \-\-pid-file=\fIfilename\fP
Set pid file to \fIfilename\fP.
.SH SEE ALSO
.BR memcached (8),
.BR redis-server (1)
.br
.SH AUTHOR
nutcracker was written by Twitter, Inc.
================================================
FILE: notes/c-styleguide.txt
================================================
- No literal tabs. Expand tabs to 4 spaces.
- Indentation is 4 spaces.
- No more than 3 levels of indentation, otherwise you should think about
refactoring your code.
- Use one statement per line.
- Make sure that your editor does not leave space at the end of the line.
- snake_case for variable, function and file names.
- Use your own judgment when naming variables and functions. Be as Spartan
as possible. E.g.: Using name like this_variable_is_a_temporary_counter
will usually be frowned upon.
- Don’t use local variables or parameters that shadow global identifiers.
GCC’s ‘-Wshadow’ option can help you to detect this problem.
- Avoid using int, char, short, long. Instead use int8_t uint8_t, int16_t,
uint16_t, int32_t, uint32_t, int64_t, uint64_t, which are available in
<stdint.h>. However, when interfacing with system calls and libraries
you cannot get away from using int and char.
- Use bool for boolean variables. You have to include <stdbool.h>
- Avoid using a bool as type for struct member names. Instead use unsigned
1-bit bit field. E.g.:
struct foo {
unsigned is_bar:1;
};
- Always use size_t type when dealing with sizes of objects or memory ranges.
- Your code should be 64-bit and 32-bit friendly. Bear in mind problems
of printing, comparisons, and structure alignment. You have to include
<inttypes.h> to get generic format specifier macros for printing.
- 80 column line limit.
- If you have to wrap a long statement (> 80 column), put the operator at the
end of the line and indent the next line at the same column as the arguments
in the previous column. E.g.:
while (cnt < 20 && this_variable_name_is_too_long &&
ep != NULL) {
z = a + really + long + statement + that + needs + three + lines +
gets + indented + on + the + same + column + as + the +
previous + column
}
and:
int a = function(param_a, param_b, param_c, param_d, param_e, param_f,
param_g, param_h, param_i, param_j, param_k, param_l);
- Always use braces for all conditional blocks (if, switch, for, while, do).
This holds good even for single statement conditional blocks. E.g.:
if (cond) {
stmt;
}
- Placement of braces for non-function statement blocks - put opening brace
last on the line and closing brace first. E.g.:
if (x is true) {
we do y
}
- Placement of brace for functions - put the opening brace at the beginning
of the next line and closing brace first. This is useful because several
tools look for opening brace in column one to find beginning of C
functions. E.g.:
int
function(int x)
{
body of the function
}
- Closing brace is empty on a line of its own, except in cases where it is
followed by a continuation of the same statement, i.e. a "while" in a
do-statement or an "else" in an if-statement, like this:
do {
body of do-loop
} while (condition);
and,
if (x == y) {
..
} else if (x > y) {
...
} else {
....
}
- Column align switch keyword and the corresponding case/default keyword. E.g.:
switch (alphabet) {
case 'a':
case 'b':
printf("I am a or b\n");
break;
default:
break;
}
- Forever loops are done with for, and not while. E.g.:
for (;;) {
stmt;
}
- Don't use a space after a function name.
- Do not needlessly surround the return expression with parentheses.
- Use space after keywords. Exceptions are sizeof, typeof, alignof and
__attribute__, which look like functions.
- Do not add spaces around (inside) parenthesized expressions.
s = sizeof( sizeof(*p)) ); /* bad example */
s = sizeof(sizeof(*p)); /* good example */
- Casts should not be followed by space. E.g.:
int q = *(int *)&p
- There is no need to type cast when assigning a void pointer to a non-void
pointer, or vice versa.
- Avoid using goto statements. However there are some exceptions to this rule
when a single goto label within a function and one or more goto statements
come in handy when a function exits from multiple locations and some common
work such as cleanup has to be done. E.g.:
int
fun(void)
{
int result = 0;
char *buffer;
buffer = malloc(1024);
if (buffer == NULL) {
return -1;
}
if (condition1) {
while (loop1) {
...
}
result = 1;
goto out;
}
...
out:
free(buffer);
return result;
}
- When declaring pointer data, use '*' adjacent to the data name and not
adjacent to the type name. E.g.:
int
function(int *p)
{
char *p;
<body of the function>
}
- Use one space around (on each side of) most binary and ternary operators,
such as any of these:
= + - < > * / % | & ^ <= >= == != ? :
but no space after unary operators:
& * + - ~ ! sizeof typeof alignof __attribute__ defined
no space before the postfix increment & decrement unary operators:
++ --
and no space around the '.' and "->" structure member operators.
- 0 and NULL; use 0 for integers, 0.0 for doubles, NULL for pointers, and
'\0' for chars.
- Test pointers against NULL. E.g, use:
if (p == NULL)
not:
!(p)
- Do not use ! for tests unless it is a boolean. E.g. use:
if (*p == '\0')
not:
if (!*p)
- Don't use assignments inside if or while-conditions. E.g, use:
struct foo *foo;
foo = malloc(sizeof(*foo));
if (foo == NULL) {
return -1
}
not:
struct foo *foo;
if ((foo = malloc(sizeof(*foo))) == NULL) {
return -1;
}
- Don't ever use typedef for structure types. Typedefs are problematic
because they do not properly hide their underlying type; for example you
need to know if the typedef is the structure itself or a pointer to the
structure. In addition they must be declared exactly once, whereas an
incomplete structure type can be mentioned as many times as necessary.
Typedefs are difficult to use in stand-alone header files: the header
that defines the typedef must be included before the header that uses it,
or by the header that uses it (which causes namespace pollution), or
there must be a back-door mechanism for obtaining the typedef.
- The only exception for using a typedef is when you are defining a type
for a function pointer or a type for an enum. E.g.:
typedef void (*foo_handler_t)(int, void *);
or:
typedef enum types {
TYPE_1,
TYPE_2
} types_t;
- Use just one variable declaration per line when variables are part of a
struct. This leaves you room for a small comment on each item, explaining
its use. Declarations should also be aligned. E.g., use:
struct foo {
int *foo_a; /* comment for foo_a */
int foo_b; /* comment for foo_b */
unsigned foo_c:1; /* comment for foo_c */
};
and not:
struct foo {
int *foo_a, foo_b;
unsigned foo_c:1;
};
- For variable declaration outside a struct, either collect all the
declarations of the same type on a single line, or use one variable
per line if the variables purpose needs to be commented. E.g.:
char *a, *b, c;
or:
char *a, *b;
char c; /* comments for c */
- Avoid magic numbers because no-one has a clue (including the author) of
what it means after a month.
- Function definitions should start the name of the function in column
one. This is useful because it makes searching for function definitions
fairly trivial. E.g.:
static char *
concat(char *s1, char *s2)
{
body of the function
}
- Function and variables local to a file should be static.
- Separate two successive functions with one blank line.
- Include parameter names with their datatypes in function declaration. E.g.:
void function(int param);
- Functions should be short and sweet, and do just one thing. They should
fit on one or two screenfuls of text (80 x 24 screen size), and do one
thing and do that well.
The maximum length of a function is inversely proportional to the
complexity and indentation level of that function. So, if you have a
conceptually simple function that is just one long (but simple)
case-statement, where you have to do lots of small things for a lot of
different cases, it's OK to have a longer function.
Another measure of the function is the number of local variables. They
shouldn't exceed 5-10, or you're doing something wrong. Re-think the
function, and split it into smaller pieces. A human brain can
generally easily keep track of about 7 different things, anything more
and it gets confused. You know you're brilliant, but maybe you'd like
to understand what you did 2 weeks from now.
- Use const for function parameters passed by reference, if the passed
pointer has no side effect.
- C style comments only. Don't use // for single line comments. Instead
use /* ... */ style.
- For multi-line comments use the following style
/*
* This is the preferred style for multi-line
* comments in the Linux kernel source code.
* Please use it consistently.
*
* Description: A column of asterisks on the left side,
* with beginning and ending almost-blank lines.
*/
- To comment out block of code spanning several lines use preprocessor
directive "#ifdef 0 ... #endif"
- Please write a brief comment at the start of each source file, with the
file name and a line or two about the overall purpose of the file.
- All major functions should have comments describing what they do at the
head of the function. Avoid putting comments in the function body unless
absolutely needed. If possible, add a comment on what sorts of arguments
the function gets, and what the possible values of arguments mean and
what they are used for and the significance of return value if there is
one. It is not necessary to duplicate in words the meaning of the C
argument declarations, if a C type is being used in its customary fashion.
If there is anything nonstandard about its use (such as an argument of
type char * which is really the address of the second character of a
string, not the first), or any possible values that would not work the
way one would expect (such as, that strings containing newlines are not
guaranteed to work), be sure to say so. E.g.:
/*
* Try to acquire a physical address lock while a pmap is locked. If we
* fail to trylock we unlock and lock the pmap directly and cache the
* locked pa in *locked. The caller should then restart their loop in case
* the virtual to physical mapping has changed.
*
* Returns 0 on success and -1 on failure.
*/
int
vm_page_pa_tryrelock(pmap_t pmap, vm_paddr_t pa, vm_paddr_t *locked)
{
...
- The comment on a function is much clearer if you use the argument names
to speak about the argument values. The variable name itself should be
lower case, but write it in upper case when you are speaking about the
value rather than the variable itself. Thus, “the inode number NODE_NUM”
rather than “an inode”.
- Every struct definition should have an accompanying comment that
describes what it is for and how it should be used.
- Finally, while comments are absolutely important to keep the code readable,
remember that the best code is self-documenting. Giving sensible names to
types and variables is much better than using obscure names that you must
then explain through comments.
- Recommend using UPPERCASE for macro names. However, sometimes using
lowercase for macro names makes sense when macros masquerade as well-known
function calls. E.g., it makes sense to write the wrapper for the
standard free() function in lowercase to keep the readability
consistent:
#define my_free(_p) do { \
free(_p); \
(_p) = NULL; \
} while (0)
- Use enums when defining more than one related constants. All enumeration
values are in UPPERCASE.
- Avoid macros as much as possible and use inline functions, enums and const
variables wherever you can.
- For macros encapsulating compound statements, right justify the backslashes
and enclose it in do { ... } while (0)
- For parameterized macros, all the parameters used in the macro body must
be surrounded by parentheses. E.g.:
#define ADD_1(_x) ((_x) + 1)
- Use sizeof(varname) instead of sizeof(type) whenever possible. E.g.:
char *p;
p = malloc(sizeof(*p)); /* good example */
p = malloc(sizeof(char)); /* bad example */
- All variables should be declared at the beginning of a scope block {..}.
It is even preferred to declare all variables at the beginning of the
function so that all the local variable declarations is in one place and
we can see the comprehensive list in one glance.
- Global structs should be declared at the top of the file in which they
are used, or in separate header files if they are used in multiple
source files.
- Declarations of external functions and functions to appear later in the
source file should all go in one place near the beginning of the file,
somewhere before the first function definition in the file or else
should go in a header file.
- Use of extern should be considered as evil, if it is used in header files
to reference global variables.
- Don’t put extern declarations inside functions.
- Usually every *.c file should have an associated *.h file. There are some
exceptions to this rule, such as unit tests and small *.c files containing
just the main() function.
- Every header file in the source code must have preprocessor conditional
to prevent the header file from being scanned multiple times and avoiding
mutual dependency cycles. Alternatively you can use #pragma once directive,
as it avoids name clashes and increases the compile speed. E.g., for a
header file named foo.h, the entire contents of the header file must be
between the guard macros as follows:
#ifndef _FOO_H_
#define _FOO_H_
...
#endif /* _FOO_H_ */
Or,
#pragma once
#ifndef _FOO_H_
#define _FOO_H_
...
#endif /* _FOO_H_ */
- Don't use #include when a forward declaration would suffice.
- Functions defined in header files should be static inline.
- Don’t make the program ugly just to placate GCC when extra warnings options
such as ‘-Wconversion’ or ‘-Wundef’ are used. These options can help in
finding bugs, but they can also generate so many false alarms that that
it hurts readability to silence them with unnecessary casts, wrappers, and
other complications.
- Conditional compilation: when supporting configuration options already
known when building your program we prefer using if (... ) over conditional
compilation, as in the former case the compiler is able to perform more
extensive checking of all possible code paths. E.g., use:
if (HAS_FOO)
...
else
...
instead of:
#ifdef HAS_FOO
...
#else
...
#endif
A modern compiler such as GCC will generate exactly the same code in both
cases and of course, the former method assumes that HAS_FOO is defined as
either 0 or 1.
- Finally, rules are rules. Sometimes they are sensible and sometimes not
and regardless of your preference, we would like you to follow them.
A project is easier to follow if all project contributors follow the style
rules so that they can all read and understand everyone's code easily. But
remember, like all good rules, they are exceptions where it makes sense not
to be too rigid on the grounds of common sense and consistency!
================================================
FILE: notes/debug.txt
================================================
- strace
strace -o strace.txt -ttT -s 1024 -p `pgrep nutcracker`
- libyaml (yaml-0.2.5)
- yaml tokens:
0 YAML_NO_TOKEN,
1 YAML_STREAM_START_TOKEN,
2 YAML_STREAM_END_TOKEN,
3 YAML_VERSION_DIRECTIVE_TOKEN,
4 YAML_TAG_DIRECTIVE_TOKEN,
5 YAML_DOCUMENT_START_TOKEN,
6 YAML_DOCUMENT_END_TOKEN,
7 YAML_BLOCK_SEQUENCE_START_TOKEN,
8 YAML_BLOCK_MAPPING_START_TOKEN,
9 YAML_BLOCK_END_TOKEN,
10 YAML_FLOW_SEQUENCE_START_TOKEN,
11 YAML_FLOW_SEQUENCE_END_TOKEN,
12 YAML_FLOW_MAPPING_START_TOKEN,
13 YAML_FLOW_MAPPING_END_TOKEN,
14 YAML_BLOCK_ENTRY_TOKEN,
15 YAML_FLOW_ENTRY_TOKEN,
16 YAML_KEY_TOKEN,
17 YAML_VALUE_TOKEN,
18 YAML_ALIAS_TOKEN,
19 YAML_ANCHOR_TOKEN,
20 YAML_TAG_TOKEN,
21 YAML_SCALAR_TOKEN
- yaml events
0 YAML_NO_EVENT,
1 YAML_STREAM_START_EVENT,
2 YAML_STREAM_END_EVENT,
3 YAML_DOCUMENT_START_EVENT,
4 YAML_DOCUMENT_END_EVENT,
5 YAML_ALIAS_EVENT,
6 YAML_SCALAR_EVENT,
7 YAML_SEQUENCE_START_EVENT,
8 YAML_SEQUENCE_END_EVENT,
9 YAML_MAPPING_START_EVENT,
10 YAML_MAPPING_END_EVENT
- sys/queue.h
queue.h is a generic linked list library adapted from BSD. It has three
macro knobs that are useful for debugging:
- QUEUE_MACRO_SCRUB nullifies links (next and prev pointers) of deleted
elements and catches cases where we are attempting to do operations
on an element that has already been unlinked.
- QUEUE_MACRO_TRACE keeps track of __FILE__ and __LINE__ of last two
updates to the list data structure.
- QUEUE_MACRO_ASSERT verifies the sanity of list data structure on every
operation.
- valgrind
valgrind --tool=memcheck --leak-check=yes <program>
- Core dump
ulimit -c unlimited
- Generate ENOMEM to test "Out of Memory"
ulimit -m <size> # limit maximum memory size
ulimit -v <size> # limit virtual memory
- get nutcracker stats
printf "" | socat - TCP:localhost:22222 | tee stats.txt
printf "" | nc localhost 22222 | python -mjson.tool
- Signalling and Logging
SIGTTIN - To up the log level
SIGTTOU - To down the log level
SIGHUP - To reopen log file
- Error codes:
http://www.cs.utah.edu/dept/old/texinfo/glibc-manual-0.02/library_2.html
/usr/include/asm-generic/errno-base.h
/usr/include/asm-generic/errno.h
- epoll (linux)
union epoll_data {
void *ptr;
int fd;
uint32_t u32;
uint64_t u64;
};
struct epoll_event {
uint32_t events; /* epoll events */
struct epoll_data data; /* user data variable */
};
/* events */
EPOLLIN = 0x001,
EPOLLPRI = 0x002,
EPOLLOUT = 0x004,
EPOLLERR = 0x008,
EPOLLHUP = 0x010,
EPOLLRDNORM = 0x040,
EPOLLRDBAND = 0x080,
EPOLLWRNORM = 0x100,
EPOLLWRBAND = 0x200,
EPOLLMSG = 0x400,
EPOLLRDHUP = 0x2000,
EPOLLONESHOT = (1 << 30),
EPOLLET = (1 << 31)
/* opcodes */
EPOLL_CTL_ADD = 1 /* add a file descriptor to the interface */
EPOLL_CTL_DEL = 2 /* remove a file descriptor from the interface */
EPOLL_CTL_MOD = 3 /* change file descriptor epoll_event structure */
- kqueue (bsd)
struct kevent {
uintptr_t ident; /* identifier for this event */
int16_t filter; /* filter for event */
uint16_t flags; /* general flags */
uint32_t fflags; /* filter-specific flags */
intptr_t data; /* filter-specific data */
void *udata; /* opaque user data identifier */
};
/* flags / events */
EV_ADD = 0x0001 /* action - add event to kq (implies enable) */
EV_DELETE = 0x0002 /* action - delete event from kq */
EV_ENABLE = 0x0004 /* action - enable event */
EV_DISABLE = 0x0008 /* action - disable event (not reported) */
EV_RECEIPT = 0x0040 /* action - force EV_ERROR on success, data == 0 */
EV_ONESHOT = 0x0010 /* flags - only report one occurrence */
EV_CLEAR = 0x0020 /* flags - clear event state after reporting */
EV_DISPATCH = 0x0080 /* flags - disable event after reporting */
EV_SYSFLAGS = 0xF000 /* flags - reserved by system */
EV_FLAG0 = 0x1000 /* flags - filter-specific flag */
EV_FLAG1 = 0x2000 /* flags - filter-specific flag */
EV_EOF = 0x8000 /* returned values - EOF detected */
EV_ERROR = 0x4000 /* returned values - error, data contains errno */
/* filters */
EVFILT_READ (-1) /* readable */
EVFILT_WRITE (-2) /* writable */
EVFILT_AIO (-3) /* attached to aio requests */
EVFILT_VNODE (-4) /* attached to vnodes */
EVFILT_PROC (-5) /* attached to struct proc */
EVFILT_SIGNAL (-6) /* attached to struct proc */
EVFILT_TIMER (-7) /* timers */
EVFILT_MACHPORT (-8) /* mach portsets */
EVFILT_FS (-9) /* filesystem events */
EVFILT_USER (-10) /* user events */
EVFILT_VM (-12) /* virtual memory events */
EV_CLEAR behaves like EPOLLET because it resets the event after it is
returned; without this flag, the event would be repeatedly returned.
- poll (unix)
POLLIN 0x001 /* there is data to read */
POLLPRI 0x002 /* there is urgent data to read */
POLLOUT 0x004 /* writing now will not block */
POLLRDNORM 0x040 /* normal data may be read */
POLLRDBAND 0x080 /* priority data may be read */
POLLWRNORM 0x100 /* writing now will not block */
POLLWRBAND 0x200 /* priority data may be written */
POLLMSG 0x400
POLLREMOVE 0x1000
POLLRDHUP 0x2000
POLLERR 0x008 /* error condition */
POLLHUP 0x010 /* hung up */
POLLNVAL 0x020 /* invalid polling request */
- event ports (solaris)
typedef struct port_event {
int portev_events; /* event data is source specific */
ushort_t portev_source; /* event source */
ushort_t portev_pad; /* port internal use */
uintptr_t portev_object; /* source specific object */
void *portev_user; /* user cookie */
} port_event_t;
/* port sources */
PORT_SOURCE_AIO 1
PORT_SOURCE_TIMER 2
PORT_SOURCE_USER 3
PORT_SOURCE_FD 4
PORT_SOURCE_ALERT 5
PORT_SOURCE_MQ 6
PORT_SOURCE_FILE 7
================================================
FILE: notes/memcache.md
================================================
## Memcache Command Support
### Request
- Twemproxy implements only the memached ASCII commands
- Binary commands are currently unsupported
#### Ascii Storage Command
+-------------------+------------+--------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+--------------------------------------------------------------------------+
| set | Yes | set <key> <flags> <expiry> <datalen> [noreply]\r\n<data>\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| add | Yes | add <key> <flags> <expiry> <datalen> [noreply]\r\n<data>\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| replace | Yes | replace <key> <flags> <expiry> <datalen> [noreply]\r\n<data>\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| append | Yes | append <key> <flags> <expiry> <datalen> [noreply]\r\n<data>\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| prepend | Yes | prepend <key> <flags> <expiry> <datalen> [noreply]\r\n<data>\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| cas | Yes | cas <key> <flags> <expiry> <datalen> <cas> [noreply]\r\n<data>\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
* Where,
* <flags> - uint32_t : data specific client side flags
* <expiry> - uint32_t : expiration time (in seconds)
* <datalen> - uint32_t : size of the data (in bytes)
* <data> - uint8_t[]: data block
* <cas> - uint64_t
#### Ascii Retrieval Command
+-------------------+------------+--------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+--------------------------------------------------------------------------+
| get | Yes | get <key> [<key>]+\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| gets | Yes | gets <key> [<key>]+\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
#### Ascii Delete
+-------------------+------------+--------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+--------------------------------------------------------------------------+
| delete | Yes | delete <key> [noreply]\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
#### Ascii Arithmetic Command
+-------------------+------------+--------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+--------------------------------------------------------------------------+
| incr | Yes | incr <key> <value> [noreply]\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| decr | Yes | decr <key> <value> [noreply]\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
* Where,
* <value> - uint64_t
#### Ascii Misc Command
+-------------------+------------+--------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+--------------------------------------------------------------------------+
| touch | Yes | touch <key> <expiry>[noreply]\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| gat | Planned | gat <expiry> <key>+\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| gats | Planned | gats <expiry> <key>+\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| quit | Yes | quit\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| flush_all | No | flush_all [<delay>] [noreply]\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| version | Yes | version\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| verbosity | No | verbosity <num> [noreply]\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| stats | No | stats\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
| stats | No | stats <args>\r\n |
+-------------------+------------+--------------------------------------------------------------------------+
### Response
#### Error Responses
ERROR\r\n
CLIENT_ERROR [error]\r\n
SERVER_ERROR [error]\r\n
Where,
- ERROR means client sent a non-existent command name
- CLIENT_ERROR means that command sent by the client does not conform to the protocol
- SERVER_ERROR means that there was an error on the server side that made processing of the command impossible
#### Storage Command Responses
STORED\r\n
NOT_STORED\r\n
EXISTS\r\n
NOT_FOUND\r\n
Where,
- STORED indicates success.
- NOT_STORED indicates the data was not stored because condition for an add or replace wasn't met.
- EXISTS indicates that the item you are trying to store with a cas has been modified since you last fetched it.
- NOT_FOUND indicates that the item you are trying to store with a cas does not exist.
#### Delete Command Responses
NOT_FOUND\r\n
DELETED\r\n
#### Retrieval Responses
END\r\n
VALUE <key> <flags> <datalen> [<cas>]\r\n<data>\r\nEND\r\n
VALUE <key> <flags> <datalen> [<cas>]\r\n<data>\r\n[VALUE <key> <flags> <datalen> [<cas>]\r\n<data>]+\r\nEND\r\n
#### Arithmetic Responses
NOT_FOUND\r\n
<value>\r\n
Where,
- <value> - uint64_t : new key value after incr or decr operation
#### Touch Command Responses
NOT_FOUND\r\n
TOUCHED\r\n
#### Statistics Response
[STAT <name> <value>\r\n]+END\r\n
#### Misc Responses
OK\r\n
VERSION <version>\r\n
### Notes
- set always creates mapping irrespective of whether it is present on not.
- add, adds only if the mapping is not present
- replace, only replaces if the mapping is present
- append and prepend command ignore flags and expiry values
- noreply instructs the server to not send the reply even if there is an error.
- decr of 0 is 0, while incr of UINT64_MAX is 0
- maximum length of the key is 250 characters
- expiry of 0 means that item never expires, though it could be evicted from the cache
- non-zero expiry is either unix time (# seconds since 01/01/1970) or,
offset in seconds from the current time (< 60 x 60 x 24 x 30 seconds = 30 days)
- expiry time is with respect to the server (not client)
- <datalen> can be zero and when it is, the <data> block is empty.
- Thoughts:
- ascii protocol is easier to debug - think using strace or tcpdump to see
protocol on the wire, Or using telnet or netcat or socat to build memcache
requests and responses
https://stackoverflow.com/questions/2525188/are-binary-protocols-dead
- nutcracker will support the more efficient meta-text protocol after the protocol
is marked as stable and memcached servers using it have had several releases.
- https://news.ycombinator.com/item?id=1712788
================================================
FILE: notes/recommendation.md
================================================
If you are deploying nutcracker in your production environment, here are a few recommendations that might be worth considering.
## Log Level
By default debug logging is disabled in nutcracker. However, it is worthwhile running nutcracker with debug logging enabled and verbosity level set to LOG_INFO (-v 6 or --verbose=6). This in reality does not add much overhead as you only pay the cost of checking an if condition for every log line encountered during the run time.
At LOG_INFO level, nutcracker logs the life cycle of every client and server connection and important events like the server being ejected from the hash ring and so on. Eg.
[Thu Aug 2 00:03:09 2012] nc_proxy.c:336 accepted c 7 on p 6 from '127.0.0.1:54009'
[Thu Aug 2 00:03:09 2012] nc_server.c:528 connected on s 8 to server '127.0.0.1:11211:1'
[Thu Aug 2 00:03:09 2012] nc_core.c:270 req 1 on s 8 timedout
[Thu Aug 2 00:03:09 2012] nc_core.c:207 close s 8 '127.0.0.1:11211' on event 0004 eof 0 done 0 rb 0 sb 20: Connection timed out
[Thu Aug 2 00:03:09 2012] nc_server.c:406 close s 8 schedule error for req 1 len 20 type 5 from c 7: Connection timed out
[Thu Aug 2 00:03:09 2012] nc_server.c:281 update pool 0 'alpha' to delete server '127.0.0.1:11211:1' for next 2 secs
[Thu Aug 2 00:03:10 2012] nc_connection.c:314 recv on sd 7 eof rb 20 sb 35
[Thu Aug 2 00:03:10 2012] nc_request.c:334 c 7 is done
[Thu Aug 2 00:03:10 2012] nc_core.c:207 close c 7 '127.0.0.1:54009' on event 0001 eof 1 done 1 rb 20 sb 35
[Thu Aug 2 00:03:11 2012] nc_proxy.c:336 accepted c 7 on p 6 from '127.0.0.1:54011'
[Thu Aug 2 00:03:11 2012] nc_server.c:528 connected on s 8 to server '127.0.0.1:11212:1'
[Thu Aug 2 00:03:12 2012] nc_connection.c:314 recv on sd 7 eof rb 20 sb 8
[Thu Aug 2 00:03:12 2012] nc_request.c:334 c 7 is done
[Thu Aug 2 00:03:12 2012] nc_core.c:207 close c 7 '127.0.0.1:54011' on event 0001 eof 1 done 1 rb 20 sb 8
To enable debug logging, you have to compile nutcracker with logging enabled using --enable-debug=log configure option.
## Liveness
Failures are a fact of life, especially when things are distributed. To be resilient against failures, it is recommended that you configure the following keys for every server pool. Eg:
resilient_pool:
auto_eject_hosts: true
server_retry_timeout: 30000
server_failure_limit: 3
Enabling `auto_eject_hosts:` ensures that a dead server can be ejected out of the hash ring after `server_failure_limit:` consecutive failures have been encountered on that said server. A non-zero `server_retry_timeout:` ensures that we don't incorrectly mark a server as dead forever especially when the failures were really transient. The combination of `server_retry_timeout:` and `server_failure_limit:` controls the tradeoff between resiliency to permanent and transient failures.
Note that an ejected server will not be included in the hash ring for any requests until the retry timeout passes. This will lead to data partitioning as keys originally on the ejected server will now be written to a server still in the pool.
To ensure that requests always succeed in the face of server ejections (`auto_eject_hosts:` is enabled), some form of retry must be implemented at the client layer since nutcracker itself does not retry a request. This client-side retry count must be greater than `server_failure_limit:` value, which ensures that the original request has a chance to make it to a live server.
## Timeout
It is always a good idea to configure nutcracker `timeout:` for every server pool, rather than purely relying on client-side timeouts. Eg:
resilient_pool_with_timeout:
auto_eject_hosts: true
server_retry_timeout: 30000
server_failure_limit: 3
timeout: 400
Relying only on client-side timeouts has the adverse effect of the original request having timedout on the client to proxy connection, but still pending and outstanding on the proxy to server connection. This further gets exacerbated when client retries the original request.
By default, nutcracker waits indefinitely for any request sent to the server. However, when `timeout:` key is configured, a requests for which no response is received from the server in `timeout:` msec is timedout and an error response `SERVER_ERROR Connection timed out\r\n` (memcached) or `-ERR Connection timed out\r\n` (redis) is sent back to the client.
## Error Response
Whenever a request encounters failure on a server we usually send to the client a response with the general form - `SERVER_ERROR <errno description>\r\n` (memcached) or `-ERR <errno description>` (redis).
For example, when a memcache server is down, this error response is usually:
+ `SERVER_ERROR Connection refused\r\n` or,
+ `SERVER_ERROR Connection reset by peer\r\n`
When the request timedout, the response is usually:
+ `SERVER_ERROR Connection timed out\r\n`
Seeing a `SERVER_ERROR` or `-ERR` response should be considered as a transient failure by a client which makes the original request an ideal candidate for a retry.
## read, writev and mbuf
All memory for incoming requests and outgoing responses is allocated in mbuf. Mbuf enables zero copy for requests and responses flowing through the proxy. By default an mbuf is 16K bytes in size and this value can be tuned between 512 and 16M bytes using -m or --mbuf-size=N argument. Every connection has at least one mbuf allocated to it. This means that the number of concurrent connections nutcracker can support is dependent on the mbuf size. A small mbuf allows us to handle more connections, while a large mbuf allows us to read and write more data to and from kernel socket buffers.
If nutcracker is meant to handle a large number of concurrent client connections, you should set the mbuf size to 512 or 1K bytes.
## How to interpret mbuf-size=N argument?
Every client connection consumes at least one mbuf. To service a request we need two connections (one from client to proxy and another from proxy to server). So we would need two mbufs.
A fragmentable request like 'get foo bar\r\n', which btw gets fragmented to 'get foo\r\n' and 'get bar\r\n' would consume two mbuf for request and two mbuf for response. So a fragmentable request with N fragments needs N * 2 mbufs. The good thing about mbuf is that the memory comes from a reuse pool. Once a mbuf is allocated, it is never freed but just put back into the reuse pool. The bad thing is that once mbuf is allocated it is never freed, since a freed mbuf always goes back to the [reuse pool](https://github.com/twitter/twemproxy/blob/master/src/nc_mbuf.c#L23-L24). This can however be easily fixed if needed by putting a threshold parameter on the reuse pool.
So, if nutcracker is handling say 1K client connections and 100 server connections, it would consume (max(1000, 100) * 2 * mbuf-size) memory for mbuf. If we assume that clients are sending non-pipelined request, then with default mbuf-size of 16K this would in total consume 32M.
Furthermore, if on average every requests has 10 fragments, then the memory consumption would be 320M. Instead of handling 1K client connections, lets say you were handling 10K, then the memory consumption would be 3.2G. Now instead of using a default mbuf-size of 16K, you used 512 bytes, then memory consumption for the same scenario would drop to 1000 * 2 * 512 * 10 = 10M
This is the reason why for 'large number' of connections or for wide multi-get like requests, you want to choose a small value for mbuf-size like 512
## Maximum Key Length
The memcache ascii protocol [specification](notes/memcache.txt) limits the maximum length of the key to 250 characters. The key should not include whitespace, or '\r' or '\n' character. For redis, we have no such limitation. However, nutcracker requires the key to be stored in a contiguous memory region. Since all requests and responses in nutcracker are stored in mbuf, the maximum length of the redis key is limited by the size of the maximum available space for data in mbuf (mbuf_data_size()). This means that if you want your redis instances to handle large keys, you might want to choose large mbuf size set using -m or --mbuf-size=N command-line argument.
## Node Names for Consistent Hashing
The server cluster in twemproxy can either be specified as list strings in format 'host:port:weight' or 'host:port:weight name'.
servers:
- 127.0.0.1:6379:1
- 127.0.0.1:6380:1
- 127.0.0.1:6381:1
- 127.0.0.1:6382:1
Or,
servers:
- 127.0.0.1:6379:1 server1
- 127.0.0.1:6380:1 server2
- 127.0.0.1:6381:1 server3
- 127.0.0.1:6382:1 server4
In the former configuration, keys are mapped **directly** to **'host:port:weight'** triplet and in the latter they are mapped to **node names** which are then mapped to nodes i.e. host:port pair. The latter configuration gives us the freedom to relocate nodes to a different server without disturbing the hash ring and hence makes this configuration ideal when auto_eject_hosts is set to false. See [issue 25](https://github.com/twitter/twemproxy/issues/25) for details.
Note that when using node names for consistent hashing, twemproxy ignores the weight value in the 'host:port:weight name' format string.
## Hash Tags
[Hash Tags](http://antirez.com/post/redis-presharding.html) enables you to use part of the key for calculating the hash. When the hash tag is present, we use part of the key within the tag as the key to be used for consistent hashing. Otherwise, we use the full key as is. Hash tags enable you to map different keys to the same server as long as the part of the key within the tag is the same.
For example, the configuration of server pool _beta_, also shown below, specifies a two character hash_tag string - "{}". This means that keys "user:{user1}:ids" and "user:{user1}:tweets" map to the same server because we compute the hash on "user1". For a key like "user:user1:ids", we use the entire string "user:user1:ids" to compute the hash and it may map to a different server.
beta:
listen: 127.0.0.1:22122
hash: fnv1a_64
hash_tag: "{}"
distribution: ketama
auto_eject_hosts: false
timeout: 400
redis: true
servers:
- 127.0.0.1:6380:1 server1
- 127.0.0.1:6381:1 server2
- 127.0.0.1:6382:1 server3
- 127.0.0.1:6383:1 server4
## Graphing Cache-pool State
When running nutcracker in production, you often would like to know the list of live and ejected servers at any given time. You can easily answer this question, by generating a time series graph of live and/or dead servers that are part of any cache pool. To do this your graphing client must collect the following stats exposed by nutcracker:
- **server_eof** which is incremented when server closes the connection normally which should not happen because we use persistent connections.
- **server_timedout** is incremented when the connection / request to server timedout.
- **server_err** is incremented for any other kinds of errors.
So, on a given server, the cumulative number of times a server is ejected can be computed as:
```c
(server_err + server_timedout + server_eof) / server_failure_limit
```
A diff of the above value between two successive time intervals would generate a nice timeseries graph for ejected servers.
You can also graph the timestamp at which any given server was ejected by graphing `server_ejected_at` stat.
## server_connections: > 1
By design, twemproxy multiplexes several client connections over few server connections. It is important to note that **"read my last write"** constraint doesn't necessarily hold true when twemproxy is configured with `server_connections: > 1`.
To illustrate this, consider a scenario where twemproxy is configured with `server_connections: 2`. If a client makes pipelined requests with the first request in pipeline being `set foo 0 0 3\r\nbar\r\n` (write) and the second request being `get foo\r\n` (read), the expectation is that the read of key `foo` would return the value `bar`. However, with configuration of two server connections it is possible that write and read request are sent on different server connections which would mean that their completion could race with one another. In summary, if the client expects "read my last write" constraint, you either configure twemproxy to use `server_connections:1` or use clients that only make synchronous requests to twemproxy.
================================================
FILE: notes/redis.md
================================================
## Redis Command Support
### Keys Command
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| DEL | Yes | DEL key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| DUMP | Yes | DUMP key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| EXISTS | Yes | EXISTS key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| EXPIRE | Yes | EXPIRE key seconds |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| EXPIREAT | Yes | EXPIREAT key timestamp |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| KEYS | No | KEYS pattern |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| MIGRATE | No | MIGRATE host port key destination-db timeout |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| MOVE | No | MOVE key db |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| OBJECT | No | OBJECT subcommand [arguments [arguments …]] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PERSIST | Yes | PERSIST key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PEXPIRE | Yes | PEXPIRE key milliseconds |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PEXPIREAT | Yes | PEXPIREAT key milliseconds-timestamp |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PTTL | Yes | PTTL key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| RANDOMKEY | No | RANDOMKEY |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| RENAME | No | RENAME key newkey |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| RENAMENX | No | RENAMENX key newkey |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| RESTORE | Yes | RESTORE key ttl serialized-value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SORT | Yes | SORT key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] [STORE destination] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| TTL | Yes | TTL key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| TYPE | Yes | TYPE key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SCAN | No | SCAN cursor [MATCH pattern] [COUNT count] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
### Strings Command
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| APPEND | Yes | APPEND key value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BITCOUNT | Yes | BITCOUNT key [start end] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BITFIELD | Yes | BITFIELD key [GET] [SET] [INCRBY] [WRAP|SAT|FAIL] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BITOP | No | BITOP operation destkey key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BITPOS | Yes | BITPOS key bit [start] [end] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| DECR | Yes | DECR key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| DECRBY | Yes | DECRBY key decrement |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GET | Yes | GET key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GETBIT | Yes | GETBIT key offset |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GETDEL | Yes | GETDEL key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GETEX | Yes | GETEX key [EX seconds|PX milliseconds|EXAT timestamp|PXAT milliseconds-timestamp|PERSIST] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GETRANGE | Yes | GETRANGE key start end |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GETSET | Yes | GETSET key value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| INCR | Yes | INCR key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| INCRBY | Yes | INCRBY key increment |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| INCRBYFLOAT | Yes | INCRBYFLOAT key increment |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| MGET | Yes | MGET key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| MSET | Yes | MSET key value [key value …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| MSETNX | No | MSETNX key value [key value …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PSETEX | Yes | PSETEX key milliseconds value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SET | Yes | SET key value [EX seconds|PX milliseconds|EXAT timestamp|PXAT milliseconds-timestamp|KEEPTTL] [NX|XX] [GET] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SETBIT | Yes | SETBIT key offset value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SETEX | Yes | SETEX key seconds value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SETNX | Yes | SETNX key value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SETRANGE | Yes | SETRANGE key offset value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| STRALGO | No | STRALGO LCS algo-specific-argument [algo-specific-argument …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| STRLEN | Yes | STRLEN key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
* MSET support is not Atomic
### Hashes
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HDEL | Yes | HDEL key field [field ...] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HEXISTS | Yes | HEXISTS key field |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HGET | Yes | HGET key field |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HGETALL | Yes | HGETALL key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HINCRBY | Yes | HINCRBY key field increment |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HINCRBYFLOAT | Yes | HINCRBYFLOAT key field increment |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HKEYS | Yes | HKEYS key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HLEN | Yes | HLEN key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HMGET | Yes | HMGET key field [field ...] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HMSET | Yes | HMSET key field value [field value ...] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HSET | Yes | HSET key field value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HSETNX | Yes | HSETNX key field value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HVALS | Yes | HVALS key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| HSCAN | Yes | HSCAN key cursor [MATCH pattern] [COUNT count] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
### Lists
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BLMOVE | No | BLMOVE source destination LEFT|RIGHT LEFT|RIGHT timeout |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BLPOP | No | BLPOP key [key …] timeout |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BRPOP | No | BRPOP key [key …] timeout |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BRPOPLPUSH | No | BRPOPLPUSH source destination timeout |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LINDEX | Yes | LINDEX key index |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LINSERT | Yes | LINSERT key BEFORE|AFTER pivot element |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LLEN | Yes | LLEN key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LMOVE | Yes | LMOVE source destination LEFT|RIGHT LEFT|RIGHT |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LPOP | Yes | LPOP key [count] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LPOS | Yes | LPOS key element [RANK] [COUNT] [MAXLEN] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LPUSH | Yes | LPUSH key element [element …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LPUSHX | Yes | LPUSHX key element [element …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LRANGE | Yes | LRANGE key start stop |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LREM | Yes | LREM key count element |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LSET | Yes | LSET key index element |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LTRIM | Yes | LTRIM key start stop |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| RPOP | Yes | RPOP key [count] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| RPOPLPUSH | Yes* | RPOPLPUSH source destination |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| RPUSH | Yes | RPUSH key element [element …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| RPUSHX | Yes | RPUSHX key element [element …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
* RPOPLPUSH support requires that source and destination keys hash to the same server. You can ensure this by using the same [hashtag](recommendation.md#hash-tags) for source and destination key. Twemproxy does no checking on its end to verify that source and destination key hash to the same server, and the RPOPLPUSH command is forwarded to the server that the source key hashes to
### Sets
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SADD | Yes | SADD key member [member …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SCARD | Yes | SCARD key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SDIFF | Yes* | SDIFF key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SDIFFSTORE | Yes* | SDIFFSTORE destination key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SINTER | Yes* | SINTER key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SINTERSTORE | Yes* | SINTERSTORE destination key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SISMEMBER | Yes | SISMEMBER key member |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SMEMBERS | Yes | SMEMBERS key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SMISMEMBER | Yes | SMISMEMBER key member [member …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SMOVE | Yes* | SMOVE source destination member |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SPOP | Yes | SPOP key [count] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SRANDMEMBER | Yes | SRANDMEMBER key [count] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SREM | Yes | SREM key member [member …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SSCAN | Yes | SSCAN key cursor [MATCH] [COUNT] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SUNION | Yes* | SUNION key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SUNIONSTORE | Yes* | SUNIONSTORE destination key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
* SIDFF, SDIFFSTORE, SINTER, SINTERSTORE, SMOVE, SUNION and SUNIONSTORE support requires that the supplied keys hash to the same server. You can ensure this by using the same [hashtag](recommendation.md#hash-tags) for all keys in the command. Twemproxy does no checking on its end to verify that all the keys hash to the same server, and the given command is forwarded to the server that the first key hashes to.
### Sorted Sets
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BZPOPMAX | No | BZPOPMAX key [key …] timeout |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BZPOPMIN | No | BZPOPMIN key [key …] timeout |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZADD | Yes | ZADD key [NX|XX] [GT|LT] [CH] [INCR] score member [score member …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZCARD | Yes | ZCARD key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZCOUNT | Yes | ZCOUNT key min max |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZDIFF | Yes* | ZDIFF numkeys key [key …] [WITHSCORES] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZDIFFSTORE | Yes* | ZDIFFSTORE destination numkeys key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZINCRBY | Yes | ZINCRBY key increment member |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZINTER | Yes* | ZINTER numkeys key [key …] [WEIGHTS] [SUM|MIN|MAX] [WITHSCORES] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZINTERSTORE | Yes* | ZINTERSTORE destination numkeys key [key …] [WEIGHTS] [SUM|MIN|MAX] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZLEXCOUNT | Yes | ZLEXCOUNT key min max |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZMSCORE | Yes | ZMSCORE key member [member …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZPOPMAX | Yes | ZPOPMAX key [count] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZPOPMIN | Yes | ZPOPMIN key [count] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZRANDMEMBER | Yes | ZRANDMEMBER key [options] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZRANGE | Yes | ZRANGE key min max [BYSCORE|BYLEX] [REV] [LIMIT] [WITHSCORES] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZRANGEBYLEX | Yes | ZRANGEBYLEX key min max [LIMIT] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZRANGEBYSCORE | Yes | ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZRANGESTORE | Yes | ZRANGESTORE dst src min max [BYSCORE|BYLEX] [REV] [LIMIT] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZRANK | Yes | ZRANK key member |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZREM | Yes | ZREM key member [member …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZREMRANGEBYLEX | Yes | ZREMRANGEBYLEX key min max |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZREMRANGEBYRANK | Yes | ZREMRANGEBYRANK key start stop |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZREMRANGEBYSCORE | Yes | ZREMRANGEBYSCORE key min max |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZREVRANGE | Yes | ZREVRANGE key start stop [WITHSCORES] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZREVRANGEBYLEX | Yes | ZREVRANGEBYLEX key max min [LIMIT] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZREVRANGEBYSCORE | Yes | ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZREVRANK | Yes | ZREVRANK key member |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZSCAN | Yes | ZSCAN key cursor [MATCH] [COUNT] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZSCORE | Yes | ZSCORE key member |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZUNION | Yes* | ZUNION numkeys key [key …] [WEIGHTS] [SUM|MIN|MAX] [WITHSCORES] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ZUNIONSTORE | Yes* | ZUNIONSTORE destination numkeys key [key …] [WEIGHTS] [SUM|MIN|MAX] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
* ZINTERSTORE and ZUNIONSTORE support requires that the supplied keys hash to the same server. You can ensure this by using the same [hashtag](recommendation.md#hash-tags) for all keys in the command. Twemproxy does no checking on its end to verify that all the keys hash to the same server, and the given command is forwarded to the server that the first key hashes to.
### HyperLogLog
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PFADD | Yes | PFADD key element [element …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PFCOUNT | Yes | PFCOUNT key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PFMERGE | Yes* | PFMERGE destkey sourcekey [sourcekey …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
* PFMERGE support requires that the supplied keys hash to the same server. You can ensure this by using the same [hashtag](recommendation.md#hash-tags) for all keys in the command. Twemproxy does no checking on its end to verify that all the keys hash to the same server, and the given command is forwarded to the server that the first key hashes to.
### Geo Command
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GEOADD | Yes | GEOADD key [NX|XX] [CH] longitude latitude member [longitude latitude member …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GEODIST | Yes | GEODIST key member1 member2 [m|km|ft|mi] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GEOHASH | Yes | GEOHASH key member [member …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GEOPOS | Yes | GEOPOS key member [member …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GEORADIUS | Yes | GEORADIUS key longitude latitude radius m|km|ft|mi [WITHCOORD] [WITHDIST] [WITHHASH] [count] [ASC|DESC] [STORE] [STOREDIST]|
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GEORADIUSBYMEMBER | Yes | GEORADIUSBYMEMBER key member radius m|km|ft|mi [WITHCOORD] [WITHDIST] [WITHHASH] [count] [ASC|DESC] [STORE] [STOREDIST]|
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GEOSEARCH | Yes | GEOSEARCH key [FROMMEMBER] [FROMLONLAT] [circle] [box] [ASC|DESC] [count] [WITHCOORD] [WITHDIST] [WITHHASH] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| GEOSEARCHSTORE | Yes | GEOSEARCHSTORE destination source [FROMMEMBER] [FROMLONLAT] [circle] [box] [ASC|DESC] [count] [WITHCOORD] [WITHDIST] [WITHHASH] [STOREDIST]|
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
### Pub/Sub
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PSUBSCRIBE | No | PSUBSCRIBE pattern [pattern …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PUBLISH | No | PUBLISH channel message |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PUBSUB | No | PUBSUB subcommand [argument [argument …]] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PUNSUBSCRIBE | No | PUNSUBSCRIBE [pattern [pattern …]] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SUBSCRIBE | No | SUBSCRIBE channel [channel …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| UNSUBSCRIBE | No | UNSUBSCRIBE [channel [channel …]] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
### Transactions
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| DISCARD | No | DISCARD |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| EXEC | No | EXEC |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| MULTI | No | MULTI |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| UNWATCH | No | UNWATCH |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| WATCH | No | WATCH key [key …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
### Scripting
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| EVAL | Yes* | EVAL script numkeys key [key …] arg [arg …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| EVALSHA | Yes* | EVALSHA sha1 numkeys key [key …] arg [arg …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SCRIPT DEBUG | No | SCRIPT DEBUG YES|SYNC|NO |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SCRIPT EXISTS | No | SCRIPT EXISTS sha1 [sha1 …] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SCRIPT FLUSH | No | SCRIPT FLUSH [ASYNC|SYNC] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SCRIPT KILL | No | SCRIPT KILL |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SCRIPT LOAD | No | SCRIPT LOAD script |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
* EVAL and EVALSHA support is limited to scripts that take at least 1 key. If multiple keys are used, all keys must hash to the same server. You can ensure this by using the same [hashtag](recommendation.md#hash-tags) for all keys. If you use more than 1 key, the proxy does no checking to verify that all keys hash to the same server, and the entire command is forwarded to the server that the first key hashes to
### Connection
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| AUTH | No | AUTH password |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| ECHO | No | ECHO message |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| PING | Yes | PING |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| QUIT | Yes | QUIT |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SELECT | No | SELECT index |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
### Server
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BGREWRITEAOF | No | BGREWRITEAOF |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| BGSAVE | No | BGSAVE |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| CLIENT KILL | No | CLIENT KILL ip:port |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| CLIENT LIST | No | CLIENT LIST |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| CONFIG GET | No | CONFIG GET parameter |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| CONFIG SET | No | CONFIG SET parameter value |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| CONFIG RESETSTAT | No | CONFIG RESETSTAT |
+-------------------+-------------+--------------------------------------------------------------------------------------------------------------------+
| DBSIZE | No | DBSIZE |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| DEBUG OBJECT | No | DEBUG OBJECT key |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| DEBUG SEGFAULT | No | DEBUG SEGFAULT |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| FLUSHALL | No | FLUSHALL |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| FLUSHDB | No | FLUSHDB |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| INFO | No | INFO |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LASTSAVE | No | LASTSAVE |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| MONITOR | No | MONITOR |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SAVE | No | SAVE |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SHUTDOWN | No | SHUTDOWN [NOSAVE] [SAVE] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SLAVEOF | No | SLAVEOF host port |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SLOWLOG | No | SLOWLOG subcommand [argument] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| SYNC | No | SYNC |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| TIME | No | TIME |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| COMMAND | Yes | COMMAND |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| LOLWUT | Yes | LOLWUT [ argument ...] |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
* COMMAND is forwarded to an arbitrarily chosen redis backend for the benefit of tools such as `redis-cli`.
It assumes that all proxied redis servers support the same backend.
There is no post-processing; Commands that are not supported by twemproxy are not filtered out.
* LOLWUT is also forwarded to an arbitrarily chosen redis backend because it does not depend on the state of the database and the presence of the command would not affect production applications.
It can be used as a reference for adding future commands with no keys and a variable-length argument list.
Other commands such as `TIME` continue to be unsupported. They may vary based on state of the database (e.g. out of sync clocks). Using EVAL with a key known to be on the specific backend server can be done instead.
## Note
- redis commands are not case sensitive
- only vectored commands 'MGET key [key ...]', 'MSET key value [key value ...]', 'DEL key [key ...]', 'UNLINK key [key ...]', 'EXISTS key [key ...]' needs to be fragmented
## Performance
### Setup
+ redis-server running on machine A.
+ nutcracker running on machine A as a local proxy to redis-server.
+ redis-benchmark running on machine B.
+ machine A != machine B.
+ nutcracker built with --enable-debug=no
+ nutcracker running with mbuf-size of 512 (-m 512)
+ redis-server built from redis 2.6 branch
### redis-benchmark against redis-server
$ redis-benchmark -h <machine-A> -q -t set,get,incr,lpush,lpop,sadd,spop,lpush,lrange -c 100 -p 6379
SET: 89285.71 requests per second
GET: 92592.59 requests per second
INCR: 89285.71 requests per second
LPUSH: 90090.09 requests per second
LPOP: 90090.09 requests per second
SADD: 90090.09 requests per second
SPOP: 93457.95 requests per second
LPUSH (needed to benchmark LRANGE): 89285.71 requests per second
LRANGE_100 (first 100 elements): 36496.35 requests per second
LRANGE_300 (first 300 elements): 15748.03 requests per second
LRANGE_500 (first 450 elements): 11135.86 requests per second
LRANGE_600 (first 600 elements): 8650.52 requests per second
### redis-benchmark against nutcracker proxing redis-server
$ redis-benchmark -h <machine-A> -q -t set,get,incr,lpush,lpop,sadd,spop,lpush,lrange -c 100 -p 22121
SET: 85470.09 requests per second
GET: 86956.52 requests per second
INCR: 85470.09 requests per second
LPUSH: 84745.77 requests per second
LPOP: 86206.90 requests per second
SADD: 84745.77 requests per second
SPOP: 86956.52 requests per second
LPUSH (needed to benchmark LRANGE): 84745.77 requests per second
LRANGE_100 (first 100 elements): 29761.90 requests per second
LRANGE_300 (first 300 elements): 12376.24 requests per second
LRANGE_500 (first 450 elements): 8605.85 requests per second
LRANGE_600 (first 600 elements): 6587.62 requests per second
## redis-auth feature
+ you can enable redis-auth for a pool with 'redis_auth':
alpha:
listen: 127.0.0.1:22121
hash: fnv1a_64
distribution: ketama
redis: true
redis_auth: testpass
+ notice:
+ *MUST* set all redis with a same passwd, and all twemproxy with the same passwd
+ Length of password should be less than 256
## redis-sentinel feature
+ You can configure sentinel for a pool with 'sentinels' to let twemproxy works with sentinel:
sigma:
listen: 127.0.0.1:22125
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: false
redis: true
server_retry_timeout: 2000
server_failure_limit: 1
servers:
- 127.0.0.1:6379:1 server1
- 127.0.0.1:6380:1 server2
sentinels:
- 127.0.0.1:26379:1
- 127.0.0.1:26380:1
- 127.0.0.1:26381:1
+ notice:
+ You should configure all the sentinels you used. Twemproxy will connect to the alive sentinels when some are down
+ Weight of sentinel is not used. Twemproxy keeps it because of the server load code reuse
See [sentinel.md](./sentinel.md) for more details.
================================================
FILE: notes/socket.txt
================================================
- int listen(int sockfd, int backlog);
Linux: The backlog argument defines the maximum length to which the
queue of pending connections for sockfd may grow. If a connection
request arrives when the queue is full, the client may receive an error
with an indication of ECONNREFUSED or, if the underlying protocol
supports retransmission, the request may be ignored so that a later
reattempt at connection succeeds.
backlog specifies the queue length for completely established sockets
waiting to be accepted, instead of the number of incomplete connection
requests. The maximum length of the queue for incomplete sockets can
be set using /proc/sys/net/ipv4/tcp_max_syn_backlog.
If the backlog argument is greater than the value in /proc/sys/net/core/somaxconn,
then it is silently truncated to that value; the default value in this
file is 128. In kernels before 2.4.25, this limit was a hard coded value,
SOMAXCONN, with the value 128.
BSD: The backlog argument defines the maximum length the queue of pending
connections may grow to. The real maximum queue length will be 1.5 times
more than the value specified in the backlog argument. A subsequent
listen() system call on the listening socket allows the caller to change
the maximum queue length using a new backlog argument. If a connection
request arrives with the queue full the client may receive an error with
an indication of ECONNREFUSED, or, in the case of TCP, the connection
will be silently dropped.
The listen() system call appeared in 4.2BSD. The ability to configure
the maximum backlog at run-time, and to use a negative backlog to request
the maximum allowable value, was introduced in FreeBSD 2.2.
- SO_LINGER (linger) socket option
This option specifies what should happen when the socket of a type that
promises reliable delivery still has untransmitted messages when it is
closed
struct linger {
int l_onoff; /* nonzero to linger on close */
int l_linger; /* time to linger (in secs) */
};
l_onoff = 0 (default), then l_linger value is ignored and close returns
immediately. But if there is any data still remaining in the socket send
buffer, the system will try to deliver the data to the peer
l_onoff = nonzero, then close blocks until data is transmitted or the
l_linger timeout period expires
a) l_linger = 0, TCP aborts connection, discards any data still remaining
in the socket send buffer and sends RST to peer. This avoids the
TCP's TIME_WAIT state
b) l_linger = nonzero, then kernel will linger when socket is closed. If
there is any pending data in the socket send buffer, the kernel waits
until all the data is sent and acknowledged by peer TCP, or the
linger time expires
If a socket is set as nonblocking, it will not wait for close to complete
even if linger time is nonzero
- TIME_WAIT state
The end that performs active close i.e. the end that sends the first FIN
goes into TIME_WAIT state. After a FIN packet is sent to the peer and
after that peers FIN/ACK arrvies and is ACKed, we go into a TIME_WAIT
state. The duration that the end point remains in this state is 2 x MSL
(maximum segment lifetime). The reason that the duration of the TIME_WAIT
state is 2 x MSL is because the maximum amount of time a packet can wander
around a network is assumed to be MSL seconds. The factor of 2 is for the
round-trip. The recommended value for MSL is 120 seconds, but Berkeley
derived implementations normally use 30 seconds instead. This means a
TIME_WAIT delay is between 1 and 4 minutes.
For Linux, the TIME_WAIT state duration is 1 minute (net/tcp.h):
#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
* state, about 60 seconds */
TIME_WAIT state on client, combined with limited number of ephermeral ports
available for TCP connections severely limits the rate at which new
connections to the server can be created. On Linux, by default ephemeral
ports are in the range of 32768 to 61000:
$ cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
So with a TIME_WAIT state duration of 1 minute, the maximum sustained rate
for any client is ~470 new connections per second
- TCP keepalive
TCP keepalive packet (TCP packet with no data and the ACK flag turned on)
is used to assert that connection is still up and running. This is useful
because if the remote peer goes away without closing their connection, the
keepalive probe will detect this and notice that the connection is broken
even if there is no traffic on it.
Imagine, the following scenario: You have a valid TCP connection established
between two endpoints A and B. B terminates abnormally (think kernel panic
or unplugging of network cable) without sending anything over the network
to notify A that connection is broken. A, from its side, is ready to
receive data, and has no idea that B has gone away. Now B comes back up
again, and while A knows about a connection with B and still thinks that it
active, B has no such idea. A tries to send data to B over a dead
connection, and B replies with an RST packet, causing A to finally close
the connection. So, without a keepalive probe A would never close the
connection if it never sent data over it.
- There are four socket functions that pass a socket address structure from
the process to the kernel - bind, connect, sendmsg and sendto. These
function are also responsible for passing the length of the sockaddr that
they are passing (socklen_t).
There are five socket functions that pass a socket from the kernel to the
process - accept, recvfrom, recvmsg, getpeername, getsockname. The kernel
is also responsible for returning the length of the sockaddr struct that
it returns back to the userspace
Different sockaddr structs:
1. sockaddr_in
2. sockaddr_in6
3. sockaddr_un
Special types of in_addr_t
/* Address to accept any incoming messages */
#define INADDR_ANY ((in_addr_t) 0x00000000)
/* Address to send to all hosts */
#define INADDR_BROADCAST ((in_addr_t) 0xffffffff)
/* Address indicating an error return */
#define INADDR_NONE ((in_addr_t) 0xffffffff)
================================================
FILE: scripts/benchmark-mget.py
================================================
#!/usr/bin/env python
#coding: utf-8
#file : test_mget.py
#author : ning
#date : 2014-04-01 13:15:48
import os
import re
import commands
ports = [
4001, # before improve
4000, # after improve
2000 # redis
]
def system(cmd):
return commands.getoutput(cmd)
def extra(regex, text):
match = re.search(regex, text, re.DOTALL)
if match:
return match.group(1)
def testit():
for mget_size in [10, 100, 1000, 10000]:
for port in ports:
cnt = 100*1000 / mget_size
clients = 50
if mget_size == 10000:
clients = 2
cmd = 'cd /home/ning/xredis/deploy-srcs/redis-2.8.3/src && ./redis-benchmark.%d -n %d -p %d -t mget -r 1000000000 -c %d' % (mget_size, cnt, port, clients)
#print cmd
rst = system(cmd)
#100.00% <= 2 milliseconds
#28089.89 requests per second
rtime = extra('100.00% <= (\d+) milliseconds', rst)
qps = extra('([\.\d]+) requests per second', rst)
print 'mget_size=%d on %d: pqs: %s, rtime: %s' % (mget_size, port, qps, rtime)
testit()
================================================
FILE: scripts/extract_redis_commands_argcounts.php
================================================
#!/usr/bin/env php
<?php
/**
* @author Tyson Andre
*
* Heuristics to extract commands from redis-doc and determine what group of commands
* they'd fall under for twemproxy's request parsing logic.
*/
if (count($argv) !== 2) {
echo "Usage: ${argv[0]} commands.json\n";
echo "commands.json can be downloaded from https://github.com/redis/redis-doc\n";
exit(1);
}
$path = $argv[1];
$contents = file_get_contents($path);
if (!is_string($contents)) {
echo "Failed to read $path\n";
exit(1);
}
$commands = json_decode($contents, true);
uasort($commands, fn($a, $b) => version_compare($b['since'], $a['since']));
const INFINITE_ARGS = 100000;
function categorize_arg(array $arg, string $commandName): array {
$min = 1;
$max = 1;
if ($arg['multiple']) {
$min = 0;
$max = INFINITE_ARGS;
}
if ($arg['optional']) {
$min = 0;
}
if ($arg['type'] === 'key') {
return ['min_key' => $min, 'max_key' => $max];
}
return ['min_arg' => $min, 'max_arg' => $max];
}
function categorize(array $command, string $commandName): string {
$minKeyCount = 0;
$maxKeyCount = 0;
$minArgCount = 0;
$maxArgCount = 0;
$arguments = $command['arguments'] ?? [];
foreach ($arguments as $arg) {
$data = categorize_arg($arg, $commandName);
$minKeyCount += ($data['min_key'] ?? 0);
$maxKeyCount += ($data['max_key'] ?? 0);
$minArgCount += ($data['min_arg'] ?? 0);
$maxArgCount += ($data['max_arg'] ?? 0);
}
if (in_array($commandName, ['DEL', 'MGET', 'MSET', 'TOUCH', 'UNLINK'])) {
return "keyn";
}
if ($maxKeyCount > $minKeyCount || $maxArgCount > $minArgCount) {
// return "key${minKeyCount}_argx";
return "key1_argx";
}
// Assume that
// min=max for arg and key
if ($minArgCount > 0 && $minKeyCount >= 2) {
return "key1_arg" . ($minArgCount + $maxKeyCount - 1);
}
return "key${minKeyCount}_arg" . $minArgCount;
}
const KEY1 = [
'PERSIST',
'PTTL',
'TTL',
'TYPE',
'DUMP',
'DECR',
'GET',
'GETDEL',
'INCR',
'STRLEN',
'HGETALL',
'HKEYS',
'HLEN',
'HVALS',
'LLEN',
'SCARD',
'SMEMBERS',
'ZCARD',
// 'AUTH',
];
const KEY1_ARG1 = [
'EXPIRE',
'EXPIREAT',
'PEXPIRE',
'PEXPIREAT',
'MOVE',
'APPEND',
'DECRBY',
'GETBIT',
'GETSET',
'INCRBY',
'INCRBYFLOAT',
'SETNX',
'HEXISTS',
'HGET',
'HSTRLEN',
'LINDEX',
'RPOPLPUSH',
'SISMEMBER',
'ZRANK',
'ZREVRANK',
'ZSCORE',
];
const KEY1_ARG2 = [
'GETRANGE',
'PSETEX',
'SETBIT',
'SETEX',
'SETRANGE',
'HINCRBY',
'HINCRBYFLOAT',
'HSETNX',
'LRANGE',
'LREM',
'LSET',
'LTRIM',
'SMOVE',
'ZCOUNT',
'ZLEXCOUNT',
'ZINCRBY',
'ZREMRANGEBYLEX',
'ZREMRANGEBYRANK',
'ZREMRANGEBYSCORE',
];
const KEY1_ARG3 = [
'LINSERT',
'LMOVE',
];
const KEY1_ARGN = [
'SORT',
'BITCOUNT',
'BITPOS',
'BITFIELD',
'BITOP',
'EXISTS',
'GETEX',
'SET',
'HDEL',
'HMGET',
'HMSET',
'HSCAN',
'HSET',
'HRANDFIELD',
'LPUSH',
'LPUSHX',
'RPUSH',
'RPUSHX',
'LPOP',
'RPOP',
'LPOS',
'SADD',
'SDIFF',
'SDIFFSTORE',
'SINTER',
'SINTERSTORE',
'SREM',
'SUNION',
'SUNIONSTORE',
'SRANDMEMBER',
'SSCAN',
'SPOP',
'SMISMEMBER',
'PFADD',
'PFMERGE',
'PFCOUNT',
'ZADD',
'ZDIFF',
'ZDIFFSTORE',
'ZINTER',
'ZINTERSTORE',
'ZMSCORE',
'ZPOPMAX',
'ZPOPMIN',
'ZRANDMEMBER',
'ZRANGE',
'ZRANGEBYLEX',
'ZRANGEBYSCORE',
'ZRANGESTORE',
'ZREM',
'ZREVRANGE',
'ZREVRANGEBYLEX',
'ZREVRANGEBYSCORE',
'ZSCAN',
'ZUNION',
'ZUNIONSTORE',
'GEODIST',
'GEOPOS',
'GEOHASH',
'GEOADD',
'GEOSEARCH',
'RESTORE',
];
const EXPECTED_MAPS = [
'key1_arg0' => KEY1,
'key1_arg1' => KEY1_ARG1,
'key1_arg2' => KEY1_ARG2,
'key1_arg3' => KEY1_ARG3,
'key1_argx' => KEY1_ARGN,
];
function compute_types(): array {
global $commands;
$types = [];
foreach ($commands as $name => $cmd) {
// printf("%s: %s\n", $name, json_encode($cmd, JSON_PRETTY_PRINT));
try {
$type = categorize($cmd, $name);
} catch (Exception $e) {
$type = "unknown: {$e->getMessage()} " . json_encode($cmd);
}
$types[$name] = $type;
}
return $types;
}
function dump_mismatched_argument_types(array $types, array $commands): void {
foreach (EXPECTED_MAPS as $expected => $maps) {
foreach ($maps as $key) {
$actual = $types[$key];
if ($actual !== $expected) {
echo "Unexpected type for $key: got $actual, want $expected: " . json_encode($commands[$key]['arguments']) . "\n";
}
}
foreach ($types as $other_name => $type) {
if ($type === $expected && !in_array($other_name, $maps)) {
$command = $commands[$other_name];
echo "Expected $other_name in $expected: " . json_encode($command['arguments']) . "\n";
echo "> " . $command['group'] . ": " . $command['summary'] . "\n\n";
}
}
}
}
function render_arg(array $argument): string {
if ($argument['optional'] ?? false) {
unset($argument['optional']);
return '[' . render_arg($argument) . ']';
}
if ($argument['enum'] ?? null) {
return implode('|', $argument['enum']);
}
if ($argument['command'] ?? null) {
return $argument['command'];
}
$name = $argument['name'];
$repr = is_array($name) ? implode(' ', $name) : $name;
if ($argument['multiple'] ?? false) {
return "$repr [$repr …]";
}
return $repr;
}
function render_command(string $name, array $command): string {
$repr = $name;
foreach ($command['arguments'] ?? [] as $argument) {
$repr .= ' ' . render_arg($argument);
}
return $repr;
}
function center_pad(string $name, int $len) {
if (mb_strlen($name) >= $len) {
return $name;
}
$name = str_repeat(' ', ($len - mb_strlen($name)) >> 1) . $name;
$name .= str_repeat(' ', $len - mb_strlen($name));
return $name;
}
function right_pad(string $name, int $len) {
if (mb_strlen($name) >= $len) {
return $name;
}
$name .= str_repeat(' ', $len - mb_strlen($name));
return $name;
}
function dump_table(array $commands) {
$header = <<<EOT
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
| Command | Supported? | Format |
+-------------------+------------+---------------------------------------------------------------------------------------------------------------------+
EOT;
echo $header . "\n";
$rowLine = explode("\n", $header)[0];
ksort($commands);
$parts = explode('+', $rowLine);
$nameLen = strlen($parts[1]);
$supportsLen = strlen($parts[2]);
$commandLen = strlen($parts[3]);
foreach ($commands as $name => $command) {
$key = center_pad($name, 19);
$commandRepr = render_command($name, $command);
$supports = 'Yes';
printf(" |%s|%s|%s|\n", center_pad($name, $nameLen), center_pad($supports, $supportsLen), right_pad(' ' . $commandRepr, $commandLen));
echo $rowLine . "\n";
}
echo "\n";
}
function dump_table_groups(array $commands): void {
$groups = [];
foreach ($commands as $name => $command) {
$groups[$command['group']][$name] = $command;
}
foreach ($groups as $groupName => $group) {
printf("### %s Command\n\n", $groupName);
dump_table($group);
}
}
$types = compute_types();
foreach ($types as $name => $type) {
printf("%s: %s\n", $name, $type);
}
dump_mismatched_argument_types($types, $commands);
dump_table_groups($commands);
================================================
FILE: scripts/makesrpm.sh
================================================
#!/bin/bash
#-------------------------------------------------------------------------------
# Create a source RPM package
# Author: Elvin Sindrilaru <esindril@cern.ch> 2015
#-------------------------------------------------------------------------------
RCEXP='^[0-9]+\.[0-9]+\.[0-9]+\-rc.*$'
#-------------------------------------------------------------------------------
# Color console messages
#-------------------------------------------------------------------------------
NORMAL=$(tput sgr0)
GREEN=$(tput setaf 2; tput bold)
RED=$(tput setaf 1)
function green() {
echo -e "$GREEN$*$NORMAL"
}
function red() {
echo -e "$RED$*$NORMAL"
}
#-------------------------------------------------------------------------------
# Find a program
#-------------------------------------------------------------------------------
function findProg()
{
for prog in $@; do
if test -x "`which $prog 2>/dev/null`"; then
echo $prog
break
fi
done
}
#-------------------------------------------------------------------------------
# Print help
#-------------------------------------------------------------------------------
function printHelp()
{
echo "Usage:" 1>&2
echo "${0} [--help] [--source PATH] [--output PATH]" 1>&2
echo " --help prints this message" 1>&2
echo " --source PATH specify the root of the source tree" 1>&2
echo " defaults to ../" 1>&2
echo " --output PATH the directory where the source rpm" 1>&2
echo " should be stored, defaults to ." 1>&2
}
#-------------------------------------------------------------------------------
# Parse the commandline
#-------------------------------------------------------------------------------
SOURCEPATH="../"
OUTPUTPATH="."
PRINTHELP=0
while test ${#} -ne 0; do
if test x${1} = x--help; then
PRINTHELP=1
elif test x${1} = x--source; then
if test ${#} -lt 2; then
echo "--source parameter needs an argument" 1>&2
exit 1
fi
SOURCEPATH=${2}
shift
elif test x${1} = x--output; then
if test ${#} -lt 2; then
echo "--output parameter needs an argument" 1>&2
exit 1
fi
OUTPUTPATH=${2}
shift
else
PRINTHELP=1
fi
shift
done
if test $PRINTHELP -eq 1; then
printHelp
exit 0
fi
echo "[i] Working on: $SOURCEPATH"
echo "[i] Storing the output to: $OUTPUTPATH"
#-------------------------------------------------------------------------------
# Check if the source and the output dirs exist
#-------------------------------------------------------------------------------
if test ! -d $SOURCEPATH -o ! -r $SOURCEPATH; then
red "[!] Source path does not exist or is not readable" 1>&2
exit 2
fi
if test ! -d $OUTPUTPATH -o ! -w $OUTPUTPATH; then
red "[!] Output path does not exist or is not writeable" 1>&2
exit 2
fi
#-------------------------------------------------------------------------------
# Check if we have all the necassary components
#-------------------------------------------------------------------------------
if test x`findProg rpmbuild` = x; then
red "[!] Unable to find rp
gitextract_s05hoke1/
├── .clang-format.example
├── .dockerignore
├── .editorconfig
├── .github/
│ ├── ISSUE_TEMPLATE.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ └── workflows/
│ └── main.yml
├── .gitignore
├── ChangeLog
├── Dockerfile
├── LICENSE
├── Makefile.am
├── NOTICE
├── README.md
├── ci/
│ ├── Dockerfile
│ └── build-nutcracker.sh
├── conf/
│ ├── nutcracker.leaf.yml
│ ├── nutcracker.root.yml
│ └── nutcracker.yml
├── configure.ac
├── contrib/
│ ├── Makefile.am
│ └── yaml-0.2.5/
│ └── .gitignore
├── m4/
│ └── .gitignore
├── man/
│ └── nutcracker.8
├── notes/
│ ├── c-styleguide.txt
│ ├── debug.txt
│ ├── memcache.md
│ ├── recommendation.md
│ ├── redis.md
│ └── socket.txt
├── scripts/
│ ├── benchmark-mget.py
│ ├── extract_redis_commands_argcounts.php
│ ├── makesrpm.sh
│ ├── multi_get.sh
│ ├── nutcracker.init
│ ├── nutcracker.init.debian
│ ├── nutcracker.spec
│ ├── pipelined_read.sh
│ ├── pipelined_write.sh
│ ├── populate_memcached.sh
│ ├── redis-check.py
│ └── redis-check.sh
├── src/
│ ├── Makefile.am
│ ├── event/
│ │ ├── Makefile.am
│ │ ├── nc_epoll.c
│ │ ├── nc_event.h
│ │ ├── nc_evport.c
│ │ └── nc_kqueue.c
│ ├── hashkit/
│ │ ├── Makefile.am
│ │ ├── nc_crc16.c
│ │ ├── nc_crc32.c
│ │ ├── nc_fnv.c
│ │ ├── nc_hashkit.h
│ │ ├── nc_hsieh.c
│ │ ├── nc_jenkins.c
│ │ ├── nc_ketama.c
│ │ ├── nc_md5.c
│ │ ├── nc_modula.c
│ │ ├── nc_murmur.c
│ │ ├── nc_one_at_a_time.c
│ │ └── nc_random.c
│ ├── nc.c
│ ├── nc_array.c
│ ├── nc_array.h
│ ├── nc_client.c
│ ├── nc_client.h
│ ├── nc_conf.c
│ ├── nc_conf.h
│ ├── nc_connection.c
│ ├── nc_connection.h
│ ├── nc_core.c
│ ├── nc_core.h
│ ├── nc_log.c
│ ├── nc_log.h
│ ├── nc_mbuf.c
│ ├── nc_mbuf.h
│ ├── nc_message.c
│ ├── nc_message.h
│ ├── nc_proxy.c
│ ├── nc_proxy.h
│ ├── nc_queue.h
│ ├── nc_rbtree.c
│ ├── nc_rbtree.h
│ ├── nc_request.c
│ ├── nc_response.c
│ ├── nc_server.c
│ ├── nc_server.h
│ ├── nc_signal.c
│ ├── nc_signal.h
│ ├── nc_stats.c
│ ├── nc_stats.h
│ ├── nc_string.c
│ ├── nc_string.h
│ ├── nc_util.c
│ ├── nc_util.h
│ ├── proto/
│ │ ├── Makefile.am
│ │ ├── nc_memcache.c
│ │ ├── nc_proto.h
│ │ └── nc_redis.c
│ └── test_all.c
├── test_in_docker.sh
└── tests/
├── .gitignore
├── README.rst
├── conf/
│ ├── conf.py
│ ├── control.sh
│ ├── redis.conf
│ └── sentinel.conf
├── lib/
│ ├── server_modules.py
│ └── utils.py
├── log/
│ └── .gitignore
├── nosetests_verbose.sh
├── test_memcache/
│ ├── __init__.py
│ └── test_gets.py
├── test_redis/
│ ├── __init__.py
│ ├── common.py
│ ├── test_auth.py
│ ├── test_basic.py
│ ├── test_commands.py
│ ├── test_mget_large_binary.py
│ ├── test_mget_mset.py
│ ├── test_pipeline.py
│ └── test_protocol.py
└── test_system/
├── __init__.py
└── test_reload.py
SYMBOL INDEX (1129 symbols across 68 files)
FILE: scripts/benchmark-mget.py
function system (line 17) | def system(cmd):
function extra (line 20) | def extra(regex, text):
function testit (line 25) | def testit():
FILE: scripts/extract_redis_commands_argcounts.php
function categorize_arg (line 26) | function categorize_arg(array $arg, string $commandName): array {
function categorize (line 42) | function categorize(array $command, string $commandName): string {
function compute_types (line 239) | function compute_types(): array {
function dump_mismatched_argument_types (line 254) | function dump_mismatched_argument_types(array $types, array $commands): ...
function render_arg (line 272) | function render_arg(array $argument): string {
function render_command (line 292) | function render_command(string $name, array $command): string {
function center_pad (line 300) | function center_pad(string $name, int $len) {
function right_pad (line 309) | function right_pad(string $name, int $len) {
function dump_table (line 317) | function dump_table(array $commands) {
function dump_table_groups (line 341) | function dump_table_groups(array $commands): void {
FILE: src/event/nc_epoll.c
type event_base (line 24) | struct event_base
type event_base (line 27) | struct event_base
type epoll_event (line 29) | struct epoll_event
function event_base_destroy (line 68) | void
function event_add_in (line 90) | int
function event_del_in (line 119) | int
function event_add_out (line 125) | int
function event_del_out (line 155) | int
function event_add_conn (line 185) | int
function event_del_conn (line 211) | int
function event_wait (line 233) | int
function event_loop_stats (line 297) | void
FILE: src/event/nc_event.h
type event_base (line 34) | struct event_base {
type event_base (line 50) | struct event_base {
type event_base (line 63) | struct event_base {
type event_base (line 76) | struct event_base
type event_base (line 77) | struct event_base
type event_base (line 79) | struct event_base
type conn (line 79) | struct conn
type event_base (line 80) | struct event_base
type conn (line 80) | struct conn
type event_base (line 81) | struct event_base
type conn (line 81) | struct conn
type event_base (line 82) | struct event_base
type conn (line 82) | struct conn
type event_base (line 83) | struct event_base
type conn (line 83) | struct conn
type event_base (line 84) | struct event_base
type conn (line 84) | struct conn
type event_base (line 85) | struct event_base
FILE: src/event/nc_evport.c
type event_base (line 25) | struct event_base
type event_base (line 28) | struct event_base
function event_base_destroy (line 69) | void
function event_add_in (line 91) | int
function event_del_in (line 97) | int
function event_add_out (line 103) | int
function event_del_out (line 129) | int
function event_add_conn (line 155) | int
function event_del_conn (line 179) | int
function event_reassociate (line 215) | static int
function event_wait (line 241) | int
function event_loop_stats (line 351) | void
FILE: src/event/nc_kqueue.c
type event_base (line 24) | struct event_base
type event_base (line 27) | struct event_base
type kevent (line 29) | struct kevent
function event_base_destroy (line 83) | void
function event_add_in (line 106) | int
function event_del_in (line 128) | int
function event_add_out (line 150) | int
function event_del_out (line 173) | int
function event_add_conn (line 196) | int
function event_del_conn (line 212) | int
function event_wait (line 242) | int
function event_loop_stats (line 348) | void
FILE: src/hashkit/nc_crc16.c
function hash_crc16 (line 55) | uint32_t
FILE: src/hashkit/nc_crc32.c
function hash_crc32 (line 98) | uint32_t
function hash_crc32a (line 111) | uint32_t
FILE: src/hashkit/nc_fnv.c
function hash_fnv1_64 (line 25) | uint32_t
function hash_fnv1a_64 (line 39) | uint32_t
function hash_fnv1_32 (line 54) | uint32_t
function hash_fnv1a_32 (line 69) | uint32_t
FILE: src/hashkit/nc_hashkit.h
type hash_type_t (line 44) | typedef enum hash_type {
type dist_type_t (line 51) | typedef enum dist_type {
type server_pool (line 71) | struct server_pool
type continuum (line 72) | struct continuum
type server_pool (line 73) | struct server_pool
type continuum (line 74) | struct continuum
type server_pool (line 75) | struct server_pool
type continuum (line 76) | struct continuum
FILE: src/hashkit/nc_hsieh.c
function hash_hsieh (line 38) | uint32_t
FILE: src/hashkit/nc_jenkins.c
function hash_jenkins (line 75) | uint32_t
FILE: src/hashkit/nc_ketama.c
function ketama_hash (line 30) | uint32_t
function ketama_item_cmp (line 43) | static int
function rstatus_t (line 57) | rstatus_t
function ketama_dispatch (line 221) | uint32_t
FILE: src/hashkit/nc_md5.c
type MD5_u32plus (line 31) | typedef unsigned int MD5_u32plus;
type MD5_CTX (line 33) | typedef struct {
function MD5_Init (line 196) | void
function MD5_Update (line 208) | void
function MD5_Final (line 244) | void
function md5_signature (line 300) | void
function hash_md5 (line 310) | uint32_t
FILE: src/hashkit/nc_modula.c
function rstatus_t (line 28) | rstatus_t
function modula_dispatch (line 145) | uint32_t
FILE: src/hashkit/nc_murmur.c
function hash_murmur (line 37) | uint32_t
FILE: src/hashkit/nc_one_at_a_time.c
function hash_one_at_a_time (line 34) | uint32_t
FILE: src/hashkit/nc_random.c
function rstatus_t (line 28) | rstatus_t
function random_dispatch (line 135) | uint32_t
FILE: src/nc.c
type option (line 54) | struct option
function rstatus_t (line 73) | static rstatus_t
function nc_print_run (line 175) | static void
function nc_print_done (line 195) | static void
function nc_show_usage (line 201) | static void
function rstatus_t (line 234) | static rstatus_t
function nc_remove_pidfile (line 263) | static void
function nc_set_default_options (line 275) | static void
function rstatus_t (line 305) | static rstatus_t
function nc_test_conf (line 448) | static bool
function rstatus_t (line 467) | static rstatus_t
function nc_post_run (line 503) | static void
function nc_run (line 517) | static void
function main (line 539) | int
FILE: src/nc_array.c
type array (line 22) | struct array
type array (line 25) | struct array
function array_destroy (line 47) | void
function rstatus_t (line 54) | rstatus_t
function array_deinit (line 71) | void
function array_idx (line 81) | uint32_t
type array (line 101) | struct array
type array (line 126) | struct array
type array (line 139) | struct array
type array (line 152) | struct array
function array_swap (line 159) | void
function array_sort (line 173) | void
function rstatus_t (line 185) | rstatus_t
FILE: src/nc_array.h
type rstatus_t (line 24) | typedef rstatus_t (*array_each_t)(void *, void *);
type array (line 26) | struct array {
function array_null (line 35) | static inline void
function array_set (line 44) | static inline void
function array_n (line 53) | static inline uint32_t
type array (line 59) | struct array
type array (line 60) | struct array
type array (line 61) | struct array
type array (line 62) | struct array
type array (line 64) | struct array
type array (line 65) | struct array
type array (line 66) | struct array
type array (line 67) | struct array
type array (line 68) | struct array
type array (line 69) | struct array
type array (line 69) | struct array
type array (line 70) | struct array
type array (line 71) | struct array
FILE: src/nc_client.c
function client_ref (line 22) | void
function client_unref (line 49) | void
function client_active (line 68) | bool
function client_close_stats (line 95) | static void
function client_close (line 122) | void
FILE: src/nc_client.h
type conn (line 23) | struct conn
type conn (line 24) | struct conn
type conn (line 25) | struct conn
type context (line 26) | struct context
type conn (line 26) | struct conn
FILE: src/nc_conf.c
type string (line 24) | struct string
type string (line 38) | struct string
type command (line 44) | struct command
type conf_pool (line 47) | struct conf_pool
type conf_pool (line 51) | struct conf_pool
type conf_pool (line 55) | struct conf_pool
type conf_pool (line 59) | struct conf_pool
type conf_pool (line 63) | struct conf_pool
type conf_pool (line 67) | struct conf_pool
type conf_pool (line 71) | struct conf_pool
type conf_pool (line 75) | struct conf_pool
type conf_pool (line 79) | struct conf_pool
type conf_pool (line 83) | struct conf_pool
type conf_pool (line 87) | struct conf_pool
type conf_pool (line 91) | struct conf_pool
type conf_pool (line 95) | struct conf_pool
type conf_pool (line 99) | struct conf_pool
type conf_pool (line 103) | struct conf_pool
type conf_pool (line 107) | struct conf_pool
type conf_pool (line 111) | struct conf_pool
type conf_pool (line 115) | struct conf_pool
type string (line 120) | struct string
type string (line 121) | struct string
function conf_server_init (line 123) | static void
function conf_server_deinit (line 139) | static void
function rstatus_t (line 149) | rstatus_t
function rstatus_t (line 184) | static rstatus_t
function conf_pool_deinit (line 238) | static void
function rstatus_t (line 258) | rstatus_t
function conf_dump (line 326) | static void
function rstatus_t (line 375) | static rstatus_t
function conf_yaml_deinit (line 402) | static void
function rstatus_t (line 411) | static rstatus_t
function conf_token_done (line 428) | static void
function rstatus_t (line 439) | static rstatus_t
function conf_event_done (line 456) | static void
function rstatus_t (line 465) | static rstatus_t
function conf_pop_scalar (line 496) | static void
function rstatus_t (line 506) | static rstatus_t
function rstatus_t (line 547) | static rstatus_t
function rstatus_t (line 592) | static rstatus_t
function rstatus_t (line 630) | static rstatus_t
function rstatus_t (line 733) | static rstatus_t
type conf (line 761) | struct conf
type conf (line 765) | struct conf
type string (line 781) | struct string
type conf_pool (line 788) | struct conf_pool
function rstatus_t (line 813) | static rstatus_t
function rstatus_t (line 861) | static rstatus_t
function rstatus_t (line 981) | static rstatus_t
function rstatus_t (line 1123) | static rstatus_t
function conf_server_name_cmp (line 1148) | static int
function conf_pool_name_cmp (line 1156) | static int
function conf_pool_listen_cmp (line 1164) | static int
function rstatus_t (line 1172) | static rstatus_t
function rstatus_t (line 1213) | static rstatus_t
function rstatus_t (line 1300) | static rstatus_t
type conf (line 1369) | struct conf
type conf (line 1373) | struct conf
function conf_destroy (line 1414) | void
type conf (line 1431) | struct conf
type command (line 1431) | struct command
type string (line 1435) | struct string
type string (line 1436) | struct string
type string (line 1439) | struct string
type conf (line 1456) | struct conf
type command (line 1456) | struct command
type string (line 1459) | struct string
type conf_listen (line 1460) | struct conf_listen
type conf_listen (line 1465) | struct conf_listen
type conf (line 1545) | struct conf
type command (line 1545) | struct command
type array (line 1548) | struct array
type string (line 1549) | struct string
type conf_server (line 1550) | struct conf_server
type array (line 1557) | struct array
type conf (line 1685) | struct conf
type command (line 1685) | struct command
type string (line 1689) | struct string
type conf (line 1711) | struct conf
type command (line 1711) | struct command
type string (line 1715) | struct string
type conf (line 1738) | struct conf
type command (line 1738) | struct command
type string (line 1742) | struct string
type conf (line 1767) | struct conf
type command (line 1767) | struct command
type string (line 1771) | struct string
type conf (line 1796) | struct conf
type command (line 1796) | struct command
type string (line 1800) | struct string
type string (line 1801) | struct string
type string (line 1804) | struct string
FILE: src/nc_conf.h
type conf_listen (line 60) | struct conf_listen {
type conf_server (line 69) | struct conf_server {
type conf_pool (line 79) | struct conf_pool {
type conf (line 102) | struct conf {
type command (line 120) | struct command {
type conf (line 128) | struct conf
type command (line 128) | struct command
type conf (line 129) | struct conf
type command (line 129) | struct command
type conf (line 130) | struct conf
type command (line 130) | struct command
type conf (line 131) | struct conf
type command (line 131) | struct command
type conf (line 132) | struct conf
type command (line 132) | struct command
type conf (line 133) | struct conf
type command (line 133) | struct command
type conf (line 134) | struct conf
type command (line 134) | struct command
type conf (line 135) | struct conf
type command (line 135) | struct command
type conf (line 140) | struct conf
type conf (line 141) | struct conf
FILE: src/nc_connection.c
type conn_tqh (line 85) | struct conn_tqh
type context (line 93) | struct context
type conn (line 94) | struct conn
type server_pool (line 96) | struct server_pool
type server (line 101) | struct server
type conn (line 108) | struct conn
type conn (line 111) | struct conn
type conn (line 167) | struct conn
type conn (line 170) | struct conn
type conn (line 247) | struct conn
type server_pool (line 248) | struct server_pool
type conn (line 250) | struct conn
function conn_free (line 287) | static void
function conn_put (line 294) | void
function conn_init (line 311) | void
function conn_deinit (line 319) | void
function conn_recv (line 333) | ssize_t
function conn_sendv (line 383) | ssize_t
function conn_ncurr_conn (line 432) | uint32_t
function conn_ntotal_conn (line 438) | uint64_t
function conn_ncurr_cconn (line 444) | uint32_t
function conn_authenticated (line 454) | bool
FILE: src/nc_connection.h
type rstatus_t (line 23) | typedef rstatus_t (*conn_recv_t)(struct context *, struct conn*);
type msg (line 24) | struct msg
type context (line 24) | struct context
type conn (line 24) | struct conn
type context (line 25) | struct context
type conn (line 25) | struct conn
type msg (line 25) | struct msg
type msg (line 25) | struct msg
type rstatus_t (line 27) | typedef rstatus_t (*conn_send_t)(struct context *, struct conn*);
type msg (line 28) | struct msg
type context (line 28) | struct context
type conn (line 28) | struct conn
type context (line 29) | struct context
type conn (line 29) | struct conn
type msg (line 29) | struct msg
type context (line 31) | struct context
type conn (line 31) | struct conn
type conn (line 32) | struct conn
type conn (line 34) | struct conn
type conn (line 35) | struct conn
type context (line 37) | struct context
type conn (line 37) | struct conn
type msg (line 37) | struct msg
type context (line 38) | struct context
type conn (line 38) | struct conn
type server (line 38) | struct server
type conn (line 39) | struct conn
type msg (line 39) | struct msg
type msg (line 39) | struct msg
type conn (line 41) | struct conn {
type context (line 96) | struct context
type conn (line 96) | struct conn
type conn (line 97) | struct conn
type conn (line 98) | struct conn
type server_pool (line 98) | struct server_pool
type conn (line 99) | struct conn
type conn (line 100) | struct conn
type conn (line 101) | struct conn
type array (line 101) | struct array
type conn (line 107) | struct conn
FILE: src/nc_core.c
function rstatus_t (line 27) | static rstatus_t
type context (line 48) | struct context
type instance (line 49) | struct instance
type context (line 52) | struct context
function core_ctx_destroy (line 145) | static void
type context (line 158) | struct context
type instance (line 159) | struct instance
type context (line 161) | struct context
function core_stop (line 180) | void
function rstatus_t (line 189) | static rstatus_t
function rstatus_t (line 204) | static rstatus_t
function core_close (line 219) | static void
function core_error (line 249) | static void
function core_timeout (line 265) | static void
function rstatus_t (line 310) | rstatus_t
function rstatus_t (line 355) | rstatus_t
FILE: src/nc_core.h
type rstatus_t (line 69) | typedef int rstatus_t;
type err_t (line 70) | typedef int err_t;
type array (line 72) | struct array
type string (line 73) | struct string
type context (line 74) | struct context
type conn (line 75) | struct conn
type conn_tqh (line 76) | struct conn_tqh
type msg (line 77) | struct msg
type msg_tqh (line 78) | struct msg_tqh
type server (line 79) | struct server
type server_pool (line 80) | struct server_pool
type mbuf (line 81) | struct mbuf
type mhdr (line 82) | struct mhdr
type conf (line 83) | struct conf
type stats (line 84) | struct stats
type instance (line 85) | struct instance
type event_base (line 86) | struct event_base
type context (line 121) | struct context {
type instance (line 137) | struct instance {
type context (line 152) | struct context
type instance (line 152) | struct instance
type context (line 153) | struct context
type context (line 155) | struct context
FILE: src/nc_log.c
type logger (line 27) | struct logger
function log_init (line 29) | int
function log_deinit (line 50) | void
function log_reopen (line 62) | void
function log_level_up (line 77) | void
function log_level_down (line 88) | void
function log_level_set (line 99) | void
function log_stacktrace (line 108) | void
function log_loggable (line 119) | int
function _log (line 131) | void
function _log_stderr (line 173) | void
function _log_hexdump (line 204) | void
function _log_safe (line 272) | void
function _log_stderr_safe (line 305) | void
FILE: src/nc_log.h
type logger (line 23) | struct logger {
FILE: src/nc_mbuf.c
type mhdr (line 24) | struct mhdr
type mbuf (line 29) | struct mbuf
type mbuf (line 32) | struct mbuf
type mbuf (line 70) | struct mbuf
type mbuf (line 78) | struct mbuf
type mbuf (line 81) | struct mbuf
function mbuf_free (line 104) | static void
function mbuf_put (line 118) | void
function mbuf_rewind (line 134) | void
function mbuf_length (line 145) | uint32_t
function mbuf_size (line 157) | uint32_t
function mbuf_data_size (line 169) | size_t
function mbuf_insert (line 178) | void
function mbuf_remove (line 189) | void
function mbuf_copy (line 205) | void
type mbuf (line 229) | struct mbuf
type mhdr (line 230) | struct mhdr
type mbuf (line 232) | struct mbuf
function mbuf_init (line 264) | void
function mbuf_deinit (line 277) | void
FILE: src/nc_mbuf.h
type mbuf (line 23) | struct mbuf
type mbuf (line 25) | struct mbuf {
function mbuf_empty (line 42) | static inline bool
function mbuf_full (line 48) | static inline bool
type instance (line 54) | struct instance
type mbuf (line 56) | struct mbuf
type mbuf (line 57) | struct mbuf
type mbuf (line 58) | struct mbuf
type mbuf (line 59) | struct mbuf
type mbuf (line 60) | struct mbuf
type mhdr (line 62) | struct mhdr
type mbuf (line 62) | struct mbuf
type mhdr (line 63) | struct mhdr
type mbuf (line 63) | struct mbuf
type mbuf (line 64) | struct mbuf
type mbuf (line 65) | struct mbuf
type mhdr (line 65) | struct mhdr
FILE: src/nc_message.c
type msg_tqh (line 115) | struct msg_tqh
type rbtree (line 116) | struct rbtree
type rbnode (line 117) | struct rbnode
type string (line 120) | struct string
type msg (line 126) | struct msg
type rbnode (line 127) | struct rbnode
type msg (line 129) | struct msg
type msg (line 132) | struct msg
type msg (line 133) | struct msg
type msg (line 138) | struct msg
type rbnode (line 141) | struct rbnode
function msg_tmo_insert (line 151) | void
function msg_tmo_delete (line 175) | void
type msg (line 193) | struct msg
type msg (line 196) | struct msg
type keypos (line 239) | struct keypos
type msg (line 280) | struct msg
type conn (line 281) | struct conn
type msg (line 283) | struct msg
type msg (line 329) | struct msg
type msg (line 332) | struct msg
type mbuf (line 333) | struct mbuf
function msg_free (line 363) | static void
function msg_put (line 372) | void
function msg_dump (line 398) | void
function msg_init (line 423) | void
function msg_deinit (line 434) | void
type string (line 448) | struct string
function msg_empty (line 454) | bool
function msg_backend_idx (line 460) | uint32_t
type mbuf (line 469) | struct mbuf
type msg (line 470) | struct msg
type mbuf (line 472) | struct mbuf
function rstatus_t (line 492) | rstatus_t
function rstatus_t (line 516) | rstatus_t
function rstatus_t (line 540) | rstatus_t
function msg_gen_frag_id (line 569) | inline uint64_t
function rstatus_t (line 575) | static rstatus_t
function rstatus_t (line 616) | static rstatus_t
function rstatus_t (line 631) | static rstatus_t
function rstatus_t (line 666) | static rstatus_t
function rstatus_t (line 719) | rstatus_t
function rstatus_t (line 743) | static rstatus_t
function rstatus_t (line 870) | rstatus_t
function msg_set_placeholder_key (line 900) | bool msg_set_placeholder_key(struct msg *r)
FILE: src/nc_message.h
type msg (line 24) | struct msg
type rstatus_t (line 25) | typedef rstatus_t (*msg_add_auth_t)(struct context *ctx, struct conn *c_...
type rstatus_t (line 26) | typedef rstatus_t (*msg_fragment_t)(struct msg *, uint32_t, struct msg_t...
type msg (line 27) | struct msg
type rstatus_t (line 28) | typedef rstatus_t (*msg_reply_t)(struct msg *r);
type msg (line 29) | struct msg
type msg_parse_result_t (line 31) | typedef enum msg_parse_result {
type keypos (line 232) | struct keypos {
type msg (line 241) | struct msg {
type msg (line 307) | struct msg
type msg (line 308) | struct msg
type conn (line 308) | struct conn
type msg (line 309) | struct msg
type string (line 313) | struct string
type msg (line 314) | struct msg
type conn (line 314) | struct conn
type msg (line 315) | struct msg
type msg (line 316) | struct msg
type msg (line 317) | struct msg
type msg (line 318) | struct msg
type context (line 319) | struct context
type conn (line 319) | struct conn
type context (line 320) | struct context
type conn (line 320) | struct conn
type msg (line 322) | struct msg
type mbuf (line 323) | struct mbuf
type msg (line 323) | struct msg
type msg (line 324) | struct msg
type msg (line 325) | struct msg
type msg (line 326) | struct msg
type msg (line 327) | struct msg
type msg (line 329) | struct msg
type conn (line 329) | struct conn
type msg (line 330) | struct msg
type conn (line 331) | struct conn
type msg (line 331) | struct msg
type conn (line 332) | struct conn
type msg (line 332) | struct msg
type context (line 333) | struct context
type conn (line 333) | struct conn
type msg (line 333) | struct msg
type context (line 334) | struct context
type conn (line 334) | struct conn
type msg (line 334) | struct msg
type context (line 335) | struct context
type conn (line 335) | struct conn
type msg (line 335) | struct msg
type context (line 336) | struct context
type conn (line 336) | struct conn
type msg (line 336) | struct msg
type context (line 337) | struct context
type conn (line 337) | struct conn
type msg (line 337) | struct msg
type context (line 338) | struct context
type conn (line 338) | struct conn
type msg (line 338) | struct msg
type context (line 339) | struct context
type conn (line 339) | struct conn
type msg (line 339) | struct msg
type msg (line 340) | struct msg
type context (line 340) | struct context
type conn (line 340) | struct conn
type context (line 341) | struct context
type conn (line 341) | struct conn
type msg (line 341) | struct msg
type msg (line 341) | struct msg
type msg (line 342) | struct msg
type context (line 342) | struct context
type conn (line 342) | struct conn
type msg (line 343) | struct msg
type context (line 343) | struct context
type conn (line 343) | struct conn
type context (line 344) | struct context
type conn (line 344) | struct conn
type msg (line 344) | struct msg
type msg (line 346) | struct msg
type conn (line 346) | struct conn
type msg (line 347) | struct msg
type msg (line 348) | struct msg
type context (line 348) | struct context
type conn (line 348) | struct conn
type context (line 349) | struct context
type conn (line 349) | struct conn
type msg (line 349) | struct msg
type msg (line 349) | struct msg
type msg (line 350) | struct msg
type context (line 350) | struct context
type conn (line 350) | struct conn
type context (line 351) | struct context
type conn (line 351) | struct conn
type msg (line 351) | struct msg
FILE: src/nc_proxy.c
function proxy_ref (line 25) | void
function proxy_unref (line 46) | void
function proxy_close (line 63) | void
function rstatus_t (line 92) | static rstatus_t
function rstatus_t (line 123) | static rstatus_t
function rstatus_t (line 205) | rstatus_t
function rstatus_t (line 232) | rstatus_t
function rstatus_t (line 251) | rstatus_t
function proxy_deinit (line 265) | void
function rstatus_t (line 281) | static rstatus_t
function rstatus_t (line 401) | rstatus_t
FILE: src/nc_proxy.h
type conn (line 23) | struct conn
type conn (line 24) | struct conn
type context (line 25) | struct context
type conn (line 25) | struct conn
type context (line 30) | struct context
type context (line 31) | struct context
type context (line 32) | struct context
type conn (line 32) | struct conn
FILE: src/nc_queue.h
type qm_trace (line 155) | struct qm_trace {
type type (line 278) | struct type
type type (line 403) | struct type
type type (line 514) | struct type
type type (line 524) | struct type
type type (line 705) | struct type
type type (line 714) | struct type
FILE: src/nc_rbtree.c
function rbtree_node_init (line 20) | void
function rbtree_init (line 31) | void
type rbnode (line 40) | struct rbnode
type rbnode (line 41) | struct rbnode
type rbnode (line 41) | struct rbnode
type rbnode (line 52) | struct rbnode
type rbtree (line 53) | struct rbtree
type rbnode (line 55) | struct rbnode
type rbnode (line 56) | struct rbnode
function rbtree_left_rotate (line 67) | static void
function rbtree_right_rotate (line 94) | static void
function rbtree_insert (line 121) | void
function rbtree_delete (line 203) | void
FILE: src/nc_rbtree.h
type rbnode (line 27) | struct rbnode {
type rbtree (line 36) | struct rbtree {
type rbnode (line 41) | struct rbnode
type rbtree (line 42) | struct rbtree
type rbnode (line 42) | struct rbnode
type rbnode (line 43) | struct rbnode
type rbtree (line 43) | struct rbtree
type rbtree (line 44) | struct rbtree
type rbnode (line 44) | struct rbnode
type rbtree (line 45) | struct rbtree
type rbnode (line 45) | struct rbnode
FILE: src/nc_request.c
type msg (line 21) | struct msg
type conn (line 22) | struct conn
type msg (line 24) | struct msg
function req_log (line 35) | static void
function req_put (line 97) | void
function req_done (line 128) | bool
function req_error (line 219) | bool
function req_server_enqueue_imsgq (line 294) | void
function req_server_enqueue_imsgq_head (line 318) | void
function req_server_dequeue_imsgq (line 342) | void
function req_client_enqueue_omsgq (line 354) | void
function req_server_enqueue_omsgq (line 363) | void
function req_client_dequeue_omsgq (line 375) | void
function req_server_dequeue_omsgq (line 384) | void
type msg (line 398) | struct msg
type context (line 399) | struct context
type conn (line 399) | struct conn
type msg (line 401) | struct msg
function rstatus_t (line 453) | static rstatus_t
function req_filter (line 474) | static bool
function req_forward_error (line 517) | static void
function req_forward_stats (line 546) | static void
function req_forward (line 555) | static void
function req_recv_done (line 626) | void
type msg (line 706) | struct msg
type context (line 707) | struct context
type conn (line 707) | struct conn
type msg (line 710) | struct msg
function req_send_done (line 749) | void
FILE: src/nc_response.c
type msg (line 21) | struct msg
type conn (line 22) | struct conn
type msg (line 24) | struct msg
function rsp_put (line 36) | void
type msg (line 44) | struct msg
type context (line 45) | struct context
type conn (line 45) | struct conn
type msg (line 45) | struct msg
type msg (line 47) | struct msg
type msg (line 48) | struct msg
type msg (line 86) | struct msg
type context (line 87) | struct context
type conn (line 87) | struct conn
type msg (line 89) | struct msg
function rsp_filter (line 141) | static bool
function rsp_forward_stats (line 224) | static void
function rsp_forward (line 233) | static void
function rsp_recv_done (line 274) | void
type msg (line 294) | struct msg
type context (line 295) | struct context
type conn (line 295) | struct conn
type msg (line 298) | struct msg
function rsp_send_done (line 352) | void
FILE: src/nc_server.c
function server_resolve (line 25) | static void
function server_ref (line 42) | void
function server_unref (line 61) | void
function server_timeout (line 80) | int
function server_active (line 94) | bool
function rstatus_t (line 124) | static rstatus_t
function rstatus_t (line 135) | rstatus_t
function server_deinit (line 172) | void
type conn (line 186) | struct conn
type server (line 187) | struct server
type server_pool (line 189) | struct server_pool
type conn (line 190) | struct conn
function rstatus_t (line 218) | static rstatus_t
function rstatus_t (line 244) | static rstatus_t
function server_failure (line 265) | static void
function server_close_stats (line 312) | static void
function server_close (line 344) | void
function rstatus_t (line 465) | rstatus_t
function server_connected (line 548) | void
function server_ok (line 567) | void
function rstatus_t (line 584) | static rstatus_t
function server_pool_hash (line 629) | static uint32_t
function server_pool_idx (line 646) | uint32_t
type server (line 702) | struct server
type server_pool (line 703) | struct server_pool
type server (line 705) | struct server
type conn (line 717) | struct conn
type context (line 718) | struct context
type server_pool (line 718) | struct server_pool
type server (line 722) | struct server
type conn (line 723) | struct conn
function rstatus_t (line 751) | static rstatus_t
function rstatus_t (line 769) | rstatus_t
function rstatus_t (line 782) | static rstatus_t
function server_pool_disconnect (line 796) | void
function rstatus_t (line 802) | static rstatus_t
function rstatus_t (line 813) | static rstatus_t
function rstatus_t (line 825) | rstatus_t
function rstatus_t (line 848) | static rstatus_t
function rstatus_t (line 854) | rstatus_t
function server_pool_deinit (line 905) | void
FILE: src/nc_server.h
type continuum (line 64) | struct continuum {
type server (line 69) | struct server {
type server_pool (line 87) | struct server_pool {
type conn (line 127) | struct conn
type conn (line 128) | struct conn
type conn (line 129) | struct conn
type conn (line 130) | struct conn
type array (line 131) | struct array
type array (line 131) | struct array
type server_pool (line 131) | struct server_pool
type array (line 132) | struct array
type conn (line 133) | struct conn
type server (line 133) | struct server
type context (line 134) | struct context
type server (line 134) | struct server
type conn (line 134) | struct conn
type context (line 135) | struct context
type conn (line 135) | struct conn
type context (line 136) | struct context
type conn (line 136) | struct conn
type context (line 137) | struct context
type conn (line 137) | struct conn
type server_pool (line 139) | struct server_pool
type conn (line 140) | struct conn
type context (line 140) | struct context
type server_pool (line 140) | struct server_pool
type server_pool (line 141) | struct server_pool
type context (line 142) | struct context
type context (line 143) | struct context
type array (line 144) | struct array
type array (line 144) | struct array
type context (line 144) | struct context
type array (line 145) | struct array
FILE: src/nc_signal.c
type signal (line 24) | struct signal
function rstatus_t (line 36) | rstatus_t
function signal_deinit (line 61) | void
function signal_handler (line 66) | void
FILE: src/nc_signal.h
type signal (line 23) | struct signal {
FILE: src/nc_stats.c
type stats_desc (line 29) | struct stats_desc {
type stats_metric (line 35) | struct stats_metric
type stats_metric (line 39) | struct stats_metric
type stats_desc (line 45) | struct stats_desc
type stats_desc (line 49) | struct stats_desc
function stats_describe (line 54) | void
function stats_metric_init (line 74) | static void
function stats_metric_reset (line 95) | static void
function rstatus_t (line 110) | static rstatus_t
function rstatus_t (line 134) | static rstatus_t
function stats_metric_deinit (line 158) | static void
function rstatus_t (line 170) | static rstatus_t
function rstatus_t (line 190) | static rstatus_t
function stats_server_unmap (line 219) | static void
function rstatus_t (line 235) | static rstatus_t
function stats_pool_reset (line 262) | static void
function rstatus_t (line 283) | static rstatus_t
function stats_pool_unmap (line 312) | static void
function rstatus_t (line 329) | static rstatus_t
function stats_destroy_buf (line 424) | static void
function rstatus_t (line 434) | static rstatus_t
function rstatus_t (line 457) | static rstatus_t
function rstatus_t (line 480) | static rstatus_t
function rstatus_t (line 532) | static rstatus_t
function rstatus_t (line 553) | static rstatus_t
function rstatus_t (line 575) | static rstatus_t
function rstatus_t (line 612) | static rstatus_t
function stats_aggregate_metric (line 630) | static void
function stats_aggregate (line 665) | static void
function rstatus_t (line 699) | static rstatus_t
function rstatus_t (line 759) | static rstatus_t
function stats_loop_callback (line 791) | static void
function rstatus_t (line 815) | static rstatus_t
function rstatus_t (line 858) | static rstatus_t
function stats_stop_aggregator (line 881) | static void
type stats (line 891) | struct stats
type array (line 893) | struct array
type stats (line 896) | struct stats
function stats_destroy (line 972) | void
function stats_swap (line 983) | void
type stats_metric (line 1017) | struct stats_metric
type context (line 1018) | struct context
type server_pool (line 1018) | struct server_pool
type stats (line 1021) | struct stats
type stats_pool (line 1022) | struct stats_pool
type stats_metric (line 1023) | struct stats_metric
function _stats_pool_incr (line 1040) | void
function _stats_pool_decr (line 1055) | void
function _stats_pool_incr_by (line 1070) | void
function _stats_pool_decr_by (line 1085) | void
function _stats_pool_set_ts (line 1100) | void
type stats_metric (line 1115) | struct stats_metric
type context (line 1116) | struct context
type server (line 1116) | struct server
type stats (line 1119) | struct stats
type stats_pool (line 1120) | struct stats_pool
type stats_server (line 1121) | struct stats_server
type stats_metric (line 1122) | struct stats_metric
function _stats_server_incr (line 1141) | void
function _stats_server_decr (line 1156) | void
function _stats_server_incr_by (line 1171) | void
function _stats_server_decr_by (line 1186) | void
function _stats_server_set_ts (line 1201) | void
FILE: src/nc_stats.h
type stats_type_t (line 55) | typedef enum stats_type {
type stats_metric (line 63) | struct stats_metric {
type stats_server (line 72) | struct stats_server {
type stats_pool (line 77) | struct stats_pool {
type stats_buffer (line 83) | struct stats_buffer {
type stats (line 89) | struct stats {
type stats_pool_field_t (line 120) | typedef enum stats_pool_field {
type stats_server_field_t (line 127) | typedef enum stats_server_field {
type context (line 199) | struct context
type server_pool (line 199) | struct server_pool
type context (line 200) | struct context
type server_pool (line 200) | struct server_pool
type context (line 201) | struct context
type server_pool (line 201) | struct server_pool
type context (line 202) | struct context
type server_pool (line 202) | struct server_pool
type context (line 203) | struct context
type server_pool (line 203) | struct server_pool
type context (line 205) | struct context
type server (line 205) | struct server
type context (line 206) | struct context
type server (line 206) | struct server
type context (line 207) | struct context
type server (line 207) | struct server
type context (line 208) | struct context
type server (line 208) | struct server
type context (line 209) | struct context
type server (line 209) | struct server
type stats (line 211) | struct stats
type array (line 211) | struct array
type stats (line 212) | struct stats
type stats (line 213) | struct stats
FILE: src/nc_string.c
function string_init (line 40) | void
function string_deinit (line 47) | void
function string_empty (line 59) | bool
function rstatus_t (line 67) | rstatus_t
function rstatus_t (line 84) | rstatus_t
function string_compare (line 101) | int
function _safe_vsnprintf (line 194) | int
function _safe_snprintf (line 275) | int
FILE: src/nc_string.h
type string (line 27) | struct string {
type string (line 45) | struct string
type string (line 46) | struct string
type string (line 47) | struct string
type string (line 48) | struct string
type string (line 48) | struct string
type string (line 49) | struct string
type string (line 50) | struct string
type string (line 50) | struct string
FILE: src/nc_util.c
function nc_set_blocking (line 40) | int
function nc_set_nonblocking (line 53) | int
function nc_set_reuseaddr (line 66) | int
function nc_set_reuseport (line 78) | int
function nc_set_tcpnodelay (line 100) | int
function nc_set_linger (line 112) | int
function nc_set_tcpkeepalive (line 126) | int
function nc_set_sndbuf (line 133) | int
function nc_set_rcvbuf (line 143) | int
function nc_get_soerror (line 153) | int
function nc_get_sndbuf (line 170) | int
function nc_get_rcvbuf (line 187) | int
function _nc_atoi (line 204) | int
function nc_valid_port (line 228) | bool
function _nc_free (line 291) | void
function nc_stacktrace (line 299) | void
function nc_stacktrace_fd (line 323) | void
function nc_assert (line 335) | void
function _vscnprintf (line 345) | int
function _scnprintf (line 373) | int
function _nc_sendn (line 389) | ssize_t
function _nc_recvn (line 420) | ssize_t
function nc_usec_now (line 451) | int64_t
function nc_msec_now (line 472) | int64_t
function nc_resolve_inet (line 478) | static int
function nc_resolve_unix (line 548) | static int
function nc_resolve (line 576) | int
type sockaddr (line 593) | struct sockaddr
type sockinfo (line 620) | struct sockinfo
type sockaddr (line 621) | struct sockaddr
type sockaddr (line 626) | struct sockaddr
type sockinfo (line 646) | struct sockinfo
type sockaddr (line 647) | struct sockaddr
type sockaddr (line 652) | struct sockaddr
FILE: src/nc_util.h
type sockinfo (line 214) | struct sockinfo {
type string (line 224) | struct string
type sockinfo (line 224) | struct sockinfo
type sockaddr (line 225) | struct sockaddr
FILE: src/proto/nc_memcache.c
function memcache_storage (line 39) | static bool
function memcache_cas (line 62) | static bool
function memcache_retrieval (line 76) | static bool
function memcache_should_fragment (line 103) | static bool
function memcache_arithmetic (line 126) | static bool
function memcache_delete (line 145) | static bool
function memcache_touch (line 159) | static bool
function memcache_parse_req (line 169) | void
function memcache_parse_rsp (line 790) | void
function memcache_failure (line 1248) | bool
function rstatus_t (line 1254) | static rstatus_t
function rstatus_t (line 1283) | static rstatus_t
function rstatus_t (line 1386) | rstatus_t
function memcache_pre_coalesce (line 1400) | void
function rstatus_t (line 1471) | static rstatus_t
function memcache_post_coalesce (line 1568) | void
function memcache_post_connect (line 1604) | void
function memcache_swallow_msg (line 1609) | void
function rstatus_t (line 1614) | rstatus_t
function rstatus_t (line 1621) | rstatus_t
FILE: src/proto/nc_proto.h
type msg (line 145) | struct msg
type msg (line 146) | struct msg
type msg (line 147) | struct msg
type msg (line 148) | struct msg
type msg (line 149) | struct msg
type context (line 150) | struct context
type conn (line 150) | struct conn
type conn (line 150) | struct conn
type msg (line 151) | struct msg
type msg_tqh (line 151) | struct msg_tqh
type msg (line 152) | struct msg
type context (line 153) | struct context
type conn (line 153) | struct conn
type server (line 153) | struct server
type conn (line 154) | struct conn
type msg (line 154) | struct msg
type msg (line 154) | struct msg
type msg (line 156) | struct msg
type msg (line 157) | struct msg
type msg (line 158) | struct msg
type msg (line 159) | struct msg
type msg (line 160) | struct msg
type context (line 161) | struct context
type conn (line 161) | struct conn
type conn (line 161) | struct conn
type msg (line 162) | struct msg
type msg_tqh (line 162) | struct msg_tqh
type msg (line 163) | struct msg
type context (line 164) | struct context
type conn (line 164) | struct conn
type server (line 164) | struct server
type conn (line 165) | struct conn
type msg (line 165) | struct msg
type msg (line 165) | struct msg
FILE: src/proto/nc_redis.c
type msg (line 36) | struct msg
type msg (line 36) | struct msg
function redis_argz (line 42) | static bool
function redis_arg0 (line 63) | static bool
function redis_arg1 (line 105) | static bool
function redis_arg2 (line 148) | static bool
function redis_arg3 (line 188) | static bool
function redis_argn (line 207) | static bool
function redis_argx (line 299) | static bool
function redis_argkvx (line 320) | static bool
function redis_argeval (line 340) | static bool
function redis_nokey (line 355) | static bool
function redis_error (line 373) | static bool
function redis_parse_req (line 424) | void
function redis_parse_rsp (line 1964) | void
function redis_failure (line 2541) | bool
function rstatus_t (line 2565) | static rstatus_t
function redis_pre_coalesce (line 2641) | void
function rstatus_t (line 2721) | static rstatus_t
function rstatus_t (line 2817) | static rstatus_t
function rstatus_t (line 2948) | rstatus_t
function rstatus_t (line 2971) | rstatus_t
function redis_post_coalesce_mset (line 2998) | void
function redis_post_coalesce_del_or_touch (line 3011) | void
function redis_post_coalesce_mget (line 3024) | static void
function redis_post_coalesce (line 3062) | void
function rstatus_t (line 3091) | static rstatus_t
function rstatus_t (line 3134) | rstatus_t
function redis_post_connect (line 3166) | void
function redis_swallow_msg (line 3216) | void
FILE: src/test_all.c
function expect_same_int (line 10) | static void expect_same_int(int expected, int actual, const char* messag...
function expect_same_uint32_t (line 20) | static void expect_same_uint32_t(uint32_t expected, uint32_t actual, con...
function expect_same_ptr (line 31) | static void expect_same_ptr(const void *expected, const void *actual, co...
function test_hash_algorithms (line 41) | static void test_hash_algorithms(void) {
function test_config_parsing (line 62) | static void test_config_parsing(void) {
function test_redis_parse_req_success_case (line 76) | static void test_redis_parse_req_success_case(const char* data, int expe...
function test_redis_parse_req_success (line 109) | static void test_redis_parse_req_success(void) {
function test_redis_parse_rsp_success_case (line 232) | static void test_redis_parse_rsp_success_case(const char* data, int expe...
function test_redis_parse_rsp_success (line 267) | static void test_redis_parse_rsp_success(void) {
function test_redis_parse_rsp_failure_case (line 304) | static void test_redis_parse_rsp_failure_case(const char* data) {
function test_redis_parse_rsp_failure (line 338) | static void test_redis_parse_rsp_failure(void) {
function test_memcache_parse_rsp_success_case (line 354) | static void test_memcache_parse_rsp_success_case(const char* data, int e...
function test_memcache_parse_rsp_success (line 392) | static void test_memcache_parse_rsp_success(void) {
function test_memcache_parse_rsp_failure_case (line 409) | static void test_memcache_parse_rsp_failure_case(const char* data) {
function test_memcache_parse_rsp_failure (line 444) | static void test_memcache_parse_rsp_failure(void) {
function test_memcache_parse_req_success_case (line 451) | static void test_memcache_parse_req_success_case(const char* data, int e...
function test_memcache_parse_req_success (line 492) | static void test_memcache_parse_req_success(void) {
function test_memcache_parse_req_failure_case (line 523) | static void test_memcache_parse_req_failure_case(const char* data) {
function test_memcache_parse_req_failure (line 557) | static void test_memcache_parse_req_failure(void) {
function main (line 583) | int main(int argc, char **argv) {
FILE: tests/lib/server_modules.py
class Base (line 18) | class Base:
method __init__ (line 23) | def __init__(self, name, host, port, path):
method __str__ (line 38) | def __str__(self):
method deploy (line 41) | def deploy(self):
method _gen_control_script (line 52) | def _gen_control_script(self):
method start (line 63) | def start(self):
method stop (line 86) | def stop(self):
method pid (line 100) | def pid(self):
method status (line 104) | def status(self):
method _alive (line 107) | def _alive(self):
method _run (line 110) | def _run(self, raw_cmd):
method clean (line 115) | def clean(self):
method host (line 119) | def host(self):
method port (line 122) | def port(self):
class RedisServer (line 125) | class RedisServer(Base):
method __init__ (line 126) | def __init__(self, host, port, path, cluster_name, server_name, auth =...
method _info_dict (line 141) | def _info_dict(self):
method _ping (line 152) | def _ping(self):
method _alive (line 158) | def _alive(self):
method _gen_conf (line 161) | def _gen_conf(self):
method _pre_deploy (line 168) | def _pre_deploy(self):
method status (line 176) | def status(self):
method isslaveof (line 183) | def isslaveof(self, master_host, master_port):
method slaveof (line 190) | def slaveof(self, master_host, master_port):
method rediscmd (line 194) | def rediscmd(self, cmd):
class RedisSentinel (line 201) | class RedisSentinel(RedisServer):
method __init__ (line 202) | def __init__(self, host, port, path, cluster_name, server_name, master...
method _gen_conf_section (line 215) | def _gen_conf_section(self):
method _gen_conf (line 225) | def _gen_conf(self):
method _pre_deploy (line 232) | def _pre_deploy(self):
method failover (line 240) | def failover(self, server_name):
class Memcached (line 244) | class Memcached(Base):
method __init__ (line 245) | def __init__(self, host, port, path, cluster_name, server_name):
method _alive (line 254) | def _alive(self):
method _pre_deploy (line 259) | def _pre_deploy(self):
class NutCracker (line 263) | class NutCracker(Base):
method __init__ (line 264) | def __init__(self, host, port, path, cluster_name, masters, mbuf=512,
method _alive (line 288) | def _alive(self):
method _gen_conf_section (line 291) | def _gen_conf_section(self, servers):
method _gen_conf (line 296) | def _gen_conf(self):
method _pre_deploy (line 325) | def _pre_deploy(self):
method version (line 333) | def version(self):
method _info_dict (line 338) | def _info_dict(self):
method reconfig (line 348) | def reconfig(self, masters, sentinels=None):
method logfile (line 357) | def logfile(self):
method cleanlog (line 360) | def cleanlog(self):
method signal (line 364) | def signal(self, signo):
method reload (line 369) | def reload(self):
method set_config (line 372) | def set_config(self, content):
FILE: tests/lib/utils.py
function getenv (line 28) | def getenv(key, default):
function strstr (line 43) | def strstr(s1, s2):
function lets_sleep (line 46) | def lets_sleep(SLEEP_TIME = 0.1):
function TT (line 49) | def TT(template, args): #todo: modify all
function TTCMD (line 52) | def TTCMD(template, args): #todo: modify all
function nothrow (line 59) | def nothrow(ExceptionToCheck=Exception, logger=None):
function test_nothrow (line 73) | def test_nothrow():
function json_encode (line 76) | def json_encode(j):
function json_decode (line 79) | def json_decode(j):
function system (line 86) | def system(cmd, log_fun=logging.info):
function shorten (line 91) | def shorten(s, l=80):
function assert_true (line 96) | def assert_true(a):
function assert_equal (line 99) | def assert_equal(a, b):
function assert_raises (line 102) | def assert_raises(exception_cls, callable, *args, **kwargs):
function assert_fail (line 111) | def assert_fail(err_response, callable, *args, **kwargs):
FILE: tests/test_memcache/test_gets.py
function setup (line 30) | def setup():
function teardown (line 40) | def teardown():
function getconn (line 46) | def getconn():
function test_basic (line 50) | def test_basic():
function test_mget_mset (line 64) | def test_mget_mset(kv=default_kv):
function test_mget_mset_large (line 77) | def test_mget_mset_large():
function test_mget_mset_key_not_exists (line 83) | def test_mget_mset_key_not_exists(kv=default_kv):
FILE: tests/test_redis/common.py
function setup (line 31) | def setup():
function teardown (line 39) | def teardown():
function getconn (line 49) | def getconn():
FILE: tests/test_redis/test_auth.py
function setup (line 22) | def setup():
function teardown (line 30) | def teardown():
function getconn (line 37) | def getconn():
function test_auth_basic (line 55) | def test_auth_basic():
function test_nopass_on_proxy (line 89) | def test_nopass_on_proxy():
function test_badpass_on_proxy (line 102) | def test_badpass_on_proxy():
function setup_and_wait (line 116) | def setup_and_wait():
FILE: tests/test_redis/test_basic.py
function test_setget (line 7) | def test_setget():
function test_msetnx (line 14) | def test_msetnx():
function test_null_key (line 22) | def test_null_key():
function test_ping_quit (line 34) | def test_ping_quit():
function test_slow_req_lua (line 44) | def test_slow_req_lua():
function test_fast_req_lua (line 50) | def test_fast_req_lua():
function disabled_test_slow_req (line 58) | def disabled_test_slow_req():
function test_signal (line 72) | def test_signal():
function test_nc_stats (line 94) | def test_nc_stats():
function test_issue_323 (line 133) | def test_issue_323():
function setup_and_wait (line 146) | def setup_and_wait():
FILE: tests/test_redis/test_commands.py
function test_linsert (line 6) | def test_linsert():
function test_exists (line 16) | def test_exists():
function test_lpush_lrange (line 23) | def test_lpush_lrange():
function test_hscan (line 34) | def test_hscan():
function test_hscan_large (line 48) | def test_hscan_large():
function test_zscan (line 78) | def test_zscan():
function test_sscan (line 91) | def test_sscan():
FILE: tests/test_redis/test_mget_large_binary.py
function setup (line 14) | def setup():
function teardown (line 21) | def teardown():
function test_mget_binary_value (line 27) | def test_mget_binary_value(cnt=5):
FILE: tests/test_redis/test_mget_mset.py
function test_mget_mset (line 5) | def test_mget_mset(kv=default_kv):
function test_mget_mset_on_key_not_exist (line 39) | def test_mget_mset_on_key_not_exist(kv=default_kv):
function test_mget_mset_large (line 78) | def test_mget_mset_large():
function test_mget_special_key (line 83) | def test_mget_special_key(cnt=5):
function test_mget_special_key_2 (line 93) | def test_mget_special_key_2(cnt=5):
function test_mget_on_backend_down (line 103) | def test_mget_on_backend_down():
function test_mset_on_backend_down (line 132) | def test_mset_on_backend_down():
function test_mget_pipeline (line 144) | def test_mget_pipeline():
function test_multi_delete_normal (line 179) | def test_multi_delete_normal():
function test_multi_delete_on_readonly (line 194) | def test_multi_delete_on_readonly():
function test_multi_delete_on_backend_down (line 207) | def test_multi_delete_on_backend_down():
function test_multi_delete_20140525 (line 238) | def test_multi_delete_20140525():
FILE: tests/test_redis/test_pipeline.py
function test_pipeline (line 5) | def test_pipeline():
function test_invalid_pipeline (line 25) | def test_invalid_pipeline():
function test_parse_error_raised (line 48) | def test_parse_error_raised():
FILE: tests/test_redis/test_protocol.py
function get_conn (line 5) | def get_conn():
function _test (line 11) | def _test(req, resp, sleep=0):
function test_slow (line 24) | def test_slow():
function test_pingpong (line 35) | def test_pingpong():
function test_quit (line 44) | def test_quit():
function test_quit_without_recv (line 53) | def test_quit_without_recv():
function _test_bad (line 64) | def _test_bad(req):
function test_badreq (line 72) | def test_badreq():
function test_wrong_argc (line 90) | def test_wrong_argc():
FILE: tests/test_system/test_reload.py
function _setup (line 38) | def _setup():
function _teardown (line 45) | def _teardown():
function get_tcp_conn (line 50) | def get_tcp_conn(host, port):
function send_cmd (line 56) | def send_cmd(s, req, resp):
function test_reload_with_old_conf (line 62) | def test_reload_with_old_conf():
function test_new_port (line 102) | def test_new_port():
function test_pool_add_del (line 131) | def test_pool_add_del():
Condensed preview — 123 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,048K chars).
[
{
"path": ".clang-format.example",
"chars": 2405,
"preview": "# This may be useful with https://clang.llvm.org/docs/ClangFormat.html#script-for-patch-reformatting\n# but some of clang"
},
{
"path": ".dockerignore",
"chars": 539,
"preview": "Dockerfile*\n**/Dockerfile*\n*.gz\nsrc/nutredis\nsrc/nutcracker\ntests/_binaries/*\n\n### Entries copied from .gitignore\n# pyc\n"
},
{
"path": ".editorconfig",
"chars": 464,
"preview": "# https://editorconfig.org/\n\nroot = true\n\n[*]\ntrim_trailing_whitespace = true\ninsert_final_newline = true\nend_of_lin"
},
{
"path": ".github/ISSUE_TEMPLATE.md",
"chars": 302,
"preview": "One line summary of the issue here.\n\n### Expected behavior\n\nAs concisely as possible, describe the expected behavior.\n\n#"
},
{
"path": ".github/PULL_REQUEST_TEMPLATE.md",
"chars": 424,
"preview": "Problem\n\nExplain the context and why you're making that change. What is the\nproblem you're trying to solve? In some cas"
},
{
"path": ".github/workflows/main.yml",
"chars": 1380,
"preview": "# This is a basic workflow to help you get started with Actions\n\nname: CI\n\n# Controls when the action will run.\non:\n # "
},
{
"path": ".gitignore",
"chars": 771,
"preview": "# pyc\n*.pyc\n\n# Compiled Object files\n*.lo\n*.o\n\n# Compiled Dynamic libraries\n*.so\n\n# Compiled Static libraries\n*.la\n*.a\n\n"
},
{
"path": "ChangeLog",
"chars": 7885,
"preview": " 2021-13-07 Tyson Andre <tysonandre775@hotmail.com>\n * twemproxy: version 0.5.0 release\n\t Same as 0.5.0-RC1\n\n 2021"
},
{
"path": "Dockerfile",
"chars": 188,
"preview": "FROM gcc\n\nCOPY . /usr/src/twemproxy\nWORKDIR /usr/src/twemproxy\nRUN \\\n autoreconf -h && \\\n autoreconf -fvi && \\\n ./con"
},
{
"path": "LICENSE",
"chars": 10173,
"preview": "\n Apache License\n Version 2.0, January 2004\n "
},
{
"path": "Makefile.am",
"chars": 242,
"preview": "MAINTAINERCLEANFILES = Makefile.in aclocal.m4 configure config.h.in config.h.in~ stamp-h.in\n\nACLOCAL_AMFLAGS = -I m4\n\nSU"
},
{
"path": "NOTICE",
"chars": 6308,
"preview": "twemproxy is a fast and lightweight proxy for memcached protocol\nCopyright (C) 2012 Twitter, Inc.\n\nPortions of twemproxy"
},
{
"path": "README.md",
"chars": 17604,
"preview": "# twemproxy (nutcracker) [![Build Status](https://github.com/twitter/twemproxy/actions/workflows/main.yml/badge.svg?bran"
},
{
"path": "ci/Dockerfile",
"chars": 2589,
"preview": "# Dockerfile to create a slower debug build of nutcracker with assertions enabled\n# for continuous integration checks.\n#"
},
{
"path": "ci/build-nutcracker.sh",
"chars": 749,
"preview": "#!/usr/bin/env bash\n\nset -xeu\nfunction cleanup {\n rm -f *.gz\n}\ntrap cleanup EXIT\ntrap cleanup INT\n\ncleanup\n\nexport CF"
},
{
"path": "conf/nutcracker.leaf.yml",
"chars": 209,
"preview": "leaf:\n listen: 127.0.0.1:22121\n hash: fnv1a_64\n distribution: ketama\n auto_eject_hosts: true\n server_retry_timeout:"
},
{
"path": "conf/nutcracker.root.yml",
"chars": 151,
"preview": "root:\n listen: 127.0.0.1:22120\n hash: fnv1a_64\n distribution: ketama\n preconnect: true\n auto_eject_hosts: false\n s"
},
{
"path": "conf/nutcracker.yml",
"chars": 1301,
"preview": "alpha:\n listen: 127.0.0.1:22121\n hash: fnv1a_64\n distribution: ketama\n auto_eject_hosts: true\n redis: true\n server"
},
{
"path": "configure.ac",
"chars": 6130,
"preview": "# Define the package version numbers and the bug reporting address\nm4_define([NC_MAJOR], 0)\nm4_define([NC_MINOR], 5)\nm4_"
},
{
"path": "contrib/Makefile.am",
"chars": 53,
"preview": "SUBDIRS = yaml-0.2.5\n\nEXTRA_DIST = yaml-0.2.5.tar.gz\n"
},
{
"path": "contrib/yaml-0.2.5/.gitignore",
"chars": 47,
"preview": "# Ignore everything\n*\n\n# Except me\n!.gitignore\n"
},
{
"path": "m4/.gitignore",
"chars": 47,
"preview": "# Ignore everything\n*\n\n# Except me\n!.gitignore\n"
},
{
"path": "man/nutcracker.8",
"chars": 2123,
"preview": ".TH NUTCRACKER 8 \"June 13, 2013\"\n.SH NAME\nnutcracker \\- Fast, light-weight proxy for memcached and Redis\n.SH SYNOPSIS\n.B"
},
{
"path": "notes/c-styleguide.txt",
"chars": 15568,
"preview": "- No literal tabs. Expand tabs to 4 spaces.\n- Indentation is 4 spaces.\n- No more than 3 levels of indentation, otherwise"
},
{
"path": "notes/debug.txt",
"chars": 6286,
"preview": "- strace\n strace -o strace.txt -ttT -s 1024 -p `pgrep nutcracker`\n\n- libyaml (yaml-0.2.5)\n\n - yaml tokens:\n\n 0 YAML_"
},
{
"path": "notes/memcache.md",
"chars": 9551,
"preview": "## Memcache Command Support\n\n### Request\n\n- Twemproxy implements only the memached ASCII commands\n- Binary commands are "
},
{
"path": "notes/recommendation.md",
"chars": 12500,
"preview": "If you are deploying nutcracker in your production environment, here are a few recommendations that might be worth consi"
},
{
"path": "notes/redis.md",
"chars": 71189,
"preview": "## Redis Command Support\n\n### Keys Command\n\n +-------------------+------------+--------------------------------------"
},
{
"path": "notes/socket.txt",
"chars": 6283,
"preview": "- int listen(int sockfd, int backlog);\n\n Linux: The backlog argument defines the maximum length to which the\n queue "
},
{
"path": "scripts/benchmark-mget.py",
"chars": 1142,
"preview": "#!/usr/bin/env python\n#coding: utf-8\n#file : test_mget.py\n#author : ning\n#date : 2014-04-01 13:15:48\n\nimport os\nimpo"
},
{
"path": "scripts/extract_redis_commands_argcounts.php",
"chars": 8194,
"preview": "#!/usr/bin/env php\n<?php\n/**\n * @author Tyson Andre\n *\n * Heuristics to extract commands from redis-doc and determine wh"
},
{
"path": "scripts/makesrpm.sh",
"chars": 6476,
"preview": "#!/bin/bash\n#-------------------------------------------------------------------------------\n# Create a source RPM packa"
},
{
"path": "scripts/multi_get.sh",
"chars": 526,
"preview": "#!/bin/sh\n\nport=22123\nsocatopt=\"-t 20 -T 20 -b 8193 -d -d \"\nkey=\"\"\nkeys=\"\"\nget_command=\"\"\n\n# build\nfor i in `seq 1 512`;"
},
{
"path": "scripts/nutcracker.init",
"chars": 1467,
"preview": "#! /bin/sh\n#\n# chkconfig: - 55 45\n# description: Twitter's twemproxy nutcracker\n# processname: nutcracker\n# config: /et"
},
{
"path": "scripts/nutcracker.init.debian",
"chars": 1828,
"preview": "#!/bin/sh\n### BEGIN INIT INFO\n# Provides: nutcracker\n# Required-Start: $network $remote_fs $local_fs\n# Requi"
},
{
"path": "scripts/nutcracker.spec",
"chars": 7579,
"preview": "Summary: Twitter's nutcracker redis and memcached proxy\nName: nutcracker\nVersion: 0.5.0\nRelease: 1\n\nURL: https://github."
},
{
"path": "scripts/pipelined_read.sh",
"chars": 496,
"preview": "#!/bin/sh\n\nsocatopt=\"-t 4 -T 4 -b 8193 -d -d \"\n\nget_commands=\"\"\n\n# build\nfor i in `seq 1 128`; do\n if [ `expr $i % 2`"
},
{
"path": "scripts/pipelined_write.sh",
"chars": 639,
"preview": "#!/bin/sh\n\nsocatopt=\"-t 1 -T 1 -b 16384\"\n\nval=`echo 6^6^6 | bc`\nval=`printf \"%s\" \"${val}\"`\nvallen=`printf \"%s\" \"${val}\" "
},
{
"path": "scripts/populate_memcached.sh",
"chars": 495,
"preview": "#!/bin/sh\n\nport=22123\nsocatopt=\"-t 1 -T 1 -b 65537\"\n\nval=`echo 6^6^6 | bc`\nval=`printf \"%s\\r\\n\" \"${val}\"`\nvallen=`printf"
},
{
"path": "scripts/redis-check.py",
"chars": 665,
"preview": "import redis\n\nrange=100\nfactor=32\nport=22121\n\nr = redis.StrictRedis(host='localhost', port=port, db=0)\n\n# lrange\nprint ["
},
{
"path": "scripts/redis-check.sh",
"chars": 52914,
"preview": "#!/bin/sh\n\nport=6379\nport=22121\n\ndebug=\"-v -d\"\ndebug=\"-d\"\n\ntimeout=\"-t 1\"\ntimeout=\"\"\n\n# keys\n\nprintf '\\ndel\\n'\nprintf '*"
},
{
"path": "src/Makefile.am",
"chars": 2487,
"preview": "MAINTAINERCLEANFILES = Makefile.in\n\nAM_CPPFLAGS =\nif !OS_SOLARIS\nAM_CPPFLAGS += -D_GNU_SOURCE\nendif\nAM_CPPFLAGS += -I $("
},
{
"path": "src/event/Makefile.am",
"chars": 279,
"preview": "MAINTAINERCLEANFILES = Makefile.in\n\nAM_CPPFLAGS = -I $(top_srcdir)/src\n\nAM_CFLAGS = -Wall -Wshadow\nAM_CFLAGS += -Wno-unu"
},
{
"path": "src/event/nc_epoll.c",
"chars": 7596,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/event/nc_event.h",
"chars": 2711,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/event/nc_evport.c",
"chars": 10954,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2013 Twitter, Inc.\n *\n * License"
},
{
"path": "src/event/nc_kqueue.c",
"chars": 10605,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/Makefile.am",
"chars": 414,
"preview": "MAINTAINERCLEANFILES = Makefile.in\n\nAM_CPPFLAGS = -I $(top_srcdir)/src\n\nAM_CFLAGS = -Wall -Wshadow\nAM_CFLAGS += -Wno-unu"
},
{
"path": "src/hashkit/nc_crc16.c",
"chars": 3072,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_crc32.c",
"chars": 4959,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_fnv.c",
"chars": 1960,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_hashkit.h",
"chars": 3237,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_hsieh.c",
"chars": 2348,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_jenkins.c",
"chars": 7399,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_ketama.c",
"chars": 8682,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_md5.c",
"chars": 9542,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_modula.c",
"chars": 5305,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_murmur.c",
"chars": 2342,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_one_at_a_time.c",
"chars": 1377,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/hashkit/nc_random.c",
"chars": 4894,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc.c",
"chars": 15053,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_array.c",
"chars": 3706,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_array.h",
"chars": 2049,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_client.c",
"chars": 4860,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_client.h",
"chars": 932,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_conf.c",
"chars": 43961,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_conf.h",
"chars": 6159,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_connection.c",
"chars": 12558,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_connection.h",
"chars": 5181,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_core.c",
"chars": 9060,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_core.h",
"chars": 4152,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_log.c",
"chars": 7245,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_log.h",
"chars": 5598,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_mbuf.c",
"chars": 6869,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_mbuf.h",
"chars": 2074,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_message.c",
"chars": 22724,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_message.h",
"chars": 26880,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_proxy.c",
"chars": 11323,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_proxy.h",
"chars": 1133,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_queue.h",
"chars": 39228,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_rbtree.c",
"chars": 8588,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_rbtree.h",
"chars": 1704,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_request.c",
"chars": 20206,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_response.c",
"chars": 10117,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_server.c",
"chars": 23567,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_server.h",
"chars": 6579,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_signal.c",
"chars": 3101,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_signal.h",
"chars": 931,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_stats.c",
"chars": 28872,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_stats.h",
"chars": 10238,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_string.c",
"chars": 7653,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_string.h",
"chars": 4845,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_util.c",
"chars": 13333,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/nc_util.h",
"chars": 6992,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/proto/Makefile.am",
"chars": 269,
"preview": "MAINTAINERCLEANFILES = Makefile.in\n\nAM_CPPFLAGS = -I $(top_srcdir)/src\n\nAM_CFLAGS = -Wall -Wshadow\nAM_CFLAGS += -Wno-unu"
},
{
"path": "src/proto/nc_memcache.c",
"chars": 43047,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/proto/nc_proto.h",
"chars": 8599,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/proto/nc_redis.c",
"chars": 94334,
"preview": "/*\n * twemproxy - A fast and lightweight proxy for memcached protocol.\n * Copyright (C) 2011 Twitter, Inc.\n *\n * License"
},
{
"path": "src/test_all.c",
"chars": 36530,
"preview": "#include <nc_hashkit.h>\n#include <nc_conf.h>\n#include <nc_util.h>\n#include <proto/nc_proto.h>\n#include <stdio.h>\n\nstatic"
},
{
"path": "test_in_docker.sh",
"chars": 1154,
"preview": "#!/usr/bin/env bash\n# Main ci script for nutcracker tests\nset -xeu\n\nfunction print_usage() {\n echo \"Usage: $0 [REDIS_"
},
{
"path": "tests/.gitignore",
"chars": 18,
"preview": "*.pyc\n*.out\n*.log\n"
},
{
"path": "tests/README.rst",
"chars": 2196,
"preview": "Python testing facilities for twemproxy, this test suite is based on https://github.com/idning/redis-mgr\n\nTesting in doc"
},
{
"path": "tests/conf/conf.py",
"chars": 419,
"preview": "import os\nimport sys\n\nPWD = os.path.dirname(os.path.realpath(__file__))\nWORKDIR = os.path.join(PWD, '../')\n\nBINARYS = {"
},
{
"path": "tests/conf/control.sh",
"chars": 426,
"preview": "#!/bin/bash\n\nstart() \n{\n stop\n ulimit -c unlimited\n\n pushd . > /dev/null\n\n cd `dirname $$0`\n ${startcmd}\n"
},
{
"path": "tests/conf/redis.conf",
"chars": 30369,
"preview": "\n# Redis configuration file example\n# Note on units: when memory size is needed, it is possible to specify\n# it in the u"
},
{
"path": "tests/conf/sentinel.conf",
"chars": 1819,
"preview": "# Example sentinel.conf\n\n# port <sentinel-port>\n# The port that this sentinel instance will run on\nport ${port}\n\n# By de"
},
{
"path": "tests/lib/server_modules.py",
"chars": 12754,
"preview": "#!/usr/bin/env python\n#coding: utf-8\n#file : server_modules.py\n#author : ning\n#date : 2014-02-24 13:00:28\n\nimport os"
},
{
"path": "tests/lib/utils.py",
"chars": 3065,
"preview": "import os\nimport re\nimport sys\nimport time\nimport copy\nimport _thread\nimport socket\nimport threading\nimport logging\nimpo"
},
{
"path": "tests/log/.gitignore",
"chars": 6,
"preview": "*.log\n"
},
{
"path": "tests/nosetests_verbose.sh",
"chars": 269,
"preview": "#!/bin/bash -xeu\n# A simple utility script to run tests with extremely verbose output.\n\nif [[ $# == 0 ]]; then\n\techo \"Us"
},
{
"path": "tests/test_memcache/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_memcache/test_gets.py",
"chars": 2465,
"preview": "#!/usr/bin/env python3\n\nimport os\nimport sys\nimport redis\nimport memcache\n\nPWD = os.path.dirname(os.path.realpath(__file"
},
{
"path": "tests/test_redis/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_redis/common.py",
"chars": 1354,
"preview": "#!/usr/bin/env python3\n\nimport os\nimport sys\nimport redis\nimport time\n\nPWD = os.path.dirname(os.path.realpath(__file__))"
},
{
"path": "tests/test_redis/test_auth.py",
"chars": 4199,
"preview": "#!/usr/bin/env python3\n\nfrom .common import *\n\nall_redis = [\n RedisServer('127.0.0.1', 2100, '/tmp/r/redis-2100/',\n "
},
{
"path": "tests/test_redis/test_basic.py",
"chars": 3895,
"preview": "#!/usr/bin/env python\n#coding: utf-8\n\nfrom nose.tools import nottest\nfrom .common import *\n\ndef test_setget():\n r = g"
},
{
"path": "tests/test_redis/test_commands.py",
"chars": 2337,
"preview": "#!/usr/bin/env python\n#coding: utf-8\n\nfrom .common import *\n\ndef test_linsert():\n r = getconn()\n\n r.rpush('mylist'"
},
{
"path": "tests/test_redis/test_mget_large_binary.py",
"chars": 915,
"preview": "#!/usr/bin/env python\n#coding: utf-8\n\nfrom .common import *\nfrom .test_mget_mset import test_mget_mset as _mget_mset\n\n#f"
},
{
"path": "tests/test_redis/test_mget_mset.py",
"chars": 6532,
"preview": "#!/usr/bin/env python3\n\nfrom .common import *\n\ndef test_mget_mset(kv=default_kv):\n r = getconn()\n\n def insert_by_p"
},
{
"path": "tests/test_redis/test_pipeline.py",
"chars": 1534,
"preview": "#!/usr/bin/env python3\n\nfrom .common import *\n\ndef test_pipeline():\n r = getconn()\n\n pipe = r.pipeline(transaction"
},
{
"path": "tests/test_redis/test_protocol.py",
"chars": 2215,
"preview": "#!/usr/bin/env python3\nfrom .common import *\nfrom pprint import pprint\n\ndef get_conn():\n s = socket.socket(socket.AF_"
},
{
"path": "tests/test_system/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_system/test_reload.py",
"chars": 5087,
"preview": "#!/usr/bin/env python3\n#file : test_reload.py\n#author : ning\n#date : 2014-09-03 12:28:16\n\nimport os\nimport sys\nimpor"
}
]
About this extraction
This page contains the full source code of the twitter/twemproxy GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 123 files (975.9 KB), approximately 265.5k tokens, and a symbol index with 1129 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.