Showing preview only (833K chars total). Download the full file or copy to clipboard to get everything.
Repository: circonus-labs/fq
Branch: master
Commit: aa2bb0149df7
Files: 78
Total size: 800.6 KB
Directory structure:
gitextract__9lkm2sj/
├── .gitignore
├── ChangeLog.md
├── LICENSE
├── Makefile
├── README.md
├── coverity_model.c
├── docs/
│ └── fq_protocol.md
├── dtest.d
├── fq.d
├── fq.h
├── fq_bench.c
├── fq_client.c
├── fq_dtrace.blank.h
├── fq_dtrace.d
├── fq_msg.c
├── fq_rcvr.c
├── fq_sndr.c
├── fq_utils.c
├── fqc.c
├── fqd.c
├── fqd.h.in
├── fqd_ccs.c
├── fqd_config.c
├── fqd_dss.c
├── fqd_dyn_sample.c
├── fqd_http.c
├── fqd_listener.c
├── fqd_peer.c
├── fqd_private.h
├── fqd_prog.c
├── fqd_queue.c
├── fqd_queue_jlog.c
├── fqd_queue_mem.c
├── fqd_routemgr.c
├── fqs.c
├── fqtool.c
├── http_parser.c
├── http_parser.h
├── java/
│ ├── Makefile
│ ├── fq_rcvr.java
│ ├── pom.xml
│ └── src/
│ ├── com/
│ │ └── omniti/
│ │ └── labs/
│ │ ├── FqClient.java
│ │ ├── FqClientImplDebug.java
│ │ ├── FqClientImplInterface.java
│ │ ├── FqClientImplNoop.java
│ │ ├── FqCommand.java
│ │ ├── FqCommandProtocolError.java
│ │ ├── FqDataProtocolError.java
│ │ ├── FqHeartbeatException.java
│ │ └── FqMessage.java
│ └── main/
│ └── java/
│ └── com/
│ └── circonus/
│ ├── FqClient.java
│ ├── FqClientImplDebug.java
│ ├── FqClientImplInterface.java
│ ├── FqClientImplNoop.java
│ ├── FqCommand.java
│ ├── FqCommandProtocolError.java
│ ├── FqDataProtocolError.java
│ ├── FqHeartbeatException.java
│ └── FqMessage.java
├── lua/
│ ├── fq-proxy
│ ├── fq-receiver
│ ├── fq-sender
│ ├── fqclient.lua.tail
│ └── generatelua.sh
├── service-configs/
│ ├── 50-circonus-fq.preset
│ ├── circonus-fq.service
│ └── daemon_options
├── test/
│ ├── lua-support/
│ │ └── init.lua
│ ├── run-tests.sh
│ └── test_spec.lua
└── web/
├── css/
│ ├── bootstrap-theme.css
│ ├── bootstrap.css
│ ├── colorbrewer.css
│ └── theme.css
├── index.html
└── js/
├── bootstrap.js
├── colorbrewer.js
└── fq.js
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
*.a
*.o
*.lo
*.so*
fqd.h
fqd
fqc
fqs
fqtool
fq_bench
fq_dtrace.h
fq_rcvr
fq_sndr
lua/fqclient.lua
Makefile.build
Makefile.depend
*.dSYM/
.*.swp
ck-*/Makefile
ck-*/build/ck.spec
ck-*/doc/Makefile
java/classes/
java/*.class
java/fqclient.jar
*~
.autotools
.cproject
.project
.settings/
test/fqd.sqlite
test/out.log
test/test_detail.xml
================================================
FILE: ChangeLog.md
================================================
# ChangeLog
### v0.13.11
* Fix test issue where libfq could not be found.
* Minor change to compiler optimization flag. Level `O5` does not exist and is
effectively `O3`.
* Update systemd install path. The new path is equivalent to the old, but
avoids potential issues with `/lib` being a symbolic link on popular Linux
distros.
* Fix potential race in host resolution.
### v0.13.10
* Correct a build issue when libbcd support is disabled.
* Fix race condition in client connection status.
### v0.13.9
* Fix file descriptor leak on connection error.
* Correct usage of `pthread_*` return values. Remove invalid use of `volatile`.
* Add missing `volatile` in Java client, and replace `LinkedList` with
`ArrayDeque`.
### v0.13.8
* Upgrade jquery to 3.5.1 for FQ user interface
### v0.13.7
* Queue drops are tracked as `dropped_to`.
* Web UI updated to display queue drops/rate.
* -b deprecated, -B added, and BCD is disabled by default.
### v0.13.6
* Track drops to queues as `dropped_in` in status.
### v0.13.5
* Force disconnect on message read/write error.
* Reuse listener threads.
### v0.13.4
* Various code cleanups.
* Better bounds checking on auth handshake (allow full size).
* Fix BCD integration.
### v0.13.3
* Set `SO_REUSEPORT = 1` for listener.
* Add `-b` to disable BCD/backtrace integration.
### v0.13.2
* Name threads on Linux to aid debugging.
* Prevent abort when queue removal fails.
### v0.13.1
* Add libbcd support for catching faults with backtrace.
### v0.13.0
* Place fq modules in $(LIBEXECDIR)/fq
* Automatically load all available modules
### v0.12.1
* Move the `valnode_t` definition into fq.h.
* Fix hex construction macro.
* Support var-args in loadable program functions.
* Fix multi-argument parsing in routing grammar.
### v0.12.0
* Omit unneeded library dependencies on Illumos.
* Make poll() calls resume after signal interruption.
### v0.11.0
* Use socket keep-alives for client/server connections.
* Fix use-after-free bug in lua ffi client bindings.
* Fix test suite.
* Explicit registration of local function to better navigate
changing dlsym "self" personalities.
* ENABLE_DTRACE=1 Linux build flag.
### v0.10.14
* Fixes to fq-client.lua on OmniOS
### v0.10.13
* Add `fqs` tool for sending messages from stdin
* Test suite utilizing `mtevbusted` from
[libmtev](https://github.com/circonus-labs/libmtev/) (PR #37)
### v0.10.12
* Fix misuse of stack for freeing messages (0.10.11 fix was bad).
* Add Linux futex support for lower-latency idle network wake-up.
* Ensure message ordering on per-client data connections.
### v0.10.11
* Fix crash when shutting down client that has never seen a message.
### v0.10.10
* Fix source management issue. 0.10.9 tag exluded commits.
* Change message free-lists to prevent use-after-free on thread exit.
### v0.10.9
* Fix builds on newer Mac OS X
* Change message free-lists to prevent use-after-free on thread exit.
* Fix bug in server->client heartbeats not beating.
### v0.10.8
* Fix querystring parsing crash when parameters are not k=v form.
* Resume on-disk queues at the right checkpoint location.
### v0.10.7
* Fix bug in route binding prefix matching causing misdirected messages
(clients could get more than they asked for).
* Fix bug on some Linux systems regarding exposed symbols.
### v0.10.6
* Fix crashing issue enqueueing messages due to unsafe use of spsc fifos.
* Add dynamic loading of routing program extensions.
* Move the "sample" function to a dynamic extension for example purposes.
================================================
FILE: LICENSE
================================================
Copyright (c) 2013 OmniTI Computer Consulting, Inc.
All rights reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to
deal in the Software without restriction, including without limitation the
rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
sell copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
IN THE SOFTWARE.
================================================
FILE: Makefile
================================================
# If you want a verbose make (visible commands) add V=1 to you invocation
.SUFFIXES: .lo
CC=gcc
LD=gcc
LN_S=ln -s
COPT=-O3
TAR=tar
SED=sed
PREFIX=/usr/local
INCLUDEDIR=$(PREFIX)/include
LIBDIR=$(PREFIX)/lib
LIBEXECDIR=$(PREFIX)/libexec
BINDIR=$(PREFIX)/bin
SBINDIR=$(PREFIX)/sbin
VARLIBFQ=$(PREFIX)/var/lib/fq
LUADIR=$(PREFIX)/share/lua/5.1
INSTALL=install
SHLD=$(LD) -shared
MODULELD=$(LD) -shared
LIBEXT=so
SHCFLAGS=-fPIC
DTRACE=/usr/sbin/dtrace
OS=$(shell uname)
FQ_MAJOR=0
FQ_MINOR=13
FQ_MICRO=12
Q=
ifeq ($(V),)
Q=@
endif
VENDOR_CFLAGS=
VENDOR_LDFLAGS=
DTRACEFLAGS=
EXTRA_CFLAGS=$(VENDOR_CFLAGS) -g -D_REENTRANT -std=gnu99 -pedantic -Wall
EXTRA_CFLAGS+=-DVARLIBFQDIR=\"$(VARLIBFQ)\"
EXTRA_CFLAGS+=-DLIBEXECDIR=\"$(LIBEXECDIR)/fq\"
#EXTRA_CFLAGS+=-DDEBUG
CLIENT_OBJ=fq_client.o fq_msg.o fq_utils.o
CLIENT_OBJ_LO=$(CLIENT_OBJ:%.o=%.lo)
FQD_OBJ=fqd.o fqd_listener.o fqd_ccs.o fqd_dss.o fqd_config.o \
fqd_queue.o fqd_routemgr.o fqd_queue_mem.o fqd_queue_jlog.o \
fqd_http.o fqd_prog.o fqd_peer.o http_parser.o \
$(CLIENT_OBJ)
FQC_OBJ=fqc.o $(CLIENT_OBJ)
FQD_SAMPLE_OBJ=fqd_dyn_sample.lo
FQD_DTRACE_OBJ=
skip_bcd=$(NO_BCD)
ifdef skip_bcd
FQDLIBS=-ljlog -lsqlite3
EXTRA_CFLAGS+=-DNO_BCD
else
FQDLIBS=-ljlog -lsqlite3 -lbcd
endif
LIBS+=-lck
SHLDFLAGS=
ifeq ($(OS),SunOS)
SHLDFLAGS+=-R$(LIBDIR)
LIBS+=-lcrypto -lsocket -lnsl -lumem -luuid
LIBLIBS+=-luuid -lsocket -lnsl
EXTRA_CFLAGS+=-D_XOPEN_SOURCE=600
EXTRA_CFLAGS+=-D_BSD_SOURCE
EXTRA_CFLAGS+=-D__EXTENSIONS__ -DHAVE_UINTXX_T -DSIZEOF_LONG_LONG_INT=8 -m64 -D_REENTRANT -DHAVE_GETHOSTBYNAME_R
EXTRA_SHLDFLAGS=-m64
FQD_DTRACE_OBJ=fq_dtrace.o
DTRACEFLAGS=-xnolibs
else
ifeq ($(OS),Darwin)
MODULELD=ld -bundle
LOADER=-bundle_loader fqd -lc
EXTRA_CFLAGS+=-D_DARWIN_C_SOURCE -DHAVE_U_INTXX_T -DHAVE_INTXX_T -DHAVE_U_INT64_T -DHAVE_INT64_T \
-Wno-dollar-in-identifier-extension -Wno-gnu-statement-expression -Wno-deprecated-declarations
#EXTRA_CFLAGS+=-Weverything
LIBEXT=dylib
else
ifeq ($(OS),Linux)
EXTRA_CFLAGS+=-D_XOPEN_SOURCE=600
EXTRA_CFLAGS+=-D_DEFAULT_SOURCE -DBYTE_ORDER=__BYTE_ORDER -DBIG_ENDIAN=__BIG_ENDIAN
SHLDFLAGS+=-Wl,-rpath=$(LIBDIR)
LDFLAGS+=-rdynamic -export-dynamic
LIBS+=-lcrypto -lpthread -ldl -luuid -lrt
LIBLIBS+=-lpthread -luuid -lrt
DTRACE=
ifeq ($(ENABLE_DTRACE),1)
DTRACE=/bin/dtrace
FQD_DTRACE_OBJ=fq_dtrace.o
endif
else
ifeq ($(OS),FreeBSD)
SHLDFLAGS+=-Wl,-rpath=$(LIBDIR)
LDFLAGS+=-rdynamic
LIBS+=-lcrypto -lpthread -luuid -lexecinfo
LIBLIBS+=-lpthread -luuid -lexecinfo
FQD_DTRACE_OBJ=fq_dtrace.o
endif
endif
endif
endif
all: libfq.$(LIBEXT) libfq.a fqd fqc fqs fqtool fq_sndr fq_rcvr fq_bench \
fq-sample.so lua/fqclient.lua
include Makefile.depend
SHLDFLAGS+=$(VENDOR_LDFLAGS) -L$(LIBDIR)
ifeq ($(OS),Darwin)
SHLDFLAGS+=-current_version $(FQ_MAJOR).$(FQ_MINOR).$(FQ_MICRO) -install_name $(LIBDIR)/libfq.$(FQ_MAJOR).dylib
SOLONG=libfq.$(FQ_MAJOR).$(FQ_MINOR).$(FQ_MICRO).dylib
SOSHORT=libfq.$(FQ_MAJOR).dylib
LIBNAME=libfq.dylib
else
SHLDFLAGS+=-Wl,-soname,libfq.so.$(FQ_MAJOR)
SOLONG=libfq.so.$(FQ_MAJOR).$(FQ_MINOR).$(FQ_MICRO)
SOSHORT=libfq.so.$(FQ_MAJOR)
LIBNAME=libfq.so
endif
CFLAGS+=$(EXTRA_CFLAGS)
SHCFLAGS+=$(EXTRA_CFLAGS)
LDFLAGS+=$(VENDOR_LDFLAGS)
fqd.h: fqd.h.in
sed -e 's/@@FQ_MAJOR@@/'$(FQ_MAJOR)'/g;' \
-e 's/@@FQ_MINOR@@/'$(FQ_MINOR)'/g;' \
-e 's/@@FQ_MICRO@@/'$(FQ_MICRO)'/g;' < fqd.h.in > fqd.h
fq_dtrace.h: fq_dtrace.d
-$(DTRACE) $(DTRACEFLAGS) -h -o $@ -s $<
if [ ! -f $@ ]; then cp fq_dtrace.blank.h $@; fi
fq_dtrace.o: $(FQD_OBJ)
$(DTRACE) $(DTRACEFLAGS) -64 -G -s fq_dtrace.d -o $@ $(FQD_OBJ)
fq_dtrace.blank.h: fq_dtrace.h
awk 'BEGIN{print "#if 0"} /#else/,/#endif/{print}' $< > $@
fqd: $(FQD_OBJ) $(FQD_DTRACE_OBJ)
@echo " - linking $@"
$(Q)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(FQD_OBJ) $(FQD_DTRACE_OBJ) $(LIBS) $(FQDLIBS)
fqc: $(FQC_OBJ)
@echo " - linking $@"
$(Q)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(FQC_OBJ) $(LIBS)
fq-sample.so: fqd $(FQD_SAMPLE_OBJ)
$(Q)$(MODULELD) $(LOADER) $(EXTRA_SHLDFLAGS) -o $@ $(FQD_SAMPLE_OBJ)
fq_sndr: fq_sndr.o libfq.a
@echo " - linking $@"
$(Q)$(CC) $(CFLAGS) $(LDFLAGS) -L. -lfq -o $@ $^ $(LIBS)
fqs: fqs.o libfq.a
@echo " - linking $@"
$(Q)$(CC) $(CFLAGS) $(LDFLAGS) -L. -lfq -o $@ $^ $(LIBS)
fq_rcvr: fq_rcvr.o libfq.a
@echo " - linking $@"
$(Q)$(CC) $(CFLAGS) $(LDFLAGS) -L. -lfq -o $@ $^ $(LIBS)
fqtool: fqtool.o libfq.a
@echo " - linking $@"
$(Q)$(CC) $(CFLAGS) $(LDFLAGS) -L. -lfq -o $@ $^ $(LIBS)
fq_bench: fq_bench.o libfq.a
@echo " - linking $@"
$(Q)$(CC) $(CFLAGS) $(LDFLAGS) -L. -lfq -o $@ $^ $(LIBS)
libfq.$(LIBEXT): $(CLIENT_OBJ_LO)
@echo " - creating $@"
$(Q)$(SHLD) $(EXTRA_SHLDFLAGS) $(SHLDFLAGS) -o $@ $(CLIENT_OBJ_LO) $(LIBLIBS)
$(LN_S) -f $@ $(SOSHORT)
libfq.a: $(CLIENT_OBJ)
@echo " - creating $@"
$(Q)ar cr $@ $(CLIENT_OBJ)
.c.o: $<
@echo " - compiling $<"
$(Q)$(CC) $(CPPFLAGS) $(CFLAGS) $(COPT) -o $@ -c $<
.c.lo: $<
@echo " - compiling $<"
$(Q)$(CC) $(CPPFLAGS) $(SHCFLAGS) -o $@ -c $<
Makefile.depend: fq_dtrace.h fqd.h
@echo " - make depend"
$(Q)$(CC) $(CPPFLAGS) $(CFLAGS) -MM *.c > Makefile.depend
java/fqclient.jar:
(cd java && $(MAKE) fqclient.jar)
lua/fqclient.lua:
(cd lua; ./generatelua.sh)
install: all
$(INSTALL) -d $(DESTDIR)$(INCLUDEDIR)
$(INSTALL) -m 0444 fq.h $(DESTDIR)$(INCLUDEDIR)/fq.h
$(INSTALL) -d $(DESTDIR)$(LIBDIR)
$(INSTALL) -m 0444 libfq.a $(DESTDIR)$(LIBDIR)/libfq.a
$(INSTALL) -m 0555 libfq.$(LIBEXT) $(DESTDIR)$(LIBDIR)/$(SOLONG)
$(LN_S) -f $(SOLONG) $(DESTDIR)$(LIBDIR)/$(SOSHORT)
$(LN_S) -f $(SOLONG) $(DESTDIR)$(LIBDIR)/$(LIBNAME)
$(INSTALL) -d $(DESTDIR)$(LIBEXECDIR)/fq
$(INSTALL) -m 0555 fq-sample.so $(DESTDIR)$(LIBEXECDIR)/fq/fq-sample.so
$(INSTALL) -d $(DESTDIR)$(BINDIR)
$(INSTALL) -m 0555 fqtool $(DESTDIR)$(BINDIR)/fqtool
$(INSTALL) -m 0555 fqs $(DESTDIR)$(BINDIR)/fqs
$(INSTALL) -d $(DESTDIR)$(SBINDIR)
$(INSTALL) -m 0555 fqd $(DESTDIR)$(SBINDIR)/fqd
$(INSTALL) -d $(DESTDIR)$(VARLIBFQ)
$(TAR) cf - web | (cd $(DESTDIR)$(VARLIBFQ) && $(TAR) xf -)
$(INSTALL) -d $(DESTDIR)/usr/lib/dtrace
$(INSTALL) -m 0444 fq.d $(DESTDIR)/usr/lib/dtrace/fq.d
$(INSTALL) -d $(DESTDIR)$(LUADIR)
$(INSTALL) -m 0644 lua/fqclient.lua $(DESTDIR)$(LUADIR)
$(INSTALL) -m 0555 lua/fq-sender lua/fq-receiver lua/fq-proxy $(DESTDIR)$(BINDIR)
install-systemd: install
$(INSTALL) -d $(DESTDIR)/usr/lib/systemd/system
$(INSTALL) -d $(DESTDIR)/usr/lib/systemd/system-preset
$(INSTALL) -m 0644 service-configs/circonus-fq.service $(DESTDIR)/usr/lib/systemd/system/circonus-fq.service
$(INSTALL) -m 0644 service-configs/50-circonus-fq.preset $(DESTDIR)/usr/lib/systemd/system-preset/50-circonus-fq.preset
$(INSTALL) -m 0644 service-configs/daemon_options $(DESTDIR)$(VARLIBFQ)/daemon_options
clean:
rm -f *.o *.a fqc fqd fqs *.$(LIBEXT) $(SOSHORT) fq_dtrace.h lua/fqclient.lua
.PHONY: test
test: lua/fqclient.lua
./test/run-tests.sh
================================================
FILE: README.md
================================================
# fq.
<a href="https://scan.coverity.com/projects/circonus-labs-fq">
<img alt="Coverity Scan Build Status"
src="https://scan.coverity.com/projects/13357/badge.svg"/>
</a>
fq is a *brokered* message queue using a publish subscribe model. It is architected for performance and isn't (today) designed for large numbers of connected clients.
+------------+ +-----------+
|- exchange -|<-- (msg publication) --|- client0 -|
+------------+ +-----------+
|- routemap -|
+------------+
| | +---------+
| +--------------|- queue -|
| +---------+
+---------+ |
|- queue -| | +-----------+
+---------+ +---|- client1 -|
| +-----------+
| +-----------+
+--|- client2 -|
| +-----------+
|
+-----------+
|- client3 -|
+-----------+
## Terminology
### Broker
The `fqd` process. The daemon through which all knowledge passes.
### Peers
Peers are connected `fqd` processes. It is important to note that peers are unidirectional. If A peers with B, then A will act as a client to B. If you want bidirectional peering, you must specify that A peers with B and B peers with A. The system aims to prevent cyclic delivery of messages efficiently.
Adding peers is done directly via fqd's sqlite DB store:
```
; sqlite3 /var/lib/fq/fqd.sqlite
sqlite> INSERT INTO "upstream"
(host, port, source, password, exchange, program, permanent_binding)
VALUES('peerB',8765,'fqd-peera//mem:drop,private,backlog=4096','none','logging','prefix:"http.access.json."','false');
```
### Client
* [C client - libfq](https://github.com/postwait/fq/blob/master/fq.h#L164-L205)
* [Java client - fq.jar](https://github.com/postwait/fq/blob/master/java/src/com/omniti/labs/FqClientImplInterface.java)
* [Node.js client - fq](https://www.npmjs.com/package/fq)
* [Go client - fq](https://godoc.org/github.com/postwait/gofq)
* submission-only /submit API (see below)
A client is an applications connection to fq over TCP/IP to send or receive messages. A client makes two TCP/IP connections to fq. An application can present itself to fq as multiple clients at one time (by opening new pairs of connections). See Queues for reasons why.
### Exchanges
Exchanges are like buses on which messages may be sent. You cannot send a message without doing so on an exchange. Exchanges are created within fq on-demand.
### Queues
Queues are queues. If you stick something in one end, you should expect it to come out the other. A single queue may have multiple clients subscribed. When a client connects, it is attached to one and only one queue. If an application wishes to attach to more than one queue, it should present as multiple clients. Queues use a competitive consumption model meaning that if multiple clients are attached to a single queue, the messages sent to that queue will be distributed over the clients such that no two clients will see the same message.
#### Queue Types
Queues can be of type `mem` or `disk`. The contents of memory queues will not survive restarts.
Various parameters can be set on a queue using the syntax `type:param1,param2`.
#### Sharing
Queues with the `public` parameter can have multiple clients connected to them (in which case they compete for messages). If you want a private queue you can specify the `private` parameter.
#### Policy
Queues can either have a `block` or `drop` policy. The drop policy means that messages that would be routed to a queue that is full will be dropped and never delivered. The block policy will cause the publisher to wait until there is room in the queue. The block policy makes no sense on a disk queue.
#### Backlog
The `backlog=<number>` parameter will specify how many messages may be held in the queue before the block or drop policies are applied.
#### Permanence
If you want a queue to be remembered by fqd, you can specify `permanent` as a flag. If you'd like for fqd to forget the queue after all clients have disconnected, you can specify the `transient` flag. If neither flag is specified, then an existing queue will retain its previous permanence setting or a new transient queue will be created.
#### Examples:
A queue called `bob` will be in memory, allowed to have multiple clients connected to it, with a drop policy and an allowable message backlog of 100000 messages: `bob/mem:public,drop,backlog=100000`
A connection client will specify username/queue. A user "USER" connecting to the aforementioned queue would connect as `USER/bob/mem:public,drop,backlog=100000`
### Messages
Messages are, of course, a payload and metadata.
#### Message metadata
Some are set by the broker.
* sender [set by the broker]
* hops (a list of fqd via which the message passed)
Others are set by the sender.
* exchange (up to 127 bytes)
* route (up to 127 bytes)
* id (128 bits). The first 64 bits the sender shall control, the latter 64bits the broker *might* control.
### Routes and Programs
Routes and programs define how messages sent on exchanges are placed in queues:
- A receiver that connects to an fq-broker specifies a program that filters the messages on the exchange.
- A sender specifies a route for every message as part of the metadata
Programs follow the following syntax (cf. `fqd.h`):
```
PROGRAM: <prefix|exact>:string RULES*
RULE: (RULE)
RULE: (RULE && RULE)
RULE: (RULE || RULE)
RULE: EXPR
EXPR: function(args)
args: arg
args: arg, args
arg: "string"
arg: true|false
arg: [0-9][0-9]*(?:.[0-9]*)
functions are dynamically loadable with type signature
strings: s, booleans: b, integers: d
function: substr_eq(9.3,10,"tailorings",true)
C symbol: fqd_route_prog__substr_eq__ddsb(int nargs, valnode_t *args);
```
In particular:
- Every program starts with either `prefix:` or `exact:`
- The program `prefix:` matches all rules
- The program string is matched against the message route
The following rule functions are defined in `fq_prog.c`:
- `fqd_route_prog__sample__d()` -- subsample the stream
- `fqd_route_prog__route_contains__s()` -- check if route contains a string
- `fqd_route_prog__payload_prefix__s()` -- check if payload starts with prefix
- `fqd_route_prog__payload_contains__s()` -- check if payload contains a string
- `fqd_route_prog__true__()` -- always true
Examples:
- `prefix:` -- matches all messages
- `prefix:bla` or `prefix:"bla"` -- matches all messages with rules starting with the sting 'bla'
- `prefix: payload_prefix("M")` -- matches messages where the payload starts with 'M'
- `prefix:foo (payload_prefix("M") && route_contains("bar"))` -- matches messages where the payload starts with 'M' and route starts with "foo" and moreover contains "bar"
## Protocol
Information on command and message protocol is found in `docs/fq_protocol.md`
### HTTP superposition
The Fq protocol also acts as a non-compliant HTTP server (though compliant enough of most clients and browsers). Fq ships with a web UI that allows inspecting real-time state and performance.
#### GET /stats.json
exposes current exchange, queue, and client information.
#### POST /submit
An endpoint allowing message submission without a full and stateful Fq connection. It expects the following headers:
* ```X-Fq-User```,
* ```X-Fq-Route```, and
* ```X-Fq-Exchange```.
The HTTP client *MUST* provide a Content-Length header corresponding to the payload content (no chunked submission). The payload is treated as the raw message box without any special encoding.
Example:
```
curl -X POST -H "X-Fq-User: user" -H 'X-Fq-Route: bla' -H 'X-Fq-Exchange: test' localhost:8765/submit --data "TEST"
```
## Building
Requirements:
* C compiler
* GNU make
* libuuid
* sqlite3
* [jlog](https://github.com/omniti-labs/jlog)
* [libbcd](https://github.com/backtrace-labs/bcd) (optional, for crash tracing)
Generally:
```
make
make install
```
To build without libbcd support:
```
NO_BCD=1 make
```
## Debugging
FQ can be run in debug mode from the command line.
To run FQ in debug mode, kill any and all existing FQ processes, then enter the
following command:
```
fq -g fq FQ_DEBUG=<flag values> <path to fqd>/fqd -D -c <path to fqd.sqlite>/fqd.sqlite -p <port number>
```
Flag values determine debug output type and can have the following values:
```
FQ_DEBUG_MEM = 0x00000001,
FQ_DEBUG_MSG = 0x00000002,
FQ_DEBUG_ROUTE = 0x00000004,
FQ_DEBUG_IO = 0x00000008,
FQ_DEBUG_CONN = 0x00000010,
FQ_DEBUG_CONFIG = 0x00000020,
FQ_DEBUG = 0x00000040,
FQ_DEBUG_PEER = 0x00000080,
FQ_DEBUG_HTTP = 0x00000100,
FQ_DEBUG_PANIC = 0x40000000
```
To debug more than one flag, simply OR the flag values. For example, to output
connection, configuration, and route information, set `FQ_DEBUG` equal to
`0x00000034 (FQ_DEBUG_CONFIG|FQ_DEBUG_CONN|FQ_DEBUG_ROUTE)`.
For example, you can run FQ in debug mode with the variables shown below to
output configuration, connection, and route information to the console:
```
fq -g fq FQ_DEBUG=0x00000034 /opt/circonus/sbin/fqd -D -c /opt/circonus/var/lib/fq/fqd.sqlite -p 8765
```
================================================
FILE: coverity_model.c
================================================
typedef unsigned int uint32_t;
typedef _Bool bool;
void
ck_pr_dec_uint_zero(uint32_t *v, bool *zero) {
*v = (*v) - 1;
if(zero) *zero = ((*v) == 0);
}
void
ck_pr_inc_uint(uint32_t *v) {
*v = (*v) + 1;
}
uint32_t
ck_pr_load_uint(uint32_t *v) {
return *v;
}
================================================
FILE: docs/fq_protocol.md
================================================
# fq Protocol
## Client
Clients maintain two paired tcp connections to fq. After the connections are made, some preliminary data is sent over the command socket. A "plain auth" command is then issued, return a client key. The client key is then used in some preliminary data sent over the data socket, pairing the two sockets for the session.
### Prefixes
* Cmd Mode: `0xcc50cafe`
* Data Mode: `0xcc50face`
* Peer Mode: `0xcc50fade`
### Command Socket
Length | Description
---------+-----------------------------
4 bytes | Cmd Mode
### Data Socket
Length | Description
---------+-----------------------------
4 bytes | Data Mode
2 bytes | Client Key Length
variable | Client Key
## Commands
General form `(command prefix)(command)`, big endian. All non-heartbeat related commands have in-band responses corresponding to the order in which requests they were sent to fq. Heartbeat requests are used simply to tell fq to look for and to send heartbeats at a specific interval. Heartbeats should be checked for during normal command processing and not as a response to a specific request.
### Prefixes
Commad prefixes are two bytes at the beginning of the command
* Error: `0xeeee`
* Heartbeat: `0xbea7`
* Auth CMD: `0xaaaa`
* Auth Plain: `0x0000`
* Auth Response: `0xaa00`
* Heartbeat Request: `0x4848`
* Bind: `0xb171`
* Bind Request: `0xb170`
* Unbind: `0x171b`
* Unbind Request: `0x071b`
* Status: `0x57a7`
* Status Request: `0xc7a7`
#### Plain Auth
Plain Auth is a subset of Auth and will have both prefixs
##### Request
Length | Description
---------+-----------------------------
2 bytes | Auth CMD Prefix
2 bytes | Auth Plain Prefix
2 bytes | User Length
variable | User
2 bytes | Queue Length (Queue + 1 byte + Queue type)
variable | Queue
1 byte | 0
variable | Queue type ("mem" or "disk") + ":param,param"
2 bytes | Password Length (16 bit)
variable | Password
##### Response
Length | Description
---------+-----------------------------
2 bytes | Auth Response Prefix
2 bytes | Client Key Length Length
variable | Client Key (0 < length < 127 bytes)
#### Bind
##### Request
Length | Description
---------+-----------------------------
2 bytes | Bind Request Prefix
2 bytes | Flags: Peer Mode (0 or 1) | perm(0110)/trans(0100)
2 bytes | Exchange Length
2 bytes | Exchange
2 bytes | Program Length
variable | Program
##### Response
Length | Description
---------+-----------------------------
2 bytes | Bind Prefix
4 bytes | Binding ID
#### Unbind
##### Request
Length | Description
---------+-----------------------------
2 bytes | Unbind Request Prefix
4 bytes | Binding ID
2 bytes | Exchange Length
variable | Exchange
##### Response
Length | Description
---------+-----------------------------
2 bytes | Unbind Prefix
4 bytes | Binding ID
On success, the response binding id will be the same as the one sent in the request.
#### Status
##### Request
Length | Description
---------+-----------------------------
2 bytes | Status Request Prefix
##### Response
Length | Description
---------+-----------------------------
2 bytes | Status Prefix
2 bytes | Key Length
variable | Key
4 bytes | Value
… (repeat key length, key, value sets)
2 bytes | Key Length 0
The response contains serveral key-value pairs. Each key is prefixed by a length and parsing of kv pairs should continue until a key length of 0 is read.
#### Heartbeat Request
##### Request
Length | Description
---------+-----------------------------
2 bytes | Heartbeat Request Prefix
2 bytes | Heartbeat Interval (milliseconds)
##### Response
None
#### Heartbeat
##### Request
Length | Description
---------+-----------------------------
2 bytes | Heartbeat Prefix
##### Response
Length | Description
---------+-----------------------------
2 bytes | Heartbeat Prefix
## Messages
Length | Description | Note
---------+-----------------------------------+------------------------
1 byte | Exchange Length |
variable | Exchange |
1 byte | Route Length |
16 bytes | Message ID |
1 byte | Sender Length | Send for Peer Mode Only
variable | Sender | Send for Peer Mode Only
1 byte | Number of Hops | Send for Peer Mode Only
variable | Hops (numHops sets of 4-byte IPs) | Send for Peer Mode Only
4 bytes | Payload Length |
variable | Payload (<= 128kb) |
All properties will be present when receiving a message, while some propertiers are only sent when in peer mode.
================================================
FILE: dtest.d
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
fq*:::message-receive{
printf("sender: %s\n", args[2]->sender);
printf("exchange: %s\n", args[2]->exchange);
printf("route: %s\n", args[2]->route);
printf("message len: %d\n", args[2]->payload_len);
printf("message: %.*s\n", args[2]->payload_len, args[2]->payload);
printf("client: %s\n", args[0]->pretty);
printf("client: %s\n", args[1]->pretty);
printf("latency: %d\n", args[2]->latency);
}
fq*:::queue-drop{
q = ((fq_queue_t *)arg0);
printf("dropped message on queue %s\n", q->name);
}
fq*:::queue-block{
printf("blocking queue %s\n", ((fq_queue_t *)arg0)->name);
}
================================================
FILE: fq.d
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
typedef struct {
uintptr_t route;
uintptr_t sender;
uintptr_t exchange;
uintptr_t payload;
uint32_t payload_len;
uint64_t latency;
} fq_dtrace_msg_t;
typedef struct {
string route;
string exchange;
string sender;
string payload;
uint32_t payload_len;
uint64_t latency;
} fq_msg_t;
translator fq_msg_t <fq_dtrace_msg_t *m> {
route = copyinstr(*(uintptr_t *)copyin((uintptr_t)&m->route, sizeof(uintptr_t)));
exchange = copyinstr(*(uintptr_t *)copyin((uintptr_t)&m->exchange, sizeof(uintptr_t)));
sender = copyinstr(*(uintptr_t *)copyin((uintptr_t)&m->sender, sizeof(uintptr_t)));
payload_len = *(uint32_t *)copyin((uintptr_t)&m->payload_len, sizeof(uint32_t));
payload = copyinstr(*(uintptr_t *)copyin((uintptr_t)&m->payload, sizeof(uintptr_t)), *(uint32_t *)copyin((uintptr_t)&m->payload_len, sizeof(uint32_t)));
latency = *(uint64_t *)copyin((uintptr_t)&m->latency, sizeof(uint64_t));
};
typedef struct {
uintptr_t name;
int32_t isprivate;
int32_t policy;
uintptr_t type;
} fq_dtrace_queue_t;
typedef struct {
string name;
int32_t isprivate;
int32_t policy;
string type;
} fq_queue_t;
translator fq_queue_t <fq_dtrace_queue_t *m> {
name = copyinstr(*(uintptr_t *)copyin((uintptr_t)&m->name, sizeof(uintptr_t)));
type = copyinstr(*(uintptr_t *)copyin((uintptr_t)&m->type, sizeof(uintptr_t)));
isprivate = *(uint32_t *)copyin((uintptr_t)&m->isprivate, sizeof(int32_t));
policy = *(uint32_t *)copyin((uintptr_t)&m->policy, sizeof(int32_t));
};
typedef struct {
int32_t fd;
uintptr_t pretty;
} fq_dtrace_remote_anon_client_t;
typedef struct {
int32_t fd;
string pretty;
} fq_remote_anon_client_t;
typedef struct {
int32_t fd;
uintptr_t pretty;
} fq_dtrace_remote_client_t;
typedef struct {
int32_t fd;
string pretty;
} fq_remote_client_t;
typedef struct {
int32_t fd;
uintptr_t pretty;
} fq_dtrace_remote_data_client_t;
typedef struct {
int32_t fd;
string pretty;
} fq_remote_data_client_t;
translator fq_remote_anon_client_t <fq_dtrace_remote_anon_client_t *c> {
fd = *(uint32_t *)copyin((uintptr_t)&c->fd, sizeof(int32_t));
pretty = copyinstr(*(uintptr_t *)copyin((uintptr_t)&c->pretty, sizeof(uintptr_t)));
};
================================================
FILE: fq.h
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#ifndef FQ_H
#define FQ_H
#ifndef _REENTRANT
#error "You must compile with -D_REENTRANT"
#endif
#include <string.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/time.h>
#include <stdint.h>
#include <stdbool.h>
#include <stdlib.h>
#include <ck_fifo.h>
#include <ck_stack.h>
#define FQ_PROTO_CMD_MODE 0xcc50cafe
#define FQ_PROTO_DATA_MODE 0xcc50face
#define FQ_PROTO_PEER_MODE 0xcc50feed
#define FQ_PROTO_OLD_PEER_MODE 0xcc50fade
#define FQ_PROTO_READ_STAT 0x47455420 /* "GET " */
#define FQ_PROTO_HTTP_GET 0x47455420 /* "GET " */
#define FQ_PROTO_HTTP_PUT 0x50555420 /* "PUT " */
#define FQ_PROTO_HTTP_POST 0x504f5354 /* "POST" */
#define FQ_PROTO_HTTP_HEAD 0x48454144 /* "HEAD" */
#define FQ_BIND_PEER 0x00000001
#define FQ_BIND_PERM 0x00000110
#define FQ_BIND_TRANS 0x00000100
#define FQ_PROTO_ERROR 0xeeee
#define FQ_PROTO_AUTH_CMD 0xaaaa
#define FQ_PROTO_AUTH_PLAIN 0
#define FQ_PROTO_AUTH_RESP 0xaa00
#define FQ_PROTO_HBREQ 0x4848
#define FQ_PROTO_HB 0xbea7
#define FQ_PROTO_BINDREQ 0xb170
#define FQ_PROTO_BIND 0xb171
#define FQ_PROTO_UNBINDREQ 0x071b
#define FQ_PROTO_UNBIND 0x171b
#define FQ_PROTO_STATUS 0x57a7
#define FQ_PROTO_STATUSREQ 0xc7a7
#define FQ_DEFAULT_QUEUE_TYPE "mem"
#ifndef min
#define min(a,b) ((a) < (b) ? (a) : (b))
#endif
#define MAX_RK_LEN 127
/* !lua start */
typedef struct fq_rk {
unsigned char name[MAX_RK_LEN];
uint8_t len;
} fq_rk;
static inline void
fq_rk_from_str(fq_rk *rk, const char *str) {
size_t len = strlen(str);
memset(rk->name, 0, MAX_RK_LEN);
rk->len = min(len, MAX_RK_LEN - 1);
memcpy(rk->name, str, rk->len);
}
static inline int
fq_rk_cmp(const fq_rk * const a, const fq_rk * const b) {
if(a->len < b->len) return -1;
if(a->len > b->len) return 1;
return memcmp(a->name, b->name, a->len);
}
#define FQ_BIND_ILLEGAL (uint32_t)0xffffffff
typedef struct {
fq_rk exchange;
uint32_t flags;
char *program;
uint32_t out__route_id;
} fq_bind_req;
typedef struct {
fq_rk exchange;
uint32_t route_id;
uint32_t out__success;
} fq_unbind_req;
typedef struct fq_msgid {
union {
struct {
uint32_t p1; /* user(sender) */
uint32_t p2; /* user(sender) */
uint32_t p3; /* reserved */
uint32_t p4; /* reserved */
} u32;
unsigned char d[16];
} id;
} fq_msgid;
typedef struct msg_free_stacks_handle_t msg_free_stacks_handle_t;
#define MAX_HOPS 32
typedef struct fq_msg {
uint32_t hops[MAX_HOPS];
fq_rk route;
fq_rk sender;
fq_rk exchange;
fq_msgid sender_msgid;
uint32_t refcnt;
uint32_t payload_len;
uint64_t arrival_time;
ck_stack_entry_t cleanup_stack_entry;
msg_free_stacks_handle_t *cleanup_handle;
/* define a free function as an alternative to `free()` */
void (*free_fn)(struct fq_msg *m);
unsigned char payload[]; /* over allocated */
} fq_msg;
extern void fq_clear_message_cleanup_stack(void);
extern fq_msg *fq_msg_alloc(const void *payload,
size_t payload_size);
extern fq_msg *fq_msg_alloc_BLANK(size_t payload_size);
extern void fq_msg_ref(fq_msg *);
extern void fq_msg_deref(fq_msg *);
#define fq_msg_free(a) fq_msg_deref(a)
extern void fq_msg_exchange(fq_msg *, const void *key, int klen);
extern void fq_msg_route(fq_msg *, const void *key, int klen);
extern void fq_msg_id(fq_msg *, fq_msgid *id);
extern int fq_find_in_hops(uint32_t, fq_msg *);
typedef struct buffered_msg_reader buffered_msg_reader;
extern buffered_msg_reader *
fq_buffered_msg_reader_alloc(int fd, uint32_t peermode);
extern void fq_buffered_msg_reader_free(buffered_msg_reader *f);
extern int
fq_buffered_msg_read(buffered_msg_reader *f,
void (*f_msg_handler)(void *, fq_msg *),
void *);
/* frame */
/*
* 1 x uint8_t<net> hops
* hops x uint32_t<net> node
* 1 x <nstring> exchange
* 1 x fq_rk<nstring> sender
* 1 x fq_rk<nstring> route
* 1 x uint32_t<net> payload_len
* 1 x data
*/
typedef struct fq_conn_s *fq_client;
#define FQ_HOOKS_V1 1
#define FQ_HOOKS_V2 2
#define FQ_HOOKS_V3 3
#define FQ_HOOKS_V4 4
typedef struct fq_hooks {
int version;
/* V1 */
void (*auth)(fq_client, int);
void (*bind)(fq_client, fq_bind_req *);
/* V2 */
void (*unbind)(fq_client, fq_unbind_req *);
/* V3 */
int sync;
/* V4 */
bool (*message)(fq_client, fq_msg *m);
void (*cleanup)(fq_client);
void (*disconnect)(fq_client);
} fq_hooks;
extern int
fq_client_hooks(fq_client conn, fq_hooks *hooks);
extern void
fq_client_set_userdata(fq_client, void *);
extern void *
fq_client_get_userdata(fq_client);
extern int
fq_client_init(fq_client *, uint32_t peermode,
void (*)(fq_client, const char *));
extern int
fq_client_creds(fq_client,
const char *host, unsigned short port,
const char *source, const char *pass);
extern void
fq_client_status(fq_client conn,
void (*f)(char *, uint32_t, void *), void *c);
extern void
fq_client_heartbeat(fq_client conn, unsigned short ms);
extern void
fq_client_bind(fq_client conn, fq_bind_req *req);
extern void
fq_client_unbind(fq_client conn, fq_unbind_req *req);
extern void
fq_client_set_backlog(fq_client conn, uint32_t len, uint32_t stall);
extern void
fq_client_set_nonblock(fq_client conn, bool nonblock);
extern int
fq_client_connect(fq_client conn);
extern int
fq_client_publish(fq_client, fq_msg *msg);
extern fq_msg *
fq_client_receive(fq_client conn);
extern void
fq_client_destroy(fq_client conn);
extern int
fq_client_data_backlog(fq_client conn);
extern int
fq_rk_to_hex(char *buf, int len, fq_rk *k);
extern int
fq_read_status(int fd, void (*f)(char *, uint32_t, void *), void *);
extern int
fq_read_uint16(int fd, unsigned short *v);
extern int
fq_write_uint16(int fd, unsigned short hs);
extern int
fq_read_uint32(int fd, uint32_t *v);
extern int
fq_write_uint32(int fd, uint32_t hs);
extern int
fq_read_short_cmd(int fd, unsigned short buflen, void *buf);
extern int
fq_write_short_cmd(int fd, unsigned short buflen, const void *buf);
extern int
fq_read_long_cmd(int fd, int *len, void **buf);
/* This function returns 0 on success, -1 on failure or a positive
* integer indicating that a partial write as happened.
* The initial call should be made with off = 0, if a positive
* value is returned, a subsequent call should be made with
* off = (off + return value).
* The caller must be able to keep track of an accumulated offset
* in the event that several invocations are required to send the
* message.
*/
extern int
fq_client_write_msg(int fd, uint32_t peermode, fq_msg *m,
size_t off, size_t *written);
typedef enum {
FQ_POLICY_DROP = 0,
FQ_POLICY_BLOCK = 1,
} queue_policy_t;
typedef enum {
FQ_DEBUG_MEM = 0x00000001,
FQ_DEBUG_MSG = 0x00000002,
FQ_DEBUG_ROUTE = 0x00000004,
FQ_DEBUG_IO = 0x00000008,
FQ_DEBUG_CONN = 0x00000010,
FQ_DEBUG_CONFIG = 0x00000020,
FQ_DEBUG = 0x00000040,
FQ_DEBUG_PEER = 0x00000080,
FQ_DEBUG_HTTP = 0x00000100,
FQ_DEBUG_PANIC = 0x40000000
} fq_debug_bits_t;
void fq_keepalive_fd(int fd, int cnt, int idle, int invtl);
extern uint32_t fq_debug_bits;
void fq_debug_set_bits(uint32_t bits);
/* string can be integer, hex or comma separated string (e.g. "mem,io") */
void fq_debug_set_string(const char *s);
extern int
fq_debug_fl(const char *file, int line, fq_debug_bits_t, const char *fmt, ...)
__attribute__((format(printf, 4, 5)));
#define fq_debug(type, ...) do { \
if(0 != (type & fq_debug_bits)) { \
fq_debug_fl(__FILE__, __LINE__, type, __VA_ARGS__); \
} \
} while(0)
#define fq_stacktrace(b,t,s,e) do { \
if(0 != (b & fq_debug_bits)) { \
fq_debug_stacktrace(b,t,s,e); \
} \
} while(0)
/* programming:
*
* PROGRAM: <prefix|exact>:string RULES*
* RULE: (RULE)
* RULE: (RULE && RULE)
* RULE: (RULE || RULE)
* RULE: EXPR
* EXPR: function(args)
* args: arg
* args: arg, args
* arg: "string"
* arg: true|false
* arg: [0-9][0-9]*(?:.[0-9]*)
*
* functions are dynamically loadable with type signature
* strings: s, booleans: b, integers: d
* function: substr_eq(9.3,10,"tailorings",true)
* C symbol: fqd_route_prog__substr_eq__ddsb(int nargs, valnode_t *args);
* fallback symbol: fqd_route_prog_substr_eq__VA(int nargs, valnode_t *args);
*/
typedef struct valnode {
enum {
RP_VALUE_STRING = 1,
RP_VALUE_BOOLEAN = 2,
RP_VALUE_DOUBLE = 3
} value_type;
union {
char *s;
bool b;
double d;
} value;
} valnode_t;
extern void fq_debug_stacktrace(fq_debug_bits_t b, const char *tag, int start, int end);
#if defined(__MACH__)
typedef uint64_t hrtime_t;
#elif defined(BSD) || defined(__FreeBSD__)
typedef uint64_t hrtime_t;
#elif defined(linux) || defined(__linux) || defined(__linux__)
typedef uint64_t hrtime_t;
#endif
extern hrtime_t fq_gethrtime(void);
/* !lua stop */
/* DTrace helpers */
typedef struct {
char *route;
char *sender;
char *exchange;
char *payload;
uint32_t payload_len;
uint64_t latency;
} fq_dtrace_msg_t;
#define DTRACE_PACK_MSG(dmsg, msg) do { \
(dmsg)->route = (char *)(msg)->route.name; \
(dmsg)->sender = (char *)(msg)->sender.name; \
(dmsg)->exchange = (char *)(msg)->exchange.name; \
(dmsg)->payload_len = (uint32_t)(msg)->payload_len; \
(dmsg)->payload = (char *)(msg)->payload; \
(dmsg)->latency = fq_gethrtime() - (msg)->arrival_time; \
} while(0)
#define fq_assert(A) do { \
if(!(A)) { \
fq_debug_stacktrace(FQ_DEBUG_PANIC, "assert", 1, 1000); \
(void)fprintf (stderr, "%s:%s:%u: failed assertion `%s'\n", __func__, __FILE__, __LINE__, #A); \
abort(); \
} \
} while(0)
#endif
================================================
FILE: fq_bench.c
================================================
/*
* Copyright (c) 2016 Circonus
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <signal.h>
#include <string.h>
#include "fq.h"
void logger(fq_client c, const char *s) {
(void)c;
fprintf(stderr, "fq_logger: %s\n", s);
}
static void
debug_status(char *key, uint32_t value, void *unused) {
(void)unused;
fq_debug(FQ_DEBUG_CONN, " ---> %s : %u\n", key, value);
}
static void
print_rate(fq_client c, hrtime_t s, hrtime_t f, uint64_t cnt, uint64_t icnt) {
double d;
fq_client_status(c, debug_status, NULL);
if(cnt) {
d = (double)cnt * 1000000000;
d /= (double)(f-s);
printf("[%d backlog] output %0.2f msg/sec\n",
fq_client_data_backlog(c), d);
}
if(icnt) {
d = (double)icnt * 1000000000;
d /= (double)(f-s);
printf("[%d backlog] input %0.2f msg/sec\n",
fq_client_data_backlog(c), d);
}
}
static void usage(const char *prog) {
printf("%s:\n", prog);
printf("\t-H\t\tthis help message\n");
printf("\t-h\t\tthe fq host to connect to\n");
printf("\t-p <port>\tspecify connecting port (default: 8765)\n");
printf("\t-u <user>\tuser name\n");
printf("\t-P <password>\tpassword\n");
printf("\t-t <type>\t'disk' or 'mem' for type of queue to test\n");
printf("\t-q <queue>\tname of queue to test (default: benchmark_queue)\n");
printf("\t-c <count>\tnumber of messages to bench with\n");
printf("\t-s <size>\tsize of each message\n");
}
static char *host;
static int port = 8765;
static char *user;
static char *exchange;
static char *pass;
static char *type;
static char *queue_name;
static int count = 100000;
static int size = 100;
static void parse_cli(int argc, char **argv) {
int c;
const char *debug = getenv("FQ_DEBUG");
while((c = getopt(argc, argv, "Hh:p:u:P:t:q:e:c:s:")) != EOF) {
switch(c) {
case 'H':
usage(argv[0]);
exit(0);
case 'h':
free(host);
host = strdup(optarg);
break;
case 'p':
port = atoi(optarg);
break;
case 'u':
free(user);
user = strdup(optarg);
break;
case 'P':
free(pass);
pass = strdup(optarg);
break;
case 't':
free(type);
type = strdup(optarg);
break;
case 'q':
free(queue_name);
queue_name = strdup(optarg);
break;
case 'e':
free(exchange);
exchange = strdup(optarg);
break;
case 'c':
count = atoi(optarg);
break;
case 's':
size = atoi(optarg);
break;
default:
usage(argv[0]);
exit(-1);
}
}
if(debug) fq_debug_set_string(debug);
if(!host) host = strdup("localhost");
if(!user) user = strdup("user");
if(!pass) pass = strdup("pass");
if(!queue_name) queue_name = strdup("benchmark_queue");
if(!exchange) exchange = strdup("maryland");
if(!type) type = strdup("mem");
}
/**
* Benchmark a single thread against a type of queue at some host.
*/
int main(int argc, char **argv) {
char queue[256] = {0};
hrtime_t s0, s, f, f0;
uint64_t cnt = 0, icnt = 0;
int i = 0;
fq_client c;
fq_msg *m;
char *fq_debug = getenv("FQ_DEBUG");
if(fq_debug) fq_debug_set_bits(atoi(fq_debug));
signal(SIGPIPE, SIG_IGN);
fq_client_init(&c, 0, logger);
parse_cli(argc, argv);
sprintf(queue, "%s/%s/%s:public,drop,backlog=10000,permanent", user, queue_name, type);
printf("using queue -> %s\n", queue);
fq_client_creds(c, host, port, queue, pass);
fq_client_heartbeat(c, 1000);
fq_client_set_backlog(c, 10000, 100);
fq_client_connect(c);
printf("payload size -> %d\n", size);
printf("message count -> %d\n", count);
s0 = s = fq_gethrtime();
f = s;
m = fq_msg_alloc_BLANK(size);
memset(m->payload, 'X', size);
fq_msg_exchange(m, exchange, strlen(exchange));
fq_msg_route(m, "test.bench.foo", 14);
while(i < count || fq_client_data_backlog(c) > 0) {
if(i < count) {
m->arrival_time = fq_gethrtime();
fq_msg_id(m, NULL);
fq_client_publish(c, m);
cnt++;
i++;
}
else usleep(10);
if (i % 1000 == 0) {
f = fq_gethrtime();
}
if(f-s > 1000000000) {
print_rate(c, s, f, cnt, icnt);
icnt = 0;
cnt = 0;
s = f;
}
}
f0 = fq_gethrtime();
print_rate(c, s0, f0, i, 0);
free(host);
free(user);
free(pass);
free(exchange);
free(queue_name);
free(type);
return 0;
}
================================================
FILE: fq_client.c
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <ctype.h>
#include <string.h>
#include <unistd.h>
#include <sys/uio.h>
#include <fcntl.h>
#include <errno.h>
#include <pthread.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <netdb.h>
#include <poll.h>
#include <ck_fifo.h>
#include <uuid/uuid.h>
#include "fq.h"
#include "fqd.h"
#include "fqd_private.h"
static const long MAX_RESOLVE_CACHE = 1000000000; /* ns */
static __thread char thread_local_name[16];
void
fq_thread_setname(const char *format, ...) {
va_list arg;
va_start(arg, format);
char thrname[16] = "\0";
if(!format) format = "terminated";
vsnprintf(thrname, sizeof(thrname), format, arg);
#if defined(linux) || defined(__linux) || defined(__linux__)
pthread_setname_np(pthread_self(), thrname);
#endif
strlcpy(thread_local_name, thrname, sizeof(thread_local_name));
va_end(arg);
}
#define likely(x) __builtin_expect(!!(x), 1)
#define unlikely(x) __builtin_expect(!!(x), 0)
#define CONNERR_S(c) do { \
if(c->errorlog) c->errorlog(c, c->error); \
} while(0)
#define CONNERR(c, s) do { \
strlcpy(c->error, s, sizeof(c->error)); \
if(c->errorlog) c->errorlog(c, c->error); \
} while(0)
static inline int
fq_client_wfrw_internal(int fd, int needs_read, int needs_write,
uint64_t ms, int *mask) {
int rv;
struct pollfd pfd;
pfd.fd = fd;
pfd.events = 0;
if(needs_read) pfd.events |= POLLIN | POLLERR | POLLHUP;
if(needs_write) pfd.events |= POLLOUT | POLLERR | POLLHUP;
while(-1 == (rv = poll(&pfd, 1, ms)) && errno == EINTR);
if(mask) *mask = pfd.revents;
return rv;
}
struct fq_conn_s {
struct sockaddr_in remote;
char *host;
short port;
hrtime_t last_resolve;
char error[128];
char *user;
char *pass;
char *queue;
char *queue_type;
fq_rk key;
int cmd_fd;
int cmd_hb_needed;
unsigned short cmd_hb_ms;
hrtime_t cmd_hb_last;
uint32_t peermode;
int data_fd;
pthread_t worker;
pthread_t data_worker;
int stop;
ck_fifo_spsc_t cmdq;
ck_fifo_spsc_t q;
ck_fifo_spsc_t backq;
uint32_t qlen;
uint32_t qmaxlen;
uint32_t q_stall_time;
bool non_blocking;
int connected;
int data_ready;
fq_msg *tosend;
int tosend_offset;
int sync_hooks; /* should they run in the calling thread */
void (*auth_hook)(fq_client, int);
void (*bind_hook)(fq_client, fq_bind_req *);
void (*unbind_hook)(fq_client, fq_unbind_req *);
bool (*message_hook)(fq_client, fq_msg *);
void (*cleanup_hook)(fq_client);
void (*disconnect_hook)(fq_client);
void (*errorlog)(fq_client, const char *);
ck_fifo_spsc_entry_t *cmdqhead;
ck_fifo_spsc_entry_t *qhead;
ck_fifo_spsc_entry_t *backqhead;
uint32_t thrcnt;
void *userdata;
};
typedef struct fq_conn_s fq_conn_s;
typedef struct {
unsigned short cmd;
union {
struct {
uint16_t ms;
} heartbeat;
struct {
void (*callback)(char *, uint32_t, void *);
void *closure;
} status;
fq_bind_req *bind;
fq_unbind_req *unbind;
int return_value;
} data;
} cmd_instr;
static void mark_safe_free(void *pptr);
static void
fq_conn_free(fq_conn_s *conn_s) {
if(conn_s->user) free(conn_s->user);
if(conn_s->pass) free(conn_s->pass);
if(conn_s->queue) free(conn_s->queue);
if(conn_s->queue_type) free(conn_s->queue_type);
if(conn_s->tosend) fq_msg_free(conn_s->tosend);
#define SWEEP_CONN_Q(TYPE, FREE, QNAME, ENTRYNAME) do { \
TYPE tofree; \
ck_fifo_spsc_dequeue_lock(&conn_s->QNAME); \
while(ck_fifo_spsc_dequeue(&conn_s->QNAME, &tofree) == true) { \
FREE(tofree); \
} \
ck_fifo_spsc_dequeue_unlock(&conn_s->QNAME); \
ck_fifo_spsc_entry_t *garbage = NULL; \
ck_fifo_spsc_deinit(&conn_s->QNAME, &garbage); \
while (garbage != NULL) { \
ck_fifo_spsc_entry_t *gn = garbage->next; \
free(garbage); \
garbage = gn; \
} \
} while(0)
SWEEP_CONN_Q(fq_msg *, fq_msg_free, q, qhead);
SWEEP_CONN_Q(void *, mark_safe_free, backq, backqhead);
SWEEP_CONN_Q(cmd_instr *, free, cmdq, cmdqhead);
#undef SWEEP_CONN_Q
if(conn_s->cleanup_hook) conn_s->cleanup_hook(conn_s);
free(conn_s);
}
typedef enum {
AUTH_HOOK_TYPE,
CMD_HOOK_TYPE
} hook_req_type_t;
typedef struct {
hook_req_type_t type;
cmd_instr *entry;
} hook_req_t;
static void
fq_client_signal(fq_client conn, cmd_instr *e)
{
fq_conn_s *conn_s = conn;
ck_fifo_spsc_enqueue_lock(&conn_s->cmdq);
ck_fifo_spsc_entry_t *fifo_entry = ck_fifo_spsc_recycle(&conn_s->cmdq);
if (fifo_entry == NULL) {
fifo_entry = malloc(sizeof(ck_fifo_spsc_entry_t));
}
ck_fifo_spsc_enqueue(&conn_s->cmdq, fifo_entry, e);
ck_fifo_spsc_enqueue_unlock(&conn_s->cmdq);
}
static int
fq_resolve_endpoint(fq_conn_s *conn_s) {
conn_s->remote.sin_family = AF_INET;
conn_s->remote.sin_port = htons(conn_s->port);
if(inet_pton(AF_INET, conn_s->host, &conn_s->remote.sin_addr) != 1) {
#ifdef HAVE_GETHOSTBYNAME_R
struct hostent hostbuf, *hp;
struct in_addr **addr_list;
int buflen = 1024, herr;
char *buf;
if((buf = malloc(buflen)) == NULL) {
CONNERR(conn_s, "out of memory");
return -1;
}
while((hp = gethostbyname_r(conn_s->host, &hostbuf, buf, buflen, &herr)) == NULL &&
errno == ERANGE) {
buflen *= 2;
if((buf = realloc(buf, buflen)) == NULL) {
CONNERR(conn_s, "out of memory");
free(buf);
return -1;
}
}
if(!hp) {
CONNERR(conn_s, "host lookup failed");
free(buf);
return -1;
}
addr_list = (struct in_addr **)hp->h_addr_list;
if(*addr_list == 0) {
CONNERR(conn_s, "no address for host");
free(buf);
return -1;
}
memcpy(&conn_s->remote.sin_addr, *addr_list, sizeof(struct in_addr));
free(buf);
#elif defined _POSIX_C_SOURCE
struct addrinfo *result;
if (getaddrinfo(conn_s->host, NULL, NULL, &result)) {
CONNERR(conn_s, "host lookup failed");
return -1;
}
if (result) {
for (struct addrinfo *rp = result; rp; rp = rp->ai_next) {
if (rp->ai_addr) {
conn_s->remote.sin_addr = ((struct sockaddr_in*)rp->ai_addr)->sin_addr;
break;
}
}
freeaddrinfo(result);
} else {
CONNERR(conn_s, "no address for host");
return -1;
}
#else
static pthread_mutex_t guard = PTHREAD_MUTEX_INITIALIZER;
struct hostent *hp;
struct in_addr **addr_list;
pthread_mutex_lock(&guard);
hp = gethostbyname(conn_s->host);
if(!hp) {
CONNERR(conn_s, "host lookup failed");
pthread_mutex_unlock(&guard);
return -1;
}
addr_list = (struct in_addr **)hp->h_addr_list;
if(*addr_list == 0) {
CONNERR(conn_s, "no address for host");
pthread_mutex_unlock(&guard);
return -1;
}
memcpy(&conn_s->remote.sin_addr, *addr_list, sizeof(struct in_addr));
pthread_mutex_unlock(&guard);
#endif
}
conn_s->last_resolve = fq_gethrtime();
return 0;
}
static int
fq_socket_connect(fq_conn_s *conn_s) {
int fd, rv, on = 1;
/* re-resolve if we've not done so quite recently */
if(conn_s->last_resolve == 0 ||
(fq_gethrtime() - conn_s->last_resolve) > MAX_RESOLVE_CACHE) {
if(fq_resolve_endpoint(conn_s) < 0) {
return -1;
}
}
fd = socket(AF_INET, SOCK_STREAM, 0);
if(fd == -1) return -1;
rv = connect(fd, (struct sockaddr *)&conn_s->remote,
sizeof(conn_s->remote));
if(rv == -1) {
snprintf(conn_s->error, sizeof(conn_s->error),
"socket: %s", strerror(errno));
CONNERR_S(conn_s);
close(fd);
return -1;
}
/* If this fails, we ignore it */
(void)setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &on, sizeof(on));
fq_keepalive_fd(fd, 10, 5, 2);
return fd;
}
static void
fq_client_disconnect_internal(fq_conn_s *conn_s) {
if(conn_s->cmd_fd >= 0) {
int toclose = conn_s->cmd_fd;
conn_s->cmd_fd = -1;
fq_stacktrace(FQ_DEBUG_CONN, "close(cmd_fd)\n",1,2);
close(toclose);
if(conn_s->disconnect_hook) conn_s->disconnect_hook(conn_s);
}
if(conn_s->data_fd >= 0) {
int toclose = conn_s->data_fd;
conn_s->data_fd = -1;
fq_debug(FQ_DEBUG_CONN, "close(data_fd)\n");
close(toclose);
}
conn_s->data_ready = 0;
if(conn_s->cmd_hb_ms) {
unsigned short hb;
hb = conn_s->cmd_hb_ms;
conn_s->cmd_hb_ms = 0;
fq_client_heartbeat(conn_s, hb);
conn_s->cmd_hb_last = 0;
}
}
static int
fq_client_do_auth(fq_conn_s *conn_s) {
int len, qlen, qtlen;
uint16_t cmd;
char error[1024];
char queue_composed[MAX_RK_LEN*2 + 1];
if(fq_write_uint16(conn_s->cmd_fd, FQ_PROTO_AUTH_CMD)) return -1;
if(fq_write_uint16(conn_s->cmd_fd, FQ_PROTO_AUTH_PLAIN)) return -2;
if(fq_write_short_cmd(conn_s->cmd_fd, strlen(conn_s->user), conn_s->user) < 0)
return -3;
qlen = strlen(conn_s->queue);
if(qlen > MAX_RK_LEN) qlen = MAX_RK_LEN;
qtlen = strlen(conn_s->queue_type);
if(qtlen > MAX_RK_LEN) qtlen = MAX_RK_LEN;
len = qlen + qtlen + 1;
memcpy(queue_composed, conn_s->queue, qlen);
queue_composed[qlen] = '\0';
memcpy(queue_composed + qlen + 1, conn_s->queue_type, qtlen);
if(fq_write_short_cmd(conn_s->cmd_fd, len, queue_composed) < 0)
return -4;
if(fq_write_short_cmd(conn_s->cmd_fd, strlen(conn_s->pass), conn_s->pass) < 0)
return -5;
if(fq_read_uint16(conn_s->cmd_fd, &cmd)) return -6;
switch(cmd) {
case FQ_PROTO_ERROR:
len = fq_read_short_cmd(conn_s->cmd_fd, sizeof(error)-1, error);
if(conn_s->errorlog) {
if(len > (int)sizeof(error)-1) len = sizeof(error)-1;
if(len < 0) conn_s->errorlog(conn_s, "error reading error");
else conn_s->errorlog(conn_s,error);
}
return -7;
case FQ_PROTO_AUTH_RESP:
len = fq_read_short_cmd(conn_s->cmd_fd,
sizeof(conn_s->key.name), conn_s->key.name);
if(len < 0 || len > (int)sizeof(conn_s->key.name)) return -8;
conn_s->key.len = len;
#ifdef DEBUG
{
char hex[260];
if(fq_rk_to_hex(hex, sizeof(hex), &conn_s->key) >= 0)
fq_debug(FQ_DEBUG_CONN, "client keyed:\n%s\n", hex);
}
#endif
conn_s->data_ready = 1;
break;
default:
if(conn_s->errorlog) {
snprintf(error, sizeof(error),
"server auth response 0x%04x unknown\n", cmd);
conn_s->errorlog(conn_s, error);
}
return -9;
}
return 0;
}
static int
fq_client_data_connect_internal(fq_conn_s *conn_s) {
int flags;
uint32_t cmd = htonl(conn_s->peermode ? FQ_PROTO_PEER_MODE
: FQ_PROTO_DATA_MODE);
/* We don't support data connections when the cmd connection is down */
if(conn_s->cmd_fd < 0) return -1;
if(conn_s->data_fd >= 0) {
int toclose = conn_s->data_fd;
conn_s->data_fd = -1;
fq_debug(FQ_DEBUG_CONN, "close(data_fd)\n");
close(toclose);
}
conn_s->data_fd = fq_socket_connect(conn_s);
if(conn_s->data_fd < 0) goto shutdown;
fq_debug(FQ_DEBUG_CONN, "connect(data_fd) -> %d\n", conn_s->data_fd);
if(write(conn_s->data_fd, &cmd, sizeof(cmd)) != sizeof(cmd))
goto shutdown;
if(conn_s->peermode) {
if(write(conn_s->data_fd, &conn_s->peermode,
sizeof(conn_s->peermode)) != sizeof(conn_s->peermode))
goto shutdown;
}
#ifdef DEBUG
{
char hex[260];
if(fq_rk_to_hex(hex, sizeof(hex), &conn_s->key) >= 0)
fq_debug(FQ_DEBUG_CONN, "client keying:\n%s\n", hex);
}
#endif
if(fq_write_short_cmd(conn_s->data_fd,
conn_s->key.len, conn_s->key.name) < 0) {
goto shutdown;
}
conn_s->tosend_offset = 0;
if(((flags = fcntl(conn_s->data_fd, F_GETFL, 0)) == -1) ||
(fcntl(conn_s->data_fd, F_SETFL, flags | O_NONBLOCK) == -1))
goto shutdown;
return 0;
shutdown:
if(conn_s->data_fd >= 0) {
int toclose = conn_s->data_fd;
conn_s->data_fd = -1;
fq_debug(FQ_DEBUG_CONN, "close(data_fd)\n");
close(toclose);
}
return -1;
}
/* This is dastardly... we know the ptr has to be aligned,
* when we see that it isn't on dequeue, we know it is a hook_req_t
*/
#define MARKED_HOOK_REQ_PTR(p) ((void *)(((uintptr_t)p)|1))
#define CHECK_HOOK_REQ_PTR(p) (((uintptr_t)p)&1)
#define UNMARKED_HOOK_REQ_PTR(p) ((void *)(((uintptr_t)p)&~1))
static void
mark_safe_free(void *pptr) {
if(CHECK_HOOK_REQ_PTR(pptr)) {
void *ptr = UNMARKED_HOOK_REQ_PTR(pptr);
free(ptr);
}
else fq_msg_free(pptr);
}
static void
enqueue_auth_hook_req(fq_conn_s *conn_s, int rv)
{
ck_fifo_spsc_enqueue_lock(&conn_s->backq);
ck_fifo_spsc_entry_t *fifo_entry = ck_fifo_spsc_recycle(&conn_s->backq);
if (unlikely(fifo_entry == NULL)) {
fifo_entry = malloc(sizeof(ck_fifo_spsc_entry_t));
}
hook_req_t *hreq = calloc(1,sizeof(*hreq));
hreq->type = AUTH_HOOK_TYPE;
hreq->entry = calloc(1, sizeof(*hreq->entry));
hreq->entry->data.return_value = 0;
ck_fifo_spsc_enqueue(&conn_s->backq, fifo_entry, MARKED_HOOK_REQ_PTR(hreq));
ck_fifo_spsc_enqueue_unlock(&conn_s->backq);
}
static void
enqueue_cmd_hook_req(fq_conn_s *conn_s, cmd_instr *e) {
ck_fifo_spsc_enqueue_lock(&conn_s->backq);
ck_fifo_spsc_entry_t *fifo_entry = ck_fifo_spsc_recycle(&conn_s->backq);
if (unlikely(fifo_entry == NULL)) {
fifo_entry = malloc(sizeof(ck_fifo_spsc_entry_t));
}
hook_req_t *hreq = calloc(1,sizeof(*hreq));
hreq->type = CMD_HOOK_TYPE;
hreq->entry = e;
ck_fifo_spsc_enqueue(&conn_s->backq, fifo_entry, MARKED_HOOK_REQ_PTR(hreq));
ck_fifo_spsc_enqueue_unlock(&conn_s->backq);
}
static int
fq_client_connect_internal(fq_conn_s *conn_s) {
int rv = -1;
uint32_t cmd = htonl(FQ_PROTO_CMD_MODE);
fq_client_disconnect_internal(conn_s);
conn_s->cmd_fd = fq_socket_connect(conn_s);
if(conn_s->cmd_fd < 0) goto shutdown;
fq_debug(FQ_DEBUG_CONN, "connect(cmd_fd) -> %d\n", conn_s->cmd_fd);
if(write(conn_s->cmd_fd, &cmd, sizeof(cmd)) != sizeof(cmd))
goto shutdown;
if((rv = fq_client_do_auth(conn_s)) < 0) {
fq_debug(FQ_DEBUG_CONN, "fq_client_do_auth -> %d\n", rv);
goto shutdown;
}
if(conn_s->auth_hook) {
if(conn_s->sync_hooks) enqueue_auth_hook_req(conn_s, 0);
else conn_s->auth_hook((fq_client)conn_s, 0);
}
return 0;
shutdown:
if(conn_s->cmd_fd >= 0) {
int toclose = conn_s->cmd_fd;
conn_s->cmd_fd = -1;
fq_debug(FQ_DEBUG_CONN, "close(cmd_fd) (in auth)\n");
close(toclose);
if(conn_s->disconnect_hook) conn_s->disconnect_hook(conn_s);
}
if(conn_s->auth_hook) {
if(conn_s->sync_hooks) enqueue_auth_hook_req(conn_s, rv);
else conn_s->auth_hook((fq_client)conn_s, rv);
}
return -1;
}
static void
fq_client_read_complete(void *closure, fq_msg *msg) {
fq_conn_s *conn_s = (fq_conn_s *)closure;
if(conn_s->message_hook && conn_s->message_hook(conn_s, msg)) {
fq_msg_deref(msg);
}
else {
ck_fifo_spsc_enqueue_lock(&conn_s->backq);
ck_fifo_spsc_entry_t *fifo_entry = ck_fifo_spsc_recycle(&conn_s->backq);
if (unlikely(fifo_entry == NULL)) {
fifo_entry = malloc(sizeof(ck_fifo_spsc_entry_t));
}
ck_fifo_spsc_enqueue(&conn_s->backq, fifo_entry, msg);
ck_fifo_spsc_enqueue_unlock(&conn_s->backq);
}
}
static void
fq_data_worker_loop(fq_conn_s *conn_s) {
buffered_msg_reader *ctx = NULL;
ctx = fq_buffered_msg_reader_alloc(conn_s->data_fd, 1);
fq_thread_setname("fq:w:%d/%d", conn_s->cmd_fd, conn_s->data_fd);
while(conn_s->cmd_fd >= 0 && conn_s->data_fd >= 0 && conn_s->stop == 0) {
int rv;
int wait_ms = 500, needs_write = 0, mask, write_rv;
if(conn_s->tosend) goto the_thick_of_it;
ck_fifo_spsc_dequeue_lock(&conn_s->q);
while(ck_fifo_spsc_dequeue(&conn_s->q, &conn_s->tosend) == true) {
conn_s->tosend_offset = 0;
ck_pr_dec_uint(&conn_s->qlen);
the_thick_of_it:
#ifdef DEBUG
fq_debug(FQ_DEBUG_MSG, "dequeue message to submit to server\n");
#endif
write_rv = fq_client_write_msg(conn_s->data_fd, conn_s->peermode,
conn_s->tosend, conn_s->tosend_offset, NULL);
if(write_rv > 0) {
conn_s->tosend_offset += write_rv;
wait_ms = 0;
break;
}
if(write_rv < 0) {
if(errno == EAGAIN) {
needs_write = 1;
break;
}
if(conn_s->errorlog) {
char errbuf[128];
snprintf(errbuf, sizeof(errbuf), "data write error: %s\n", strerror(errno));
conn_s->errorlog(conn_s, errbuf);
}
ck_fifo_spsc_dequeue_unlock(&conn_s->q);
goto finish;
}
fq_msg_deref(conn_s->tosend);
conn_s->tosend = NULL;
conn_s->tosend_offset = 0;
wait_ms = 0;
}
ck_fifo_spsc_dequeue_unlock(&conn_s->q);
rv = fq_client_wfrw_internal(conn_s->data_fd, 1, needs_write, wait_ms, &mask);
fq_debug(FQ_DEBUG_CONN, "fq_client_wfrw_internal(data:%d) -> %d\n", conn_s->data_fd, rv);
if(rv < 0) {
if(conn_s->errorlog) {
char errbuf[128];
snprintf(errbuf, sizeof(errbuf), "data read error: %s\n", strerror(errno));
conn_s->errorlog(conn_s, errbuf);
}
goto finish;
}
if(rv > 0 && (mask & POLLIN)) {
if(fq_buffered_msg_read(ctx, fq_client_read_complete, conn_s) < 0) {
if(conn_s->errorlog) conn_s->errorlog(conn_s, "data read: end-of-line\n");
goto finish;
}
}
}
finish:
conn_s->data_ready = 0;
fq_clear_message_cleanup_stack();
if(ctx) fq_buffered_msg_reader_free(ctx);
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "cmd_fd -> %d, stop -> %d\n", conn_s->cmd_fd, conn_s->stop);
#endif
}
static void *
fq_data_worker(void *u) {
int backoff = 0;
bool zero;
fq_conn_s *conn_s = (fq_conn_s *)u;
ck_pr_inc_uint(&conn_s->thrcnt);
while(conn_s->stop == 0) {
if(conn_s->data_ready) {
if(fq_client_data_connect_internal(conn_s) == 0) {
backoff = 0; /* we're good, restart our backoff */
}
fq_debug(FQ_DEBUG_IO, "[data] connected\n");
fq_data_worker_loop(conn_s);
fq_debug(FQ_DEBUG_IO, "[data] connection failed: %s\n", conn_s->error);
}
if(backoff) usleep(backoff + (4096 - (lrand48()%8192))); /* +/- 4ms */
else backoff = 16384;
if(backoff < 1000000) backoff += (backoff >> 4);
}
if(conn_s->data_fd >= 0) {
int toclose = conn_s->data_fd;
conn_s->data_fd = -1;
fq_debug(FQ_DEBUG_CONN, "close(data_fd)\n");
close(toclose);
}
ck_pr_dec_uint_zero(&conn_s->thrcnt, &zero);
if(zero) fq_conn_free(conn_s);
return (void *)NULL;
}
#define MAX_PENDING 128
static void *
fq_conn_worker(void *u) {
int backoff = 0, i;
bool zero;
fq_conn_s *conn_s = (fq_conn_s *)u;
cmd_instr *last_entries[MAX_PENDING] = { NULL };
int last_entry_idx = 0;
int next_entry_idx = 0;
#define SAVE_ENTRY(e) do { \
if(last_entries[next_entry_idx] != NULL) { \
CONNERR(conn_s, "exceed max cmd pipeline"); \
goto restart; \
} \
last_entries[next_entry_idx++] = e; \
next_entry_idx = next_entry_idx % MAX_PENDING; \
} while(0)
#define last_entry last_entries[last_entry_idx]
#define PROCESS_ENTRY(e, should_free) do { \
fq_assert(last_entries[last_entry_idx] == e); \
if(should_free) free(last_entries[last_entry_idx]); \
last_entries[last_entry_idx++] = NULL; \
last_entry_idx = last_entry_idx % MAX_PENDING; \
} while(0)
cmd_instr *entry;
fq_thread_setname("fq:c");
ck_pr_inc_uint(&conn_s->thrcnt);
while(conn_s->stop == 0) {
int wait_ms = 50;
if(fq_client_connect_internal(conn_s) == 0) {
fq_thread_setname("fq:c:%d", conn_s->cmd_fd);
backoff = 0; /* we're good, restart our backoff */
} else {
fq_thread_setname("fq:c");
}
while(conn_s->data_ready && conn_s->stop == 0) {
hrtime_t t;
unsigned long long hb_us;
struct timeval tv;
int rv;
ck_fifo_spsc_dequeue_lock(&conn_s->cmdq);
while(ck_fifo_spsc_dequeue(&conn_s->cmdq, &entry) == true) {
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "client acting on user req 0x%04x\n", entry->cmd);
#endif
switch(entry->cmd) {
case FQ_PROTO_STATUSREQ:
fq_debug(FQ_DEBUG_CONN, "sending status request\n");
if(fq_write_uint16(conn_s->cmd_fd, entry->cmd)) {
free(entry);
CONNERR(conn_s, "write failed (statusreq)");
goto restart;
}
SAVE_ENTRY(entry);
break;
case FQ_PROTO_HBREQ:
fq_debug(FQ_DEBUG_CONN, "sending heartbeat request\n");
if(fq_write_uint16(conn_s->cmd_fd, entry->cmd) ||
fq_write_uint16(conn_s->cmd_fd, entry->data.heartbeat.ms)) {
free(entry);
CONNERR(conn_s, "write failed (hbreq)");
goto restart;
}
conn_s->cmd_hb_ms = entry->data.heartbeat.ms;
tv.tv_sec = (unsigned long)entry->data.heartbeat.ms / 1000;
tv.tv_usec = 1000UL * (entry->data.heartbeat.ms % 1000);
if(setsockopt(conn_s->cmd_fd, SOL_SOCKET, SO_RCVTIMEO,
&tv, sizeof(tv)))
CONNERR(conn_s, strerror(errno));
tv.tv_sec = (unsigned long)entry->data.heartbeat.ms / 1000;
tv.tv_usec = 1000UL * (entry->data.heartbeat.ms % 1000);
if(setsockopt(conn_s->cmd_fd, SOL_SOCKET, SO_SNDTIMEO,
&tv, sizeof(tv)))
CONNERR(conn_s, strerror(errno));
conn_s->cmd_hb_last = fq_gethrtime();
free(entry);
break;
case FQ_PROTO_BINDREQ:
{
unsigned short flags = entry->data.bind->flags;
if(fq_write_uint16(conn_s->cmd_fd, entry->cmd) ||
fq_write_uint16(conn_s->cmd_fd, flags) ||
fq_write_short_cmd(conn_s->cmd_fd,
entry->data.bind->exchange.len,
entry->data.bind->exchange.name) < 0 ||
fq_write_short_cmd(conn_s->cmd_fd,
strlen(entry->data.bind->program),
entry->data.bind->program) < 0) {
CONNERR(conn_s, "write failed (bindreq)");
goto restart;
}
SAVE_ENTRY(entry);
}
break;
case FQ_PROTO_UNBINDREQ:
{
if(fq_write_uint16(conn_s->cmd_fd, entry->cmd) ||
fq_write_uint32(conn_s->cmd_fd, entry->data.unbind->route_id) ||
fq_write_short_cmd(conn_s->cmd_fd,
entry->data.unbind->exchange.len,
entry->data.unbind->exchange.name) < 0) {
CONNERR(conn_s, "write failed (unbindreq)");
goto restart;
}
SAVE_ENTRY(entry);
}
break;
default:
CONNERR(conn_s, "unknown user-side cmd");
free(entry);
}
}
ck_fifo_spsc_dequeue_unlock(&conn_s->cmdq);
if(conn_s->cmd_hb_needed) {
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "-> heartbeat\n");
#endif
if(fq_write_uint16(conn_s->cmd_fd, FQ_PROTO_HB)) {
CONNERR(conn_s, "write failed (hb)");
break;
}
conn_s->cmd_hb_needed = 0;
}
rv = fq_client_wfrw_internal(conn_s->cmd_fd, 1, 0, wait_ms, NULL);
if(rv == 0) {
wait_ms = (wait_ms >> 2) + wait_ms;
if(conn_s->cmd_hb_ms && wait_ms > conn_s->cmd_hb_ms)
wait_ms = conn_s->cmd_hb_ms;
if(wait_ms > 1000) wait_ms = 1000;
}
else wait_ms = 50;
fq_debug(FQ_DEBUG_CONN, "fq_client_wfrw_internal(cmd:%d) -> %d\n", conn_s->cmd_fd, rv);
t = fq_gethrtime();
hb_us = (unsigned long long)conn_s->cmd_hb_ms * 3 * 1000000ULL;
if(conn_s->cmd_hb_last && hb_us &&
conn_s->cmd_hb_last < (unsigned int) (t - hb_us)) {
char errbuf[256];
snprintf(errbuf, sizeof(errbuf), "heartbeat failed [%llu - %llu = %llu]",
(unsigned long long)t, (unsigned long long)conn_s->cmd_hb_last,
(unsigned long long)(t - conn_s->cmd_hb_last));
CONNERR(conn_s, errbuf);
break;
}
if(rv < 0) {
CONNERR(conn_s, strerror(errno));
break;
}
if(rv > 0) {
bool handled = false;
uint16_t hb;
if(fq_read_uint16(conn_s->cmd_fd, &hb)) break;
switch(hb) {
case FQ_PROTO_HB:
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "<- heartbeat\n");
#endif
conn_s->cmd_hb_last = fq_gethrtime();
conn_s->cmd_hb_needed = 1;
break;
case FQ_PROTO_STATUS:
if(!last_entry || last_entry->cmd != FQ_PROTO_STATUSREQ) {
CONNERR(conn_s, "protocol violation (status unexpected)");
goto restart;
}
if(fq_read_status(conn_s->cmd_fd,
last_entry->data.status.callback,
last_entry->data.status.closure))
goto restart;
PROCESS_ENTRY(last_entry, 1);
break;
case FQ_PROTO_BIND:
if(!last_entry || last_entry->cmd != FQ_PROTO_BINDREQ) {
CONNERR(conn_s, "protocol violation (bind unexpected)");
goto restart;
}
if(fq_read_uint32(conn_s->cmd_fd,
&last_entry->data.bind->out__route_id)) {
if(conn_s->bind_hook) {
if(conn_s->sync_hooks) {
enqueue_cmd_hook_req(conn_s, last_entry);
PROCESS_ENTRY(last_entry, 0);
handled = true;
}
else conn_s->bind_hook((fq_client)conn_s, last_entry->data.bind);
}
CONNERR(conn_s, "read failed (bind)");
goto restart;
}
if(conn_s->bind_hook) {
if(conn_s->sync_hooks) {
enqueue_cmd_hook_req(conn_s, last_entry);
PROCESS_ENTRY(last_entry, 0);
handled = true;
}
else conn_s->bind_hook((fq_client)conn_s, last_entry->data.bind);
}
if(!handled) PROCESS_ENTRY(last_entry, 1);
break;
case FQ_PROTO_UNBIND:
if(!last_entry || last_entry->cmd != FQ_PROTO_UNBINDREQ) {
CONNERR(conn_s, "protocol violation (unbind unexpected)");
goto restart;
}
if(fq_read_uint32(conn_s->cmd_fd,
&last_entry->data.unbind->out__success)) {
if(conn_s->unbind_hook) {
if(conn_s->sync_hooks) {
enqueue_cmd_hook_req(conn_s, last_entry);
PROCESS_ENTRY(last_entry, 0);
handled = true;
}
conn_s->unbind_hook((fq_client)conn_s, last_entry->data.unbind);
}
CONNERR(conn_s, "read failed (unbind)");
goto restart;
}
if(conn_s->unbind_hook) {
if(conn_s->sync_hooks) {
enqueue_cmd_hook_req(conn_s, last_entry);
PROCESS_ENTRY(last_entry, 0);
handled = true;
last_entry = NULL;
}
conn_s->unbind_hook((fq_client)conn_s, last_entry->data.unbind);
}
if(!handled) PROCESS_ENTRY(last_entry,1);
break;
default:
CONNERR(conn_s, "protocol violation");
goto restart;
break;
}
}
}
fq_debug(FQ_DEBUG_CONN, "[cmd] connection failed: %s\n", conn_s->error);
if(backoff) usleep(backoff + (4096 - (lrand48()%8192))); /* +/- 4ms */
else backoff = 16384;
if(backoff < 1000000) backoff += (backoff >> 4);
restart:
/* drain the queue.. we're going to make a new connection */
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "[cmd] draining cmds\n");
#endif
last_entry = NULL;
for(i=0;i<MAX_PENDING;i++) {
if(last_entries[i]) {
free(last_entries[i]);
last_entries[i] = NULL;
}
}
while(ck_fifo_spsc_dequeue(&conn_s->cmdq, &entry) == true) {
free(entry);
}
}
fq_client_disconnect_internal(conn_s);
ck_pr_dec_uint_zero(&conn_s->thrcnt, &zero);
if(zero) fq_conn_free(conn_s);
return (void *)NULL;
}
int
fq_client_init(fq_client *conn_ptr, uint32_t peermode,
void (*logger)(fq_client, const char *)) {
fq_conn_s *conn_s;
conn_s = *conn_ptr = calloc(1, sizeof(*conn_s));
if(!conn_s) return -1;
/* make the sockets as disconnected */
conn_s->cmd_fd = conn_s->data_fd = -1;
conn_s->peermode = peermode;
conn_s->errorlog = logger;
conn_s->thrcnt = 1;
return 0;
}
void
fq_client_set_userdata(fq_client conn, void *d) {
fq_conn_s *conn_s = (fq_conn_s *)conn;
conn_s->userdata = d;
}
void *
fq_client_get_userdata(fq_client conn) {
fq_conn_s *conn_s = (fq_conn_s *)conn;
return conn_s->userdata;
}
int
fq_client_hooks(fq_client conn, fq_hooks *hooks) {
fq_conn_s *conn_s = (fq_conn_s *)conn;
switch(hooks->version) {
case FQ_HOOKS_V4:
conn_s->message_hook = hooks->message;
conn_s->cleanup_hook = hooks->cleanup;
conn_s->disconnect_hook = hooks->disconnect;
case FQ_HOOKS_V3:
conn_s->sync_hooks = hooks->sync;
case FQ_HOOKS_V2:
conn_s->unbind_hook = hooks->unbind;
case FQ_HOOKS_V1:
conn_s->auth_hook = hooks->auth;
conn_s->bind_hook = hooks->bind;
break;
default:
return -1;
}
return 0;
}
int
fq_client_creds(fq_client conn, const char *host, unsigned short port,
const char *sender, const char *pass) {
char qname[39];
fq_conn_s *conn_s;
conn_s = conn;
if(conn_s->user) {
CONNERR(conn_s, "fq_client_creds already called");
return -1;
}
/* mark the sockets as disconnected */
conn_s->cmd_fd = conn_s->data_fd = -1;
/* parse the user info */
conn_s->user = strdup(sender);
conn_s->queue = strchr(conn_s->user, '/');
if(conn_s->queue) {
*conn_s->queue++ = '\0';
conn_s->queue_type = strchr(conn_s->queue, '/');
if(conn_s->queue_type) {
*conn_s->queue_type++ = '\0';
}
}
if(!conn_s->queue || conn_s->queue[0] == '\0') {
char *cp;
uuid_t out;
uuid_generate(out);
qname[0] = 'q'; qname[1] = '-';
uuid_unparse(out, qname+2);
for(cp=qname;*cp;cp++) *cp = tolower(*cp);
conn_s->queue = qname;
}
conn_s->queue_type = strdup(conn_s->queue_type ?
conn_s->queue_type :
FQ_DEFAULT_QUEUE_TYPE);
conn_s->queue = strdup(conn_s->queue);
conn_s->pass = strdup(pass);
conn_s->cmdqhead = malloc(sizeof(ck_fifo_spsc_entry_t));
ck_fifo_spsc_init(&conn_s->cmdq, conn_s->cmdqhead);
conn_s->qhead = malloc(sizeof(ck_fifo_spsc_entry_t));
ck_fifo_spsc_init(&conn_s->q, conn_s->qhead);
conn_s->backqhead = malloc(sizeof(ck_fifo_spsc_entry_t));
ck_fifo_spsc_init(&conn_s->backq, conn_s->backqhead);
conn_s->host = strdup(host);
conn_s->port = port;
return 0;
}
void
fq_client_status(fq_client conn,
void (*f)(char *, uint32_t, void *), void *c) {
fq_conn_s *conn_s = conn;
cmd_instr *e;
if(conn_s == 0 || conn_s->cmd_fd < 0) return;
e = malloc(sizeof(*e));
e->cmd = FQ_PROTO_STATUSREQ;
e->data.status.callback = f;
e->data.status.closure = c;
fq_client_signal(conn, e);
}
void
fq_client_heartbeat(fq_client conn, unsigned short heartbeat_ms) {
fq_conn_s *conn_s = conn;
cmd_instr *e;
if(conn_s->cmd_fd < 0) return;
e = malloc(sizeof(*e));
e->cmd = FQ_PROTO_HBREQ;
e->data.heartbeat.ms = heartbeat_ms;
fq_client_signal(conn, e);
}
void
fq_client_bind(fq_client conn, fq_bind_req *req) {
fq_conn_s *conn_s = conn;
cmd_instr *e;
if(conn_s->cmd_fd < 0) return;
e = malloc(sizeof(*e));
e->cmd = FQ_PROTO_BINDREQ;
e->data.bind = req;
fq_client_signal(conn, e);
}
void
fq_client_unbind(fq_client conn, fq_unbind_req *req) {
fq_conn_s *conn_s = conn;
cmd_instr *e;
if(conn_s->cmd_fd < 0) return;
e = malloc(sizeof(*e));
e->cmd = FQ_PROTO_UNBINDREQ;
e->data.unbind = req;
fq_client_signal(conn, e);
}
void
fq_client_set_backlog(fq_client conn, uint32_t len, uint32_t stall) {
fq_conn_s *conn_s = conn;
conn_s->qmaxlen = len;
conn_s->q_stall_time = stall;
}
void
fq_client_set_nonblock(fq_client conn, bool nonblock) {
fq_conn_s *conn_s = conn;
conn_s->non_blocking = nonblock;
}
int
fq_client_connect(fq_client conn) {
pthread_attr_t attr;
fq_conn_s *conn_s = conn;
if(conn_s->connected != 0) return -1;
conn_s->connected = 1;
if(pthread_attr_init(&attr) != 0) {
CONNERR(conn_s, "pthread_attr_init failed");
return -1;
}
if(pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED) != 0) {
CONNERR(conn_s, "pthread_attr_setdetachstate failed");
return -1;
}
if(pthread_create(&conn_s->worker, NULL, fq_conn_worker, conn_s) != 0) {
CONNERR(conn_s, "could not start command thread");
conn_s->stop = 1;
return -1;
}
if(pthread_create(&conn_s->data_worker, NULL, fq_data_worker, conn_s) != 0) {
CONNERR(conn_s, "could not start data thread");
conn_s->stop = 1;
return -1;
}
return 0;
}
void
fq_client_destroy(fq_client conn) {
bool zero;
fq_conn_s *conn_s = conn;
conn_s->stop = 1;
ck_pr_dec_uint_zero(&conn_s->thrcnt, &zero);
if(zero) fq_conn_free(conn_s);
}
int
fq_client_data_backlog(fq_client conn) {
fq_conn_s *conn_s = conn;
return ck_pr_load_uint(&conn_s->qlen);
}
int
fq_client_publish(fq_client conn, fq_msg *msg) {
fq_conn_s *conn_s = conn;
ck_fifo_spsc_enqueue_lock(&conn_s->q);
if(conn_s->non_blocking && conn_s->qlen >= conn_s->qmaxlen) {
ck_fifo_spsc_enqueue_unlock(&conn_s->q);
return -1;
}
ck_fifo_spsc_entry_t *fifo_entry = ck_fifo_spsc_recycle(&conn_s->q);
if (unlikely(fifo_entry == NULL)) {
fifo_entry = malloc(sizeof(ck_fifo_spsc_entry_t));
}
while(conn_s->qlen >= conn_s->qmaxlen) {
if(conn_s->q_stall_time > 0) usleep(conn_s->q_stall_time);
else ck_pr_stall();
}
fq_msg_ref(msg);
ck_fifo_spsc_enqueue(&conn_s->q, fifo_entry, msg);
ck_fifo_spsc_enqueue_unlock(&conn_s->q);
ck_pr_inc_uint(&conn_s->qlen);
return 1;
}
fq_msg *fq_client_receive(fq_client conn) {
bool success;
fq_conn_s *conn_s = conn;
fq_msg *m = NULL;
ck_fifo_spsc_dequeue_lock(&conn_s->backq);
success = ck_fifo_spsc_dequeue(&conn_s->backq, &m);
ck_fifo_spsc_dequeue_unlock(&conn_s->backq);
if(success && m && CHECK_HOOK_REQ_PTR(m)) {
hook_req_t *hreq = UNMARKED_HOOK_REQ_PTR(m);
m = NULL;
cmd_instr *entry = hreq->entry;
switch(hreq->type) {
case AUTH_HOOK_TYPE:
if(conn_s->sync_hooks && conn_s->auth_hook)
conn_s->auth_hook(conn_s, entry->data.return_value);
break;
case CMD_HOOK_TYPE:
switch(entry->cmd) {
case FQ_PROTO_BINDREQ:
if(conn_s->sync_hooks && conn_s->bind_hook)
conn_s->bind_hook(conn_s, entry->data.bind);
break;
case FQ_PROTO_UNBINDREQ:
if(conn_s->sync_hooks && conn_s->unbind_hook)
conn_s->unbind_hook(conn_s, entry->data.unbind);
break;
default:
snprintf(conn_s->error, sizeof(conn_s->error),
"sync cmd feedback unknown: %x\n", entry->cmd);
if(conn_s->errorlog) conn_s->errorlog(conn_s, conn_s->error);
}
break;
}
free(entry);
free(hreq);
}
return m;
}
================================================
FILE: fq_dtrace.blank.h
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#if 0
#else
#define FQ_QUEUE_ENQUEUE(arg0, arg1) \
do { \
} while (0)
#define FQ_QUEUE_ENQUEUE_ENABLED() (0)
#define FQ_CLIENT_AUTH(arg0) \
do { \
} while (0)
#define FQ_CLIENT_AUTH_ENABLED() (0)
#define FQ_CLIENT_AUTH_DATA(arg0) \
do { \
} while (0)
#define FQ_CLIENT_AUTH_DATA_ENABLED() (0)
#define FQ_CLIENT_CONNECT(arg0, arg1) \
do { \
} while (0)
#define FQ_CLIENT_CONNECT_ENABLED() (0)
#define FQ_CLIENT_DISCONNECT(arg0, arg1) \
do { \
} while (0)
#define FQ_CLIENT_DISCONNECT_ENABLED() (0)
#define FQ_CONFIG_ROTATE(arg0) \
do { \
} while (0)
#define FQ_CONFIG_ROTATE_ENABLED() (0)
#define FQ_MESSAGE_DELIVER(arg0, arg1, arg2) \
do { \
} while (0)
#define FQ_MESSAGE_DELIVER_ENABLED() (0)
#define FQ_MESSAGE_RECEIVE(arg0, arg1, arg2) \
do { \
} while (0)
#define FQ_MESSAGE_RECEIVE_ENABLED() (0)
#define FQ_QUEUE_BLOCK(arg0, arg1) \
do { \
} while (0)
#define FQ_QUEUE_BLOCK_ENABLED() (0)
#define FQ_QUEUE_CREATE_FAILURE(arg0, arg1, arg2) \
do { \
} while (0)
#define FQ_QUEUE_CREATE_FAILURE_ENABLED() (0)
#define FQ_QUEUE_CREATE_SUCCESS(arg0, arg1, arg2, arg3, arg4, arg5) \
do { \
} while (0)
#define FQ_QUEUE_CREATE_SUCCESS_ENABLED() (0)
#define FQ_QUEUE_DESTROY(arg0, arg1) \
do { \
} while (0)
#define FQ_QUEUE_DESTROY_ENABLED() (0)
#define FQ_QUEUE_DROP(arg0, arg1) \
do { \
} while (0)
#define FQ_QUEUE_DROP_ENABLED() (0)
#define FQ_ROUTE_PROGRAM_ENTRY(arg0, arg1) \
do { \
} while (0)
#define FQ_ROUTE_PROGRAM_ENTRY_ENABLED() (0)
#define FQ_ROUTE_PROGRAM_RETURN(arg0, arg1, arg2) \
do { \
} while (0)
#define FQ_ROUTE_PROGRAM_RETURN_ENABLED() (0)
#endif /* !defined(DTRACE_PROBES_DISABLED) || !DTRACE_PROBES_DISABLED */
================================================
FILE: fq_dtrace.d
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
typedef struct {
int dummy;
} fq_dtrace_msg_t;
typedef struct {
int dummy;
} fq_msg_t;
typedef struct {
int dummy;
} fq_dtrace_queue_t;
typedef struct {
int dummy;
} fq_queue_t;
typedef struct {
int dummy;
} fq_dtrace_remote_client_t;
typedef struct {
int dummy;
} fq_remote_client_t;
typedef struct {
int dummy;
} fq_dtrace_remote_anon_client_t;
typedef struct {
int dummy;
} fq_remote_anon_client_t;
typedef struct {
int dummy;
} fq_dtrace_remote_data_client_t;
typedef struct {
int dummy;
} fq_remote_data_client_t;
provider fq {
probe client__connect(fq_dtrace_remote_anon_client_t *c, int m) :
(fq_remote_anon_client_t *c, int m);
probe client__disconnect(fq_dtrace_remote_anon_client_t *c, int m) :
(fq_remote_anon_client_t *c, int m);
probe client__auth(fq_dtrace_remote_client_t *c) :
(fq_remote_client_t *c);
probe client__auth__data(fq_dtrace_remote_data_client_t *c) :
(fq_remote_data_client_t *c);
probe queue__create__success(int, char *, int, char *, int, int);
probe queue__create__failure(int, char *, char *);
probe queue__destroy(int, char *);
probe queue__drop(fq_dtrace_queue_t *q, fq_dtrace_msg_t *m) :
(fq_queue_t *q, fq_msg_t *m);
probe queue__block(fq_dtrace_queue_t *q, fq_dtrace_msg_t *m) :
(fq_queue_t *q, fq_msg_t *m);
probe queue__enqueue(fq_dtrace_queue_t *q, fq_dtrace_msg_t *m) :
(fq_queue_t *q, fq_msg_t *m);
probe config__rotate(int);
probe message__receive(fq_dtrace_remote_client_t *c,
fq_dtrace_remote_data_client_t *d,
fq_dtrace_msg_t *m) :
(fq_remote_client_t *c,
fq_remote_data_client_t *d,
fq_msg_t *m);
probe message__deliver(fq_dtrace_remote_client_t *c,
fq_dtrace_remote_data_client_t *d,
fq_dtrace_msg_t *m) :
(fq_remote_client_t *c,
fq_remote_data_client_t *d,
fq_msg_t *m);
probe route__program__entry(char *p, fq_dtrace_msg_t *m) :
(char *p, fq_msg_t *m);
probe route__program__return(char *p, fq_dtrace_msg_t *m, int32_t u) :
(char *p, fq_msg_t *m, int32_t u);
};
#pragma D attributes Evolving/Evolving/ISA provider fq provider
#pragma D attributes Private/Private/Unknown provider fq module
#pragma D attributes Private/Private/Unknown provider fq function
#pragma D attributes Private/Private/ISA provider fq name
#pragma D attributes Evolving/Evolving/ISA provider fq args
================================================
FILE: fq_msg.c
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include <pthread.h>
#include <stdio.h>
#include <string.h>
#include <stddef.h>
#include <stdlib.h>
#include "fq.h"
#include "ck_pr.h"
#include "ck_malloc.h"
#include "ck_hs.h"
#include "ck_fifo.h"
#define MSG_ALIGN sizeof(void *)
#define MAX_FREE_LIST_SIZE 1000000
#define unlikely(x) __builtin_expect(!!(x), 0)
static fq_msgid local_msgid = {
.id = {
.u32 = {
.p1 = 0,
.p2 = 0,
.p3 = 0,
.p4 = 0
}
}
};
static void
pull_next_local_msgid(fq_msgid *msgid) {
uint32_t last;
fq_msgid g;
again:
memcpy(&g, &local_msgid, sizeof(fq_msgid));
memcpy(msgid, &g, sizeof(fq_msgid));
last = ck_pr_faa_32(&local_msgid.id.u32.p1, 1);
msgid->id.u32.p1 = last + 1;
if(last == 0xffffffffUL) {
last = ck_pr_faa_32(&local_msgid.id.u32.p2, 1);
msgid->id.u32.p2 = last + 1;
if(last == 0xffffffffUL) {
last = ck_pr_faa_32(&local_msgid.id.u32.p3, 1);
msgid->id.u32.p3 = last + 1;
if(last == 0xffffffffUL) {
last = ck_pr_faa_32(&local_msgid.id.u32.p4, 1);
msgid->id.u32.p4 = last + 1;
}
}
}
if(msgid->id.u32.p4 < g.id.u32.p4) goto again;
if(msgid->id.u32.p4 > g.id.u32.p4) return;
if(msgid->id.u32.p3 < g.id.u32.p3) goto again;
if(msgid->id.u32.p3 > g.id.u32.p3) return;
if(msgid->id.u32.p2 < g.id.u32.p2) goto again;
if(msgid->id.u32.p2 > g.id.u32.p2) return;
if(msgid->id.u32.p1 > g.id.u32.p1) return;
goto again;
}
static inline fq_msg*
msg_allocate(const size_t s)
{
fq_msg *m = calloc(1, sizeof(fq_msg) + s);
if(!m) return NULL;
m->payload_len = s;
return m;
}
static inline void
msg_free(fq_msg *m)
{
if (m->free_fn != NULL) {
m->free_fn(m);
} else {
free(m);
}
}
fq_msg *
fq_msg_alloc(const void *data, size_t s) {
fq_msg *m = msg_allocate(s);
if (unlikely(m == NULL)) {
return NULL;
}
if(s) memcpy(m->payload, data, s);
#ifdef DEBUG
fq_debug(FQ_DEBUG_MSG, "msg(%p) -> alloc\n", (void *)m);
#endif
m->arrival_time = fq_gethrtime();
m->refcnt = 1;
return m;
}
fq_msg *
fq_msg_alloc_BLANK(size_t s) {
fq_msg *m = msg_allocate(s);
if (unlikely(m == NULL)) {
return NULL;
}
#ifdef DEBUG
fq_debug(FQ_DEBUG_MSG, "msg(%p) -> alloc\n", (void *)m);
#endif
m->arrival_time = fq_gethrtime();
m->refcnt = 1;
return m;
}
void
fq_msg_ref(fq_msg *msg) {
ck_pr_inc_uint(&msg->refcnt);
#ifdef DEBUG
fq_debug(FQ_DEBUG_MSG, "msg(%p) -> ref: %d\n", (void *)msg, msg->refcnt);
#endif
}
void
fq_msg_deref(fq_msg *msg) {
bool zero;
ck_pr_dec_uint_zero(&msg->refcnt, &zero);
if(zero) {
#ifdef DEBUG
fq_debug(FQ_DEBUG_MSG, "msg(%p) -> free\n", (void *)msg);
#endif
msg_free(msg);
}
#ifdef DEBUG
else {
fq_debug(FQ_DEBUG_MSG, "msg(%p) -> deref: %d\n", (void *)msg, msg->refcnt);
}
#endif
}
void
fq_msg_exchange(fq_msg *msg, const void *exchange, int rlen) {
if(rlen <= 0) {
msg->exchange.len = 0;
return;
}
if(rlen > MAX_RK_LEN) rlen = MAX_RK_LEN;
msg->exchange.len = rlen;
memcpy(msg->exchange.name, exchange, rlen);
}
void
fq_msg_route(fq_msg *msg, const void *route, int rlen) {
if(rlen <= 0) {
msg->route.len = 0;
return;
}
if(rlen > MAX_RK_LEN) rlen = MAX_RK_LEN;
msg->route.len = rlen;
memcpy(msg->route.name, route, rlen);
}
void
fq_msg_id(fq_msg *msg, fq_msgid *id) {
if(!id) pull_next_local_msgid(&msg->sender_msgid);
else memcpy(&msg->sender_msgid, id, sizeof(*id));
}
================================================
FILE: fq_rcvr.c
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <string.h>
#include "fq.h"
char *exchange = "maryland";
char *program = "prefix:\"\"";
int output = 1;
void logger(fq_client, const char *);
void logger(fq_client c, const char *s) {
(void)c;
fprintf(stderr, "fq_logger: %s\n", s);
}
static void
print_rate(fq_client c, hrtime_t s, hrtime_t f, uint64_t cnt, uint64_t icnt) {
double d;
if(cnt) {
d = (double)cnt * 1000000000;
d /= (double)(f-s);
printf("[%d backlog] output %0.2f msg/sec\n",
fq_client_data_backlog(c), d);
}
if(icnt) {
d = (double)icnt * 1000000000;
d /= (double)(f-s);
printf("[%d backlog] input %0.2f msg/sec\n",
fq_client_data_backlog(c), d);
}
}
static void
my_auth_handler(fq_client c, int error) {
fq_bind_req *breq;
if(error) return;
printf("attempting bind\n");
breq = malloc(sizeof(*breq));
memset(breq, 0, sizeof(*breq));
int exchange_len = strlen(exchange);
memcpy(breq->exchange.name, exchange, exchange_len);
breq->exchange.len = exchange_len;
breq->flags = FQ_BIND_TRANS;
breq->program = program;
fq_client_bind(c, breq);
}
static void
my_bind_handler(fq_client c, fq_bind_req *breq) {
(void)c;
printf("route set -> %u\n", breq->out__route_id);
if(breq->out__route_id == FQ_BIND_ILLEGAL) {
fprintf(stderr, "Failure to bind...\n");
exit(-1);
}
}
fq_hooks hooks = {
.version = FQ_HOOKS_V1,
.auth = my_auth_handler,
.bind = my_bind_handler
};
int main(int argc, char **argv) {
hrtime_t s, f;
uint64_t cnt = 0, icnt = 0, icnt_total = 0;
int rcvd = 0;
fq_client c;
fq_msg *m;
char *fq_debug = getenv("FQ_DEBUG");
if(fq_debug) fq_debug_set_bits(atoi(fq_debug));
signal(SIGPIPE, SIG_IGN);
fq_client_init(&c, 0, logger);
if(fq_client_hooks(c, &hooks)) {
fprintf(stderr, "Can't register hooks\n");
exit(-1);
}
char *host = "localhost";
int port = 8765;
char *user = "guest";
char *pass = "guest";
int o;
while(-1 != (o = getopt(argc, argv, "h:p:u:P:e:b:s"))) {
switch(o) {
case 'h': host = strdup(optarg); break;
case 'p': port = atoi(optarg); break;
case 'u': user = strdup(optarg); break;
case 'P': pass = strdup(optarg); break;
case 'e': exchange = strdup(optarg); break;
case 'b': program = strdup(optarg); break;
case 's': output = 2; break;
default:
fprintf(stderr, "%s [-h host] [-p port] [-u user] [-P pass] [-e exchange] [-b program] [-s]\n",
argv[0]);
exit(-1);
break;
}
}
fq_client_hooks(c, &hooks);
fq_client_creds(c, host, port, user, pass);
fq_client_heartbeat(c, 1000);
fq_client_set_backlog(c, 10000, 100);
fq_client_connect(c);
s = fq_gethrtime();
while(1) {
f = fq_gethrtime();
while(NULL != (m = fq_client_receive(c))) {
icnt++;
icnt_total++;
rcvd++;
if(output == 1) {
int ending = m->payload[m->payload_len-1] == '\n' ? 1 : 0;
printf("[%.*s] %.*s\n", m->route.len, m->route.name, m->payload_len - ending, m->payload);
}
fq_msg_deref(m);
}
usleep(1000);
if(f-s > 1000000000) {
if(output == 2) {
print_rate(c, s, f, cnt, icnt);
printf("total: %llu\n", (unsigned long long)icnt_total);
}
icnt = 0;
cnt = 0;
s = f;
}
}
(void) argc;
return 0;
}
================================================
FILE: fq_sndr.c
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <string.h>
#include "fq.h"
#define SEND_COUNT 1000
int send_count = SEND_COUNT;
void logger(fq_client c, const char *);
void logger(fq_client c, const char *s) {
(void)c;
fprintf(stderr, "fq_logger: %s\n", s);
}
static void
debug_status(char *key, uint32_t value, void *unused) {
(void)unused;
fq_debug(FQ_DEBUG_CONN, " ---> %s : %u\n", key, value);
}
static void
print_rate(fq_client c, hrtime_t s, hrtime_t f, uint64_t cnt, uint64_t icnt) {
double d;
fq_client_status(c, debug_status, NULL);
if(cnt) {
d = (double)cnt * 1000000000;
d /= (double)(f-s);
printf("[%d backlog] output %0.2f msg/sec\n",
fq_client_data_backlog(c), d);
}
if(icnt) {
d = (double)icnt * 1000000000;
d /= (double)(f-s);
printf("[%d backlog] input %0.2f msg/sec\n",
fq_client_data_backlog(c), d);
}
}
int main(int argc, char **argv) {
hrtime_t s0, s, f, f0;
uint64_t cnt = 0, icnt = 0;
int psize = 0, i = 0;
fq_client c;
fq_msg *m;
char *fq_debug = getenv("FQ_DEBUG");
if(fq_debug) fq_debug_set_bits(atoi(fq_debug));
signal(SIGPIPE, SIG_IGN);
fq_client_init(&c, 0, logger);
if(argc < 5) {
fprintf(stderr, "%s <host> <port> <user> <pass> [size [count]]\n",
argv[0]);
exit(-1);
}
fq_client_creds(c, argv[1], atoi(argv[2]), argv[3], argv[4]);
fq_client_heartbeat(c, 1000);
fq_client_set_backlog(c, 10000, 100);
fq_client_connect(c);
if(argc > 5) {
psize = atoi(argv[5]);
}
printf("payload size -> %d\n", psize);
if(argc > 6) {
send_count = atoi(argv[6]);
}
printf("message count -> %d\n", send_count);
s0 = s = fq_gethrtime();
while(i < send_count || fq_client_data_backlog(c) > 0) {
if(i < send_count) {
m = fq_msg_alloc_BLANK(psize);
memset(m->payload, 0, psize);
fq_msg_exchange(m, "maryland", 8);
fq_msg_route(m, "test.prefix.boo", 15);
fq_msg_id(m, NULL);
fq_client_publish(c, m);
cnt++;
i++;
fq_msg_free(m);
}
else usleep(100);
f = fq_gethrtime();
if(f-s > 1000000000) {
print_rate(c, s, f, cnt, icnt);
icnt = 0;
cnt = 0;
s = f;
}
}
f0 = fq_gethrtime();
print_rate(c, s0, f0, i, 0);
(void) argc;
return 0;
}
================================================
FILE: fq_utils.c
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include "fq.h"
#include "fqd.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <strings.h>
#include <ctype.h>
#include <unistd.h>
#include <sys/uio.h>
#include <sys/socket.h>
#include <errno.h>
#include <pthread.h>
#include <stdarg.h>
#include <execinfo.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <inttypes.h>
#include <assert.h>
static inline int msg_free_stack_select(ssize_t in);
typedef struct free_message_stack {
ck_stack_t stack;
uint32_t size;
uint32_t max_size;
size_t alloc_size;
} free_message_stack;
/* We support separate stacks for separate msg sizes...
* containers are from 2^10 (1k) to 2^16 (65k).
* Messages are allocated from the smallest stack that can contain them.
* Otherwise, they are traditionally allocated.
*/
#define MSG_FREE_BASE 10
#define MSG_FREE_CEILING 16
#define MSG_FREE_STACKS (MSG_FREE_CEILING-MSG_FREE_BASE+1)
/* the handles are TLS, they on heap and allocated TLS, and freed to a pool.
* once a handle is allocated it never actually freed. */
struct msg_free_stacks_handle_t {
free_message_stack *stacks[MSG_FREE_STACKS];
bool valid;
};
struct handle_free_list {
msg_free_stacks_handle_t *handle;
struct handle_free_list *next;
};
/* this is actually in <sys/sysmacros.h> on illumos but flagged off for some reason */
#ifndef container_of
#define container_of(m, s, name) \
(void *)((uintptr_t)(m) - (uintptr_t)offsetof(s, name))
#endif
uint32_t fq_debug_bits = FQ_DEBUG_PANIC;
void fq_debug_set_bits(uint32_t bits) {
fq_debug_bits = bits | FQ_DEBUG_PANIC;
}
static void
fq_init_free_message_stack(free_message_stack *stack, const size_t max_free_count,
const size_t alloc_size)
{
ck_stack_init(&stack->stack);
stack->size = 0;
stack->max_size = max_free_count;
stack->alloc_size = alloc_size;
}
static inline fq_msg *
fq_pop_free_message_stack(struct free_message_stack *stack)
{
fq_msg *rv = NULL;
if (stack == NULL) {
return rv;
}
ck_stack_entry_t *ce = ck_stack_pop_mpmc(&stack->stack);
if (ce != NULL) {
ck_pr_dec_32(&stack->size);
rv = container_of(ce, fq_msg, cleanup_stack_entry);
}
return rv;
}
static inline void
fq_push_free_message_stack(struct free_message_stack *stack, fq_msg *m)
{
if (stack == NULL) {
return;
}
while(ck_pr_load_32(&stack->size) > stack->max_size) {
ck_stack_entry_t *ce = ck_stack_pop_mpmc(&stack->stack);
if (ce != NULL) {
fq_msg *m = container_of(ce, fq_msg, cleanup_stack_entry);
free(m);
ck_pr_dec_32(&stack->size);
}
else break;
}
uint32_t c = ck_pr_load_32(&stack->size);
if (c >= stack->max_size) {
free(m);
return;
}
ck_pr_inc_32(&stack->size);
ck_stack_push_mpmc(&stack->stack, &m->cleanup_stack_entry);
}
static void
fq_free_msg_fn(fq_msg *m)
{
if (m->cleanup_handle && m->cleanup_handle->valid) {
int idx = msg_free_stack_select(m->payload_len);
if(idx >= 0) {
fq_push_free_message_stack(m->cleanup_handle->stacks[idx], m);
return;
}
}
free(m);
}
void fq_debug_set_string(const char *s) {
char *lastsep, *tok = NULL;
char copy[128];
unsigned long nv;
int slen;
if(!s) return;
/* then comma separated named */
slen = strlen(s);
if(slen < 0 || slen > sizeof(copy) - 1) return;
/* copy including null terminator */
memcpy(copy,s,slen+1);
/* First try decimal */
nv = strtoul(copy,&lastsep,10);
if(*lastsep == '\0') {
fq_debug_set_bits(nv);
return;
}
/* Then try hex */
nv = strtoul(copy,&lastsep,16);
if(*lastsep == '\0') {
fq_debug_set_bits(nv);
return;
}
for (tok = strtok_r(copy, ",", &lastsep);
tok;
tok = strtok_r(NULL, ",", &lastsep)) {
#define SETBIT(tok, A) do { \
if(!strcasecmp(tok, #A + 9)) fq_debug_bits |= A; \
} while(0)
SETBIT(tok, FQ_DEBUG_MEM);
SETBIT(tok, FQ_DEBUG_MSG);
SETBIT(tok, FQ_DEBUG_ROUTE);
SETBIT(tok, FQ_DEBUG_IO);
SETBIT(tok, FQ_DEBUG_CONN);
SETBIT(tok, FQ_DEBUG_CONFIG);
SETBIT(tok, FQ_DEBUG_PEER);
SETBIT(tok, FQ_DEBUG_HTTP);
if(lastsep == NULL) break;
}
}
#define IN_READ_BUFFER_SIZE 1024*128
#define FREE_MSG_LIST_SIZE 100000000 /* in bytes */
#define CAPPED(x) (((x)<(MAX_MESSAGE_SIZE))?(x):(MAX_MESSAGE_SIZE))
struct buffered_msg_reader {
unsigned char scratch[IN_READ_BUFFER_SIZE];
int fd;
int off;
uint32_t peermode;
ssize_t nread;
ssize_t into_body;
fq_msg *copy;
fq_msg *msg;
};
static struct handle_free_list *free_message_handle_list = NULL;
static pthread_mutex_t free_message_handle_list_lock = PTHREAD_MUTEX_INITIALIZER;
static inline msg_free_stacks_handle_t *free_message_handle_acquire(void) {
msg_free_stacks_handle_t *a = NULL;
pthread_mutex_lock(&free_message_handle_list_lock);
if(free_message_handle_list) {
struct handle_free_list *tofree = free_message_handle_list;
free_message_handle_list = tofree->next;
a = tofree->handle;
free(tofree);
}
pthread_mutex_unlock(&free_message_handle_list_lock);
if(!a) a = calloc(1, sizeof(*a));
a->valid = true;
return a;
}
static inline int msg_free_stack_select(ssize_t in) {
int i;
if(in <= (1 << MSG_FREE_BASE)) return 0;
in--;
in >>= MSG_FREE_BASE+1;
for(i = 1; i < MSG_FREE_STACKS && in; i++, in >>= 1);
if(i < MSG_FREE_STACKS) return i;
return -1;
}
static __thread msg_free_stacks_handle_t *tls_free_message_handle = NULL;
buffered_msg_reader *fq_buffered_msg_reader_alloc(int fd, uint32_t peermode) {
buffered_msg_reader *br;
br = calloc(1, sizeof(*br));
br->fd = fd;
br->peermode = peermode;
br->msg = fq_msg_alloc_BLANK(0);
return br;
}
void fq_buffered_msg_reader_free(buffered_msg_reader *f) {
assert(f->msg->refcnt == 1);
fq_msg_deref(f->msg);
if(f->copy) fq_msg_deref(f->copy);
free(f);
}
static int
parse_message_headers(int peermode, unsigned char *d, int dlen,
fq_msg *msg) {
int ioff = 0;
unsigned char exchange_len, route_len, sender_len, nhops;
#define BAIL_UNLESS_LEFT(d) do { \
if((dlen-ioff) < (int)(d)) return 0; \
} while(0)
BAIL_UNLESS_LEFT(sizeof(exchange_len));
memcpy(&exchange_len, d+ioff, sizeof(exchange_len));
ioff += sizeof(exchange_len);
if(exchange_len > sizeof(msg->exchange.name)) return -1;
msg->exchange.len = exchange_len;
BAIL_UNLESS_LEFT(exchange_len);
memcpy(msg->exchange.name, d+ioff, exchange_len);
ioff += exchange_len;
BAIL_UNLESS_LEFT(sizeof(route_len));
memcpy(&route_len, d+ioff, sizeof(route_len));
ioff += sizeof(route_len);
if(route_len > sizeof(msg->route.name)) return -2;
msg->route.len = route_len;
BAIL_UNLESS_LEFT(route_len);
memcpy(msg->route.name, d+ioff, route_len);
ioff += route_len;
BAIL_UNLESS_LEFT(sizeof(msg->sender_msgid));
memcpy(&msg->sender_msgid, d+ioff, sizeof(msg->sender_msgid));
ioff += sizeof(msg->sender_msgid);
if(peermode) {
/* Peer mode includes the sender and the hops */
BAIL_UNLESS_LEFT(sizeof(sender_len));
memcpy(&sender_len, d+ioff, sizeof(sender_len));
ioff += sizeof(sender_len);
if(sender_len > sizeof(msg->sender.name)) return -3;
msg->sender.len = sender_len;
BAIL_UNLESS_LEFT(sender_len);
memcpy(msg->sender.name, d+ioff, sender_len);
ioff += sender_len;
BAIL_UNLESS_LEFT(sizeof(nhops));
memcpy(&nhops, d+ioff, sizeof(nhops));
ioff += sizeof(nhops);
if(nhops > MAX_HOPS) return -4;
if(nhops > 0) {
BAIL_UNLESS_LEFT(sizeof(uint32_t) * nhops);
memcpy(msg->hops, d+ioff, sizeof(uint32_t) * nhops);
ioff += sizeof(uint32_t) * nhops;
}
}
BAIL_UNLESS_LEFT(sizeof(msg->payload_len));
memcpy(&msg->payload_len, d+ioff, sizeof(msg->payload_len));
msg->payload_len = ntohl(msg->payload_len);
ioff += sizeof(msg->payload_len);
return ioff;
}
void
fq_clear_message_cleanup_stack()
{
int i;
if(tls_free_message_handle == NULL) return;
tls_free_message_handle->valid = false;
for(i=0; i<MSG_FREE_STACKS; i++) {
if (tls_free_message_handle->stacks[i]) {
tls_free_message_handle->stacks[i]->size = 0;
ck_stack_entry_t *ce = ck_stack_batch_pop_mpmc(&tls_free_message_handle->stacks[i]->stack);
while (ce != NULL) {
fq_msg *m = container_of(ce, fq_msg, cleanup_stack_entry);
ce = ce->next;
free(m);
}
}
}
struct handle_free_list *node = calloc(1, sizeof(*node));
pthread_mutex_lock(&free_message_handle_list_lock);
node->handle = tls_free_message_handle;
node->next = free_message_handle_list;
free_message_handle_list = node;
pthread_mutex_unlock(&free_message_handle_list_lock);
}
/*
* return 0: keep going (to write path)
* return -1: busted
*
* Read into one of N buffers so the processing thread
* can do the work separate from the read
*/
int
fq_buffered_msg_read(buffered_msg_reader *f,
void (*f_msg_handler)(void *, fq_msg *),
void *closure) {
int rv;
static char scratch_buf[IN_READ_BUFFER_SIZE];
while(f->into_body < f->msg->payload_len) {
fq_assert(f->copy);
/* we need to be reading a largish payload */
if(f->into_body >= MAX_MESSAGE_SIZE) {
/* read into a scratch buffer */
size_t readsize = f->copy->payload_len - f->into_body;
if(readsize > sizeof(scratch_buf)) readsize = sizeof(scratch_buf);
while((rv = read(f->fd, scratch_buf, readsize)) == -1 && errno == EINTR);
}
else {
while((rv = read(f->fd, f->copy->payload + f->into_body,
CAPPED(f->copy->payload_len) - f->into_body)) == -1 && errno == EINTR);
}
if(rv < 0 && errno == EAGAIN) return 0;
if(rv <= 0) {
fq_debug(FQ_DEBUG_IO, "read error: %s\n", rv < 0 ? strerror(errno) : "end-of-line");
return -1;
}
fq_debug(FQ_DEBUG_MSG, "%p <-- %d bytes for payload\n", (void *)f, rv);
f->into_body += rv;
if(f->into_body == f->copy->payload_len) {
f->into_body = 0;
goto message_done;
}
}
while((rv = read(f->fd, f->scratch+f->nread, sizeof(f->scratch)-f->nread)) == -1 &&
errno == EINTR);
fq_debug(FQ_DEBUG_IO, "%p <-- %d bytes @ %d (%d)\n", (void *)f, rv, (int)f->nread,
(int)f->nread + ((rv > 0) ? rv : 0));
if(rv == -1 && errno == EAGAIN) return 0;
if(rv <= 0) return -1;
f->nread += rv;
while(f->nread>0) {
uint32_t body_available;
int body_start;
body_start = parse_message_headers(f->peermode,
f->scratch+f->off, f->nread-f->off,
f->msg);
f->into_body = 0;
fq_debug(FQ_DEBUG_MSG, "%d = parse(+%d, %d) -> %d\n",
body_start, f->off, (int)f->nread-f->off,
body_start ? (int)f->msg->payload_len : 0);
if(body_start < 0) return -1;
if(!body_start) {
fq_debug(FQ_DEBUG_MSG, "incomplete message header...\n");
memmove(f->scratch, f->scratch + f->off, f->nread - f->off);
f->nread -= f->off;
f->off = 0;
return 0;
}
free_message_stack *tls_free_message_stack = NULL;
int msg_stack_idx = msg_free_stack_select(f->msg->payload_len);
if(msg_stack_idx >= 0) {
if(tls_free_message_handle == NULL)
tls_free_message_handle = free_message_handle_acquire();
if(tls_free_message_handle->stacks[msg_stack_idx] == NULL) {
/* lazy create/init the cleanup stack */
tls_free_message_handle->stacks[msg_stack_idx] = malloc(sizeof(free_message_stack));
fq_init_free_message_stack(tls_free_message_handle->stacks[msg_stack_idx],
FREE_MSG_LIST_SIZE/(1 << (msg_stack_idx + MSG_FREE_BASE)),
(1 << (msg_stack_idx + MSG_FREE_BASE)));
}
tls_free_message_stack = tls_free_message_handle->stacks[msg_stack_idx];
}
if(tls_free_message_stack) {
/* We have a message... or the formal beginnings of one */
f->copy = fq_pop_free_message_stack(tls_free_message_stack);
if (f->copy == NULL) {
/* ran out of entries in free list */
f->copy = fq_msg_alloc_BLANK(tls_free_message_stack->alloc_size);
if (f->copy == NULL) {
/* this is bad, we can't alloc */
fq_debug(FQ_DEBUG_MSG, "unable to malloc, OOM?\n");
return -1;
}
}
/* always 1 as this msg only lives until it's copied by a worker thread */
memcpy(f->copy, f->msg, sizeof(fq_msg));
f->copy->refcnt = 1;
f->copy->free_fn = fq_free_msg_fn;
} else {
f->copy = fq_msg_alloc_BLANK(CAPPED(f->msg->payload_len));
if (f->copy == NULL) {
/* this is bad, we can't alloc */
fq_debug(FQ_DEBUG_MSG, "unable to malloc, OOM?\n");
return -1;
}
memcpy(f->copy, f->msg, sizeof(fq_msg));
f->copy->refcnt = 1;
f->copy->free_fn = NULL;
}
/* assign the cleanup stack for this message */
f->copy->cleanup_handle = tls_free_message_stack ? tls_free_message_handle : NULL;
memset(&f->copy->cleanup_stack_entry, 0, sizeof(ck_stack_entry_t));
f->off += body_start;
body_available = f->nread - f->off;
if(f->copy->payload_len < body_available) body_available = f->copy->payload_len;
memcpy(f->copy->payload, f->scratch+f->off, CAPPED(body_available));
if(body_available == f->copy->payload_len) {
f->off += body_available;
message_done:
f->copy->refcnt = 1;
f->copy->payload_len = CAPPED(f->copy->payload_len);
fq_debug(FQ_DEBUG_MSG, "message read... injecting\n");
f->copy->arrival_time = fq_gethrtime();
f_msg_handler(closure, f->copy);
f->copy = NULL;
memset(f->msg, 0, sizeof(fq_msg));
/* It is still allocated and we are the sole owner, refcnt must be 1 */
f->msg->refcnt = 1;
}
else {
f->nread = 0;
f->off = 0;
f->into_body = body_available;
fq_debug(FQ_DEBUG_MSG, "incomplete message... (%d needed)\n",
(int)f->msg->payload_len - (int)f->into_body);
return 0;
}
}
return 0;
}
#if defined(BSD) || defined(__FreeBSD__)
#include <time.h>
#define NANOSEC 1000000000
hrtime_t fq_gethrtime() {
struct timespec ts;
clock_gettime(CLOCK_UPTIME,&ts);
return (((u_int64_t) ts.tv_sec) * NANOSEC + ts.tv_nsec);
}
#elif defined(linux) || defined(__linux) || defined(__linux__)
#include <time.h>
hrtime_t fq_gethrtime() {
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC_RAW, &ts);
return ((ts.tv_sec * 1000000000) + ts.tv_nsec);
}
#elif defined(__MACH__)
#include <mach/mach.h>
#include <mach/mach_time.h>
static int initialized = 0;
static mach_timebase_info_data_t sTimebaseInfo;
hrtime_t fq_gethrtime() {
uint64_t t;
if(!initialized) {
if(sTimebaseInfo.denom == 0)
(void) mach_timebase_info(&sTimebaseInfo);
}
t = mach_absolute_time();
return t * sTimebaseInfo.numer / sTimebaseInfo.denom;
}
#elif defined(__sun)
inline hrtime_t fq_gethrtime() {
return gethrtime();
}
#else
#error "Unknown platform for clock implementation"
#endif
int fq_rk_to_hex(char *buf, int len, fq_rk *k) {
int i;
unsigned char *bout = (unsigned char *)buf;
if(k->len * 2 + 4 > len) return -1;
*bout++ = '0';
*bout++ = 'x';
for (i=0; i<k->len; i++) {
snprintf((char *)bout, 3, "%02x", k->name[i]);
bout+=2;
}
*bout = '\0';
return (bout - (unsigned char *)buf);
}
int
fq_read_uint16(int fd, unsigned short *v) {
unsigned short nlen;
int rv;
while((rv = read(fd, &nlen, sizeof(nlen))) == -1 && errno == EINTR);
if(rv != sizeof(nlen)) return -1;
*v = ntohs(nlen);
return 0;
}
int
fq_write_uint16(int fd, unsigned short v) {
uint16_t nv;
int rv;
nv = htons(v);
while((rv = write(fd, &nv, sizeof(nv))) == -1 && errno == EINTR);
return (rv == sizeof(nv)) ? 0 : -1;
}
int
fq_read_uint32(int fd, uint32_t *v) {
uint32_t nlen;
int rv;
while((rv = read(fd, &nlen, sizeof(nlen))) == -1 && errno == EINTR);
if(rv != sizeof(nlen)) return -1;
*v = ntohl(nlen);
return 0;
}
int
fq_write_uint32(int fd, uint32_t v) {
uint32_t nv;
int rv;
nv = htonl(v);
while((rv = write(fd, &nv, sizeof(nv))) == -1 && errno == EINTR);
return (rv == sizeof(nv)) ? 0 : -1;
}
int
fq_read_short_cmd(int fd, unsigned short buflen, void *buf) {
void *tgt = buf;
unsigned char scratch[0xffff];
unsigned short nlen, len;
int rv;
while((rv = read(fd, &nlen, sizeof(nlen))) == -1 && errno == EINTR);
if(rv < 0 || rv != sizeof(nlen)) return -1;
len = ntohs(nlen);
if(len == 0) return 0;
if(len > buflen)
tgt = scratch;
while((rv = read(fd, tgt, len)) == -1 && errno == EINTR);
if(rv != len) {
return -1;
}
if(tgt != buf) memcpy(buf, tgt, buflen); /* truncated */
return rv;
}
int
fq_read_status(int fd, void (*f)(char *, uint32_t, void *), void *closure) {
while(1) {
char key[0x10000];
int len;
uint32_t value;
len = fq_read_short_cmd(fd, 0xffff, key);
if(len < 0) return -1;
if(len == 0) break;
key[len] = '\0';
if(fq_read_uint32(fd, &value) < 0) return -1;
f(key, value, closure);
}
return 0;
}
int
fq_write_short_cmd(int fd, unsigned short buflen, const void *buf) {
unsigned short nlen;
int rv;
nlen = htons(buflen);
while((rv = write(fd, &nlen, sizeof(nlen))) == -1 && errno == EINTR);
if(rv != sizeof(nlen)) return -1;
if(buflen == 0) return 0;
while((rv = write(fd, buf, buflen)) == -1 && errno == EINTR);
if(rv != buflen) return -1;
return rv;
}
int
fq_read_long_cmd(int fd, int *rlen, void **rbuf) {
unsigned int nlen;
int rv, len;
while((rv = read(fd, &nlen, sizeof(nlen))) == -1 && errno == EINTR);
if(rv < 0 || rv != sizeof(nlen)) return -1;
len = ntohl(nlen);
*rlen = 0;
*rbuf = NULL;
if(len < 0) {
return -1;
}
else if(len > 0) {
*rbuf = malloc(len);
while((rv = read(fd, *rbuf, len)) == -1 && errno == EINTR);
if(rv != len) {
free(*rbuf);
*rlen = 0;
*rbuf = NULL;
return -1;
}
*rlen = rv;
}
return *rlen;
}
int
fq_debug_fl(const char *file, int line, fq_debug_bits_t b, const char *fmt, ...) {
int rv;
va_list argp;
static hrtime_t epoch = 0;
hrtime_t now;
char fmtstring[1024];
uint64_t p = (uint64_t)pthread_self();
uint32_t ps = p & 0xffffffff;
(void)b;
now = fq_gethrtime();
if(!epoch) epoch = now;
snprintf(fmtstring, sizeof(fmtstring), "[%" PRIu64 "] [%08x] %s",
(uint64_t)((now-epoch)/1000), ps, fmt);
va_start(argp, fmt);
rv = vfprintf(stderr, fmtstring, argp);
va_end(argp);
(void)file;
(void)line;
return rv;
}
void
fq_debug_stacktrace(fq_debug_bits_t b, const char *tag, int start, int end) {
#define STACK_DEPTH 16
int i, cnt;
void *bti[STACK_DEPTH + 1], **bt = bti+1;
char **btname;
cnt = backtrace(bti, STACK_DEPTH + 1);
if(cnt < 1) {
fq_debug(b, "track trace failed\n");
return;
}
btname = backtrace_symbols(bt, cnt);
if(start > cnt) start = cnt;
if(end > cnt) end = cnt;
for(i=start;i!=end;i += (start > end) ? -1 : 1) {
if(btname && btname[i])
fq_debug(b, "[%2d] %s %s\n", i, tag, btname[i]);
else
fq_debug(b, "[%2d] %s %p\n", i, tag, bt[i]);
}
if(btname) free(btname);
}
int fq_serialize(struct iovec **vecs, int *vec_count, uint32_t peermode, size_t off, fq_msg *m)
{
int i, writev_start = 0, idx = 0;
size_t expect = 0;
uint32_t data_len = htonl(m->payload_len);
uint8_t nhops = 0;
uint8_t sender_len = m->sender.len;
uint8_t exchange_len = m->exchange.len;
uint8_t route_len = m->route.len;
if (vecs == NULL) {
return -1;
}
*vec_count = 7 + (peermode ? 4 : 0);
/* 7 for normal + 4 for peer */
*vecs = calloc(*vec_count, sizeof(struct iovec));
struct iovec *pv = *vecs;
expect = 1 + m->exchange.len + 1 + m->route.len +
sizeof(m->sender_msgid) +
sizeof(data_len) + m->payload_len;
if(peermode) {
for(i = 0; i < MAX_HOPS; i++) {
if(m->hops[i] == 0) break;
nhops++;
}
expect += 1 + m->sender.len + 1 + (nhops * sizeof(uint32_t));
}
fq_assert(off < expect);
expect -= off;
pv[idx ].iov_len = 1;
pv[idx++].iov_base = &exchange_len;
pv[idx ].iov_len = m->exchange.len;
pv[idx++].iov_base = m->exchange.name;
pv[idx ].iov_len = 1;
pv[idx++].iov_base = &route_len;
pv[idx ].iov_len = m->route.len;
pv[idx++].iov_base = m->route.name;
pv[idx ].iov_len = sizeof(m->sender_msgid);
pv[idx++].iov_base = &m->sender_msgid;
if(peermode) {
pv[idx ].iov_len = 1;
pv[idx++].iov_base = &sender_len;
pv[idx ].iov_len = m->sender.len;
pv[idx++].iov_base = m->sender.name;
pv[idx ].iov_len = 1;
pv[idx++].iov_base = &nhops;
pv[idx ].iov_len = nhops * sizeof(uint32_t);
pv[idx++].iov_base = m->hops;
}
pv[idx ].iov_len = sizeof(data_len);
pv[idx++].iov_base = &data_len;
pv[idx ].iov_len = m->payload_len;
pv[idx++].iov_base = m->payload;
if(off > 0) {
for(i = 0; i < idx; i++) {
if(off >= pv[i].iov_len) {
off -= pv[i].iov_len;
writev_start++;
}
else {
pv[i].iov_len -= off;
pv[i].iov_base = ((unsigned char *)pv[i].iov_base) + off;
off = 0;
break;
}
}
}
return 0;
}
int
fq_client_write_msg(int fd, uint32_t peermode, fq_msg *m, size_t off, size_t *written) {
struct iovec pv[11]; /* 7 for normal + 4 for peer */
int rv, i, writev_start = 0, idx = 0;
size_t expect;
unsigned char nhops = 0;
unsigned char sender_len = m->sender.len;
unsigned char exchange_len = m->exchange.len;
unsigned char route_len = m->route.len;
uint32_t data_len = htonl(m->payload_len);
expect = 1 + m->exchange.len + 1 + m->route.len +
sizeof(m->sender_msgid) +
sizeof(data_len) + m->payload_len;
if(peermode) {
for(i=0;i<MAX_HOPS;i++) {
if(m->hops[i] == 0) break;
nhops++;
}
expect += 1 + m->sender.len + 1 + (nhops * sizeof(uint32_t));
}
fq_assert(off < expect);
expect -= off;
pv[idx ].iov_len = 1;
pv[idx++].iov_base = &exchange_len;
pv[idx ].iov_len = m->exchange.len;
pv[idx++].iov_base = m->exchange.name;
pv[idx ].iov_len = 1;
pv[idx++].iov_base = &route_len;
pv[idx ].iov_len = m->route.len;
pv[idx++].iov_base = m->route.name;
pv[idx ].iov_len = sizeof(m->sender_msgid);
pv[idx++].iov_base = &m->sender_msgid;
if(peermode) {
pv[idx ].iov_len = 1;
pv[idx++].iov_base = &sender_len;
pv[idx ].iov_len = m->sender.len;
pv[idx++].iov_base = m->sender.name;
pv[idx ].iov_len = 1;
pv[idx++].iov_base = &nhops;
pv[idx ].iov_len = nhops * sizeof(uint32_t);
pv[idx++].iov_base = m->hops;
}
pv[idx ].iov_len = sizeof(data_len);
pv[idx++].iov_base = &data_len;
pv[idx ].iov_len = m->payload_len;
pv[idx++].iov_base = m->payload;
if(off > 0) {
for(i=0;i<idx;i++) {
if(off >= pv[i].iov_len) {
off -= pv[i].iov_len;
writev_start++;
}
else {
pv[i].iov_len -= off;
pv[i].iov_base = ((unsigned char *)pv[i].iov_base) + off;
off = 0;
break;
}
}
}
rv = writev(fd, pv+writev_start, idx-writev_start);
fq_debug(FQ_DEBUG_IO, "writev(%d bytes [%d data]) -> %d\n",
(int)expect, (int)m->payload_len, rv);
if(rv > 0 && written) *written = rv;
if(rv != (int)expect) {
return rv;
}
if(rv == 0) return -1;
return 0;
}
int
fq_find_in_hops(uint32_t needle, fq_msg *m) {
int i;
for(i=0; i<MAX_HOPS; i++) {
if(m->hops[i] == needle) return i;
}
return -1;
}
void
fq_keepalive_fd(int fd, int cnt, int idle, int invtl) {
int optval = 1;
socklen_t optlen = sizeof(optval);
if(setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &optval, optlen) < 0) {
fq_debug(FQ_DEBUG_CONN, "client(%d) keepalive failed: %s\n", fd, strerror(errno));
}
#if defined(SOL_TCP)
#if defined(TCP_KEEPCNT)
optval = cnt;
if(setsockopt(fd, SOL_TCP, TCP_KEEPCNT, &optval, optlen) < 0) {
fq_debug(FQ_DEBUG_CONN, "client(%d) keepcnt failed : %s\n", fd, strerror(errno));
}
#endif
#if defined(TCP_KEEPIDLE)
optval = idle;
if(setsockopt(fd, SOL_TCP, TCP_KEEPIDLE, &optval, optlen) < 0) {
fq_debug(FQ_DEBUG_CONN, "client(%d) keepidle failed : %s\n", fd, strerror(errno));
}
#endif
#if defined(TCP_KEEPINTVL)
optval = intvl;
if(setsockopt(fd, SOL_TCP, TCP_KEEPINTVL, &optval, optlen) < 0) {
fq_debug(FQ_DEBUG_CONN, "client(%d) keepidle failed : %s\n", fd, strerror(errno));
}
#endif
#endif
}
================================================
FILE: fqc.c
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <signal.h>
#include <string.h>
#include "fq.h"
#define SEND_COUNT 1000
int send_count = SEND_COUNT;
void logger(fq_client, const char *);
void logger(fq_client c, const char *s) {
(void)c;
fprintf(stderr, "fq_logger: %s\n", s);
}
static void
print_rate(fq_client c, hrtime_t s, hrtime_t f, uint64_t cnt, uint64_t icnt) {
double d;
if(cnt) {
d = (double)cnt * 1000000000;
d /= (double)(f-s);
printf("[%d backlog] output %0.2f msg/sec\n",
fq_client_data_backlog(c), d);
}
if(icnt) {
d = (double)icnt * 1000000000;
d /= (double)(f-s);
printf("[%d backlog] input %0.2f msg/sec\n",
fq_client_data_backlog(c), d);
}
}
int main(int argc, char **argv) {
hrtime_t s0, s, f, f0;
uint64_t cnt = 0, icnt = 0;
int psize = 0, i = 0, rcvd = 0;
fq_client c;
fq_bind_req breq;
fq_msg *m;
signal(SIGPIPE, SIG_IGN);
fq_client_init(&c, 0, logger);
if(argc < 5) {
fprintf(stderr, "%s <host> <port> <user> <pass> [size [count]]\n",
argv[0]);
exit(-1);
}
fq_client_creds(c, argv[1], atoi(argv[2]), argv[3], argv[4]);
fq_client_heartbeat(c, 1000);
fq_client_set_backlog(c, 10000, 100);
fq_client_connect(c);
memset(&breq, 0, sizeof(breq));
memcpy(breq.exchange.name, "maryland", 8);
breq.exchange.len = 8;
breq.flags = 0;
breq.program = (char *)"prefix:\"test.prefix.\"";
fq_client_bind(c, &breq);
while(ck_pr_load_32(&breq.out__route_id) == 0) usleep(100);
printf("route set -> %u\n", breq.out__route_id);
if(breq.out__route_id == FQ_BIND_ILLEGAL) {
fprintf(stderr, "Failure to bind...\n");
exit(-1);
}
if(argc > 5) {
psize = atoi(argv[5]);
if(psize <= 0 || psize > 100000000) {
fprintf(stderr, "invalid size must be > 0 and < 100000000\n");
exit(-1);
}
}
printf("payload size -> %d\n", psize);
if(argc > 6) {
send_count = atoi(argv[6]);
if(send_count <= 0 || send_count > 10000000) {
fprintf(stderr, "invalid send count must be > 0 and < 10000000\n");
exit(-1);
}
}
printf("message count -> %d\n", send_count);
s0 = s = fq_gethrtime();
while(i < send_count || fq_client_data_backlog(c) > 0) {
if(i < send_count) {
m = fq_msg_alloc_BLANK(psize);
memset(m->payload, 0, psize);
fq_msg_exchange(m, "maryland", 8);
fq_msg_route(m, "test.prefix.foo", 15);
fq_msg_id(m, NULL);
fq_client_publish(c, m);
cnt++;
i++;
fq_msg_free(m);
}
else usleep(100);
f = fq_gethrtime();
while(NULL != (m = fq_client_receive(c))) {
icnt++;
rcvd++;
fq_msg_deref(m);
}
if(f-s > 1000000000) {
print_rate(c, s, f, cnt, icnt);
icnt = 0;
cnt = 0;
s = f;
}
}
f0 = fq_gethrtime();
print_rate(c, s0, f0, i, 0);
do {
icnt=0;
while(NULL != (m = fq_client_receive(c))) {
icnt++;
rcvd++;
fq_msg_deref(m);
}
} while(rcvd < send_count);
f0 = fq_gethrtime();
print_rate(c, s0, f0, 0, rcvd);
printf("Total received during test: %d\n", rcvd);
(void) argc;
return 0;
}
================================================
FILE: fqd.c
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <unistd.h>
#include <errno.h>
#include <sys/types.h>
#include <fcntl.h>
#include <pthread.h>
#include <netdb.h>
#include <sys/socket.h>
#include <arpa/inet.h>
#include <dlfcn.h>
#include "getopt.h"
#include "fqd.h"
#include "fqd_private.h"
#ifndef NO_BCD
#include <bcd.h>
static void bcd_signal_handler(int s, siginfo_t *si, void *unused) {
(void)si;
(void)unused;
bcd_fatal("This is a fatal crash");
signal(s, SIG_DFL);
return;
}
static void
bcd_setup_sigaction(void)
{
struct sigaction sa;
int signals[] = {
SIGSEGV,
SIGFPE,
SIGABRT,
SIGBUS,
SIGILL,
SIGFPE
};
sa.sa_sigaction = bcd_signal_handler;
sa.sa_flags = SA_SIGINFO | SA_ONSTACK;
for (size_t i = 0; i < sizeof(signals) / sizeof(*signals); i++) {
if (sigaction(signals[i], &sa, NULL) == -1) {
fprintf(stderr, "warning: failed to set signal "
"handler %d\n", signals[i]);
}
}
return;
}
#else
typedef void *bcd_t;
#endif
static uint32_t nodeid = 0;
static unsigned short port = 8765;
static int foreground = 0;
static bool usebcd = false;
static int worker_threads = 1;
static char *config_path = NULL;
static char *queue_path = NULL;
static char *libexecdir = NULL;
static bcd_t global_bcd = { 0 };
#define die(str) do { \
fprintf(stderr, "%s: %s\n", str, strerror(errno)); \
exit(-1); \
} while(0)
void fqd_bcd_attach(void) {
if(!usebcd) return;
#ifndef NO_BCD
bcd_error_t error;
if (bcd_attach(&global_bcd, &error) == -1) {
fprintf(stderr, "error: %s\n",
bcd_error_message(&error));
exit(-1);
}
#endif
}
static void *listener_thread(void *unused) {
(void)unused;
fqd_start_worker_threads(worker_threads);
fprintf(stderr, "Listening on port: %d\n", port);
fqd_listener(NULL, port);
fqd_stop_worker_threads();
return NULL;
}
static void usage(const char *prog) {
printf("%s:\n", prog);
printf("\t-h\t\tthis help message\n");
printf("\t-D\t\trun in the foreground\n");
printf("\t-B\t\tenable BCD backtrace reporting\n");
printf("\t-t <count>\tnumber of worker threads to use (default 1)\n");
printf("\t-n <ip>\t\tnode self identifier (IPv4)\n");
printf("\t-p <port>\tspecify listening port (default: 8765)\n");
printf("\t-c <file>\tlocation of the configdb\n");
printf("\t-q <dir>\twhere persistent queues are stored\n");
printf("\t-w <dir>\twhere files for web services are available\n");
printf("\t-v <flags>\tprint additional debugging information, by overriding FQ_DEBUG (cf. fq.h)\n");
printf("\t-l <dir>\tuse this dir for relative module loads\n");
printf("\t-m <module>\tmodule to load\n");
}
static void parse_cli(int argc, char **argv) {
int c;
char *debug = NULL;
if(getenv("FQ_DEBUG")) {
debug = strdup(getenv("FQ_DEBUG"));
}
libexecdir = strdup(LIBEXECDIR);
while((c = getopt(argc, argv, "Bbl:m:hDt:n:p:q:c:w:v:")) != EOF) {
switch(c) {
case 'B':
usebcd = true;
break;
case 'b':
usebcd = false;
break;
case 'l':
free(libexecdir);
libexecdir = strdup(optarg);
break;
case 'm':
fqd_route_load_module(libexecdir, optarg, ".so");
break;
case 'q':
free(queue_path);
queue_path = strdup(optarg);
break;
case 'w':
fqd_http_set_root(optarg);
break;
case 'c':
free(config_path);
config_path = strdup(optarg);
break;
case 'D':
foreground = 1;
break;
case 't':
worker_threads = atoi(optarg);
break;
case 'h':
usage(argv[0]);
exit(0);
case 'n':
if(inet_pton(AF_INET, optarg, &nodeid) != 1) {
fprintf(stderr, "Bad argument to -n, must be an IPv4 address.\n");
exit(-1);
}
if(nodeid == 0 || nodeid == htonl(0x7f000001)) {
fprintf(stderr, "nodeid cannot be INADDR_ANY or loopback\n");
exit(-1);
}
break;
case 'p':
port = atoi(optarg);
break;
case 'v':
free(debug);
debug = strdup(optarg);
break;
default:
usage(argv[0]);
exit(-1);
}
}
if(debug) fq_debug_set_string(debug);
free(debug);
}
static uint32_t get_my_ip(void) {
uint32_t ip;
struct hostent *h;
char buff[128];
gethostname(buff, sizeof(buff));
h = gethostbyname(buff);
if(h && h->h_addrtype == AF_INET && h->h_length == 4) {
memcpy(&ip, h->h_addr_list[0], h->h_length);
if(ip == htonl(0x7f000001)) return 0;
return ip;
}
return 0;
}
int main(int argc, char **argv) {
nodeid = get_my_ip();
parse_cli(argc,argv);
global_functions_init(libexecdir);
if(nodeid == 0) {
fprintf(stderr, "Could not determine host address, use -n <ip>\n");
exit(-1);
}
signal(SIGPIPE, SIG_IGN);
if(!foreground) {
int pid, fd;
/* Handle stdin/stdout/stderr */
fd = open("/dev/null", O_RDONLY);
if(fd < 0 || dup2(fd, STDIN_FILENO) < 0) die("Failed to setup stdin");
close(fd);
fd = open("/dev/null", O_WRONLY);
if(fd < 0 || dup2(fd, STDOUT_FILENO) < 0 || dup2(fd, STDERR_FILENO) < 0)
die("Failed to setup std{out,err}");
close(fd);
/* daemonize */
pid = fork();
if(pid < 0) die("Failed to fork");
if(pid > 0) exit(0);
setsid();
pid = fork();
if(pid < 0) die("Failed to fork");
if(pid > 0) exit(0);
/* run */
}
#ifndef NO_BCD
bcd_error_t error;
if(usebcd) {
struct bcd_config config;
/* Initialize BCD configuration. See bcd.h for options */
if (bcd_config_init(&config, &error) == -1)
goto fatal;
/* Initialize the library. */
if (bcd_init(&config, &error) == -1)
goto fatal;
/* Initialize a handle to BCD. This should be called by every thread interacting with BCD. */
if (bcd_attach(&global_bcd, &error) == -1)
goto fatal;
if (bcd_kv(&global_bcd, "application", "fqd", &error) == -1)
goto fatal;
if (bcd_kv(&global_bcd, "version", FQ_VERSION, &error) == -1)
goto fatal;
bcd_setup_sigaction();
}
#endif
fqd_config_init(nodeid, config_path, queue_path);
listener_thread(NULL);
fprintf(stderr, "Listener thread could not start. Exiting.\n");
exit(0);
return 0;
#ifndef NO_BCD
fatal:
fprintf(stderr, "error: %s\n",
bcd_error_message(&error));
exit(-1);
#endif
}
================================================
FILE: fqd.h.in
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#ifndef FQD_H
#define FQD_H
#ifndef _REENTRANT
#error "You must compile with -D_REENTRANT"
#endif
#include <stdint.h>
#include <stdbool.h>
#include <stdlib.h>
#include <sys/time.h>
#include <netinet/in.h>
#define FQ_VERSION_MAJOR @@FQ_MAJOR@@
#define FQ_VERSION_MINOR @@FQ_MINOR@@
#define FQ_VERSION_PATCH @@FQ_MICRO@@
#define FQ_VERSION "@@FQ_MAJOR@@.@@FQ_MINOR@@.@@FQ_MICRO@@"
#include "fq.h"
#ifndef VARLIBFQDIR
#define VARLIBFQDIR "/var/lib/fq"
#endif
#define MAX_MESSAGE_SIZE (1 << 28) /* 256MB */
typedef void * fqd_queue_impl_data;
typedef struct fqd_queue_impl {
const char *name;
fqd_queue_impl_data (*setup)(fq_rk *, uint32_t *count);
void (*enqueue)(fqd_queue_impl_data, fq_msg *);
fq_msg *(*dequeue)(fqd_queue_impl_data);
void (*dispose)(fq_rk *, fqd_queue_impl_data);
int (*add_checkpoint)(fqd_queue_impl_data, const char *name, const fq_msgid *id);
int (*remove_checkpoint)(fqd_queue_impl_data, const char *name);
int (*reset_checkpoint)(fqd_queue_impl_data, const char *name);
} fqd_queue_impl;
/* implememted in fqd_queue_mem.c */
extern fqd_queue_impl fqd_queue_mem_impl;
/* implememted in fqd_queue_jlog.c */
extern fqd_queue_impl fqd_queue_jlog_impl;
typedef struct fqd_queue fqd_queue;
typedef struct fqd_route_rules fqd_route_rules;
typedef struct fqd_route_rule fqd_route_rule;
typedef struct fqd_config fqd_config;
typedef struct fqd_exchange_stats {
uint64_t n_messages;
uint64_t n_bytes;
uint64_t n_routed;
uint64_t n_no_route;
uint64_t n_dropped;
uint64_t n_size_dropped;
uint64_t n_no_exchange;
uint64_t n_loops;
} fqd_exchange_stats_t;
typedef struct fqd_exchange {
fq_rk exchange;
fqd_exchange_stats_t *stats;
fqd_route_rules *set;
} fqd_exchange;
extern int fqd_queue_write_json(int fd, fqd_queue *q);
extern int fqd_queue_sprint(char *buf, int len, fqd_queue *q);
extern void fqd_queue_ref(fqd_queue *);
extern bool fqd_queue_deref(fqd_queue *);
extern int fqd_queue_cmp(const fqd_queue *, const fqd_queue *);
extern int fqd_config_make_perm_queue(fqd_queue *q);
extern int fqd_config_make_trans_queue(fqd_queue *q);
extern int fqd_config_make_perm_binding(fq_rk *exchange, fqd_queue *q,
int peermode, const char *program);
extern int fqd_config_make_trans_binding(fq_rk *exchange, fqd_queue *q,
int peermode, const char *program);
#define CLIENT_SHARED \
uint32_t refcnt; \
int fd; \
struct timeval connect_time; \
struct sockaddr_in remote; \
hrtime_t last_activity; \
hrtime_t last_heartbeat; \
char pretty[80];
typedef struct {
CLIENT_SHARED
} remote_anon_client;
typedef struct {
CLIENT_SHARED
uint32_t mode;
uint32_t peer_id;
uint32_t no_exchange;
uint32_t no_route;
uint32_t routed;
uint32_t dropped;
uint32_t size_dropped;
uint32_t msgs_in;
uint32_t msgs_out;
uint32_t octets_in;
uint32_t octets_out;
} remote_data_client;
typedef struct remote_client {
CLIENT_SHARED
fq_rk user;
fq_rk key;
fqd_queue *queue;
remote_data_client *data;
unsigned short heartbeat_ms;
} remote_client;
/* You can read around in this... but can't modify it */
extern void fqd_config_init(uint32_t, const char *config_path,
const char *queue_path);
extern int fqd_config_construct_queue_path(char *, size_t, fq_rk *);
extern uint32_t fqd_config_get_nodeid(void);
extern fqd_config *fqd_config_get(void);
extern void fqd_config_release(fqd_config *);
extern int fqd_config_register_client(remote_client *, uint64_t *gen);
extern int fqd_config_deregister_client(remote_client *, uint64_t *gen);
extern fqd_queue *fqd_config_register_queue(fqd_queue *, uint64_t *gen);
extern int fqd_config_deregister_queue(fqd_queue *, uint64_t *gen);
extern fqd_queue *fqd_config_get_registered_queue(fqd_config *, fq_rk *);
extern remote_client *fqd_config_get_registered_client(fqd_config *, fq_rk *key);
extern fqd_exchange *fqd_config_get_exchange(fqd_config *c, fq_rk *exchange);
extern void fqd_size_dropped(uint64_t);
extern void fqd_exchange_messages(fqd_exchange *, uint64_t);
extern void fqd_exchange_message_octets(fqd_exchange *, uint64_t);
extern void fqd_exchange_no_route(fqd_exchange *, uint64_t);
extern void fqd_exchange_routed(fqd_exchange *, uint64_t);
extern void fqd_exchange_dropped(fqd_exchange *, uint64_t);
extern void fqd_exchange_no_exchange(fqd_exchange *, uint64_t);
extern uint32_t fqd_config_bind(fq_rk *exchange, uint16_t flags,
const char *program,
fqd_queue *q, uint64_t *gen);
extern int fqd_config_unbind(fq_rk *exchange, uint32_t route_id,
fqd_queue *q, uint64_t *gen);
extern void fqd_config_wait(uint64_t gen, int us);
extern void fqd_config_http_stats(remote_client *client);
extern void fqd_command_and_control_server(remote_client *);
extern void fqd_data_subscription_server(remote_data_client *);
extern int fqd_listener(const char *ip, unsigned short port);
extern void fqd_remote_client_ref(remote_client *);
extern bool fqd_remote_client_deref(remote_client *);
extern fq_rk *fqd_queue_name(fqd_queue *q);
extern fqd_queue *fqd_queue_get(fq_rk *, const char *, const char *,
int, char *);
extern uint32_t fqd_queue_get_backlog_limit(fqd_queue *);
extern void fqd_queue_set_backlog_limit(fqd_queue *, uint32_t);
extern queue_policy_t fqd_queue_get_policy(fqd_queue *);
extern void fqd_queue_set_policy(fqd_queue *, queue_policy_t);
extern void fqd_queue_enqueue(fqd_queue *q, fq_msg *m, int *dropped);
extern fq_msg *fqd_queue_dequeue(fqd_queue *q);
extern int fqd_queue_register_client(fqd_queue *q, remote_client *c);
extern bool fqd_queue_deregister_client(fqd_queue *q, remote_client *c);
extern void fqd_inject_message(remote_data_client *c, fq_msg *m);
extern struct fqd_route_rule *
fqd_routemgr_compile(const char *program, int peermode, fqd_queue *q);
extern void fqd_routemgr_rule_free(fqd_route_rule *rule);
extern fqd_route_rules *fqd_routemgr_ruleset_alloc(void);
extern uint32_t fqd_routemgr_ruleset_add_rule(fqd_route_rules *set,
fqd_route_rule *r, int *isnew);
extern int
fqd_routemgr_drop_rules_by_route_id(fqd_route_rules *set, fqd_queue *q,
uint32_t route_id);
extern int
fqd_routemgr_perm_route_id(fqd_route_rules *set, uint32_t route_id);
extern int
fqd_routemgr_trans_route_id(fqd_route_rules *set, uint32_t route_id);
extern void
fqd_routemgr_drop_rules_by_queue(fqd_route_rules *set, fqd_queue *q);
extern fqd_route_rules *fqd_routemgr_ruleset_copy(fqd_route_rules *set);
extern void fqd_routemgr_ruleset_free(fqd_route_rules *set);
extern int
fqd_add_peer(uint64_t gen,
const char *host, int port,
const char *user, const char *pass,
fq_rk *exchange, const char *prog,
bool perm);
extern int
fqd_remove_peers(uint64_t current_gen);
extern int
fqd_remove_peer(const char *host, int port,
const char *user, const char *pass,
fq_rk *exchange, const char *prog);
#define ERRTOFD(fd, error) do { \
(void)fq_write_uint16(fd, htons(FQ_PROTO_ERROR)); \
(void)fq_write_short_cmd(fd, strlen(error), error); \
} while(0)
/* programming:
*
* PROGRAM: <prefix|exact>:string RULES*
* RULE: (RULE)
* RULE: (RULE && RULE)
* RULE: (RULE || RULE)
* RULE: EXPR
* EXPR: function(args)
* args: arg
* args: arg, args
* arg: "string"
* arg: true|false
* arg: [0-9][0-9]*(?:.[0-9]*)
*
* functions are dynamically loadable with type signature
* strings: s, booleans: b, integers: d
* function: substr_eq(9.3,10,"tailorings",true)
* C symbol: fqd_route_prog__substr_eq__ddsb(int nargs, valnode_t *args);
* fallback symbol: fqd_route_prog_substr_eq__VA(int nargs, valnode_t *args);
*/
#define MAX_VALNODE_ARGS 16
typedef struct exprnode {
bool (*match)(fq_msg *m, int nargs, valnode_t *args);
int nargs;
valnode_t *args;
} exprnode_t;
typedef struct rulenode {
uint32_t refcnt;
char oper;
struct rulenode *left;
struct rulenode *right;
struct exprnode *expr;
} rulenode_t;
/* DTrace helpers */
typedef struct {
int fd;
char *pretty;
} fq_dtrace_remote_anon_client_t;
typedef struct {
int fd;
char *pretty;
char *user;
} fq_dtrace_remote_client_t;
typedef struct {
int fd;
char *pretty;
uint32_t mode;
uint32_t no_exchange;
uint32_t no_route;
uint32_t routed;
uint32_t dropped;
uint32_t size_dropped;
uint32_t msgs_in;
uint32_t msgs_out;
} fq_dtrace_remote_data_client_t;
#define DTRACE_PACK_ANON_CLIENT(dc, c) do { \
(dc)->fd = (int32_t)(c)->fd; \
(dc)->pretty = (c)->pretty; \
} while(0)
#define DTRACE_PACK_CLIENT(dc, c) do { \
(dc)->fd = (int32_t)(c)->fd; \
(dc)->pretty = (c)->pretty; \
(dc)->user = (char *)(c)->user.name; \
} while(0)
#define DTRACE_PACK_DATA_CLIENT(dc, c) do { \
(dc)->fd = (int32_t)(c)->fd; \
(dc)->pretty = (c)->pretty; \
(dc)->mode = (c)->mode; \
(dc)->no_exchange = (c)->no_exchange; \
(dc)->no_route = (c)->no_route; \
(dc)->routed = (c)->routed; \
(dc)->dropped = (c)->dropped; \
(dc)->size_dropped = (c)->size_dropped; \
(dc)->msgs_in = (c)->msgs_in; \
(dc)->msgs_out = (c)->msgs_out; \
} while(0)
typedef struct {
char *name;
int32_t private;
int32_t policy;
char *type;
} fq_dtrace_queue_t;
void fqd_queue_dtrace_pack(fq_dtrace_queue_t *, fqd_queue *);
void fqd_http_loop(remote_client *c, uint32_t bytes_four);
void fqd_http_set_root(const char *newpath);
#define DTRACE_PACK_QUEUE(dq, c) fqd_queue_dtrace_pack(dq, c)
#endif
================================================
FILE: fqd_ccs.c
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include "fqd.h"
#include "fqd_private.h"
#include "fq_dtrace.h"
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <poll.h>
#include <errno.h>
#include <uuid/uuid.h>
#ifdef __MACH__
static int mkkey(void *ptr, int len) {
static FILE *_random;
if(_random == NULL) _random = fopen("/dev/random", "r");
if(_random == NULL) return -1;
unsigned char *ucp = ptr;
for(int i=0; i<len; i++) ucp[i] = fgetc(_random);
return 0;
}
#else
#include <openssl/rand.h>
static int mkkey(void *ptr, int len) {
if(RAND_bytes(ptr, len) != 1) {
#if OPENSSL_VERSION_NUMBER < 0x1010100fL
if(RAND_pseudo_bytes(ptr, len) != 1) {
return -1;
}
#else
return -1;
#endif
}
return 0;
}
#endif
static int
fqd_ccs_auth(remote_client *client) {
uint16_t cmd, method;
fq_rk queue_name;
if(fq_read_uint16(client->fd, &cmd) ||
ntohs(cmd) != FQ_PROTO_AUTH_CMD) {
ERRTOFD(client->fd, "auth command expected");
return -1;
}
if(fq_read_uint16(client->fd, &method)) {
ERRTOFD(client->fd, "auth method read failed");
return -2;
}
method = ntohs(method);
if(method == 0) {
char buf[128];
unsigned char pass[10240];
char queue_detail[1024], *end_of_qd;
char *qtype = NULL, *qparams = NULL;
char *replace_params = NULL;
int len;
len = fq_read_short_cmd(client->fd, sizeof(client->user.name),
client->user.name);
if(len < 0 || len > (int)sizeof(client->user.name)) {
ERRTOFD(client->fd, "user name is too long");
return -3;
}
client->user.len = len & 0xff;
len = fq_read_short_cmd(client->fd, sizeof(queue_detail)-1,
queue_detail);
if(len < 0) return -4;
if(len >= sizeof(queue_detail)) {
ERRTOFD(client->fd, "queue detail is too long");
return -4;
}
queue_detail[len] = '\0';
end_of_qd = memchr(queue_detail, '\0', len);
if(!end_of_qd) {
if(len < 0 || len > (int)sizeof(queue_name.name)) {
ERRTOFD(client->fd, "queue name is too long");
return -4;
}
queue_name.len = len & 0xff;
memcpy(queue_name.name, queue_detail, queue_name.len);
if(queue_name.len < sizeof(queue_name.name))
memset(queue_name.name + queue_name.len, 0,
sizeof(queue_name.name) - queue_name.len);
}
else if(end_of_qd - queue_detail <= 0xff) {
queue_name.len = end_of_qd - queue_detail;
memcpy(queue_name.name, queue_detail, queue_name.len);
if(queue_name.len < sizeof(queue_name.name))
memset(queue_name.name + queue_name.len, 0,
sizeof(queue_name.name) - queue_name.len);
qtype = end_of_qd + 1;
if(*qtype) qparams = strchr(qtype, ':');
else qtype = NULL;
if(qparams) *qparams++ = '\0';
}
else {
ERRTOFD(client->fd, "pass field is too long");
return -4;
}
if(queue_name.len == 0) {
uuid_t autogen;
static const char *DYNAMIC_QUEUE_FORCE_OPTIONS = "transient,private";
int rlen = strlen(DYNAMIC_QUEUE_FORCE_OPTIONS)+1;
if(!qparams || *qparams == '\0') {
replace_params = malloc(rlen);
memcpy(replace_params, DYNAMIC_QUEUE_FORCE_OPTIONS, rlen);
}
else {
rlen += strlen(qparams)+1;
replace_params = malloc(rlen);
snprintf(replace_params, rlen, "%s,%s",
qparams, DYNAMIC_QUEUE_FORCE_OPTIONS);
}
qparams = replace_params;
uuid_generate(autogen);
memcpy(queue_name.name, "auto-", 5);
uuid_unparse_lower(autogen, (void *)(queue_name.name+5));
queue_name.len = 5 + 36; /* 5 + 36 uuid, no trailing \0 */
}
len = fq_read_short_cmd(client->fd, sizeof(pass), pass);
if(len < 0 || len > (int)sizeof(pass)) {
ERRTOFD(client->fd, "queue name is too long");
free(replace_params);
return -4;
}
client->queue = fqd_queue_get(&queue_name, qtype, qparams,
sizeof(buf), buf);
if(client->queue == NULL) {
ERRTOFD(client->fd, buf);
free(replace_params);
return -6;
}
/* do AUTH */
buf[0] = '\0';
inet_ntop(AF_INET, &client->remote.sin_addr, buf, sizeof(buf));
snprintf(client->pretty, sizeof(client->pretty), "%.*s/%.*s@%s:%d",
client->user.len, client->user.name,
queue_name.len, queue_name.name,
buf, ntohs(client->remote.sin_port));
if(FQ_CLIENT_AUTH_ENABLED()) {
fq_dtrace_remote_client_t dclient;
DTRACE_PACK_CLIENT(&dclient, client);
FQ_CLIENT_AUTH(&dclient);
}
free(replace_params);
return 0;
}
ERRTOFD(client->fd, "unsupported auth method");
return -1;
}
static int
fqd_ccs_key_client(remote_client *client) {
int fd = client->fd;
client->key.len = sizeof(client->key.name);
if(mkkey(client->key.name, client->key.len) != 0) {
ERRTOFD(fd, "can't generate random key");
return -1;
}
if(fqd_queue_register_client(client->queue, client)) {
ERRTOFD(fd, "can't add you to queue");
return -1;
}
if(fq_write_uint16(client->fd, FQ_PROTO_AUTH_RESP) ||
fq_write_short_cmd(client->fd,
client->key.len, client->key.name) < 0) {
return -2;
}
#ifdef DEBUG
{
char hex[260];
if(fq_rk_to_hex(hex, sizeof(hex), &client->key) >= 0)
fq_debug(FQ_DEBUG_CONN, "client keyed:\n%s\n", hex);
}
#endif
return 0;
}
static int
fqd_ccs_heartbeat(remote_client *client) {
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "heartbeat -> %s\n", client->pretty);
#endif
return fq_write_uint16(client->fd, FQ_PROTO_HB);
}
static int
fqd_css_status(remote_client *client) {
remote_data_client *data = client->data;
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "status -> %s\n", client->pretty);
#endif
if(fq_write_uint16(client->fd, FQ_PROTO_STATUS) < 0) return -1;
#define write_uintkey(name, v) do { \
if(fq_write_short_cmd(client->fd, strlen(name), name) < 0) return -1; \
if(fq_write_uint32(client->fd, v) < 0) return -1; \
} while(0)
if(client->queue) write_uintkey("dropped_in", client->queue->dropped_to);
if(data) {
write_uintkey("no_exchange", data->no_exchange);
write_uintkey("no_route", data->no_route);
write_uintkey("routed", data->routed);
write_uintkey("dropped", data->dropped);
write_uintkey("size_dropped", data->size_dropped);
write_uintkey("msgs_in", data->msgs_in);
write_uintkey("msgs_out", data->msgs_out);
write_uintkey("octets_in", data->octets_in);
write_uintkey("octets_out", data->octets_out);
}
if(fq_write_uint16(client->fd, 0) < 0) return -1;
return 0;
}
static int
fqd_ccs_loop(remote_client *client) {
int poll_timeout = 10;
while(1) {
int rv;
struct pollfd pfd;
uint16_t cmd;
hrtime_t t;
pfd.fd = client->fd;
pfd.events = POLLIN|POLLHUP;
pfd.revents = 0;
rv = poll(&pfd, 1, poll_timeout);
if(rv < 0) {
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "poll() failed on %s: %s\n", client->pretty,
strerror(errno));
#endif
break;
}
if(rv > 0) poll_timeout = 10;
else poll_timeout *= 2;
if(poll_timeout > 4000) poll_timeout = 4000;
/* we must wake up often enough to emit our heartbeat */
if(client->heartbeat_ms && poll_timeout > client->heartbeat_ms) poll_timeout = client->heartbeat_ms;
t = fq_gethrtime();
unsigned long long hb_ns = ((unsigned long long)client->heartbeat_ms) * 1000000ULL;
long long hb_age_ns = t - client->last_heartbeat;
if(client->heartbeat_ms && hb_age_ns > hb_ns) {
if(fqd_ccs_heartbeat(client)) break;
client->last_heartbeat = t;
}
if(hb_ns && client->last_activity < (t - hb_ns * 3)) {
ERRTOFD(client->fd, "client heartbeat failed");
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "heartbeat [%dms] failed from %s [%lld]\n", client->heartbeat_ms,
client->pretty, hb_age_ns);
#endif
break;
}
if(rv > 0) {
if(fq_read_uint16(client->fd, &cmd) != 0) break;
client->last_activity = fq_gethrtime();
switch(cmd) {
case FQ_PROTO_HB:
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "heartbeat <- %s\n", client->pretty);
#endif
break;
case FQ_PROTO_HBREQ:
{
uint16_t ms;
if(fq_read_uint16(client->fd, &ms) < 0) return -1;
#ifdef DEBUG
fq_debug(FQ_DEBUG_CONN, "setting client(%p) heartbeat to %d\n",
(void *)client, ms);
#endif
client->heartbeat_ms = ms;
break;
}
case FQ_PROTO_STATUSREQ:
if(fqd_css_status(client)) return -1;
break;
case FQ_PROTO_BINDREQ:
{
int len;
uint16_t flags;
uint32_t route_id;
uint64_t cgen;
char program[0xffff];
fq_rk exchange;
if(fq_read_uint16(client->fd, &flags)) return -1;
len = fq_read_short_cmd(client->fd, sizeof(exchange.name),
exchange.name);
if(len < 0 || len > (int)sizeof(exchange.name)) return -3;
exchange.len = len & 0xff;
len = fq_read_short_cmd(client->fd, sizeof(program)-1, program);
if(len < 0 || len > (int)sizeof(program)-1) return -1;
program[len] = '\0';
route_id = fqd_config_bind(&exchange, flags, program,
client->queue, &cgen);
if(route_id != FQ_BIND_ILLEGAL)
fqd_config_wait(cgen, 100);
if(fq_write_uint16(client->fd, FQ_PROTO_BIND) != 0) return -1;
if(fq_write_uint32(client->fd, route_id) != 0) return -1;
break;
}
case FQ_PROTO_UNBINDREQ:
{
uint32_t route_id;
fq_rk exchange;
int success, len;
if(fq_read_uint32(client->fd, &route_id)) return -1;
len = fq_read_short_cmd(client->fd, sizeof(exchange.name),
exchange.name);
if(len < 0 || len > (int)sizeof(exchange.name)) return -1;
exchange.len = len & 0xff;
success = fqd_config_unbind(&exchange, route_id, client->queue, NULL);
if(fq_write_uint16(client->fd, FQ_PROTO_UNBIND) != 0) return -1;
if(fq_write_uint32(client->fd, success ? route_id : FQ_BIND_ILLEGAL))
return -1;
break;
}
default:
return -1;
}
}
}
return -1;
}
extern void
fqd_command_and_control_server(remote_client *client) {
/* auth */
int rv, registered = 0;
uint64_t cgen;
fq_debug(FQ_DEBUG_CONN, "--> ccs thread\n");
if((rv = fqd_ccs_auth(client)) != 0) {
fq_debug(FQ_DEBUG_CONN, "client auth failed: %d\n", rv);
(void)rv;
goto out;
}
if(fqd_config_register_client(client, &cgen)) {
fq_debug(FQ_DEBUG_CONN, "client registration failed\n");
goto out;
}
fq_thread_setname("fqd:ccs:%s", client->pretty);
registered = 1;
fqd_config_wait(cgen, 100);
if(fqd_ccs_key_client(client) != 0) {
fq_debug(FQ_DEBUG_CONN, "client keying failed: %d\n", rv);
goto out;
}
fqd_ccs_heartbeat(client);
fqd_ccs_loop(client);
out:
if(registered) fqd_config_deregister_client(client, NULL);
fq_debug(FQ_DEBUG_CONN, "<-- ccs thread\n");
}
================================================
FILE: fqd_config.c
================================================
/*
* Copyright (c) 2013 OmniTI Computer Consulting, Inc.
* All rights reserved.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to
* deal in the Software without restriction, including without limitation the
* rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
* sell copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
* IN THE SOFTWARE.
*/
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include <errno.h>
#include <ck_pr.h>
#include <sqlite3.h>
#include "fq.h"
#include "fqd.h"
#include "fqd_private.h"
#include "fq_dtrace.h"
#define CONFIG_RING_SIZE 3
#define CONFIG_ROTATE_NS (100*1000*1000) /*100ms*/
#define DEFAULT_CLIENT_CNT 128
const char *fq_version_string = FQ_VERSION;
const char *fqd_config_path = VARLIBFQDIR "/fqd.sqlite";
const char *fqd_queue_path = VARLIBFQDIR "/queues";
/* A ring of three configs
*
* [cycleout] [currentread] [currentwrite]
*
*/
fqd_exchange_stats_t global_counters = { 0, 0, 0, 0, 0, 0 };
struct fqd_config {
uint64_t gen;
int n_clients;
remote_client **clients;
int n_queues;
fqd_queue **queues;
int n_exchanges;
fqd_exchange **exchanges;
};
static sqlite3 *configdb = NULL;
static uint64_t global_gen = 0;
static uint32_t global_nodeid = 0;
uint32_t fqd_config_get_nodeid() { return global_nodeid; }
typedef struct fqd_config_ref {
fqd_config config;
uint32_t readers;
uint32_t dirty;
} fqd_config_ref;
static struct {
fqd_config_ref configs[CONFIG_RING_SIZE];
/* protected by writelock */
pthread_mutex_t writelock;
uint32_t current_config;
/* end writelock protected things */
} global_config;
#define FQGC(i) global_config.configs[i]
static void *config_rotation(void *);
static void setup_config(void);
static void setup_initial_config(void);
void
fqd_config_init(uint32_t nodeid, const char *config_path, const char *qpath) {
int i;
pthread_t t;
pthread_attr_t attr;
global_nodeid = nodeid;
if(config_path) fqd_config_path = config_path;
if(qpath) fqd_queue_path = qpath;
memset(&global_config, 0, sizeof(global_config));
pthread_mutex_init(&global_config.writelock, NULL);
for(i=0;i<CONFIG_RING_SIZE;i++)
global_config.configs[i].config.gen = ++global_gen;
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
pthread_create(&t, &attr, config_rotation, NULL);
setup_config();
}
extern fqd_config *
fqd_config_get() {
int lcc = global_config.current_config;
ck_pr_inc_32(&global_config.configs[lcc].readers);
return (fqd_config *)&global_config.configs[lcc];
}
extern void
fqd_config_release(fqd_config *fake) {
fqd_config_ref *real = (fqd_config_ref *)fake;
ck_pr_dec_32(&real->readers);
}
int
fqd_config_construct_queue_path(char *path, size_t pathlen,
fq_rk *qname) {
int i;
char *qout, qhex[MAX_RK_LEN * 2 + 1];
qout = qhex;
for(i=0; i<qname->len; i++) {
snprintf(qout, 3, "%02x", (int)qname->name[i]);
qout += 2;
}
*qout = '\0';
return snprintf(path, pathlen, "%s/%s", fqd_queue_path, qhex);
}
fqd_queue *
fqd_config_get_registered_queue(fqd_config *c, fq_rk *qname) {
int i;
fqd_queue *q = NULL;
for(i=0;i<c->n_queues;i++) {
if(c->queues[i] && fq_rk_cmp(qname, fqd_queue_name(c->queues[i])) == 0) {
q = c->queues[i];
break;
}
}
fq_debug(FQ_DEBUG_CONFIG, "referencing queue -> (%p)\n", (void *)q);
return q;
}
remote_client *
fqd_config_get_registered_client(fqd_config *c, fq_rk *key) {
int i;
remote_client *client = NULL;
for(i=0;i<c->n_clients;i++) {
if(c->clients[i] && fq_rk_cmp(key, &c->clients[i]->key) == 0) {
client = c->clients[i];
break;
}
}
return client;
}
fqd_exchange *
fqd_config_get_exchange(fqd_config *c, fq_rk *exchange) {
int i;
for(i=0;i<c->n_exchanges;i++)
if(c->exchanges[i] &&
fq_rk_cmp(exchange, &c->exchanges[i]->exchange) == 0)
return c->exchanges[i];
return NULL;
}
/* This is static b/c no one but us should be calling it
* we we need to hold a lock whilst calling it.
*/
static fqd_exchange *
fqd_config_add_exchange(fqd_config *c, fq_rk *exchange) {
int i;
for(i=0;i<c->n_exchanges;i++) {
if(c->exchanges[i] == NULL) break;
if(fq_rk_cmp(exchange, &c->exchanges[i]->exchange) == 0)
return c->exchanges[i];
}
if(i == c->n_exchanges) {
fqd_exchange **nlist;
int ncnt = c->n_exchanges * 2;
if(ncnt == 0) ncnt = 16;
nlist = calloc(ncnt, sizeof(*c->exchanges));
if(c->n_exchanges) {
memcpy(nlist, c->exchanges, c->n_exchanges * sizeof(*c->exchanges));
free(c->exchanges);
}
c->n_exchanges = ncnt;
c->exchanges = nlist;
}
c->exchanges[i] = calloc(1, sizeof(*c->exchanges[i]));
memcpy(&c->exchanges[i]->exchange, exchange, sizeof(*exchange));
c->exchanges[i]->stats = calloc(1, sizeof(*c->exchanges[i]->stats));
c->exchanges[i]->set = fqd_routemgr_ruleset_alloc();
fq_debug(FQ_DEBUG_CONFIG, "Adding new exchange[%.*s] -> %d\n",
exchange->len, exchange->name, i);
return c->exchanges[i];
}
void fqd_config_wait(uint64_t gen, int us) {
while(1) {
int which;
which = ck_pr_load_uint(&global_config.current_config);
if(FQGC(which).config.gen >= gen) return;
if(us>0) usleep(us);
}
}
/* config modification */
#define BEGIN_CONFIG_MODIFY(conf) \
fqd_config_ref *conf ## _ref; \
fqd_config *conf; \
pthread_mutex_lock(&global_config.writelock); \
conf ## _ref = &FQGC((global_config.current_config + 1) % CONFIG_RING_SIZE); \
conf = &conf ## _ref->config
#define MARK_CONFIG(conf) do { conf ## _ref->dirty = 1; } while(0)
#define END_CONFIG_MODIFY() pthread_mutex_unlock(&global_config.writelock)
extern uint32_t
fqd_config_bind(fq_rk *exchange, uint16_t flags, const char *program,
fqd_queue *q, uint64_t *gen) {
uint32_t route_id;
fqd_exchange *x;
fqd_route_rule *rule;
int peermode = ((flags & FQ_BIND_PEER) == FQ_BIND_PEER);
int isnew = 0;
rule = fqd_routemgr_compile(program, peermode, q);
if(!rule) return FQ_BIND_ILLEGAL;
BEGIN_CONFIG_MODIFY(config);
x = fqd_config_get_exchange(config, exchange);
if(!x) x = fqd_config_add_exchange(config, exchange);
route_id = fqd_routemgr_ruleset_add_rule(x->set, rule, &isnew);
if(flags & FQ_BIND_PERM) {
if((flags & FQ_BIND_PERM) == FQ_BIND_PERM) {
fqd_routemgr_perm_route_id(x->set, route_id);
}
else if((flags & FQ_BIND_PERM) == FQ_BIND_TRANS) {
fqd_routemgr_trans_route_id(x->set, route_id);
}
}
fq_debug(FQ_DEBUG_CONFIG,
"rule %u \"%s\" for exchange \"%.*s\" -> Q[%p]\n", route_id,
program, exchange->len, exchange->name, (void *)q);
if(gen) *gen = config->gen;
MARK_CONFIG(config);
END_CONFIG_MODIFY();
/* if these bits are set, we have configdb work to do */
if(flags & FQ_BIND_PERM) {
if((flags & FQ_BIND_PERM) == FQ_BIND_PERM) {
fqd_config_make_perm_binding(exchange, q, peermode, program);
}
else if((flags & FQ_BIND_PERM) == FQ_BIND_TRANS) {
fqd_config_make_trans_binding(exchange, q, peermode, program);
}
}
return route_id;
}
extern int
fqd_config_unbind(fq_rk *exchange, uint32_t route_id,
fqd_queue *c, uint64_t *gen) {
int i, dropped = 0;
BEGIN_CONFIG_MODIFY(config);
for(i=0;i<config->n_exchanges;i++) {
if(config->exchanges[i] != NULL &&
fq_rk_cmp(exchange, &config->exchanges[i]->exchange) == 0) {
dropped = fqd_routemgr_drop_rules_by_route_id(config->exchanges[i]->set,
c, route_id);
if(gen) *gen = config->gen;
break;
}
}
if(dropped) MARK_CONFIG(config);
END_CONFIG_MODIFY();
fq_debug(FQ_DEBUG_CONFIG,
"unbind rule %u %s for exchange \"%.*s\" -> Q[%p]\n", route_id,
dropped ? "successful" : "failed", exchange->len, exchange->name,
(void *)c);
return dropped;
}
extern int
fqd_config_register_client(remote_client *c, uint64_t *gen) {
int i, rv = 0, available_slot = -1;
BEGIN_CONFIG_MODIFY(config);
for(i=0; i<config->n_clients; i++) {
fq_assert(c != config->clients[i]);
if(available_slot == -1 && config->clients[i] == NULL)
available_slot = i;
}
if(available_slot < 0) {
remote_client **f;
f = calloc(sizeof(*f), config->n_clients + 128);
if(f == NULL) goto oom;
if(config->n_clients)
memcpy(f, config->clients, sizeof(*f) * config->n_clients);
available_slot = config->n_clients;
config->n_clients += 128;
free(config->clients);
config->clients = f;
}
config->clients[available_slot] = c;
fq_debug(FQ_DEBUG_CONFIG, "registering client -> (%p:%s)\n", (void *)c, c->pretty);
fqd_remote_client_ref(c);
if(gen) *gen = config->gen;
MARK_CONFIG(config);
rv = 0;
oom:
END_CONFIG_MODIFY();
return rv;
}
extern int
fqd_config_deregister_client(remote_client *c, uint64_t *gen) {
int i;
remote_client *toderef = NULL;
BEGIN_CONFIG_MODIFY(config);
for(i=0; i<config->n_clients; i++) {
if(c == config->clients[i]) {
config->clients[i] = NULL;
toderef = c;
fq_debug(FQ_DEBUG_CONFIG, "deregistering client -> (%p:%s)\n", (void *)c, c->pretty);
break;
}
}
if(i == config->n_clients)
fq_debug(FQ_DEBUG_CONFIG,
"FAILED deregistering client -> (%p:%s)\n", (void *)c, c->pretty);
fq_assert(i != config->n_clients);
MARK_CONFIG(config);
if(gen) *gen = config->gen;
END_CONFIG_MODIFY();
if(toderef) {
/* Do this work without holding the lock */
if(toderef->queue) {
if(fqd_queue_deregister_client(toderef->queue, c)) {
fqd_config_deregister_queue(toderef->queue, NULL);
}
}
toderef->queue = NULL;
fqd_remote_client_deref(toderef);
}
return 0;
}
extern fqd_queue *
fqd_config_register_queue(fqd_queue *c, uint64_t *gen) {
int i, available_slot = -1;
BEGIN_CONFIG_MODIFY(config);
for(i=0; i<config->n_queues; i++) {
if(config->queues[i] && fqd_queue_cmp(c, config->queues[i]) == 0) {
if(gen) *gen = config->gen;
c = config->queues[i];
goto out;
}
if(available_slot == -1 && config->queues[i] == NULL)
available_slot = i;
}
if(available_slot < 0) {
fqd_queue **f;
f = calloc(sizeof(*f), config->n_queues + 128);
if(f == NULL) goto out;
if(config->n_queues)
memcpy(f, config->queues, sizeof(*f) * config->n_queues);
available_slot = config->n_queues;
config->n_queues += 128;
free(config->queues);
config->queues = f;
}
config->queues[available_slot] = c;
fqd_queue_ref(c);
if(gen) *gen = config->gen;
MARK_CONFIG(config);
out:
END_CONFIG_MODIFY();
fq_debug(FQ_DEBUG_CONFIG, "registering queue (%s) -> (%p:%.*s)\n",
(available_slot == -1) ? "old" : "new", (void *)c,
c->name.len, c->name.name);
return c;
}
extern int
fqd_config_deregister_queue(fqd_queue *c, uint64_t *gen) {
int i;
fqd_queue *toderef = NULL;
BEGIN_CONFIG_MODIFY(config);
for(i=0; i<config->n_queues; i++) {
if(config->queues[i] && fqd_queue_cmp(c, config->queues[i]) == 0) {
config->queues[i] = NULL;
toderef = c;
fq_debug(FQ_DEBUG_CONFIG, "deregistering queue -> (%p:%.*s)\n", (void *)c, c->name.len, c->name.name);
break;
}
}
if(toderef) {
for(i=0;i<config->n_exchanges;i++) {
if(config->exchanges[i] != NULL) {
fqd_routemgr_drop_rules_by_queue(config->exchanges[i]->set, toderef);
}
}
MARK_CONFIG(config);
}
else {
fq_debug(FQ_DEBUG_CONFIG, "FAILED deregistering queue -> (%p:%.*s)\n", (void *)c, c->name.len, c->name.name);
}
if(gen) *gen = config->gen;
END_CONFIG_MODIFY();
if(toderef)
fqd_queue_deref(toderef);
return 0;
}
/* This section deals with managing the rings */
static void
fqd_internal_copy_config(fqd_config_ref *src, fqd_config_ref *tgt) {
int i;
/* First clients */
if(tgt->config.clients) {
for(i=0;i<tgt->config.n_clients;i++)
if(tgt->config.clients[i])
fqd_remote_client_deref(tgt->config.clients[i]);
free(tgt->config.clients);
tgt->config.clients = NULL;
}
if(src->config.clients) {
tgt->config.n_clients = src->config.n_clients;
tgt->config.clients =
malloc(sizeof(*tgt->config.clients) * tgt->config.n_clients);
fq_assert(tgt->config.clients);
memcpy(tgt->config.clients, src->config.clients,
sizeof(*tgt->config.clients) * tgt->config.n_clients);
for(i=0;i<tgt->config.n_clients;i++)
if(tgt->config.clients[i])
fqd_remote_client_ref(tgt->config.clients[i]);
}
/* Now the same thing of queues */
if(tgt->config.queues) {
for(i=0;i<tgt->config.n_queues;i++)
if(tgt->config.queues[i])
fqd_queue_deref(tgt->config.queues[i]);
free(tgt->config.queues);
tgt->config.queues = NULL;
}
if(src->config.queues) {
tgt->config.n_queues = src->config.n_queues;
tgt->config.queues =
malloc(sizeof(*tgt->config.queues) * tgt->config.n_queues);
fq_assert(tgt->config.queues);
memcpy(tgt->config.queues, src->config.queues,
sizeof(*tgt->config.queues) * tgt->config.n_queues);
for(i=0;i<tgt->config.n_queues;i++)
if(tgt->config.queues[i])
fqd_queue_ref(tgt->config.queues[i]);
}
/* next the exchang/routemaps */
if(tgt->config.exchanges) {
for(i=0;i<tgt->config.n_exchanges;i++) {
if(tgt->config.exchanges[i] && tgt->config.exchanges[i]->set) {
fqd_routemgr_ruleset_free(tgt->config.exchanges[i]->set);
free(tgt->config.exchanges[i]);
}
}
free(tgt->config.exchanges);
tgt->config.exchanges = NULL;
}
if(src->config.exchanges) {
tgt->config.n_exchanges = src->config.n_exchanges;
tgt->config.exchanges =
malloc(sizeof(*tgt->config.exchanges) * tgt->config.n_exchanges);
fq_assert(tgt->config.exchanges);
for(i=0;i<tgt->config.n_exchanges;i++) {
if(src->config.exchanges[i]) {
tgt->config.exchanges[i] = malloc(sizeof(*tgt->config.exchanges[i]));
memcpy(tgt->config.exchanges[i], src->config.exchanges[i],
sizeof(*tgt->config.exchanges[i]));
tgt->config.exchanges[i]->set =
fqd_routemgr_ruleset_copy(src->config.exchanges[i]->set);
}
else tgt->config.exchanges[i] = NULL;
}
}
}
static void
fixup_config_write_context(void) {
uint32_t current, next, nextnext;
current = global_config.current_config;
next = (current + 1) % CONFIG_RING_SIZE;
nextnext = (current + 2) % CONFIG_RING_SIZE;
FQ_CONFIG_ROTATE(FQGC(next).dirty);
if(!FQGC(next).dirty) return;
fq_debug(FQ_DEBUG_CONFIG, "Swapping to next running config\n");
pthread_mutex_lock(&global_config.writelock);
/* We've locked writing... let the world use the new config */
global_config.current_config = next;
/* Wait until the next(next) has no readers so we can copy into it */
while(ck_pr_load_uint(&FQGC(nextnext).readers) != 0)
ck_pr_stall();
/* Safe to do the copy */
fqd_internal_copy_config(&FQGC(next), &FQGC(nextnext));
/* Mark that new write target as clean */
FQGC(nextnext).config.gen = ++global_gen;
FQGC(nextnext).dirty = 0;
pthread_mutex_unlock(&global_config.writelock);
fq_debug(FQ_DEBUG_CONFIG, "Swapped to next running config\n");
}
static void *config_rotation(void *unused) {
fq_thread_setname("fqd:config");
fqd_bcd_attach();
while(1) {
fixup_config_write_context();
usleep(CONFIG_ROTATE_NS / 1000);
}
(void)unused;
return NULL;
}
#define cprintf(client, fmt, ...) do { \
char scratch[1024]; \
int len; \
len = snprintf(scratch, sizeof(scratch), fmt, __VA_ARGS__); \
while(write(client->fd, scratch, len) == -1 && errno == EINTR); \
} while(0)
#define cwrite(client, str) write(client->fd, str, strlen(str))
int fqd_config_http_routes(struct fqd_route_rule *r, int rv, void *closure) {
remote_client *client = closure;
char *program_encoded, *cp, *tcp;
int len;
len = strlen(r->program)*2+1;
program_encoded = malloc(len);
for(cp = r->program, tcp = program_encoded; *cp; cp++) {
switch(*cp) {
case '\n': *tcp++ = '\\'; *tcp++ = 'n'; break;
case '\r': *tcp++ = '\\'; *tcp++ = 'r'; break;
case '\t': *tcp++ = '\\'; *tcp++ = 't'; break;
case '\\':
case '\"':
*tcp++ = '\\'; *tcp++ = *cp; break;
default: *tcp++ = *cp; break;
}
}
*tcp = '\0';
cprintf(client, " %s\"%u\": {\n", rv ? "," : " ", r->route_id);
cprintf(client, " \"route_id\": %u,\n", r->route_id);
cprintf(client, " \"prefix\": \"%.*s\",\n", r->prefix.len, r->prefix.name);
cprintf(client, " \"queue\": \"%.*s\",\n", r->queue->name.len, r->queue->name.name);
cprintf(client, " \"permanent\": %s,\n", r->permanent ? "true" : "false");
cprintf(client, " \"invocations\": %llu,\n", (unsigned long long)r->stats->invocations);
cprintf(client, " \"avg_ns\": %u,\n", r->stats->avg_ns);
cprintf(client, " \"program\": \"%s\"\n", program_encoded);
cwrite(client, " }\n");
free(program_encoded);
return 1;
}
void fqd_config_http_stats(remote_client *client) {
int i;
const char *headers = "HTTP/1.0 200 OK\r\nConnection: close\r\nContent-Type: application/json\r\n\r\n";
fqd_config *config;
while(write(client->fd, headers, strlen(headers)) == -1 && errno == EINTR);
config = fqd_config_get();
cwrite(client, "{\n");
cprintf(client, " \"version\": \"%s\",\n", fq_version_string);
cwrite(client, " \"exchanges\": {\n");
for(i=0;i<config->n_exchanges;i++) {
if(config->exchanges[i]) {
fqd_exchange *e = config->exchanges[i];
cprintf(client, " \"%.*s\": {\n", e->exchange.len, e->exchange.name);
cprintf(client, " \"messages\": %llu,\n", (long long unsigned int) e->stats->n_messages);
cprintf(client, " \"octets\": %llu,\n", (long long unsigned int) e->stats->n_bytes);
cprintf(client, " \"no_route\": %llu,\n", (long long unsigned int) e->stats->n_no_route);
cprintf(client, " \"routed\": %llu,\n", (long long unsigned int) e->stats->n_routed);
cprintf(client, " \"dropped\": %llu,\n", (long long unsigned int) e->stats->n_dropped);
cwrite(client, " \"routes\": {\n");
for_each_route_rule_do(e->set, fqd_config_http_routes, client);
cwrite(client, " }\n");
cwrite(client, " },\n");
}
}
cwrite(client, " \"_aggregate\": {\n");
cprintf(client, " \"no_exchange\": %llu,\n", (long long unsigned int) global_counters.n_no_exchange);
cprintf(client, " \"messages\": %llu,\n", (long long unsigned int) global_counters.n_messages);
cprintf(client, " \"octets\": %llu,\n", (long long unsigned int) global_counters.n_bytes);
cprintf(client, " \"no_route\": %llu,\n", (long long unsigned int) global_counters.n_no_route);
cprintf(client, " \"routed\": %llu,\n", (long long unsigned int) global_counters.n_routed);
cprintf(client, " \"dropped\": %llu,\n", (long long unsigned int) global_counters.n_dropped);
cprintf(client, " \"size_dropped\": %llu\n", (long long unsigned int) global_counters.n_size_dropped);
cwrite(client, " }\n");
cwrite(client, " },\n");
cwrite(client, " \"queues\": {\n");
int seen = 0;
for(i=0;i<config->n_queues;i++) {
if(config->queues[i]) {
fqd_queue *q = config->queues[i];
fq_rk *qname = fqd_queue_name(q);
if(seen++) cwrite(client, ",\n");
cprintf(client, " \"%.*s\": \n", qname->len, qname->name);
fqd_queue_write_json(client->fd, q);
}
}
cwrite(client, " }\n");
cwrite(client, "}\n");
fqd_config_release(config);
}
void fqd_size_dropped(uint64_t n) {
ck_pr_add_64(&global_counters.n_size_dropped, n);
}
void fqd_exchange_messages(fqd_exchange *e, uint64_t n) {
if(e) ck_pr_add_64(&e->stats->n_messages, n);
ck_pr_add_64(&global_counters.n_messages, n);
}
void fqd_exchange_message_octets(fqd_exchange *e, uint64_t n) {
if(e) ck_pr_add_64(&e->stats->n_bytes, n);
ck_pr_add_64(&global_counters.n_bytes, n);
}
void fqd_exchange_no_route(fqd_exchange *e, uint64_t n) {
if(e) ck_pr_add_64(&e->stats->n_no_route, n);
ck_pr_add_64(&global_counters.n_no_route, n);
}
void fqd_exchange_routed(fqd_exchange *e, uint64_t n) {
fq_assert(e);
ck_pr_add_64(&e->stats->n_routed, n);
ck_pr_add_64(&global_counters.n_routed, n);
}
void fqd_exchange_dropped(fqd_exchange *e, uint64_t n) {
if(e) ck_pr_add_64(&e->stats->n_dropped, n);
ck_pr_add_64(&global_counters.n_dropped, n);
}
void fqd_exchange_no_exchange(fqd_exchange *e, uint64_t n) {
fq_assert(!e);
ck_pr_add_64(&global_counters.n_no_exchange, n);
}
#define bail(...) do {fprintf(stderr, __VA_ARGS__); exit(-2);} while(0)
static void setup_initial_config() {
char *SQL, *errmsg = NULL;
int rv;
int flags = SQLITE_OPEN_CREATE|SQLITE_OPEN_READWRITE|SQLITE_OPEN_EXCLUSIVE;
if((rv = sqlite3_open_v2(fqd_config_path, &configdb, flags, NULL)) != 0)
bail("... failed to open %s: %s\n", fqd_config_path,
sqlite3_errmsg(configdb));
sqlite3_exec(configdb, "PRAGMA foreign_keys = ON", 0, 0, &errmsg);
if(errmsg) bail("sqlite error: %s\n", sqlite3_errmsg(configdb));
SQL = sqlite3_mprintf(
"CREATE TABLE queue (name TEXT NOT NULL PRIMARY KEY,"
" type TEXT NOT NULL DEFAULT \"mem\", attributes TEXT)"
);
sqlite3_exec(configdb, SQL, 0, 0, &errmsg);
sqlite3_free(SQL);
if(errmsg && strcmp(errmsg, "table queue already exists"))
bail("sqlite error: %s\n", sqlite3_errmsg(configdb));
if(errmsg) sqlite3_free(errmsg);
SQL = sqlite3_mprintf(
"CREATE TABLE binding ( "
" exchange TEXT NOT NULL, "
" queue TEXT NOT NULL, "
" peermode BOOLEAN NOT NULL DEFAULT FALSE, program TEXT, "
" UNIQUE(exchange, queue, peermode, program), "
" FOREIGN KEY(queue) REFERENCES queue(name) "
")"
);
sqlite3_exec(configdb, SQL, 0, 0, &errmsg);
sqlite3_free(SQL);
if(errmsg && strcmp(errmsg, "table binding already exists"))
bail("sqlite error: %s\n", sqlite3_errmsg(configdb));
if(errmsg) sqlite3_free(errmsg);
SQL = sqlite3_mprintf(
"CREATE TABLE upstream ( "
" host TEXT NOT NULL, "
" port INTEGER NOT NULL DEFAULT 8765, "
" source TEXT NOT NULL, "
" password TEXT NOT NULL, "
" exchange TEXT NOT NULL, "
" program TEXT NOT NULL DEFAULT '', "
" permanent_binding BOOLEAN NOT NULL DEFAULT FALSE, "
" UNIQUE(host, port, source, password, exchange, program, permanent_binding) "
")"
);
sqlite3_exec(configdb, SQL, 0, 0, &errmsg);
sqlite3_free(SQL);
if(errmsg && strcmp(errmsg, "table upstream already exists"))
bail("sqlite error: %s\n", sqlite3_errmsg(configdb));
if(errmsg) sqlite3_free(errmsg);
}
int fqd_config_make_perm_queue(fqd_queue *q) {
sqlite3_stmt *stmt;
fq_rk *qname;
const char *insertSQL;
char qtype[1024], *attrs;
fqd_queue_sprint(qtype, sizeof(qtype), q);
attrs = strchr(qtype, ':');
if(attrs == NULL) return -1;
*attrs++ = '\0';
insertSQL = "INSERT INTO queue VALUES(?,?,?)";
qname = fqd_queue_name(q);
sqlite3_prepare_v2(configdb, insertSQL, strlen(insertSQL), &stmt, NULL);
sqlite3_bind_text(stmt, 1, (char *)qname->name, qname->len, NULL);
sqlite3_bind_text(stmt, 2, qtype, strlen(qtype), NULL);
sqlite3_bind_text(stmt, 3, attrs, strlen(attrs), NULL);
switch(sqlite3_step(stmt)) {
case SQLITE_DONE:
if(sqlite3_changes(configdb) > 0) {
fq_debug(FQ_DEBUG_CONFIG, "Queue %.*s made permanent\n",
qname->len, qname->name);
fqd_queue_ref(q);
}
break;
default:
fq_debug(FQ_DEBUG_CONFIG, "Queue %.*s not made permanent: %s\n",
qname->len, qname->name, sqlite3_errmsg(configdb));
break;
}
sqlite3_finalize(stmt);
return 0;
}
int fqd_config_make_trans_queue(fqd_queue *q) {
sqlite3_stmt *stmt;
fq_rk *qname;
const char *insertSQL;
char qtype[1024], *attrs;
fqd_queue_sprint(qtype, sizeof(qtype), q);
attrs = strchr(qtype, ':');
if(attrs == NULL) return -1;
*attrs++ = '\0';
insertSQL = "DELETE FROM queue WHERE name = ?";
qname = fqd_queue_name(q);
sqlite3_prepare_v2(configdb, insertSQL, strlen(insertSQL), &stmt, NULL);
sqlite3_bind_text(stmt, 1, (char *)qname->name, qname->len, NULL);
switch(sqlite3_step(stmt)) {
case SQLITE_DONE:
if(sqlite3_changes(configdb) > 0) {
fq_debug(FQ_DEBUG_CONFIG, "Queue %.*s made transient\n",
qname->len, qname->name);
fqd_queue_deref(q);
break;
}
fq_debug(FQ_DEBUG_CONFIG, "Queue %.*s not made transient: not found\n",
qname->len, qname->name);
break;
default:
fq_debug(FQ_DEBUG_CONFIG, "Queue %.*s not made transient: %s\n",
qname->len, qname->name, sqlite3_errmsg(configdb));
break;
}
sqlite3_finalize(stmt);
return 0;
}
static int sql_make_queues(void *c, int n, char **row, char **col) {
fqd_queue *queue;
char err[1024];
fq_rk q;
fq_assert(n == 3);
(void)c;
(void)col;
q.len = strlen(row[0]);
if(q.len != strlen(row[0])) return 0;
memcpy(q.name, row[0], q.len);
queue = fqd_queue_get(&q, row[1], row[2], sizeof(err), err);
if(!queue) {
fprintf(stderr, "queue(%s) -> %s\n", row[0], err);
return 0;
}
fqd_queue_ref(queue);
return 0;
}
static uint64_t peer_generation = 0;
static int sql_make_peers(void *c, int n, char **row, char **col) {
fq_rk exchange;
char *host = row[0], *source = row[2], *pass = row[3], *prog = row[5];
int port = atoi(row[1]);
bool perm = !strcmp(row[6],"true");
exchange.len = strlen(row[4]);
if(exchange.len != strlen(row[4])) return 0;
memcpy(exchange.name, row[4], exchange.len);
fqd_add_peer(peer_generation, host, port, source, pass, &exchange, prog, perm);
return 0;
}
int fqd_config_make_perm_binding(fq_rk *exchange, fqd_queue *q,
int peermode, const char *program) {
sqlite3_stmt *stmt;
fq_rk *qname;
const char *insertSQL;
const char *pmstr = peermode ? "true" : "false";
char qtype[1024], *attrs;
fqd_queue_sprint(qtype, sizeof(qtype), q);
attrs = strchr(qtype, ':');
if(attrs == NULL) return -1;
*attrs++ = '\0';
insertSQL = "INSERT INTO binding (exchange,queue,peermode,program) "
"VALUES(?,?,?,?)";
qname = fqd_queue_name(q);
sqlite3_prepare_v2(configdb, insertSQL, strlen(insertSQL), &stmt, NULL);
sqlite3_bind_text(stmt, 1, (char *)exchange->name, exchange->len, NULL);
sqlite3_bind_text(stmt, 2, (char *)qname->name, qname->len, NULL);
sqlite3_bind_text(stmt, 3, pmstr, strlen(pmstr), NULL);
sqlite3_bind_text(stmt, 4, program, strlen(program), NULL);
switch(sqlite3_step(stmt)) {
case SQLITE_DONE:
if(sqlite3_changes(configdb) > 0) {
fq_debug(FQ_DEBUG_CONFIG, "Binding %.*s made permanent\n",
qname->len, qname->name);
fqd_queue_ref(q);
}
break;
default:
fq_debug(FQ_DEBUG_CONFIG, "Binding %.*s not made permanent: %s\n",
qname->len, qname->name, sqlite3_errmsg(configdb));
break;
}
sqlite3_finalize(stmt);
return 0;
}
int fqd_config_make_trans_binding(fq_rk *exchange, fqd_queue *q,
int peermode, const char *program) {
sqlite3_stmt *stmt;
fq_rk *qname;
const char *delSQL;
const char *pmstr = peermode ? "true" : "false";
char qtype[1024], *attrs;
fqd_queue_sprint(qtype, sizeof(qtype), q);
attrs = strchr(qtype, ':');
if(attrs != NULL) *attrs++ = '\0';
delSQL = "DELETE FROM binding WHERE exchange=? AND queue=? "
" AND peermode=? AND program=?";
qname = fqd_queue_name(q);
sqlite3_prepare_v2(configdb, delSQL, strlen(delSQL), &stmt, NULL);
sqlite3_bind_text(stmt, 1, (char *)exchange->name, exchange->len, NULL);
sqlite3_bind_text(stmt, 2, (char *)qname->name, qname->len, NULL);
sqlite3_bind_text(stmt, 3, pmstr, strlen(pmstr), NULL);
sqlite3_bind_text(stmt, 4, program, strlen(program), NULL);
switch(sqlite3_step(stmt)) {
case SQLITE_DONE:
if(sqlite3_changes(configdb) > 0) {
fq_debug(FQ_DEBUG_CONFIG, "Binding %.*s made transient\n",
qname->len, qname->name);
fqd_queue_ref(q);
break;
}
fq_debug(FQ_DEBUG_CONFIG, "Binding %.*s not made transient: not found\n",
qname->len, qname->name);
break;
default:
fq_debug(FQ_DEBUG_CONFIG, "Binding %.*s not made transient: %s\n",
qname->len, qname->name, sqlite3_errmsg(configdb));
break;
}
sqlite3_finalize(stmt);
return 0;
}
static int sql_make_bindings(void *c, int n, char **row, char **col) {
int *nbindings = (int *)c;
fqd_queue *queue;
fq_rk q, x;
uint16_t flags;
fq_assert(n == 4);
(void)c;
(void)col;
x.len = strlen(row[0]);
if(x.len != strlen(row[0])) return 0;
memcpy(x.name, row[0], x.len);
q.len = strlen(row[1]);
if(q.len != strlen(row[1])) return 0;
memcpy(q.name, row[1], q.len);
BEGIN_CONFIG_MODIFY(config);
queue = fqd_config_get_registered_queue(config, &q);
MARK_CONFIG(config);
END_CONFIG_MODIFY();
if(queue == NULL) return 1;
flags = !strcmp(row[2],"true") ? FQ_BIND_PEER : 0;
flags |= FQ_BIND_PERM;
fqd_config_bind(&x, flags, row[3], queue, NULL);
(*nbindings)++;
return 0;
}
static void
fqd_refresh_peers(bool fatal) {
char *errmsg = NULL;
sqlite3_exec(configdb,
"SELECT host, port, source, password, exchange, program, permanent_binding FROM upstream",
sql_make_peers, NULL, &errmsg
);
if(errmsg) {
if(fatal) bail("sqlite error: %s\n", sqlite3_errmsg(configdb));
fq_debug(FQ_DEBUG_PEER, "sqlite error: %s\n", sqlite3_errmsg(configdb));
}
}
static void *
fqd_peer_config_maintenance(void *c) {
fq_thread_setname("fqd:peer_config");
fqd_bcd_attach();
while(1) {
peer_generation++;
fqd_refresh_peers(0);
fqd_remove_peers(peer_generation);
sleep(1);
}
return NULL;
}
static void setup_config() {
pthread_t tid;
pthread_attr_t attr;
fqd_config *config;
int i, nexchanges = 0, nqueues = 0, nbindings = 0;
char *errmsg = NULL;
int flags = SQLITE_OPEN_READWRITE|SQLITE_OPEN_EXCLUSIVE;
if(sqlite3_config(SQLITE_CONFIG_SERIALIZED) != SQLITE_OK) {
bail("... failed to set sqlite3 threadsafety\n");
}
fprintf(stderr, "Opening configdb %s\n", fqd_config_path);
if(sqlite3_open_v2(fqd_config_path, &configdb, flags, NULL)) {
flags = SQLITE_OPEN_CREATE|SQLITE_OPEN_READWRITE|SQLITE_OPEN_EXCLUSIVE;
if(sqlite3_open_v2(fqd_config_path, &configdb, flags, NULL))
bail("... failed to open %s: %s\n", fqd_config_path,
sqlite3_errmsg(configdb));
}
setup_initial_config();
sqlite3_exec(configdb,
"SELECT name, type, attributes FROM queue",
sql_make_queues, NULL, &errmsg
);
if(errmsg) bail("sqlite error: %s\n", sqlite3_errmsg(configdb));
sqlite3_exec(configdb,
"SELECT exchange, queue, peermode, program FROM binding",
sql_make_bindings, &nbindings, &errmsg
);
if(errmsg) bail("sqlite error: %s\n", sqlite3_errmsg(configdb));
/* Summarize */
{
BEGIN_CONFIG_MODIFY(tc);
(void)tc;
MARK_CONFIG(tc);
gitextract__9lkm2sj/
├── .gitignore
├── ChangeLog.md
├── LICENSE
├── Makefile
├── README.md
├── coverity_model.c
├── docs/
│ └── fq_protocol.md
├── dtest.d
├── fq.d
├── fq.h
├── fq_bench.c
├── fq_client.c
├── fq_dtrace.blank.h
├── fq_dtrace.d
├── fq_msg.c
├── fq_rcvr.c
├── fq_sndr.c
├── fq_utils.c
├── fqc.c
├── fqd.c
├── fqd.h.in
├── fqd_ccs.c
├── fqd_config.c
├── fqd_dss.c
├── fqd_dyn_sample.c
├── fqd_http.c
├── fqd_listener.c
├── fqd_peer.c
├── fqd_private.h
├── fqd_prog.c
├── fqd_queue.c
├── fqd_queue_jlog.c
├── fqd_queue_mem.c
├── fqd_routemgr.c
├── fqs.c
├── fqtool.c
├── http_parser.c
├── http_parser.h
├── java/
│ ├── Makefile
│ ├── fq_rcvr.java
│ ├── pom.xml
│ └── src/
│ ├── com/
│ │ └── omniti/
│ │ └── labs/
│ │ ├── FqClient.java
│ │ ├── FqClientImplDebug.java
│ │ ├── FqClientImplInterface.java
│ │ ├── FqClientImplNoop.java
│ │ ├── FqCommand.java
│ │ ├── FqCommandProtocolError.java
│ │ ├── FqDataProtocolError.java
│ │ ├── FqHeartbeatException.java
│ │ └── FqMessage.java
│ └── main/
│ └── java/
│ └── com/
│ └── circonus/
│ ├── FqClient.java
│ ├── FqClientImplDebug.java
│ ├── FqClientImplInterface.java
│ ├── FqClientImplNoop.java
│ ├── FqCommand.java
│ ├── FqCommandProtocolError.java
│ ├── FqDataProtocolError.java
│ ├── FqHeartbeatException.java
│ └── FqMessage.java
├── lua/
│ ├── fq-proxy
│ ├── fq-receiver
│ ├── fq-sender
│ ├── fqclient.lua.tail
│ └── generatelua.sh
├── service-configs/
│ ├── 50-circonus-fq.preset
│ ├── circonus-fq.service
│ └── daemon_options
├── test/
│ ├── lua-support/
│ │ └── init.lua
│ ├── run-tests.sh
│ └── test_spec.lua
└── web/
├── css/
│ ├── bootstrap-theme.css
│ ├── bootstrap.css
│ ├── colorbrewer.css
│ └── theme.css
├── index.html
└── js/
├── bootstrap.js
├── colorbrewer.js
└── fq.js
SYMBOL INDEX (743 symbols across 48 files)
FILE: coverity_model.c
type _Bool (line 2) | typedef _Bool bool;
function ck_pr_dec_uint_zero (line 4) | void
function ck_pr_inc_uint (line 10) | void
function ck_pr_load_uint (line 15) | uint32_t
FILE: fq.h
type fq_rk (line 78) | typedef struct fq_rk {
function fq_rk_from_str (line 83) | static inline void
function fq_rk_cmp (line 91) | static inline int
type fq_bind_req (line 100) | typedef struct {
type fq_unbind_req (line 108) | typedef struct {
type fq_msgid (line 115) | typedef struct fq_msgid {
type msg_free_stacks_handle_t (line 127) | typedef struct msg_free_stacks_handle_t msg_free_stacks_handle_t;
type fq_msg (line 130) | typedef struct fq_msg {
type buffered_msg_reader (line 161) | typedef struct buffered_msg_reader buffered_msg_reader;
type fq_conn_s (line 183) | struct fq_conn_s
type fq_hooks (line 189) | typedef struct fq_hooks {
type queue_policy_t (line 296) | typedef enum {
type fq_debug_bits_t (line 301) | typedef enum {
type valnode_t (line 359) | typedef struct valnode {
type hrtime_t (line 375) | typedef uint64_t hrtime_t;
type hrtime_t (line 377) | typedef uint64_t hrtime_t;
type hrtime_t (line 379) | typedef uint64_t hrtime_t;
type fq_dtrace_msg_t (line 386) | typedef struct {
FILE: fq_bench.c
function logger (line 31) | void logger(fq_client c, const char *s) {
function debug_status (line 36) | static void
function print_rate (line 42) | static void
function usage (line 60) | static void usage(const char *prog) {
function parse_cli (line 83) | static void parse_cli(int argc, char **argv) {
function main (line 142) | int main(int argc, char **argv) {
FILE: fq_client.c
function fq_thread_setname (line 50) | void
function fq_client_wfrw_internal (line 77) | static inline int
type fq_conn_s (line 91) | struct fq_conn_s {
type fq_conn_s (line 137) | typedef struct fq_conn_s fq_conn_s;
type cmd_instr (line 139) | typedef struct {
function fq_conn_free (line 156) | static void
type hook_req_type_t (line 189) | typedef enum {
type hook_req_t (line 194) | typedef struct {
function fq_client_signal (line 199) | static void
function fq_resolve_endpoint (line 212) | static int
function fq_socket_connect (line 294) | static int
function fq_client_disconnect_internal (line 321) | static void
function fq_client_do_auth (line 346) | static int
function fq_client_data_connect_internal (line 404) | static int
function mark_safe_free (line 464) | static void
function enqueue_auth_hook_req (line 472) | static void
function enqueue_cmd_hook_req (line 488) | static void
function fq_client_connect_internal (line 502) | static int
function fq_client_read_complete (line 537) | static void
function fq_data_worker_loop (line 556) | static void
type timeval (line 700) | struct timeval
function fq_client_init (line 928) | int
function fq_client_set_userdata (line 942) | void
function fq_client_hooks (line 954) | int
function fq_client_creds (line 976) | int
function fq_client_status (line 1030) | void
function fq_client_heartbeat (line 1042) | void
function fq_client_bind (line 1052) | void
function fq_client_unbind (line 1062) | void
function fq_client_set_backlog (line 1073) | void
function fq_client_set_nonblock (line 1080) | void
function fq_client_connect (line 1086) | int
function fq_client_destroy (line 1114) | void
function fq_client_data_backlog (line 1123) | int
function fq_client_publish (line 1128) | int
function fq_msg (line 1150) | fq_msg *fq_client_receive(fq_client conn) {
FILE: fq_msg.c
function pull_next_local_msgid (line 51) | static void
function fq_msg (line 82) | static inline fq_msg*
function msg_free (line 91) | static inline void
function fq_msg (line 101) | fq_msg *
function fq_msg (line 116) | fq_msg *
function fq_msg_ref (line 130) | void
function fq_msg_deref (line 137) | void
function fq_msg_exchange (line 154) | void
function fq_msg_route (line 164) | void
function fq_msg_id (line 174) | void
FILE: fq_rcvr.c
function logger (line 37) | void logger(fq_client c, const char *s) {
function print_rate (line 41) | static void
function my_auth_handler (line 59) | static void
function my_bind_handler (line 76) | static void
function main (line 92) | int main(int argc, char **argv) {
FILE: fq_sndr.c
function logger (line 35) | void logger(fq_client c, const char *s) {
function debug_status (line 39) | static void
function print_rate (line 44) | static void
function main (line 61) | int main(int argc, char **argv) {
FILE: fq_utils.c
type free_message_stack (line 46) | typedef struct free_message_stack {
type msg_free_stacks_handle_t (line 66) | struct msg_free_stacks_handle_t {
type handle_free_list (line 71) | struct handle_free_list {
function fq_debug_set_bits (line 84) | void fq_debug_set_bits(uint32_t bits) {
function fq_init_free_message_stack (line 88) | static void
function fq_msg (line 98) | static inline fq_msg *
function fq_push_free_message_stack (line 114) | static inline void
function fq_free_msg_fn (line 140) | static void
function fq_debug_set_string (line 153) | void fq_debug_set_string(const char *s) {
type buffered_msg_reader (line 201) | struct buffered_msg_reader {
type handle_free_list (line 213) | struct handle_free_list
function msg_free_stacks_handle_t (line 216) | static inline msg_free_stacks_handle_t *free_message_handle_acquire(void) {
function msg_free_stack_select (line 231) | static inline int msg_free_stack_select(ssize_t in) {
function buffered_msg_reader (line 244) | buffered_msg_reader *fq_buffered_msg_reader_alloc(int fd, uint32_t peerm...
function fq_buffered_msg_reader_free (line 252) | void fq_buffered_msg_reader_free(buffered_msg_reader *f) {
function parse_message_headers (line 258) | static int
function fq_clear_message_cleanup_stack (line 323) | void
function fq_buffered_msg_read (line 355) | int
function hrtime_t (line 496) | hrtime_t fq_gethrtime() {
function hrtime_t (line 503) | hrtime_t fq_gethrtime() {
function hrtime_t (line 514) | hrtime_t fq_gethrtime() {
function hrtime_t (line 524) | inline hrtime_t fq_gethrtime() {
function fq_rk_to_hex (line 531) | int fq_rk_to_hex(char *buf, int len, fq_rk *k) {
function fq_read_uint16 (line 544) | int
function fq_write_uint16 (line 553) | int
function fq_read_uint32 (line 561) | int
function fq_write_uint32 (line 570) | int
function fq_read_short_cmd (line 578) | int
function fq_read_status (line 597) | int
function fq_write_short_cmd (line 613) | int
function fq_read_long_cmd (line 626) | int
function fq_debug_fl (line 652) | int
function fq_debug_stacktrace (line 676) | void
function fq_serialize (line 699) | int fq_serialize(struct iovec **vecs, int *vec_count, uint32_t peermode,...
function fq_client_write_msg (line 773) | int
function fq_find_in_hops (line 846) | int
function fq_keepalive_fd (line 855) | void
FILE: fqc.c
function logger (line 35) | void logger(fq_client c, const char *s) {
function print_rate (line 39) | static void
function main (line 55) | int main(int argc, char **argv) {
FILE: fqd.c
function bcd_signal_handler (line 42) | static void bcd_signal_handler(int s, siginfo_t *si, void *unused) {
function bcd_setup_sigaction (line 50) | static void
function fqd_bcd_attach (line 94) | void fqd_bcd_attach(void) {
function usage (line 114) | static void usage(const char *prog) {
function parse_cli (line 129) | static void parse_cli(int argc, char **argv) {
function get_my_ip (line 196) | static uint32_t get_my_ip(void) {
function main (line 209) | int main(int argc, char **argv) {
FILE: fqd_ccs.c
function mkkey (line 39) | static int mkkey(void *ptr, int len) {
function mkkey (line 49) | static int mkkey(void *ptr, int len) {
function fqd_ccs_auth (line 63) | static int
function fqd_ccs_key_client (line 181) | static int
function fqd_ccs_heartbeat (line 212) | static int
function fqd_css_status (line 220) | static int
function fqd_ccs_loop (line 247) | static int
function fqd_command_and_control_server (line 357) | extern void
FILE: fqd_config.c
type fqd_config (line 56) | struct fqd_config {
function fqd_config_get_nodeid (line 69) | uint32_t fqd_config_get_nodeid() { return global_nodeid; }
type fqd_config_ref (line 71) | typedef struct fqd_config_ref {
function fqd_config_init (line 92) | void
function fqd_config (line 115) | extern fqd_config *
function fqd_config_release (line 122) | extern void
function fqd_config_construct_queue_path (line 128) | int
function fqd_queue (line 142) | fqd_queue *
function remote_client (line 156) | remote_client *
function fqd_exchange (line 169) | fqd_exchange *
function fqd_exchange (line 182) | static fqd_exchange *
function fqd_config_wait (line 211) | void fqd_config_wait(uint64_t gen, int us) {
function fqd_config_bind (line 229) | extern uint32_t
function fqd_config_unbind (line 270) | extern int
function fqd_config_register_client (line 293) | extern int
function fqd_config_deregister_client (line 324) | extern int
function fqd_queue (line 358) | extern fqd_queue *
function fqd_config_deregister_queue (line 394) | extern int
function fqd_internal_copy_config (line 426) | static void
function fixup_config_write_context (line 498) | static void
function fqd_config_http_routes (line 548) | int fqd_config_http_routes(struct fqd_route_rule *r, int rv, void *closu...
function fqd_config_http_stats (line 581) | void fqd_config_http_stats(remote_client *client) {
function fqd_size_dropped (line 631) | void fqd_size_dropped(uint64_t n) {
function fqd_exchange_messages (line 634) | void fqd_exchange_messages(fqd_exchange *e, uint64_t n) {
function fqd_exchange_message_octets (line 638) | void fqd_exchange_message_octets(fqd_exchange *e, uint64_t n) {
function fqd_exchange_no_route (line 642) | void fqd_exchange_no_route(fqd_exchange *e, uint64_t n) {
function fqd_exchange_routed (line 646) | void fqd_exchange_routed(fqd_exchange *e, uint64_t n) {
function fqd_exchange_dropped (line 651) | void fqd_exchange_dropped(fqd_exchange *e, uint64_t n) {
function fqd_exchange_no_exchange (line 655) | void fqd_exchange_no_exchange(fqd_exchange *e, uint64_t n) {
function setup_initial_config (line 662) | static void setup_initial_config() {
function fqd_config_make_perm_queue (line 717) | int fqd_config_make_perm_queue(fqd_queue *q) {
function fqd_config_make_trans_queue (line 749) | int fqd_config_make_trans_queue(fqd_queue *q) {
function sql_make_queues (line 781) | static int sql_make_queues(void *c, int n, char **row, char **col) {
function sql_make_peers (line 802) | static int sql_make_peers(void *c, int n, char **row, char **col) {
function fqd_config_make_perm_binding (line 815) | int fqd_config_make_perm_binding(fq_rk *exchange, fqd_queue *q,
function fqd_config_make_trans_binding (line 850) | int fqd_config_make_trans_binding(fq_rk *exchange, fqd_queue *q,
function sql_make_bindings (line 888) | static int sql_make_bindings(void *c, int n, char **row, char **col) {
function fqd_refresh_peers (line 918) | static void
function setup_config (line 943) | static void setup_config() {
FILE: fqd_dss.c
function futex (line 48) | static int
type incoming_message (line 71) | struct incoming_message
type incoming_message (line 80) | struct incoming_message
type timespec (line 126) | struct timespec
type timespec (line 131) | struct timespec
type timespec (line 137) | struct timespec
function fqd_start_worker_threads (line 167) | void
function fqd_stop_worker_threads (line 188) | void
function fqd_queue_message_process (line 199) | static void
function fqd_dss_read_complete (line 242) | static void
function fqd_data_driver (line 287) | static void
function fqd_data_subscription_server (line 391) | void
FILE: fqd_dyn_sample.c
function fqd_route_prog__sample__d (line 28) | bool
FILE: fqd_http.c
function fqd_http_set_root (line 51) | void fqd_http_set_root(const char *newpath) {
function ends_with (line 57) | static inline int ends_with(const char *str, const char *end) {
type http_req (line 76) | struct http_req {
type http_req (line 97) | struct http_req
type http_req (line 98) | struct http_req
type http_req (line 99) | struct http_req
type http_req (line 100) | struct http_req
function ht_free (line 106) | static void
type ck_malloc (line 110) | struct ck_malloc
function http_req_clean (line 115) | static void
function store_kv (line 161) | static void
function fqd_http_message_url (line 196) | static int
function fqd_http_message_status (line 221) | static int
function fqd_http_message_body (line 229) | static int
function fqd_http_message_header_field (line 243) | static int
type http_req (line 254) | struct http_req
function fqd_http_message_header_value (line 266) | static int
function fqd_http_message_headers_complete (line 292) | static int
function fqd_http_jsend (line 313) | static void
function fqd_http_message_complete (line 344) | static int
function fqd_http_add_checkpoint (line 429) | static int
function fqd_http_remove_checkpoint (line 515) | static int
function fqd_http_reset_to_checkpoint (line 559) | static int
function fqd_http_submit_msg (line 602) | static int
function fqd_http_loop (line 674) | void
FILE: fqd_listener.c
function fqd_remote_client_ref (line 45) | void
function fqd_remote_client_deref (line 49) | bool
function service_connection (line 63) | static void
type fqd_ccs_work_queue_t (line 163) | typedef struct fqd_ccs_work_queue {
function fqd_ccs_enqueue_work (line 173) | static void
function remote_anon_client (line 201) | remote_anon_client *
function fqd_listener (line 231) | int
FILE: fqd_peer.c
type peer_binding_info (line 46) | typedef struct {
type fqd_peer_connection (line 56) | typedef struct peer_connection {
function peercmp (line 71) | static int
function fqd_peer_auth_hook (line 85) | static void
function fqd_peer_bind_hook (line 111) | static void
function fqd_peer_unbind_hook (line 134) | static void
function fqd_peer_cleanup_hook (line 140) | static void
function fqd_peer_disconnect_hook (line 156) | static void
function fqd_peer_message_hook (line 166) | static bool
function peerlog (line 197) | static void
function fqd_peer_start (line 203) | static void
function fqd_peer_stop (line 213) | static void
function fqd_peer_online_bind (line 218) | static void
function fqd_peer_online_unbind (line 231) | static void
function fqd_add_peer (line 245) | int
function fqd_remove_peers (line 310) | int
function fqd_remove_peer (line 344) | int
FILE: fqd_private.h
type fqd_route_stats (line 31) | struct fqd_route_stats {
type fqd_route_rule (line 37) | struct fqd_route_rule {
type prefix_jumptable (line 50) | struct prefix_jumptable {
type fqd_route_rules (line 61) | struct fqd_route_rules {
type fqd_queue (line 65) | struct fqd_queue {
type fqd_route_rules (line 86) | struct fqd_route_rules
type fqd_route_rule (line 87) | struct fqd_route_rule
function strlcpy (line 102) | static inline size_t strlcpy(char *dst, const char *src, size_t size)
function strlcat (line 113) | static inline size_t strlcat(char *dst, const char *src, size_t size)
FILE: fqd_prog.c
function fqd_route_load_module (line 38) | void fqd_route_load_module(const char *libexecdir, const char *file, con...
function global_functions_init (line 51) | void global_functions_init(const char *libexecdir) {
function fqd_route_prog__true__ (line 75) | bool fqd_route_prog__true__(fq_msg *m, int nargs, valnode_t *args) {
function fqd_route_prog__route_contains__s (line 83) | bool
function fqd_route_prog__payload_prefix__s (line 93) | bool
function fqd_route_prog__payload_contains__s (line 105) | bool
FILE: fqd_queue.c
function fqd_queue_write_json (line 49) | int fqd_queue_write_json(int fd, fqd_queue *q) {
function fqd_queue_sprint (line 91) | int fqd_queue_sprint(char *buf, int len, fqd_queue *q) {
function fqd_queue_dtrace_pack (line 99) | void fqd_queue_dtrace_pack(fq_dtrace_queue_t *d, fqd_queue *s) {
function fq_rk (line 108) | fq_rk *
function fqd_queue_enqueue (line 113) | void
function fq_msg (line 165) | fq_msg *
function fqd_queue_register_client (line 175) | int
function fqd_queue_deregister_client (line 196) | bool
function fqd_queue_cmp (line 216) | int
function fqd_queue_ref (line 221) | void
function fqd_queue_deref (line 227) | bool
function fqd_queue_get_backlog_limit (line 242) | uint32_t
function fqd_queue_set_backlog_limit (line 246) | void
function queue_policy_t (line 250) | queue_policy_t
function fqd_queue_set_policy (line 254) | void
function fqd_queue_free (line 258) | static void
function fqd_queue (line 265) | fqd_queue *
FILE: fqd_queue_jlog.c
type queue_jlog (line 36) | struct queue_jlog {
function queue_jlog_enqueue (line 69) | static void queue_jlog_enqueue(fqd_queue_impl_data f, fq_msg *m) {
function fq_msg (line 78) | static fq_msg *queue_jlog_dequeue(fqd_queue_impl_data f) {
function queue_log_add_checkpoint (line 155) | static int queue_log_add_checkpoint(fqd_queue_impl_data data, const char...
function queue_log_remove_checkpoint (line 212) | static int queue_log_remove_checkpoint(fqd_queue_impl_data data, const c...
function queue_log_reset_to_checkpoint (line 227) | static int queue_log_reset_to_checkpoint(fqd_queue_impl_data data, const...
function write_sig (line 255) | static int write_sig(struct queue_jlog *d) {
function read_sig (line 265) | static int read_sig(struct queue_jlog *d, uuid_t out) {
function fqd_queue_impl_data (line 275) | static fqd_queue_impl_data queue_jlog_setup(fq_rk *qname, uint32_t *coun...
function multi_unlink (line 339) | static int
function queue_jlog_dispose (line 348) | static void queue_jlog_dispose(fq_rk *qname, fqd_queue_impl_data f)
FILE: fqd_queue_mem.c
type queue_mem (line 28) | struct queue_mem {
function queue_mem_enqueue (line 34) | static void queue_mem_enqueue(fqd_queue_impl_data f, fq_msg *m) {
function fq_msg (line 46) | static fq_msg *queue_mem_dequeue(fqd_queue_impl_data f) {
function fqd_queue_impl_data (line 58) | static fqd_queue_impl_data queue_mem_setup(fq_rk *qname, uint32_t *count) {
function queue_mem_dispose (line 67) | static void queue_mem_dispose(fq_rk *qname, fqd_queue_impl_data f) {
function queue_mem_add_checkpoint (line 85) | static int queue_mem_add_checkpoint(fqd_queue_impl_data data, const char...
function queue_mem_remove_checkpoint (line 90) | static int queue_mem_remove_checkpoint(fqd_queue_impl_data data, const c...
function queue_mem_reset_to_checkpoint (line 95) | static int queue_mem_reset_to_checkpoint(fqd_queue_impl_data data, const...
FILE: fqd_routemgr.c
type dl_handles (line 40) | struct dl_handles {
function fqd_routemgr_add_handle (line 45) | void fqd_routemgr_add_handle(void *handle) {
type function_registry_entry (line 52) | struct function_registry_entry {
function gen_free (line 58) | static void gen_free(void *p, size_t b, bool r) { (void)b; (void)r; free...
type ck_malloc (line 59) | struct ck_malloc
function __ck_hash_name (line 60) | static unsigned long __ck_hash_name(const void *v, unsigned long seed) {
function __ck_compare_name (line 68) | static bool __ck_compare_name(const void *a, const void *b) {
function global_function_register (line 74) | void global_function_register(const char *name, void (*f)(void)) {
type function_registry_entry (line 93) | struct function_registry_entry
function apply_compiled_program_node (line 105) | static bool
function apply_compiled_program (line 118) | static bool
type queue_target (line 131) | struct queue_target {
function already_queue_target (line 138) | static bool
function add_queue_target (line 150) | static void
function internal_jt_do (line 167) | static int
function for_each_route_rule_do (line 183) | int
function walk_jump_table (line 188) | static void
function fqd_inject_message (line 229) | void
function is_term_char (line 288) | static inline int is_term_char(char a, const char *ts, int tslen) {
type fqd_route_rule (line 334) | struct fqd_route_rule
type fqd_route_rule (line 339) | struct fqd_route_rule
function fqd_routemgr_rule_free (line 377) | void
type fqd_route_rules (line 393) | struct fqd_route_rules
type fqd_route_rules (line 395) | struct fqd_route_rules
function walk_jump_table_setp_by_route_id (line 397) | static int
function walk_jump_table_drop_rules_by_route_id (line 420) | static int
function walk_jump_table_drop_rules_by_queue (line 451) | static void
function fqd_routemgr_set_permanence_by_route_id (line 475) | static int
function fqd_routemgr_perm_route_id (line 480) | int
function fqd_routemgr_trans_route_id (line 484) | int
function fqd_routemgr_drop_rules_by_route_id (line 488) | int
function fqd_routemgr_drop_rules_by_queue (line 495) | void
type prefix_jumptable (line 501) | struct prefix_jumptable
type prefix_jumptable (line 502) | struct prefix_jumptable
type prefix_jumptable (line 507) | struct prefix_jumptable
function fqd_routemgr_ruleset_add_rule (line 561) | uint32_t
function rulenode_t (line 585) | static rulenode_t *
function fqd_route_rule (line 591) | static fqd_route_rule *
function copy_jt (line 606) | static void
function fqd_route_rules (line 634) | fqd_route_rules *
function free_jt (line 642) | static void
function fqd_routemgr_ruleset_free (line 662) | void
function prog_free (line 668) | static void
function expr_free (line 679) | static void
function is_valid_term_char (line 693) | static int is_valid_term_char(char ch, bool first) {
function rule_getterm (line 701) | static int rule_getterm(const char **cp, char *term, int len) {
function rule_getstring (line 713) | static int rule_getstring(const char **cp, valnode_t *arg) {
function exprnode_t (line 747) | static exprnode_t *
function rulenode_t (line 819) | rulenode_t *
function rulenode_t (line 920) | static rulenode_t *
FILE: fqs.c
function logger (line 17) | void logger(fq_client c, const char *s) {
function usage (line 21) | static void usage(const char *prog) {
function main (line 31) | int main(int argc, char **argv) {
FILE: fqtool.c
function logger (line 41) | void logger(fq_client c, const char *s) {
function my_auth_handler (line 63) | static void
function my_bind_handler (line 86) | static void
function my_message_handler (line 99) | static bool
function usage (line 169) | int usage(const char *prog) {
function main (line 189) | int main(int argc, char **argv) {
FILE: http_parser.c
type state (line 280) | enum state
type header_states (line 364) | enum header_states
type http_host_state (line 394) | enum http_host_state
function parse_url_char (line 479) | static enum state
function http_parser_execute (line 627) | size_t http_parser_execute (http_parser *parser,
function http_message_needs_eof (line 2079) | int
function http_should_keep_alive (line 2102) | int
type http_method (line 2122) | enum http_method
function http_parser_init (line 2128) | void
function http_parser_settings_init (line 2139) | void
type http_errno (line 2146) | enum http_errno
type http_errno (line 2152) | enum http_errno
function http_parse_host_char (line 2157) | static enum http_host_state
function http_parse_host (line 2222) | static int
function http_parser_parse_url (line 2295) | int
function http_parser_pause (line 2394) | void
function http_body_is_final (line 2408) | int
function http_parser_version (line 2413) | unsigned long
FILE: http_parser.h
type __int8 (line 36) | typedef __int8 int8_t;
type __int16 (line 38) | typedef __int16 int16_t;
type __int32 (line 40) | typedef __int32 int32_t;
type __int64 (line 42) | typedef __int64 int64_t;
type http_parser (line 66) | typedef struct http_parser http_parser;
type http_parser_settings (line 67) | typedef struct http_parser_settings http_parser_settings;
type http_method (line 123) | enum http_method
type http_parser_type (line 131) | enum http_parser_type { HTTP_REQUEST, HTTP_RESPONSE, HTTP_BOTH }
type flags (line 135) | enum flags
type http_errno (line 194) | enum http_errno {
type http_parser (line 204) | struct http_parser {
type http_parser_settings (line 234) | struct http_parser_settings {
type http_parser_url_fields (line 246) | enum http_parser_url_fields
type http_parser_url (line 265) | struct http_parser_url {
type http_parser_type (line 288) | enum http_parser_type
type http_method (line 313) | enum http_method
type http_errno (line 316) | enum http_errno
type http_errno (line 319) | enum http_errno
type http_parser_url (line 324) | struct http_parser_url
FILE: java/fq_rcvr.java
class fq_rcvr (line 8) | public class fq_rcvr {
class FqTest (line 9) | private static class FqTest extends FqClientImplDebug {
method setClient (line 14) | public void setClient(FqClient c) { client = c; }
method dispatch (line 15) | public void dispatch(FqMessage m) {
method dispatchAuth (line 19) | public void dispatchAuth(FqCommand.Auth a) {
method showRate (line 28) | public void showRate() {
method main (line 37) | public static void main(String args[]) {
FILE: java/src/com/omniti/labs/FqClient.java
class FqClient (line 48) | public class FqClient {
method initialize (line 96) | private void initialize(FqClientImplInterface _impl, int _mode, int bs...
method FqClient (line 110) | public FqClient(FqClientImplInterface _impl)
method FqClient (line 114) | public FqClient(FqClientImplInterface _impl, int _mode)
method FqClient (line 118) | public FqClient(FqClientImplInterface _impl, int _mode, int _bsize)
method isPeermode (line 122) | public boolean isPeermode() { return (mode == FQ_PROTO_PEER_MODE); }
method getImpl (line 123) | public FqClientImplInterface getImpl() { return impl; }
method setHeartbeat (line 124) | public void setHeartbeat(short ms) {
method setHeartbeat (line 130) | public void setHeartbeat(int ms) {
method set_backlog (line 133) | public void set_backlog(int len, int stall) {
method send (line 139) | public void send(FqCommand cmd) {
method send (line 142) | public void send(FqMessage m) {
method creds (line 145) | public void creds(int _port, String _source, String _pass)
method creds (line 149) | public void creds(String _host, String _source, String _pass)
method creds (line 153) | public void creds(String _host, int _port,
method client_do_auth (line 178) | private boolean client_do_auth() throws IOException, FqCommandProtocol...
method cmd_read_short_bytearray (line 186) | public byte[] cmd_read_short_bytearray() throws IOException {
method cmd_read_short_string (line 200) | public String cmd_read_short_string() throws IOException {
method cmd_read (line 205) | public ByteBuffer cmd_read(int len) throws IOException {
method data_write (line 216) | public long data_write(ByteBuffer bb) throws IOException {
method data_write (line 219) | public long data_write(ByteBuffer[] bb) throws IOException {
method cmd_write (line 222) | public int cmd_write(ByteBuffer bb) throws IOException {
method client_data_connect_internal (line 225) | private boolean client_data_connect_internal() {
method reset (line 247) | private void reset() {
method client_connect_internal (line 254) | private boolean client_connect_internal() {
method connect (line 277) | public void connect() {
method recvHeartbeat (line 295) | public void recvHeartbeat() {
method sendHeartbeat (line 298) | private void sendHeartbeat() throws IOException, FqHeartbeatException {
method data_backlog (line 314) | public int data_backlog() { return qlen.get(); }
method worker (line 315) | private void worker() {
method data_worker_sender (line 349) | private void data_worker_sender() {
method waitForData (line 372) | private void waitForData(long ms) throws IOException {
method blockingRead (line 381) | public int blockingRead(byte dst[], int offset, int len) throws IOExce...
method fill_data_buffer (line 400) | public ByteBuffer fill_data_buffer(boolean force) throws IOException {
method data_worker_receiver (line 409) | private void data_worker_receiver() {
method data_worker (line 427) | private void data_worker() {
method back_worker (line 468) | private void back_worker() {
method shutdown (line 477) | public void shutdown() {
FILE: java/src/com/omniti/labs/FqClientImplDebug.java
class FqClientImplDebug (line 29) | public class FqClientImplDebug implements FqClientImplInterface {
method setClient (line 31) | public void setClient(FqClient c) throws InUseException {
method genericError (line 35) | protected void genericError(Throwable e) {
method connectError (line 38) | public void connectError(Throwable e) { genericError(e); }
method commandError (line 39) | public void commandError(Throwable e) { genericError(e); }
method dataError (line 40) | public void dataError(Throwable e) { genericError(e); }
method dispatch (line 41) | public void dispatch(FqMessage m) {
method dispatch (line 47) | public void dispatch(FqCommand cmd) {
method dispatchAuth (line 50) | public void dispatchAuth(FqCommand.Auth cmd) {
method dispatchHeartbeatRequest (line 53) | public void dispatchHeartbeatRequest(FqCommand.HeartbeatRequest cmd) {
method dispatchHeartbeat (line 56) | public void dispatchHeartbeat(FqCommand.Heartbeat cmd) { dispatch(cmd); }
method dispatchBindRequest (line 57) | public void dispatchBindRequest(FqCommand.BindRequest cmd) {
method dispatchUnbindRequest (line 60) | public void dispatchUnbindRequest(FqCommand.UnbindRequest cmd) {
method dispatchStatusRequest (line 63) | public void dispatchStatusRequest(FqCommand.StatusRequest cmd) {
FILE: java/src/com/omniti/labs/FqClientImplInterface.java
type FqClientImplInterface (line 28) | public interface FqClientImplInterface {
class InUseException (line 29) | public class InUseException extends Exception { }
method setClient (line 30) | public void setClient(FqClient c) throws InUseException;
method connectError (line 31) | public void connectError(Throwable e);
method commandError (line 32) | public void commandError(Throwable e);
method dataError (line 33) | public void dataError(Throwable e);
method dispatch (line 34) | public void dispatch(FqMessage m);
method dispatch (line 35) | public void dispatch(FqCommand cmd);
method dispatchAuth (line 36) | public void dispatchAuth(FqCommand.Auth cmd);
method dispatchHeartbeat (line 37) | public void dispatchHeartbeat(FqCommand.Heartbeat cmd);
method dispatchHeartbeatRequest (line 38) | public void dispatchHeartbeatRequest(FqCommand.HeartbeatRequest cmd);
method dispatchBindRequest (line 39) | public void dispatchBindRequest(FqCommand.BindRequest cmd);
method dispatchUnbindRequest (line 40) | public void dispatchUnbindRequest(FqCommand.UnbindRequest cmd);
method dispatchStatusRequest (line 41) | public void dispatchStatusRequest(FqCommand.StatusRequest cmd);
FILE: java/src/com/omniti/labs/FqClientImplNoop.java
class FqClientImplNoop (line 26) | public class FqClientImplNoop implements FqClientImplInterface {
method setClient (line 28) | public void setClient(FqClient c) throws InUseException {
method genericError (line 32) | protected void genericError(Throwable e) { }
method connectError (line 33) | public void connectError(Throwable e) { genericError(e); }
method commandError (line 34) | public void commandError(Throwable e) { genericError(e); }
method dataError (line 35) | public void dataError(Throwable e) { genericError(e); }
method dispatch (line 36) | public void dispatch(FqMessage m) { }
method dispatch (line 37) | public void dispatch(FqCommand cmd) { }
method dispatchAuth (line 38) | public void dispatchAuth(FqCommand.Auth cmd) { dispatch(cmd); }
method dispatchHeartbeatRequest (line 39) | public void dispatchHeartbeatRequest(FqCommand.HeartbeatRequest cmd) {
method dispatchHeartbeat (line 42) | public void dispatchHeartbeat(FqCommand.Heartbeat cmd) { dispatch(cmd); }
method dispatchBindRequest (line 43) | public void dispatchBindRequest(FqCommand.BindRequest cmd) { dispatch(...
method dispatchUnbindRequest (line 44) | public void dispatchUnbindRequest(FqCommand.UnbindRequest cmd) { dispa...
method dispatchStatusRequest (line 45) | public void dispatchStatusRequest(FqCommand.StatusRequest cmd) { dispa...
FILE: java/src/com/omniti/labs/FqCommand.java
class FqCommand (line 36) | public abstract class FqCommand {
method cmd (line 52) | public abstract short cmd();
method response_cmd (line 53) | public short response_cmd() { return cmd(); }
method compose (line 54) | public void compose() { }
method send (line 55) | public void send(FqClient c) throws IOException {
method hasInBandResponse (line 66) | public abstract boolean hasInBandResponse();
method getShortCmd (line 69) | public Short getShortCmd(FqClient c)
method process (line 84) | public void process(FqClient c) throws IOException, FqCommandProtocolE...
method alloc (line 95) | protected void alloc(int size) {
method FqCommand (line 99) | public FqCommand(int size) {
method FqCommand (line 102) | public FqCommand() {
class Heartbeat (line 105) | public static class Heartbeat extends FqCommand {
method Heartbeat (line 106) | public Heartbeat() { super(0); }
method hasInBandResponse (line 107) | public boolean hasInBandResponse() { return false; }
method cmd (line 108) | public short cmd() { return FQ_PROTO_HB; }
class HeartbeatRequest (line 110) | public static class HeartbeatRequest extends FqCommand {
method HeartbeatRequest (line 112) | public HeartbeatRequest(int _ms) {
method hasInBandResponse (line 116) | public boolean hasInBandResponse() { return false; }
method cmd (line 117) | public short cmd() { return FQ_PROTO_HBREQ; }
method compose (line 118) | public void compose() { bb.putShort(ms); }
class Auth (line 120) | public static abstract class Auth extends FqCommand {
method success (line 122) | public boolean success() { return (key != null); }
method getKey (line 123) | public byte[] getKey() { return key; }
class PlainAuth (line 125) | public static class PlainAuth extends Auth {
method PlainAuth (line 131) | public PlainAuth(String user, String pass,
method cmd (line 144) | public short cmd() { return FQ_PROTO_AUTH_CMD; }
method hasInBandResponse (line 145) | public boolean hasInBandResponse() { return true; }
method compose (line 146) | public void compose() {
method process (line 157) | public void process(FqClient c) throws IOException, FqCommandProtoco...
class BindRequest (line 179) | public static class BindRequest extends FqCommand {
method BindRequest (line 188) | public BindRequest(byte _exchange[], String _program, short _flags) {
method BindRequest (line 198) | public BindRequest(byte _exchange[], String _program, boolean _peerm...
method BindRequest (line 201) | public BindRequest(String exchange, String p, boolean m) {
method cmd (line 204) | public short cmd() { return FQ_PROTO_BINDREQ; }
method response_cmd (line 205) | public short response_cmd() { return FQ_PROTO_BIND; }
method hasInBandResponse (line 206) | public boolean hasInBandResponse() { return true; }
method compose (line 207) | public void compose() {
method process (line 214) | public void process(FqClient c) throws IOException, FqCommandProtoco...
method getBinding (line 223) | public Integer getBinding() { return binding; }
method getExchange (line 224) | public byte[] getExchange() { return exchange; }
class UnbindRequest (line 226) | public static class UnbindRequest extends FqCommand {
method UnbindRequest (line 230) | public UnbindRequest(BindRequest b) {
method cmd (line 238) | public short cmd() { return FQ_PROTO_UNBINDREQ; }
method response_cmd (line 239) | public short response_cmd() { return FQ_PROTO_UNBIND; }
method hasInBandResponse (line 240) | public boolean hasInBandResponse() { return true; }
method compose (line 241) | public void compose() {
method process (line 246) | public void process(FqClient c) throws IOException, FqCommandProtoco...
method getBinding (line 255) | public Integer getBinding() { return bind.getBinding(); }
method getSuccess (line 256) | public boolean getSuccess() {
class StatusRequest (line 261) | public static class StatusRequest extends FqCommand {
method StatusRequest (line 264) | public StatusRequest() { super(0); }
method cmd (line 265) | public short cmd() { return FQ_PROTO_STATUSREQ; }
method response_cmd (line 266) | public short response_cmd() { return FQ_PROTO_STATUS; }
method hasInBandResponse (line 267) | public boolean hasInBandResponse() { return true; }
method process (line 268) | public void process(FqClient c) throws IOException, FqCommandProtoco...
method getDate (line 284) | public Date getDate() { return last_update; }
method getMap (line 285) | public Map<String,Long> getMap() { return status; }
FILE: java/src/com/omniti/labs/FqCommandProtocolError.java
class FqCommandProtocolError (line 26) | public class FqCommandProtocolError extends Exception {
method toString (line 30) | public String toString() {
method FqCommandProtocolError (line 35) | public FqCommandProtocolError(short _expected, short _recvd) {
method FqCommandProtocolError (line 39) | public FqCommandProtocolError(short _expected) {
method FqCommandProtocolError (line 43) | public FqCommandProtocolError(String _msg) {
FILE: java/src/com/omniti/labs/FqDataProtocolError.java
class FqDataProtocolError (line 26) | public class FqDataProtocolError extends Exception {
method FqDataProtocolError (line 27) | public FqDataProtocolError(String a) {
FILE: java/src/com/omniti/labs/FqHeartbeatException.java
class FqHeartbeatException (line 26) | public class FqHeartbeatException extends Exception {
method FqHeartbeatException (line 27) | public FqHeartbeatException() { super(); }
FILE: java/src/com/omniti/labs/FqMessage.java
class FqMessage (line 33) | public class FqMessage {
class MsgId (line 34) | public static class MsgId {
method MsgId (line 36) | public MsgId(byte v[]) {
method setRoute (line 58) | public void setRoute(byte[] _r) { route = _r; route_len = _r.length; }
method setSender (line 59) | public void setSender(byte[] _r) { sender = _r; sender_len = _r.length; }
method setExchange (line 60) | public void setExchange(byte[] _r) { exchange = _r; exchange_len = _r....
method setMsgId (line 61) | public void setMsgId() {
method setPayload (line 68) | public void setPayload(byte[] _r) { payload = _r; payload_len = _r.len...
method getRoute (line 70) | public String getRoute() { return new String(route, StandardCharsets.U...
method getExchange (line 71) | public String getExchange() { return new String(exchange, StandardChar...
method getSender (line 72) | public String getSender() { return new String(sender, StandardCharsets...
method getMsgId (line 73) | public MsgId getMsgId() { return sender_msgid; }
method getPayload (line 74) | public byte[] getPayload() { return payload; }
method getPath (line 75) | public InetAddress[] getPath() { return hops; }
method isComplete (line 77) | public boolean isComplete(boolean peermode) {
method isComplete (line 88) | public boolean isComplete() { return _complete; }
method read (line 89) | public boolean read(FqClient c) throws IOException, FqDataProtocolError {
method readInternal (line 120) | private boolean readInternal(FqClient c, ByteBuffer bb)
method send (line 209) | public boolean send(FqClient c) throws IOException, FqDataProtocolError {
FILE: java/src/main/java/com/circonus/FqClient.java
class FqClient (line 49) | public class FqClient {
method initialize (line 97) | private void initialize(FqClientImplInterface _impl, int _mode, int bs...
method FqClient (line 111) | public FqClient(FqClientImplInterface _impl)
method FqClient (line 115) | public FqClient(FqClientImplInterface _impl, int _mode)
method FqClient (line 119) | public FqClient(FqClientImplInterface _impl, int _mode, int _bsize)
method isPeermode (line 123) | public boolean isPeermode() { return (mode == FQ_PROTO_PEER_MODE); }
method getImpl (line 124) | public FqClientImplInterface getImpl() { return impl; }
method setHeartbeat (line 125) | public void setHeartbeat(short ms) {
method setHeartbeat (line 131) | public void setHeartbeat(int ms) {
method set_backlog (line 134) | public void set_backlog(int len, int stall) {
method send (line 140) | public void send(FqCommand cmd) {
method send (line 143) | public void send(FqMessage m) {
method creds (line 146) | public void creds(int _port, String _source, String _pass)
method creds (line 150) | public void creds(String _host, String _source, String _pass)
method creds (line 154) | public void creds(String _host, int _port,
method client_do_auth (line 179) | private boolean client_do_auth() throws IOException, FqCommandProtocol...
method cmd_read_short_bytearray (line 187) | public byte[] cmd_read_short_bytearray() throws IOException {
method cmd_read_short_string (line 201) | public String cmd_read_short_string() throws IOException {
method cmd_read (line 206) | public ByteBuffer cmd_read(int len) throws IOException {
method data_write (line 217) | public long data_write(ByteBuffer bb) throws IOException {
method data_write (line 220) | public long data_write(ByteBuffer[] bb) throws IOException {
method cmd_write (line 223) | public int cmd_write(ByteBuffer bb) throws IOException {
method client_data_connect_internal (line 226) | private boolean client_data_connect_internal() {
method reset (line 248) | private void reset() {
method client_connect_internal (line 255) | private boolean client_connect_internal() {
method connect (line 278) | public void connect() {
method recvHeartbeat (line 296) | public void recvHeartbeat() {
method sendHeartbeat (line 299) | private void sendHeartbeat() throws IOException, FqHeartbeatException {
method data_backlog (line 315) | public int data_backlog() { return qlen.get(); }
method worker (line 316) | private void worker() {
method data_worker_sender (line 350) | private void data_worker_sender() {
method waitForData (line 373) | private void waitForData(long ms) throws IOException {
method blockingRead (line 382) | public int blockingRead(byte dst[], int offset, int len) throws IOExce...
method fill_data_buffer (line 401) | public ByteBuffer fill_data_buffer(boolean force) throws IOException {
method data_worker_receiver (line 410) | private void data_worker_receiver() {
method data_worker (line 428) | private void data_worker() {
method back_worker (line 469) | private void back_worker() {
method shutdown (line 478) | public void shutdown() {
FILE: java/src/main/java/com/circonus/FqClientImplDebug.java
class FqClientImplDebug (line 30) | public class FqClientImplDebug implements FqClientImplInterface {
method setClient (line 32) | public void setClient(FqClient c) throws InUseException {
method genericError (line 36) | protected void genericError(Throwable e) {
method connectError (line 39) | public void connectError(Throwable e) { genericError(e); }
method commandError (line 40) | public void commandError(Throwable e) { genericError(e); }
method dataError (line 41) | public void dataError(Throwable e) { genericError(e); }
method dispatch (line 42) | public void dispatch(FqMessage m) {
method dispatch (line 48) | public void dispatch(FqCommand cmd) {
method dispatchAuth (line 51) | public void dispatchAuth(FqCommand.Auth cmd) {
method dispatchHeartbeatRequest (line 54) | public void dispatchHeartbeatRequest(FqCommand.HeartbeatRequest cmd) {
method dispatchHeartbeat (line 57) | public void dispatchHeartbeat(FqCommand.Heartbeat cmd) { dispatch(cmd); }
method dispatchBindRequest (line 58) | public void dispatchBindRequest(FqCommand.BindRequest cmd) {
method dispatchUnbindRequest (line 61) | public void dispatchUnbindRequest(FqCommand.UnbindRequest cmd) {
method dispatchStatusRequest (line 64) | public void dispatchStatusRequest(FqCommand.StatusRequest cmd) {
FILE: java/src/main/java/com/circonus/FqClientImplInterface.java
type FqClientImplInterface (line 29) | public interface FqClientImplInterface {
class InUseException (line 30) | public class InUseException extends Exception { }
method setClient (line 31) | public void setClient(FqClient c) throws InUseException;
method connectError (line 32) | public void connectError(Throwable e);
method commandError (line 33) | public void commandError(Throwable e);
method dataError (line 34) | public void dataError(Throwable e);
method dispatch (line 35) | public void dispatch(FqMessage m);
method dispatch (line 36) | public void dispatch(FqCommand cmd);
method dispatchAuth (line 37) | public void dispatchAuth(FqCommand.Auth cmd);
method dispatchHeartbeat (line 38) | public void dispatchHeartbeat(FqCommand.Heartbeat cmd);
method dispatchHeartbeatRequest (line 39) | public void dispatchHeartbeatRequest(FqCommand.HeartbeatRequest cmd);
method dispatchBindRequest (line 40) | public void dispatchBindRequest(FqCommand.BindRequest cmd);
method dispatchUnbindRequest (line 41) | public void dispatchUnbindRequest(FqCommand.UnbindRequest cmd);
method dispatchStatusRequest (line 42) | public void dispatchStatusRequest(FqCommand.StatusRequest cmd);
FILE: java/src/main/java/com/circonus/FqClientImplNoop.java
class FqClientImplNoop (line 27) | public class FqClientImplNoop implements FqClientImplInterface {
method setClient (line 29) | public void setClient(FqClient c) throws InUseException {
method genericError (line 33) | protected void genericError(Throwable e) { }
method connectError (line 34) | public void connectError(Throwable e) { genericError(e); }
method commandError (line 35) | public void commandError(Throwable e) { genericError(e); }
method dataError (line 36) | public void dataError(Throwable e) { genericError(e); }
method dispatch (line 37) | public void dispatch(FqMessage m) { }
method dispatch (line 38) | public void dispatch(FqCommand cmd) { }
method dispatchAuth (line 39) | public void dispatchAuth(FqCommand.Auth cmd) { dispatch(cmd); }
method dispatchHeartbeatRequest (line 40) | public void dispatchHeartbeatRequest(FqCommand.HeartbeatRequest cmd) {
method dispatchHeartbeat (line 43) | public void dispatchHeartbeat(FqCommand.Heartbeat cmd) { dispatch(cmd); }
method dispatchBindRequest (line 44) | public void dispatchBindRequest(FqCommand.BindRequest cmd) { dispatch(...
method dispatchUnbindRequest (line 45) | public void dispatchUnbindRequest(FqCommand.UnbindRequest cmd) { dispa...
method dispatchStatusRequest (line 46) | public void dispatchStatusRequest(FqCommand.StatusRequest cmd) { dispa...
FILE: java/src/main/java/com/circonus/FqCommand.java
class FqCommand (line 37) | public abstract class FqCommand {
method cmd (line 53) | public abstract short cmd();
method response_cmd (line 54) | public short response_cmd() { return cmd(); }
method compose (line 55) | public void compose() { }
method send (line 56) | public void send(FqClient c) throws IOException {
method hasInBandResponse (line 67) | public abstract boolean hasInBandResponse();
method getShortCmd (line 70) | public Short getShortCmd(FqClient c)
method process (line 85) | public void process(FqClient c) throws IOException, FqCommandProtocolE...
method alloc (line 96) | protected void alloc(int size) {
method FqCommand (line 100) | public FqCommand(int size) {
method FqCommand (line 103) | public FqCommand() {
class Heartbeat (line 106) | public static class Heartbeat extends FqCommand {
method Heartbeat (line 107) | public Heartbeat() { super(0); }
method hasInBandResponse (line 108) | public boolean hasInBandResponse() { return false; }
method cmd (line 109) | public short cmd() { return FQ_PROTO_HB; }
class HeartbeatRequest (line 111) | public static class HeartbeatRequest extends FqCommand {
method HeartbeatRequest (line 113) | public HeartbeatRequest(int _ms) {
method hasInBandResponse (line 117) | public boolean hasInBandResponse() { return false; }
method cmd (line 118) | public short cmd() { return FQ_PROTO_HBREQ; }
method compose (line 119) | public void compose() { bb.putShort(ms); }
class Auth (line 121) | public static abstract class Auth extends FqCommand {
method success (line 123) | public boolean success() { return (key != null); }
method getKey (line 124) | public byte[] getKey() { return key; }
class PlainAuth (line 126) | public static class PlainAuth extends Auth {
method PlainAuth (line 132) | public PlainAuth(String user, String pass,
method cmd (line 145) | public short cmd() { return FQ_PROTO_AUTH_CMD; }
method hasInBandResponse (line 146) | public boolean hasInBandResponse() { return true; }
method compose (line 147) | public void compose() {
method process (line 158) | public void process(FqClient c) throws IOException, FqCommandProtoco...
class BindRequest (line 180) | public static class BindRequest extends FqCommand {
method BindRequest (line 189) | public BindRequest(byte _exchange[], String _program, short _flags) {
method BindRequest (line 199) | public BindRequest(byte _exchange[], String _program, boolean _peerm...
method BindRequest (line 202) | public BindRequest(String exchange, String p, boolean m) {
method cmd (line 205) | public short cmd() { return FQ_PROTO_BINDREQ; }
method response_cmd (line 206) | public short response_cmd() { return FQ_PROTO_BIND; }
method hasInBandResponse (line 207) | public boolean hasInBandResponse() { return true; }
method compose (line 208) | public void compose() {
method process (line 215) | public void process(FqClient c) throws IOException, FqCommandProtoco...
method getBinding (line 224) | public Integer getBinding() { return binding; }
method getExchange (line 225) | public byte[] getExchange() { return exchange; }
class UnbindRequest (line 227) | public static class UnbindRequest extends FqCommand {
method UnbindRequest (line 231) | public UnbindRequest(BindRequest b) {
method cmd (line 239) | public short cmd() { return FQ_PROTO_UNBINDREQ; }
method response_cmd (line 240) | public short response_cmd() { return FQ_PROTO_UNBIND; }
method hasInBandResponse (line 241) | public boolean hasInBandResponse() { return true; }
method compose (line 242) | public void compose() {
method process (line 247) | public void process(FqClient c) throws IOException, FqCommandProtoco...
method getBinding (line 256) | public Integer getBinding() { return bind.getBinding(); }
method getSuccess (line 257) | public boolean getSuccess() {
class StatusRequest (line 262) | public static class StatusRequest extends FqCommand {
method StatusRequest (line 265) | public StatusRequest() { super(0); }
method cmd (line 266) | public short cmd() { return FQ_PROTO_STATUSREQ; }
method response_cmd (line 267) | public short response_cmd() { return FQ_PROTO_STATUS; }
method hasInBandResponse (line 268) | public boolean hasInBandResponse() { return true; }
method process (line 269) | public void process(FqClient c) throws IOException, FqCommandProtoco...
method getDate (line 285) | public Date getDate() { return last_update; }
method getMap (line 286) | public Map<String,Long> getMap() { return status; }
FILE: java/src/main/java/com/circonus/FqCommandProtocolError.java
class FqCommandProtocolError (line 27) | public class FqCommandProtocolError extends Exception {
method toString (line 31) | public String toString() {
method FqCommandProtocolError (line 36) | public FqCommandProtocolError(short _expected, short _recvd) {
method FqCommandProtocolError (line 40) | public FqCommandProtocolError(short _expected) {
method FqCommandProtocolError (line 44) | public FqCommandProtocolError(String _msg) {
FILE: java/src/main/java/com/circonus/FqDataProtocolError.java
class FqDataProtocolError (line 27) | public class FqDataProtocolError extends Exception {
method FqDataProtocolError (line 28) | public FqDataProtocolError(String a) {
FILE: java/src/main/java/com/circonus/FqHeartbeatException.java
class FqHeartbeatException (line 27) | public class FqHeartbeatException extends Exception {
method FqHeartbeatException (line 28) | public FqHeartbeatException() { super(); }
FILE: java/src/main/java/com/circonus/FqMessage.java
class FqMessage (line 34) | public class FqMessage {
class MsgId (line 35) | public static class MsgId {
method MsgId (line 37) | public MsgId(byte v[]) {
method setRoute (line 59) | public void setRoute(byte[] _r) { route = _r; route_len = _r.length; }
method setSender (line 60) | public void setSender(byte[] _r) { sender = _r; sender_len = _r.length; }
method setExchange (line 61) | public void setExchange(byte[] _r) { exchange = _r; exchange_len = _r....
method setMsgId (line 62) | public void setMsgId() {
method setPayload (line 69) | public void setPayload(byte[] _r) { payload = _r; payload_len = _r.len...
method getRoute (line 71) | public String getRoute() { return new String(route, StandardCharsets.U...
method getExchange (line 72) | public String getExchange() { return new String(exchange, StandardChar...
method getSender (line 73) | public String getSender() { return new String(sender, StandardCharsets...
method getMsgId (line 74) | public MsgId getMsgId() { return sender_msgid; }
method getPayload (line 75) | public byte[] getPayload() { return payload; }
method getPath (line 76) | public InetAddress[] getPath() { return hops; }
method isComplete (line 78) | public boolean isComplete(boolean peermode) {
method isComplete (line 89) | public boolean isComplete() { return _complete; }
method read (line 90) | public boolean read(FqClient c) throws IOException, FqDataProtocolError {
method readInternal (line 121) | private boolean readInternal(FqClient c, ByteBuffer bb)
method send (line 210) | public boolean send(FqClient c) throws IOException, FqDataProtocolError {
FILE: web/js/bootstrap.js
function transitionEnd (line 33) | function transitionEnd() {
function removeElement (line 118) | function removeElement() {
function clearMenus (line 764) | function clearMenus() {
function getParent (line 775) | function getParent($this) {
function complete (line 1336) | function complete() {
function ScrollSpy (line 1604) | function ScrollSpy(element, options) {
function next (line 1805) | function next() {
FILE: web/js/fq.js
function pretty_number (line 16) | function pretty_number(a) {
function hexify (line 33) | function hexify(a) {
function $badge (line 41) | function $badge(n) {
function $label (line 44) | function $label(n, type) {
function alphaKeys (line 48) | function alphaKeys(obj) {
function rate_calc (line 56) | function rate_calc(type, name, data, part) {
function rate_queue_calc (line 65) | function rate_queue_calc(type, name, data, i, part) {
function update_exchange (line 75) | function update_exchange(name,detail) {
function update_routes (line 104) | function update_routes(name,detail) {
function clear_exchanges (line 142) | function clear_exchanges() {
function mk_client (line 148) | function mk_client(c) {
function update_queue_row (line 152) | function update_queue_row(name,detail) {
function refresh_stats (line 227) | function refresh_stats() {
Condensed preview — 78 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (862K chars).
[
{
"path": ".gitignore",
"chars": 334,
"preview": "*.a\n*.o\n*.lo\n*.so*\nfqd.h\nfqd\nfqc\nfqs\nfqtool\nfq_bench\nfq_dtrace.h\nfq_rcvr\nfq_sndr\nlua/fqclient.lua\nMakefile.build\nMakefil"
},
{
"path": "ChangeLog.md",
"chars": 3593,
"preview": "# ChangeLog\n\n### v0.13.11\n\n* Fix test issue where libfq could not be found.\n* Minor change to compiler optimization flag"
},
{
"path": "LICENSE",
"chars": 1097,
"preview": "Copyright (c) 2013 OmniTI Computer Consulting, Inc.\nAll rights reserved.\n\nPermission is hereby granted, free of charge, "
},
{
"path": "Makefile",
"chars": 6807,
"preview": "# If you want a verbose make (visible commands) add V=1 to you invocation\n\n.SUFFIXES: .lo\n\nCC=gcc\nLD=gcc\nLN_S=ln -s\nCOPT"
},
{
"path": "README.md",
"chars": 9302,
"preview": "# fq.\n\n<a href=\"https://scan.coverity.com/projects/circonus-labs-fq\">\n <img alt=\"Coverity Scan Build Status\"\n src"
},
{
"path": "coverity_model.c",
"chars": 267,
"preview": "typedef unsigned int uint32_t;\ntypedef _Bool bool;\n\nvoid\nck_pr_dec_uint_zero(uint32_t *v, bool *zero) {\n *v = (*v) - 1;"
},
{
"path": "docs/fq_protocol.md",
"chars": 4985,
"preview": "# fq Protocol\n\n## Client\n\nClients maintain two paired tcp connections to fq. After the connections are made, some prelim"
},
{
"path": "dtest.d",
"chars": 1761,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n * \n * Permission is hereby granted, f"
},
{
"path": "fq.d",
"chars": 3425,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fq.h",
"chars": 11077,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fq_bench.c",
"chars": 5554,
"preview": "/*\n * Copyright (c) 2016 Circonus\n * All rights reserved.\n *\n * Permission is hereby granted, free of charge, to any per"
},
{
"path": "fq_client.c",
"chars": 37009,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fq_dtrace.blank.h",
"chars": 2813,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fq_dtrace.d",
"chars": 3700,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fq_msg.c",
"chars": 4564,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fq_rcvr.c",
"chars": 4550,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fq_sndr.c",
"chars": 3495,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fq_utils.c",
"chars": 25671,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqc.c",
"chars": 4323,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd.c",
"chars": 7599,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd.h.in",
"chars": 10868,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_ccs.c",
"chars": 12484,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_config.c",
"chars": 32587,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_dss.c",
"chars": 13255,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_dyn_sample.c",
"chars": 1462,
"preview": "/*\n * Copyright (c) 2014 Circonus, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, free of charge, to a"
},
{
"path": "fqd_http.c",
"chars": 21389,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_listener.c",
"chars": 8154,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_peer.c",
"chars": 11615,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_private.h",
"chars": 3633,
"preview": "/*\n * Copyright (c) 2013 Circonus, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, free of charge, to a"
},
{
"path": "fqd_prog.c",
"chars": 3899,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_queue.c",
"chars": 12063,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_queue_jlog.c",
"chars": 11615,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_queue_mem.c",
"chars": 3537,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqd_routemgr.c",
"chars": 27764,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "fqs.c",
"chars": 2899,
"preview": "/*\n * fqs\n *\n * fqs reads messages from stdin, one pre line, and sends it to the specified fq exchange.\n *\n */\n\n#define "
},
{
"path": "fqtool.c",
"chars": 8846,
"preview": "/*\n * Copyright (c) 2014 Circonus, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, free of charge, to a"
},
{
"path": "http_parser.c",
"chars": 69822,
"preview": "/* Based on src/http/ngx_http_parse.c from NGINX copyright Igor Sysoev\n *\n * Additional changes are licensed under the s"
},
{
"path": "http_parser.h",
"chars": 12673,
"preview": "/* Copyright Joyent, Inc. and other Node contributors. All rights reserved.\n *\n * Permission is hereby granted, free of "
},
{
"path": "java/Makefile",
"chars": 687,
"preview": "JAVAC=javac\nJAR=jar\n\nJAVAFILES=\\\n\tcom/omniti/labs/FqClient.java \\\n\tcom/omniti/labs/FqClientImplDebug.java \\\n\tcom/omniti/"
},
{
"path": "java/fq_rcvr.java",
"chars": 1617,
"preview": "import com.circonus.FqClient;\nimport com.circonus.FqClientImplDebug;\nimport com.circonus.FqClientImplNoop;\nimport com.ci"
},
{
"path": "java/pom.xml",
"chars": 4097,
"preview": "<project>\n <modelVersion>4.0.0</modelVersion>\n <groupId>com.circonus</groupId>\n <artifactId>fqclient</artifactId>\n <"
},
{
"path": "java/src/com/omniti/labs/FqClient.java",
"chars": 15673,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "java/src/com/omniti/labs/FqClientImplDebug.java",
"chars": 2879,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "java/src/com/omniti/labs/FqClientImplInterface.java",
"chars": 1945,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "java/src/com/omniti/labs/FqClientImplNoop.java",
"chars": 2214,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "java/src/com/omniti/labs/FqCommand.java",
"chars": 10165,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "java/src/com/omniti/labs/FqCommandProtocolError.java",
"chars": 1799,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "java/src/com/omniti/labs/FqDataProtocolError.java",
"chars": 1303,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "java/src/com/omniti/labs/FqHeartbeatException.java",
"chars": 1289,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "java/src/com/omniti/labs/FqMessage.java",
"chars": 8257,
"preview": "/*\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n *\n * Permission is hereby granted, fr"
},
{
"path": "java/src/main/java/com/circonus/FqClient.java",
"chars": 15699,
"preview": "/*\n * Copyright (c) 2016 Circonus, Inc.\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n "
},
{
"path": "java/src/main/java/com/circonus/FqClientImplDebug.java",
"chars": 2913,
"preview": "/*\n * Copyright (c) 2016 Circonus, Inc.\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n "
},
{
"path": "java/src/main/java/com/circonus/FqClientImplInterface.java",
"chars": 1976,
"preview": "/*\n * Copyright (c) 2016 Circonus, Inc.\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n "
},
{
"path": "java/src/main/java/com/circonus/FqClientImplNoop.java",
"chars": 2248,
"preview": "/*\n * Copyright (c) 2016 Circonus, Inc.\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n "
},
{
"path": "java/src/main/java/com/circonus/FqCommand.java",
"chars": 10196,
"preview": "/*\n * Copyright (c) 2016 Circonus, Inc.\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n "
},
{
"path": "java/src/main/java/com/circonus/FqCommandProtocolError.java",
"chars": 1833,
"preview": "/*\n * Copyright (c) 2016 Circonus, Inc.\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n "
},
{
"path": "java/src/main/java/com/circonus/FqDataProtocolError.java",
"chars": 1337,
"preview": "/*\n * Copyright (c) 2016 Circonus, Inc.\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n "
},
{
"path": "java/src/main/java/com/circonus/FqHeartbeatException.java",
"chars": 1323,
"preview": "/*\n * Copyright (c) 2016 Circonus, Inc.\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n "
},
{
"path": "java/src/main/java/com/circonus/FqMessage.java",
"chars": 8288,
"preview": "/*\n * Copyright (c) 2016 Circonus, Inc.\n * Copyright (c) 2013 OmniTI Computer Consulting, Inc.\n * All rights reserved.\n "
},
{
"path": "lua/fq-proxy",
"chars": 2666,
"preview": "#!/usr/bin/env luajit\nlocal fqclient = require 'fqclient'\n\nlocal usage = [[\nArguments:\n --src_host (default loca"
},
{
"path": "lua/fq-receiver",
"chars": 2251,
"preview": "#!/usr/bin/env luajit\nlocal fqclient = require 'fqclient'\n\nlocal usage = [[\nArguments:\n --host (default localhos"
},
{
"path": "lua/fq-sender",
"chars": 1785,
"preview": "#!/usr/bin/env luajit\nlocal fqclient = require 'fqclient'\nlocal ffi = require 'ffi';\n\nlocal USAGE = [[\nArguments:\n --ho"
},
{
"path": "lua/fqclient.lua.tail",
"chars": 4443,
"preview": "\nlocal gettimeofday_struct = ffi.new(\"timeval\")\nlocal function gettimeofday()\n ffi.C.gettimeofday(gettimeofday_struct, "
},
{
"path": "lua/generatelua.sh",
"chars": 2369,
"preview": "#!/usr/bin/env bash\n\nset -o errexit # Exit script on first error.\nset -o nounset # Treat references to unset variabl"
},
{
"path": "service-configs/50-circonus-fq.preset",
"chars": 55,
"preview": "# Ship disabled by default\ndisable circonus-fq.service\n"
},
{
"path": "service-configs/circonus-fq.service",
"chars": 325,
"preview": "[Unit]\nDescription=FQ\nAfter=network.target\n\n[Service]\nUser=fq\nWorkingDirectory=/opt/circonus/var/lib/fq\nEnvironmentFile="
},
{
"path": "service-configs/daemon_options",
"chars": 54,
"preview": "# Additional fqd commandline arguments\nDAEMON_OPTS=\"\"\n"
},
{
"path": "test/lua-support/init.lua",
"chars": 1,
"preview": "\n"
},
{
"path": "test/run-tests.sh",
"chars": 98,
"preview": "cd \"$(dirname $0)\"\n\nLD_LIBRARY_PATH=\"../\"\nexport LD_LIBRARY_PATH\n\n/opt/circonus/bin/mtevbusted $@\n"
},
{
"path": "test/test_spec.lua",
"chars": 4081,
"preview": "local fqclient = require(\"../lua/fqclient.lua\")\n\nlocal function mkreader(exchange, program)\n local key_auth = mtev.uuid"
},
{
"path": "web/css/bootstrap-theme.css",
"chars": 16819,
"preview": ".btn-default,\n.btn-primary,\n.btn-success,\n.btn-info,\n.btn-warning,\n.btn-danger {\n text-shadow: 0 -1px 0 rgba(0, 0, 0, 0"
},
{
"path": "web/css/bootstrap.css",
"chars": 119892,
"preview": "/*!\n * Bootstrap v3.0.0\n *\n * Copyright 2013 Twitter, Inc\n * Licensed under the Apache License v2.0\n * http://www.apache"
},
{
"path": "web/css/colorbrewer.css",
"chars": 59899,
"preview": "/* This product includes color specifications and designs developed by Cynthia Brewer (http://colorbrewer.org/). */\n.YlG"
},
{
"path": "web/css/theme.css",
"chars": 1778,
"preview": "body {\n padding-top: 75px;\n padding-bottom: 30px;\n}\n\n.container > div.navbar-header {\n background: transparent url(.."
},
{
"path": "web/index.html",
"chars": 8977,
"preview": "<!DOCTYPE html>\n<html lang=\"en\">\n <head>\n <meta charset=\"utf-8\">\n <meta name=\"viewport\" content=\"width=device-wid"
},
{
"path": "web/js/bootstrap.js",
"chars": 58458,
"preview": "/**\n* bootstrap.js v3.0.0 by @fat and @mdo\n* Copyright 2013 Twitter Inc.\n* http://www.apache.org/licenses/LICENSE-2.0\n*/"
},
{
"path": "web/js/colorbrewer.js",
"chars": 18990,
"preview": "// This product includes color specifications and designs developed by Cynthia Brewer (http://colorbrewer.org/).\nvar col"
},
{
"path": "web/js/fq.js",
"chars": 8687,
"preview": "Array.prototype.equals = function (array) {\n if (!array) return false;\n if (this.length != array.length) return fa"
}
]
About this extraction
This page contains the full source code of the circonus-labs/fq GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 78 files (800.6 KB), approximately 252.1k tokens, and a symbol index with 743 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.