Repository: kristoff-it/redis-cuckoofilter
Branch: master
Commit: a678ee5ed395
Files: 9
Total size: 107.1 KB
Directory structure:
gitextract_w82oi8x6/
├── .gitignore
├── LICENSE
├── README.md
├── RELEASENOTES
└── src/
├── lib/
│ ├── redismodule.h
│ └── zig-cuckoofilter.zig
├── redis-cuckoofilter.zig
├── redismodule.zig
└── t_cuckoofilter.zig
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
zig-cache/
# Prerequisites
*.d
# Object files
*.o
*.ko
*.obj
*.elf
# Linker output
*.ilk
*.map
*.exp
# Precompiled Headers
*.gch
*.pch
# Libraries
*.lib
*.a
*.la
*.lo
# Shared objects (inc. Windows DLLs)
*.dll
*.so
*.xo
*.so.*
*.dylib
# Executables
*.exe
*.out
*.app
*.i*86
*.x86_64
*.hex
# Debug files
*.dSYM/
*.su
*.idb
*.pdb
# Kernel Module Compile Results
*.mod*
*.cmd
.tmp_versions/
modules.order
Module.symvers
Mkfile.old
dkms.conf
*.rdb
.python-version
.DS_Store
*.pyc
.cache/v/cache/lastfailed
redis-cuckoofilter.h
*.zip
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2019 Loris Cro
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
<h1 align="center">redis-cuckoofilter</h1>
<p align="center">
<a href="https://github.com/kristoff-it/redis-cuckoofilter/releases/latest"><img src="https://badgen.net/github/release/kristoff-it/redis-cuckoofilter"/></a>
<a href="LICENSE"><img src="https://badgen.net/github/license/kristoff-it/zig-cuckoofilter" /></a>
<a href="https://github.com/kristoff-it/zig-cuckoofilter"><img src="https://badgen.net/badge/based%20on/zig-cuckoofilter" /></a>
<a href="https://twitter.com/croloris"><img src="https://badgen.net/badge/twitter/@croloris/1DA1F2?icon&label" /></a>
</p>
<p align="center">
Hashing-function agnostic Cuckoo filters for Redis.
</p>
What's a Cuckoo Filter?
-----------------------
Cuckoo filters are a probabilistic data structure that allows you to test for
membership of an element in a set without having to hold the whole set in
memory.
This is done at the cost of having a probability of getting a false positive
response, which, in other words, means that they can only answer "Definitely no"
or "Probably yes". The false positive probability is roughly inversely related
to how much memory you are willing to allocate to the filter.
The most iconic data structure used for this kind of task are Bloom filters
but Cuckoo filters boast both better practical performance and efficiency, and,
more importantly, the ability of **deleting elements from the filter**.
Bloom filters only support insertion of new items.
Some extensions of Bloom filters have the ability of deleting items but they
achieve so at the expense of precision or memory usage, resulting in a far worse
tradeoff compared to what Cuckoo filters offer.
What Makes This Redis Module Interesting
----------------------------------------
Cuckoo filters offer a very interesting division of labour between server and
clients.
Since Cuckoo filters rely on a single hashing of the original item you want to
insert, it is possible to off-load that part of the computation to the client.
In practical terms it means that instead of sending the whole item to Redis, the
clients send `hash` and `fingeprint` of the original item.
### What are the advantages of doing so?
- You need to push trough the cable a constant amount of data per item instead
of N bytes *(Redis is a remote service afterall, you're going through a UNIX
socket at the very least)*.
- To perform well, Cuckoo filters rely on a good choice of fingerprint for each
item and it should not be left to the library.
- **The hash function can be decided by you, meaning that this module is
hashing-function agnostic**.
The last point is the most important one.
It allows you to be more flexible in case you need to reason about item hashes
across different clients potentially written in different languages.
Additionally, different hashing function families specialize on different use
cases that might interest you or not. For example some work best for small data
(< 7 bytes), some the opposite. Some focus more on performance at the expense of
more collisions, while some others behave better than the rest on peculiar
platforms.
[This blogpost](http://aras-p.info/blog/2016/08/09/More-Hash-Function-Tests/)
shows a few benchmarks of different hashing function families.
Considering all of that, the choice of hashing and fingerprinting functions has
to be up to you.
*For the internal partial hashing that has to happen when reallocating a
fingerprint server-side, this implementation uses FNV1a which is robust and fast
for 1 byte inputs (the size of a fingerprint).*
*Thanks to how Cuckoo filters work, that choice is completely transparent to the
clients.*
Installation
------------
1. Download a precompiled binary from the
[Release section](https://github.com/kristoff-it/redis-cuckoofilter/releases/)
of this repo or compile it yourself (instructions at the end of this README).
2. Put `libredis-cuckoofilter.so` module in a folder readable by your Redis
server.
3. To try out the module you can send
`MODULE LOAD /path/to/libredis-cuckoofilter.so` using redis-cli or a client of
your choice.
4. Once you save on disk a key containing a Cuckoo filter you will need to add
`loadmodule /path/to/libredis-cuckoofilter.so` to your `redis.conf`, otherwise
Redis will not load complaining that it doesn't know how to read some data
from the `.rdb` file.
Quickstart
----------
```
redis-cli> MODULE LOAD /path/to/libredis-cuckoofilter.so
OK
redis-cli> CF.INIT test 64K
OK
redis-cli> CF.ADD test 5366164415461427448 97
OK
redis-cli> CF.CHECK test 5366164415461427448 97
(integer) 1
redis-cli> CF.REM test 5366164415461427448 97
OK
redis-cli> CF.CHECK test 5366164415461427448 97
(integer) 0
```
Client-side quickstart
----------------------
```python
import redis
r = redis.Redis()
# Load the module if you haven't done so already
r.execute_command("module", "load", "/path/to/libredis-cuckoofilter.so")
# Create a filter
r.execute_command("cf.init", "test", "64k")
# Define a fingerprinting function, for hashing we'll use python's builtin `hash()`
def fingerprint(x):
return ord(x[0]) # takes the first byte and returns its numerical value
item = "banana"
# Add an item to the filter
r.execute_command("cf.add", "test", hash(item), fingerprint(item))
# Check for its presence
r.execute_command("cf.check", "test", hash(item), finterprint(item)) # => true
# Check for a non-existing item
r.execute_command("cf.check", "test", hash("apple"), fingerprint("apple")) # => false
```
Fingerprint size and error rates
--------------------------------
In Cuckoo filters the number of bytes that we decide to use as fingerprint
will directly impact the maximum false positive error rate of a given filter.
This implementation supports 1, 2 and 4-byte wide fingerprints.
### 1 (3% error)
Error % -> `3.125e-02 (~0.03, i.e. 3%)`
### 2 (0.01% error)
Error % -> `1.22070312e-04 (~0.0001, i.e. 0.01%))`
### 4 (0.0000001% error)
Error % -> `9.31322574e-10 (~0.000000001, i.e. 0.0000001%)`
Complete command list
---------------------
### - `CF.SIZEFOR universe [fpsize] [EXACT]`
#### Complexity: O(1)
#### Example: `CF.SIZEFOR 1000 2 EXACT`
Returns the correct size for a filter that must hold at most `universe` items.
Default `fpsize` is 1, specify a different value if you need an error rate lower
than 3%.
Cuckoo filters should never be filled over 80% of their maximum theoretical capacity
both for performance reasons and because a filter that approaces 100% fill rate will
start refusing inserts with a `ERR too full` error.
This command will automatically pad `universe` for you. Use `EXACT` if you don't want
that behavior.
### - `CF.CAPACITY size [fpsize]`
#### Complexity: O(1)
#### Example: `CF.CAPACITY 4G 2`
Returns the theoretical maximum number of items that can be added to a filter of given
`size` and `fpsize`. Default `fpsize` is 1.
### - `CF.INIT key size [fpsize]`
#### Complexity: O(size)
#### Example: `CF.INIT mykey 64K`
Instantiates a new filter. Use `CF.SIZEFOR` to know the correct value for `size`.
Supported sizes are a power of 2 in this range: `1K .. 8G`.
Default error rate is 3%, use `fpsize` to specify a different target error rate.
### - `CF.ADD key hash fp`
#### Complexity: O(1)
#### Example `CF.ADD mykey 100 97`
Adds a new item to the filter. Both `hash` and `fp` must be numbers.
In particular, `hash` has to be a 64bit representable number, while `fp`
should be a `fpsize` representable number. As an example, a filter with
`fpsize` set to `1` will cause the maximum recommended value of `fp` to be `255`.
The `fp` argument is a `u32` so `(2^32)-1` is its maximum valid value, but when
`fpsize` is lower than `4`, high bits will be truncated (e.g. `-1 == 255` when
`fpsize == 1`).
You can use both signed and unsigned values as long as you are consistent
in their use. Internally all values will be transalted to unsigned.
If a filter is undersized/overfilled or you are adding multiple copies of
the same item or, worse, you're not properely handling information entropy,
this command will return `ERR too full`.
Read the extented example in
[kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter)
to learn more about misusage scenarios.
### - `CF.REM key hash fp`
#### Complexity: O(1)
#### Example `CF.REM mykey 100 97`
Deletes an item. Accepts the same arguments as `CF.ADD`.
WARNING: this command must be used to only delete items that were
previously inserted. Trying to delete non-existing items will corrupt the
filter and cause it to lockdown. When that happens all command will start
returning `ERR broken`, because at that point it will be impossible to
know what the correct state would be. Incurring in `ERR broken` is
a usage error and should never happen. Read the extented example in
[kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter)
to learn more about misusage scenarios.
### - `CF.CHECK key hash fp`
#### Complexity: O(1)
#### Example `CF.CHECK mykey 100 97`
Checks if an item is present in the filter or not. Returns `1` for the
positive case and `0` otherwise. Accepts the same arguments as `CF.ADD`.
### - `CF.COUNT key`
#### Complexity: O(1)
#### Example: `CF.COUNT mykey`
Returns the number of items present in the filter.
### - `CF.ISBROKEN key`
#### Complexity: O(1)
#### Example: `CF.ISBROKEN mykey`
Returns `1` if the filter was broken because of misusage of `CF.REM`,
returns `0` otherwise. A broken filter cannot be fixed and will start
returning `ERR broken` from most comamnds.
### - `CF.ISTOOFULL key`
#### Complexity: O(1)
#### Example: `CF.ISTOOFULL mykey`
Returns `1` if the filter is too full, returns `0` otherwise.
This command can return `1` even if you never received a
`ERR too full` from a call to `CF.ADD`.
Read the extented example in
[kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter)
to learn more about misusage scenarios.
### - `CF.FIXTOOFULL key`
#### Complexity: O(1) big constant
#### Example: `CF.FIXTOOFULL mykey`
If you are adding and also **deleting** items from the filter
but in a moment of *congestion* you ended up ovferfilling the filter,
this command can help re-distribute some items to fix the situation.
It's not a command you should ever rely on because it should never
be needed if you properly sized your filter using `CF.SIZEFOR`.
Read the extented example in
[kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter)
to learn more about misusage scenarios.
Advanced usage
--------------
Checkout
[kristoff-it/zig-cuckoofilter](https://github.com/kristoff-it/zig-cuckoofilter)
for more information about advanced usage of Cuckoo filters and
how to deal (and most importantly, prevent) failure scenarios.
Planned Features
----------------
- Advanced client-side syncrhonization
Given that now the logic is bundled in zig-cuckoofilter and that
it can now be used by any C ABI compatible target (checkout the
repo for examples in C, JS, Python and Go), combined with Streams
it would be possible to keep a client-side Cuckoo filter synced
with one in Redis, allowing clients to keep reads locally and
asyncrhonously sync with Redis to obtain new updates to the filter.
Compiling
---------
Download the latest Zig compiler version from http://ziglang.org.
### To compile for your native platform
```sh
$ zig build-lib -dynamic -isystem src --release-fast src/redis-cuckoofilter.zig
```
### To cross-compile
```sh
$ zig build-lib -dynamic -isystem src --release-fast -target x86_64-linux --library c src/redis-cuckoofilter.zig
```
Use `zig targets` for the complete list of available targets.
License
-------
MIT License
Copyright (c) 2019 Loris Cro
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: RELEASENOTES
================================================
-- 1.1.1
- Update documentation for CF.CAPACITY as it was out of sync with the code.
- Bundled a new build of the binaries in the release. Apparently I failed to
include up-to-date binaries in the latest release, leaving surprise bugs
for unsuspecting users (thanks @shahasim).
-- 1.1.0
- Fix bug that made fingerprints 1byte long even when specifying otherwise
(thanks @migolovanov).
- Make negative hash/fp finally work as intented: if the value is negative
it gets casted to unsigned and 2complement is applied.
-- 1.0.1
- Fix parsing of negative hash/fp
- Remove `random` attribute to write commands
Forgot to do that in the previous release, where insertion became fully
deterministic.
-- 1.0.0
- Now viable for production
With the new features this release can be considered stable enough
for production use.
- Complete rewrite of the module in Zig
Great improvement, nothing much else to say.
- Now using zig-cuckoofilter as backing library
zig-cuckoofilter is a production-ready, sans-io implementation
of Cuckoo Filters written by me. Check it out if you need support
for Cuckoo Filters on the client-side. Works with any C ABI compatile
target (via both static and dynamic linking) or language (via CFFI).
- New APIs for humans to help you size the filter
Use CF.SIZEFOR and CF.CAPACITY to properly size the filter.
No more reading through the documentation to know how to
properly configure your filters!
- New APIs
Check the README for a complete list of available commands.
- More flexible `hash` and `fp` parsing
You can now use both signed and unsigned representations without
any worry, as long as you are consistent. All values will be
internally converted to unsigned.
- Fully deterministic support for insertions
Each key saves the state of the PRNG, making the insert history
reproducible.
- Support for Redis Replication
With fully deterministic support it was now trivial to add support
for Redis replicas. Enjoy your now more resilient Cuckoo filters!
- Abandoned the idea of supporting resizing
Resizing comes at a price and goes against both Redis' and this
module's philosophy. This module is for advanced users so you
are expected to know how to size a filter properly. If you are in a
situation where you prefer dynamic sizing, then it's up to you to
orchestrate multiple filters manually (which is the only option
and exactly what the library would have done for you). So you
could start with one filter and then add more as more items come
in. You will have then to check all filters for presence, thus
changing the asymptotic properties of the resulting `MULTICHECK`
operation.
- Abandoned the idea of supporting filters for multisets
Cuckoo Filters are mainly useful because they allow you to delete
items and I haven't found a semantically sound way of mixing
deletions with multiset item-counting.
- Started investigating client-side libraries
Given that now the logic is bundled in zig-cuckoofilter and that
it can now be used by any C ABI compatible target (checkout the
repo for examples in C, JS, Python and Go), combined with Streams
it would be possible to keep a client-side Cuckoo filter synced
with one in Redis, allowing clients to keep reads locally and
asyncrhonously sync with Redis to obtain new updates to the filter.
-- 0.9
- Added fingerprint size option to `cf.init`
After some reasoning, I concluded that having the user choose
fingerprint size is the best choice in terms of API design.
Users care about error rate but it is not possible to have them
directly decide it as it is the result of choosing fingerprint size
and bucket size, settings that in turn define many other performance
characteristics of the filter.
Considering Redis' nature and common usage patterns,
it's very important to keep the memory layout efficient in terms of
access. This means having a few reasonably useful settings that allow
keeping all buckets word or half-word aligned.
A fpsize setting seems reasonable considering that users already
have to implement a fingerprinting function and should be well aware
of how long the fingerprint should be.
With this new setting, users can create a filter with 1, 2 or 4 bytes
long fingerprints which in turn cap the filter's error rate respectively
to 0.03, 0.0001 and 0.000000001.
The first error rate (0.03) is the point where Cuckoo filters become
more efficient than Bloom filters, the others follow from doubling the
fingerprint size.
For 1 and 2 fingerprint size, bucket size is 4.
For 4, bucketsize is 2.
This makes all bucket fetches to be performed in a single word read.
- Added reference python implementation and testing
Adding variable fingerprint size made the C code much more complex.
To keep reasonably well of what the code is supposed to be doing, I
wrote a pure python implementation that is both very easy to understand
and very slow.
The standard selftest suite has been also adapted to the python code
and a "lockstep" testing methodology has devised: each test operation
applied to the Redis module is also applied to the python instance and
before proceeding to the next, both filters are serialized and confronted
to make sure all operations produce byte-by-byte equal results.
This test is very slow but ensures a reasonable certainty that the Python
and C code behave consistently. This paired with the fact that the Python
code is very easy to understand should help build and test new features in
the future.
-- 0.2
- Removed bucket size option from `cf.init`
The reasoning being that the option makes more sense once we decide that
we also want to parametrize the fingerprint size as the two options
are intertwined (bigger bucket size <==> longer fingerprints). 1 byte
fingerprints and 4 buckets work well together (cf. the paper) so it's
good enough for now and makes the interface more appealing.
This also changed how the filter is saved to disk so the module will
refuse to load 0.1 filters. No backwards compatibility is going to be
implemented until we reach 1.0
-- 0.1
- Init
================================================
FILE: src/lib/redismodule.h
================================================
#ifndef REDISMODULE_H
#define REDISMODULE_H
#include <sys/types.h>
#include <stdint.h>
#include <stdio.h>
/* ---------------- Defines common between core and modules --------------- */
/* Error status return values. */
#define REDISMODULE_OK 0
#define REDISMODULE_ERR 1
/* API versions. */
#define REDISMODULE_APIVER_1 1
/* API flags and constants */
#define REDISMODULE_READ (1<<0)
#define REDISMODULE_WRITE (1<<1)
#define REDISMODULE_LIST_HEAD 0
#define REDISMODULE_LIST_TAIL 1
/* Key types. */
#define REDISMODULE_KEYTYPE_EMPTY 0
#define REDISMODULE_KEYTYPE_STRING 1
#define REDISMODULE_KEYTYPE_LIST 2
#define REDISMODULE_KEYTYPE_HASH 3
#define REDISMODULE_KEYTYPE_SET 4
#define REDISMODULE_KEYTYPE_ZSET 5
#define REDISMODULE_KEYTYPE_MODULE 6
/* Reply types. */
#define REDISMODULE_REPLY_UNKNOWN -1
#define REDISMODULE_REPLY_STRING 0
#define REDISMODULE_REPLY_ERROR 1
#define REDISMODULE_REPLY_INTEGER 2
#define REDISMODULE_REPLY_ARRAY 3
#define REDISMODULE_REPLY_NULL 4
/* Postponed array length. */
#define REDISMODULE_POSTPONED_ARRAY_LEN -1
/* Expire */
#define REDISMODULE_NO_EXPIRE -1
/* Sorted set API flags. */
#define REDISMODULE_ZADD_XX (1<<0)
#define REDISMODULE_ZADD_NX (1<<1)
#define REDISMODULE_ZADD_ADDED (1<<2)
#define REDISMODULE_ZADD_UPDATED (1<<3)
#define REDISMODULE_ZADD_NOP (1<<4)
/* Hash API flags. */
#define REDISMODULE_HASH_NONE 0
#define REDISMODULE_HASH_NX (1<<0)
#define REDISMODULE_HASH_XX (1<<1)
#define REDISMODULE_HASH_CFIELDS (1<<2)
#define REDISMODULE_HASH_EXISTS (1<<3)
/* Context Flags: Info about the current context returned by
* RM_GetContextFlags(). */
/* The command is running in the context of a Lua script */
#define REDISMODULE_CTX_FLAGS_LUA (1<<0)
/* The command is running inside a Redis transaction */
#define REDISMODULE_CTX_FLAGS_MULTI (1<<1)
/* The instance is a master */
#define REDISMODULE_CTX_FLAGS_MASTER (1<<2)
/* The instance is a slave */
#define REDISMODULE_CTX_FLAGS_SLAVE (1<<3)
/* The instance is read-only (usually meaning it's a slave as well) */
#define REDISMODULE_CTX_FLAGS_READONLY (1<<4)
/* The instance is running in cluster mode */
#define REDISMODULE_CTX_FLAGS_CLUSTER (1<<5)
/* The instance has AOF enabled */
#define REDISMODULE_CTX_FLAGS_AOF (1<<6)
/* The instance has RDB enabled */
#define REDISMODULE_CTX_FLAGS_RDB (1<<7)
/* The instance has Maxmemory set */
#define REDISMODULE_CTX_FLAGS_MAXMEMORY (1<<8)
/* Maxmemory is set and has an eviction policy that may delete keys */
#define REDISMODULE_CTX_FLAGS_EVICT (1<<9)
/* Redis is out of memory according to the maxmemory flag. */
#define REDISMODULE_CTX_FLAGS_OOM (1<<10)
/* Less than 25% of memory available according to maxmemory. */
#define REDISMODULE_CTX_FLAGS_OOM_WARNING (1<<11)
/* The command was sent over the replication link. */
#define REDISMODULE_CTX_FLAGS_REPLICATED (1<<12)
#define REDISMODULE_NOTIFY_GENERIC (1<<2) /* g */
#define REDISMODULE_NOTIFY_STRING (1<<3) /* $ */
#define REDISMODULE_NOTIFY_LIST (1<<4) /* l */
#define REDISMODULE_NOTIFY_SET (1<<5) /* s */
#define REDISMODULE_NOTIFY_HASH (1<<6) /* h */
#define REDISMODULE_NOTIFY_ZSET (1<<7) /* z */
#define REDISMODULE_NOTIFY_EXPIRED (1<<8) /* x */
#define REDISMODULE_NOTIFY_EVICTED (1<<9) /* e */
#define REDISMODULE_NOTIFY_STREAM (1<<10) /* t */
#define REDISMODULE_NOTIFY_KEY_MISS (1<<11) /* m */
#define REDISMODULE_NOTIFY_ALL (REDISMODULE_NOTIFY_GENERIC | REDISMODULE_NOTIFY_STRING | REDISMODULE_NOTIFY_LIST | REDISMODULE_NOTIFY_SET | REDISMODULE_NOTIFY_HASH | REDISMODULE_NOTIFY_ZSET | REDISMODULE_NOTIFY_EXPIRED | REDISMODULE_NOTIFY_EVICTED | REDISMODULE_NOTIFY_STREAM | REDISMODULE_NOTIFY_KEY_MISS) /* A */
/* A special pointer that we can use between the core and the module to signal
* field deletion, and that is impossible to be a valid pointer. */
#define REDISMODULE_HASH_DELETE ((RedisModuleString*)(long)1)
/* Error messages. */
#define REDISMODULE_ERRORMSG_WRONGTYPE "WRONGTYPE Operation against a key holding the wrong kind of value"
#define REDISMODULE_POSITIVE_INFINITE (1.0/0.0)
#define REDISMODULE_NEGATIVE_INFINITE (-1.0/0.0)
/* Cluster API defines. */
#define REDISMODULE_NODE_ID_LEN 40
#define REDISMODULE_NODE_MYSELF (1<<0)
#define REDISMODULE_NODE_MASTER (1<<1)
#define REDISMODULE_NODE_SLAVE (1<<2)
#define REDISMODULE_NODE_PFAIL (1<<3)
#define REDISMODULE_NODE_FAIL (1<<4)
#define REDISMODULE_NODE_NOFAILOVER (1<<5)
#define REDISMODULE_CLUSTER_FLAG_NONE 0
#define REDISMODULE_CLUSTER_FLAG_NO_FAILOVER (1<<1)
#define REDISMODULE_CLUSTER_FLAG_NO_REDIRECTION (1<<2)
#define REDISMODULE_NOT_USED(V) ((void) V)
/* This type represents a timer handle, and is returned when a timer is
* registered and used in order to invalidate a timer. It's just a 64 bit
* number, because this is how each timer is represented inside the radix tree
* of timers that are going to expire, sorted by expire time. */
typedef uint64_t RedisModuleTimerID;
/* CommandFilter Flags */
/* Do filter RedisModule_Call() commands initiated by module itself. */
#define REDISMODULE_CMDFILTER_NOSELF (1<<0)
/* ------------------------- End of common defines ------------------------ */
#ifndef REDISMODULE_CORE
typedef long long mstime_t;
/* Incomplete structures for compiler checks but opaque access. */
typedef struct RedisModuleCtx RedisModuleCtx;
typedef struct RedisModuleKey RedisModuleKey;
typedef struct RedisModuleString RedisModuleString;
typedef struct RedisModuleCallReply RedisModuleCallReply;
typedef struct RedisModuleIO RedisModuleIO;
typedef struct RedisModuleType RedisModuleType;
typedef struct RedisModuleDigest RedisModuleDigest;
typedef struct RedisModuleBlockedClient RedisModuleBlockedClient;
typedef struct RedisModuleClusterInfo RedisModuleClusterInfo;
typedef struct RedisModuleDict RedisModuleDict;
typedef struct RedisModuleDictIter RedisModuleDictIter;
typedef struct RedisModuleCommandFilterCtx RedisModuleCommandFilterCtx;
typedef struct RedisModuleCommandFilter RedisModuleCommandFilter;
typedef int (*RedisModuleCmdFunc)(RedisModuleCtx *ctx, RedisModuleString **argv, int argc);
typedef void (*RedisModuleDisconnectFunc)(RedisModuleCtx *ctx, RedisModuleBlockedClient *bc);
typedef int (*RedisModuleNotificationFunc)(RedisModuleCtx *ctx, int type, const char *event, RedisModuleString *key);
typedef void *(*RedisModuleTypeLoadFunc)(RedisModuleIO *rdb, int encver);
typedef void (*RedisModuleTypeSaveFunc)(RedisModuleIO *rdb, void *value);
typedef void (*RedisModuleTypeRewriteFunc)(RedisModuleIO *aof, RedisModuleString *key, void *value);
typedef size_t (*RedisModuleTypeMemUsageFunc)(const void *value);
typedef void (*RedisModuleTypeDigestFunc)(RedisModuleDigest *digest, void *value);
typedef void (*RedisModuleTypeFreeFunc)(void *value);
typedef void (*RedisModuleClusterMessageReceiver)(RedisModuleCtx *ctx, const char *sender_id, uint8_t type, const unsigned char *payload, uint32_t len);
typedef void (*RedisModuleTimerProc)(RedisModuleCtx *ctx, void *data);
typedef void (*RedisModuleCommandFilterFunc) (RedisModuleCommandFilterCtx *filter);
#define REDISMODULE_TYPE_METHOD_VERSION 1
typedef struct RedisModuleTypeMethods {
uint64_t version;
RedisModuleTypeLoadFunc rdb_load;
RedisModuleTypeSaveFunc rdb_save;
RedisModuleTypeRewriteFunc aof_rewrite;
RedisModuleTypeMemUsageFunc mem_usage;
RedisModuleTypeDigestFunc digest;
RedisModuleTypeFreeFunc free;
} RedisModuleTypeMethods;
#define REDISMODULE_GET_API(name) \
RedisModule_GetApi("RedisModule_" #name, ((void **)&RedisModule_ ## name))
#define REDISMODULE_API_FUNC(x) (*x)
void *REDISMODULE_API_FUNC(RedisModule_Alloc)(size_t bytes);
void *REDISMODULE_API_FUNC(RedisModule_Realloc)(void *ptr, size_t bytes);
void REDISMODULE_API_FUNC(RedisModule_Free)(void *ptr);
void *REDISMODULE_API_FUNC(RedisModule_Calloc)(size_t nmemb, size_t size);
char *REDISMODULE_API_FUNC(RedisModule_Strdup)(const char *str);
int REDISMODULE_API_FUNC(RedisModule_GetApi)(const char *, void *);
int REDISMODULE_API_FUNC(RedisModule_CreateCommand)(RedisModuleCtx *ctx, const char *name, RedisModuleCmdFunc cmdfunc, const char *strflags, int firstkey, int lastkey, int keystep);
void REDISMODULE_API_FUNC(RedisModule_SetModuleAttribs)(RedisModuleCtx *ctx, const char *name, int ver, int apiver);
int REDISMODULE_API_FUNC(RedisModule_IsModuleNameBusy)(const char *name);
int REDISMODULE_API_FUNC(RedisModule_WrongArity)(RedisModuleCtx *ctx);
int REDISMODULE_API_FUNC(RedisModule_ReplyWithLongLong)(RedisModuleCtx *ctx, long long ll);
int REDISMODULE_API_FUNC(RedisModule_GetSelectedDb)(RedisModuleCtx *ctx);
int REDISMODULE_API_FUNC(RedisModule_SelectDb)(RedisModuleCtx *ctx, int newid);
void *REDISMODULE_API_FUNC(RedisModule_OpenKey)(RedisModuleCtx *ctx, RedisModuleString *keyname, int mode);
void REDISMODULE_API_FUNC(RedisModule_CloseKey)(RedisModuleKey *kp);
int REDISMODULE_API_FUNC(RedisModule_KeyType)(RedisModuleKey *kp);
size_t REDISMODULE_API_FUNC(RedisModule_ValueLength)(RedisModuleKey *kp);
int REDISMODULE_API_FUNC(RedisModule_ListPush)(RedisModuleKey *kp, int where, RedisModuleString *ele);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_ListPop)(RedisModuleKey *key, int where);
RedisModuleCallReply *REDISMODULE_API_FUNC(RedisModule_Call)(RedisModuleCtx *ctx, const char *cmdname, const char *fmt, ...);
const char *REDISMODULE_API_FUNC(RedisModule_CallReplyProto)(RedisModuleCallReply *reply, size_t *len);
void REDISMODULE_API_FUNC(RedisModule_FreeCallReply)(RedisModuleCallReply *reply);
int REDISMODULE_API_FUNC(RedisModule_CallReplyType)(RedisModuleCallReply *reply);
long long REDISMODULE_API_FUNC(RedisModule_CallReplyInteger)(RedisModuleCallReply *reply);
size_t REDISMODULE_API_FUNC(RedisModule_CallReplyLength)(RedisModuleCallReply *reply);
RedisModuleCallReply *REDISMODULE_API_FUNC(RedisModule_CallReplyArrayElement)(RedisModuleCallReply *reply, size_t idx);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateString)(RedisModuleCtx *ctx, const char *ptr, size_t len);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringFromLongLong)(RedisModuleCtx *ctx, long long ll);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringFromString)(RedisModuleCtx *ctx, const RedisModuleString *str);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringPrintf)(RedisModuleCtx *ctx, const char *fmt, ...);
void REDISMODULE_API_FUNC(RedisModule_FreeString)(RedisModuleCtx *ctx, RedisModuleString *str);
const char *REDISMODULE_API_FUNC(RedisModule_StringPtrLen)(const RedisModuleString *str, size_t *len);
int REDISMODULE_API_FUNC(RedisModule_ReplyWithError)(RedisModuleCtx *ctx, const char *err);
int REDISMODULE_API_FUNC(RedisModule_ReplyWithSimpleString)(RedisModuleCtx *ctx, const char *msg);
int REDISMODULE_API_FUNC(RedisModule_ReplyWithArray)(RedisModuleCtx *ctx, long len);
void REDISMODULE_API_FUNC(RedisModule_ReplySetArrayLength)(RedisModuleCtx *ctx, long len);
int REDISMODULE_API_FUNC(RedisModule_ReplyWithStringBuffer)(RedisModuleCtx *ctx, const char *buf, size_t len);
int REDISMODULE_API_FUNC(RedisModule_ReplyWithString)(RedisModuleCtx *ctx, RedisModuleString *str);
int REDISMODULE_API_FUNC(RedisModule_ReplyWithNull)(RedisModuleCtx *ctx);
int REDISMODULE_API_FUNC(RedisModule_ReplyWithDouble)(RedisModuleCtx *ctx, double d);
int REDISMODULE_API_FUNC(RedisModule_ReplyWithCallReply)(RedisModuleCtx *ctx, RedisModuleCallReply *reply);
int REDISMODULE_API_FUNC(RedisModule_StringToLongLong)(const RedisModuleString *str, long long *ll);
int REDISMODULE_API_FUNC(RedisModule_StringToDouble)(const RedisModuleString *str, double *d);
void REDISMODULE_API_FUNC(RedisModule_AutoMemory)(RedisModuleCtx *ctx);
int REDISMODULE_API_FUNC(RedisModule_Replicate)(RedisModuleCtx *ctx, const char *cmdname, const char *fmt, ...);
int REDISMODULE_API_FUNC(RedisModule_ReplicateVerbatim)(RedisModuleCtx *ctx);
const char *REDISMODULE_API_FUNC(RedisModule_CallReplyStringPtr)(RedisModuleCallReply *reply, size_t *len);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CreateStringFromCallReply)(RedisModuleCallReply *reply);
int REDISMODULE_API_FUNC(RedisModule_DeleteKey)(RedisModuleKey *key);
int REDISMODULE_API_FUNC(RedisModule_UnlinkKey)(RedisModuleKey *key);
int REDISMODULE_API_FUNC(RedisModule_StringSet)(RedisModuleKey *key, RedisModuleString *str);
char *REDISMODULE_API_FUNC(RedisModule_StringDMA)(RedisModuleKey *key, size_t *len, int mode);
int REDISMODULE_API_FUNC(RedisModule_StringTruncate)(RedisModuleKey *key, size_t newlen);
mstime_t REDISMODULE_API_FUNC(RedisModule_GetExpire)(RedisModuleKey *key);
int REDISMODULE_API_FUNC(RedisModule_SetExpire)(RedisModuleKey *key, mstime_t expire);
int REDISMODULE_API_FUNC(RedisModule_ZsetAdd)(RedisModuleKey *key, double score, RedisModuleString *ele, int *flagsptr);
int REDISMODULE_API_FUNC(RedisModule_ZsetIncrby)(RedisModuleKey *key, double score, RedisModuleString *ele, int *flagsptr, double *newscore);
int REDISMODULE_API_FUNC(RedisModule_ZsetScore)(RedisModuleKey *key, RedisModuleString *ele, double *score);
int REDISMODULE_API_FUNC(RedisModule_ZsetRem)(RedisModuleKey *key, RedisModuleString *ele, int *deleted);
void REDISMODULE_API_FUNC(RedisModule_ZsetRangeStop)(RedisModuleKey *key);
int REDISMODULE_API_FUNC(RedisModule_ZsetFirstInScoreRange)(RedisModuleKey *key, double min, double max, int minex, int maxex);
int REDISMODULE_API_FUNC(RedisModule_ZsetLastInScoreRange)(RedisModuleKey *key, double min, double max, int minex, int maxex);
int REDISMODULE_API_FUNC(RedisModule_ZsetFirstInLexRange)(RedisModuleKey *key, RedisModuleString *min, RedisModuleString *max);
int REDISMODULE_API_FUNC(RedisModule_ZsetLastInLexRange)(RedisModuleKey *key, RedisModuleString *min, RedisModuleString *max);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_ZsetRangeCurrentElement)(RedisModuleKey *key, double *score);
int REDISMODULE_API_FUNC(RedisModule_ZsetRangeNext)(RedisModuleKey *key);
int REDISMODULE_API_FUNC(RedisModule_ZsetRangePrev)(RedisModuleKey *key);
int REDISMODULE_API_FUNC(RedisModule_ZsetRangeEndReached)(RedisModuleKey *key);
int REDISMODULE_API_FUNC(RedisModule_HashSet)(RedisModuleKey *key, int flags, ...);
int REDISMODULE_API_FUNC(RedisModule_HashGet)(RedisModuleKey *key, int flags, ...);
int REDISMODULE_API_FUNC(RedisModule_IsKeysPositionRequest)(RedisModuleCtx *ctx);
void REDISMODULE_API_FUNC(RedisModule_KeyAtPos)(RedisModuleCtx *ctx, int pos);
unsigned long long REDISMODULE_API_FUNC(RedisModule_GetClientId)(RedisModuleCtx *ctx);
int REDISMODULE_API_FUNC(RedisModule_GetContextFlags)(RedisModuleCtx *ctx);
void *REDISMODULE_API_FUNC(RedisModule_PoolAlloc)(RedisModuleCtx *ctx, size_t bytes);
RedisModuleType *REDISMODULE_API_FUNC(RedisModule_CreateDataType)(RedisModuleCtx *ctx, const char *name, int encver, RedisModuleTypeMethods *typemethods);
int REDISMODULE_API_FUNC(RedisModule_ModuleTypeSetValue)(RedisModuleKey *key, RedisModuleType *mt, void *value);
RedisModuleType *REDISMODULE_API_FUNC(RedisModule_ModuleTypeGetType)(RedisModuleKey *key);
void *REDISMODULE_API_FUNC(RedisModule_ModuleTypeGetValue)(RedisModuleKey *key);
void REDISMODULE_API_FUNC(RedisModule_SaveUnsigned)(RedisModuleIO *io, uint64_t value);
uint64_t REDISMODULE_API_FUNC(RedisModule_LoadUnsigned)(RedisModuleIO *io);
void REDISMODULE_API_FUNC(RedisModule_SaveSigned)(RedisModuleIO *io, int64_t value);
int64_t REDISMODULE_API_FUNC(RedisModule_LoadSigned)(RedisModuleIO *io);
void REDISMODULE_API_FUNC(RedisModule_EmitAOF)(RedisModuleIO *io, const char *cmdname, const char *fmt, ...);
void REDISMODULE_API_FUNC(RedisModule_SaveString)(RedisModuleIO *io, RedisModuleString *s);
void REDISMODULE_API_FUNC(RedisModule_SaveStringBuffer)(RedisModuleIO *io, const char *str, size_t len);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_LoadString)(RedisModuleIO *io);
char *REDISMODULE_API_FUNC(RedisModule_LoadStringBuffer)(RedisModuleIO *io, size_t *lenptr);
void REDISMODULE_API_FUNC(RedisModule_SaveDouble)(RedisModuleIO *io, double value);
double REDISMODULE_API_FUNC(RedisModule_LoadDouble)(RedisModuleIO *io);
void REDISMODULE_API_FUNC(RedisModule_SaveFloat)(RedisModuleIO *io, float value);
float REDISMODULE_API_FUNC(RedisModule_LoadFloat)(RedisModuleIO *io);
void REDISMODULE_API_FUNC(RedisModule_Log)(RedisModuleCtx *ctx, const char *level, const char *fmt, ...);
void REDISMODULE_API_FUNC(RedisModule_LogIOError)(RedisModuleIO *io, const char *levelstr, const char *fmt, ...);
int REDISMODULE_API_FUNC(RedisModule_StringAppendBuffer)(RedisModuleCtx *ctx, RedisModuleString *str, const char *buf, size_t len);
void REDISMODULE_API_FUNC(RedisModule_RetainString)(RedisModuleCtx *ctx, RedisModuleString *str);
int REDISMODULE_API_FUNC(RedisModule_StringCompare)(RedisModuleString *a, RedisModuleString *b);
RedisModuleCtx *REDISMODULE_API_FUNC(RedisModule_GetContextFromIO)(RedisModuleIO *io);
const RedisModuleString *REDISMODULE_API_FUNC(RedisModule_GetKeyNameFromIO)(RedisModuleIO *io);
long long REDISMODULE_API_FUNC(RedisModule_Milliseconds)(void);
void REDISMODULE_API_FUNC(RedisModule_DigestAddStringBuffer)(RedisModuleDigest *md, unsigned char *ele, size_t len);
void REDISMODULE_API_FUNC(RedisModule_DigestAddLongLong)(RedisModuleDigest *md, long long ele);
void REDISMODULE_API_FUNC(RedisModule_DigestEndSequence)(RedisModuleDigest *md);
RedisModuleDict *REDISMODULE_API_FUNC(RedisModule_CreateDict)(RedisModuleCtx *ctx);
void REDISMODULE_API_FUNC(RedisModule_FreeDict)(RedisModuleCtx *ctx, RedisModuleDict *d);
uint64_t REDISMODULE_API_FUNC(RedisModule_DictSize)(RedisModuleDict *d);
int REDISMODULE_API_FUNC(RedisModule_DictSetC)(RedisModuleDict *d, void *key, size_t keylen, void *ptr);
int REDISMODULE_API_FUNC(RedisModule_DictReplaceC)(RedisModuleDict *d, void *key, size_t keylen, void *ptr);
int REDISMODULE_API_FUNC(RedisModule_DictSet)(RedisModuleDict *d, RedisModuleString *key, void *ptr);
int REDISMODULE_API_FUNC(RedisModule_DictReplace)(RedisModuleDict *d, RedisModuleString *key, void *ptr);
void *REDISMODULE_API_FUNC(RedisModule_DictGetC)(RedisModuleDict *d, void *key, size_t keylen, int *nokey);
void *REDISMODULE_API_FUNC(RedisModule_DictGet)(RedisModuleDict *d, RedisModuleString *key, int *nokey);
int REDISMODULE_API_FUNC(RedisModule_DictDelC)(RedisModuleDict *d, void *key, size_t keylen, void *oldval);
int REDISMODULE_API_FUNC(RedisModule_DictDel)(RedisModuleDict *d, RedisModuleString *key, void *oldval);
RedisModuleDictIter *REDISMODULE_API_FUNC(RedisModule_DictIteratorStartC)(RedisModuleDict *d, const char *op, void *key, size_t keylen);
RedisModuleDictIter *REDISMODULE_API_FUNC(RedisModule_DictIteratorStart)(RedisModuleDict *d, const char *op, RedisModuleString *key);
void REDISMODULE_API_FUNC(RedisModule_DictIteratorStop)(RedisModuleDictIter *di);
int REDISMODULE_API_FUNC(RedisModule_DictIteratorReseekC)(RedisModuleDictIter *di, const char *op, void *key, size_t keylen);
int REDISMODULE_API_FUNC(RedisModule_DictIteratorReseek)(RedisModuleDictIter *di, const char *op, RedisModuleString *key);
void *REDISMODULE_API_FUNC(RedisModule_DictNextC)(RedisModuleDictIter *di, size_t *keylen, void **dataptr);
void *REDISMODULE_API_FUNC(RedisModule_DictPrevC)(RedisModuleDictIter *di, size_t *keylen, void **dataptr);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_DictNext)(RedisModuleCtx *ctx, RedisModuleDictIter *di, void **dataptr);
RedisModuleString *REDISMODULE_API_FUNC(RedisModule_DictPrev)(RedisModuleCtx *ctx, RedisModuleDictIter *di, void **dataptr);
int REDISMODULE_API_FUNC(RedisModule_DictCompareC)(RedisModuleDictIter *di, const char *op, void *key, size_t keylen);
int REDISMODULE_API_FUNC(RedisModule_DictCompare)(RedisModuleDictIter *di, const char *op, RedisModuleString *key);
/* Experimental APIs */
#ifdef REDISMODULE_EXPERIMENTAL_API
#define REDISMODULE_EXPERIMENTAL_API_VERSION 3
RedisModuleBlockedClient *REDISMODULE_API_FUNC(RedisModule_BlockClient)(RedisModuleCtx *ctx, RedisModuleCmdFunc reply_callback, RedisModuleCmdFunc timeout_callback, void (*free_privdata)(RedisModuleCtx*,void*), long long timeout_ms);
int REDISMODULE_API_FUNC(RedisModule_UnblockClient)(RedisModuleBlockedClient *bc, void *privdata);
int REDISMODULE_API_FUNC(RedisModule_IsBlockedReplyRequest)(RedisModuleCtx *ctx);
int REDISMODULE_API_FUNC(RedisModule_IsBlockedTimeoutRequest)(RedisModuleCtx *ctx);
void *REDISMODULE_API_FUNC(RedisModule_GetBlockedClientPrivateData)(RedisModuleCtx *ctx);
RedisModuleBlockedClient *REDISMODULE_API_FUNC(RedisModule_GetBlockedClientHandle)(RedisModuleCtx *ctx);
int REDISMODULE_API_FUNC(RedisModule_AbortBlock)(RedisModuleBlockedClient *bc);
RedisModuleCtx *REDISMODULE_API_FUNC(RedisModule_GetThreadSafeContext)(RedisModuleBlockedClient *bc);
void REDISMODULE_API_FUNC(RedisModule_FreeThreadSafeContext)(RedisModuleCtx *ctx);
void REDISMODULE_API_FUNC(RedisModule_ThreadSafeContextLock)(RedisModuleCtx *ctx);
void REDISMODULE_API_FUNC(RedisModule_ThreadSafeContextUnlock)(RedisModuleCtx *ctx);
int REDISMODULE_API_FUNC(RedisModule_SubscribeToKeyspaceEvents)(RedisModuleCtx *ctx, int types, RedisModuleNotificationFunc cb);
int REDISMODULE_API_FUNC(RedisModule_BlockedClientDisconnected)(RedisModuleCtx *ctx);
void REDISMODULE_API_FUNC(RedisModule_RegisterClusterMessageReceiver)(RedisModuleCtx *ctx, uint8_t type, RedisModuleClusterMessageReceiver callback);
int REDISMODULE_API_FUNC(RedisModule_SendClusterMessage)(RedisModuleCtx *ctx, char *target_id, uint8_t type, unsigned char *msg, uint32_t len);
int REDISMODULE_API_FUNC(RedisModule_GetClusterNodeInfo)(RedisModuleCtx *ctx, const char *id, char *ip, char *master_id, int *port, int *flags);
char **REDISMODULE_API_FUNC(RedisModule_GetClusterNodesList)(RedisModuleCtx *ctx, size_t *numnodes);
void REDISMODULE_API_FUNC(RedisModule_FreeClusterNodesList)(char **ids);
RedisModuleTimerID REDISMODULE_API_FUNC(RedisModule_CreateTimer)(RedisModuleCtx *ctx, mstime_t period, RedisModuleTimerProc callback, void *data);
int REDISMODULE_API_FUNC(RedisModule_StopTimer)(RedisModuleCtx *ctx, RedisModuleTimerID id, void **data);
int REDISMODULE_API_FUNC(RedisModule_GetTimerInfo)(RedisModuleCtx *ctx, RedisModuleTimerID id, uint64_t *remaining, void **data);
const char *REDISMODULE_API_FUNC(RedisModule_GetMyClusterID)(void);
size_t REDISMODULE_API_FUNC(RedisModule_GetClusterSize)(void);
void REDISMODULE_API_FUNC(RedisModule_GetRandomBytes)(unsigned char *dst, size_t len);
void REDISMODULE_API_FUNC(RedisModule_GetRandomHexChars)(char *dst, size_t len);
void REDISMODULE_API_FUNC(RedisModule_SetDisconnectCallback)(RedisModuleBlockedClient *bc, RedisModuleDisconnectFunc callback);
void REDISMODULE_API_FUNC(RedisModule_SetClusterFlags)(RedisModuleCtx *ctx, uint64_t flags);
int REDISMODULE_API_FUNC(RedisModule_ExportSharedAPI)(RedisModuleCtx *ctx, const char *apiname, void *func);
void *REDISMODULE_API_FUNC(RedisModule_GetSharedAPI)(RedisModuleCtx *ctx, const char *apiname);
RedisModuleCommandFilter *REDISMODULE_API_FUNC(RedisModule_RegisterCommandFilter)(RedisModuleCtx *ctx, RedisModuleCommandFilterFunc cb, int flags);
int REDISMODULE_API_FUNC(RedisModule_UnregisterCommandFilter)(RedisModuleCtx *ctx, RedisModuleCommandFilter *filter);
int REDISMODULE_API_FUNC(RedisModule_CommandFilterArgsCount)(RedisModuleCommandFilterCtx *fctx);
const RedisModuleString *REDISMODULE_API_FUNC(RedisModule_CommandFilterArgGet)(RedisModuleCommandFilterCtx *fctx, int pos);
int REDISMODULE_API_FUNC(RedisModule_CommandFilterArgInsert)(RedisModuleCommandFilterCtx *fctx, int pos, RedisModuleString *arg);
int REDISMODULE_API_FUNC(RedisModule_CommandFilterArgReplace)(RedisModuleCommandFilterCtx *fctx, int pos, RedisModuleString *arg);
int REDISMODULE_API_FUNC(RedisModule_CommandFilterArgDelete)(RedisModuleCommandFilterCtx *fctx, int pos);
#endif
/* This is included inline inside each Redis module. */
static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int apiver) __attribute__((unused));
static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int ver, int apiver) {
void *getapifuncptr = ((void**)ctx)[0];
RedisModule_GetApi = (int (*)(const char *, void *)) (unsigned long)getapifuncptr;
REDISMODULE_GET_API(Alloc);
REDISMODULE_GET_API(Calloc);
REDISMODULE_GET_API(Free);
REDISMODULE_GET_API(Realloc);
REDISMODULE_GET_API(Strdup);
REDISMODULE_GET_API(CreateCommand);
REDISMODULE_GET_API(SetModuleAttribs);
REDISMODULE_GET_API(IsModuleNameBusy);
REDISMODULE_GET_API(WrongArity);
REDISMODULE_GET_API(ReplyWithLongLong);
REDISMODULE_GET_API(ReplyWithError);
REDISMODULE_GET_API(ReplyWithSimpleString);
REDISMODULE_GET_API(ReplyWithArray);
REDISMODULE_GET_API(ReplySetArrayLength);
REDISMODULE_GET_API(ReplyWithStringBuffer);
REDISMODULE_GET_API(ReplyWithString);
REDISMODULE_GET_API(ReplyWithNull);
REDISMODULE_GET_API(ReplyWithCallReply);
REDISMODULE_GET_API(ReplyWithDouble);
REDISMODULE_GET_API(ReplySetArrayLength);
REDISMODULE_GET_API(GetSelectedDb);
REDISMODULE_GET_API(SelectDb);
REDISMODULE_GET_API(OpenKey);
REDISMODULE_GET_API(CloseKey);
REDISMODULE_GET_API(KeyType);
REDISMODULE_GET_API(ValueLength);
REDISMODULE_GET_API(ListPush);
REDISMODULE_GET_API(ListPop);
REDISMODULE_GET_API(StringToLongLong);
REDISMODULE_GET_API(StringToDouble);
REDISMODULE_GET_API(Call);
REDISMODULE_GET_API(CallReplyProto);
REDISMODULE_GET_API(FreeCallReply);
REDISMODULE_GET_API(CallReplyInteger);
REDISMODULE_GET_API(CallReplyType);
REDISMODULE_GET_API(CallReplyLength);
REDISMODULE_GET_API(CallReplyArrayElement);
REDISMODULE_GET_API(CallReplyStringPtr);
REDISMODULE_GET_API(CreateStringFromCallReply);
REDISMODULE_GET_API(CreateString);
REDISMODULE_GET_API(CreateStringFromLongLong);
REDISMODULE_GET_API(CreateStringFromString);
REDISMODULE_GET_API(CreateStringPrintf);
REDISMODULE_GET_API(FreeString);
REDISMODULE_GET_API(StringPtrLen);
REDISMODULE_GET_API(AutoMemory);
REDISMODULE_GET_API(Replicate);
REDISMODULE_GET_API(ReplicateVerbatim);
REDISMODULE_GET_API(DeleteKey);
REDISMODULE_GET_API(UnlinkKey);
REDISMODULE_GET_API(StringSet);
REDISMODULE_GET_API(StringDMA);
REDISMODULE_GET_API(StringTruncate);
REDISMODULE_GET_API(GetExpire);
REDISMODULE_GET_API(SetExpire);
REDISMODULE_GET_API(ZsetAdd);
REDISMODULE_GET_API(ZsetIncrby);
REDISMODULE_GET_API(ZsetScore);
REDISMODULE_GET_API(ZsetRem);
REDISMODULE_GET_API(ZsetRangeStop);
REDISMODULE_GET_API(ZsetFirstInScoreRange);
REDISMODULE_GET_API(ZsetLastInScoreRange);
REDISMODULE_GET_API(ZsetFirstInLexRange);
REDISMODULE_GET_API(ZsetLastInLexRange);
REDISMODULE_GET_API(ZsetRangeCurrentElement);
REDISMODULE_GET_API(ZsetRangeNext);
REDISMODULE_GET_API(ZsetRangePrev);
REDISMODULE_GET_API(ZsetRangeEndReached);
REDISMODULE_GET_API(HashSet);
REDISMODULE_GET_API(HashGet);
REDISMODULE_GET_API(IsKeysPositionRequest);
REDISMODULE_GET_API(KeyAtPos);
REDISMODULE_GET_API(GetClientId);
REDISMODULE_GET_API(GetContextFlags);
REDISMODULE_GET_API(PoolAlloc);
REDISMODULE_GET_API(CreateDataType);
REDISMODULE_GET_API(ModuleTypeSetValue);
REDISMODULE_GET_API(ModuleTypeGetType);
REDISMODULE_GET_API(ModuleTypeGetValue);
REDISMODULE_GET_API(SaveUnsigned);
REDISMODULE_GET_API(LoadUnsigned);
REDISMODULE_GET_API(SaveSigned);
REDISMODULE_GET_API(LoadSigned);
REDISMODULE_GET_API(SaveString);
REDISMODULE_GET_API(SaveStringBuffer);
REDISMODULE_GET_API(LoadString);
REDISMODULE_GET_API(LoadStringBuffer);
REDISMODULE_GET_API(SaveDouble);
REDISMODULE_GET_API(LoadDouble);
REDISMODULE_GET_API(SaveFloat);
REDISMODULE_GET_API(LoadFloat);
REDISMODULE_GET_API(EmitAOF);
REDISMODULE_GET_API(Log);
REDISMODULE_GET_API(LogIOError);
REDISMODULE_GET_API(StringAppendBuffer);
REDISMODULE_GET_API(RetainString);
REDISMODULE_GET_API(StringCompare);
REDISMODULE_GET_API(GetContextFromIO);
REDISMODULE_GET_API(GetKeyNameFromIO);
REDISMODULE_GET_API(Milliseconds);
REDISMODULE_GET_API(DigestAddStringBuffer);
REDISMODULE_GET_API(DigestAddLongLong);
REDISMODULE_GET_API(DigestEndSequence);
REDISMODULE_GET_API(CreateDict);
REDISMODULE_GET_API(FreeDict);
REDISMODULE_GET_API(DictSize);
REDISMODULE_GET_API(DictSetC);
REDISMODULE_GET_API(DictReplaceC);
REDISMODULE_GET_API(DictSet);
REDISMODULE_GET_API(DictReplace);
REDISMODULE_GET_API(DictGetC);
REDISMODULE_GET_API(DictGet);
REDISMODULE_GET_API(DictDelC);
REDISMODULE_GET_API(DictDel);
REDISMODULE_GET_API(DictIteratorStartC);
REDISMODULE_GET_API(DictIteratorStart);
REDISMODULE_GET_API(DictIteratorStop);
REDISMODULE_GET_API(DictIteratorReseekC);
REDISMODULE_GET_API(DictIteratorReseek);
REDISMODULE_GET_API(DictNextC);
REDISMODULE_GET_API(DictPrevC);
REDISMODULE_GET_API(DictNext);
REDISMODULE_GET_API(DictPrev);
REDISMODULE_GET_API(DictCompare);
REDISMODULE_GET_API(DictCompareC);
#ifdef REDISMODULE_EXPERIMENTAL_API
REDISMODULE_GET_API(GetThreadSafeContext);
REDISMODULE_GET_API(FreeThreadSafeContext);
REDISMODULE_GET_API(ThreadSafeContextLock);
REDISMODULE_GET_API(ThreadSafeContextUnlock);
REDISMODULE_GET_API(BlockClient);
REDISMODULE_GET_API(UnblockClient);
REDISMODULE_GET_API(IsBlockedReplyRequest);
REDISMODULE_GET_API(IsBlockedTimeoutRequest);
REDISMODULE_GET_API(GetBlockedClientPrivateData);
REDISMODULE_GET_API(GetBlockedClientHandle);
REDISMODULE_GET_API(AbortBlock);
REDISMODULE_GET_API(SetDisconnectCallback);
REDISMODULE_GET_API(SubscribeToKeyspaceEvents);
REDISMODULE_GET_API(BlockedClientDisconnected);
REDISMODULE_GET_API(RegisterClusterMessageReceiver);
REDISMODULE_GET_API(SendClusterMessage);
REDISMODULE_GET_API(GetClusterNodeInfo);
REDISMODULE_GET_API(GetClusterNodesList);
REDISMODULE_GET_API(FreeClusterNodesList);
REDISMODULE_GET_API(CreateTimer);
REDISMODULE_GET_API(StopTimer);
REDISMODULE_GET_API(GetTimerInfo);
REDISMODULE_GET_API(GetMyClusterID);
REDISMODULE_GET_API(GetClusterSize);
REDISMODULE_GET_API(GetRandomBytes);
REDISMODULE_GET_API(GetRandomHexChars);
REDISMODULE_GET_API(SetClusterFlags);
REDISMODULE_GET_API(ExportSharedAPI);
REDISMODULE_GET_API(GetSharedAPI);
REDISMODULE_GET_API(RegisterCommandFilter);
REDISMODULE_GET_API(UnregisterCommandFilter);
REDISMODULE_GET_API(CommandFilterArgsCount);
REDISMODULE_GET_API(CommandFilterArgGet);
REDISMODULE_GET_API(CommandFilterArgInsert);
REDISMODULE_GET_API(CommandFilterArgReplace);
REDISMODULE_GET_API(CommandFilterArgDelete);
#endif
if (RedisModule_IsModuleNameBusy && RedisModule_IsModuleNameBusy(name)) return REDISMODULE_ERR;
RedisModule_SetModuleAttribs(ctx,name,ver,apiver);
return REDISMODULE_OK;
}
#else
/* Things only defined for the modules core, not exported to modules
* including this file. */
#define RedisModuleString robj
#endif /* REDISMODULE_CORE */
#endif /* REDISMOUDLE_H */
================================================
FILE: src/lib/zig-cuckoofilter.zig
================================================
const builtin = @import("builtin");
const std = @import("std");
const testing = std.testing;
const FREE_SLOT = 0;
// Use the provided function to re-seed the default PRNG implementation.
var xoro = std.rand.Xoroshiro128.init(42);
pub fn seed_default_prng(seed: u64) void {
xoro.seed(seed);
}
// If you want to read the state of the default PRNG impl:
pub fn get_default_prng_state() [2]u64 {
return xoro.s;
}
// If you want to set the state of the default PRNG impl:
pub fn set_default_prng_state(s: [2]u64) void {
xoro.s = s;
}
// Default PRNG implementation.
// By overriding .rand_fn you can provide your own custom PRNG implementation.
// Useful in adversarial situations (CSPRNG) or when you need deterministic behavior
// from the filter, as it requires to be able to save and restore the PRNG's state.
// You can use Filter<X>.RandomFn to see the function type you need to fulfill.
fn XoroRandFnImpl(comptime T: type) type {
return struct {
fn random() T {
return xoro.random.int(T);
}
};
}
// Supported CuckooFilter implementations.
// Bucket size is chosen mainly to keep the bucket 64bit word-sized, or under.
// This way reading a bucket requires a single memory fetch.
// Filter8 does not have 8-fp wide buckets to keep the error rate under 3%,
// as this is the cutoff point where Cuckoo Filters become more space-efficient
// than Bloom. We also enforce memory alignment.
pub const Filter8 = CuckooFilter(u8, 4);
pub const Filter16 = CuckooFilter(u16, 4);
pub const Filter32 = CuckooFilter(u32, 2);
fn CuckooFilter(comptime Tfp: type, comptime buckSize: usize) type {
return struct {
homeless_fp: Tfp,
homeless_bucket_idx: usize,
buckets: []align(Align) Bucket,
fpcount: usize,
broken: bool,
rand_fn: ?RandomFn,
pub const FPType = Tfp;
pub const Align = std.math.min(@alignOf(usize), @alignOf(@IntType(false, buckSize * @typeInfo(Tfp).Int.bits)));
pub const MaxError = 2.0 * @intToFloat(f32, buckSize) / @intToFloat(f32, 1 << @typeInfo(Tfp).Int.bits);
pub const RandomFn = fn () BucketSizeType;
const BucketSizeType = @IntType(false, comptime std.math.log2(buckSize));
const Bucket = [buckSize]Tfp;
const MinSize = @sizeOf(Tfp) * buckSize * 2;
const Self = @This();
const ScanMode = enum {
Set,
Force,
Delete,
Search,
};
pub fn size_for(min_capacity: usize) usize {
return size_for_exactly(min_capacity + @divTrunc(min_capacity, 5));
}
pub fn size_for_exactly(min_capacity: usize) usize {
var res = std.math.pow(usize, 2, std.math.log2(min_capacity));
if (res != min_capacity) res <<= 1;
const requested_size = res * @sizeOf(Tfp);
return if (MinSize > requested_size) MinSize else requested_size;
}
pub fn capacity(size: usize) usize {
return size / @sizeOf(Tfp);
}
// Use bytesToBuckets when you have persisted the filter and need to restore it.
// This will allow to cast back your bytes slice in the correct type for the .buckets
// property. Make sure you still have the right alignment when loading the data back!
pub fn bytesToBuckets(memory: []align(Align) u8) ![]align(Align) Bucket {
const not_pow2 = memory.len != std.math.pow(usize, 2, std.math.log2(memory.len));
if (not_pow2 or memory.len < MinSize) return error.BadLength;
return @bytesToSlice(Bucket, memory);
}
pub fn init(memory: []align(Align) u8) !Self {
for (memory) |*x| x.* = 0;
return Self{
.homeless_fp = FREE_SLOT,
.homeless_bucket_idx = undefined,
.buckets = try bytesToBuckets(memory),
.fpcount = 0,
.broken = false,
.rand_fn = null,
};
}
pub fn count(self: *Self) !usize {
return if (self.broken) error.Broken else self.fpcount;
}
pub fn maybe_contains(self: *Self, hash: u64, fingerprint: Tfp) !bool {
const fp = if (FREE_SLOT == fingerprint) 1 else fingerprint;
const bucket_idx = hash & (self.buckets.len - 1);
// Try primary bucket
if (fp == self.scan(bucket_idx, fp, .Search, FREE_SLOT)) return true;
// Try alt bucket
const alt_bucket_idx = self.compute_alt_bucket_idx(bucket_idx, fp);
if (fp == self.scan(alt_bucket_idx, fp, .Search, FREE_SLOT)) return true;
// Try homeless slot
return if (self.is_homeless_fp(bucket_idx, alt_bucket_idx, fp)) true else if (self.broken) error.Broken else false;
}
pub fn remove(self: *Self, hash: u64, fingerprint: Tfp) !void {
if (self.broken) return error.Broken;
const fp = if (FREE_SLOT == fingerprint) 1 else fingerprint;
const bucket_idx = hash & (self.buckets.len - 1);
// Try primary bucket
if (fp == self.scan(bucket_idx, fp, .Delete, FREE_SLOT)) {
self.fpcount -= 1;
return;
}
// Try alt bucket
const alt_bucket_idx = self.compute_alt_bucket_idx(bucket_idx, fp);
if (fp == self.scan(alt_bucket_idx, fp, .Delete, FREE_SLOT)) {
self.fpcount -= 1;
return;
}
// Try homeless slot
if (self.is_homeless_fp(bucket_idx, alt_bucket_idx, fp)) {
self.homeless_fp = FREE_SLOT;
self.fpcount -= 1;
return;
}
// Oh no...
self.broken = true;
return error.Broken;
}
pub fn add(self: *Self, hash: u64, fingerprint: Tfp) !void {
if (self.broken) return error.Broken;
const fp = if (FREE_SLOT == fingerprint) 1 else fingerprint;
const bucket_idx = hash & (self.buckets.len - 1);
// Try primary bucket
if (FREE_SLOT == self.scan(bucket_idx, FREE_SLOT, .Set, fp)) {
self.fpcount += 1;
return;
}
// If too tull already, try to add the fp to the secondary slot without forcing
const alt_bucket_idx = self.compute_alt_bucket_idx(bucket_idx, fp);
if (FREE_SLOT != self.homeless_fp) {
if (FREE_SLOT == self.scan(alt_bucket_idx, FREE_SLOT, .Set, fp)) {
self.fpcount += 1;
return;
} else return error.TooFull;
}
// We are now willing to force the insertion
self.homeless_bucket_idx = alt_bucket_idx;
self.homeless_fp = fp;
self.fpcount += 1;
var i: usize = 0;
while (i < 500) : (i += 1) {
self.homeless_bucket_idx = self.compute_alt_bucket_idx(self.homeless_bucket_idx, self.homeless_fp);
self.homeless_fp = self.scan(self.homeless_bucket_idx, FREE_SLOT, .Force, self.homeless_fp);
if (FREE_SLOT == self.homeless_fp) return;
}
// If we went over the while loop, now the homeless slot is occupied.
}
pub fn is_broken(self: *Self) bool {
return self.broken;
}
pub fn is_toofull(self: *Self) bool {
return FREE_SLOT != self.homeless_fp;
}
pub fn fix_toofull(self: *Self) !void {
if (FREE_SLOT == self.homeless_fp) return else {
const homeless_fp = self.homeless_fp;
self.homeless_fp = FREE_SLOT;
try self.add(self.homeless_bucket_idx, homeless_fp);
if (FREE_SLOT != self.homeless_fp) return error.TooFull;
}
}
inline fn is_homeless_fp(self: *Self, bucket_idx: usize, alt_bucket_idx: usize, fp: Tfp) bool {
const same_main = (self.homeless_bucket_idx == bucket_idx);
const same_alt = (self.homeless_bucket_idx == alt_bucket_idx);
const same_fp = (self.homeless_fp == fp);
return (same_fp and (same_main or same_alt));
}
inline fn compute_alt_bucket_idx(self: *Self, bucket_idx: usize, fp: Tfp) usize {
const fpSize = @sizeOf(Tfp);
const FNV_OFFSET = 14695981039346656037;
const FNV_PRIME = 1099511628211;
// Note: endianess
const bytes = @ptrCast(*const [fpSize]u8, &fp).*;
var res: usize = FNV_OFFSET;
comptime var i = 0;
inline while (i < fpSize) : (i += 1) {
res ^= bytes[i];
res *%= FNV_PRIME;
}
return (bucket_idx ^ res) & (self.buckets.len - 1);
}
inline fn scan(self: *Self, bucket_idx: u64, fp: Tfp, comptime mode: ScanMode, val: Tfp) Tfp {
comptime var i = 0;
// Search the bucket
var bucket = &self.buckets[bucket_idx];
inline while (i < buckSize) : (i += 1) {
if (bucket[i] == fp) {
switch (mode) {
.Search => {},
.Delete => bucket[i] = FREE_SLOT,
.Set => bucket[i] = val,
.Force => bucket[i] = val,
}
return fp;
}
}
switch (mode) {
.Search => return FREE_SLOT,
.Delete => return FREE_SLOT,
.Set => return 1,
.Force => {
// We did not find any free slot, so we must now evict.
const slot = if (self.rand_fn) |rfn| rfn() else XoroRandFnImpl(BucketSizeType).random();
const evicted = bucket[slot];
bucket[slot] = val;
return evicted;
},
}
}
};
}
test "Hx == (Hy XOR hash(fp))" {
var memory: [1 << 20]u8 align(Filter8.Align) = undefined;
var cf = Filter8.init(memory[0..]) catch unreachable;
testing.expect(0 == cf.compute_alt_bucket_idx(cf.compute_alt_bucket_idx(0, 'x'), 'x'));
testing.expect(1 == cf.compute_alt_bucket_idx(cf.compute_alt_bucket_idx(1, 'x'), 'x'));
testing.expect(42 == cf.compute_alt_bucket_idx(cf.compute_alt_bucket_idx(42, 'x'), 'x'));
testing.expect(500 == cf.compute_alt_bucket_idx(cf.compute_alt_bucket_idx(500, 'x'), 'x'));
testing.expect(5000 == cf.compute_alt_bucket_idx(cf.compute_alt_bucket_idx(5000, 'x'), 'x'));
testing.expect(10585 == cf.compute_alt_bucket_idx(cf.compute_alt_bucket_idx(10585, 'x'), 'x'));
testing.expect(10586 == cf.compute_alt_bucket_idx(cf.compute_alt_bucket_idx(10586, 'x'), 'x'));
testing.expect(18028 == cf.compute_alt_bucket_idx(cf.compute_alt_bucket_idx(18028, 'x'), 'x'));
testing.expect((1 << 15) - 1 == cf.compute_alt_bucket_idx(cf.compute_alt_bucket_idx((1 << 15) - 1, 'x'), 'x'));
}
fn test_not_broken(cf: var) void {
testing.expect(false == cf.maybe_contains(2, 'a') catch unreachable);
testing.expect(0 == cf.count() catch unreachable);
cf.add(2, 'a') catch unreachable;
testing.expect(cf.maybe_contains(2, 'a') catch unreachable);
testing.expect(false == cf.maybe_contains(0, 'a') catch unreachable);
testing.expect(false == cf.maybe_contains(1, 'a') catch unreachable);
testing.expect(1 == cf.count() catch unreachable);
cf.remove(2, 'a') catch unreachable;
testing.expect(false == cf.maybe_contains(2, 'a') catch unreachable);
testing.expect(0 == cf.count() catch unreachable);
}
test "is not completely broken" {
var memory: [16]u8 align(Filter8.Align) = undefined;
var cf = Filter8.init(memory[0..]) catch unreachable;
test_not_broken(&cf);
}
const Version = struct {
Tfp: type,
buckLen: usize,
cftype: type,
};
const SupportedVersions = []Version{
Version{ .Tfp = u8, .buckLen = 4, .cftype = Filter8 },
Version{ .Tfp = u16, .buckLen = 4, .cftype = Filter16 },
Version{ .Tfp = u32, .buckLen = 2, .cftype = Filter32 },
};
test "generics are not completely broken" {
inline for (SupportedVersions) |v| {
var memory: [1024]u8 align(v.cftype.Align) = undefined;
var cf = v.cftype.init(memory[0..]) catch unreachable;
test_not_broken(&cf);
}
}
test "too full when adding too many copies" {
inline for (SupportedVersions) |v| {
var memory: [1024]u8 align(v.cftype.Align) = undefined;
var cf = v.cftype.init(memory[0..]) catch unreachable;
var i: usize = 0;
while (i < v.buckLen * 2) : (i += 1) {
cf.add(0, 1) catch unreachable;
}
testing.expect(!cf.is_toofull());
// The first time we go over-board we can still occupy
// the homeless slot, so this won't fail:
cf.add(0, 1) catch unreachable;
testing.expect(cf.is_toofull());
// We now are really full.
testing.expectError(error.TooFull, cf.add(0, 1));
testing.expect(cf.is_toofull());
testing.expectError(error.TooFull, cf.add(0, 1));
testing.expect(cf.is_toofull());
testing.expectError(error.TooFull, cf.add(0, 1));
testing.expect(cf.is_toofull());
i = 0;
while (i < v.buckLen * 2) : (i += 1) {
cf.add(2, 1) catch unreachable;
}
// Homeless slot is already occupied.
testing.expectError(error.TooFull, cf.add(2, 1));
testing.expectError(error.TooFull, cf.add(2, 1));
testing.expectError(error.TooFull, cf.add(2, 1));
// Try to fix the situation
testing.expect(cf.is_toofull());
// This should fail
testing.expectError(error.TooFull, cf.fix_toofull());
// Make it fixable
cf.remove(0, 1) catch unreachable;
cf.fix_toofull() catch unreachable;
testing.expect(!cf.is_toofull());
cf.add(2, 1) catch unreachable;
testing.expect(cf.is_toofull());
// Delete now all instances except the homeless one
i = 0;
while (i < v.buckLen * 2) : (i += 1) {
cf.remove(2, 1) catch unreachable;
}
// Should be able to find the homeless fp
testing.expect(cf.maybe_contains(2, 1) catch unreachable);
// Delete it and try to find it
cf.remove(2, 1) catch unreachable;
testing.expect(false == cf.maybe_contains(2, 1) catch unreachable);
}
}
test "properly breaks when misused" {
inline for (SupportedVersions) |v| {
var memory: [1024]u8 align(v.cftype.Align) = undefined;
var cf = v.cftype.init(memory[0..]) catch unreachable;
var fp = @intCast(v.Tfp, 1);
testing.expectError(error.Broken, cf.remove(2, 1));
testing.expectError(error.Broken, cf.add(2, 1));
testing.expectError(error.Broken, cf.count());
testing.expectError(error.Broken, cf.maybe_contains(2, 1));
}
}
fn TestSet(comptime Tfp: type) type {
const ItemSet = std.hash_map.AutoHashMap(u64, Tfp);
return struct {
items: ItemSet,
false_positives: ItemSet,
const Self = @This();
fn init(iterations: usize, false_positives: usize, allocator: *std.mem.Allocator) Self {
var item_set = ItemSet.init(allocator);
var false_set = ItemSet.init(allocator);
return Self{
.items = blk: {
var i: usize = 0;
while (i < iterations) : (i += 1) {
var hash = xoro.random.int(u64);
while (item_set.contains(hash)) {
hash = xoro.random.int(u64);
}
_ = item_set.put(hash, xoro.random.int(Tfp)) catch unreachable;
}
break :blk item_set;
},
.false_positives = blk: {
var i: usize = 0;
while (i < false_positives) : (i += 1) {
var hash = xoro.random.int(u64);
while (item_set.contains(hash) or false_set.contains(hash)) {
hash = xoro.random.int(u64);
}
_ = false_set.put(hash, xoro.random.int(Tfp)) catch unreachable;
}
break :blk false_set;
},
};
}
};
}
test "small stress test" {
const iterations = 60000;
const false_positives = 10000;
var direct_allocator = std.heap.DirectAllocator.init();
//defer direct_allocator.deinit();
inline for (SupportedVersions) |v| {
var test_cases = TestSet(v.Tfp).init(iterations, false_positives, &direct_allocator.allocator);
var iit = test_cases.items.iterator();
var fit = test_cases.false_positives.iterator();
//defer test_cases.items.deinit();
//defer test_cases.false_positives.deinit();
// Build an appropriately-sized filter
var memory: [v.cftype.size_for(iterations)]u8 align(v.cftype.Align) = undefined;
var cf = v.cftype.init(memory[0..]) catch unreachable;
// Test all items for presence (should all be false)
{
iit.reset();
while (iit.next()) |item| {
testing.expect(!(cf.maybe_contains(item.key, item.value) catch unreachable));
}
}
// Add all items (should not fail)
{
iit.reset();
var iters: usize = 0;
while (iit.next()) |item| {
testing.expect(iters == cf.count() catch unreachable);
cf.add(item.key, item.value) catch unreachable;
iters += 1;
}
testing.expect(iters == cf.count() catch unreachable);
}
// Test that memory contains the right number of elements
{
var count: usize = 0;
for (@bytesToSlice(v.Tfp, memory)) |byte| {
if (byte != 0) {
count += 1;
}
}
testing.expect(iterations == count);
testing.expect(iterations == cf.count() catch unreachable);
}
// Test all items for presence (should all be true)
{
iit.reset();
while (iit.next()) |item| {
testing.expect(cf.maybe_contains(item.key, item.value) catch unreachable);
}
}
// Delete half the elements and ensure they are not found
// (there could be false positives depending on fill lvl)
{
iit.reset();
const max = @divTrunc(iterations, 2);
var count: usize = 0;
var false_count: usize = 0;
while (iit.next()) |item| {
count += 1;
if (count >= max) break;
testing.expect(cf.maybe_contains(item.key, item.value) catch unreachable);
cf.remove(item.key, item.value) catch unreachable;
testing.expect(iterations - count == cf.count() catch unreachable);
if (cf.maybe_contains(item.key, item.value) catch unreachable) false_count += 1;
}
testing.expect(false_count < @divTrunc(iterations, 40)); // < 2.5%
iit.reset();
count = 0;
false_count = 0;
while (iit.next()) |item| {
count += 1;
if (count >= max) break;
if (cf.maybe_contains(item.key, item.value) catch unreachable) false_count += 1;
}
testing.expect(false_count < @divTrunc(iterations, 40)); // < 2.5%
}
// Test false positive elements
{
fit.reset();
var false_count: usize = 0;
while (fit.next()) |item| {
if (cf.maybe_contains(item.key, item.value) catch unreachable) false_count += 1;
}
testing.expect(false_count < @divTrunc(iterations, 40)); // < 2.5%
}
// Add deleted elements back in and test that all are present
{
iit.reset();
const max = @divTrunc(iterations, 2);
var count: usize = 0;
var false_count: usize = 0;
while (iit.next()) |item| {
count += 1;
if (count >= max) break;
cf.add(item.key, item.value) catch unreachable;
testing.expect(cf.maybe_contains(item.key, item.value) catch unreachable);
}
}
// Test false positive elements (again)
{
fit.reset();
var false_count: usize = 0;
while (fit.next()) |item| {
if (cf.maybe_contains(item.key, item.value) catch unreachable) false_count += 1;
}
testing.expect(false_count < @divTrunc(iterations, 40)); // < 2.5%
}
// Delete all items
{
iit.reset();
var iters: usize = 0;
while (iit.next()) |item| {
cf.remove(item.key, item.value) catch unreachable;
iters += 1;
}
}
// Test that memory contains 0 elements
{
var count: usize = 0;
for (@bytesToSlice(v.Tfp, memory)) |fprint| {
if (fprint != 0) {
count += 1;
}
}
testing.expect(0 == count);
testing.expect(0 == cf.count() catch unreachable);
}
// Test all items for presence (should all be false)
{
iit.reset();
while (iit.next()) |item| {
testing.expect(!(cf.maybe_contains(item.key, item.value) catch unreachable));
}
}
}
}
================================================
FILE: src/redis-cuckoofilter.zig
================================================
const builtin = @import("builtin");
const std = @import("std");
const mem = std.mem;
const redis = @import("./redismodule.zig");
const cuckoo = @import("./lib/zig-cuckoofilter.zig");
const t_ccf = @import("./t_cuckoofilter.zig");
// We save the initial state of Xoroshiro seeded at 42 at compile-time,
// used to initialize the prng state for each new cuckoofilter key.
// This way we are able to provide fully deterministic behavior.
const XoroDefaultState: [2]u64 = comptime std.rand.Xoroshiro128.init(42).s;
// Compares two strings ignoring case (ascii strings, not fancy unicode strings).
// Used by commands to check if a given flag (e.g. NX, EXACT, ...) was given as an arugment.
// Specialzied version where one string is comptime known (and all uppercase).
inline fn insensitive_eql(comptime uppercase: []const u8, str: []const u8) bool {
comptime {
var i = 0;
while (i < uppercase.len) : (i += 1) {
if (uppercase[i] >= 'a' and uppercase[i] <= 'z') {
@compileError("`insensitive_eql` requires the first argument to be all uppercase");
}
}
}
if (uppercase.len != str.len) return false;
if (uppercase.len < 10) {
comptime var i = 0;
inline while (i < uppercase.len) : (i += 1) {
const val = if (str[i] >= 'a' and str[i] <= 'z') str[i] - 32 else str[i];
if (val != uppercase[i]) return false;
}
} else {
var i = 0;
while (i < uppercase.len) : (i += 1) {
const val = if (str[i] >= 'a' and str[i] <= 'z') str[i] - 32 else str[i];
if (val != uppercase[i]) return false;
}
}
return true;
}
// Given a byte count, returns the equivalent string size (e.g. 1024 -> "1K")
// Specialized version for some powers of 2
fn size2str(size: usize, buf: *[5]u8) ![]u8 {
var pow_1024: usize = 0;
var num = size;
while (num >= 1024) {
num = try std.math.divExact(usize, num, 1024);
pow_1024 += 1;
}
var letter: u8 = undefined;
switch (pow_1024) {
0 => return error.TooSmall,
1 => letter = 'K',
2 => letter = 'M',
3 => letter = 'G',
else => return error.TooBig,
}
// We want to stop at 8G
if (pow_1024 == 3 and num > 8) return error.TooBig;
return switch (num) {
1, 2, 4, 8, 16, 32, 64, 128, 256, 512 => try std.fmt.bufPrint(buf, "{}{c}\x00", num, letter),
else => error.Error,
};
}
// Given a size, returns the equivalent byte count (e.g. "1K" -> 1024)
// Specialized version for some powers of 2
fn str2size(str: []const u8) !usize {
if (str.len < 2 or str.len > 4) return error.Error;
var pow_1024: usize = undefined;
switch (str[str.len - 1]) {
'k', 'K' => pow_1024 = 1024,
'm', 'M' => pow_1024 = 1024 * 1024,
'g', 'G' => pow_1024 = 1024 * 1024 * 1024,
else => return error.Error,
}
// Parse the numeric part
const num: usize = try std.fmt.parseInt(usize, str[0..(str.len - 1)], 10);
// We want to stop at 8G
if (pow_1024 == 3 and num > 8) return error.Error;
return switch (num) {
1, 2, 4, 8, 16, 32, 64, 128, 256, 512 => try std.math.mul(usize, num, pow_1024),
else => error.Error,
};
}
// Parses hash and fp values, used by most commands.
inline fn parse_args(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int, hash: *u64, fp: *u32) !void {
if (argc != 4) {
_ = redis.RedisModule_WrongArity.?(ctx);
return error.Error;
}
// Parse hash as u64
var hash_len: usize = undefined;
var hash_str = redis.RedisModule_StringPtrLen.?(argv[2], &hash_len);
var hash_start: usize = 0;
if (hash_len > 0 and hash_str[0] == '-') {
// Shift forward by 1 to skip the negative sign
hash_start = 1;
}
hash.* = std.fmt.parseInt(u64, hash_str[hash_start..hash_len], 10) catch |err| {
_ = switch (err) {
error.Overflow => redis.RedisModule_ReplyWithError.?(ctx, c"ERR hash overflows u64"),
error.InvalidCharacter => redis.RedisModule_ReplyWithError.?(ctx, c"ERR hash contains bad character"),
};
return error.Error;
};
// The hash was a negative number
if (hash_start == 1) {
hash.* = std.math.sub(u64, std.math.maxInt(u64), hash.*) catch {
_ = redis.RedisModule_ReplyWithError.?(ctx, c"ERR hash underflows u64");
return error.Error;
};
}
// Parse fp as u32
var fp_len: usize = undefined;
var fp_str = redis.RedisModule_StringPtrLen.?(argv[3], &fp_len);
var fp_start: usize = 0;
if (fp_len > 0 and fp_str[0] == '-') {
// Shift forward by 1 to skip the negative sign
fp_start = 1;
}
fp.* = @bitCast(u32, std.fmt.parseInt(i32, fp_str[fp_start..fp_len], 10) catch |err| {
_ = switch (err) {
error.Overflow => redis.RedisModule_ReplyWithError.?(ctx, c"ERR fp overflows u32"),
error.InvalidCharacter => redis.RedisModule_ReplyWithError.?(ctx, c"ERR fp contains bad character"),
};
return error.Error;
});
// The fp was a negative number
if (fp_start == 1) {
fp.* = std.math.sub(u32, std.math.maxInt(u32), fp.*) catch {
_ = redis.RedisModule_ReplyWithError.?(ctx, c"ERR fp underflows u32");
return error.Error;
};
}
}
// Registers the module and its commands.
export fn RedisModule_OnLoad(ctx: *redis.RedisModuleCtx, argv: [*c]*redis.RedisModuleString, argc: c_int) c_int {
if (redis.RedisModule_Init(ctx, c"cuckoofilter", 1, redis.REDISMODULE_APIVER_1) == redis.REDISMODULE_ERR) {
return redis.REDISMODULE_ERR;
}
// Register our custom types
t_ccf.RegisterTypes(ctx) catch return redis.REDISMODULE_ERR;
// Register our commands
registerCommand(ctx, c"cf.init", CF_INIT, c"write deny-oom", 1, 1, 1) catch return redis.REDISMODULE_ERR;
registerCommand(ctx, c"cf.rem", CF_REM, c"write fast", 1, 1, 1) catch return redis.REDISMODULE_ERR;
registerCommand(ctx, c"cf.add", CF_ADD, c"write fast", 1, 1, 1) catch return redis.REDISMODULE_ERR;
registerCommand(ctx, c"cf.fixtoofull", CF_FIXTOOFULL, c"write fast", 1, 1, 1) catch return redis.REDISMODULE_ERR;
registerCommand(ctx, c"cf.check", CF_CHECK, c"readonly fast", 1, 1, 1) catch return redis.REDISMODULE_ERR;
registerCommand(ctx, c"cf.count", CF_COUNT, c"readonly fast", 1, 1, 1) catch return redis.REDISMODULE_ERR;
registerCommand(ctx, c"cf.isbroken", CF_ISBROKEN, c"readonly fast", 1, 1, 1) catch return redis.REDISMODULE_ERR;
registerCommand(ctx, c"cf.istoofull", CF_ISTOOFULL, c"readonly fast", 1, 1, 1) catch return redis.REDISMODULE_ERR;
registerCommand(ctx, c"cf.capacity", CF_CAPACITY, c"fast allow-loading allow-stale", 0, 0, 0) catch return redis.REDISMODULE_ERR;
registerCommand(ctx, c"cf.sizefor", CF_SIZEFOR, c"fast allow-loading allow-stale", 0, 0, 0) catch return redis.REDISMODULE_ERR;
return redis.REDISMODULE_OK;
}
inline fn registerCommand(ctx: *redis.RedisModuleCtx, cmd: [*c]const u8, func: redis.RedisModuleCmdFunc, mode: [*c]const u8, firstkey: c_int, lastkey: c_int, keystep: c_int) !void {
const err = redis.RedisModule_CreateCommand.?(ctx, cmd, func, mode, firstkey, lastkey, keystep);
if (err == redis.REDISMODULE_ERR) return error.Error;
}
// CF.INIT size [fpsize]
export fn CF_INIT(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
if (argc != 3 and argc != 4) return redis.RedisModule_WrongArity.?(ctx);
// size argument
var size_len: usize = undefined;
const size_str = redis.RedisModule_StringPtrLen.?(argv[2], &size_len)[0..size_len];
const size = str2size(size_str) catch return redis.RedisModule_ReplyWithError.?(ctx, c"ERR bad size");
// fpsize argument
var fp_size = "1"[0..];
if (argc == 4) {
var fp_size_len: usize = undefined;
fp_size = redis.RedisModule_StringPtrLen.?(argv[3], &fp_size_len)[0..fp_size_len];
}
// Obtain the key from Redis.
var key = @ptrCast(?*redis.RedisModuleKey, redis.RedisModule_OpenKey.?(ctx, argv[1], redis.REDISMODULE_READ | redis.REDISMODULE_WRITE));
defer redis.RedisModule_CloseKey.?(key);
var keyType = redis.RedisModule_KeyType.?(key);
if (keyType != redis.REDISMODULE_KEYTYPE_EMPTY) return redis.RedisModule_ReplyWithError.?(ctx, c"ERR key already exists");
// New Cuckoo Filter!
if (fp_size.len != 1) return redis.RedisModule_ReplyWithError.?(ctx, c"ERR bad fpsize");
return switch (fp_size[0]) {
'1' => do_init(t_ccf.Filter8, ctx, key, size),
'2' => do_init(t_ccf.Filter16, ctx, key, size),
'4' => do_init(t_ccf.Filter32, ctx, key, size),
else => return redis.RedisModule_ReplyWithError.?(ctx, c"ERR bad fpsize"),
};
}
inline fn do_init(comptime CFType: type, ctx: ?*redis.RedisModuleCtx, key: ?*redis.RedisModuleKey, size: usize) c_int {
var cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), redis.RedisModule_Alloc.?(@sizeOf(CFType))));
var buckets = @ptrCast([*]u8, @alignCast(@alignOf(usize), redis.RedisModule_Alloc.?(size)));
const realCFType = @typeOf(cf.cf);
cf.s = XoroDefaultState;
cf.cf = realCFType.init(buckets[0..size]) catch return redis.RedisModule_ReplyWithError.?(ctx, c"ERR could not create filter");
switch (CFType) {
t_ccf.Filter8 => _ = redis.RedisModule_ModuleTypeSetValue.?(key, t_ccf.Type8, cf),
t_ccf.Filter16 => _ = redis.RedisModule_ModuleTypeSetValue.?(key, t_ccf.Type16, cf),
t_ccf.Filter32 => _ = redis.RedisModule_ModuleTypeSetValue.?(key, t_ccf.Type32, cf),
else => unreachable,
}
_ = redis.RedisModule_ReplicateVerbatim.?(ctx);
return redis.RedisModule_ReplyWithSimpleString.?(ctx, c"OK");
}
// CF.ADD key hash fp
export fn CF_ADD(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
var hash: u64 = undefined;
var fp: u32 = undefined;
parse_args(ctx, argv, argc, &hash, &fp) catch return redis.REDISMODULE_OK;
var key = @ptrCast(?*redis.RedisModuleKey, redis.RedisModule_OpenKey.?(ctx, argv[1], redis.REDISMODULE_READ | redis.REDISMODULE_WRITE));
defer redis.RedisModule_CloseKey.?(key);
if (redis.RedisModule_KeyType.?(key) == redis.REDISMODULE_KEYTYPE_EMPTY)
return redis.RedisModule_ReplyWithError.?(ctx, c"ERR key does not exist");
const keyType = redis.RedisModule_ModuleTypeGetType.?(key);
return if (keyType == t_ccf.Type8) do_add(t_ccf.Filter8, ctx, key, hash, fp) else if (keyType == t_ccf.Type16) do_add(t_ccf.Filter16, ctx, key, hash, fp) else if (keyType == t_ccf.Type32) do_add(t_ccf.Filter32, ctx, key, hash, fp) else redis.RedisModule_ReplyWithError.?(ctx, redis.REDISMODULE_ERRORMSG_WRONGTYPE);
}
inline fn do_add(comptime CFType: type, ctx: ?*redis.RedisModuleCtx, key: ?*redis.RedisModuleKey, hash: u64, fp: u32) c_int {
const cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), redis.RedisModule_ModuleTypeGetValue.?(key)));
const realCFType = @typeOf(cf.cf);
cuckoo.set_default_prng_state(cf.s);
defer {
cf.s = cuckoo.get_default_prng_state();
}
_ = redis.RedisModule_ReplicateVerbatim.?(ctx);
return if (cf.cf.add(hash, @truncate(realCFType.FPType, fp)))
redis.RedisModule_ReplyWithSimpleString.?(ctx, c"OK")
else |err| switch (err) {
error.Broken => redis.RedisModule_ReplyWithError.?(ctx, c"ERR filter is broken"),
error.TooFull => redis.RedisModule_ReplyWithError.?(ctx, c"ERR too full"),
};
}
// CF.CHECK key hash fp
export fn CF_CHECK(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
var hash: u64 = undefined;
var fp: u32 = undefined;
parse_args(ctx, argv, argc, &hash, &fp) catch return redis.REDISMODULE_OK;
var key = @ptrCast(?*redis.RedisModuleKey, redis.RedisModule_OpenKey.?(ctx, argv[1], redis.REDISMODULE_READ | redis.REDISMODULE_WRITE));
defer redis.RedisModule_CloseKey.?(key);
if (redis.RedisModule_KeyType.?(key) == redis.REDISMODULE_KEYTYPE_EMPTY)
return redis.RedisModule_ReplyWithError.?(ctx, c"ERR key does not exist");
const keyType = redis.RedisModule_ModuleTypeGetType.?(key);
return if (keyType == t_ccf.Type8) do_check(t_ccf.Filter8, ctx, key, hash, fp) else if (keyType == t_ccf.Type16) do_check(t_ccf.Filter16, ctx, key, hash, fp) else if (keyType == t_ccf.Type32) do_check(t_ccf.Filter32, ctx, key, hash, fp) else redis.RedisModule_ReplyWithError.?(ctx, redis.REDISMODULE_ERRORMSG_WRONGTYPE);
}
inline fn do_check(comptime CFType: type, ctx: ?*redis.RedisModuleCtx, key: ?*redis.RedisModuleKey, hash: u64, fp: u32) c_int {
const cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), redis.RedisModule_ModuleTypeGetValue.?(key)));
const realCFType = @typeOf(cf.cf);
return if (cf.cf.maybe_contains(hash, @truncate(realCFType.FPType, fp))) |maybe_found|
redis.RedisModule_ReplyWithSimpleString.?(ctx, if (maybe_found) c"1" else c"0")
else |err| switch (err) {
error.Broken => redis.RedisModule_ReplyWithError.?(ctx, c"ERR filter is broken"),
};
}
// CF.REM key hash fp
export fn CF_REM(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
var hash: u64 = undefined;
var fp: u32 = undefined;
parse_args(ctx, argv, argc, &hash, &fp) catch return redis.REDISMODULE_OK;
var key = @ptrCast(?*redis.RedisModuleKey, redis.RedisModule_OpenKey.?(ctx, argv[1], redis.REDISMODULE_READ | redis.REDISMODULE_WRITE));
defer redis.RedisModule_CloseKey.?(key);
if (redis.RedisModule_KeyType.?(key) == redis.REDISMODULE_KEYTYPE_EMPTY)
return redis.RedisModule_ReplyWithError.?(ctx, c"ERR key does not exist");
const keyType = redis.RedisModule_ModuleTypeGetType.?(key);
return if (keyType == t_ccf.Type8) do_rem(t_ccf.Filter8, ctx, key, hash, fp) else if (keyType == t_ccf.Type16) do_rem(t_ccf.Filter16, ctx, key, hash, fp) else if (keyType == t_ccf.Type32) do_rem(t_ccf.Filter32, ctx, key, hash, fp) else redis.RedisModule_ReplyWithError.?(ctx, redis.REDISMODULE_ERRORMSG_WRONGTYPE);
}
inline fn do_rem(comptime CFType: type, ctx: ?*redis.RedisModuleCtx, key: ?*redis.RedisModuleKey, hash: u64, fp: u32) c_int {
const cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), redis.RedisModule_ModuleTypeGetValue.?(key)));
const realCFType = @typeOf(cf.cf);
_ = redis.RedisModule_ReplicateVerbatim.?(ctx);
return if (cf.cf.remove(hash, @truncate(realCFType.FPType, fp)))
redis.RedisModule_ReplyWithSimpleString.?(ctx, c"OK")
else |err| switch (err) {
error.Broken => redis.RedisModule_ReplyWithError.?(ctx, c"ERR filter is broken"),
};
}
// CF.FIXTOOFULL key
export fn CF_FIXTOOFULL(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
if (argc != 2) return redis.RedisModule_WrongArity.?(ctx);
var key = @ptrCast(?*redis.RedisModuleKey, redis.RedisModule_OpenKey.?(ctx, argv[1], redis.REDISMODULE_READ | redis.REDISMODULE_WRITE));
defer redis.RedisModule_CloseKey.?(key);
if (redis.RedisModule_KeyType.?(key) == redis.REDISMODULE_KEYTYPE_EMPTY)
return redis.RedisModule_ReplyWithError.?(ctx, c"ERR key does not exist");
const keyType = redis.RedisModule_ModuleTypeGetType.?(key);
return if (keyType == t_ccf.Type8) do_fixtoofull(t_ccf.Filter8, ctx, key) else if (keyType == t_ccf.Type16) do_fixtoofull(t_ccf.Filter16, ctx, key) else if (keyType == t_ccf.Type32) do_fixtoofull(t_ccf.Filter32, ctx, key) else redis.RedisModule_ReplyWithError.?(ctx, redis.REDISMODULE_ERRORMSG_WRONGTYPE);
}
inline fn do_fixtoofull(comptime CFType: type, ctx: ?*redis.RedisModuleCtx, key: ?*redis.RedisModuleKey) c_int {
const cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), redis.RedisModule_ModuleTypeGetValue.?(key)));
const realCFType = @typeOf(cf.cf);
cuckoo.set_default_prng_state(cf.s);
defer {
cf.s = cuckoo.get_default_prng_state();
}
_ = redis.RedisModule_ReplicateVerbatim.?(ctx);
return if (cf.cf.fix_toofull())
redis.RedisModule_ReplyWithSimpleString.?(ctx, c"OK")
else |err| switch (err) {
error.Broken => redis.RedisModule_ReplyWithError.?(ctx, c"ERR filter is broken"),
error.TooFull => redis.RedisModule_ReplyWithError.?(ctx, c"ERR too full"),
};
}
// CF.COUNT key
export fn CF_COUNT(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
if (argc != 2) return redis.RedisModule_WrongArity.?(ctx);
var key = @ptrCast(?*redis.RedisModuleKey, redis.RedisModule_OpenKey.?(ctx, argv[1], redis.REDISMODULE_READ | redis.REDISMODULE_WRITE));
defer redis.RedisModule_CloseKey.?(key);
if (redis.RedisModule_KeyType.?(key) == redis.REDISMODULE_KEYTYPE_EMPTY)
return redis.RedisModule_ReplyWithError.?(ctx, c"ERR key does not exist");
const keyType = redis.RedisModule_ModuleTypeGetType.?(key);
return if (keyType == t_ccf.Type8) do_count(t_ccf.Filter8, ctx, key) else if (keyType == t_ccf.Type16) do_count(t_ccf.Filter16, ctx, key) else if (keyType == t_ccf.Type32) do_count(t_ccf.Filter32, ctx, key) else redis.RedisModule_ReplyWithError.?(ctx, redis.REDISMODULE_ERRORMSG_WRONGTYPE);
}
inline fn do_count(comptime CFType: type, ctx: ?*redis.RedisModuleCtx, key: ?*redis.RedisModuleKey) c_int {
const cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), redis.RedisModule_ModuleTypeGetValue.?(key)));
return if (cf.cf.count()) |count|
redis.RedisModule_ReplyWithLongLong.?(ctx, @intCast(c_longlong, count))
else |err| switch (err) {
error.Broken => redis.RedisModule_ReplyWithError.?(ctx, c"ERR filter is broken"),
};
}
// CF.ISTOOFULL key
export fn CF_ISTOOFULL(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
if (argc != 2) return redis.RedisModule_WrongArity.?(ctx);
var key = @ptrCast(?*redis.RedisModuleKey, redis.RedisModule_OpenKey.?(ctx, argv[1], redis.REDISMODULE_READ | redis.REDISMODULE_WRITE));
defer redis.RedisModule_CloseKey.?(key);
if (redis.RedisModule_KeyType.?(key) == redis.REDISMODULE_KEYTYPE_EMPTY)
return redis.RedisModule_ReplyWithError.?(ctx, c"ERR key does not exist");
const keyType = redis.RedisModule_ModuleTypeGetType.?(key);
return if (keyType == t_ccf.Type8) do_istoofull(t_ccf.Filter8, ctx, key) else if (keyType == t_ccf.Type16) do_istoofull(t_ccf.Filter16, ctx, key) else if (keyType == t_ccf.Type32) do_istoofull(t_ccf.Filter32, ctx, key) else redis.RedisModule_ReplyWithError.?(ctx, redis.REDISMODULE_ERRORMSG_WRONGTYPE);
}
inline fn do_istoofull(comptime CFType: type, ctx: ?*redis.RedisModuleCtx, key: ?*redis.RedisModuleKey) c_int {
const cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), redis.RedisModule_ModuleTypeGetValue.?(key)));
return redis.RedisModule_ReplyWithSimpleString.?(ctx, if (cf.cf.is_toofull()) c"1" else c"0");
}
// CF.ISBROKEN key
export fn CF_ISBROKEN(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
if (argc != 2) return redis.RedisModule_WrongArity.?(ctx);
var key = @ptrCast(?*redis.RedisModuleKey, redis.RedisModule_OpenKey.?(ctx, argv[1], redis.REDISMODULE_READ | redis.REDISMODULE_WRITE));
defer redis.RedisModule_CloseKey.?(key);
if (redis.RedisModule_KeyType.?(key) == redis.REDISMODULE_KEYTYPE_EMPTY)
return redis.RedisModule_ReplyWithError.?(ctx, c"ERR key does not exist");
const keyType = redis.RedisModule_ModuleTypeGetType.?(key);
return if (keyType == t_ccf.Type8) do_isbroken(t_ccf.Filter8, ctx, key) else if (keyType == t_ccf.Type16) do_isbroken(t_ccf.Filter16, ctx, key) else if (keyType == t_ccf.Type32) do_isbroken(t_ccf.Filter32, ctx, key) else redis.RedisModule_ReplyWithError.?(ctx, redis.REDISMODULE_ERRORMSG_WRONGTYPE);
}
inline fn do_isbroken(comptime CFType: type, ctx: ?*redis.RedisModuleCtx, key: ?*redis.RedisModuleKey) c_int {
const cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), redis.RedisModule_ModuleTypeGetValue.?(key)));
return redis.RedisModule_ReplyWithSimpleString.?(ctx, if (cf.cf.is_broken()) c"1" else c"0");
}
// CF.CAPACITY size [fpsize]
export fn CF_CAPACITY(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
if (argc != 2 and argc != 3) return redis.RedisModule_WrongArity.?(ctx);
// SIZE argument
var size_len: usize = undefined;
const size_str = redis.RedisModule_StringPtrLen.?(argv[1], &size_len)[0..size_len];
// FPSIZE argument
var fp_size = "1"[0..];
if (argc == 3) {
var fpsize_len: usize = undefined;
fp_size = redis.RedisModule_StringPtrLen.?(argv[2], &fpsize_len)[0..fpsize_len];
}
const size = str2size(size_str) catch return redis.RedisModule_ReplyWithError.?(ctx, c"ERR bad size");
if (fp_size.len != 1) return redis.RedisModule_ReplyWithError.?(ctx, c"ERR bad fpsize");
return switch (fp_size[0]) {
'1' => redis.RedisModule_ReplyWithLongLong.?(ctx, @intCast(c_longlong, cuckoo.Filter8.capacity(size))),
'2' => redis.RedisModule_ReplyWithLongLong.?(ctx, @intCast(c_longlong, cuckoo.Filter16.capacity(size))),
'4' => redis.RedisModule_ReplyWithLongLong.?(ctx, @intCast(c_longlong, cuckoo.Filter32.capacity(size))),
else => redis.RedisModule_ReplyWithError.?(ctx, c"ERR bad fpsize"),
};
}
// CF.SIZEFOR universe [fpsize] [EXACT]
export fn CF_SIZEFOR(ctx: ?*redis.RedisModuleCtx, argv: [*c]?*redis.RedisModuleString, argc: c_int) c_int {
if (argc != 2 and argc != 3 and argc != 4) return redis.RedisModule_WrongArity.?(ctx);
// Parse universe
var uni_len: usize = undefined;
var uni_str = redis.RedisModule_StringPtrLen.?(argv[1], &uni_len);
const universe = std.fmt.parseInt(usize, uni_str[0..uni_len], 10) catch |err| return switch (err) {
error.Overflow => redis.RedisModule_ReplyWithError.?(ctx, c"ERR universe overflows usize"),
error.InvalidCharacter => redis.RedisModule_ReplyWithError.?(ctx, c"ERR universe contains bad character"),
};
// Parse fpsize and EXACT
var fp_size = "1"[0..];
var exact = false;
if (argc > 2) {
var arg2len: usize = undefined;
const arg2 = redis.RedisModule_StringPtrLen.?(argv[2], &arg2len)[0..arg2len];
if (insensitive_eql("EXACT", arg2)) exact = true else fp_size = arg2;
}
if (argc == 4) {
if (exact) return redis.RedisModule_ReplyWithError.?(ctx, c"ERR bad fpsize");
var arg3len: usize = undefined;
const arg3 = redis.RedisModule_StringPtrLen.?(argv[3], &arg3len)[0..arg3len];
if (insensitive_eql("EXACT", arg3)) exact = true else return redis.RedisModule_WrongArity.?(ctx);
}
if (fp_size.len != 1) return redis.RedisModule_ReplyWithError.?(ctx, c"ERR bad fpsize");
return switch (fp_size[0]) {
'1' => do_sizefor(cuckoo.Filter8, ctx, universe, exact),
'2' => do_sizefor(cuckoo.Filter16, ctx, universe, exact),
'4' => do_sizefor(cuckoo.Filter32, ctx, universe, exact),
else => redis.RedisModule_ReplyWithError.?(ctx, c"ERR bad fpsize"),
};
}
fn do_sizefor(comptime CFType: type, ctx: ?*redis.RedisModuleCtx, universe: usize, exact: bool) c_int {
var buf: [5]u8 = undefined;
const size = if (exact) CFType.size_for_exactly(universe) else CFType.size_for(universe);
_ = size2str(std.math.max(1024, size), &buf) catch |err| switch (err) {
error.TooBig => return redis.RedisModule_ReplyWithError.?(ctx, c"ERR unsupported resulting size (8G max)"),
else => return redis.RedisModule_ReplyWithError.?(ctx, c"ERR unexpected error. Please report it at github.com/kristoff-it/redis-cuckoofilter"),
};
return redis.RedisModule_ReplyWithSimpleString.?(ctx, @ptrCast([*c]const u8, &buf));
}
test "insensitive_eql" {
std.testing.expect(insensitive_eql("EXACT", "EXACT"));
std.testing.expect(insensitive_eql("EXACT", "exact"));
std.testing.expect(insensitive_eql("EXACT", "exacT"));
std.testing.expect(insensitive_eql("EXACT", "eXacT"));
std.testing.expect(insensitive_eql("", ""));
std.testing.expect(insensitive_eql("1", "1"));
std.testing.expect(insensitive_eql("1234", "1234"));
std.testing.expect(!insensitive_eql("", "1"));
std.testing.expect(!insensitive_eql("1", "2"));
std.testing.expect(!insensitive_eql("1", "12"));
std.testing.expect(!insensitive_eql("EXACT", "AXACT"));
std.testing.expect(!insensitive_eql("EXACT", "ESACT"));
std.testing.expect(!insensitive_eql("EXACT", "EXECT"));
std.testing.expect(!insensitive_eql("EXACT", "EXAZT"));
std.testing.expect(!insensitive_eql("EXACT", "EXACZ"));
}
test "size2str" {
var buf: [5]u8 = undefined;
std.testing.expect(mem.eql(u8, "1K\x00", size2str(1024 * 1, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "2K\x00", size2str(1024 * 2, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "4K\x00", size2str(1024 * 4, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "8K\x00", size2str(1024 * 8, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "16K\x00", size2str(1024 * 16, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "32K\x00", size2str(1024 * 32, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "64K\x00", size2str(1024 * 64, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "128K\x00", size2str(1024 * 128, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "256K\x00", size2str(1024 * 256, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "512K\x00", size2str(1024 * 512, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "1M\x00", size2str(1024 * 1024 * 1, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "2M\x00", size2str(1024 * 1024 * 2, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "4M\x00", size2str(1024 * 1024 * 4, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "8M\x00", size2str(1024 * 1024 * 8, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "16M\x00", size2str(1024 * 1024 * 16, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "32M\x00", size2str(1024 * 1024 * 32, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "64M\x00", size2str(1024 * 1024 * 64, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "128M\x00", size2str(1024 * 1024 * 128, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "256M\x00", size2str(1024 * 1024 * 256, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "512M\x00", size2str(1024 * 1024 * 512, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "1G\x00", size2str(1024 * 1024 * 1024 * 1, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "2G\x00", size2str(1024 * 1024 * 1024 * 2, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "4G\x00", size2str(1024 * 1024 * 1024 * 4, &buf) catch unreachable));
std.testing.expect(mem.eql(u8, "8G\x00", size2str(1024 * 1024 * 1024 * 8, &buf) catch unreachable));
std.testing.expectError(error.TooSmall, size2str(1, &buf));
std.testing.expectError(error.UnexpectedRemainder, size2str(1025, &buf));
std.testing.expectError(error.TooSmall, size2str(0, &buf));
}
test "str2size" {
std.testing.expect(1024 * 1 == str2size("1K") catch unreachable);
std.testing.expect(1024 * 2 == str2size("2K") catch unreachable);
std.testing.expect(1024 * 4 == str2size("4K") catch unreachable);
std.testing.expect(1024 * 8 == str2size("8K") catch unreachable);
std.testing.expect(1024 * 16 == str2size("16K") catch unreachable);
std.testing.expect(1024 * 32 == str2size("32K") catch unreachable);
std.testing.expect(1024 * 64 == str2size("64K") catch unreachable);
std.testing.expect(1024 * 128 == str2size("128K") catch unreachable);
std.testing.expect(1024 * 256 == str2size("256K") catch unreachable);
std.testing.expect(1024 * 512 == str2size("512K") catch unreachable);
std.testing.expect(1024 * 1024 * 1 == str2size("1M") catch unreachable);
std.testing.expect(1024 * 1024 * 2 == str2size("2M") catch unreachable);
std.testing.expect(1024 * 1024 * 4 == str2size("4M") catch unreachable);
std.testing.expect(1024 * 1024 * 8 == str2size("8M") catch unreachable);
std.testing.expect(1024 * 1024 * 16 == str2size("16M") catch unreachable);
std.testing.expect(1024 * 1024 * 32 == str2size("32M") catch unreachable);
std.testing.expect(1024 * 1024 * 64 == str2size("64M") catch unreachable);
std.testing.expect(1024 * 1024 * 128 == str2size("128M") catch unreachable);
std.testing.expect(1024 * 1024 * 256 == str2size("256M") catch unreachable);
std.testing.expect(1024 * 1024 * 512 == str2size("512M") catch unreachable);
std.testing.expect(1024 * 1024 * 1024 * 1 == str2size("1G") catch unreachable);
std.testing.expect(1024 * 1024 * 1024 * 2 == str2size("2G") catch unreachable);
std.testing.expect(1024 * 1024 * 1024 * 4 == str2size("4G") catch unreachable);
std.testing.expect(1024 * 1024 * 1024 * 8 == str2size("8G") catch unreachable);
std.testing.expectError(error.Error, str2size(""));
std.testing.expectError(error.Error, str2size("5K"));
std.testing.expectError(error.Error, str2size("55"));
std.testing.expectError(error.Error, str2size("800G"));
}
================================================
FILE: src/redismodule.zig
================================================
pub usingnamespace @cImport({
@cInclude("./lib/redismodule.h");
});
================================================
FILE: src/t_cuckoofilter.zig
================================================
const cuckoo = @import("./lib/zig-cuckoofilter.zig");
const redis = @import("./redismodule.zig");
pub const CUCKOO_FILTER_ENCODING_VERSION = 2;
pub var Type8: ?*redis.RedisModuleType = null;
pub var Type16: ?*redis.RedisModuleType = null;
pub var Type32: ?*redis.RedisModuleType = null;
// `s` is the prng state. It's persisted for each key
// in order to provide fully deterministic behavior for
// insertions.
pub const Filter8 = struct {
s: [2]u64,
cf: cuckoo.Filter8,
};
pub const Filter16 = struct {
s: [2]u64,
cf: cuckoo.Filter16,
};
pub const Filter32 = struct {
s: [2]u64,
cf: cuckoo.Filter32,
};
pub fn RegisterTypes(ctx: *redis.RedisModuleCtx) !void {
// 8 bit fingerprint
Type8 = redis.RedisModule_CreateDataType.?(ctx, c"ccf-kff-1", CUCKOO_FILTER_ENCODING_VERSION, &redis.RedisModuleTypeMethods{
.version = redis.REDISMODULE_TYPE_METHOD_VERSION,
.rdb_load = CFLoad8,
.rdb_save = CFSave8,
.aof_rewrite = CFRewrite,
.free = CFFree8,
.mem_usage = CFMemUsage8,
.digest = CFDigest,
});
if (Type8 == null) return error.RegisterError;
// 16 bit fingerprint
Type16 = redis.RedisModule_CreateDataType.?(ctx, c"ccf-kff-2", CUCKOO_FILTER_ENCODING_VERSION, &redis.RedisModuleTypeMethods{
.version = redis.REDISMODULE_TYPE_METHOD_VERSION,
.rdb_load = CFLoad16,
.rdb_save = CFSave16,
.aof_rewrite = CFRewrite,
.free = CFFree16,
.mem_usage = CFMemUsage16,
.digest = CFDigest,
});
if (Type16 == null) return error.RegisterError;
// 32 bit fingerprint
Type32 = redis.RedisModule_CreateDataType.?(ctx, c"ccf-kff-4", CUCKOO_FILTER_ENCODING_VERSION, &redis.RedisModuleTypeMethods{
.version = redis.REDISMODULE_TYPE_METHOD_VERSION,
.rdb_load = CFLoad32,
.rdb_save = CFSave32,
.aof_rewrite = CFRewrite,
.free = CFFree32,
.mem_usage = CFMemUsage32,
.digest = CFDigest,
});
if (Type32 == null) return error.RegisterError;
}
export fn CFLoad8(rdb: ?*redis.RedisModuleIO, encver: c_int) ?*c_void {
return CFLoadImpl(Filter8, rdb, encver);
}
export fn CFLoad16(rdb: ?*redis.RedisModuleIO, encver: c_int) ?*c_void {
return CFLoadImpl(Filter16, rdb, encver);
}
export fn CFLoad32(rdb: ?*redis.RedisModuleIO, encver: c_int) ?*c_void {
return CFLoadImpl(Filter32, rdb, encver);
}
inline fn CFLoadImpl(comptime CFType: type, rdb: ?*redis.RedisModuleIO, encver: c_int) ?*c_void {
if (encver != CUCKOO_FILTER_ENCODING_VERSION) {
// We should actually log an error here, or try to implement
// the ability to load older versions of our data structure.
return null;
}
// Allocate cf struct
var cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), redis.RedisModule_Alloc.?(@sizeOf(CFType))));
// Load
const realCFType = @typeOf(cf.cf);
cf.* = CFType{
.s = [2]u64{ redis.RedisModule_LoadUnsigned.?(rdb), redis.RedisModule_LoadUnsigned.?(rdb) },
.cf = realCFType{
.rand_fn = null,
.homeless_fp = @intCast(realCFType.FPType, redis.RedisModule_LoadUnsigned.?(rdb)),
.homeless_bucket_idx = @intCast(usize, redis.RedisModule_LoadUnsigned.?(rdb)),
.fpcount = redis.RedisModule_LoadUnsigned.?(rdb),
.broken = redis.RedisModule_LoadUnsigned.?(rdb) != 0,
.buckets = blk: {
var bytes_len: usize = undefined;
const buckets_ptr = @alignCast(@alignOf(usize), redis.RedisModule_LoadStringBuffer.?(rdb, &bytes_len))[0..bytes_len];
break :blk realCFType.bytesToBuckets(buckets_ptr) catch @panic("trying to load corrupted buckets from RDB!");
},
},
};
return cf;
}
export fn CFSave8(rdb: ?*redis.RedisModuleIO, value: ?*c_void) void {
CFSaveImpl(Filter8, rdb, value);
}
export fn CFSave16(rdb: ?*redis.RedisModuleIO, value: ?*c_void) void {
CFSaveImpl(Filter16, rdb, value);
}
export fn CFSave32(rdb: ?*redis.RedisModuleIO, value: ?*c_void) void {
CFSaveImpl(Filter32, rdb, value);
}
inline fn CFSaveImpl(comptime CFType: type, rdb: ?*redis.RedisModuleIO, value: ?*c_void) void {
const cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), value));
// Write cuckoo struct data
redis.RedisModule_SaveUnsigned.?(rdb, cf.s[0]);
redis.RedisModule_SaveUnsigned.?(rdb, cf.s[1]);
redis.RedisModule_SaveUnsigned.?(rdb, cf.cf.homeless_fp);
redis.RedisModule_SaveUnsigned.?(rdb, cf.cf.homeless_bucket_idx);
redis.RedisModule_SaveUnsigned.?(rdb, cf.cf.fpcount);
if (cf.cf.broken) redis.RedisModule_SaveUnsigned.?(rdb, 1) else redis.RedisModule_SaveUnsigned.?(rdb, 0);
// Write buckets
const bytes = @sliceToBytes(cf.cf.buckets);
redis.RedisModule_SaveStringBuffer.?(rdb, bytes.ptr, bytes.len);
}
export fn CFFree8(cf: ?*c_void) void {
CFFreeImpl(Filter8, cf);
}
export fn CFFree16(cf: ?*c_void) void {
CFFreeImpl(Filter16, cf);
}
export fn CFFree32(cf: ?*c_void) void {
CFFreeImpl(Filter32, cf);
}
inline fn CFFreeImpl(comptime CFType: type, value: ?*c_void) void {
const cf = @ptrCast(*CFType, @alignCast(@alignOf(usize), value));
redis.RedisModule_Free.?(cf.cf.buckets.ptr);
redis.RedisModule_Free.?(cf);
}
export fn CFMemUsage8(value: ?*const c_void) usize {
return CFMemUsageImpl(Filter8, value);
}
export fn CFMemUsage16(value: ?*const c_void) usize {
return CFMemUsageImpl(Filter16, value);
}
export fn CFMemUsage32(value: ?*const c_void) usize {
return CFMemUsageImpl(Filter32, value);
}
inline fn CFMemUsageImpl(comptime CFType: type, value: ?*const c_void) usize {
const cf = @ptrCast(*const CFType, @alignCast(@alignOf(usize), value));
const realCFType = @typeOf(cf.cf);
return @sizeOf(CFType) + (@sliceToBytes(cf.cf.buckets).len * @sizeOf(realCFType.FPType));
}
export fn CFRewrite(aof: ?*redis.RedisModuleIO, key: ?*redis.RedisModuleString, value: ?*c_void) void {}
export fn CFDigest(digest: ?*redis.RedisModuleDigest, value: ?*c_void) void {}
gitextract_w82oi8x6/
├── .gitignore
├── LICENSE
├── README.md
├── RELEASENOTES
└── src/
├── lib/
│ ├── redismodule.h
│ └── zig-cuckoofilter.zig
├── redis-cuckoofilter.zig
├── redismodule.zig
└── t_cuckoofilter.zig
SYMBOL INDEX (17 symbols across 1 files)
FILE: src/lib/redismodule.h
type RedisModuleTimerID (line 134) | typedef uint64_t RedisModuleTimerID;
type mstime_t (line 145) | typedef long long mstime_t;
type RedisModuleCtx (line 148) | typedef struct RedisModuleCtx RedisModuleCtx;
type RedisModuleKey (line 149) | typedef struct RedisModuleKey RedisModuleKey;
type RedisModuleString (line 150) | typedef struct RedisModuleString RedisModuleString;
type RedisModuleCallReply (line 151) | typedef struct RedisModuleCallReply RedisModuleCallReply;
type RedisModuleIO (line 152) | typedef struct RedisModuleIO RedisModuleIO;
type RedisModuleType (line 153) | typedef struct RedisModuleType RedisModuleType;
type RedisModuleDigest (line 154) | typedef struct RedisModuleDigest RedisModuleDigest;
type RedisModuleBlockedClient (line 155) | typedef struct RedisModuleBlockedClient RedisModuleBlockedClient;
type RedisModuleClusterInfo (line 156) | typedef struct RedisModuleClusterInfo RedisModuleClusterInfo;
type RedisModuleDict (line 157) | typedef struct RedisModuleDict RedisModuleDict;
type RedisModuleDictIter (line 158) | typedef struct RedisModuleDictIter RedisModuleDictIter;
type RedisModuleCommandFilterCtx (line 159) | typedef struct RedisModuleCommandFilterCtx RedisModuleCommandFilterCtx;
type RedisModuleCommandFilter (line 160) | typedef struct RedisModuleCommandFilter RedisModuleCommandFilter;
type RedisModuleTypeMethods (line 176) | typedef struct RedisModuleTypeMethods {
function RedisModule_Init (line 361) | static int RedisModule_Init(RedisModuleCtx *ctx, const char *name, int v...
Condensed preview — 9 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (113K chars).
[
{
"path": ".gitignore",
"chars": 540,
"preview": "zig-cache/\n# Prerequisites\n*.d\n\n# Object files\n*.o\n*.ko\n*.obj\n*.elf\n\n# Linker output\n*.ilk\n*.map\n*.exp\n\n# Precompiled He"
},
{
"path": "LICENSE",
"chars": 1066,
"preview": "MIT License\n\nCopyright (c) 2019 Loris Cro\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\n"
},
{
"path": "README.md",
"chars": 12838,
"preview": "\n<h1 align=\"center\">redis-cuckoofilter</h1>\n<p align=\"center\">\n <a href=\"https://github.com/kristoff-it/redis-cuckoof"
},
{
"path": "RELEASENOTES",
"chars": 6242,
"preview": "-- 1.1.1\n\t- Update documentation for CF.CAPACITY as it was out of sync with the code.\n\t- Bundled a new build of the bina"
},
{
"path": "src/lib/redismodule.h",
"chars": 31221,
"preview": "#ifndef REDISMODULE_H\n#define REDISMODULE_H\n\n#include <sys/types.h>\n#include <stdint.h>\n#include <stdio.h>\n\n/* ---------"
},
{
"path": "src/lib/zig-cuckoofilter.zig",
"chars": 22024,
"preview": "const builtin = @import(\"builtin\");\nconst std = @import(\"std\");\nconst testing = std.testing;\n\nconst FREE_SLOT = 0;\n\n// U"
},
{
"path": "src/redis-cuckoofilter.zig",
"chars": 29574,
"preview": "const builtin = @import(\"builtin\");\nconst std = @import(\"std\");\nconst mem = std.mem;\nconst redis = @import(\"./redismodul"
},
{
"path": "src/redismodule.zig",
"chars": 72,
"preview": "pub usingnamespace @cImport({\n @cInclude(\"./lib/redismodule.h\");\n});\n"
},
{
"path": "src/t_cuckoofilter.zig",
"chars": 6090,
"preview": "const cuckoo = @import(\"./lib/zig-cuckoofilter.zig\");\nconst redis = @import(\"./redismodule.zig\");\n\npub const CUCKOO_FILT"
}
]
About this extraction
This page contains the full source code of the kristoff-it/redis-cuckoofilter GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 9 files (107.1 KB), approximately 29.5k tokens, and a symbol index with 17 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.