Showing preview only (632K chars total). Download the full file or copy to clipboard to get everything.
Repository: goossaert/kingdb
Branch: master
Commit: 58994280e789
Files: 75
Total size: 605.4 KB
Directory structure:
gitextract_ytklrwk_/
├── .gitignore
├── 3rdparty/
│ ├── leveldb/
│ │ ├── AUTHORS
│ │ └── LICENSE
│ ├── lz4/
│ │ └── LICENSE
│ ├── murmurhash3/
│ │ └── LICENSE
│ ├── xxhash/
│ │ └── LICENSE
│ └── zlib/
│ └── LICENSE
├── LICENSE
├── Makefile
├── README.md
├── algorithm/
│ ├── coding.cc
│ ├── coding.h
│ ├── compressor.cc
│ ├── compressor.h
│ ├── crc32c.cc
│ ├── crc32c.h
│ ├── endian.cc
│ ├── endian.h
│ ├── hash.cc
│ ├── hash.h
│ ├── lz4.cc
│ ├── lz4.h
│ ├── murmurhash3.cc
│ ├── murmurhash3.h
│ ├── xxhash.cc
│ └── xxhash.h
├── cache/
│ ├── rate_limiter.h
│ ├── write_buffer.cc
│ └── write_buffer.h
├── doc/
│ ├── bench/
│ │ ├── benchmarks.md
│ │ ├── db_bench_kingdb.cc
│ │ └── generate_benchmarks_table.py
│ ├── kingdb.md
│ └── kingserver.md
├── include/
│ └── kingdb/
│ └── kdb.h
├── interface/
│ ├── database.cc
│ ├── database.h
│ ├── iterator.h
│ ├── kingdb.h
│ ├── multipart.h
│ └── snapshot.h
├── network/
│ ├── client.h
│ ├── client_main.cc
│ ├── server.cc
│ ├── server.h
│ └── server_main.cc
├── storage/
│ ├── format.h
│ ├── hstable_manager.h
│ ├── resource_manager.h
│ └── storage_engine.h
├── thread/
│ ├── event_manager.h
│ ├── threadpool.h
│ └── threadstorage.h
├── unit-tests/
│ ├── client_embedded.cc
│ ├── dummy_interface.h
│ ├── dummy_storage_engine.h
│ ├── dummy_storage_engine_index.h
│ ├── kingdb_user.cc
│ ├── test_compression.cc
│ ├── test_db.cc
│ ├── testharness.cc
│ └── testharness.h
└── util/
├── byte_array.h
├── config_parser.h
├── debug.cc
├── debug.h
├── file.h
├── filepool.h
├── logger.cc
├── logger.h
├── options.h
├── order.h
├── status.cc
├── status.h
└── version.h
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
# Compiled Object files
*.slo
*.lo
*.o
*.obj
# Compiled Dynamic libraries
*.so
*.dylib
*.dll
# Compiled Static libraries
*.lai
*.la
*.a
*.lib
# Executables
*.exe
*.out
*.app
# Python
*.pyc
# Vim
*~
*.*.swp
================================================
FILE: 3rdparty/leveldb/AUTHORS
================================================
# Names should be added to this file like so:
# Name or Organization <email address>
Google Inc.
# Initial version authors:
Jeffrey Dean <jeff@google.com>
Sanjay Ghemawat <sanjay@google.com>
# Partial list of contributors:
Kevin Regan <kevin.d.regan@gmail.com>
Johan Bilien <jobi@litl.com>
================================================
FILE: 3rdparty/leveldb/LICENSE
================================================
Copyright (c) 2011 The LevelDB Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
================================================
FILE: 3rdparty/lz4/LICENSE
================================================
LZ4 Library
Copyright (c) 2011-2014, Yann Collet
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
================================================
FILE: 3rdparty/murmurhash3/LICENSE
================================================
MurmurHash3 was written by Austin Appleby, and is placed in the public
domain. The author hereby disclaims copyright to this source code.
================================================
FILE: 3rdparty/xxhash/LICENSE
================================================
xxHash Library
Copyright (c) 2012-2014, Yann Collet
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
================================================
FILE: 3rdparty/zlib/LICENSE
================================================
/* zlib.h -- interface of the 'zlib' general purpose compression library
version 1.2.8, April 28th, 2013
Copyright (C) 1995-2013 Jean-loup Gailly and Mark Adler
This software is provided 'as-is', without any express or implied
warranty. In no event will the authors be held liable for any damages
arising from the use of this software.
Permission is granted to anyone to use this software for any purpose,
including commercial applications, and to alter it and redistribute it
freely, subject to the following restrictions:
1. The origin of this software must not be misrepresented; you must not
claim that you wrote the original software. If you use this software
in a product, an acknowledgment in the product documentation would be
appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be
misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.
Jean-loup Gailly Mark Adler
jloup@gzip.org madler@alumni.caltech.edu
*/
================================================
FILE: LICENSE
================================================
Copyright (c) 2014, Emmanuel Goossaert
All rights reserved.
Use of KingDB is governed by the BSD 3-Clause License.
KingDB uses/derives code from third-party open source projects. For the complete
list of those projects and their respective licenses, refer to the end of the
LICENSE file.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name of KingDB nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
----------------------------------------------------------------
Third-party source code
Below is the list of the projects from which KingDB derives code, and the
location of their respective LICENSE files.
- LevelDB, under BSD 3-Clause License: /3rdparty/leveldb/
- LZ4, under BSD 2-Clause License: /3rdparty/lz4/
- Murmurhash3, public domain: /3rdparty/murmurhash3/
- xxHash, under BSD 2-Clause License: /3rdparty/xxhash/
- zlib, under zlib/libpng License: /3rdparty/zlib/
================================================
FILE: Makefile
================================================
CC=g++
INCLUDES=-I/usr/local/include/ -I/opt/local/include/ -I. -I./include/
LDFLAGS=-g -L/usr/local/lib/ -L/opt/local/lib/ -lpthread
LDFLAGS_CLIENT=-g -L/usr/local/lib/ -L/opt/local/lib/ -lpthread -lmemcached -fPIC
SOURCES=interface/database.cc util/logger.cc util/status.cc util/debug.cc network/server.cc cache/write_buffer.cc algorithm/endian.cc algorithm/compressor.cc algorithm/murmurhash3.cc algorithm/xxhash.cc algorithm/crc32c.cc algorithm/lz4.cc algorithm/hash.cc algorithm/coding.cc unit-tests/testharness.cc
SOURCES_MAIN=network/server_main.cc
SOURCES_CLIENT=network/client_main.cc
SOURCES_CLIENT_EMB=unit-tests/client_embedded.cc
SOURCES_TEST_COMPRESSION=unit-tests/test_compression.cc
SOURCES_TEST_DB=unit-tests/test_db.cc
OBJECTS=$(SOURCES:.cc=.o)
OBJECTS_MAIN=$(SOURCES_MAIN:.cc=.o)
OBJECTS_CLIENT=$(SOURCES_CLIENT:.cc=.o)
OBJECTS_CLIENT_EMB=$(SOURCES_CLIENT_EMB:.cc=.o)
OBJECTS_TEST_COMPRESSION=$(SOURCES_TEST_COMPRESSION:.cc=.o)
OBJECTS_TEST_DB=$(SOURCES_TEST_DB:.cc=.o)
EXECUTABLE=kingserver
CLIENT_NETWORK=client_network
CLIENT_EMB=client_emb
TEST_COMPRESSION=test_compression
TEST_DB=test_db
LIBRARY=libkingdb.a
PREFIX=/usr/local
BINDIR=$(PREFIX)/bin
INCLUDEDIR=$(PREFIX)/include/kingdb
LIBDIR=$(PREFIX)/lib
GCC_WALL=-Waddress -Warray-bounds=1 -Wc++11-compat -Wc++14-compat -Wchar-subscripts -Wenum-compare -Wimplicit-int -Wimplicit-function-declaration -Wcomment -Wformat -Wmain -Wmaybe-uninitialized -Wmissing-braces -Wnonnull -Wopenmp-simd -Wparentheses -Wpointer-sign -Wreorder -Wreturn-type -Wsequence-point -Wsign-compare -Wstrict-aliasing -Wstrict-overflow=1 -Wswitch -Wtrigraphs -Wuninitialized -Wunknown-pragmas -Wunused-function -Wunused-label -Wunused-value -Wunused-variable -Wvolatile-register-var
CFLAGS=-Wall -std=c++11 -MMD -MP -c
all: CFLAGS += -O2
all: $(SOURCES) $(LIBRARY) $(EXECUTABLE) $(CLIENT_EMB) $(TEST_COMPRESSION) $(TEST_DB)
debug: CFLAGS += -DDEBUG -g
debug: LDFLAGS+= -lprofiler
debug: $(SOURCES) $(LIBRARY) $(EXECUTABLE) $(CLIENT_EMB) $(TEST_COMPRESSION) $(TEST_DB)
client: CFLAGS += -O2
client: $(SOURCES) $(CLIENT_NETWORK)
client-debug: CFLAGS += -DDEBUG -g
client-debug: LDFLAGS_CLIENT += -lprofiler
client-debug: $(SOURCES) $(CLIENT_NETWORK)
threadsanitize: CFLAGS += -DDEBUG -g -fsanitize=thread -O2 -pie -fPIC
threadsanitize: LDFLAGS += -pie -ltsan
threadsanitize: LDFLAGS_CLIENT += -pie -ltsan
threadsanitize: $(SOURCES) $(LIBRARY) $(EXECUTABLE) $(CLIENT_EMB) $(CLIENT_NETWORK) $(TEST_COMPRESSION) $(TEST_DB)
$(EXECUTABLE): $(OBJECTS) $(OBJECTS_MAIN)
$(CC) $(OBJECTS) $(OBJECTS_MAIN) -o $@ $(LDFLAGS)
$(CLIENT_NETWORK): $(OBJECTS) $(OBJECTS_CLIENT)
$(CC) $(OBJECTS) $(OBJECTS_CLIENT) -o $@ $(LDFLAGS_CLIENT)
$(CLIENT_EMB): $(OBJECTS) $(OBJECTS_CLIENT_EMB)
$(CC) $(OBJECTS) $(OBJECTS_CLIENT_EMB) -o $@ $(LDFLAGS)
$(TEST_COMPRESSION): $(OBJECTS) $(OBJECTS_TEST_COMPRESSION)
$(CC) $(OBJECTS) $(OBJECTS_TEST_COMPRESSION) -o $@ $(LDFLAGS)
$(TEST_DB): $(OBJECTS) $(OBJECTS_TEST_DB)
$(CC) $(OBJECTS) $(OBJECTS_TEST_DB) -o $@ $(LDFLAGS)
$(LIBRARY): $(OBJECTS)
rm -f $@
ar -rs $@ $(OBJECTS)
install: $(EXECUTABLE)
install $(EXECUTABLE) $(BINDIR)
cp libkingdb.a $(LIBDIR)
rm -rf $(INCLUDEDIR)
mkdir $(INCLUDEDIR) $(INCLUDEDIR)/util $(INCLUDEDIR)/thread $(INCLUDEDIR)/interface $(INCLUDEDIR)/storage $(INCLUDEDIR)/algorithm $(INCLUDEDIR)/cache
cp algorithm/*.h $(INCLUDEDIR)/algorithm
cp cache/*.h $(INCLUDEDIR)/cache
cp include/kingdb/*.h $(INCLUDEDIR)
cp interface/*.h $(INCLUDEDIR)/interface
cp util/*.h $(INCLUDEDIR)/util
cp thread/*.h $(INCLUDEDIR)/thread
cp storage/*.h $(INCLUDEDIR)/storage
.cc.o:
$(CC) $(CFLAGS) $(INCLUDES) $< -o $@
clean:
rm -f $(EXECUTABLE) $(CLIENT_NETWORK) $(CLIENT_EMB) $(TEST_COMPRESSION) $(TEST_DB) $(LIBRARY)
find . -name \.*.*.swp* -type f -print0 | xargs -0 rm -f
find . -name \*.d -type f -print0 | xargs -0 rm -f
find . -name \*.o -type f -print0 | xargs -0 rm -f
find . -name \*~ -type f -print0 | xargs -0 rm -f
find . -name \*-e -type f -print0 | xargs -0 rm -f
-include $(SOURCES:%.cc=%.d)
================================================
FILE: README.md
================================================
KingDB
======
###What is KingDB?
**KingDB** is a fast on-disk persistent key-value store. You can embed it or use it as a library in your C++ applications.
**KingServer** is a server application that embeds KingDB and implements the Memcached protocol. It allows you to access your data through a network interface using whatever programming language you want. KingServer is not a distributed system: the data lives in a single machine.
**WARNING:** This is still alpha code. Even though unit-tests are covering the core components, make sure you run tests for your environment before using KingDB in production.
###Why use KingDB?
- Fast for heavy write workloads and random reads.
- The architecture, code, and data format are simple.
- Multipart API to read and write large entries in smaller parts.
- Multiple threads can access the same database safely.
- Crash-proof: nothing ever gets overwritten.
- Iterators and read-only consistent snapshots.
- Compaction happens in a background thread, and does not block reads or writes.
- The data format allows hot backups to be made.
- Covered by unit tests.
###How fast is KingDB?
KingDB was benchmarked using the same test suite as LevelDB. On a Linux CentOS 6.5, for entries with 16-byte keys and 100-byte values (50 bytes after compression), the performance was:
| Workload | Operations per second |
| ------------------: | :-------------------- |
| Sequential reads | 104k |
| Random reads | 203k |
| Sequential writes | 233k |
| Random writes | 251k |
| Overwrite | 250k |
For more details and a comparison with LevelDB, you can refer to the full [KingDB benchmarks](doc/bench/benchmarks.md).
###Where is the documentation?
You can learn more in the [KingDB documentation](doc/kingdb.md) and the [KingServer documentation](doc/kingserver.md).
###How to install KingDB?
You can find installation instructions in the [installation section](doc/kingdb.md#2-how-to-install-kingdb) of the documentation.
KingDB has no external dependencies and has been tested on:
- Mac OS X 10.9.5 with Apple LLVM version 6.0 (clang-600.0.51)
- Linux Ubuntu 14.04 x64 with GCC 4.9.2
- Linux Ubuntu 15.04 x64 with GCC 4.9.2-10ubuntu13
- Linux CentOS 6.5 x86\_64 with GCC 4.9.2
If you are using GCC, update the Makefile and add \-fno\-builtin\-memcmp in the CFLAGS, and if you have tcmalloc on your system, add \-ltcmalloc to the LDFLAGS. This will give you a nice performance speed\-up.
###Where to get help?
You can get help on the [KingDB mailing list](https://groups.google.com/d/forum/kingdb). To ask a question, simply [join the list](https://groups.google.com/d/forum/kingdb/join).
================================================
FILE: algorithm/coding.cc
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
// The code below was copied from LevelDB. A few changes were applied to make it
// self-sufficient and part of KingDB.
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#include "algorithm/coding.h"
namespace kdb {
void EncodeFixed32(char* buf, uint32_t value) {
if (kdb::kLittleEndian) {
memcpy(buf, &value, sizeof(value));
} else {
buf[0] = value & 0xff;
buf[1] = (value >> 8) & 0xff;
buf[2] = (value >> 16) & 0xff;
buf[3] = (value >> 24) & 0xff;
}
}
void EncodeFixed64(char* buf, uint64_t value) {
if (kdb::kLittleEndian) {
memcpy(buf, &value, sizeof(value));
} else {
buf[0] = value & 0xff;
buf[1] = (value >> 8) & 0xff;
buf[2] = (value >> 16) & 0xff;
buf[3] = (value >> 24) & 0xff;
buf[4] = (value >> 32) & 0xff;
buf[5] = (value >> 40) & 0xff;
buf[6] = (value >> 48) & 0xff;
buf[7] = (value >> 56) & 0xff;
}
}
void GetFixed32(const char* buf, uint32_t* value) {
if (kdb::kLittleEndian) {
memcpy(value, buf, sizeof(*value));
} else {
// NOTE: this code has not been tested yet
*value = (uint32_t)buf[0]
| (uint32_t)buf[1] << 8
| (uint32_t)buf[2] << 16
| (uint32_t)buf[3] << 24;
}
}
void GetFixed64(const char* buf, uint64_t* value) {
if (kdb::kLittleEndian) {
memcpy(value, buf, sizeof(*value));
} else {
// NOTE: this code has not been tested yet
*value = (uint64_t)buf[0]
| (uint64_t)buf[1] << 8
| (uint64_t)buf[2] << 16
| (uint64_t)buf[3] << 24
| (uint64_t)buf[4] << 32
| (uint64_t)buf[5] << 40
| (uint64_t)buf[6] << 48
| (uint64_t)buf[7] << 56;
}
}
void PutFixed32(std::string* dst, uint32_t value) {
char buf[sizeof(value)];
EncodeFixed32(buf, value);
dst->append(buf, sizeof(buf));
}
void PutFixed64(std::string* dst, uint64_t value) {
char buf[sizeof(value)];
EncodeFixed64(buf, value);
dst->append(buf, sizeof(buf));
}
char* EncodeVarint32(char* dst, uint32_t v) {
// Operate on characters as unsigneds
unsigned char* ptr = reinterpret_cast<unsigned char*>(dst);
static const int B = 128;
if (v < (1<<7)) {
*(ptr++) = v;
} else if (v < (1<<14)) {
*(ptr++) = v | B;
*(ptr++) = v>>7;
} else if (v < (1<<21)) {
*(ptr++) = v | B;
*(ptr++) = (v>>7) | B;
*(ptr++) = v>>14;
} else if (v < (1<<28)) {
*(ptr++) = v | B;
*(ptr++) = (v>>7) | B;
*(ptr++) = (v>>14) | B;
*(ptr++) = v>>21;
} else {
*(ptr++) = v | B;
*(ptr++) = (v>>7) | B;
*(ptr++) = (v>>14) | B;
*(ptr++) = (v>>21) | B;
*(ptr++) = v>>28;
}
return reinterpret_cast<char*>(ptr);
}
void PutVarint32(std::string* dst, uint32_t v) {
char buf[5];
char* ptr = EncodeVarint32(buf, v);
dst->append(buf, ptr - buf);
}
char* EncodeVarint64(char* dst, uint64_t v) {
static const int B = 128;
unsigned char* ptr = reinterpret_cast<unsigned char*>(dst);
while (v >= B) {
*(ptr++) = (v & (B-1)) | B;
v >>= 7;
}
*(ptr++) = static_cast<unsigned char>(v);
return reinterpret_cast<char*>(ptr);
}
void PutVarint64(std::string* dst, uint64_t v) {
char buf[10];
char* ptr = EncodeVarint64(buf, v);
dst->append(buf, ptr - buf);
}
int VarintLength(uint64_t v) {
int len = 1;
while (v >= 128) {
v >>= 7;
len++;
}
return len;
}
const char* GetVarint32PtrFallback(const char* p,
const char* limit,
uint32_t* value) {
uint32_t result = 0;
for (uint32_t shift = 0; shift <= 28 && p < limit; shift += 7) {
uint32_t byte = *(reinterpret_cast<const unsigned char*>(p));
p++;
if (byte & 128) {
// More bytes are present
result |= ((byte & 127) << shift);
} else {
result |= (byte << shift);
*value = result;
return reinterpret_cast<const char*>(p);
}
}
return NULL;
}
int GetVarint32(char* data, uint64_t size, uint32_t* value) {
const char* p = data;
const char* limit = p + size;
const char* q = GetVarint32Ptr(p, limit, value);
if (q == NULL) {
return -1;
} else {
return q - p;
}
}
const char* GetVarint64Ptr(const char* p, const char* limit, uint64_t* value) {
uint64_t result = 0;
for (uint32_t shift = 0; shift <= 63 && p < limit; shift += 7) {
uint64_t byte = *(reinterpret_cast<const unsigned char*>(p));
p++;
if (byte & 128) {
// More bytes are present
result |= ((byte & 127) << shift);
} else {
result |= (byte << shift);
*value = result;
return reinterpret_cast<const char*>(p);
}
}
return NULL;
}
int GetVarint64(char* data, uint64_t size, uint64_t* value) {
const char* p = data;
const char* limit = p + size;
const char* q = GetVarint64Ptr(p, limit, value);
if (q == NULL) {
return -1;
} else {
return q - p;
}
}
} // namespace kdb
================================================
FILE: algorithm/coding.h
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
// The code below was copied from LevelDB. A few changes were applied to make it
// self-sufficient and part of KingDB.
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
// Endian-neutral encoding:
// * Fixed-length numbers are encoded with least-significant byte first
// * In addition we support variable length "varint" encoding
// * Strings are encoded prefixed by their length in varint format
#ifndef KINGDB_CODING_H_
#define KINGDB_CODING_H_
#include "util/debug.h"
#include <stdint.h>
#include <string.h>
#include <string>
#include "algorithm/endian.h"
#include "util/status.h"
namespace kdb {
// Standard Put... routines append to a string
extern void PutFixed32(std::string* dst, uint32_t value);
extern void PutFixed64(std::string* dst, uint64_t value);
extern void PutVarint32(std::string* dst, uint32_t value);
extern void PutVarint64(std::string* dst, uint64_t value);
extern int GetVarint32(char* input, uint64_t size, uint32_t* value);
extern int GetVarint64(char* input, uint64_t size, uint64_t* value);
// Pointer-based variants of GetVarint... These either store a value
// in *v and return a pointer just past the parsed value, or return
// NULL on error. These routines only look at bytes in the range
// [p..limit-1]
extern const char* GetVarint32Ptr(const char* p,const char* limit, uint32_t* v);
extern const char* GetVarint64Ptr(const char* p,const char* limit, uint64_t* v);
// Returns the length of the varint32 or varint64 encoding of "v"
extern int VarintLength(uint64_t v);
// Lower-level versions of Put... that write directly into a character buffer
// REQUIRES: dst has enough space for the value being written
extern void EncodeFixed32(char* dst, uint32_t value);
extern void EncodeFixed64(char* dst, uint64_t value);
extern void GetFixed32(const char* src, uint32_t* value);
extern void GetFixed64(const char* src, uint64_t* value);
// Lower-level versions of Put... that write directly into a character buffer
// and return a pointer just past the last byte written.
// REQUIRES: dst has enough space for the value being written
extern char* EncodeVarint32(char* dst, uint32_t value);
extern char* EncodeVarint64(char* dst, uint64_t value);
// Lower-level versions of Get... that read directly from a character buffer
// without any bounds checking.
inline uint32_t DecodeFixed32(const char* ptr) {
if (kdb::kLittleEndian) {
// Load the raw bytes
uint32_t result;
memcpy(&result, ptr, sizeof(result)); // gcc optimizes this to a plain load
return result;
} else {
return ((static_cast<uint32_t>(static_cast<unsigned char>(ptr[0])))
| (static_cast<uint32_t>(static_cast<unsigned char>(ptr[1])) << 8)
| (static_cast<uint32_t>(static_cast<unsigned char>(ptr[2])) << 16)
| (static_cast<uint32_t>(static_cast<unsigned char>(ptr[3])) << 24));
}
}
inline uint64_t DecodeFixed64(const char* ptr) {
if (kdb::kLittleEndian) {
// Load the raw bytes
uint64_t result;
memcpy(&result, ptr, sizeof(result)); // gcc optimizes this to a plain load
return result;
} else {
uint64_t lo = DecodeFixed32(ptr);
uint64_t hi = DecodeFixed32(ptr + 4);
return (hi << 32) | lo;
}
}
// Internal routine for use by fallback path of GetVarint32Ptr
extern const char* GetVarint32PtrFallback(const char* p,
const char* limit,
uint32_t* value);
inline const char* GetVarint32Ptr(const char* p,
const char* limit,
uint32_t* value) {
if (p < limit) {
uint32_t result = *(reinterpret_cast<const unsigned char*>(p));
if ((result & 128) == 0) {
*value = result;
return p + 1;
}
}
return GetVarint32PtrFallback(p, limit, value);
}
} // namespace kdb
#endif // KINGDB_CODING_H_
================================================
FILE: algorithm/compressor.cc
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
#include "algorithm/compressor.h"
namespace kdb {
void CompressorLZ4::ResetThreadLocalStorage() {
ts_compress_.reset();
ts_uncompress_.reset();
}
Status CompressorLZ4::Compress(char *source,
uint64_t size_source,
char **dest,
uint64_t *size_dest) {
/*
if (size_source < 8) {
*dest = nullptr;
*size_dest = 0;
return Status::OK();
}
*/
uint32_t bound = LZ4_compressBound(size_source);
*size_dest = 0;
*dest = new char[8 + bound];
int ret = LZ4_compress_limitedOutput(source, (*dest) + 8, size_source, bound);
if (ret <= 0) {
delete[] *dest;
return Status::IOError("LZ4_compress_limitedOutput() failed");
}
uint32_t size_compressed = ret + 8;
uint32_t size_compressed_stored = size_compressed;
// If the frame was compressed to a size larger than the original data,
// we just copy the original data.
if ((uint64_t)ret > size_source) {
if (size_source > 8 + bound) {
delete[] *dest;
*dest = new char[8 + size_source];
}
memcpy((*dest) + 8, source, size_source);
size_compressed = size_source + 8;
size_compressed_stored = 0;
}
// NOTE: Yes, storing 64 bits into 32, but overflows will not happens as
// size_source is limited to db.storage.maximum_part_size
uint32_t size_source_32 = size_source;
EncodeFixed32((*dest), size_compressed_stored);
EncodeFixed32((*dest) + 4, size_source_32);
// NOTE: small entries don't need to have the compressed and source sizes
// in front of the frame, this is just a waste of storage space.
// Maybe have a special type of entry, like 'small' or 'self-contained',
// which would indicate that the frame doesn't have the sizes.
log::trace("CompressorLZ4::Compress()", "size_compressed:%u size_source:%u", size_compressed, size_source_32);
uint64_t size_compressed_total = ts_compress_.get() + size_compressed;
ts_compress_.put(size_compressed_total);
*size_dest = size_compressed;
return Status::OK();
}
bool CompressorLZ4::IsUncompressionDone(uint64_t size_source_total) {
uint64_t offset_uncompress = ts_uncompress_.get();
if (offset_uncompress == size_source_total) return true;
return false;
}
Status CompressorLZ4::Uncompress(char *source,
uint64_t size_source_total,
char **dest,
uint64_t *size_dest,
char **frame_out,
uint64_t *size_frame_out,
bool do_memory_allocation // = true by dafault
) {
uint64_t offset_uncompress = ts_uncompress_.get();
log::trace("CompressorLZ4::Uncompress()", "in %" PRIu64 " %" PRIu64, offset_uncompress, size_source_total);
if (do_memory_allocation) {
*dest = nullptr;
}
if (offset_uncompress == size_source_total) return Status::Done();
uint32_t size_source, size_compressed;
GetFixed32(source + offset_uncompress, &size_compressed);
GetFixed32(source + offset_uncompress + 4, &size_source);
if (size_compressed > 0) {
// the frame contains compressed data
size_compressed -= 8;
*size_dest = 0;
if (do_memory_allocation) {
*dest = new char[size_source];
}
int size = size_compressed;
log::trace("CompressorLZ4::Uncompress()", "ptr:%p size:%d size_source:%d offset:%" PRIu64, source + offset_uncompress + 8, size, size_source, offset_uncompress);
int ret = LZ4_decompress_safe_partial(source + offset_uncompress + 8,
*dest,
size,
size_source,
size_source);
if (ret <= 0) {
if (do_memory_allocation) {
delete[] (*dest);
*dest = nullptr;
}
return Status::IOError("LZ4_decompress_safe_partial() failed");
}
*size_dest = ret;
} else {
// the frame contains uncompressed data
size_compressed = size_source;
*size_dest = size_source;
if (do_memory_allocation) {
*dest = new char[size_source];
}
memcpy(*dest, source + offset_uncompress + 8, size_source);
}
crc32_.stream(source + offset_uncompress, size_compressed + 8);
*frame_out = source + offset_uncompress;
*size_frame_out = size_compressed + 8;
log::trace("CompressorLZ4::Uncompress()", "crc32:0x%" PRIx64 " frame_ptr:%p frame_size:%" PRIu64, crc32_.get(), *frame_out, *size_frame_out);
offset_uncompress += size_compressed + 8;
ts_uncompress_.put(offset_uncompress);
log::trace("CompressorLZ4::Uncompress()", "out %" PRIu64 " %" PRIu64, offset_uncompress, size_source_total);
return Status::OK();
}
Status CompressorLZ4::UncompressByteArray(ByteArray& value,
bool do_checksum_verification,
ByteArray* value_uncompressed) {
Status s;
if (do_checksum_verification) {
crc32_.ResetThreadLocalStorage();
crc32_.put(value.checksum_initial());
}
bool is_compressed = value.is_compressed();
bool is_compression_disabled = false;
uint64_t offset_in = 0;
uint64_t offset_out = 0;
ResetThreadLocalStorage();
*value_uncompressed = ByteArray::NewAllocatedMemoryByteArray(value.size());
value_uncompressed->set_size(value.size());
value_uncompressed->set_size_compressed(0);
while (true) {
if (is_compressed && !is_compression_disabled) {
if (IsUncompressionDone(value.size_compressed())) {
if ( !do_checksum_verification
|| crc32_.get() == value.checksum()) {
log::debug("CompressorLZ4::UncompressByteArray()", "Good CRC32 - stored:0x%08" PRIx64 " computed:0x%08" PRIx64 "\n", value.checksum(), crc32_.get());
return Status::OK();
} else {
log::debug("CompressorLZ4::UncompressByteArray()", "Bad CRC32 - stored:0x%08" PRIx64 " computed:0x%08" PRIx64 "\n", value.checksum(), crc32_.get());
return Status::IOError("Invalid checksum.");
}
}
if (HasFrameHeaderDisabledCompression(value.data() + offset_in)) {
log::debug("CompressorLZ4::UncompressByteArray()", "Finds that compression is disabled\n");
is_compression_disabled = true;
if (do_checksum_verification) {
crc32_.stream(value.data() + offset_in, size_frame_header());
}
offset_in += size_frame_header();
}
if (!is_compression_disabled) {
char *frame;
uint64_t size_frame;
uint64_t size_out;
char *buffer_out = value_uncompressed->data() + offset_out;
log::trace("CompressorLZ4::UncompressByteArray()", "before uncompress");
Status s = Uncompress(value.data(),
value.size_compressed(),
&buffer_out,
&size_out,
&frame,
&size_frame,
false);
//chunk_ = NewShallowCopyByteArray(data_out, size_out);
if (s.IsDone()) {
return Status::OK();
} else if (s.IsOK()) {
if (do_checksum_verification) {
crc32_.stream(frame, size_frame);
}
} else {
return s;
}
offset_in += size_frame;
offset_out += size_out;
}
}
if (!is_compressed || is_compression_disabled) {
log::trace("CompressorLZ4::UncompressByteArray()", "No compression or compression disabled");
uint64_t size_left;
if (is_compressed && is_compression_disabled) {
size_left = value.size_compressed();
} else {
size_left = value.size();
}
if (offset_in == size_left) {
log::trace("CompressorLZ4::UncompressByteArray()", "Has gotten all the data");
return Status::OK();
}
char* data_left = value.data() + offset_in;
size_t step = 1024*1024;
size_t size_current = offset_in + step < size_left ? step : size_left - offset_in;
if (do_checksum_verification) {
crc32_.stream(data_left, size_current);
}
memcpy(value_uncompressed->data() + offset_out, data_left, size_current);
//chunk_ = value;
//chunk_.increment_offset(offset);
//chunk_.set_size(size_current);
//chunk_.set_size_compressed(0);
offset_in += size_current;
offset_out += size_current;
log::trace("CompressorLZ4::UncompressByteArray()", "Done with handling uncompressed data");
return Status::OK();
}
}
return true;
}
};
================================================
FILE: algorithm/compressor.h
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
#ifndef KINGDB_COMPRESSOR_H_
#define KINGDB_COMPRESSOR_H_
#include "util/debug.h"
#include <algorithm>
#include <map>
#include <cinttypes>
#include "algorithm/lz4.h"
#include "util/logger.h"
#include "util/status.h"
#include "util/byte_array.h"
#include "thread/threadstorage.h"
#include "algorithm/crc32c.h"
namespace kdb {
/* Depending on whether or not compression is enabled, and if compression is
* even possible (i.e. the data is not incompressible), the data stored for an
* entry will be stored with the appropriate format.
*
* 1. Compression is disabled
* - The space reserved on secondary storage for the entry's value is of the
* exact size of the value itself, and the data of the value is stored
* contiguously in that space.
* - The entry is marked as being 'compacted', because it uses exactly the space
* that was allocated for it.
* - The checksum calculation is done over the raw data, i.e. the data of the
* value itself.
*
* 2. Compression is enabled
* - Data is stored in 'frames'. Each frame contains a chunk of compressed data,
* with its own compression dictionary (depending on the compression algorithm
* being used).
* - The space reserved on secondary storage for the entry's value is of the
* size of the raw value data, to which a padding is added. That padding must
* be at least as large as the size of a frame header, which is 8 bytes.
* - The entry is marked as being 'uncompacted', the space reserved on secondary
* storage is likely to be more than the space the compressed value data is
* actually using (in the case of incompressible data, the allocated space may
* be just enough).
* - Each frame is as follows:
* Frame header:
* * size_compressed: size of compressed data in bytes] (4 bytes, 32
* unsigned integer)
* * size_raw: size of raw data in bytes] (4 bytes, 32 unsigned integer)
* Frame data:
* * compressed data (of size 'size_compressed' bytes)
* - Each chunk of data is compressed, and if the size of the output byte array
* is larger than the raw data, then the data is stored uncompressed: the
* compressed data in the frame simply contains the raw data itself. And the
* 'size_compressed' field in the frame header is set to 0 to indicate that
* the data stored is the raw data.
* - The checksum calculation is done over the frames, i.e. the compressed data
* along with the frame headers.
*
* 3. The data or part of the data is incompresssible
* - If the data is incompressible, i.e. the compressed data takes more space
* than the raw data, then we have a problem. The solution is to allocate at
* least the size of one frame header, which guarantees that in the worst
* case, the can be stored as it is, uncompressed.
* - As the compressed frames are being computed, the following check is
* performed:
* * size_frame_header = 8
* * size_remaining = (total size of raw value)
* - (current offset in the raw value)
* => size of raw data that is left to come
* * size_chunk
* => size of the chunk of raw value currently incoming
* * space_left = (total size of raw value)
* + (size of reserved padding)
* - (offset where the next write will go on secondary storage)
* => size of the space left on secondary storage where data can be
* written for this entry.
* * size_compressed: size of the current chunk of data after it was
* compressed (including the frame header)
* * if ( size_remaining - size_chunk + size_frame_header
* > space_left - size_compressed) {
* - cancel the compression of the current chunk
* - disable compression for the rest of the streaming of this entry,
* and store the rest of the incoming data as raw.
* }
* => This if-statement is testing, considering that the current chunk
* is compressed, if there will be enough space to store the rest
* of the entry as raw data when the next chunk comes in, including
* a frame header. If not, then the compression needs to be disabled
* immediately. Given that the padding is at least as large as the
* size of a frame header, and that the test is performed over the
* next chunk, any point of the compression stream will have enough
* space to store the entry as raw data if it needs to.
* - If compression is disabled mid-flight, the frame header is a sequence of
* null bytes. That way, the decompression process will be able to identify
* where compression was disabled, and copy the raw data directly on-ward.
* */
class CompressorLZ4 {
public:
CompressorLZ4() {
}
// Added an empty copy assignment operator to avoid error messages of the type:
// "object of type '...' cannot be assigned because its copy assignment
// operator is implicitly deleted"
CompressorLZ4& operator=(const CompressorLZ4& r) {
if(&r == this) return *this;
return *this;
}
virtual ~CompressorLZ4() {
//log::emerg("CompressorLZ4()::dtor", "call");
}
void ResetThreadLocalStorage();
Status Compress(char *raw_in,
uint64_t size_raw_in,
char **compressed_out,
uint64_t *size_compressed_out
);
bool IsUncompressionDone(uint64_t size_source);
Status Uncompress(char *source,
uint64_t size_source,
char **dest,
uint64_t *size_dest,
char **frame_out,
uint64_t *size_frame_out,
bool do_memory_allocation=true
);
Status UncompressByteArray(ByteArray& value,
bool do_checksum_verification,
ByteArray* value_uncompressed);
void DisableCompressionInFrameHeader(char* frame) {
for (uint64_t i = 0; i < size_frame_header(); i++) frame[i] = 0;
}
bool HasFrameHeaderDisabledCompression(char *frame) {
for (uint64_t i = 0; i < size_frame_header(); i++) {
if(frame[i] != 0) return false;
}
return true;
}
uint64_t thread_local_handler(std::map<std::thread::id, uint64_t>& status,
std::mutex& mutex,
uint64_t value,
bool apply);
uint64_t size_compressed() { return ts_compress_.get(); }
uint64_t MaxInputSize() {
return LZ4_MAX_INPUT_SIZE;
}
uint64_t size_frame_header() { return 8; };
uint64_t size_uncompressed_frame(uint64_t size_data) { return size_data + 8; }
void AdjustCompressedSize(int64_t inc) {
int64_t size = ts_compress_.get() + inc;
ts_compress_.put(size);
}
private:
ThreadStorage ts_compress_;
ThreadStorage ts_uncompress_;
CRC32 crc32_;
};
};
#endif // KINGDB_COMPRESSOR_H_
================================================
FILE: algorithm/crc32c.cc
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
// The code below was copied from zlib and LevelDB. A few changes were
// applied to make it self-sufficient and part of KingDB.
// zlib.h -- interface of the 'zlib' general purpose compression library
// version 1.2.8, April 28th, 2013
// Copyright (C) 1995-2013 Jean-loup Gailly and Mark Adler
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
//
// A portable implementation of crc32c, optimized to handle
// four bytes at a time.
#include "algorithm/crc32c.h"
#include <stdint.h>
namespace kdb {
namespace crc32c {
static const uint32_t table0_[256] = {
0x00000000, 0xf26b8303, 0xe13b70f7, 0x1350f3f4,
0xc79a971f, 0x35f1141c, 0x26a1e7e8, 0xd4ca64eb,
0x8ad958cf, 0x78b2dbcc, 0x6be22838, 0x9989ab3b,
0x4d43cfd0, 0xbf284cd3, 0xac78bf27, 0x5e133c24,
0x105ec76f, 0xe235446c, 0xf165b798, 0x030e349b,
0xd7c45070, 0x25afd373, 0x36ff2087, 0xc494a384,
0x9a879fa0, 0x68ec1ca3, 0x7bbcef57, 0x89d76c54,
0x5d1d08bf, 0xaf768bbc, 0xbc267848, 0x4e4dfb4b,
0x20bd8ede, 0xd2d60ddd, 0xc186fe29, 0x33ed7d2a,
0xe72719c1, 0x154c9ac2, 0x061c6936, 0xf477ea35,
0xaa64d611, 0x580f5512, 0x4b5fa6e6, 0xb93425e5,
0x6dfe410e, 0x9f95c20d, 0x8cc531f9, 0x7eaeb2fa,
0x30e349b1, 0xc288cab2, 0xd1d83946, 0x23b3ba45,
0xf779deae, 0x05125dad, 0x1642ae59, 0xe4292d5a,
0xba3a117e, 0x4851927d, 0x5b016189, 0xa96ae28a,
0x7da08661, 0x8fcb0562, 0x9c9bf696, 0x6ef07595,
0x417b1dbc, 0xb3109ebf, 0xa0406d4b, 0x522bee48,
0x86e18aa3, 0x748a09a0, 0x67dafa54, 0x95b17957,
0xcba24573, 0x39c9c670, 0x2a993584, 0xd8f2b687,
0x0c38d26c, 0xfe53516f, 0xed03a29b, 0x1f682198,
0x5125dad3, 0xa34e59d0, 0xb01eaa24, 0x42752927,
0x96bf4dcc, 0x64d4cecf, 0x77843d3b, 0x85efbe38,
0xdbfc821c, 0x2997011f, 0x3ac7f2eb, 0xc8ac71e8,
0x1c661503, 0xee0d9600, 0xfd5d65f4, 0x0f36e6f7,
0x61c69362, 0x93ad1061, 0x80fde395, 0x72966096,
0xa65c047d, 0x5437877e, 0x4767748a, 0xb50cf789,
0xeb1fcbad, 0x197448ae, 0x0a24bb5a, 0xf84f3859,
0x2c855cb2, 0xdeeedfb1, 0xcdbe2c45, 0x3fd5af46,
0x7198540d, 0x83f3d70e, 0x90a324fa, 0x62c8a7f9,
0xb602c312, 0x44694011, 0x5739b3e5, 0xa55230e6,
0xfb410cc2, 0x092a8fc1, 0x1a7a7c35, 0xe811ff36,
0x3cdb9bdd, 0xceb018de, 0xdde0eb2a, 0x2f8b6829,
0x82f63b78, 0x709db87b, 0x63cd4b8f, 0x91a6c88c,
0x456cac67, 0xb7072f64, 0xa457dc90, 0x563c5f93,
0x082f63b7, 0xfa44e0b4, 0xe9141340, 0x1b7f9043,
0xcfb5f4a8, 0x3dde77ab, 0x2e8e845f, 0xdce5075c,
0x92a8fc17, 0x60c37f14, 0x73938ce0, 0x81f80fe3,
0x55326b08, 0xa759e80b, 0xb4091bff, 0x466298fc,
0x1871a4d8, 0xea1a27db, 0xf94ad42f, 0x0b21572c,
0xdfeb33c7, 0x2d80b0c4, 0x3ed04330, 0xccbbc033,
0xa24bb5a6, 0x502036a5, 0x4370c551, 0xb11b4652,
0x65d122b9, 0x97baa1ba, 0x84ea524e, 0x7681d14d,
0x2892ed69, 0xdaf96e6a, 0xc9a99d9e, 0x3bc21e9d,
0xef087a76, 0x1d63f975, 0x0e330a81, 0xfc588982,
0xb21572c9, 0x407ef1ca, 0x532e023e, 0xa145813d,
0x758fe5d6, 0x87e466d5, 0x94b49521, 0x66df1622,
0x38cc2a06, 0xcaa7a905, 0xd9f75af1, 0x2b9cd9f2,
0xff56bd19, 0x0d3d3e1a, 0x1e6dcdee, 0xec064eed,
0xc38d26c4, 0x31e6a5c7, 0x22b65633, 0xd0ddd530,
0x0417b1db, 0xf67c32d8, 0xe52cc12c, 0x1747422f,
0x49547e0b, 0xbb3ffd08, 0xa86f0efc, 0x5a048dff,
0x8ecee914, 0x7ca56a17, 0x6ff599e3, 0x9d9e1ae0,
0xd3d3e1ab, 0x21b862a8, 0x32e8915c, 0xc083125f,
0x144976b4, 0xe622f5b7, 0xf5720643, 0x07198540,
0x590ab964, 0xab613a67, 0xb831c993, 0x4a5a4a90,
0x9e902e7b, 0x6cfbad78, 0x7fab5e8c, 0x8dc0dd8f,
0xe330a81a, 0x115b2b19, 0x020bd8ed, 0xf0605bee,
0x24aa3f05, 0xd6c1bc06, 0xc5914ff2, 0x37faccf1,
0x69e9f0d5, 0x9b8273d6, 0x88d28022, 0x7ab90321,
0xae7367ca, 0x5c18e4c9, 0x4f48173d, 0xbd23943e,
0xf36e6f75, 0x0105ec76, 0x12551f82, 0xe03e9c81,
0x34f4f86a, 0xc69f7b69, 0xd5cf889d, 0x27a40b9e,
0x79b737ba, 0x8bdcb4b9, 0x988c474d, 0x6ae7c44e,
0xbe2da0a5, 0x4c4623a6, 0x5f16d052, 0xad7d5351
};
static const uint32_t table1_[256] = {
0x00000000, 0x13a29877, 0x274530ee, 0x34e7a899,
0x4e8a61dc, 0x5d28f9ab, 0x69cf5132, 0x7a6dc945,
0x9d14c3b8, 0x8eb65bcf, 0xba51f356, 0xa9f36b21,
0xd39ea264, 0xc03c3a13, 0xf4db928a, 0xe7790afd,
0x3fc5f181, 0x2c6769f6, 0x1880c16f, 0x0b225918,
0x714f905d, 0x62ed082a, 0x560aa0b3, 0x45a838c4,
0xa2d13239, 0xb173aa4e, 0x859402d7, 0x96369aa0,
0xec5b53e5, 0xfff9cb92, 0xcb1e630b, 0xd8bcfb7c,
0x7f8be302, 0x6c297b75, 0x58ced3ec, 0x4b6c4b9b,
0x310182de, 0x22a31aa9, 0x1644b230, 0x05e62a47,
0xe29f20ba, 0xf13db8cd, 0xc5da1054, 0xd6788823,
0xac154166, 0xbfb7d911, 0x8b507188, 0x98f2e9ff,
0x404e1283, 0x53ec8af4, 0x670b226d, 0x74a9ba1a,
0x0ec4735f, 0x1d66eb28, 0x298143b1, 0x3a23dbc6,
0xdd5ad13b, 0xcef8494c, 0xfa1fe1d5, 0xe9bd79a2,
0x93d0b0e7, 0x80722890, 0xb4958009, 0xa737187e,
0xff17c604, 0xecb55e73, 0xd852f6ea, 0xcbf06e9d,
0xb19da7d8, 0xa23f3faf, 0x96d89736, 0x857a0f41,
0x620305bc, 0x71a19dcb, 0x45463552, 0x56e4ad25,
0x2c896460, 0x3f2bfc17, 0x0bcc548e, 0x186eccf9,
0xc0d23785, 0xd370aff2, 0xe797076b, 0xf4359f1c,
0x8e585659, 0x9dface2e, 0xa91d66b7, 0xbabffec0,
0x5dc6f43d, 0x4e646c4a, 0x7a83c4d3, 0x69215ca4,
0x134c95e1, 0x00ee0d96, 0x3409a50f, 0x27ab3d78,
0x809c2506, 0x933ebd71, 0xa7d915e8, 0xb47b8d9f,
0xce1644da, 0xddb4dcad, 0xe9537434, 0xfaf1ec43,
0x1d88e6be, 0x0e2a7ec9, 0x3acdd650, 0x296f4e27,
0x53028762, 0x40a01f15, 0x7447b78c, 0x67e52ffb,
0xbf59d487, 0xacfb4cf0, 0x981ce469, 0x8bbe7c1e,
0xf1d3b55b, 0xe2712d2c, 0xd69685b5, 0xc5341dc2,
0x224d173f, 0x31ef8f48, 0x050827d1, 0x16aabfa6,
0x6cc776e3, 0x7f65ee94, 0x4b82460d, 0x5820de7a,
0xfbc3faf9, 0xe861628e, 0xdc86ca17, 0xcf245260,
0xb5499b25, 0xa6eb0352, 0x920cabcb, 0x81ae33bc,
0x66d73941, 0x7575a136, 0x419209af, 0x523091d8,
0x285d589d, 0x3bffc0ea, 0x0f186873, 0x1cbaf004,
0xc4060b78, 0xd7a4930f, 0xe3433b96, 0xf0e1a3e1,
0x8a8c6aa4, 0x992ef2d3, 0xadc95a4a, 0xbe6bc23d,
0x5912c8c0, 0x4ab050b7, 0x7e57f82e, 0x6df56059,
0x1798a91c, 0x043a316b, 0x30dd99f2, 0x237f0185,
0x844819fb, 0x97ea818c, 0xa30d2915, 0xb0afb162,
0xcac27827, 0xd960e050, 0xed8748c9, 0xfe25d0be,
0x195cda43, 0x0afe4234, 0x3e19eaad, 0x2dbb72da,
0x57d6bb9f, 0x447423e8, 0x70938b71, 0x63311306,
0xbb8de87a, 0xa82f700d, 0x9cc8d894, 0x8f6a40e3,
0xf50789a6, 0xe6a511d1, 0xd242b948, 0xc1e0213f,
0x26992bc2, 0x353bb3b5, 0x01dc1b2c, 0x127e835b,
0x68134a1e, 0x7bb1d269, 0x4f567af0, 0x5cf4e287,
0x04d43cfd, 0x1776a48a, 0x23910c13, 0x30339464,
0x4a5e5d21, 0x59fcc556, 0x6d1b6dcf, 0x7eb9f5b8,
0x99c0ff45, 0x8a626732, 0xbe85cfab, 0xad2757dc,
0xd74a9e99, 0xc4e806ee, 0xf00fae77, 0xe3ad3600,
0x3b11cd7c, 0x28b3550b, 0x1c54fd92, 0x0ff665e5,
0x759baca0, 0x663934d7, 0x52de9c4e, 0x417c0439,
0xa6050ec4, 0xb5a796b3, 0x81403e2a, 0x92e2a65d,
0xe88f6f18, 0xfb2df76f, 0xcfca5ff6, 0xdc68c781,
0x7b5fdfff, 0x68fd4788, 0x5c1aef11, 0x4fb87766,
0x35d5be23, 0x26772654, 0x12908ecd, 0x013216ba,
0xe64b1c47, 0xf5e98430, 0xc10e2ca9, 0xd2acb4de,
0xa8c17d9b, 0xbb63e5ec, 0x8f844d75, 0x9c26d502,
0x449a2e7e, 0x5738b609, 0x63df1e90, 0x707d86e7,
0x0a104fa2, 0x19b2d7d5, 0x2d557f4c, 0x3ef7e73b,
0xd98eedc6, 0xca2c75b1, 0xfecbdd28, 0xed69455f,
0x97048c1a, 0x84a6146d, 0xb041bcf4, 0xa3e32483
};
static const uint32_t table2_[256] = {
0x00000000, 0xa541927e, 0x4f6f520d, 0xea2ec073,
0x9edea41a, 0x3b9f3664, 0xd1b1f617, 0x74f06469,
0x38513ec5, 0x9d10acbb, 0x773e6cc8, 0xd27ffeb6,
0xa68f9adf, 0x03ce08a1, 0xe9e0c8d2, 0x4ca15aac,
0x70a27d8a, 0xd5e3eff4, 0x3fcd2f87, 0x9a8cbdf9,
0xee7cd990, 0x4b3d4bee, 0xa1138b9d, 0x045219e3,
0x48f3434f, 0xedb2d131, 0x079c1142, 0xa2dd833c,
0xd62de755, 0x736c752b, 0x9942b558, 0x3c032726,
0xe144fb14, 0x4405696a, 0xae2ba919, 0x0b6a3b67,
0x7f9a5f0e, 0xdadbcd70, 0x30f50d03, 0x95b49f7d,
0xd915c5d1, 0x7c5457af, 0x967a97dc, 0x333b05a2,
0x47cb61cb, 0xe28af3b5, 0x08a433c6, 0xade5a1b8,
0x91e6869e, 0x34a714e0, 0xde89d493, 0x7bc846ed,
0x0f382284, 0xaa79b0fa, 0x40577089, 0xe516e2f7,
0xa9b7b85b, 0x0cf62a25, 0xe6d8ea56, 0x43997828,
0x37691c41, 0x92288e3f, 0x78064e4c, 0xdd47dc32,
0xc76580d9, 0x622412a7, 0x880ad2d4, 0x2d4b40aa,
0x59bb24c3, 0xfcfab6bd, 0x16d476ce, 0xb395e4b0,
0xff34be1c, 0x5a752c62, 0xb05bec11, 0x151a7e6f,
0x61ea1a06, 0xc4ab8878, 0x2e85480b, 0x8bc4da75,
0xb7c7fd53, 0x12866f2d, 0xf8a8af5e, 0x5de93d20,
0x29195949, 0x8c58cb37, 0x66760b44, 0xc337993a,
0x8f96c396, 0x2ad751e8, 0xc0f9919b, 0x65b803e5,
0x1148678c, 0xb409f5f2, 0x5e273581, 0xfb66a7ff,
0x26217bcd, 0x8360e9b3, 0x694e29c0, 0xcc0fbbbe,
0xb8ffdfd7, 0x1dbe4da9, 0xf7908dda, 0x52d11fa4,
0x1e704508, 0xbb31d776, 0x511f1705, 0xf45e857b,
0x80aee112, 0x25ef736c, 0xcfc1b31f, 0x6a802161,
0x56830647, 0xf3c29439, 0x19ec544a, 0xbcadc634,
0xc85da25d, 0x6d1c3023, 0x8732f050, 0x2273622e,
0x6ed23882, 0xcb93aafc, 0x21bd6a8f, 0x84fcf8f1,
0xf00c9c98, 0x554d0ee6, 0xbf63ce95, 0x1a225ceb,
0x8b277743, 0x2e66e53d, 0xc448254e, 0x6109b730,
0x15f9d359, 0xb0b84127, 0x5a968154, 0xffd7132a,
0xb3764986, 0x1637dbf8, 0xfc191b8b, 0x595889f5,
0x2da8ed9c, 0x88e97fe2, 0x62c7bf91, 0xc7862def,
0xfb850ac9, 0x5ec498b7, 0xb4ea58c4, 0x11abcaba,
0x655baed3, 0xc01a3cad, 0x2a34fcde, 0x8f756ea0,
0xc3d4340c, 0x6695a672, 0x8cbb6601, 0x29faf47f,
0x5d0a9016, 0xf84b0268, 0x1265c21b, 0xb7245065,
0x6a638c57, 0xcf221e29, 0x250cde5a, 0x804d4c24,
0xf4bd284d, 0x51fcba33, 0xbbd27a40, 0x1e93e83e,
0x5232b292, 0xf77320ec, 0x1d5de09f, 0xb81c72e1,
0xccec1688, 0x69ad84f6, 0x83834485, 0x26c2d6fb,
0x1ac1f1dd, 0xbf8063a3, 0x55aea3d0, 0xf0ef31ae,
0x841f55c7, 0x215ec7b9, 0xcb7007ca, 0x6e3195b4,
0x2290cf18, 0x87d15d66, 0x6dff9d15, 0xc8be0f6b,
0xbc4e6b02, 0x190ff97c, 0xf321390f, 0x5660ab71,
0x4c42f79a, 0xe90365e4, 0x032da597, 0xa66c37e9,
0xd29c5380, 0x77ddc1fe, 0x9df3018d, 0x38b293f3,
0x7413c95f, 0xd1525b21, 0x3b7c9b52, 0x9e3d092c,
0xeacd6d45, 0x4f8cff3b, 0xa5a23f48, 0x00e3ad36,
0x3ce08a10, 0x99a1186e, 0x738fd81d, 0xd6ce4a63,
0xa23e2e0a, 0x077fbc74, 0xed517c07, 0x4810ee79,
0x04b1b4d5, 0xa1f026ab, 0x4bdee6d8, 0xee9f74a6,
0x9a6f10cf, 0x3f2e82b1, 0xd50042c2, 0x7041d0bc,
0xad060c8e, 0x08479ef0, 0xe2695e83, 0x4728ccfd,
0x33d8a894, 0x96993aea, 0x7cb7fa99, 0xd9f668e7,
0x9557324b, 0x3016a035, 0xda386046, 0x7f79f238,
0x0b899651, 0xaec8042f, 0x44e6c45c, 0xe1a75622,
0xdda47104, 0x78e5e37a, 0x92cb2309, 0x378ab177,
0x437ad51e, 0xe63b4760, 0x0c158713, 0xa954156d,
0xe5f54fc1, 0x40b4ddbf, 0xaa9a1dcc, 0x0fdb8fb2,
0x7b2bebdb, 0xde6a79a5, 0x3444b9d6, 0x91052ba8
};
static const uint32_t table3_[256] = {
0x00000000, 0xdd45aab8, 0xbf672381, 0x62228939,
0x7b2231f3, 0xa6679b4b, 0xc4451272, 0x1900b8ca,
0xf64463e6, 0x2b01c95e, 0x49234067, 0x9466eadf,
0x8d665215, 0x5023f8ad, 0x32017194, 0xef44db2c,
0xe964b13d, 0x34211b85, 0x560392bc, 0x8b463804,
0x924680ce, 0x4f032a76, 0x2d21a34f, 0xf06409f7,
0x1f20d2db, 0xc2657863, 0xa047f15a, 0x7d025be2,
0x6402e328, 0xb9474990, 0xdb65c0a9, 0x06206a11,
0xd725148b, 0x0a60be33, 0x6842370a, 0xb5079db2,
0xac072578, 0x71428fc0, 0x136006f9, 0xce25ac41,
0x2161776d, 0xfc24ddd5, 0x9e0654ec, 0x4343fe54,
0x5a43469e, 0x8706ec26, 0xe524651f, 0x3861cfa7,
0x3e41a5b6, 0xe3040f0e, 0x81268637, 0x5c632c8f,
0x45639445, 0x98263efd, 0xfa04b7c4, 0x27411d7c,
0xc805c650, 0x15406ce8, 0x7762e5d1, 0xaa274f69,
0xb327f7a3, 0x6e625d1b, 0x0c40d422, 0xd1057e9a,
0xaba65fe7, 0x76e3f55f, 0x14c17c66, 0xc984d6de,
0xd0846e14, 0x0dc1c4ac, 0x6fe34d95, 0xb2a6e72d,
0x5de23c01, 0x80a796b9, 0xe2851f80, 0x3fc0b538,
0x26c00df2, 0xfb85a74a, 0x99a72e73, 0x44e284cb,
0x42c2eeda, 0x9f874462, 0xfda5cd5b, 0x20e067e3,
0x39e0df29, 0xe4a57591, 0x8687fca8, 0x5bc25610,
0xb4868d3c, 0x69c32784, 0x0be1aebd, 0xd6a40405,
0xcfa4bccf, 0x12e11677, 0x70c39f4e, 0xad8635f6,
0x7c834b6c, 0xa1c6e1d4, 0xc3e468ed, 0x1ea1c255,
0x07a17a9f, 0xdae4d027, 0xb8c6591e, 0x6583f3a6,
0x8ac7288a, 0x57828232, 0x35a00b0b, 0xe8e5a1b3,
0xf1e51979, 0x2ca0b3c1, 0x4e823af8, 0x93c79040,
0x95e7fa51, 0x48a250e9, 0x2a80d9d0, 0xf7c57368,
0xeec5cba2, 0x3380611a, 0x51a2e823, 0x8ce7429b,
0x63a399b7, 0xbee6330f, 0xdcc4ba36, 0x0181108e,
0x1881a844, 0xc5c402fc, 0xa7e68bc5, 0x7aa3217d,
0x52a0c93f, 0x8fe56387, 0xedc7eabe, 0x30824006,
0x2982f8cc, 0xf4c75274, 0x96e5db4d, 0x4ba071f5,
0xa4e4aad9, 0x79a10061, 0x1b838958, 0xc6c623e0,
0xdfc69b2a, 0x02833192, 0x60a1b8ab, 0xbde41213,
0xbbc47802, 0x6681d2ba, 0x04a35b83, 0xd9e6f13b,
0xc0e649f1, 0x1da3e349, 0x7f816a70, 0xa2c4c0c8,
0x4d801be4, 0x90c5b15c, 0xf2e73865, 0x2fa292dd,
0x36a22a17, 0xebe780af, 0x89c50996, 0x5480a32e,
0x8585ddb4, 0x58c0770c, 0x3ae2fe35, 0xe7a7548d,
0xfea7ec47, 0x23e246ff, 0x41c0cfc6, 0x9c85657e,
0x73c1be52, 0xae8414ea, 0xcca69dd3, 0x11e3376b,
0x08e38fa1, 0xd5a62519, 0xb784ac20, 0x6ac10698,
0x6ce16c89, 0xb1a4c631, 0xd3864f08, 0x0ec3e5b0,
0x17c35d7a, 0xca86f7c2, 0xa8a47efb, 0x75e1d443,
0x9aa50f6f, 0x47e0a5d7, 0x25c22cee, 0xf8878656,
0xe1873e9c, 0x3cc29424, 0x5ee01d1d, 0x83a5b7a5,
0xf90696d8, 0x24433c60, 0x4661b559, 0x9b241fe1,
0x8224a72b, 0x5f610d93, 0x3d4384aa, 0xe0062e12,
0x0f42f53e, 0xd2075f86, 0xb025d6bf, 0x6d607c07,
0x7460c4cd, 0xa9256e75, 0xcb07e74c, 0x16424df4,
0x106227e5, 0xcd278d5d, 0xaf050464, 0x7240aedc,
0x6b401616, 0xb605bcae, 0xd4273597, 0x09629f2f,
0xe6264403, 0x3b63eebb, 0x59416782, 0x8404cd3a,
0x9d0475f0, 0x4041df48, 0x22635671, 0xff26fcc9,
0x2e238253, 0xf36628eb, 0x9144a1d2, 0x4c010b6a,
0x5501b3a0, 0x88441918, 0xea669021, 0x37233a99,
0xd867e1b5, 0x05224b0d, 0x6700c234, 0xba45688c,
0xa345d046, 0x7e007afe, 0x1c22f3c7, 0xc167597f,
0xc747336e, 0x1a0299d6, 0x782010ef, 0xa565ba57,
0xbc65029d, 0x6120a825, 0x0302211c, 0xde478ba4,
0x31035088, 0xec46fa30, 0x8e647309, 0x5321d9b1,
0x4a21617b, 0x9764cbc3, 0xf54642fa, 0x2803e842
};
// Used to fetch a naturally-aligned 32-bit word in little endian byte-order
static inline uint32_t LE_LOAD32(const uint8_t *p) {
return DecodeFixed32(reinterpret_cast<const char*>(p));
}
uint32_t Extend(uint32_t crc, const char* buf, size_t size) {
const uint8_t *p = reinterpret_cast<const uint8_t *>(buf);
const uint8_t *e = p + size;
uint32_t l = crc ^ 0xffffffffu;
#define STEP1 do { \
int c = (l & 0xff) ^ *p++; \
l = table0_[c] ^ (l >> 8); \
} while (0)
#define STEP4 do { \
uint32_t c = l ^ LE_LOAD32(p); \
p += 4; \
l = table3_[c & 0xff] ^ \
table2_[(c >> 8) & 0xff] ^ \
table1_[(c >> 16) & 0xff] ^ \
table0_[c >> 24]; \
} while (0)
// Point x at first 4-byte aligned byte in string. This might be
// just past the end of the string.
const uintptr_t pval = reinterpret_cast<uintptr_t>(p);
const uint8_t* x = reinterpret_cast<const uint8_t*>(((pval + 3) >> 2) << 2);
if (x <= e) {
// Process bytes until finished or p is 4-byte aligned
while (p != x) {
STEP1;
}
}
// Process bytes 16 at a time
while ((e-p) >= 16) {
STEP4; STEP4; STEP4; STEP4;
}
// Process bytes 4 at a time
while ((e-p) >= 4) {
STEP4;
}
// Process the last few bytes
while (p != e) {
STEP1;
}
#undef STEP4
#undef STEP1
return l ^ 0xffffffffu;
}
// For crc32_combine
ulong gf2_matrix_times (ulong *mat, ulong vec)
{
ulong sum = 0;
while (vec) {
if (vec & 1)
sum ^= *mat;
vec >>= 1;
mat++;
}
return sum;
}
void gf2_matrix_square (ulong *square, ulong *mat)
{
int n;
for (n = 0; n < GF2_DIM; n++)
square[n] = gf2_matrix_times(mat, mat[n]);
}
ulong Combine(ulong crc1, ulong crc2, ulong len2)
{
// WARNING:
// This method is *very* slow. It was used to combine the crc32 of the
// header and the crc32 of the content of an entry, and after doing some
// profiling, it turned out that Combine() was accountable for 40% of all
// the CPU usage in KingDB during write-intensive workloads. The data format
// was then changed appropriately so that the use of Combine() would no
// longer be needed. If Combine() needs to be used again, its usage must
// *absolutely* be carefully profiled and monitored, as it can have a
// dramatic impact on performance.
//
// NOTE: The original zlib code had a polynomial 'odd[0]' equal to
// 0xedb88320L, which is the value for the CRC32 checksum.
// I changed it for 0x82f63b78, which is the reversed polynomial for
// the CRC32C (Castagnoli) checksum, which is the one used in the
// code that I copied from LevelDB. The polynomial was found in:
// http://en.wikipedia.org/wiki/Cyclic_redundancy_check
int n;
ulong row;
ulong even[GF2_DIM]; /* even-power-of-two zeros operator */
ulong odd[GF2_DIM]; /* odd-power-of-two zeros operator */
/* degenerate case */
if (len2 == 0)
return crc1;
/* put operator for one zero bit in odd */
odd[0] = 0x82f63b78; /* CRC-32 polynomial */
row = 1;
for (n = 1; n < GF2_DIM; n++) {
odd[n] = row;
row <<= 1;
}
/* put operator for two zero bits in even */
gf2_matrix_square(even, odd);
/* put operator for four zero bits in odd */
gf2_matrix_square(odd, even);
/* apply len2 zeros to crc1 (first square will put the operator for one
zero byte, eight zero bits, in even) */
do {
/* apply zeros operator for this bit of len2 */
gf2_matrix_square(even, odd);
if (len2 & 1)
crc1 = gf2_matrix_times(even, crc1);
len2 >>= 1;
/* if no more bits set, then done */
if (len2 == 0)
break;
/* another iteration of the loop with odd and even swapped */
gf2_matrix_square(odd, even);
if (len2 & 1)
crc1 = gf2_matrix_times(odd, crc1);
len2 >>= 1;
/* if no more bits set, then done */
} while (len2 != 0);
/* return combined crc */
crc1 ^= crc2;
return crc1;
}
/* 8-bit CRC with polynomial x^8+x^6+x^3+x^2+1, 0x14D.
Chosen based on Koopman, et al. (0xA6 in his notation = 0x14D >> 1):
http://www.ece.cmu.edu/~koopman/roses/dsn04/koopman04_crc_poly_embedded.pdf
*/
static unsigned char crc8_table[] = {
0x00, 0x3e, 0x7c, 0x42, 0xf8, 0xc6, 0x84, 0xba, 0x95, 0xab, 0xe9, 0xd7,
0x6d, 0x53, 0x11, 0x2f, 0x4f, 0x71, 0x33, 0x0d, 0xb7, 0x89, 0xcb, 0xf5,
0xda, 0xe4, 0xa6, 0x98, 0x22, 0x1c, 0x5e, 0x60, 0x9e, 0xa0, 0xe2, 0xdc,
0x66, 0x58, 0x1a, 0x24, 0x0b, 0x35, 0x77, 0x49, 0xf3, 0xcd, 0x8f, 0xb1,
0xd1, 0xef, 0xad, 0x93, 0x29, 0x17, 0x55, 0x6b, 0x44, 0x7a, 0x38, 0x06,
0xbc, 0x82, 0xc0, 0xfe, 0x59, 0x67, 0x25, 0x1b, 0xa1, 0x9f, 0xdd, 0xe3,
0xcc, 0xf2, 0xb0, 0x8e, 0x34, 0x0a, 0x48, 0x76, 0x16, 0x28, 0x6a, 0x54,
0xee, 0xd0, 0x92, 0xac, 0x83, 0xbd, 0xff, 0xc1, 0x7b, 0x45, 0x07, 0x39,
0xc7, 0xf9, 0xbb, 0x85, 0x3f, 0x01, 0x43, 0x7d, 0x52, 0x6c, 0x2e, 0x10,
0xaa, 0x94, 0xd6, 0xe8, 0x88, 0xb6, 0xf4, 0xca, 0x70, 0x4e, 0x0c, 0x32,
0x1d, 0x23, 0x61, 0x5f, 0xe5, 0xdb, 0x99, 0xa7, 0xb2, 0x8c, 0xce, 0xf0,
0x4a, 0x74, 0x36, 0x08, 0x27, 0x19, 0x5b, 0x65, 0xdf, 0xe1, 0xa3, 0x9d,
0xfd, 0xc3, 0x81, 0xbf, 0x05, 0x3b, 0x79, 0x47, 0x68, 0x56, 0x14, 0x2a,
0x90, 0xae, 0xec, 0xd2, 0x2c, 0x12, 0x50, 0x6e, 0xd4, 0xea, 0xa8, 0x96,
0xb9, 0x87, 0xc5, 0xfb, 0x41, 0x7f, 0x3d, 0x03, 0x63, 0x5d, 0x1f, 0x21,
0x9b, 0xa5, 0xe7, 0xd9, 0xf6, 0xc8, 0x8a, 0xb4, 0x0e, 0x30, 0x72, 0x4c,
0xeb, 0xd5, 0x97, 0xa9, 0x13, 0x2d, 0x6f, 0x51, 0x7e, 0x40, 0x02, 0x3c,
0x86, 0xb8, 0xfa, 0xc4, 0xa4, 0x9a, 0xd8, 0xe6, 0x5c, 0x62, 0x20, 0x1e,
0x31, 0x0f, 0x4d, 0x73, 0xc9, 0xf7, 0xb5, 0x8b, 0x75, 0x4b, 0x09, 0x37,
0x8d, 0xb3, 0xf1, 0xcf, 0xe0, 0xde, 0x9c, 0xa2, 0x18, 0x26, 0x64, 0x5a,
0x3a, 0x04, 0x46, 0x78, 0xc2, 0xfc, 0xbe, 0x80, 0xaf, 0x91, 0xd3, 0xed,
0x57, 0x69, 0x2b, 0x15};
uint8_t crc8(unsigned crc, unsigned char *data, size_t len)
{
unsigned char *end;
if (len == 0)
return crc;
crc ^= 0xff;
end = data + len;
do {
crc = crc8_table[crc ^ *data++];
} while (data < end);
return crc ^ 0xff;
}
uint8_t crc8(unsigned crc, char *data, size_t len)
{
return crc8(crc, reinterpret_cast<unsigned char*>(data), len);
}
} // namespace crc32c
} // namespace kdb
================================================
FILE: algorithm/crc32c.h
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
// The code below was copied from zlib and LevelDB. A few changes were
// applied to make it self-sufficient and part of KingDB.
// zlib.h -- interface of the 'zlib' general purpose compression library
// version 1.2.8, April 28th, 2013
// Copyright (C) 1995-2013 Jean-loup Gailly and Mark Adler
// Copyright (c) 2011 The LevelDB Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file. See the AUTHORS file for names of contributors.
#ifndef KINGDB_CRC32_H_
#define KINGDB_CRC32_H_
#include "util/debug.h"
#include <stddef.h>
#include <stdint.h>
#include <string.h>
#include "util/logger.h"
#include "algorithm/endian.h"
#include "algorithm/coding.h"
#include "thread/threadstorage.h"
namespace kdb {
namespace crc32c {
// Return the crc32c of concat(A, data[0,n-1]) where init_crc is the
// crc32c of some string A. Extend() is often used to maintain the
// crc32c of a stream of data.
extern uint32_t Extend(uint32_t init_crc, const char* data, size_t n);
// Return the crc32c of data[0,n-1]
inline uint32_t Value(const char* data, size_t n) {
return Extend(0, data, n);
}
static const uint32_t kMaskDelta = 0xa282ead8ul;
// Return a masked representation of crc.
//
// Motivation: it is problematic to compute the CRC of a string that
// contains embedded CRCs. Therefore we recommend that CRCs stored
// somewhere (e.g., in files) should be masked before being stored.
inline uint32_t Mask(uint32_t crc) {
// Rotate right by 15 bits and add a constant.
return ((crc >> 15) | (crc << 17)) + kMaskDelta;
}
// Return the crc whose masked representation is masked_crc.
inline uint32_t Unmask(uint32_t masked_crc) {
uint32_t rot = masked_crc - kMaskDelta;
return ((rot >> 17) | (rot << 15));
}
// For crc32_combine
typedef uint32_t ulong;
typedef int64_t I64;
ulong Combine(ulong crc1, ulong crc2, ulong len2);
#define GF2_DIM 32
// 8-bit CRC
uint8_t crc8(unsigned crc, unsigned char *data, size_t len);
uint8_t crc8(unsigned crc, char *data, size_t len);
} // namespace crc32c
class CRC32 {
public:
CRC32() {}
~CRC32() {
}
// Added an empty copy assignment operator to avoid error messages of the type:
// "object of type '...' cannot be assigned because its copy assignment
// operator is implicitly deleted"
CRC32& operator=(const CRC32& r) {
if(&r == this) return *this;
return *this;
}
void stream(const char* data, size_t n) {
//log::trace("CRC32", "size: %zu", n);
uint64_t c = ts_.get();
uint32_t c32 = c;
uint32_t c_new = crc32c::Extend(c32, data, n);
ts_.put(c_new);
}
uint32_t get() { return ts_.get(); }
void put(uint32_t c32) { ts_.put(c32); }
void ResetThreadLocalStorage() { ts_.reset(); }
virtual uint64_t MaxInputSize() { return std::numeric_limits<int32_t>::max(); }
private:
kdb::ThreadStorage ts_;
};
} // namespace kdb
#endif // KINGDB_CRC32_H_
================================================
FILE: algorithm/endian.cc
================================================
#include "algorithm/endian.h"
namespace kdb {
endian_t getEndianness() {
if ((0xffffffff & 1) == kBytesLittleEndian) {
return kBytesLittleEndian;
} else if ((0xffffffff & 1) == kBytesBigEndian) {
return kBytesBigEndian;
} else if ((0xffffffff & 1) == kBytesLittleEndianWord) {
return kBytesLittleEndianWord;
} else if ((0xffffffff & 1) == kBytesBigEndianWord) {
return kBytesBigEndianWord;
}
return kBytesUnknownEndian;
}
const bool kLittleEndian = (getEndianness() == kBytesLittleEndian);
const bool kBigEndian = (getEndianness() == kBytesBigEndian);
} // namespace kdb
================================================
FILE: algorithm/endian.h
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
#ifndef KINGDB_ENDIAN_H_
#define KINGDB_ENDIAN_H_
#include "util/debug.h"
#include <stdint.h>
namespace kdb {
enum endian_t : uint32_t {
kBytesLittleEndian = 0x00000001, // byte-swapped little-endian
kBytesBigEndian = 0x01000000, // byte-swapped big-endian
kBytesLittleEndianWord = 0x00010000, // word-swapped little-endian
kBytesBigEndianWord = 0x00000100, // word-swapped big-endian
kBytesUnknownEndian = 0xffffffff
};
//TODO: linux compatibility
/*
constexpr endian_t getEndianness() {
if ((0xffffffff & 1) == kBytesLittleEndian) {
return kBytesLittleEndian;
} else if ((0xffffffff & 1) == kBytesBigEndian) {
return kBytesBigEndian;
} else if ((0xffffffff & 1) == kBytesLittleEndianWord) {
return kBytesLittleEndianWord;
} else if ((0xffffffff & 1) == kBytesBigEndianWord) {
return kBytesBigEndianWord;
}
return kBytesUnknownEndian;
}
*/
endian_t getEndianness();
extern const bool kLittleEndian;
extern const bool kBigEndian;
};
#endif // KINGDB_ENDIAN_H_
================================================
FILE: algorithm/hash.cc
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
#include "algorithm/hash.h"
namespace kdb {
uint64_t MurmurHash3::HashFunction(const char *data, uint32_t len) {
// NOTE: You may need to change the seed, which by default is 0
static char hash[16];
static uint64_t output;
// NOTE: Beware, the len in MurmurHash3_x64_128 is an 'int', not a 'uint32_t'
MurmurHash3_x64_128(data, len, 0, hash);
memcpy(&output, hash, 8);
return output;
}
uint64_t xxHash::HashFunction(const char *data, uint32_t len) {
// NOTE: You may need to change the seed, which by default is 0
return XXH64(data, len, 0);
}
Hash* MakeHash(HashType ht) {
if (ht == kMurmurHash3_64) {
return new MurmurHash3();
} else if (ht == kxxHash_64) {
return new xxHash();
} else {
log::emerg("Hash", "Unknown hashing function: [%d]", ht);
exit(-1);
}
}
} // namespace kdb
================================================
FILE: algorithm/hash.h
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
#ifndef KINGDB_HASH_H_
#define KINGDB_HASH_H_
#include "util/debug.h"
#include <string>
#include <cstdint>
#include <limits>
#include <string.h>
#include "util/logger.h"
#include "util/options.h"
#include "algorithm/murmurhash3.h"
#include "algorithm/xxhash.h"
namespace kdb {
class Hash {
public:
Hash() {}
virtual ~Hash() {}
virtual uint64_t HashFunction(const char *data, uint32_t len) = 0;
virtual uint64_t MaxInputSize() = 0;
};
class MurmurHash3: public Hash {
public:
MurmurHash3() {}
virtual ~MurmurHash3() {}
virtual uint64_t HashFunction(const char *data, uint32_t len);
virtual uint64_t MaxInputSize() { return std::numeric_limits<int32_t>::max(); }
};
class xxHash: public Hash {
public:
xxHash() {}
virtual ~xxHash() {}
virtual uint64_t HashFunction(const char *data, uint32_t len);
virtual uint64_t MaxInputSize() { return std::numeric_limits<int32_t>::max(); }
};
Hash* MakeHash(HashType ht);
} // namespace kdb
#endif // KINGDB_HASH_H_
================================================
FILE: algorithm/lz4.cc
================================================
/*
LZ4 - Fast LZ compression algorithm
Copyright (C) 2011-2014, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
/**************************************
Tuning parameters
**************************************/
/*
* HEAPMODE :
* Select how default compression functions will allocate memory for their hash table,
* in memory stack (0:default, fastest), or in memory heap (1:requires memory allocation (malloc)).
*/
#define HEAPMODE 0
/**************************************
CPU Feature Detection
**************************************/
/* 32 or 64 bits ? */
#if (defined(__x86_64__) || defined(_M_X64) || defined(_WIN64) \
|| defined(__powerpc64__) || defined(__powerpc64le__) \
|| defined(__ppc64__) || defined(__ppc64le__) \
|| defined(__PPC64__) || defined(__PPC64LE__) \
|| defined(__ia64) || defined(__itanium__) || defined(_M_IA64) ) /* Detects 64 bits mode */
# define LZ4_ARCH64 1
#else
# define LZ4_ARCH64 0
#endif
#define LZ4_32BITS (sizeof(void*)==4)
#define LZ4_64BITS (sizeof(void*)==8)
/*
* Little Endian or Big Endian ?
* Overwrite the #define below if you know your architecture endianess
*/
#include <stdlib.h> /* Apparently required to detect endianess */
#if defined (__GLIBC__)
# include <endian.h>
# if (__BYTE_ORDER == __BIG_ENDIAN)
# define LZ4_BIG_ENDIAN 1
# endif
#elif (defined(__BIG_ENDIAN__) || defined(__BIG_ENDIAN) || defined(_BIG_ENDIAN)) && !(defined(__LITTLE_ENDIAN__) || defined(__LITTLE_ENDIAN) || defined(_LITTLE_ENDIAN))
# define LZ4_BIG_ENDIAN 1
#elif defined(__sparc) || defined(__sparc__) \
|| defined(__powerpc__) || defined(__ppc__) || defined(__PPC__) \
|| defined(__hpux) || defined(__hppa) \
|| defined(_MIPSEB) || defined(__s390__)
# define LZ4_BIG_ENDIAN 1
#else
/* Little Endian assumed. PDP Endian and other very rare endian format are unsupported. */
#endif
/*
* Unaligned memory access is automatically enabled for "common" CPU, such as x86.
* For others CPU, such as ARM, the compiler may be more cautious, inserting unnecessary extra code to ensure aligned access property
* If you know your target CPU supports unaligned memory access, you want to force this option manually to improve performance
*/
#if defined(__ARM_FEATURE_UNALIGNED)
# define LZ4_FORCE_UNALIGNED_ACCESS 1
#endif
/* Define this parameter if your target system or compiler does not support hardware bit count */
#if defined(_MSC_VER) && defined(_WIN32_WCE) /* Visual Studio for Windows CE does not support Hardware bit count */
# define LZ4_FORCE_SW_BITCOUNT
#endif
/*
* BIG_ENDIAN_NATIVE_BUT_INCOMPATIBLE :
* This option may provide a small boost to performance for some big endian cpu, although probably modest.
* You may set this option to 1 if data will remain within closed environment.
* This option is useless on Little_Endian CPU (such as x86)
*/
/* #define BIG_ENDIAN_NATIVE_BUT_INCOMPATIBLE 1 */
/**************************************
Compiler Options
**************************************/
#if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */
/* "restrict" is a known keyword */
#else
# define restrict /* Disable restrict */
#endif
#ifdef _MSC_VER /* Visual Studio */
# define FORCE_INLINE static __forceinline
# include <intrin.h> /* For Visual 2005 */
# if LZ4_ARCH64 /* 64-bits */
# pragma intrinsic(_BitScanForward64) /* For Visual 2005 */
# pragma intrinsic(_BitScanReverse64) /* For Visual 2005 */
# else /* 32-bits */
# pragma intrinsic(_BitScanForward) /* For Visual 2005 */
# pragma intrinsic(_BitScanReverse) /* For Visual 2005 */
# endif
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
#else
# ifdef __GNUC__
# define FORCE_INLINE static inline __attribute__((always_inline))
# else
# define FORCE_INLINE static inline
# endif
#endif
#ifdef _MSC_VER /* Visual Studio */
# define lz4_bswap16(x) _byteswap_ushort(x)
#else
# define lz4_bswap16(x) ((unsigned short int) ((((x) >> 8) & 0xffu) | (((x) & 0xffu) << 8)))
#endif
#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
#if (GCC_VERSION >= 302) || (__INTEL_COMPILER >= 800) || defined(__clang__)
# define expect(expr,value) (__builtin_expect ((expr),(value)) )
#else
# define expect(expr,value) (expr)
#endif
#define likely(expr) expect((expr) != 0, 1)
#define unlikely(expr) expect((expr) != 0, 0)
/**************************************
Memory routines
**************************************/
#include <stdlib.h> /* malloc, calloc, free */
#define ALLOCATOR(n,s) calloc(n,s)
#define FREEMEM free
#include <string.h> /* memset, memcpy */
#define MEM_INIT memset
/**************************************
Includes
**************************************/
#include "algorithm/lz4.h"
/**************************************
Basic Types
**************************************/
#if defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */
# include <stdint.h>
typedef uint8_t BYTE;
typedef uint16_t U16;
typedef uint32_t U32;
typedef int32_t S32;
typedef uint64_t U64;
#else
typedef unsigned char BYTE;
typedef unsigned short U16;
typedef unsigned int U32;
typedef signed int S32;
typedef unsigned long long U64;
#endif
#if defined(__GNUC__) && !defined(LZ4_FORCE_UNALIGNED_ACCESS)
# define _PACKED __attribute__ ((packed))
#else
# define _PACKED
#endif
#if !defined(LZ4_FORCE_UNALIGNED_ACCESS) && !defined(__GNUC__)
# if defined(__IBMC__) || defined(__SUNPRO_C) || defined(__SUNPRO_CC)
# pragma pack(1)
# else
# pragma pack(push, 1)
# endif
#endif
typedef struct { U16 v; } _PACKED U16_S;
typedef struct { U32 v; } _PACKED U32_S;
typedef struct { U64 v; } _PACKED U64_S;
typedef struct {size_t v;} _PACKED size_t_S;
#if !defined(LZ4_FORCE_UNALIGNED_ACCESS) && !defined(__GNUC__)
# if defined(__SUNPRO_C) || defined(__SUNPRO_CC)
# pragma pack(0)
# else
# pragma pack(pop)
# endif
#endif
#define A16(x) (((U16_S *)(x))->v)
#define A32(x) (((U32_S *)(x))->v)
#define A64(x) (((U64_S *)(x))->v)
#define AARCH(x) (((size_t_S *)(x))->v)
/**************************************
Constants
**************************************/
#define LZ4_HASHLOG (LZ4_MEMORY_USAGE-2)
#define HASHTABLESIZE (1 << LZ4_MEMORY_USAGE)
#define HASH_SIZE_U32 (1 << LZ4_HASHLOG)
#define MINMATCH 4
#define COPYLENGTH 8
#define LASTLITERALS 5
#define MFLIMIT (COPYLENGTH+MINMATCH)
static const int LZ4_minLength = (MFLIMIT+1);
#define KB *(1U<<10)
#define MB *(1U<<20)
#define GB *(1U<<30)
#define LZ4_64KLIMIT ((64 KB) + (MFLIMIT-1))
#define SKIPSTRENGTH 6 /* Increasing this value will make the compression run slower on incompressible data */
#define MAXD_LOG 16
#define MAX_DISTANCE ((1 << MAXD_LOG) - 1)
#define ML_BITS 4
#define ML_MASK ((1U<<ML_BITS)-1)
#define RUN_BITS (8-ML_BITS)
#define RUN_MASK ((1U<<RUN_BITS)-1)
/**************************************
Structures and local types
**************************************/
typedef struct {
U32 hashTable[HASH_SIZE_U32];
U32 currentOffset;
U32 initCheck;
const BYTE* dictionary;
const BYTE* bufferStart;
U32 dictSize;
} LZ4_stream_t_internal;
typedef enum { notLimited = 0, limitedOutput = 1 } limitedOutput_directive;
typedef enum { byPtr, byU32, byU16 } tableType_t;
typedef enum { noDict = 0, withPrefix64k, usingExtDict } dict_directive;
typedef enum { noDictIssue = 0, dictSmall } dictIssue_directive;
typedef enum { endOnOutputSize = 0, endOnInputSize = 1 } endCondition_directive;
typedef enum { full = 0, partial = 1 } earlyEnd_directive;
/**************************************
Architecture-specific macros
**************************************/
#define STEPSIZE sizeof(size_t)
#define LZ4_COPYSTEP(d,s) { AARCH(d) = AARCH(s); d+=STEPSIZE; s+=STEPSIZE; }
#define LZ4_COPY8(d,s) { LZ4_COPYSTEP(d,s); if (STEPSIZE<8) LZ4_COPYSTEP(d,s); }
#if (defined(LZ4_BIG_ENDIAN) && !defined(BIG_ENDIAN_NATIVE_BUT_INCOMPATIBLE))
# define LZ4_READ_LITTLEENDIAN_16(d,s,p) { U16 v = A16(p); v = lz4_bswap16(v); d = (s) - v; }
# define LZ4_WRITE_LITTLEENDIAN_16(p,i) { U16 v = (U16)(i); v = lz4_bswap16(v); A16(p) = v; p+=2; }
#else /* Little Endian */
# define LZ4_READ_LITTLEENDIAN_16(d,s,p) { d = (s) - A16(p); }
# define LZ4_WRITE_LITTLEENDIAN_16(p,v) { A16(p) = v; p+=2; }
#endif
/**************************************
Macros
**************************************/
#define LZ4_STATIC_ASSERT(c) { enum { LZ4_static_assert = 1/(!!(c)) }; } /* use only *after* variable declarations */
#if LZ4_ARCH64 || !defined(__GNUC__)
# define LZ4_WILDCOPY(d,s,e) { do { LZ4_COPY8(d,s) } while (d<e); } /* at the end, d>=e; */
#else
# define LZ4_WILDCOPY(d,s,e) { if (likely(e-d <= 8)) LZ4_COPY8(d,s) else do { LZ4_COPY8(d,s) } while (d<e); }
#endif
/****************************
Private local functions
****************************/
#if LZ4_ARCH64
static int LZ4_NbCommonBytes (register U64 val)
{
# if defined(LZ4_BIG_ENDIAN)
# if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT)
unsigned long r = 0;
_BitScanReverse64( &r, val );
return (int)(r>>3);
# elif defined(__GNUC__) && (GCC_VERSION >= 304) && !defined(LZ4_FORCE_SW_BITCOUNT)
return (__builtin_clzll(val) >> 3);
# else
int r;
if (!(val>>32)) { r=4; } else { r=0; val>>=32; }
if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; }
r += (!val);
return r;
# endif
# else
# if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT)
unsigned long r = 0;
_BitScanForward64( &r, val );
return (int)(r>>3);
# elif defined(__GNUC__) && (GCC_VERSION >= 304) && !defined(LZ4_FORCE_SW_BITCOUNT)
return (__builtin_ctzll(val) >> 3);
# else
static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, 0, 3, 1, 3, 1, 4, 2, 7, 0, 2, 3, 6, 1, 5, 3, 5, 1, 3, 4, 4, 2, 5, 6, 7, 7, 0, 1, 2, 3, 3, 4, 6, 2, 6, 5, 5, 3, 4, 5, 6, 7, 1, 2, 4, 6, 4, 4, 5, 7, 2, 6, 5, 7, 6, 7, 7 };
return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58];
# endif
# endif
}
#else
static int LZ4_NbCommonBytes (register U32 val)
{
# if defined(LZ4_BIG_ENDIAN)
# if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT)
unsigned long r = 0;
_BitScanReverse( &r, val );
return (int)(r>>3);
# elif defined(__GNUC__) && (GCC_VERSION >= 304) && !defined(LZ4_FORCE_SW_BITCOUNT)
return (__builtin_clz(val) >> 3);
# else
int r;
if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; }
r += (!val);
return r;
# endif
# else
# if defined(_MSC_VER) && !defined(LZ4_FORCE_SW_BITCOUNT)
unsigned long r;
_BitScanForward( &r, val );
return (int)(r>>3);
# elif defined(__GNUC__) && (GCC_VERSION >= 304) && !defined(LZ4_FORCE_SW_BITCOUNT)
return (__builtin_ctz(val) >> 3);
# else
static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, 3, 2, 2, 1, 3, 2, 0, 1, 3, 3, 1, 2, 2, 2, 2, 0, 3, 1, 2, 0, 1, 0, 1, 1 };
return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27];
# endif
# endif
}
#endif
/********************************
Compression functions
********************************/
int LZ4_versionNumber (void) { return LZ4_VERSION_NUMBER; }
int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); }
static int LZ4_hashSequence(U32 sequence, tableType_t tableType)
{
if (tableType == byU16)
return (((sequence) * 2654435761U) >> ((MINMATCH*8)-(LZ4_HASHLOG+1)));
else
return (((sequence) * 2654435761U) >> ((MINMATCH*8)-LZ4_HASHLOG));
}
static int LZ4_hashPosition(const BYTE* p, tableType_t tableType) { return LZ4_hashSequence(A32(p), tableType); }
static void LZ4_putPositionOnHash(const BYTE* p, U32 h, void* tableBase, tableType_t tableType, const BYTE* srcBase)
{
switch (tableType)
{
case byPtr: { const BYTE** hashTable = (const BYTE**) tableBase; hashTable[h] = p; break; }
case byU32: { U32* hashTable = (U32*) tableBase; hashTable[h] = (U32)(p-srcBase); break; }
case byU16: { U16* hashTable = (U16*) tableBase; hashTable[h] = (U16)(p-srcBase); break; }
}
}
static void LZ4_putPosition(const BYTE* p, void* tableBase, tableType_t tableType, const BYTE* srcBase)
{
U32 h = LZ4_hashPosition(p, tableType);
LZ4_putPositionOnHash(p, h, tableBase, tableType, srcBase);
}
static const BYTE* LZ4_getPositionOnHash(U32 h, void* tableBase, tableType_t tableType, const BYTE* srcBase)
{
if (tableType == byPtr) { const BYTE** hashTable = (const BYTE**) tableBase; return hashTable[h]; }
if (tableType == byU32) { U32* hashTable = (U32*) tableBase; return hashTable[h] + srcBase; }
{ U16* hashTable = (U16*) tableBase; return hashTable[h] + srcBase; } /* default, to ensure a return */
}
static const BYTE* LZ4_getPosition(const BYTE* p, void* tableBase, tableType_t tableType, const BYTE* srcBase)
{
U32 h = LZ4_hashPosition(p, tableType);
return LZ4_getPositionOnHash(h, tableBase, tableType, srcBase);
}
static unsigned LZ4_count(const BYTE* pIn, const BYTE* pRef, const BYTE* pInLimit)
{
const BYTE* const pStart = pIn;
while (likely(pIn<pInLimit-(STEPSIZE-1)))
{
size_t diff = AARCH(pRef) ^ AARCH(pIn);
if (!diff) { pIn+=STEPSIZE; pRef+=STEPSIZE; continue; }
pIn += LZ4_NbCommonBytes(diff);
return (unsigned)(pIn - pStart);
}
if (LZ4_64BITS) if ((pIn<(pInLimit-3)) && (A32(pRef) == A32(pIn))) { pIn+=4; pRef+=4; }
if ((pIn<(pInLimit-1)) && (A16(pRef) == A16(pIn))) { pIn+=2; pRef+=2; }
if ((pIn<pInLimit) && (*pRef == *pIn)) pIn++;
return (unsigned)(pIn - pStart);
}
static int LZ4_compress_generic(
void* ctx,
const char* source,
char* dest,
int inputSize,
int maxOutputSize,
limitedOutput_directive outputLimited,
tableType_t tableType,
dict_directive dict,
dictIssue_directive dictIssue)
{
LZ4_stream_t_internal* const dictPtr = (LZ4_stream_t_internal*)ctx;
const BYTE* ip = (const BYTE*) source;
const BYTE* base;
const BYTE* lowLimit;
const BYTE* const lowRefLimit = ip - dictPtr->dictSize;
const BYTE* const dictionary = dictPtr->dictionary;
const BYTE* const dictEnd = dictionary + dictPtr->dictSize;
const size_t dictDelta = dictEnd - (const BYTE*)source;
const BYTE* anchor = (const BYTE*) source;
const BYTE* const iend = ip + inputSize;
const BYTE* const mflimit = iend - MFLIMIT;
const BYTE* const matchlimit = iend - LASTLITERALS;
BYTE* op = (BYTE*) dest;
BYTE* const olimit = op + maxOutputSize;
const int skipStrength = SKIPSTRENGTH;
U32 forwardH;
size_t refDelta=0;
/* Init conditions */
if ((U32)inputSize > (U32)LZ4_MAX_INPUT_SIZE) return 0; /* Unsupported input size, too large (or negative) */
switch(dict)
{
case noDict:
default:
base = (const BYTE*)source;
lowLimit = (const BYTE*)source;
break;
case withPrefix64k:
base = (const BYTE*)source - dictPtr->currentOffset;
lowLimit = (const BYTE*)source - dictPtr->dictSize;
break;
case usingExtDict:
base = (const BYTE*)source - dictPtr->currentOffset;
lowLimit = (const BYTE*)source;
break;
}
if ((tableType == byU16) && (inputSize>=(int)LZ4_64KLIMIT)) return 0; /* Size too large (not within 64K limit) */
if (inputSize<LZ4_minLength) goto _last_literals; /* Input too small, no compression (all literals) */
/* First Byte */
LZ4_putPosition(ip, ctx, tableType, base);
ip++; forwardH = LZ4_hashPosition(ip, tableType);
/* Main Loop */
for ( ; ; )
{
const BYTE* ref;
BYTE* token;
{
const BYTE* forwardIp = ip;
unsigned step=1;
unsigned searchMatchNb = (1U << skipStrength);
/* Find a match */
do {
U32 h = forwardH;
ip = forwardIp;
forwardIp += step;
step = searchMatchNb++ >> skipStrength;
if (unlikely(forwardIp > mflimit)) goto _last_literals;
ref = LZ4_getPositionOnHash(h, ctx, tableType, base);
if (dict==usingExtDict)
{
if (ref<(const BYTE*)source)
{
refDelta = dictDelta;
lowLimit = dictionary;
}
else
{
refDelta = 0;
lowLimit = (const BYTE*)source;
}
}
forwardH = LZ4_hashPosition(forwardIp, tableType);
LZ4_putPositionOnHash(ip, h, ctx, tableType, base);
} while ( ((dictIssue==dictSmall) ? (ref < lowRefLimit) : 0)
|| ((tableType==byU16) ? 0 : (ref + MAX_DISTANCE < ip))
|| (A32(ref+refDelta) != A32(ip)) );
}
/* Catch up */
while ((ip>anchor) && (ref+refDelta > lowLimit) && (unlikely(ip[-1]==ref[refDelta-1]))) { ip--; ref--; }
{
/* Encode Literal length */
unsigned litLength = (unsigned)(ip - anchor);
token = op++;
if ((outputLimited) && (unlikely(op + litLength + (2 + 1 + LASTLITERALS) + (litLength/255) > olimit)))
return 0; /* Check output limit */
if (litLength>=RUN_MASK)
{
int len = (int)litLength-RUN_MASK;
*token=(RUN_MASK<<ML_BITS);
for(; len >= 255 ; len-=255) *op++ = 255;
*op++ = (BYTE)len;
}
else *token = (BYTE)(litLength<<ML_BITS);
/* Copy Literals */
{ BYTE* end = op+litLength; LZ4_WILDCOPY(op,anchor,end); op=end; }
}
_next_match:
/* Encode Offset */
LZ4_WRITE_LITTLEENDIAN_16(op, (U16)(ip-ref));
/* Encode MatchLength */
{
unsigned matchLength;
if ((dict==usingExtDict) && (lowLimit==dictionary))
{
const BYTE* limit;
ref += refDelta;
limit = ip + (dictEnd-ref);
if (limit > matchlimit) limit = matchlimit;
matchLength = LZ4_count(ip+MINMATCH, ref+MINMATCH, limit);
ip += MINMATCH + matchLength;
if (ip==limit)
{
unsigned more = LZ4_count(ip, (const BYTE*)source, matchlimit);
matchLength += more;
ip += more;
}
}
else
{
matchLength = LZ4_count(ip+MINMATCH, ref+MINMATCH, matchlimit);
ip += MINMATCH + matchLength;
}
if (matchLength>=ML_MASK)
{
if ((outputLimited) && (unlikely(op + (1 + LASTLITERALS) + (matchLength>>8) > olimit)))
return 0; /* Check output limit */
*token += ML_MASK;
matchLength -= ML_MASK;
for (; matchLength >= 510 ; matchLength-=510) { *op++ = 255; *op++ = 255; }
if (matchLength >= 255) { matchLength-=255; *op++ = 255; }
*op++ = (BYTE)matchLength;
}
else *token += (BYTE)(matchLength);
}
anchor = ip;
/* Test end of chunk */
if (ip > mflimit) break;
/* Fill table */
LZ4_putPosition(ip-2, ctx, tableType, base);
/* Test next position */
ref = LZ4_getPosition(ip, ctx, tableType, base);
if (dict==usingExtDict)
{
if (ref<(const BYTE*)source)
{
refDelta = dictDelta;
lowLimit = dictionary;
}
else
{
refDelta = 0;
lowLimit = (const BYTE*)source;
}
}
LZ4_putPosition(ip, ctx, tableType, base);
if ( ((dictIssue==dictSmall) ? (ref>=lowRefLimit) : 1)
&& (ref+MAX_DISTANCE>=ip)
&& (A32(ref+refDelta)==A32(ip)) )
{ token=op++; *token=0; goto _next_match; }
/* Prepare next loop */
forwardH = LZ4_hashPosition(++ip, tableType);
}
_last_literals:
/* Encode Last Literals */
{
int lastRun = (int)(iend - anchor);
if ((outputLimited) && (((char*)op - dest) + lastRun + 1 + ((lastRun+255-RUN_MASK)/255) > (U32)maxOutputSize))
return 0; /* Check output limit */
if (lastRun>=(int)RUN_MASK) { *op++=(RUN_MASK<<ML_BITS); lastRun-=RUN_MASK; for(; lastRun >= 255 ; lastRun-=255) *op++ = 255; *op++ = (BYTE) lastRun; }
else *op++ = (BYTE)(lastRun<<ML_BITS);
memcpy(op, anchor, iend - anchor);
op += iend-anchor;
}
/* End */
return (int) (((char*)op)-dest);
}
int LZ4_compress(const char* source, char* dest, int inputSize)
{
#if (HEAPMODE)
void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U32, 4); /* Aligned on 4-bytes boundaries */
#else
U32 ctx[LZ4_STREAMSIZE_U32] = {0}; /* Ensure data is aligned on 4-bytes boundaries */
#endif
int result;
if (inputSize < (int)LZ4_64KLIMIT)
result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue);
else
result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, 0, notLimited, LZ4_64BITS ? byU32 : byPtr, noDict, noDictIssue);
#if (HEAPMODE)
FREEMEM(ctx);
#endif
return result;
}
int LZ4_compress_limitedOutput(const char* source, char* dest, int inputSize, int maxOutputSize)
{
#if (HEAPMODE)
void* ctx = ALLOCATOR(LZ4_STREAMSIZE_U32, 4); /* Aligned on 4-bytes boundaries */
#else
U32 ctx[LZ4_STREAMSIZE_U32] = {0}; /* Ensure data is aligned on 4-bytes boundaries */
#endif
int result;
if (inputSize < (int)LZ4_64KLIMIT)
result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue);
else
result = LZ4_compress_generic((void*)ctx, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64BITS ? byU32 : byPtr, noDict, noDictIssue);
#if (HEAPMODE)
FREEMEM(ctx);
#endif
return result;
}
/*****************************************
Experimental : Streaming functions
*****************************************/
/*
* LZ4_initStream
* Use this function once, to init a newly allocated LZ4_stream_t structure
* Return : 1 if OK, 0 if error
*/
void LZ4_resetStream (LZ4_stream_t* LZ4_stream)
{
MEM_INIT(LZ4_stream, 0, sizeof(LZ4_stream_t));
}
LZ4_stream_t* LZ4_createStream(void)
{
LZ4_stream_t* lz4s = (LZ4_stream_t*)ALLOCATOR(4, LZ4_STREAMSIZE_U32);
LZ4_resetStream(lz4s);
return lz4s;
}
int LZ4_freeStream (LZ4_stream_t* LZ4_stream)
{
FREEMEM(LZ4_stream);
return (0);
}
int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int dictSize)
{
LZ4_stream_t_internal* dict = (LZ4_stream_t_internal*) LZ4_dict;
const BYTE* p = (const BYTE*)dictionary;
const BYTE* const dictEnd = p + dictSize;
const BYTE* base;
LZ4_STATIC_ASSERT(LZ4_STREAMSIZE >= sizeof(LZ4_stream_t_internal)); /* A compilation error here means LZ4_STREAMSIZE is not large enough */
if (dict->initCheck) LZ4_resetStream(LZ4_dict); /* Uninitialized structure detected */
if (dictSize < MINMATCH)
{
dict->dictionary = NULL;
dict->dictSize = 0;
return 1;
}
if (p <= dictEnd - 64 KB) p = dictEnd - 64 KB;
base = p - dict->currentOffset;
dict->dictionary = p;
dict->dictSize = (U32)(dictEnd - p);
dict->currentOffset += dict->dictSize;
while (p <= dictEnd-MINMATCH)
{
LZ4_putPosition(p, dict, byU32, base);
p+=3;
}
return 1;
}
static void LZ4_renormDictT(LZ4_stream_t_internal* LZ4_dict, const BYTE* src)
{
if ((LZ4_dict->currentOffset > 0x80000000) ||
((size_t)LZ4_dict->currentOffset > (size_t)src)) /* address space overflow */
{
/* rescale hash table */
U32 delta = LZ4_dict->currentOffset - 64 KB;
const BYTE* dictEnd = LZ4_dict->dictionary + LZ4_dict->dictSize;
int i;
for (i=0; i<HASH_SIZE_U32; i++)
{
if (LZ4_dict->hashTable[i] < delta) LZ4_dict->hashTable[i]=0;
else LZ4_dict->hashTable[i] -= delta;
}
LZ4_dict->currentOffset = 64 KB;
if (LZ4_dict->dictSize > 64 KB) LZ4_dict->dictSize = 64 KB;
LZ4_dict->dictionary = dictEnd - LZ4_dict->dictSize;
}
}
FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const char* source, char* dest, int inputSize,
int maxOutputSize, limitedOutput_directive limit)
{
LZ4_stream_t_internal* streamPtr = (LZ4_stream_t_internal*)LZ4_stream;
const BYTE* const dictEnd = streamPtr->dictionary + streamPtr->dictSize;
const BYTE* smallest = (const BYTE*) source;
if (streamPtr->initCheck) return 0; /* Uninitialized structure detected */
if ((streamPtr->dictSize>0) && (smallest>dictEnd)) smallest = dictEnd;
LZ4_renormDictT(streamPtr, smallest);
/* Check overlapping input/dictionary space */
{
const BYTE* sourceEnd = (const BYTE*) source + inputSize;
if ((sourceEnd > streamPtr->dictionary) && (sourceEnd < dictEnd))
{
streamPtr->dictSize = (U32)(dictEnd - sourceEnd);
if (streamPtr->dictSize > 64 KB) streamPtr->dictSize = 64 KB;
if (streamPtr->dictSize < 4) streamPtr->dictSize = 0;
streamPtr->dictionary = dictEnd - streamPtr->dictSize;
}
}
/* prefix mode : source data follows dictionary */
if (dictEnd == (const BYTE*)source)
{
int result;
if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset))
result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, dictSmall);
else
result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, withPrefix64k, noDictIssue);
streamPtr->dictSize += (U32)inputSize;
streamPtr->currentOffset += (U32)inputSize;
return result;
}
/* external dictionary mode */
{
int result;
if ((streamPtr->dictSize < 64 KB) && (streamPtr->dictSize < streamPtr->currentOffset))
result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, dictSmall);
else
result = LZ4_compress_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limit, byU32, usingExtDict, noDictIssue);
streamPtr->dictionary = (const BYTE*)source;
streamPtr->dictSize = (U32)inputSize;
streamPtr->currentOffset += (U32)inputSize;
return result;
}
}
int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize)
{
return LZ4_compress_continue_generic(LZ4_stream, source, dest, inputSize, 0, notLimited);
}
int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize)
{
return LZ4_compress_continue_generic(LZ4_stream, source, dest, inputSize, maxOutputSize, limitedOutput);
}
/* Hidden debug function, to force separate dictionary mode */
int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* source, char* dest, int inputSize)
{
LZ4_stream_t_internal* streamPtr = (LZ4_stream_t_internal*)LZ4_dict;
int result;
const BYTE* const dictEnd = streamPtr->dictionary + streamPtr->dictSize;
const BYTE* smallest = dictEnd;
if (smallest > (const BYTE*) source) smallest = (const BYTE*) source;
LZ4_renormDictT((LZ4_stream_t_internal*)LZ4_dict, smallest);
result = LZ4_compress_generic(LZ4_dict, source, dest, inputSize, 0, notLimited, byU32, usingExtDict, noDictIssue);
streamPtr->dictionary = (const BYTE*)source;
streamPtr->dictSize = (U32)inputSize;
streamPtr->currentOffset += (U32)inputSize;
return result;
}
int LZ4_saveDict (LZ4_stream_t* LZ4_dict, char* safeBuffer, int dictSize)
{
LZ4_stream_t_internal* dict = (LZ4_stream_t_internal*) LZ4_dict;
const BYTE* previousDictEnd = dict->dictionary + dict->dictSize;
if ((U32)dictSize > 64 KB) dictSize = 64 KB; /* useless to define a dictionary > 64 KB */
if ((U32)dictSize > dict->dictSize) dictSize = dict->dictSize;
memcpy(safeBuffer, previousDictEnd - dictSize, dictSize);
dict->dictionary = (const BYTE*)safeBuffer;
dict->dictSize = (U32)dictSize;
return 1;
}
/****************************
Decompression functions
****************************/
/*
* This generic decompression function cover all use cases.
* It shall be instanciated several times, using different sets of directives
* Note that it is essential this generic function is really inlined,
* in order to remove useless branches during compilation optimisation.
*/
FORCE_INLINE int LZ4_decompress_generic(
const char* source,
char* dest,
int inputSize,
int outputSize, /* If endOnInput==endOnInputSize, this value is the max size of Output Buffer. */
int endOnInput, /* endOnOutputSize, endOnInputSize */
int partialDecoding, /* full, partial */
int targetOutputSize, /* only used if partialDecoding==partial */
int dict, /* noDict, withPrefix64k, usingExtDict */
const char* dictStart, /* only if dict==usingExtDict */
int dictSize /* note : = 0 if noDict */
)
{
/* Local Variables */
const BYTE* restrict ip = (const BYTE*) source;
const BYTE* ref;
const BYTE* const iend = ip + inputSize;
BYTE* op = (BYTE*) dest;
BYTE* const oend = op + outputSize;
BYTE* cpy;
BYTE* oexit = op + targetOutputSize;
const BYTE* const lowLimit = (const BYTE*)dest - dictSize;
const BYTE* const dictEnd = (const BYTE*)dictStart + dictSize;
const size_t dec32table[] = {4-0, 4-3, 4-2, 4-3, 4-0, 4-0, 4-0, 4-0}; /* note : static reduces speed for LZ4_decompress_safe() on GCC64 */
static const size_t dec64table[] = {0, 0, 0, (size_t)-1, 0, 1, 2, 3};
const int safeDecode = (endOnInput==endOnInputSize);
const int checkOffset = ((safeDecode) && (dictSize < (int)(64 KB)));
/* Special cases */
if ((partialDecoding) && (oexit> oend-MFLIMIT)) oexit = oend-MFLIMIT; /* targetOutputSize too high => decode everything */
if ((endOnInput) && (unlikely(outputSize==0))) return ((inputSize==1) && (*ip==0)) ? 0 : -1; /* Empty output buffer */
if ((!endOnInput) && (unlikely(outputSize==0))) return (*ip==0?1:-1);
/* Main Loop */
while (1)
{
unsigned token;
size_t length;
/* get runlength */
token = *ip++;
if ((length=(token>>ML_BITS)) == RUN_MASK)
{
unsigned s;
do
{
s = *ip++;
length += s;
}
while (likely((endOnInput)?ip<iend-RUN_MASK:1) && (s==255));
if ((safeDecode) && LZ4_32BITS && unlikely((size_t)(op+length)<(size_t)(op))) goto _output_error; /* overflow detection */
if ((safeDecode) && LZ4_32BITS && unlikely((size_t)(ip+length)<(size_t)(ip))) goto _output_error; /* overflow detection */
}
/* copy literals */
cpy = op+length;
if (((endOnInput) && ((cpy>(partialDecoding?oexit:oend-MFLIMIT)) || (ip+length>iend-(2+1+LASTLITERALS))) )
|| ((!endOnInput) && (cpy>oend-COPYLENGTH)))
{
if (partialDecoding)
{
if (cpy > oend) goto _output_error; /* Error : write attempt beyond end of output buffer */
if ((endOnInput) && (ip+length > iend)) goto _output_error; /* Error : read attempt beyond end of input buffer */
}
else
{
if ((!endOnInput) && (cpy != oend)) goto _output_error; /* Error : block decoding must stop exactly there */
if ((endOnInput) && ((ip+length != iend) || (cpy > oend))) goto _output_error; /* Error : input must be consumed */
}
memcpy(op, ip, length);
ip += length;
op += length;
break; /* Necessarily EOF, due to parsing restrictions */
}
LZ4_WILDCOPY(op, ip, cpy); ip -= (op-cpy); op = cpy;
/* get offset */
LZ4_READ_LITTLEENDIAN_16(ref,cpy,ip); ip+=2;
if ((checkOffset) && (unlikely(ref < lowLimit))) goto _output_error; /* Error : offset outside destination buffer */
/* get matchlength */
if ((length=(token&ML_MASK)) == ML_MASK)
{
unsigned s;
do
{
if ((endOnInput) && (ip > iend-LASTLITERALS)) goto _output_error;
s = *ip++;
length += s;
} while (s==255);
if ((safeDecode) && LZ4_32BITS && unlikely((size_t)(op+length)<(size_t)op)) goto _output_error; /* overflow detection */
}
/* check external dictionary */
if ((dict==usingExtDict) && (ref < (BYTE* const)dest))
{
if (unlikely(op+length+MINMATCH > oend-LASTLITERALS)) goto _output_error;
if (length+MINMATCH <= (size_t)(dest-(char*)ref))
{
ref = dictEnd - (dest-(char*)ref);
memcpy(op, ref, length+MINMATCH);
op += length+MINMATCH;
}
else
{
size_t copySize = (size_t)(dest-(char*)ref);
memcpy(op, dictEnd - copySize, copySize);
op += copySize;
copySize = length+MINMATCH - copySize;
if (copySize > (size_t)((char*)op-dest)) /* overlap */
{
BYTE* const endOfMatch = op + copySize;
const BYTE* copyFrom = (BYTE*)dest;
while (op < endOfMatch) *op++ = *copyFrom++;
}
else
{
memcpy(op, dest, copySize);
op += copySize;
}
}
continue;
}
/* copy repeated sequence */
if (unlikely((op-ref)<(int)STEPSIZE))
{
const size_t dec64 = dec64table[LZ4_32BITS ? 0 : op-ref];
op[0] = ref[0];
op[1] = ref[1];
op[2] = ref[2];
op[3] = ref[3];
ref += dec32table[op-ref];
A32(op+4) = A32(ref);
op += STEPSIZE; ref -= dec64;
} else { LZ4_COPYSTEP(op,ref); }
cpy = op + length - (STEPSIZE-4);
if (unlikely(cpy>oend-COPYLENGTH-(STEPSIZE-4)))
{
if (cpy > oend-LASTLITERALS) goto _output_error; /* Error : last 5 bytes must be literals */
if (op<oend-COPYLENGTH) LZ4_WILDCOPY(op, ref, (oend-COPYLENGTH));
while(op<cpy) *op++=*ref++;
op=cpy;
continue;
}
LZ4_WILDCOPY(op, ref, cpy);
op=cpy; /* correction */
}
/* end of decoding */
if (endOnInput)
return (int) (((char*)op)-dest); /* Nb of output bytes decoded */
else
return (int) (((char*)ip)-source); /* Nb of input bytes read */
/* Overflow error detected */
_output_error:
return (int) (-(((char*)ip)-source))-1;
}
int LZ4_decompress_safe(const char* source, char* dest, int compressedSize, int maxDecompressedSize)
{
return LZ4_decompress_generic(source, dest, compressedSize, maxDecompressedSize, endOnInputSize, full, 0, noDict, NULL, 0);
}
int LZ4_decompress_safe_partial(const char* source, char* dest, int compressedSize, int targetOutputSize, int maxDecompressedSize)
{
return LZ4_decompress_generic(source, dest, compressedSize, maxDecompressedSize, endOnInputSize, partial, targetOutputSize, noDict, NULL, 0);
}
int LZ4_decompress_fast(const char* source, char* dest, int originalSize)
{
return LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize, full, 0, withPrefix64k, NULL, 64 KB);
}
/* streaming decompression functions */
typedef struct
{
const char* dictionary;
int dictSize;
} LZ4_streamDecode_t_internal;
/*
* If you prefer dynamic allocation methods,
* LZ4_createStreamDecode()
* provides a pointer (void*) towards an initialized LZ4_streamDecode_t structure.
*/
LZ4_streamDecode_t* LZ4_createStreamDecode(void)
{
LZ4_streamDecode_t* lz4s = (LZ4_streamDecode_t*) ALLOCATOR(sizeof(U32), LZ4_STREAMDECODESIZE_U32);
MEM_INIT(lz4s, 0, LZ4_STREAMDECODESIZE);
return lz4s;
}
int LZ4_freeStreamDecode (LZ4_streamDecode_t* LZ4_stream)
{
FREEMEM(LZ4_stream);
return 0;
}
/*
* LZ4_setStreamDecode
* Use this function to instruct where to find the dictionary
* This function is not necessary if previous data is still available where it was decoded.
* Loading a size of 0 is allowed (same effect as no dictionary).
* Return : 1 if OK, 0 if error
*/
int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dictionary, int dictSize)
{
LZ4_streamDecode_t_internal* lz4sd = (LZ4_streamDecode_t_internal*) LZ4_streamDecode;
lz4sd->dictionary = dictionary;
lz4sd->dictSize = dictSize;
return 1;
}
/*
*_continue() :
These decoding functions allow decompression of multiple blocks in "streaming" mode.
Previously decoded blocks must still be available at the memory position where they were decoded.
If it's not possible, save the relevant part of decoded data into a safe buffer,
and indicate where it stands using LZ4_setDictDecode()
*/
int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int compressedSize, int maxOutputSize)
{
LZ4_streamDecode_t_internal* lz4sd = (LZ4_streamDecode_t_internal*) LZ4_streamDecode;
int result;
result = LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, usingExtDict, lz4sd->dictionary, lz4sd->dictSize);
if (result <= 0) return result;
if (lz4sd->dictionary + lz4sd->dictSize == dest)
{
lz4sd->dictSize += result;
}
else
{
lz4sd->dictionary = dest;
lz4sd->dictSize = result;
}
return result;
}
int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int originalSize)
{
LZ4_streamDecode_t_internal* lz4sd = (LZ4_streamDecode_t_internal*) LZ4_streamDecode;
int result;
result = LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize, full, 0, usingExtDict, lz4sd->dictionary, lz4sd->dictSize);
if (result <= 0) return result;
if (lz4sd->dictionary + lz4sd->dictSize == dest)
{
lz4sd->dictSize += result;
}
else
{
lz4sd->dictionary = dest;
lz4sd->dictSize = result;
}
return result;
}
/*
Advanced decoding functions :
*_usingDict() :
These decoding functions work the same as "_continue" ones,
the dictionary must be explicitly provided within parameters
*/
int LZ4_decompress_safe_usingDict(const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize)
{
return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, usingExtDict, dictStart, dictSize);
}
int LZ4_decompress_fast_usingDict(const char* source, char* dest, int originalSize, const char* dictStart, int dictSize)
{
return LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize, full, 0, usingExtDict, dictStart, dictSize);
}
/***************************************************
Obsolete Functions
***************************************************/
/*
These function names are deprecated and should no longer be used.
They are only provided here for compatibility with older user programs.
- LZ4_uncompress is totally equivalent to LZ4_decompress_fast
- LZ4_uncompress_unknownOutputSize is totally equivalent to LZ4_decompress_safe
*/
int LZ4_uncompress (const char* source, char* dest, int outputSize) { return LZ4_decompress_fast(source, dest, outputSize); }
int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize) { return LZ4_decompress_safe(source, dest, isize, maxOutputSize); }
/* Obsolete Streaming functions */
int LZ4_sizeofStreamState() { return LZ4_STREAMSIZE; }
static void LZ4_init(LZ4_stream_t_internal* lz4ds, const BYTE* base)
{
MEM_INIT(lz4ds, 0, LZ4_STREAMSIZE);
lz4ds->bufferStart = base;
}
int LZ4_resetStreamState(void* state, const char* inputBuffer)
{
if ((((size_t)state) & 3) != 0) return 1; /* Error : pointer is not aligned on 4-bytes boundary */
LZ4_init((LZ4_stream_t_internal*)state, (const BYTE*)inputBuffer);
return 0;
}
void* LZ4_create (const char* inputBuffer)
{
void* lz4ds = ALLOCATOR(4, LZ4_STREAMSIZE_U32);
LZ4_init ((LZ4_stream_t_internal*)lz4ds, (const BYTE*)inputBuffer);
return lz4ds;
}
char* LZ4_slideInputBuffer (void* LZ4_Data)
{
LZ4_stream_t_internal* lz4ds = (LZ4_stream_t_internal*)LZ4_Data;
LZ4_saveDict((LZ4_stream_t*)LZ4_Data, (char*)lz4ds->bufferStart, 64 KB);
return (char*)(lz4ds->bufferStart + 64 KB);
}
/* Obsolete compresson functions using User-allocated state */
int LZ4_sizeofState() { return LZ4_STREAMSIZE; }
int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize)
{
if (((size_t)(state)&3) != 0) return 0; /* Error : state is not aligned on 4-bytes boundary */
MEM_INIT(state, 0, LZ4_STREAMSIZE);
if (inputSize < (int)LZ4_64KLIMIT)
return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, byU16, noDict, noDictIssue);
else
return LZ4_compress_generic(state, source, dest, inputSize, 0, notLimited, LZ4_64BITS ? byU32 : byPtr, noDict, noDictIssue);
}
int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize)
{
if (((size_t)(state)&3) != 0) return 0; /* Error : state is not aligned on 4-bytes boundary */
MEM_INIT(state, 0, LZ4_STREAMSIZE);
if (inputSize < (int)LZ4_64KLIMIT)
return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, byU16, noDict, noDictIssue);
else
return LZ4_compress_generic(state, source, dest, inputSize, maxOutputSize, limitedOutput, LZ4_64BITS ? byU32 : byPtr, noDict, noDictIssue);
}
/* Obsolete streaming decompression functions */
int LZ4_decompress_safe_withPrefix64k(const char* source, char* dest, int compressedSize, int maxOutputSize)
{
return LZ4_decompress_generic(source, dest, compressedSize, maxOutputSize, endOnInputSize, full, 0, withPrefix64k, NULL, 64 KB);
}
int LZ4_decompress_fast_withPrefix64k(const char* source, char* dest, int originalSize)
{
return LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize, full, 0, withPrefix64k, NULL, 64 KB);
}
================================================
FILE: algorithm/lz4.h
================================================
/*
LZ4 - Fast LZ compression algorithm
Header File
Copyright (C) 2011-2014, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- LZ4 source repository : http://code.google.com/p/lz4/
- LZ4 public forum : https://groups.google.com/forum/#!forum/lz4c
*/
#pragma once
#if defined (__cplusplus)
extern "C" {
#endif
/**************************************
Version
**************************************/
#define LZ4_VERSION_MAJOR 1 /* for major interface/format changes */
#define LZ4_VERSION_MINOR 3 /* for minor interface/format changes */
#define LZ4_VERSION_RELEASE 0 /* for tweaks, bug-fixes, or development */
#define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE)
int LZ4_versionNumber (void);
/**************************************
Tuning parameter
**************************************/
/*
* LZ4_MEMORY_USAGE :
* Memory usage formula : N->2^N Bytes (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.)
* Increasing memory usage improves compression ratio
* Reduced memory usage can improve speed, due to cache effect
* Default value is 14, for 16KB, which nicely fits into Intel x86 L1 cache
*/
#define LZ4_MEMORY_USAGE 14
/**************************************
Simple Functions
**************************************/
int LZ4_compress (const char* source, char* dest, int inputSize);
int LZ4_decompress_safe (const char* source, char* dest, int compressedSize, int maxDecompressedSize);
/*
LZ4_compress() :
Compresses 'inputSize' bytes from 'source' into 'dest'.
Destination buffer must be already allocated,
and must be sized to handle worst cases situations (input data not compressible)
Worst case size evaluation is provided by function LZ4_compressBound()
inputSize : Max supported value is LZ4_MAX_INPUT_SIZE
return : the number of bytes written in buffer dest
or 0 if the compression fails
LZ4_decompress_safe() :
compressedSize : is obviously the source size
maxDecompressedSize : is the size of the destination buffer, which must be already allocated.
return : the number of bytes decompressed into the destination buffer (necessarily <= maxDecompressedSize)
If the destination buffer is not large enough, decoding will stop and output an error code (<0).
If the source stream is detected malformed, the function will stop decoding and return a negative result.
This function is protected against buffer overflow exploits :
it never writes outside of output buffer, and never reads outside of input buffer.
Therefore, it is protected against malicious data packets.
*/
/*
Note :
Should you prefer to explicitly allocate compression-table memory using your own allocation method,
use the streaming functions provided below, simply reset the memory area between each call to LZ4_compress_continue()
*/
/**************************************
Advanced Functions
**************************************/
#define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */
#define LZ4_COMPRESSBOUND(isize) ((unsigned int)(isize) > (unsigned int)LZ4_MAX_INPUT_SIZE ? 0 : (isize) + ((isize)/255) + 16)
/*
LZ4_compressBound() :
Provides the maximum size that LZ4 may output in a "worst case" scenario (input data not compressible)
primarily useful for memory allocation of output buffer.
macro is also provided when result needs to be evaluated at compilation (such as stack memory allocation).
isize : is the input size. Max supported value is LZ4_MAX_INPUT_SIZE
return : maximum output size in a "worst case" scenario
or 0, if input size is too large ( > LZ4_MAX_INPUT_SIZE)
*/
int LZ4_compressBound(int isize);
/*
LZ4_compress_limitedOutput() :
Compress 'inputSize' bytes from 'source' into an output buffer 'dest' of maximum size 'maxOutputSize'.
If it cannot achieve it, compression will stop, and result of the function will be zero.
This function never writes outside of provided output buffer.
inputSize : Max supported value is LZ4_MAX_INPUT_VALUE
maxOutputSize : is the size of the destination buffer (which must be already allocated)
return : the number of bytes written in buffer 'dest'
or 0 if the compression fails
*/
int LZ4_compress_limitedOutput (const char* source, char* dest, int inputSize, int maxOutputSize);
/*
LZ4_compress_withState() :
Same compression functions, but using an externally allocated memory space to store compression state.
Use LZ4_sizeofState() to know how much memory must be allocated,
and then, provide it as 'void* state' to compression functions.
*/
int LZ4_sizeofState(void);
int LZ4_compress_withState (void* state, const char* source, char* dest, int inputSize);
int LZ4_compress_limitedOutput_withState (void* state, const char* source, char* dest, int inputSize, int maxOutputSize);
/*
LZ4_decompress_fast() :
originalSize : is the original and therefore uncompressed size
return : the number of bytes read from the source buffer (in other words, the compressed size)
If the source stream is detected malformed, the function will stop decoding and return a negative result.
Destination buffer must be already allocated. Its size must be a minimum of 'originalSize' bytes.
note : This function fully respect memory boundaries for properly formed compressed data.
It is a bit faster than LZ4_decompress_safe().
However, it does not provide any protection against intentionnally modified data stream (malicious input).
Use this function in trusted environment only (data to decode comes from a trusted source).
*/
int LZ4_decompress_fast (const char* source, char* dest, int originalSize);
/*
LZ4_decompress_safe_partial() :
This function decompress a compressed block of size 'compressedSize' at position 'source'
into destination buffer 'dest' of size 'maxDecompressedSize'.
The function tries to stop decompressing operation as soon as 'targetOutputSize' has been reached,
reducing decompression time.
return : the number of bytes decoded in the destination buffer (necessarily <= maxDecompressedSize)
Note : this number can be < 'targetOutputSize' should the compressed block to decode be smaller.
Always control how many bytes were decoded.
If the source stream is detected malformed, the function will stop decoding and return a negative result.
This function never writes outside of output buffer, and never reads outside of input buffer. It is therefore protected against malicious data packets
*/
int LZ4_decompress_safe_partial (const char* source, char* dest, int compressedSize, int targetOutputSize, int maxDecompressedSize);
/***********************************************
Experimental Streaming Compression Functions
***********************************************/
#define LZ4_STREAMSIZE_U32 ((1 << (LZ4_MEMORY_USAGE-2)) + 8)
#define LZ4_STREAMSIZE (LZ4_STREAMSIZE_U32 * sizeof(unsigned int))
/*
* LZ4_stream_t
* information structure to track an LZ4 stream.
* important : init this structure content before first use !
*/
typedef struct { unsigned int table[LZ4_STREAMSIZE_U32]; } LZ4_stream_t;
/*
* LZ4_resetStream
* Use this function to init a newly allocated LZ4_stream_t structure
* You can also reset an existing LZ4_stream_t structure
*/
void LZ4_resetStream (LZ4_stream_t* LZ4_stream);
/*
* If you prefer dynamic allocation methods,
* LZ4_createStream will allocate and initialize an LZ4_stream_t structure
* LZ4_freeStream releases its memory.
*/
LZ4_stream_t* LZ4_createStream(void);
int LZ4_freeStream (LZ4_stream_t* LZ4_stream);
/*
* LZ4_loadDict
* Use this function to load a static dictionary into LZ4_stream.
* Any previous data will be forgotten, only 'dictionary' will remain in memory.
* Loading a size of 0 is allowed.
* Return : 1 if OK, 0 if error
*/
int LZ4_loadDict (LZ4_stream_t* LZ4_stream, const char* dictionary, int dictSize);
/*
* LZ4_compress_continue
* Compress data block 'source', using blocks compressed before as dictionary to improve compression ratio
* Previous data blocks are assumed to still be present at their previous location.
*/
int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize);
/*
* LZ4_compress_limitedOutput_continue
* Same as before, but also specify a maximum target compressed size (maxOutputSize)
* If objective cannot be met, compression exits, and returns a zero.
*/
int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const char* source, char* dest, int inputSize, int maxOutputSize);
/*
* LZ4_saveDict
* If previously compressed data block is not guaranteed to remain available at its memory location
* save it into a safe place (char* safeBuffer)
* Note : you don't need to call LZ4_loadDict() afterwards,
* dictionary is immediately usable, you can therefore call again LZ4_compress_continue()
* Return : 1 if OK, 0 if error
* Note : any dictSize > 64 KB will be interpreted as 64KB.
*/
int LZ4_saveDict (LZ4_stream_t* LZ4_stream, char* safeBuffer, int dictSize);
/************************************************
Experimental Streaming Decompression Functions
************************************************/
#define LZ4_STREAMDECODESIZE_U32 4
#define LZ4_STREAMDECODESIZE (LZ4_STREAMDECODESIZE_U32 * sizeof(unsigned int))
/*
* LZ4_streamDecode_t
* information structure to track an LZ4 stream.
* important : init this structure content using LZ4_setStreamDecode or memset() before first use !
*/
typedef struct { unsigned int table[LZ4_STREAMDECODESIZE_U32]; } LZ4_streamDecode_t;
/*
* LZ4_setStreamDecode
* Use this function to instruct where to find the dictionary.
* This function can be used to specify a static dictionary,
* or to instruct where to find some previously decoded data saved into a different memory space.
* Setting a size of 0 is allowed (same effect as no dictionary).
* Return : 1 if OK, 0 if error
*/
int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const char* dictionary, int dictSize);
/*
* If you prefer dynamic allocation methods,
* LZ4_createStreamDecode will allocate and initialize an LZ4_streamDecode_t structure
* LZ4_freeStreamDecode releases its memory.
*/
LZ4_streamDecode_t* LZ4_createStreamDecode(void);
int LZ4_freeStreamDecode (LZ4_streamDecode_t* LZ4_stream);
/*
*_continue() :
These decoding functions allow decompression of multiple blocks in "streaming" mode.
Previously decoded blocks must still be available at the memory position where they were decoded.
If it's not possible, save the relevant part of decoded data into a safe buffer,
and indicate where its new address using LZ4_setDictDecode()
*/
int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int compressedSize, int maxOutputSize);
int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, const char* source, char* dest, int originalSize);
/*
Advanced decoding functions :
*_usingDict() :
These decoding functions work the same as
a combination of LZ4_setDictDecode() followed by LZ4_decompress_x_continue()
all together into a single function call.
It doesn't use nor update an LZ4_streamDecode_t structure.
*/
int LZ4_decompress_safe_usingDict (const char* source, char* dest, int compressedSize, int maxOutputSize, const char* dictStart, int dictSize);
int LZ4_decompress_fast_usingDict (const char* source, char* dest, int originalSize, const char* dictStart, int dictSize);
/**************************************
Obsolete Functions
**************************************/
/*
Obsolete decompression functions
These function names are deprecated and should no longer be used.
They are only provided here for compatibility with older user programs.
- LZ4_uncompress is the same as LZ4_decompress_fast
- LZ4_uncompress_unknownOutputSize is the same as LZ4_decompress_safe
These function prototypes are now disabled; uncomment them if you really need them.
It is highly recommended to stop using these functions and migrated to newer ones */
/* int LZ4_uncompress (const char* source, char* dest, int outputSize); */
/* int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, int isize, int maxOutputSize); */
/*
* If you prefer dynamic allocation methods,
* LZ4_createStreamDecode()
* provides a pointer (void*) towards an initialized LZ4_streamDecode_t structure.
* LZ4_free just frees it.
*/
/* void* LZ4_createStreamDecode(void); */
/*int LZ4_free (void* LZ4_stream); yes, it's the same one as for compression */
/* Obsolete streaming functions; use new streaming interface whenever possible */
void* LZ4_create (const char* inputBuffer);
int LZ4_sizeofStreamState(void);
int LZ4_resetStreamState(void* state, const char* inputBuffer);
char* LZ4_slideInputBuffer (void* state);
/* Obsolete streaming decoding functions */
int LZ4_decompress_safe_withPrefix64k (const char* source, char* dest, int compressedSize, int maxOutputSize);
int LZ4_decompress_fast_withPrefix64k (const char* source, char* dest, int originalSize);
#if defined (__cplusplus)
}
#endif
================================================
FILE: algorithm/murmurhash3.cc
================================================
//-----------------------------------------------------------------------------
// MurmurHash3 was written by Austin Appleby, and is placed in the public
// domain. The author hereby disclaims copyright to this source code.
// Note - The x86 and x64 versions do _not_ produce the same results, as the
// algorithms are optimized for their respective platforms. You can still
// compile and run any of them on any platform, but your performance with the
// non-native version will be less than optimal.
#include "algorithm/murmurhash3.h"
//-----------------------------------------------------------------------------
// Platform-specific functions and macros
// Microsoft Visual Studio
#if defined(_MSC_VER)
#define FORCE_INLINE __forceinline
#include <stdlib.h>
#define ROTL32(x,y) _rotl(x,y)
#define ROTL64(x,y) _rotl64(x,y)
#define BIG_CONSTANT(x) (x)
// Other compilers
#else // defined(_MSC_VER)
#define FORCE_INLINE inline __attribute__((always_inline))
inline uint32_t rotl32 ( uint32_t x, int8_t r )
{
return (x << r) | (x >> (32 - r));
}
inline uint64_t rotl64 ( uint64_t x, int8_t r )
{
return (x << r) | (x >> (64 - r));
}
#define ROTL32(x,y) rotl32(x,y)
#define ROTL64(x,y) rotl64(x,y)
#define BIG_CONSTANT(x) (x##LLU)
#endif // !defined(_MSC_VER)
//-----------------------------------------------------------------------------
// Block read - if your platform needs to do endian-swapping or can only
// handle aligned reads, do the conversion here
FORCE_INLINE uint32_t getblock32 ( const uint32_t * p, int i )
{
return p[i];
}
FORCE_INLINE uint64_t getblock64 ( const uint64_t * p, int i )
{
return p[i];
}
//-----------------------------------------------------------------------------
// Finalization mix - force all bits of a hash block to avalanche
FORCE_INLINE uint32_t fmix32 ( uint32_t h )
{
h ^= h >> 16;
h *= 0x85ebca6b;
h ^= h >> 13;
h *= 0xc2b2ae35;
h ^= h >> 16;
return h;
}
//----------
FORCE_INLINE uint64_t fmix64 ( uint64_t k )
{
k ^= k >> 33;
k *= BIG_CONSTANT(0xff51afd7ed558ccd);
k ^= k >> 33;
k *= BIG_CONSTANT(0xc4ceb9fe1a85ec53);
k ^= k >> 33;
return k;
}
//-----------------------------------------------------------------------------
void MurmurHash3_x86_32 ( const void * key, int len,
uint32_t seed, void * out )
{
const uint8_t * data = (const uint8_t*)key;
const int nblocks = len / 4;
uint32_t h1 = seed;
const uint32_t c1 = 0xcc9e2d51;
const uint32_t c2 = 0x1b873593;
//----------
// body
const uint32_t * blocks = (const uint32_t *)(data + nblocks*4);
for(int i = -nblocks; i; i++)
{
uint32_t k1 = getblock32(blocks,i);
k1 *= c1;
k1 = ROTL32(k1,15);
k1 *= c2;
h1 ^= k1;
h1 = ROTL32(h1,13);
h1 = h1*5+0xe6546b64;
}
//----------
// tail
const uint8_t * tail = (const uint8_t*)(data + nblocks*4);
uint32_t k1 = 0;
switch(len & 3)
{
case 3: k1 ^= tail[2] << 16;
case 2: k1 ^= tail[1] << 8;
case 1: k1 ^= tail[0];
k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1;
};
//----------
// finalization
h1 ^= len;
h1 = fmix32(h1);
*(uint32_t*)out = h1;
}
//-----------------------------------------------------------------------------
void MurmurHash3_x86_128 ( const void * key, const int len,
uint32_t seed, void * out )
{
const uint8_t * data = (const uint8_t*)key;
const int nblocks = len / 16;
uint32_t h1 = seed;
uint32_t h2 = seed;
uint32_t h3 = seed;
uint32_t h4 = seed;
const uint32_t c1 = 0x239b961b;
const uint32_t c2 = 0xab0e9789;
const uint32_t c3 = 0x38b34ae5;
const uint32_t c4 = 0xa1e38b93;
//----------
// body
const uint32_t * blocks = (const uint32_t *)(data + nblocks*16);
for(int i = -nblocks; i; i++)
{
uint32_t k1 = getblock32(blocks,i*4+0);
uint32_t k2 = getblock32(blocks,i*4+1);
uint32_t k3 = getblock32(blocks,i*4+2);
uint32_t k4 = getblock32(blocks,i*4+3);
k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1;
h1 = ROTL32(h1,19); h1 += h2; h1 = h1*5+0x561ccd1b;
k2 *= c2; k2 = ROTL32(k2,16); k2 *= c3; h2 ^= k2;
h2 = ROTL32(h2,17); h2 += h3; h2 = h2*5+0x0bcaa747;
k3 *= c3; k3 = ROTL32(k3,17); k3 *= c4; h3 ^= k3;
h3 = ROTL32(h3,15); h3 += h4; h3 = h3*5+0x96cd1c35;
k4 *= c4; k4 = ROTL32(k4,18); k4 *= c1; h4 ^= k4;
h4 = ROTL32(h4,13); h4 += h1; h4 = h4*5+0x32ac3b17;
}
//----------
// tail
const uint8_t * tail = (const uint8_t*)(data + nblocks*16);
uint32_t k1 = 0;
uint32_t k2 = 0;
uint32_t k3 = 0;
uint32_t k4 = 0;
switch(len & 15)
{
case 15: k4 ^= tail[14] << 16;
case 14: k4 ^= tail[13] << 8;
case 13: k4 ^= tail[12] << 0;
k4 *= c4; k4 = ROTL32(k4,18); k4 *= c1; h4 ^= k4;
case 12: k3 ^= tail[11] << 24;
case 11: k3 ^= tail[10] << 16;
case 10: k3 ^= tail[ 9] << 8;
case 9: k3 ^= tail[ 8] << 0;
k3 *= c3; k3 = ROTL32(k3,17); k3 *= c4; h3 ^= k3;
case 8: k2 ^= tail[ 7] << 24;
case 7: k2 ^= tail[ 6] << 16;
case 6: k2 ^= tail[ 5] << 8;
case 5: k2 ^= tail[ 4] << 0;
k2 *= c2; k2 = ROTL32(k2,16); k2 *= c3; h2 ^= k2;
case 4: k1 ^= tail[ 3] << 24;
case 3: k1 ^= tail[ 2] << 16;
case 2: k1 ^= tail[ 1] << 8;
case 1: k1 ^= tail[ 0] << 0;
k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1;
};
//----------
// finalization
h1 ^= len; h2 ^= len; h3 ^= len; h4 ^= len;
h1 += h2; h1 += h3; h1 += h4;
h2 += h1; h3 += h1; h4 += h1;
h1 = fmix32(h1);
h2 = fmix32(h2);
h3 = fmix32(h3);
h4 = fmix32(h4);
h1 += h2; h1 += h3; h1 += h4;
h2 += h1; h3 += h1; h4 += h1;
((uint32_t*)out)[0] = h1;
((uint32_t*)out)[1] = h2;
((uint32_t*)out)[2] = h3;
((uint32_t*)out)[3] = h4;
}
//-----------------------------------------------------------------------------
void MurmurHash3_x64_128 ( const void * key, const int len,
const uint32_t seed, void * out )
{
const uint8_t * data = (const uint8_t*)key;
const int nblocks = len / 16;
uint64_t h1 = seed;
uint64_t h2 = seed;
const uint64_t c1 = BIG_CONSTANT(0x87c37b91114253d5);
const uint64_t c2 = BIG_CONSTANT(0x4cf5ad432745937f);
//----------
// body
const uint64_t * blocks = (const uint64_t *)(data);
for(int i = 0; i < nblocks; i++)
{
uint64_t k1 = getblock64(blocks,i*2+0);
uint64_t k2 = getblock64(blocks,i*2+1);
k1 *= c1; k1 = ROTL64(k1,31); k1 *= c2; h1 ^= k1;
h1 = ROTL64(h1,27); h1 += h2; h1 = h1*5+0x52dce729;
k2 *= c2; k2 = ROTL64(k2,33); k2 *= c1; h2 ^= k2;
h2 = ROTL64(h2,31); h2 += h1; h2 = h2*5+0x38495ab5;
}
//----------
// tail
const uint8_t * tail = (const uint8_t*)(data + nblocks*16);
uint64_t k1 = 0;
uint64_t k2 = 0;
switch(len & 15)
{
case 15: k2 ^= ((uint64_t)tail[14]) << 48;
case 14: k2 ^= ((uint64_t)tail[13]) << 40;
case 13: k2 ^= ((uint64_t)tail[12]) << 32;
case 12: k2 ^= ((uint64_t)tail[11]) << 24;
case 11: k2 ^= ((uint64_t)tail[10]) << 16;
case 10: k2 ^= ((uint64_t)tail[ 9]) << 8;
case 9: k2 ^= ((uint64_t)tail[ 8]) << 0;
k2 *= c2; k2 = ROTL64(k2,33); k2 *= c1; h2 ^= k2;
case 8: k1 ^= ((uint64_t)tail[ 7]) << 56;
case 7: k1 ^= ((uint64_t)tail[ 6]) << 48;
case 6: k1 ^= ((uint64_t)tail[ 5]) << 40;
case 5: k1 ^= ((uint64_t)tail[ 4]) << 32;
case 4: k1 ^= ((uint64_t)tail[ 3]) << 24;
case 3: k1 ^= ((uint64_t)tail[ 2]) << 16;
case 2: k1 ^= ((uint64_t)tail[ 1]) << 8;
case 1: k1 ^= ((uint64_t)tail[ 0]) << 0;
k1 *= c1; k1 = ROTL64(k1,31); k1 *= c2; h1 ^= k1;
};
//----------
// finalization
h1 ^= len; h2 ^= len;
h1 += h2;
h2 += h1;
h1 = fmix64(h1);
h2 = fmix64(h2);
h1 += h2;
h2 += h1;
((uint64_t*)out)[0] = h1;
((uint64_t*)out)[1] = h2;
}
//-----------------------------------------------------------------------------
================================================
FILE: algorithm/murmurhash3.h
================================================
//-----------------------------------------------------------------------------
// MurmurHash3 was written by Austin Appleby, and is placed in the public
// domain. The author hereby disclaims copyright to this source code.
#ifndef _MURMURHASH3_H_
#define _MURMURHASH3_H_
//-----------------------------------------------------------------------------
// Platform-specific functions and macros
// Microsoft Visual Studio
#if defined(_MSC_VER)
typedef unsigned char uint8_t;
typedef unsigned long uint32_t;
typedef unsigned __int64 uint64_t;
// Other compilers
#else // defined(_MSC_VER)
#include <stdint.h>
#endif // !defined(_MSC_VER)
//-----------------------------------------------------------------------------
void MurmurHash3_x86_32 ( const void * key, int len, uint32_t seed, void * out );
void MurmurHash3_x86_128 ( const void * key, int len, uint32_t seed, void * out );
void MurmurHash3_x64_128 ( const void * key, int len, uint32_t seed, void * out );
//-----------------------------------------------------------------------------
#endif // _MURMURHASH3_H_
================================================
FILE: algorithm/xxhash.cc
================================================
/*
xxHash - Fast Hash algorithm
Copyright (C) 2012-2014, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- xxHash source repository : http://code.google.com/p/xxhash/
*/
//**************************************
// Tuning parameters
//**************************************
// Unaligned memory access is automatically enabled for "common" CPU, such as x86.
// For others CPU, the compiler will be more cautious, and insert extra code to ensure aligned access is respected.
// If you know your target CPU supports unaligned memory access, you want to force this option manually to improve performance.
// You can also enable this parameter if you know your input data will always be aligned (boundaries of 4, for U32).
#if defined(__ARM_FEATURE_UNALIGNED) || defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64)
# define XXH_USE_UNALIGNED_ACCESS 1
#endif
// XXH_ACCEPT_NULL_INPUT_POINTER :
// If the input pointer is a null pointer, xxHash default behavior is to trigger a memory access error, since it is a bad pointer.
// When this option is enabled, xxHash output for null input pointers will be the same as a null-length input.
// This option has a very small performance cost (only measurable on small inputs).
// By default, this option is disabled. To enable it, uncomment below define :
// #define XXH_ACCEPT_NULL_INPUT_POINTER 1
// XXH_FORCE_NATIVE_FORMAT :
// By default, xxHash library provides endian-independant Hash values, based on little-endian convention.
// Results are therefore identical for little-endian and big-endian CPU.
// This comes at a performance cost for big-endian CPU, since some swapping is required to emulate little-endian format.
// Should endian-independance be of no importance for your application, you may set the #define below to 1.
// It will improve speed for Big-endian CPU.
// This option has no impact on Little_Endian CPU.
#define XXH_FORCE_NATIVE_FORMAT 0
//**************************************
// Compiler Specific Options
//**************************************
// Disable some Visual warning messages
#ifdef _MSC_VER // Visual Studio
# pragma warning(disable : 4127) // disable: C4127: conditional expression is constant
#endif
#ifdef _MSC_VER // Visual Studio
# define FORCE_INLINE static __forceinline
#else
# ifdef __GNUC__
# define FORCE_INLINE static inline __attribute__((always_inline))
# else
# define FORCE_INLINE static inline
# endif
#endif
//**************************************
// Includes & Memory related functions
//**************************************
#include "algorithm/xxhash.h"
// Modify the local functions below should you wish to use some other memory related routines
// for malloc(), free()
#include <stdlib.h>
FORCE_INLINE void* XXH_malloc(size_t s) { return malloc(s); }
FORCE_INLINE void XXH_free (void* p) { free(p); }
// for memcpy()
#include <string.h>
FORCE_INLINE void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); }
//**************************************
// Basic Types
//**************************************
#if defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L // C99
# include <stdint.h>
typedef uint8_t BYTE;
typedef uint16_t U16;
typedef uint32_t U32;
typedef int32_t S32;
typedef uint64_t U64;
#else
typedef unsigned char BYTE;
typedef unsigned short U16;
typedef unsigned int U32;
typedef signed int S32;
typedef unsigned long long U64;
#endif
#if defined(__GNUC__) && !defined(XXH_USE_UNALIGNED_ACCESS)
# define _PACKED __attribute__ ((packed))
#else
# define _PACKED
#endif
#if !defined(XXH_USE_UNALIGNED_ACCESS) && !defined(__GNUC__)
# ifdef __IBMC__
# pragma pack(1)
# else
# pragma pack(push, 1)
# endif
#endif
typedef struct _U32_S { U32 v; } _PACKED U32_S;
typedef struct _U64_S { U64 v; } _PACKED U64_S;
#if !defined(XXH_USE_UNALIGNED_ACCESS) && !defined(__GNUC__)
# pragma pack(pop)
#endif
#define A32(x) (((U32_S *)(x))->v)
#define A64(x) (((U64_S *)(x))->v)
//***************************************
// Compiler-specific Functions and Macros
//***************************************
#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
// Note : although _rotl exists for minGW (GCC under windows), performance seems poor
#if defined(_MSC_VER)
# define XXH_rotl32(x,r) _rotl(x,r)
# define XXH_rotl64(x,r) _rotl64(x,r)
#else
# define XXH_rotl32(x,r) ((x << r) | (x >> (32 - r)))
# define XXH_rotl64(x,r) ((x << r) | (x >> (64 - r)))
#endif
#if defined(_MSC_VER) // Visual Studio
# define XXH_swap32 _byteswap_ulong
# define XXH_swap64 _byteswap_uint64
#elif GCC_VERSION >= 403
# define XXH_swap32 __builtin_bswap32
# define XXH_swap64 __builtin_bswap64
#else
static inline U32 XXH_swap32 (U32 x) {
return ((x << 24) & 0xff000000 ) |
((x << 8) & 0x00ff0000 ) |
((x >> 8) & 0x0000ff00 ) |
((x >> 24) & 0x000000ff );}
static inline U64 XXH_swap64 (U64 x) {
return ((x << 56) & 0xff00000000000000ULL) |
((x << 40) & 0x00ff000000000000ULL) |
((x << 24) & 0x0000ff0000000000ULL) |
((x << 8) & 0x000000ff00000000ULL) |
((x >> 8) & 0x00000000ff000000ULL) |
((x >> 24) & 0x0000000000ff0000ULL) |
((x >> 40) & 0x000000000000ff00ULL) |
((x >> 56) & 0x00000000000000ffULL);}
#endif
//**************************************
// Constants
//**************************************
#define PRIME32_1 2654435761U
#define PRIME32_2 2246822519U
#define PRIME32_3 3266489917U
#define PRIME32_4 668265263U
#define PRIME32_5 374761393U
#define PRIME64_1 11400714785074694791ULL
#define PRIME64_2 14029467366897019727ULL
#define PRIME64_3 1609587929392839161ULL
#define PRIME64_4 9650029242287828579ULL
#define PRIME64_5 2870177450012600261ULL
//**************************************
// Architecture Macros
//**************************************
typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;
#ifndef XXH_CPU_LITTLE_ENDIAN // It is possible to define XXH_CPU_LITTLE_ENDIAN externally, for example using a compiler switch
static const int one = 1;
# define XXH_CPU_LITTLE_ENDIAN (*(char*)(&one))
#endif
//**************************************
// Macros
//**************************************
#define XXH_STATIC_ASSERT(c) { enum { XXH_static_assert = 1/(!!(c)) }; } // use only *after* variable declarations
//****************************
// Memory reads
//****************************
typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;
FORCE_INLINE U32 XXH_readLE32_align(const U32* ptr, XXH_endianess endian, XXH_alignment align)
{
if (align==XXH_unaligned)
return endian==XXH_littleEndian ? A32(ptr) : XXH_swap32(A32(ptr));
else
return endian==XXH_littleEndian ? *ptr : XXH_swap32(*ptr);
}
FORCE_INLINE U32 XXH_readLE32(const U32* ptr, XXH_endianess endian) { return XXH_readLE32_align(ptr, endian, XXH_unaligned); }
FORCE_INLINE U64 XXH_readLE64_align(const U64* ptr, XXH_endianess endian, XXH_alignment align)
{
if (align==XXH_unaligned)
return endian==XXH_littleEndian ? A64(ptr) : XXH_swap64(A64(ptr));
else
return endian==XXH_littleEndian ? *ptr : XXH_swap64(*ptr);
}
FORCE_INLINE U64 XXH_readLE64(const U64* ptr, XXH_endianess endian) { return XXH_readLE64_align(ptr, endian, XXH_unaligned); }
//****************************
// Simple Hash Functions
//****************************
FORCE_INLINE U32 XXH32_endian_align(const void* input, unsigned int len, U32 seed, XXH_endianess endian, XXH_alignment align)
{
const BYTE* p = (const BYTE*)input;
const BYTE* bEnd = p + len;
U32 h32;
#define XXH_get32bits(p) XXH_readLE32_align((const U32*)p, endian, align)
#ifdef XXH_ACCEPT_NULL_INPUT_POINTER
if (p==NULL) { len=0; bEnd=p=(const BYTE*)(size_t)16; }
#endif
if (len>=16)
{
const BYTE* const limit = bEnd - 16;
U32 v1 = seed + PRIME32_1 + PRIME32_2;
U32 v2 = seed + PRIME32_2;
U32 v3 = seed + 0;
U32 v4 = seed - PRIME32_1;
do
{
v1 += XXH_get32bits(p) * PRIME32_2; v1 = XXH_rotl32(v1, 13); v1 *= PRIME32_1; p+=4;
v2 += XXH_get32bits(p) * PRIME32_2; v2 = XXH_rotl32(v2, 13); v2 *= PRIME32_1; p+=4;
v3 += XXH_get32bits(p) * PRIME32_2; v3 = XXH_rotl32(v3, 13); v3 *= PRIME32_1; p+=4;
v4 += XXH_get32bits(p) * PRIME32_2; v4 = XXH_rotl32(v4, 13); v4 *= PRIME32_1; p+=4;
} while (p<=limit);
h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7) + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18);
}
else
{
h32 = seed + PRIME32_5;
}
h32 += (U32) len;
while (p<=bEnd-4)
{
h32 += XXH_get32bits(p) * PRIME32_3;
h32 = XXH_rotl32(h32, 17) * PRIME32_4 ;
p+=4;
}
while (p<bEnd)
{
h32 += (*p) * PRIME32_5;
h32 = XXH_rotl32(h32, 11) * PRIME32_1 ;
p++;
}
h32 ^= h32 >> 15;
h32 *= PRIME32_2;
h32 ^= h32 >> 13;
h32 *= PRIME32_3;
h32 ^= h32 >> 16;
return h32;
}
U32 XXH32(const void* input, unsigned int len, U32 seed)
{
#if 0
// Simple version, good for code maintenance, but unfortunately slow for small inputs
void* state = XXH32_init(seed);
XXH32_update(state, input, len);
return XXH32_digest(state);
#else
XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
# if !defined(XXH_USE_UNALIGNED_ACCESS)
if ((((size_t)input) & 3) == 0) // Input is aligned, let's leverage the speed advantage
{
if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned);
else
return XXH32_endian_align(input, len, seed, XXH_bigEndian, XXH_aligned);
}
# endif
if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
return XXH32_endian_align(input, len, seed, XXH_littleEndian, XXH_unaligned);
else
return XXH32_endian_align(input, len, seed, XXH_bigEndian, XXH_unaligned);
#endif
}
FORCE_INLINE U64 XXH64_endian_align(const void* input, unsigned int len, U64 seed, XXH_endianess endian, XXH_alignment align)
{
const BYTE* p = (const BYTE*)input;
const BYTE* bEnd = p + len;
U64 h64;
#define XXH_get64bits(p) XXH_readLE64_align((const U64*)p, endian, align)
#ifdef XXH_ACCEPT_NULL_INPUT_POINTER
if (p==NULL) { len=0; bEnd=p=(const BYTE*)(size_t)32; }
#endif
if (len>=32)
{
const BYTE* const limit = bEnd - 32;
U64 v1 = seed + PRIME64_1 + PRIME64_2;
U64 v2 = seed + PRIME64_2;
U64 v3 = seed + 0;
U64 v4 = seed - PRIME64_1;
do
{
v1 += XXH_get64bits(p) * PRIME64_2; p+=8; v1 = XXH_rotl64(v1, 31); v1 *= PRIME64_1;
v2 += XXH_get64bits(p) * PRIME64_2; p+=8; v2 = XXH_rotl64(v2, 31); v2 *= PRIME64_1;
v3 += XXH_get64bits(p) * PRIME64_2; p+=8; v3 = XXH_rotl64(v3, 31); v3 *= PRIME64_1;
v4 += XXH_get64bits(p) * PRIME64_2; p+=8; v4 = XXH_rotl64(v4, 31); v4 *= PRIME64_1;
} while (p<=limit);
h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
v1 *= PRIME64_2; v1 = XXH_rotl64(v1, 31); v1 *= PRIME64_1; h64 ^= v1;
h64 = h64 * PRIME64_1 + PRIME64_4;
v2 *= PRIME64_2; v2 = XXH_rotl64(v2, 31); v2 *= PRIME64_1; h64 ^= v2;
h64 = h64 * PRIME64_1 + PRIME64_4;
v3 *= PRIME64_2; v3 = XXH_rotl64(v3, 31); v3 *= PRIME64_1; h64 ^= v3;
h64 = h64 * PRIME64_1 + PRIME64_4;
v4 *= PRIME64_2; v4 = XXH_rotl64(v4, 31); v4 *= PRIME64_1; h64 ^= v4;
h64 = h64 * PRIME64_1 + PRIME64_4;
}
else
{
h64 = seed + PRIME64_5;
}
h64 += (U64) len;
while (p<=bEnd-8)
{
U64 k1 = XXH_get64bits(p);
k1 *= PRIME64_2; k1 = XXH_rotl64(k1,31); k1 *= PRIME64_1; h64 ^= k1;
h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4;
p+=8;
}
#if 1
if (p<=bEnd-4)
{
h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1;
h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
p+=4;
}
while (p<bEnd)
{
h64 ^= (*p) * PRIME64_5;
h64 = XXH_rotl64(h64, 11) * PRIME64_1;
p++;
}
#else
if( p<bEnd )
{
// Copy the remaining bytes
U64 k1 = PRIME64_5;
BYTE* _k1 = (BYTE*)&k1;
switch( (size_t)(bEnd-p)) // Yes, I tried multiple ways of doing this memcopy
{
case 7: *_k1++ = *p++;
case 6: *_k1++ = *p++;
case 5: *_k1++ = *p++;
case 4: *_k1++ = *p++;
case 3: *_k1++ = *p++;
case 2: *_k1++ = *p++;
case 1: *_k1++ = *p++;
}
k1 *= PRIME64_5; k1 = XXH_rotl64(k1,31); k1 *= PRIME64_3; h64 ^= k1;
h64 = XXH_rotl64(h64, 11) * PRIME64_1 + PRIME64_4;
}
#endif
h64 ^= h64 >> 33;
h64 *= PRIME64_2;
h64 ^= h64 >> 29;
h64 *= PRIME64_3;
h64 ^= h64 >> 32;
return h64;
}
unsigned long long XXH64(const void* input, unsigned int len, unsigned long long seed)
{
XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
# if !defined(XXH_USE_UNALIGNED_ACCESS)
if ((((size_t)input) & 7)==0) // Input is aligned, let's leverage the speed advantage
{
if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
return XXH64_endian_align(input, len, seed, XXH_littleEndian, XXH_aligned);
else
return XXH64_endian_align(input, len, seed, XXH_bigEndian, XXH_aligned);
}
# endif
if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
return XXH64_endian_align(input, len, seed, XXH_littleEndian, XXH_unaligned);
else
return XXH64_endian_align(input, len, seed, XXH_bigEndian, XXH_unaligned);
}
//****************************
// Advanced Hash Functions
//****************************
struct XXH_state32_t
{
U64 total_len;
U32 seed;
U32 v1;
U32 v2;
U32 v3;
U32 v4;
int memsize;
char memory[16];
};
struct XXH_state64_t
{
U64 total_len;
U64 seed;
U64 v1;
U64 v2;
U64 v3;
U64 v4;
int memsize;
char memory[32];
};
int XXH32_sizeofState(void)
{
XXH_STATIC_ASSERT(XXH32_SIZEOFSTATE >= sizeof(struct XXH_state32_t)); // A compilation error here means XXH32_SIZEOFSTATE is not large enough
return sizeof(struct XXH_state32_t);
}
int XXH64_sizeofState(void)
{
XXH_STATIC_ASSERT(XXH64_SIZEOFSTATE >= sizeof(struct XXH_state64_t)); // A compilation error here means XXH64_SIZEOFSTATE is not large enough
return sizeof(struct XXH_state64_t);
}
XXH_errorcode XXH32_resetState(void* state_in, U32 seed)
{
struct XXH_state32_t * state = (struct XXH_state32_t *) state_in;
state->seed = seed;
state->v1 = seed + PRIME32_1 + PRIME32_2;
state->v2 = seed + PRIME32_2;
state->v3 = seed + 0;
state->v4 = seed - PRIME32_1;
state->total_len = 0;
state->memsize = 0;
return XXH_OK;
}
XXH_errorcode XXH64_resetState(void* state_in, unsigned long long seed)
{
struct XXH_state64_t * state = (struct XXH_state64_t *) state_in;
state->seed = seed;
state->v1 = seed + PRIME64_1 + PRIME64_2;
state->v2 = seed + PRIME64_2;
state->v3 = seed + 0;
state->v4 = seed - PRIME64_1;
state->total_len = 0;
state->memsize = 0;
return XXH_OK;
}
void* XXH32_init (U32 seed)
{
void* state = XXH_malloc (sizeof(struct XXH_state32_t));
XXH32_resetState(state, seed);
return state;
}
void* XXH64_init (unsigned long long seed)
{
void* state = XXH_malloc (sizeof(struct XXH_state64_t));
XXH64_resetState(state, seed);
return state;
}
FORCE_INLINE XXH_errorcode XXH32_update_endian (void* state_in, const void* input, int len, XXH_endianess endian)
{
struct XXH_state32_t * state = (struct XXH_state32_t *) state_in;
const BYTE* p = (const BYTE*)input;
const BYTE* const bEnd = p + len;
#ifdef XXH_ACCEPT_NULL_INPUT_POINTER
if (input==NULL) return XXH_ERROR;
#endif
state->total_len += len;
if (state->memsize + len < 16) // fill in tmp buffer
{
XXH_memcpy(state->memory + state->memsize, input, len);
state->memsize += len;
return XXH_OK;
}
if (state->memsize) // some data left from previous update
{
XXH_memcpy(state->memory + state->memsize, input, 16-state->memsize);
{
const U32* p32 = (const U32*)state->memory;
state->v1 += XXH_readLE32(p32, endian) * PRIME32_2; state->v1 = XXH_rotl32(state->v1, 13); state->v1 *= PRIME32_1; p32++;
state->v2 += XXH_readLE32(p32, endian) * PRIME32_2; state->v2 = XXH_rotl32(state->v2, 13); state->v2 *= PRIME32_1; p32++;
state->v3 += XXH_readLE32(p32, endian) * PRIME32_2; state->v3 = XXH_rotl32(state->v3, 13); state->v3 *= PRIME32_1; p32++;
state->v4 += XXH_readLE32(p32, endian) * PRIME32_2; state->v4 = XXH_rotl32(state->v4, 13); state->v4 *= PRIME32_1; p32++;
}
p += 16-state->memsize;
state->memsize = 0;
}
if (p <= bEnd-16)
{
const BYTE* const limit = bEnd - 16;
U32 v1 = state->v1;
U32 v2 = state->v2;
U32 v3 = state->v3;
U32 v4 = state->v4;
do
{
v1 += XXH_readLE32((const U32*)p, endian) * PRIME32_2; v1 = XXH_rotl32(v1, 13); v1 *= PRIME32_1; p+=4;
v2 += XXH_readLE32((const U32*)p, endian) * PRIME32_2; v2 = XXH_rotl32(v2, 13); v2 *= PRIME32_1; p+=4;
v3 += XXH_readLE32((const U32*)p, endian) * PRIME32_2; v3 = XXH_rotl32(v3, 13); v3 *= PRIME32_1; p+=4;
v4 += XXH_readLE32((const U32*)p, endian) * PRIME32_2; v4 = XXH_rotl32(v4, 13); v4 *= PRIME32_1; p+=4;
} while (p<=limit);
state->v1 = v1;
state->v2 = v2;
state->v3 = v3;
state->v4 = v4;
}
if (p < bEnd)
{
XXH_memcpy(state->memory, p, bEnd-p);
state->memsize = (int)(bEnd-p);
}
return XXH_OK;
}
XXH_errorcode XXH32_update (void* state_in, const void* input, unsigned int len)
{
XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
return XXH32_update_endian(state_in, input, len, XXH_littleEndian);
else
return XXH32_update_endian(state_in, input, len, XXH_bigEndian);
}
FORCE_INLINE U32 XXH32_intermediateDigest_endian (void* state_in, XXH_endianess endian)
{
struct XXH_state32_t * state = (struct XXH_state32_t *) state_in;
const BYTE * p = (const BYTE*)state->memory;
BYTE* bEnd = (BYTE*)state->memory + state->memsize;
U32 h32;
if (state->total_len >= 16)
{
h32 = XXH_rotl32(state->v1, 1) + XXH_rotl32(state->v2, 7) + XXH_rotl32(state->v3, 12) + XXH_rotl32(state->v4, 18);
}
else
{
h32 = state->seed + PRIME32_5;
}
h32 += (U32) state->total_len;
while (p<=bEnd-4)
{
h32 += XXH_readLE32((const U32*)p, endian) * PRIME32_3;
h32 = XXH_rotl32(h32, 17) * PRIME32_4;
p+=4;
}
while (p<bEnd)
{
h32 += (*p) * PRIME32_5;
h32 = XXH_rotl32(h32, 11) * PRIME32_1;
p++;
}
h32 ^= h32 >> 15;
h32 *= PRIME32_2;
h32 ^= h32 >> 13;
h32 *= PRIME32_3;
h32 ^= h32 >> 16;
return h32;
}
U32 XXH32_intermediateDigest (void* state_in)
{
XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
return XXH32_intermediateDigest_endian(state_in, XXH_littleEndian);
else
return XXH32_intermediateDigest_endian(state_in, XXH_bigEndian);
}
U32 XXH32_digest (void* state_in)
{
U32 h32 = XXH32_intermediateDigest(state_in);
XXH_free(state_in);
return h32;
}
FORCE_INLINE XXH_errorcode XXH64_update_endian (void* state_in, const void* input, int len, XXH_endianess endian)
{
struct XXH_state64_t * state = (struct XXH_state64_t *) state_in;
const BYTE* p = (const BYTE*)input;
const BYTE* const bEnd = p + len;
#ifdef XXH_ACCEPT_NULL_INPUT_POINTER
if (input==NULL) return XXH_ERROR;
#endif
state->total_len += len;
if (state->memsize + len < 32) // fill in tmp buffer
{
XXH_memcpy(state->memory + state->memsize, input, len);
state->memsize += len;
return XXH_OK;
}
if (state->memsize) // some data left from previous update
{
XXH_memcpy(state->memory + state->memsize, input, 32-state->memsize);
{
const U64* p64 = (const U64*)state->memory;
state->v1 += XXH_readLE64(p64, endian) * PRIME64_2; state->v1 = XXH_rotl64(state->v1, 31); state->v1 *= PRIME64_1; p64++;
state->v2 += XXH_readLE64(p64, endian) * PRIME64_2; state->v2 = XXH_rotl64(state->v2, 31); state->v2 *= PRIME64_1; p64++;
state->v3 += XXH_readLE64(p64, endian) * PRIME64_2; state->v3 = XXH_rotl64(state->v3, 31); state->v3 *= PRIME64_1; p64++;
state->v4 += XXH_readLE64(p64, endian) * PRIME64_2; state->v4 = XXH_rotl64(state->v4, 31); state->v4 *= PRIME64_1; p64++;
}
p += 32-state->memsize;
state->memsize = 0;
}
if (p <= bEnd-32)
{
const BYTE* const limit = bEnd - 32;
U64 v1 = state->v1;
U64 v2 = state->v2;
U64 v3 = state->v3;
U64 v4 = state->v4;
do
{
v1 += XXH_readLE64((const U64*)p, endian) * PRIME64_2; v1 = XXH_rotl64(v1, 31); v1 *= PRIME64_1; p+=8;
v2 += XXH_readLE64((const U64*)p, endian) * PRIME64_2; v2 = XXH_rotl64(v2, 31); v2 *= PRIME64_1; p+=8;
v3 += XXH_readLE64((const U64*)p, endian) * PRIME64_2; v3 = XXH_rotl64(v3, 31); v3 *= PRIME64_1; p+=8;
v4 += XXH_readLE64((const U64*)p, endian) * PRIME64_2; v4 = XXH_rotl64(v4, 31); v4 *= PRIME64_1; p+=8;
} while (p<=limit);
state->v1 = v1;
state->v2 = v2;
state->v3 = v3;
state->v4 = v4;
}
if (p < bEnd)
{
XXH_memcpy(state->memory, p, bEnd-p);
state->memsize = (int)(bEnd-p);
}
return XXH_OK;
}
XXH_errorcode XXH64_update (void* state_in, const void* input, unsigned int len)
{
XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
return XXH64_update_endian(state_in, input, len, XXH_littleEndian);
else
return XXH64_update_endian(state_in, input, len, XXH_bigEndian);
}
FORCE_INLINE U64 XXH64_intermediateDigest_endian (void* state_in, XXH_endianess endian)
{
struct XXH_state64_t * state = (struct XXH_state64_t *) state_in;
const BYTE * p = (const BYTE*)state->memory;
BYTE* bEnd = (BYTE*)state->memory + state->memsize;
U64 h64;
if (state->total_len >= 32)
{
U64 v1 = state->v1;
U64 v2 = state->v2;
U64 v3 = state->v3;
U64 v4 = state->v4;
h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
v1 *= PRIME64_2; v1 = XXH_rotl64(v1, 31); v1 *= PRIME64_1; h64 ^= v1;
h64 = h64*PRIME64_1 + PRIME64_4;
v2 *= PRIME64_2; v2 = XXH_rotl64(v2, 31); v2 *= PRIME64_1; h64 ^= v2;
h64 = h64*PRIME64_1 + PRIME64_4;
v3 *= PRIME64_2; v3 = XXH_rotl64(v3, 31); v3 *= PRIME64_1; h64 ^= v3;
h64 = h64*PRIME64_1 + PRIME64_4;
v4 *= PRIME64_2; v4 = XXH_rotl64(v4, 31); v4 *= PRIME64_1; h64 ^= v4;
h64 = h64*PRIME64_1 + PRIME64_4;
}
else
{
h64 = state->seed + PRIME64_5;
}
h64 += (U64) state->total_len;
while (p<=bEnd-8)
{
U64 k1 = XXH_readLE64((const U64*)p, endian);
k1 *= PRIME64_2; k1 = XXH_rotl64(k1,31); k1 *= PRIME64_1; h64 ^= k1;
h64 = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4;
p+=8;
}
if (p<=bEnd-4)
{
h64 ^= (U64)(XXH_readLE32((const U32*)p, endian)) * PRIME64_1;
h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
p+=4;
}
while (p<bEnd)
{
h64 ^= (*p) * PRIME64_5;
h64 = XXH_rotl64(h64, 11) * PRIME64_1;
p++;
}
h64 ^= h64 >> 33;
h64 *= PRIME64_2;
h64 ^= h64 >> 29;
h64 *= PRIME64_3;
h64 ^= h64 >> 32;
return h64;
}
unsigned long long XXH64_intermediateDigest (void* state_in)
{
XXH_endianess endian_detected = (XXH_endianess)XXH_CPU_LITTLE_ENDIAN;
if ((endian_detected==XXH_littleEndian) || XXH_FORCE_NATIVE_FORMAT)
return XXH64_intermediateDigest_endian(state_in, XXH_littleEndian);
else
return XXH64_intermediateDigest_endian(state_in, XXH_bigEndian);
}
unsigned long long XXH64_digest (void* state_in)
{
U64 h64 = XXH64_intermediateDigest(state_in);
XXH_free(state_in);
return h64;
}
================================================
FILE: algorithm/xxhash.h
================================================
/*
xxHash - Extremely Fast Hash algorithm
Header File
Copyright (C) 2012-2014, Yann Collet.
BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
You can contact the author at :
- xxHash source repository : http://code.google.com/p/xxhash/
*/
/* Notice extracted from xxHash homepage :
xxHash is an extremely fast Hash algorithm, running at RAM speed limits.
It also successfully passes all tests from the SMHasher suite.
Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz)
Name Speed Q.Score Author
xxHash 5.4 GB/s 10
CrapWow 3.2 GB/s 2 Andrew
MumurHash 3a 2.7 GB/s 10 Austin Appleby
SpookyHash 2.0 GB/s 10 Bob Jenkins
SBox 1.4 GB/s 9 Bret Mulvey
Lookup3 1.2 GB/s 9 Bob Jenkins
SuperFastHash 1.2 GB/s 1 Paul Hsieh
CityHash64 1.05 GB/s 10 Pike & Alakuijala
FNV 0.55 GB/s 5 Fowler, Noll, Vo
CRC32 0.43 GB/s 9
MD5-32 0.33 GB/s 10 Ronald L. Rivest
SHA1-32 0.28 GB/s 10
Q.Score is a measure of quality of the hash function.
It depends on successfully passing SMHasher test set.
10 is a perfect score.
*/
#pragma once
#if defined (__cplusplus)
extern "C" {
#endif
/*****************************
Type
*****************************/
typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode;
/*****************************
Simple Hash Functions
*****************************/
unsigned int XXH32 (const void* input, unsigned int len, unsigned int seed);
unsigned long long XXH64 (const void* input, unsigned int len, unsigned long long seed);
/*
XXH32() :
Calculate the 32-bits hash of sequence of length "len" stored at memory address "input".
The memory between input & input+len must be valid (allocated and read-accessible).
"seed" can be used to alter the result predictably.
This function successfully passes all SMHasher tests.
Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s
Note that "len" is type "int", which means it is limited to 2^31-1.
If your data is larger, use the advanced functions below.
XXH64() :
Calculate the 64-bits hash of sequence of length "len" stored at memory address "input".
*/
/*****************************
Advanced Hash Functions
*****************************/
void* XXH32_init (unsigned int seed);
XXH_errorcode XXH32_update (void* state, const void* input, unsigned int len);
unsigned int XXH32_digest (void* state);
void* XXH64_init (unsigned long long seed);
XXH_errorcode XXH64_update (void* state, const void* input, unsigned int len);
unsigned long long XXH64_digest (void* state);
/*
These functions calculate the xxhash of an input provided in several small packets,
as opposed to an input provided as a single block.
It must be started with :
void* XXHnn_init()
The function returns a pointer which holds the state of calculation.
This pointer must be provided as "void* state" parameter for XXHnn_update().
XXHnn_update() can be called as many times as necessary.
The user must provide a valid (allocated) input.
The function returns an error code, with 0 meaning OK, and any other value meaning there is an error.
Note that "len" is type "int", which means it is limited to 2^31-1.
If your data is larger, it is recommended to chunk your data into blocks
of size for example 2^30 (1GB) to avoid any "int" overflow issue.
Finally, you can end the calculation anytime, by using XXHnn_digest().
This function returns the final nn-bits hash.
You must provide the same "void* state" parameter created by XXHnn_init().
Memory will be freed by XXHnn_digest().
*/
int XXH32_sizeofState(void);
XXH_errorcode XXH32_resetState(void* state, unsigned int seed);
#define XXH32_SIZEOFSTATE 48
typedef struct { long long ll[(XXH32_SIZEOFSTATE+(sizeof(long long)-1))/sizeof(long long)]; } XXH32_stateSpace_t;
int XXH64_sizeofState(void);
XXH_errorcode XXH64_resetState(void* state, unsigned long long seed);
#define XXH64_SIZEOFSTATE 88
typedef struct { long long ll[(XXH64_SIZEOFSTATE+(sizeof(long long)-1))/sizeof(long long)]; } XXH64_stateSpace_t;
/*
These functions allow user application to make its own allocation for state.
XXHnn_sizeofState() is used to know how much space must be allocated for the xxHash nn-bits state.
Note that the state must be aligned to access 'long long' fields. Memory must be allocated and referenced by a pointer.
This pointer must then be provided as 'state' into XXHnn_resetState(), which initializes the state.
For static allocation purposes (such as allocation on stack, or freestanding systems without malloc()),
use the structure XXHnn_stateSpace_t, which will ensure that memory space is large enough and correctly aligned to access 'long long' fields.
*/
unsigned int XXH32_intermediateDigest (void* state);
unsigned long long XXH64_intermediateDigest (void* state);
/*
This function does the same as XXHnn_digest(), generating a nn-bit hash,
but preserve memory context.
This way, it becomes possible to generate intermediate hashes, and then continue feeding data with XXHnn_update().
To free memory context, use XXHnn_digest(), or free().
*/
#if defined (__cplusplus)
}
#endif
================================================
FILE: cache/rate_limiter.h
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
#ifndef KINGDB_RATE_LIMITER_H_
#define KINGDB_RATE_LIMITER_H_
#include "util/debug.h"
#include <cinttypes>
namespace kdb {
class RateLimiter {
public:
RateLimiter(uint64_t rate_limit)
: rate_limit_(rate_limit),
rate_incoming_(250 * 1024 * 1024),
rate_incoming_adjusted_(0),
rate_writing_default_(1 * 1024 * 1024),
epoch_last_(0),
epoch_current_(0),
duration_slept_(0),
bytes_per_microsecond_(5)
{
}
~RateLimiter() {
}
void Tick(uint64_t bytes_incoming) {
epoch_current_ = std::time(0);
if (epoch_current_ != epoch_last_) {
rate_incoming_adjusted_ = rate_incoming_ + bytes_per_microsecond_ * duration_slept_;
log::trace("RateLimiter::Tick()",
"rate_incoming_: %" PRIu64 " rate_incoming_adjusted_:%" PRIu64,
rate_incoming_, rate_incoming_adjusted_);
duration_slept_ = 0;
rate_incoming_ = 0;
epoch_last_ = epoch_current_;
uint64_t rate_writing = GetWritingRate();
double ratio = (double)rate_incoming_adjusted_ / (double)rate_writing;
if (ratio > 1.0) {
// The rate of incoming data is greater than the rate at which data
// can be written, therefore the number of bytes for each microsecond
// slept must be decreased: this means that less bytes will trigger the
// same amount of microseconds slept, which will increase the amount
// of time spent sleeping, and bring the rate of incoming data closer
// to the rate at which data is written.
if (ratio > 1.50) {
bytes_per_microsecond_ *= 0.75;
} else if (ratio > 1.10) {
bytes_per_microsecond_ *= 0.95;
} else if (ratio > 1.05) {
bytes_per_microsecond_ *= 0.99;
} else {
bytes_per_microsecond_ *= 0.995;
}
if (bytes_per_microsecond_ <= 5) bytes_per_microsecond_ += 1;
log::trace("RateLimiter::Tick()", "decreasing");
} else {
// The rate of incoming data is lower than the rate at which data
// can be written, therefore bytes_per_microsecond_ needs to be
// increased.
if (ratio < 0.5) {
bytes_per_microsecond_ *= 1.25;
} else if (ratio < 0.90) {
bytes_per_microsecond_ *= 1.05;
} else if (ratio < 0.95) {
bytes_per_microsecond_ *= 1.01;
} else {
bytes_per_microsecond_ *= 1.005;
}
log::trace("RateLimiter::Tick()", "increasing");
if (bytes_per_microsecond_ <= 5) bytes_per_microsecond_ += 1;
}
log::trace("RateLimiter::Tick()",
"limit rate: bytes_per_microsecond_: %" PRIu64 " rate_writing:%" PRIu64,
bytes_per_microsecond_, rate_writing);
}
mutex_throttling_.lock();
rate_incoming_ += bytes_incoming;
uint64_t sleep_microseconds = 0;
if (bytes_per_microsecond_ > 0) {
sleep_microseconds = bytes_incoming / bytes_per_microsecond_;
}
mutex_throttling_.unlock();
if (sleep_microseconds > 50000) sleep_microseconds = 50000;
if (sleep_microseconds) {
log::trace("RateLimiter::Tick()",
"bytes_per_microsecond_: %" PRIu64 ", sleep_microseconds: %" PRIu64,
bytes_per_microsecond_, sleep_microseconds);
std::chrono::microseconds duration(sleep_microseconds);
std::this_thread::sleep_for(duration);
duration_slept_ += sleep_microseconds;
}
}
void WriteStart() {
auto epoch = std::chrono::system_clock::now().time_since_epoch();
epoch_write_start_ = std::chrono::duration_cast<std::chrono::milliseconds>(epoch).count();
}
void WriteEnd(uint64_t num_bytes_written) {
auto epoch = std::chrono::system_clock::now().time_since_epoch();
epoch_write_end_ = std::chrono::duration_cast<std::chrono::milliseconds>(epoch).count();
uint64_t rate_writing = num_bytes_written;
mutex_throttling_.lock();
if (epoch_write_start_ == epoch_write_end_) {
rate_writing = num_bytes_written;
} else {
double duration = ((double)epoch_write_end_ - (double)epoch_write_start_) / 1000;
rate_writing = (uint64_t)((double)num_bytes_written / (double)duration);
}
StoreWritingRate(rate_writing);
mutex_throttling_.unlock();
}
void StoreWritingRate(uint64_t rate) {
if (rates_.size() >= 10) {
rates_.erase(rates_.begin());
}
rates_.push_back(rate);
}
uint64_t GetWritingRate() {
// If no rate has been stored yet, a default value is used.
if (rates_.size() == 0) return rate_writing_default_;
// The writting rate is an average: this allows to cope with irregularities
// in the throughput and prevent the rate limiter to fall into a flapping
// effect, in which it would limit the throughput either way too much,
// or not enough. This average allow the bandwitdth to converge smoothly.
uint64_t sum = 0;
for (size_t i = 0; i < rates_.size(); i++) {
sum += rates_[i];
}
uint64_t rate_limit_current = sum / rates_.size();
if (rate_limit_ > 0 && rate_limit_ < rate_limit_current) {
return rate_limit_;
} else if (rate_limit_current > 0){
return rate_limit_current;
} else {
return rate_writing_default_;
}
}
uint64_t epoch_write_start_;
uint64_t epoch_write_end_;
uint64_t rate_limit_;
uint64_t rate_incoming_;
uint64_t rate_incoming_adjusted_;
uint64_t rate_writing_;
uint64_t rate_writing_default_;
uint64_t epoch_last_;
uint64_t epoch_current_;
uint64_t duration_slept_;
uint64_t bytes_per_microsecond_;
std::mutex mutex_throttling_;
std::vector<uint64_t> rates_;
};
} // namespace kdb
#endif // KINGDB_RATE_LIMITER_H_
================================================
FILE: cache/write_buffer.cc
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
#include "cache/write_buffer.h"
namespace kdb {
void WriteBuffer::Flush() {
std::unique_lock<std::mutex> lock_flush(mutex_flush_level2_);
if (IsStopRequestedAndBufferEmpty()) return;
// NOTE: Doing the flushing and waiting twice, in case the two buffers,
// 'live' and 'copy', have items. This is a quick hack and a better
// solution should be investigated.
for (auto i = 0; i < 2; i++) {
log::debug("LOCK", "2 lock");
cv_flush_.notify_one();
cv_flush_done_.wait_for(lock_flush, std::chrono::milliseconds(db_options_.internal__close_timeout));
}
log::trace("WriteBuffer::Flush()", "end");
}
Status WriteBuffer::Get(ReadOptions& read_options, ByteArray& key, ByteArray* value_out) {
// NOTE: The lookups is done by iterating over vectors, which is fine
// as long as the buffer doesn't become too big. This may need to be
// changed for something faster at some point.
// TODO: make sure the live buffer doesn't need to be protected by a mutex in
// order to be accessed -- right now I'm relying on timing, but that may
// be too weak to guarantee proper access
// TODO: for items being stored that are not small enough, only parts will
// be found in the buffers -- should the kv-store return "not found"
// or should it try to send the data from the disk and the partially
// available parts in the buffer?
if (IsStopRequested()) return Status::IOError("Cannot handle request: WriteBuffer is closing");
// read the "live" buffer
mutex_live_write_level1_.lock();
log::debug("LOCK", "1 lock");
mutex_indices_level3_.lock();
log::debug("LOCK", "3 lock");
auto& buffer_live = buffers_[im_live_];
int num_items = buffer_live.size();
mutex_indices_level3_.unlock();
log::debug("LOCK", "3 unlock");
mutex_live_write_level1_.unlock();
log::debug("LOCK", "1 unlock");
bool found = false;
Order order_found;
for (int i = 0; i < num_items; i++) {
auto& order = buffer_live[i];
if (order.key == key) {
found = true;
order_found = order;
}
}
if (found) {
log::debug("WriteBuffer::Get()", "found in buffer_live");
if ( order_found.type == OrderType::Put
&& order_found.IsSelfContained()) {
*value_out = order_found.chunk;
(*value_out).set_size(order_found.size_value);
(*value_out).set_size_compressed(order_found.size_value_compressed);
return Status::OK();
} else if (order_found.type == OrderType::Delete) {
return Status::DeleteOrder();
} else {
return Status::NotFound("Unable to find entry");
}
}
// prepare to read the "copy" buffer
log::debug("LOCK", "4 lock");
mutex_copy_write_level4_.lock();
log::debug("LOCK", "5 lock");
mutex_copy_read_level5_.lock();
num_readers_ += 1;
mutex_copy_read_level5_.unlock();
log::debug("LOCK", "5 unlock");
mutex_copy_write_level4_.unlock();
log::debug("LOCK", "4 unlock");
// read from "copy" buffer
found = false;
log::debug("LOCK", "3 lock");
mutex_indices_level3_.lock();
auto& buffer_copy = buffers_[im_copy_];
mutex_indices_level3_.unlock();
log::debug("LOCK", "3 unlock");
for (auto& order: buffer_copy) {
if (order.key == key) {
found = true;
order_found = order;
}
}
Status s;
if (found) log::debug("WriteBuffer::Get()", "found in buffer_copy");
if ( found
&& order_found.type == OrderType::Put
&& order_found.IsSelfContained()) {
*value_out = order_found.chunk;
(*value_out).set_size(order_found.size_value);
(*value_out).set_size_compressed(order_found.size_value_compressed);
} else if ( found
&& order_found.type == OrderType::Delete) {
s = Status::DeleteOrder();
} else {
s = Status::NotFound("Unable to find entry");
}
// exit the "copy" buffer
log::debug("LOCK", "5 lock");
mutex_copy_read_level5_.lock();
num_readers_ -= 1;
mutex_copy_read_level5_.unlock();
log::debug("LOCK", "5 unlock");
cv_read_.notify_one();
return s;
}
Status WriteBuffer::Put(WriteOptions& write_options, ByteArray& key, ByteArray& chunk) {
//return Write(OrderType::Put, key, value);
return Status::InvalidArgument("WriteBuffer::Put() is not implemented");
}
Status WriteBuffer::PutPart(WriteOptions& write_options,
ByteArray& key,
ByteArray& chunk,
uint64_t offset_chunk,
uint64_t size_value,
uint64_t size_value_compressed,
uint32_t crc32) {
return WritePart(write_options,
OrderType::Put,
key,
chunk,
offset_chunk,
size_value,
size_value_compressed,
crc32
);
}
Status WriteBuffer::Delete(WriteOptions& write_options, ByteArray& key) {
auto empty = ByteArray::NewEmptyByteArray();
return WritePart(write_options, OrderType::Delete, key, empty, 0, 0, 0, 0);
}
Status WriteBuffer::WritePart(const WriteOptions& write_options,
const OrderType& op,
ByteArray& key,
ByteArray& chunk,
uint64_t offset_chunk,
uint64_t size_value,
uint64_t size_value_compressed,
uint32_t crc32) {
if (IsStopRequested()) return Status::IOError("Cannot handle request: WriteBuffer is closing");
log::trace("WriteBuffer::WritePart()",
"key:[%s] | size chunk:%" PRIu64 ", total size value:%" PRIu64 " offset_chunk:%" PRIu64 " sizeOfBuffer:%d",
key.ToString().c_str(), chunk.size(), size_value, offset_chunk, buffers_[im_live_].size());
bool is_first_part = (offset_chunk == 0);
bool is_large = key.size() + size_value > db_options_.storage__hstable_size;
uint64_t bytes_arriving = 0;
if (is_first_part) bytes_arriving += key.size();
bytes_arriving += chunk.size();
if (UseRateLimiter()) rate_limiter_.Tick(bytes_arriving);
// TODO: here the buffer index im_live_ is called outside of the level 2 and 3 mutexes is this really safe?
log::debug("LOCK", "1 lock");
std::unique_lock<std::mutex> lock_live(mutex_live_write_level1_);
mutex_indices_level3_.lock();
buffers_[im_live_].push_back(Order{std::this_thread::get_id(),
write_options,
op,
key,
chunk,
offset_chunk,
size_value,
size_value_compressed,
crc32,
is_large});
sizes_[im_live_] += bytes_arriving;
uint64_t size_buffer_live = sizes_[im_live_];
mutex_indices_level3_.unlock();
/*
if (buffers_[im_live_].size()) {
for(auto &p: buffers_[im_live_]) {
log::trace("WriteBuffer::WritePart()",
"Write() ITEM key_ptr:[%p] key:[%s] | size chunk:%d, total size value:%d offset_chunk:%" PRIu64 " sizeOfBuffer:%d sizes_[im_live_]:%d",
p.key, p.key->ToString().c_str(), p.chunk->size(), p.size_value, p.offset_chunk, buffers_[im_live_].size(), sizes_[im_live_]);
}
} else {
log::trace("WriteBuffer::WritePart()", "Write() ITEM no buffers_[im_live_]");
}
*/
if (size_buffer_live > buffer_size_) {
log::trace("WriteBuffer::WritePart()", "trying to swap");
mutex_flush_level2_.lock();
log::debug("LOCK", "2 lock");
log::debug("LOCK", "3 lock");
std::unique_lock<std::mutex> lock_swap(mutex_indices_level3_);
cv_flush_.notify_one();
log::debug("LOCK", "3 unlock");
mutex_flush_level2_.unlock();
log::debug("LOCK", "2 unlock");
} else {
log::trace("WriteBuffer::WritePart()", "will not swap");
}
log::debug("LOCK", "1 unlock");
return Status::OK();
}
void WriteBuffer::ProcessingLoop() {
while(true) {
bool force_sync = false;
log::trace("WriteBuffer", "ProcessingLoop() - start");
log::debug("LOCK", "2 lock");
std::unique_lock<std::mutex> lock_flush(mutex_flush_level2_);
while (sizes_[im_live_] == 0) {
log::trace("WriteBuffer", "ProcessingLoop() - wait - %" PRIu64 " %" PRIu64, buffers_[im_copy_].size(), buffers_[im_live_].size());
std::cv_status status = cv_flush_.wait_for(lock_flush, std::chrono::milliseconds(db_options_.write_buffer__flush_timeout));
if (IsStopRequestedAndBufferEmpty()) return;
if (status == std::cv_status::no_timeout) {
force_sync = true;
}
}
mutex_indices_level3_.lock();
if (sizes_[im_copy_] == 0) {
std::swap(im_live_, im_copy_);
}
mutex_indices_level3_.unlock();
log::trace("WriteBuffer", "ProcessingLoop() - start swap - %" PRIu64 " %" PRIu64, buffers_[im_copy_].size(), buffers_[im_live_].size());
// Notify the storage engine that the buffer can be flushed
log::trace("BM", "WAIT: Get()-flush_buffer");
if (UseRateLimiter()) rate_limiter_.WriteStart();
if (force_sync && buffers_[im_copy_].size()) {
buffers_[im_copy_][0].write_options.sync = true;
}
event_manager_->flush_buffer.StartAndBlockUntilDone(buffers_[im_copy_]);
// Wait for the index to notify the buffer manager
log::trace("BM", "WAIT: Get()-clear_buffer");
event_manager_->clear_buffer.Wait();
event_manager_->clear_buffer.Done();
// Wait for readers
// TODO: the cleaning of the flush buffer shouldn't be done in one go but
// in multiple iterations just like the transfer of indexes is being done,
// so that the readers are never blocked for too long.
log::debug("LOCK", "4 lock");
mutex_copy_write_level4_.lock();
while(true) {
log::debug("LOCK", "5 lock");
std::unique_lock<std::mutex> lock_read(mutex_copy_read_level5_);
if (num_readers_ == 0) break;
log::debug("WriteBuffer", "ProcessingLoop() - wait for lock_read");
cv_read_.wait(lock_read);
}
log::debug("LOCK", "5 unlock");
if (UseRateLimiter()) rate_limiter_.WriteEnd(sizes_[im_copy_]);
log::trace("WriteBuffer", "ProcessingLoop() bytes_in_buffer: %" PRIu64 " rate_writing: %" PRIu64, sizes_[im_copy_], rate_limiter_.GetWritingRate());
// Clear flush buffer
log::debug("WriteBuffer::ProcessingLoop()", "clear flush buffer");
/*
if (buffers_[im_copy_].size()) {
for(auto &p: buffers_[im_copy_]) {
log::trace("WriteBuffer", "ProcessingLoop() ITEM im_copy - key:[%s] | size chunk:%d, total size value:%d offset_chunk:%" PRIu64 " sizeOfBuffer:%d sizes_[im_copy_]:%d", p.key.ToString().c_str(), p.chunk.size(), p.size_value, p.offset_chunk, buffers_[im_copy_].size(), sizes_[im_copy_]);
}
} else {
log::trace("WriteBuffer", "ProcessingLoop() ITEM no buffers_[im_copy_]");
}
if (buffers_[im_live_].size()) {
for(auto &p: buffers_[im_live_]) {
log::trace("WriteBuffer", "ProcessingLoop() ITEM im_live - key:[%s] | size chunk:%d, total size value:%d offset_chunk:%" PRIu64 " sizeOfBuffer:%d sizes_[im_live_]:%d", p.key.ToString().c_str(), p.chunk.size(), p.size_value, p.offset_chunk, buffers_[im_live_].size(), sizes_[im_live_]);
}
} else {
log::trace("WriteBuffer", "ProcessingLoop() ITEM no buffers_[im_live_]");
}
*/
// Note: the call to clear() can delete a lot of allocated memory at once,
// which may block all other operations for a while: this may benefit
// from throttling (using db_options_.internal__num_iterations_per_lock)
sizes_[im_copy_] = 0;
buffers_[im_copy_].clear();
log::trace("WriteBuffer", "ProcessingLoop() - end swap - %" PRIu64 " %" PRIu64, buffers_[im_copy_].size(), buffers_[im_live_].size());
mutex_copy_write_level4_.unlock();
log::debug("LOCK", "4 unlock");
log::debug("LOCK", "2 unlock");
cv_flush_done_.notify_all();
if (IsStopRequestedAndBufferEmpty()) return;
}
}
} // namespace kdb
================================================
FILE: cache/write_buffer.h
================================================
// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.
// Use of this source code is governed by the BSD 3-Clause License,
// that can be found in the LICENSE file.
#ifndef KINGDB_WRITE_BUFFER_H_
#define KINGDB_WRITE_BUFFER_H_
#include "util/debug.h"
#include <cinttypes>
#include <thread>
#include <map>
#include <array>
#include <string>
#include <vector>
#include <chrono>
#include <ctime>
#include "util/options.h"
#include "util/byte_array.h"
#include "util/order.h"
#include "cache/rate_limiter.h"
#include "thread/event_manager.h"
namespace kdb {
class WriteBuffer {
public:
WriteBuffer(const DatabaseOptions& db_options,
EventManager *event_manager)
: db_options_(db_options),
event_manager_(event_manager),
rate_limiter_(db_options.rate_limit_incoming) {
stop_requested_ = false;
im_live_ = 0;
im_copy_ = 1;
sizes_[im_live_] = 0;
sizes_[im_copy_] = 0;
num_readers_ = 0;
buffer_size_ = db_options_.write_buffer__size / 2;
thread_buffer_handler_ = std::thread(&WriteBuffer::ProcessingLoop, this);
is_closed_ = false;
log::debug("WriteBuffer::ctor()", "WriteBuffer::ctor() %" PRIu64 " - %s\n", db_options_.rate_limit_incoming, db_options_.write_buffer__mode_str.c_str());
if (db_options_.write_buffer__mode == kWriteBufferModeAdaptive) {
log::debug("WriteBuffer::ctor()", "WriteBuffer::ctor() write buffer mode adaptive\n");
} else {
log::debug("WriteBuffer::ctor()", "WriteBuffer::ctor() write buffer mode direct\n");
}
}
~WriteBuffer() { Close(); }
Status Get(ReadOptions& read_options, ByteArray& key, ByteArray* value_out);
Status Put(WriteOptions& write_options, ByteArray& key, ByteArray& chunk);
Status PutPart(WriteOptions& write_options,
ByteArray& key,
ByteArray& chunk,
uint64_t offset_chunk,
uint64_t size_value,
uint64_t size_value_compressed,
uint32_t crc32);
Status Delete(WriteOptions& write_options, ByteArray& key);
void Flush();
void Close () {
std::unique_lock<std::mutex> lock(mutex_close_);
if (is_closed_) return;
is_closed_ = true;
Stop();
Flush();
cv_flush_.notify_one();
thread_buffer_handler_.join();
}
bool UseRateLimiter() {
if ( db_options_.write_buffer__mode == kWriteBufferModeAdaptive
|| db_options_.rate_limit_incoming > 0) {
return true;
}
return false;
}
bool IsStopRequestedAndBufferEmpty() {
return ( IsStopRequested()
&& buffers_[im_live_].empty()
&& buffers_[im_copy_].empty());
}
bool IsStopRequested() { return stop_requested_; }
void Stop() { stop_requested_ = true; }
bool stop_requested_;
private:
Status WritePart(const WriteOptions& write_options,
const OrderType& op,
ByteArray& key,
ByteArray& chunk,
uint64_t offset_chunk,
uint64_t size_value,
uint64_t size_value_compressed,
uint32_t crc32);
void ProcessingLoop();
DatabaseOptions db_options_;
int im_live_;
int im_copy_;
uint64_t buffer_size_;
int num_readers_;
std::array<std::vector<Order>, 2> buffers_;
std::array<int, 2> sizes_;
bool is_closed_;
std::mutex mutex_close_;
std::thread thread_buffer_handler_;
EventManager *event_manager_;
RateLimiter rate_limiter_;
// Using a lock hierarchy to avoid deadlocks
std::mutex mutex_live_write_level1_;
std::mutex mutex_flush_level2_;
std::mutex mutex_indices_level3_;
std::mutex mutex_copy_write_level4_;
std::mutex mutex_copy_read_level5_;
std::condition_variable cv_flush_;
std::condition_variable cv_flush_done_;
std::condition_variable cv_read_;
};
} // namespace kdb
#endif // KINGDB_WRITE_BUFFER_H_
================================================
FILE: doc/bench/benchmarks.md
================================================
Benchmarks of KingDB v0.9.0 and LevelDB v1.18
=============================================
Date: April 9, 2015
##TL;DR
Benchmarks were run for different value sizes, from 100 bytes to 1MB, using the benchmark framework of LevelDB. The benchmark program for KingDB can be found in `doc/bench/db_bench_kingdb.cc`
- Random writes: KingDB is faster than LevelDB by 2-10x.
- Sequential writes: KingDB and LevelDB are equivalent.
- Overwrites: KingDB is faster than LevelDB by 2-14x
- Random reads: Before compaction, KingDB is faster than LevelDB by 1.5-3x. However after compaction, LevelDB is faster than KingDB by around 1.7x.
- Sequential reads: LevelDB is faster than KingDB by 3-50x. Yes, KingDB is really bad for sequential reads.
- Compaction: LevelDB has faster compactions than KingDB, by 3-4x.
##Description
The benchmarks were made over two systems, a Linux CentOS 6.4 and a Mac OS X 10.9.5.
###Linux
- Operating System: CentOS 6.4
- RAM: 128GB DDR3
- CPU: 12-core Intel Xeon E5-2640 @ 2.50GHz
- Compiler: GCC 4.9.2
- Compiler and linker options: \-O2 \-fno-builtin-memcmp \-ltcmalloc
- Storage: hard drive
- File system: ext3
###Mac
- Operating System: OS X 10.9.5
- RAM: 8GB DDR3
- CPU: 4-core Intel Core i7 @ 2.3GHz
- Compiler: Apple LLVM version 6.0
- Compiler and linker options: \-O2
- Storage: solid state drive
- File system: HFS+
###Key-value store options
- LevelDB v1.18 has a write cache of 4MB, and uses Snappy v1.1.2 for compression.
- KingDB v0.9.0 has a write cache of 4MB, uses LZ4 v1.3.0 for compression.
##Results
For each workload, multiple value sizes are tested. The performance of LevelDB and KingDB are compared using [fold change](http://en.wikipedia.org/wiki/Fold_change) metrics. The first fold change column refers to the Linux sytem, and the second fold change column refers to the Mac system. A positive fold change means that KingDB is faster than LevelDB, and a negative fold change means that LevelDB is faster than KingDB.
Except for the value size and the fold changes, the metric for all the values in table are in number of operations per second (ops/second).
| Workload | Value size | LevelDB Linux | KingDB Linux | fold change | LevelDB Mac | KingDB Mac | fold change |
| --------------: | ---------: | -------------: | -------------: | ----------: | -------------: | -------------: | ----------: |
| fillseq | 100b | 265957 | 233808 | -1.14x | 224064 | 255885 | 1.14x |
| fillseq | 1kb | 95721 | 99127 | 1.04x | 117827 | 151057 | 1.28x |
| fillseq | 100kb | 1605 | 1827 | 1.14x | 1646 | 2953 | 1.79x |
| fillseq | 256kb | 674 | 670 | -1.01x | 690 | 1303 | 1.89x |
| fillseq | 512kb | 336 | 368 | 1.10x | 317 | 649 | 2.05x |
| fillseq | 1mb | 169 | 180 | 1.07x | 140 | 313 | 2.24x |
| fillrandom | 100b | 127551 | 251
gitextract_ytklrwk_/
├── .gitignore
├── 3rdparty/
│ ├── leveldb/
│ │ ├── AUTHORS
│ │ └── LICENSE
│ ├── lz4/
│ │ └── LICENSE
│ ├── murmurhash3/
│ │ └── LICENSE
│ ├── xxhash/
│ │ └── LICENSE
│ └── zlib/
│ └── LICENSE
├── LICENSE
├── Makefile
├── README.md
├── algorithm/
│ ├── coding.cc
│ ├── coding.h
│ ├── compressor.cc
│ ├── compressor.h
│ ├── crc32c.cc
│ ├── crc32c.h
│ ├── endian.cc
│ ├── endian.h
│ ├── hash.cc
│ ├── hash.h
│ ├── lz4.cc
│ ├── lz4.h
│ ├── murmurhash3.cc
│ ├── murmurhash3.h
│ ├── xxhash.cc
│ └── xxhash.h
├── cache/
│ ├── rate_limiter.h
│ ├── write_buffer.cc
│ └── write_buffer.h
├── doc/
│ ├── bench/
│ │ ├── benchmarks.md
│ │ ├── db_bench_kingdb.cc
│ │ └── generate_benchmarks_table.py
│ ├── kingdb.md
│ └── kingserver.md
├── include/
│ └── kingdb/
│ └── kdb.h
├── interface/
│ ├── database.cc
│ ├── database.h
│ ├── iterator.h
│ ├── kingdb.h
│ ├── multipart.h
│ └── snapshot.h
├── network/
│ ├── client.h
│ ├── client_main.cc
│ ├── server.cc
│ ├── server.h
│ └── server_main.cc
├── storage/
│ ├── format.h
│ ├── hstable_manager.h
│ ├── resource_manager.h
│ └── storage_engine.h
├── thread/
│ ├── event_manager.h
│ ├── threadpool.h
│ └── threadstorage.h
├── unit-tests/
│ ├── client_embedded.cc
│ ├── dummy_interface.h
│ ├── dummy_storage_engine.h
│ ├── dummy_storage_engine_index.h
│ ├── kingdb_user.cc
│ ├── test_compression.cc
│ ├── test_db.cc
│ ├── testharness.cc
│ └── testharness.h
└── util/
├── byte_array.h
├── config_parser.h
├── debug.cc
├── debug.h
├── file.h
├── filepool.h
├── logger.cc
├── logger.h
├── options.h
├── order.h
├── status.cc
├── status.h
└── version.h
SYMBOL INDEX (349 symbols across 58 files)
FILE: algorithm/coding.cc
type kdb (line 14) | namespace kdb {
function EncodeFixed32 (line 16) | void EncodeFixed32(char* buf, uint32_t value) {
function EncodeFixed64 (line 27) | void EncodeFixed64(char* buf, uint64_t value) {
function GetFixed32 (line 42) | void GetFixed32(const char* buf, uint32_t* value) {
function GetFixed64 (line 54) | void GetFixed64(const char* buf, uint64_t* value) {
function PutFixed32 (line 71) | void PutFixed32(std::string* dst, uint32_t value) {
function PutFixed64 (line 77) | void PutFixed64(std::string* dst, uint64_t value) {
function PutVarint32 (line 111) | void PutVarint32(std::string* dst, uint32_t v) {
function PutVarint64 (line 128) | void PutVarint64(std::string* dst, uint64_t v) {
function VarintLength (line 134) | int VarintLength(uint64_t v) {
function GetVarint32 (line 162) | int GetVarint32(char* data, uint64_t size, uint32_t* value) {
function GetVarint64 (line 190) | int GetVarint64(char* data, uint64_t size, uint64_t* value) {
FILE: algorithm/coding.h
function namespace (line 28) | namespace kdb {
FILE: algorithm/compressor.cc
type kdb (line 7) | namespace kdb {
function Status (line 15) | Status CompressorLZ4::Compress(char *source,
function Status (line 75) | Status CompressorLZ4::Uncompress(char *source,
function Status (line 140) | Status CompressorLZ4::UncompressByteArray(ByteArray& value,
FILE: algorithm/compressor.h
function namespace (line 22) | namespace kdb {
FILE: algorithm/crc32c.cc
type kdb (line 23) | namespace kdb {
type crc32c (line 24) | namespace crc32c {
function LE_LOAD32 (line 292) | static inline uint32_t LE_LOAD32(const uint8_t *p) {
function Extend (line 296) | uint32_t Extend(uint32_t crc, const char* buf, size_t size) {
function ulong (line 344) | ulong gf2_matrix_times (ulong *mat, ulong vec)
function gf2_matrix_square (line 356) | void gf2_matrix_square (ulong *square, ulong *mat)
function ulong (line 363) | ulong Combine(ulong crc1, ulong crc2, ulong len2)
function crc8 (line 463) | uint8_t crc8(unsigned crc, unsigned char *data, size_t len)
function crc8 (line 477) | uint8_t crc8(unsigned crc, char *data, size_t len)
FILE: algorithm/crc32c.h
function namespace (line 29) | namespace kdb {
FILE: algorithm/endian.cc
type kdb (line 3) | namespace kdb {
function endian_t (line 5) | endian_t getEndianness() {
FILE: algorithm/endian.h
function namespace (line 11) | namespace kdb {
FILE: algorithm/hash.cc
type kdb (line 7) | namespace kdb {
function Hash (line 24) | Hash* MakeHash(HashType ht) {
FILE: algorithm/hash.h
function namespace (line 19) | namespace kdb {
FILE: algorithm/lz4.cc
function LZ4_NbCommonBytes (line 303) | static int LZ4_NbCommonBytes (register U64 val)
function LZ4_NbCommonBytes (line 335) | static int LZ4_NbCommonBytes (register U32 val)
function LZ4_versionNumber (line 370) | int LZ4_versionNumber (void) { return LZ4_VERSION_NUMBER; }
function LZ4_compressBound (line 371) | int LZ4_compressBound(int isize) { return LZ4_COMPRESSBOUND(isize); }
function LZ4_hashSequence (line 373) | static int LZ4_hashSequence(U32 sequence, tableType_t tableType)
function LZ4_hashPosition (line 381) | static int LZ4_hashPosition(const BYTE* p, tableType_t tableType) { retu...
function LZ4_putPositionOnHash (line 383) | static void LZ4_putPositionOnHash(const BYTE* p, U32 h, void* tableBase,...
function LZ4_putPosition (line 393) | static void LZ4_putPosition(const BYTE* p, void* tableBase, tableType_t ...
function BYTE (line 399) | static const BYTE* LZ4_getPositionOnHash(U32 h, void* tableBase, tableTy...
function BYTE (line 406) | static const BYTE* LZ4_getPosition(const BYTE* p, void* tableBase, table...
function LZ4_count (line 412) | static unsigned LZ4_count(const BYTE* pIn, const BYTE* pRef, const BYTE*...
function LZ4_compress_generic (line 431) | static int LZ4_compress_generic(
function LZ4_compress (line 644) | int LZ4_compress(const char* source, char* dest, int inputSize)
function LZ4_compress_limitedOutput (line 664) | int LZ4_compress_limitedOutput(const char* source, char* dest, int input...
function LZ4_resetStream (line 694) | void LZ4_resetStream (LZ4_stream_t* LZ4_stream)
function LZ4_stream_t (line 699) | LZ4_stream_t* LZ4_createStream(void)
function LZ4_freeStream (line 706) | int LZ4_freeStream (LZ4_stream_t* LZ4_stream)
function LZ4_loadDict (line 713) | int LZ4_loadDict (LZ4_stream_t* LZ4_dict, const char* dictionary, int di...
function LZ4_renormDictT (line 746) | static void LZ4_renormDictT(LZ4_stream_t_internal* LZ4_dict, const BYTE*...
function FORCE_INLINE (line 767) | FORCE_INLINE int LZ4_compress_continue_generic (void* LZ4_stream, const ...
function LZ4_compress_continue (line 817) | int LZ4_compress_continue (LZ4_stream_t* LZ4_stream, const char* source,...
function LZ4_compress_limitedOutput_continue (line 822) | int LZ4_compress_limitedOutput_continue (LZ4_stream_t* LZ4_stream, const...
function LZ4_compress_forceExtDict (line 829) | int LZ4_compress_forceExtDict (LZ4_stream_t* LZ4_dict, const char* sourc...
function LZ4_saveDict (line 849) | int LZ4_saveDict (LZ4_stream_t* LZ4_dict, char* safeBuffer, int dictSize)
function FORCE_INLINE (line 876) | FORCE_INLINE int LZ4_decompress_generic(
function LZ4_decompress_safe (line 1045) | int LZ4_decompress_safe(const char* source, char* dest, int compressedSi...
function LZ4_decompress_safe_partial (line 1050) | int LZ4_decompress_safe_partial(const char* source, char* dest, int comp...
function LZ4_decompress_fast (line 1055) | int LZ4_decompress_fast(const char* source, char* dest, int originalSize)
function LZ4_streamDecode_t (line 1073) | LZ4_streamDecode_t* LZ4_createStreamDecode(void)
function LZ4_freeStreamDecode (line 1080) | int LZ4_freeStreamDecode (LZ4_streamDecode_t* LZ4_stream)
function LZ4_setStreamDecode (line 1093) | int LZ4_setStreamDecode (LZ4_streamDecode_t* LZ4_streamDecode, const cha...
function LZ4_decompress_safe_continue (line 1108) | int LZ4_decompress_safe_continue (LZ4_streamDecode_t* LZ4_streamDecode, ...
function LZ4_decompress_fast_continue (line 1128) | int LZ4_decompress_fast_continue (LZ4_streamDecode_t* LZ4_streamDecode, ...
function LZ4_decompress_safe_usingDict (line 1156) | int LZ4_decompress_safe_usingDict(const char* source, char* dest, int co...
function LZ4_decompress_fast_usingDict (line 1161) | int LZ4_decompress_fast_usingDict(const char* source, char* dest, int or...
function LZ4_uncompress (line 1176) | int LZ4_uncompress (const char* source, char* dest, int outputSize) { re...
function LZ4_uncompress_unknownOutputSize (line 1177) | int LZ4_uncompress_unknownOutputSize (const char* source, char* dest, in...
function LZ4_sizeofStreamState (line 1182) | int LZ4_sizeofStreamState() { return LZ4_STREAMSIZE; }
function LZ4_init (line 1184) | static void LZ4_init(LZ4_stream_t_internal* lz4ds, const BYTE* base)
function LZ4_resetStreamState (line 1190) | int LZ4_resetStreamState(void* state, const char* inputBuffer)
function LZ4_sizeofState (line 1215) | int LZ4_sizeofState() { return LZ4_STREAMSIZE; }
function LZ4_compress_withState (line 1217) | int LZ4_compress_withState (void* state, const char* source, char* dest,...
function LZ4_compress_limitedOutput_withState (line 1228) | int LZ4_compress_limitedOutput_withState (void* state, const char* sourc...
function LZ4_decompress_safe_withPrefix64k (line 1241) | int LZ4_decompress_safe_withPrefix64k(const char* source, char* dest, in...
function LZ4_decompress_fast_withPrefix64k (line 1246) | int LZ4_decompress_fast_withPrefix64k(const char* source, char* dest, in...
FILE: algorithm/lz4.h
type LZ4_stream_t (line 183) | typedef struct { unsigned int table[LZ4_STREAMSIZE_U32]; } LZ4_stream_t;
type LZ4_streamDecode_t (line 246) | typedef struct { unsigned int table[LZ4_STREAMDECODESIZE_U32]; } LZ4_str...
FILE: algorithm/murmurhash3.cc
function rotl32 (line 34) | inline uint32_t rotl32 ( uint32_t x, int8_t r )
function rotl64 (line 39) | inline uint64_t rotl64 ( uint64_t x, int8_t r )
function FORCE_INLINE (line 55) | FORCE_INLINE uint32_t getblock32 ( const uint32_t * p, int i )
function FORCE_INLINE (line 60) | FORCE_INLINE uint64_t getblock64 ( const uint64_t * p, int i )
function FORCE_INLINE (line 68) | FORCE_INLINE uint32_t fmix32 ( uint32_t h )
function FORCE_INLINE (line 81) | FORCE_INLINE uint64_t fmix64 ( uint64_t k )
function MurmurHash3_x86_32 (line 94) | void MurmurHash3_x86_32 ( const void * key, int len,
function MurmurHash3_x86_128 (line 150) | void MurmurHash3_x86_128 ( const void * key, const int len,
function MurmurHash3_x64_128 (line 255) | void MurmurHash3_x64_128 ( const void * key, const int len,
FILE: algorithm/xxhash.cc
function FORCE_INLINE (line 86) | FORCE_INLINE void* XXH_malloc(size_t s) { return malloc(s); }
function FORCE_INLINE (line 87) | FORCE_INLINE void XXH_free (void* p) { free(p); }
function FORCE_INLINE (line 90) | FORCE_INLINE void* XXH_memcpy(void* dest, const void* src, size_t size) ...
type _U32_S (line 125) | struct _U32_S { U32 v; }
type _U64_S (line 126) | struct _U64_S { U64 v; }
function U32 (line 157) | static inline U32 XXH_swap32 (U32 x) {
function U64 (line 162) | static inline U64 XXH_swap64 (U64 x) {
function FORCE_INLINE (line 210) | FORCE_INLINE U32 XXH_readLE32_align(const U32* ptr, XXH_endianess endian...
function FORCE_INLINE (line 218) | FORCE_INLINE U32 XXH_readLE32(const U32* ptr, XXH_endianess endian) { re...
function FORCE_INLINE (line 220) | FORCE_INLINE U64 XXH_readLE64_align(const U64* ptr, XXH_endianess endian...
function FORCE_INLINE (line 228) | FORCE_INLINE U64 XXH_readLE64(const U64* ptr, XXH_endianess endian) { re...
function FORCE_INLINE (line 234) | FORCE_INLINE U32 XXH32_endian_align(const void* input, unsigned int len,...
function U32 (line 294) | U32 XXH32(const void* input, unsigned int len, U32 seed)
function FORCE_INLINE (line 321) | FORCE_INLINE U64 XXH64_endian_align(const void* input, unsigned int len,...
function XXH64 (line 427) | unsigned long long XXH64(const void* input, unsigned int len, unsigned l...
type XXH_state32_t (line 451) | struct XXH_state32_t
type XXH_state64_t (line 463) | struct XXH_state64_t
function XXH32_sizeofState (line 476) | int XXH32_sizeofState(void)
function XXH64_sizeofState (line 482) | int XXH64_sizeofState(void)
function XXH_errorcode (line 489) | XXH_errorcode XXH32_resetState(void* state_in, U32 seed)
function XXH_errorcode (line 502) | XXH_errorcode XXH64_resetState(void* state_in, unsigned long long seed)
type XXH_state32_t (line 518) | struct XXH_state32_t
type XXH_state64_t (line 525) | struct XXH_state64_t
function FORCE_INLINE (line 531) | FORCE_INLINE XXH_errorcode XXH32_update_endian (void* state_in, const vo...
function XXH_errorcode (line 595) | XXH_errorcode XXH32_update (void* state_in, const void* input, unsigned ...
function FORCE_INLINE (line 607) | FORCE_INLINE U32 XXH32_intermediateDigest_endian (void* state_in, XXH_en...
function U32 (line 649) | U32 XXH32_intermediateDigest (void* state_in)
function U32 (line 660) | U32 XXH32_digest (void* state_in)
function FORCE_INLINE (line 670) | FORCE_INLINE XXH_errorcode XXH64_update_endian (void* state_in, const vo...
function XXH_errorcode (line 734) | XXH_errorcode XXH64_update (void* state_in, const void* input, unsigned ...
function FORCE_INLINE (line 746) | FORCE_INLINE U64 XXH64_intermediateDigest_endian (void* state_in, XXH_en...
function XXH64_intermediateDigest (line 813) | unsigned long long XXH64_intermediateDigest (void* state_in)
function XXH64_digest (line 824) | unsigned long long XXH64_digest (void* state_in)
FILE: algorithm/xxhash.h
type XXH_errorcode (line 70) | typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode;
type XXH32_stateSpace_t (line 135) | typedef struct { long long ll[(XXH32_SIZEOFSTATE+(sizeof(long long)-1))/...
type XXH64_stateSpace_t (line 141) | typedef struct { long long ll[(XXH64_SIZEOFSTATE+(sizeof(long long)-1))/...
FILE: cache/rate_limiter.h
function namespace (line 11) | namespace kdb {
FILE: cache/write_buffer.cc
type kdb (line 7) | namespace kdb {
function Status (line 23) | Status WriteBuffer::Get(ReadOptions& read_options, ByteArray& key, Byt...
function Status (line 124) | Status WriteBuffer::Put(WriteOptions& write_options, ByteArray& key, B...
function Status (line 130) | Status WriteBuffer::PutPart(WriteOptions& write_options,
function Status (line 149) | Status WriteBuffer::Delete(WriteOptions& write_options, ByteArray& key) {
function Status (line 155) | Status WriteBuffer::WritePart(const WriteOptions& write_options,
FILE: cache/write_buffer.h
function namespace (line 24) | namespace kdb {
FILE: doc/bench/db_bench_kingdb.cc
function DBSynchronize (line 95) | inline
function Compact (line 102) | inline
type leveldb (line 109) | namespace leveldb {
class RandomGenerator (line 113) | class RandomGenerator {
method RandomGenerator (line 119) | RandomGenerator() {
method Generate (line 134) | kdb::ByteArray Generate(int len) {
function Slice (line 144) | static Slice TrimSpace(Slice s) {
class Benchmark (line 158) | class Benchmark {
method PrintHeader (line 176) | void PrintHeader() {
method PrintWarnings (line 194) | void PrintWarnings() {
method PrintEnvironment (line 206) | void PrintEnvironment() {
method Start (line 240) | void Start() {
method FinishedSingleOp (line 250) | void FinishedSingleOp() {
method Stop (line 276) | void Stop(const Slice& name) {
type Order (line 306) | enum Order {
type DBState (line 310) | enum DBState {
method Benchmark (line 315) | Benchmark()
method Run (line 341) | void Run() {
method EraseDB (line 409) | void EraseDB(char *dbname) {
method Open (line 426) | void Open(bool sync) {
method Write (line 457) | void Write(bool sync, Order order, DBState state,
method ReadSequential (line 491) | void ReadSequential() {
method ReadRandom (line 505) | void ReadRandom() {
function main (line 525) | int main(int argc, char** argv) {
FILE: doc/bench/generate_benchmarks_table.py
function fold_change (line 16) | def fold_change(ldb, kdb):
function perc_change (line 26) | def perc_change(ldb, kdb):
FILE: interface/database.cc
type kdb (line 7) | namespace kdb {
function Status (line 9) | Status Database::GetRaw(ReadOptions& read_options,
function Status (line 77) | Status Database::Get(ReadOptions& read_options, ByteArray& key, ByteAr...
function Status (line 82) | Status Database::Put(WriteOptions& write_options, ByteArray& key, Byte...
function Status (line 87) | Status Database::PutPart(WriteOptions& write_options,
function Status (line 128) | Status Database::PutPartValidSize(WriteOptions& write_options,
function Status (line 279) | Status Database::Delete(WriteOptions& write_options,
function Snapshot (line 301) | Snapshot Database::NewSnapshot() {
function KingDB (line 330) | KingDB* Database::NewSnapshotPointer() {
function Iterator (line 361) | Iterator Database::NewIterator(ReadOptions& read_options) {
FILE: interface/database.h
function namespace (line 37) | namespace kdb {
FILE: interface/iterator.h
function namespace (line 19) | namespace kdb {
function ByteArray (line 216) | ByteArray GetKey() {
function ByteArray (line 221) | ByteArray GetValue() {
function MultipartReader (line 246) | MultipartReader GetMultipartValue() {
function Status (line 250) | Status GetStatus() {
function class (line 277) | class SequentialIterator: public IteratorResource {
function SetParentSnapshot (line 325) | void SetParentSnapshot(KingDB *snapshot) {
function Begin (line 329) | void Begin() {
function IsValid (line 345) | bool IsValid() {
function Next (line 352) | bool Next() {
function ByteArray (line 441) | ByteArray GetKey() {
function ByteArray (line 446) | ByteArray GetValue() {
function MultipartReader (line 471) | MultipartReader GetMultipartValue() {
function Status (line 475) | Status GetStatus() {
function class (line 508) | class Iterator {
FILE: interface/kingdb.h
function namespace (line 13) | namespace kdb {
FILE: interface/multipart.h
function namespace (line 27) | namespace kdb {
FILE: interface/snapshot.h
function namespace (line 18) | namespace kdb {
function virtual (line 134) | virtual void Flush() {}
function virtual (line 135) | virtual void Compact() {}
FILE: network/client.h
function namespace (line 35) | namespace kdb {
function class (line 145) | class ClientTask: public Task {
FILE: network/client_main.cc
function show_usage (line 5) | void show_usage(char *program_name) {
function main (line 9) | int main(int argc, char **argv) {
FILE: network/server.cc
type kdb (line 13) | namespace kdb {
type sockaddr (line 290) | struct sockaddr
type sockaddr_in (line 293) | struct sockaddr_in
type sockaddr_in6 (line 295) | struct sockaddr_in6
function Status (line 299) | Status Server::Start(ServerOptions& server_options,
type addrinfo (line 327) | struct addrinfo
type sockaddr_storage (line 393) | struct sockaddr_storage
type sockaddr (line 415) | struct sockaddr
type sockaddr (line 419) | struct sockaddr
FILE: network/server.h
function namespace (line 39) | namespace kdb {
function class (line 62) | class Server {
FILE: network/server_main.cc
function termination_signal_handler (line 22) | void termination_signal_handler(int signal) {
function crash_signal_handler (line 27) | void crash_signal_handler(int sig) {
function daemonize (line 39) | int daemonize() {
function main (line 71) | int main(int argc, char** argv) {
FILE: storage/format.h
function namespace (line 26) | namespace kdb {
type EntryFooter (line 259) | struct EntryFooter {
type DatabaseOptionEncoder (line 268) | struct DatabaseOptionEncoder {
function EncodeTo (line 317) | static uint32_t EncodeTo(const struct DatabaseOptions *input, char* buff...
function GetFixedSize (line 333) | static uint32_t GetFixedSize() {
type FileType (line 340) | enum FileType {
type HSTableHeader (line 347) | struct HSTableHeader {
function IsTypeLarge (line 373) | bool IsTypeLarge() {
function IsTypeCompacted (line 377) | bool IsTypeCompacted() {
function IsFileVersionSupported (line 382) | bool IsFileVersionSupported() {
function IsFileVersionNewer (line 387) | bool IsFileVersionNewer() {
type HSTableHeader (line 399) | struct HSTableHeader
type DatabaseOptions (line 400) | struct DatabaseOptions
function EncodeTo (line 416) | static uint32_t EncodeTo(const struct HSTableHeader *input, const struct...
function GetFixedSize (line 427) | static uint32_t GetFixedSize() {
type HSTableFooterFlags (line 432) | enum HSTableFooterFlags {
function IsTypeCompacted (line 437) | struct HSTableFooter {
function SetFlagHasPaddingInValues (line 459) | void SetFlagHasPaddingInValues() {
function SetFlagHasInvalidEntries (line 463) | void SetFlagHasInvalidEntries() {
function Status (line 467) | static Status DecodeFrom(const char* buffer_in,
function EncodeTo (line 480) | static uint32_t EncodeTo(const struct HSTableFooter *input,
function GetFixedSize (line 491) | static uint32_t GetFixedSize() {
function EncodeTo (line 497) | struct OffsetArrayRow {
FILE: storage/hstable_manager.h
function namespace (line 37) | namespace kdb {
function Status (line 362) | Status FlushOffsetArray() {
function Status (line 381) | Status WriteOffsetArray(int fd,
function Status (line 1188) | Status DeleteAllLockedFiles(std::string& dbname) {
FILE: storage/resource_manager.h
function namespace (line 35) | namespace kdb {
FILE: storage/storage_engine.h
function namespace (line 42) | namespace kdb {
function Status (line 459) | Status GetEntry(ReadOptions& read_options,
function IsLocationLastInIndex (line 523) | bool IsLocationLastInIndex(uint64_t location, ByteArray& key) {
function IsFileLarge (line 535) | bool IsFileLarge(uint32_t fileid) {
function Status (line 539) | Status Compaction(std::string dbname,
type stat (line 781) | struct stat
type HSTableHeader (line 807) | struct HSTableHeader
type HSTableFooter (line 813) | struct HSTableFooter
function Status (line 1158) | Status ReleaseAllSnapshots() {
FILE: thread/event_manager.h
function namespace (line 14) | namespace kdb {
FILE: thread/threadpool.h
function namespace (line 17) | namespace kdb {
function AddTask (line 93) | void AddTask(Task* task) {
function Start (line 99) | void Start() {
function Stop (line 108) | void Stop() {
FILE: thread/threadstorage.h
function namespace (line 13) | namespace kdb {
FILE: unit-tests/client_embedded.cc
function handler (line 34) | void handler(int sig) {
function main (line 45) | int main() {
FILE: unit-tests/dummy_interface.h
function namespace (line 15) | namespace kdb {
FILE: unit-tests/kingdb_user.cc
function main (line 9) | int main() {
FILE: unit-tests/test_compression.cc
function VerifyValue (line 19) | int VerifyValue(const std::string& key, int size_value, const char* valu...
function main (line 43) | int main() {
FILE: unit-tests/test_db.cc
type kdb (line 35) | namespace kdb {
class KeyGenerator (line 37) | class KeyGenerator {
class SequentialKeyGenerator (line 44) | class SequentialKeyGenerator: public KeyGenerator {
method GetKey (line 48) | virtual std::string GetKey(uint64_t thread_id, uint64_t index, int s...
class RandomKeyGenerator (line 57) | class RandomKeyGenerator: public KeyGenerator {
method RandomKeyGenerator (line 59) | RandomKeyGenerator() {
method GetKey (line 65) | virtual std::string GetKey(uint64_t thread_id, uint64_t index, int s...
class DataGenerator (line 81) | class DataGenerator {
class IncompressibleDataGenerator (line 87) | class IncompressibleDataGenerator: public DataGenerator {
method IncompressibleDataGenerator (line 89) | IncompressibleDataGenerator() {
method GenerateData (line 95) | virtual void GenerateData(char *data, int size) {
class CompressibleDataGenerator (line 108) | class CompressibleDataGenerator: public DataGenerator {
method CompressibleDataGenerator (line 110) | CompressibleDataGenerator() {
method GenerateData (line 116) | virtual void GenerateData(char *data, int size) {
class DBTest (line 137) | class DBTest {
method DBTest (line 139) | DBTest() {
method Open (line 147) | void Open(bool erase_db=true) {
method Close (line 157) | void Close(bool erase_db=true) {
method OpenWithoutErasingDB (line 164) | void OpenWithoutErasingDB() {
method CloseWithoutErasingDB (line 168) | void CloseWithoutErasingDB() {
method DeleteDatabaseOptionsFile (line 172) | void DeleteDatabaseOptionsFile() {
method ResetAllOptions (line 179) | void ResetAllOptions() {
method IterateOverOptions (line 185) | bool IterateOverOptions() {
method EraseDB (line 268) | void EraseDB() {
method Get (line 284) | kdb::Status Get(ReadOptions& read_options_, const std::string& key, ...
method Put (line 288) | kdb::Status Put(const std::string& key, const std::string& value) {
function TEST (line 303) | TEST(DBTest, CloseAndReopen) {
function TEST (line 329) | TEST(DBTest, RepairInvalidDatabaseOptionFile) {
function TEST (line 356) | TEST(DBTest, KeysWithNullBytes) {
function TEST (line 401) | TEST(DBTest, TestStringInterface){
function TEST (line 436) | TEST(DBTest, MultipartReader) {
function TEST (line 467) | TEST(DBTest, SingleThreadSmallEntries) {
function TEST (line 530) | TEST(DBTest, SingleThreadSnapshot) {
function TEST (line 594) | TEST(DBTest, SingleThreadSmallEntriesCompaction) {
function TEST (line 659) | TEST(DBTest, SequentialIterator) {
function run_command (line 733) | std::string run_command(const char* command) {
function exists_program (line 756) | bool exists_program(const char* program_name) {
function compute_external_md5 (line 767) | std::string compute_external_md5(const char* filepath) {
function TEST (line 793) | TEST(DBTest, SingleThreadSingleLargeEntry) {
function TEST (line 882) | TEST(DBTest, FileUtil) {
function handler (line 901) | void handler(int sig) {
function main (line 913) | int main() {
FILE: unit-tests/testharness.cc
type kdb (line 14) | namespace kdb {
type test (line 15) | namespace test {
type Test (line 18) | struct Test {
function RegisterTest (line 26) | bool RegisterTest(const char* base, const char* name, void (*func)()) {
function RunAllTests (line 38) | int RunAllTests() {
function TmpDir (line 62) | std::string TmpDir() {
function RandomSeed (line 70) | int RandomSeed() {
FILE: unit-tests/testharness.h
function namespace (line 29) | namespace kdb {
FILE: util/byte_array.h
function namespace (line 26) | namespace kdb {
function class (line 158) | class PointerByteArrayResource: public ByteArrayResource {
function virtual (line 169) | virtual const char* data_const() { return data_; }
function virtual (line 170) | virtual uint64_t size() { return size_; }
function virtual (line 171) | virtual const uint64_t size_const() { return size_; }
function virtual (line 172) | virtual uint64_t size_compressed() { return size_compressed_; }
function virtual (line 173) | virtual const uint64_t size_compressed_const() { return size_compressed_; }
function class (line 182) | class ByteArray {
function ByteArray (line 302) | inline ByteArray NewShallowCopyByteArray(char* data, uint64_t size) {
function ByteArray (line 306) | inline ByteArray NewDeepCopyByteArray(const char* data, uint64_t size) {
function ByteArray (line 310) | inline ByteArray NewDeepCopyByteArray(const std::string& str) {
function ByteArray (line 314) | inline ByteArray NewMmappedByteArray(const std::string& filepath, uint64...
function ByteArray (line 318) | inline ByteArray NewPointerByteArray(const char* data, uint64_t size) {
FILE: util/config_parser.h
function class (line 27) | class Parameter {
function class (line 163) | class UnsignedInt64Parameter: public Parameter {
function class (line 198) | class DoubleParameter: public Parameter {
function class (line 227) | class StringParameter: public Parameter {
function AddParameter (line 260) | void AddParameter(Parameter* parameter) {
function SetDefaultValue (line 267) | void SetDefaultValue(const std::string name, const std::string default_v...
function FoundAllMandatoryParameters (line 279) | bool FoundAllMandatoryParameters() {
function PrintUsage (line 317) | void PrintUsage() {
type stat (line 431) | struct stat
FILE: util/debug.cc
type kdb (line 7) | namespace kdb {
function PrintHex (line 9) | void PrintHex(const char* buffer, int size) {
FILE: util/debug.h
function namespace (line 20) | namespace kdb {
FILE: util/file.h
function namespace (line 23) | namespace kdb {
FILE: util/filepool.h
function namespace (line 33) | namespace kdb {
FILE: util/logger.cc
type kdb (line 7) | namespace kdb {
FILE: util/logger.h
function namespace (line 28) | namespace kdb {
FILE: util/options.h
function namespace (line 11) | namespace kdb {
type ServerOptions (line 217) | struct ServerOptions {
FILE: util/order.h
function namespace (line 28) | namespace kdb {
FILE: util/status.cc
type kdb (line 7) | namespace kdb {
FILE: util/status.h
function namespace (line 18) | namespace kdb {
function Status (line 34) | static Status OK() { return Status(); }
function Status (line 36) | static Status Done() { return Status(kDone); }
function Status (line 38) | static Status MultipartRequired() { return Status(kMultipartRequired); }
function Status (line 40) | static Status DeleteOrder() { return Status(kDeleteOrder); }
type Code (line 72) | enum Code {
FILE: util/version.h
function namespace (line 8) | namespace kdb {
Condensed preview — 75 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (646K chars).
[
{
"path": ".gitignore",
"chars": 211,
"preview": "# Compiled Object files\n*.slo\n*.lo\n*.o\n*.obj\n\n# Compiled Dynamic libraries\n*.so\n*.dylib\n*.dll\n\n# Compiled Static librari"
},
{
"path": "3rdparty/leveldb/AUTHORS",
"chars": 293,
"preview": "# Names should be added to this file like so:\n# Name or Organization <email address>\n\nGoogle Inc.\n\n# Initial version aut"
},
{
"path": "3rdparty/leveldb/LICENSE",
"chars": 1484,
"preview": "Copyright (c) 2011 The LevelDB Authors. All rights reserved.\n\nRedistribution and use in source and binary forms, with or"
},
{
"path": "3rdparty/lz4/LICENSE",
"chars": 1311,
"preview": "LZ4 Library\nCopyright (c) 2011-2014, Yann Collet\nAll rights reserved.\n\nRedistribution and use in source and binary forms"
},
{
"path": "3rdparty/murmurhash3/LICENSE",
"chars": 138,
"preview": "MurmurHash3 was written by Austin Appleby, and is placed in the public\ndomain. The author hereby disclaims copyright to "
},
{
"path": "3rdparty/xxhash/LICENSE",
"chars": 1314,
"preview": "xxHash Library\nCopyright (c) 2012-2014, Yann Collet\nAll rights reserved.\n\nRedistribution and use in source and binary fo"
},
{
"path": "3rdparty/zlib/LICENSE",
"chars": 1106,
"preview": "/* zlib.h -- interface of the 'zlib' general purpose compression library\n version 1.2.8, April 28th, 2013\n\n Copyright "
},
{
"path": "LICENSE",
"chars": 2190,
"preview": "Copyright (c) 2014, Emmanuel Goossaert\nAll rights reserved.\n\nUse of KingDB is governed by the BSD 3-Clause License.\nKing"
},
{
"path": "Makefile",
"chars": 4054,
"preview": "CC=g++\nINCLUDES=-I/usr/local/include/ -I/opt/local/include/ -I. -I./include/\nLDFLAGS=-g -L/usr/local/lib/ -L/opt/local/l"
},
{
"path": "README.md",
"chars": 2764,
"preview": "KingDB\n======\n\n###What is KingDB?\n\n**KingDB** is a fast on-disk persistent key-value store. You can embed it or use it a"
},
{
"path": "algorithm/coding.cc",
"chars": 5249,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "algorithm/coding.h",
"chars": 4206,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "algorithm/compressor.cc",
"chars": 8904,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "algorithm/compressor.h",
"chars": 7119,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "algorithm/crc32c.cc",
"chars": 20451,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "algorithm/crc32c.h",
"chars": 3121,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "algorithm/endian.cc",
"chars": 603,
"preview": "#include \"algorithm/endian.h\"\n\nnamespace kdb {\n\nendian_t getEndianness() {\n if ((0xffffffff & 1) == kBytesLittleEndian)"
},
{
"path": "algorithm/endian.h",
"chars": 1209,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "algorithm/hash.cc",
"chars": 1003,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "algorithm/hash.h",
"chars": 1163,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "algorithm/lz4.cc",
"chars": 45208,
"preview": "/*\n LZ4 - Fast LZ compression algorithm\n Copyright (C) 2011-2014, Yann Collet.\n BSD 2-Clause License (http://www.o"
},
{
"path": "algorithm/lz4.h",
"chars": 14814,
"preview": "/*\n LZ4 - Fast LZ compression algorithm\n Header File\n Copyright (C) 2011-2014, Yann Collet.\n BSD 2-Clause Licens"
},
{
"path": "algorithm/murmurhash3.cc",
"chars": 7911,
"preview": "//-----------------------------------------------------------------------------\n// MurmurHash3 was written by Austin App"
},
{
"path": "algorithm/murmurhash3.h",
"chars": 1086,
"preview": "//-----------------------------------------------------------------------------\n// MurmurHash3 was written by Austin App"
},
{
"path": "algorithm/xxhash.cc",
"chars": 27033,
"preview": "/*\r\nxxHash - Fast Hash algorithm\r\nCopyright (C) 2012-2014, Yann Collet.\r\nBSD 2-Clause License (http://www.opensource.org"
},
{
"path": "algorithm/xxhash.h",
"chars": 6868,
"preview": "/*\r\n xxHash - Extremely Fast Hash algorithm\r\n Header File\r\n Copyright (C) 2012-2014, Yann Collet.\r\n BSD 2-Clause"
},
{
"path": "cache/rate_limiter.h",
"chars": 5903,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "cache/write_buffer.cc",
"chars": 12399,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "cache/write_buffer.h",
"chars": 3902,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "doc/bench/benchmarks.md",
"chars": 8149,
"preview": "Benchmarks of KingDB v0.9.0 and LevelDB v1.18 \n=============================================\n\nDate: April 9, 2015\n\n##TL;"
},
{
"path": "doc/bench/db_bench_kingdb.cc",
"chars": 17254,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "doc/bench/generate_benchmarks_table.py",
"chars": 3888,
"preview": "#!/usr/bin/python\n\n# This program reads outputs from the LevelDB benchmarks and generates\n# a table in Markdown format t"
},
{
"path": "doc/kingdb.md",
"chars": 23820,
"preview": "Documentation of KingDB v0.9.0\n==============================\n\n##Table of Contents\n\n**[1. Why use KingDB?](#1-why-use-ki"
},
{
"path": "doc/kingserver.md",
"chars": 14285,
"preview": "Documentation of KingServer v0.9.0\n==================================\n\n##Table of Contents\n\n**[1. What is KingServer?](#"
},
{
"path": "include/kingdb/kdb.h",
"chars": 496,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "interface/database.cc",
"chars": 14375,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "interface/database.h",
"chars": 10742,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "interface/iterator.h",
"chars": 16838,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "interface/kingdb.h",
"chars": 3077,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "interface/multipart.h",
"chars": 6673,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "interface/snapshot.h",
"chars": 6288,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "network/client.h",
"chars": 12010,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "network/client_main.cc",
"chars": 1820,
"preview": "#include \"network/client.h\"\n\n#include <util/config_parser.h>\n\nvoid show_usage(char *program_name) {\n printf(\"Example: %"
},
{
"path": "network/server.cc",
"chars": 15752,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "network/server.h",
"chars": 2808,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "network/server_main.cc",
"chars": 7378,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "storage/format.h",
"chars": 16714,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "storage/hstable_manager.h",
"chars": 56876,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "storage/resource_manager.h",
"chars": 6809,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "storage/storage_engine.h",
"chars": 55629,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "thread/event_manager.h",
"chars": 1511,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "thread/threadpool.h",
"chars": 3763,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "thread/threadstorage.h",
"chars": 1534,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "unit-tests/client_embedded.cc",
"chars": 3370,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "unit-tests/dummy_interface.h",
"chars": 1484,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "unit-tests/dummy_storage_engine.h",
"chars": 1922,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "unit-tests/dummy_storage_engine_index.h",
"chars": 5354,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "unit-tests/kingdb_user.cc",
"chars": 1217,
"preview": "#include <kingdb/kdb.h>\n\n// Compiling:\n// - LLVM:\n// $ g++ -std=c++11 -I/usr/local/include/kingdb -lkingdb kingdb_user.c"
},
{
"path": "unit-tests/test_compression.cc",
"chars": 4031,
"preview": "#include \"algorithm/compressor.h\"\n#include \"algorithm/lz4.h\"\n\n\nchar* MakeValue(const std::string& key, int size_value) {"
},
{
"path": "unit-tests/test_db.cc",
"chars": 26702,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "unit-tests/testharness.cc",
"chars": 1899,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "unit-tests/testharness.h",
"chars": 5006,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/byte_array.h",
"chars": 9526,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/config_parser.h",
"chars": 16695,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/debug.cc",
"chars": 679,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/debug.h",
"chars": 606,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/file.h",
"chars": 7414,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/filepool.h",
"chars": 5235,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/logger.cc",
"chars": 430,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/logger.h",
"chars": 8421,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/options.h",
"chars": 12888,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/order.h",
"chars": 1400,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/status.cc",
"chars": 1268,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/status.h",
"chars": 2364,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
},
{
"path": "util/version.h",
"chars": 470,
"preview": "// Copyright (c) 2014, Emmanuel Goossaert. All rights reserved.\n// Use of this source code is governed by the BSD 3-Clau"
}
]
About this extraction
This page contains the full source code of the goossaert/kingdb GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 75 files (605.4 KB), approximately 170.7k tokens, and a symbol index with 349 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.