# sled 1.0 architecture
## in-memory
* Lock-free B+ tree index, extracted into the [`concurrent-map`](https://github.com/komora-io/concurrent-map) crate.
* The lowest key from each leaf is stored in this in-memory index.
* To read any leaf that is not already cached in memory, at most one disk read will be required.
* RwLock-backed leaves, using the ArcRwLock from the [`parking_lot`](https://github.com/Amanieu/parking_lot) crate. As a `Db` grows, leaf contention tends to go down in most use cases. But this may be revisited over time if many users have issues with RwLock-related contention. Avoiding full RCU for updates on the leaves results in many of the performance benefits over sled 0.34, with significantly lower memory pressure.
* A simple but very high performance epoch-based reclamation technique is used for safely deferring frees of in-memory index data and reuse of on-disk heap slots, extracted into the [`ebr`](https://github.com/komora-io/ebr) crate.
* A scan-resistant LRU is used for handling eviction. By default, 20% of the cache is reserved for leaves that are accessed at most once. This is configurable via `Config.entry_cache_percent`. This is handled by the extracted [`cache-advisor`](https://github.com/komora-io/cache-advisor) crate. The overall cache size is set by the `Config.cache_size` configurable.
## write path
* This is where things get interesting. There is no traditional WAL. There is no LSM. Only metadata is logged atomically after objects are written in parallel.
* The important guarantees are:
* all previous writes are durable after a call to `Db::flush` (This is also called periodically in the background by a flusher thread)
* all write batches written using `Db::apply_batch` are either 100% visible or 0% visible after crash recovery. If it was followed by a flush that returned `Ok(())` it is guaranteed to be present.
* Atomic ([linearizable](https://jepsen.io/consistency/models/linearizable)) durability is provided by marking dirty leaves as participants in "flush epochs" and performing atomic batch writes of the full epoch at a time, in order. Each call to `Db::flush` advances the current flush epoch by 1.
* The atomic write consists in the following steps:
1. User code or the background flusher thread calls `Db::flush`.
1. In parallel (via [rayon](https://docs.rs/rayon)) serialize and compress each dirty leaf with zstd (configurable via `Config.zstd_compression_level`).
1. Based on the size of the bytes for each object, choose the smallest heap file slot that can hold the full set of bytes. This is an on-disk slab allocator.
1. Slab slots are not power-of-two sized, but tend to increase in size by around 20% from one to the next, resulting in far lower fragmentation than typical page-oriented heaps with either constant-size or power-of-two sized leaves.
1. Write the object to the allocated slot from the rayon threadpool.
1. After all writes, fsync the heap files that were written to.
1. If any writes were written to the end of the heap file, causing it to grow, fsync the directory that stores all heap files.
1. After the writes are stable, it is now safe to write an atomic metadata batch that records the location of each written leaf in the heap. This is a simple framed batch of `(low_key, slab_slot)` tuples that are initially written to a log, but eventually merged into a simple snapshot file for the metadata store once the log becomes larger than the snapshot file.
1. Fsync of the metadata log file.
1. Fsync of the metadata log directory.
1. After the atomic metadata batch write, the previously occupied slab slots are marked for future reuse with the epoch-based reclamation system. After all threads that may have witnessed the previous location have finished their work, the slab slot is added to the free `BinaryHeap` of the slot that it belongs to so that it may be reused in future atomic write batches.
1. Return `Ok(())` to the caller of `Db::flush`.
* Writing objects before the metadata write is random, but modern SSDs handle this well. Even though the SSD's FTL will be working harder to defragment things periodically than if we wrote a few megabytes sequentially with each write, the data that the FTL will be copying will be mostly live due to the eager leaf write-backs.
## recovery
* Recovery involves simply reading the atomic metadata store that records the low key for each written leaf as well as its location and mapping it into the in-memory index. Any gaps in the slabs are then used as free slots.
* Any write that failed to complete its entire atomic writebatch is treated as if it never happened, because no user-visible flush ever returned successfully.
* Rayon is also used here for parallelizing reads of this metadata. In general, this is extremely fast compared to the previous sled recovery process.
## tuning
* The larger the `LEAF_FANOUT` const generic on the high-level `Db` struct (default `1024`), the smaller the in-memory leaf index and the better the compression ratio of the on-disk file, but the more expensive it will be to read the entire leaf off of disk and decompress it.
* You can choose to turn the `LEAF_FANOUT` relatively low to make the system behave more like an Index+Log architecture, but overall disk size will grow and write performance will decrease.
* NB: changing `LEAF_FANOUT` after writing data is not supported.
================================================
FILE: CHANGELOG.md
================================================
# Unreleased
## New Features
* #1178 batches and transactions are now unified for subscribers.
* #1231 `Tree::get_zero_copy` allows for reading a value directly
in-place without making an `IVec` first.
* #1250 the global `print_profile` function has been added
which is enabled when compiling with the `metrics` feature.
* #1254 `IVec` data will now always have an alignment of 8,
which may enable interesting architecture-specific use cases.
* #1307 & #1315 `Db::contains_tree` can be used to see if a
`Tree` with a given name already exists.
## Improvements
* #1214 a new slab-style storage engine has been added which
replaces the previous file-per-blob technique for storing
large pages.
* #1231 tree nodes now get merged into a single-allocation
representation that is able to dynamically avoid various
overheads, resulting in significant efficiency improvements.
## Breaking Changes
* #1400 Bump MSRV to 1.57.
* #1399 Thread support is now required on all platforms.
* #1135 The "no_metrics" anti-feature has been replaced with
the "metrics" positive feature.
* #1178 the `Event` enum has become a unified struct that allows
subscribers to iterate over each (Tree, key, optional value)
involved in single key operations, batches, or transactions in
a unified way.
* #1178 the `Event::key` method has been removed in favor of the
new more comprehensive `iter` method.
* #1214 The deprecated `Config::build` method has been removed.
* #1248 The deprecated `Tree::set` method has been removed.
* #1248 The deprecated `Tree::del` method has been removed.
* #1250 The `Config::print_profile_on_drop` method has been
removed in favor of the global `print_profile` function.
* #1252 The deprecated `Db::open` method has been removed.
* #1252 The deprecated `Config::segment_cleanup_skew` method
has been removed.
* #1252 The deprecated `Config::segment_cleanup_threshold`
method has been removed.
* #1252 The deprecated `Config::snapshot_path` method has
been removed.
* #1253 The `IVec::subslice` method has been removed.
* #1275 Keys and values are now limited to 128gb on 64-bit
platforms and 512mb on 32-bit platforms.
* #1281 `Config`'s `cache_capacity` is now a usize, as u64
doesn't make sense for things that must fit in memory anyway.
* #1314 `Subscriber::next_timeout` now requires a mutable self
reference.
* #1349 The "measure_allocs" feature has been removed.
* #1354 `Error` has been modified to be Copy, removing all
heap-allocated variants.
## Bug Fixes
* #1202 Fix a space leak where blobs were not
removed when replaced by another blob.
* #1229 the powerful ALICE crash consistency tool has been
used to discover several crash vulnerabilities, now fixed.
# 0.34.7
## Bug Fixes
* #1314 Fix a bug in Subscriber's Future impl.
# 0.34.6
## Improvements
* documentation improved
# 0.34.5
## Improvements
* #1164 widen some trait bounds on trees and batches
# 0.34.4
## New Features
* #1151 `Send` is implemented for `Iter`
* #1167 added `Tree::first` and `Tree::last` functions
to retrieve the first or last items in a `Tree`, unless
the `Tree` is empty.
## Bug Fixes
* #1159 dropping a `Db` instance will no-longer
prematurely shut-down the background flusher
thread.
* #1168 fixed an issue that was causing panics during
recovery in 32-bit code.
* #1170 when encountering corrupted storage data,
the recovery process will panic less often.
# 0.34.3
## New Features
* #1146 added `TransactionalTree::generate_id`
# 0.34.2
## Improvements
* #1133 transactions and writebatch performance has been
significantly improved by removing a bottleneck in
the atomic batch stability tracking code.
# 0.34.1
## New Features
* #1136 Added the `TransactionalTree::flush` method to
flush the underlying database after the transaction
commits and before the transaction returns.
# 0.34
## Improvements
* #1132 implemented From for io::Error to
reduce friction in some situations.
## Breaking Changes
* #1131 transactions performed on `Tree`s from different
`Db`s will now safely fail.
* #1131 transactions may now only be performed on tuples
of up to 14 elements. For higher numbers, please use
slices.
# 0.33
## Breaking Changes
* #1125 the backtrace crate has been made optional, which
cuts several seconds off compilation time, but may cause
breakage if you interacted with the backtrace field
of corruption-related errors.
## Bug Fixes
* #1128 `Tree::pop_min` and `Tree::pop_max` had a bug where
they were not atomic.
# 0.32.1
## New Features
* #1116 `IVec::subslice` has been added to facilitate
creating zero-copy subsliced `IVec`s that are backed
by the same data.
## Bug Fixes
* #1120 Fixed a use-after-free caused by missing `ref` keyword
on a `Copy` type in a pattern match in `IVec::as_mut`.
* #1108 conversions from `Box<[u8]>` to `IVec` are fixed.
# 0.32
## New Features
* #1079 `Transactional` is now implemented for
`[&Tree]` and `[Tree]` so you can avoid the
previous friction of using tuples, as was
necessary previously.
* #1058 The minimum supported Rust version (MSRV)
is now 1.39.0.
* #1037 `Subscriber` now implements `Future` (non-fused)
so prefix watching may now be iterated over via
`while let Some(event) = (&mut subscriber).await {}`
## Improvements
* #965 concurrency control is now dynamically enabled
for atomic point operations, so that it may be
avoided unless transactional functionality is
being used in the system. This significantly
increases performance for workloads that do not
use transactions.
* A number of memory optimizations have been implemented.
* Disk usage has been significantly reduced for many
workloads.
* #1016 On 64-bit systems, we can now store 1-2 trillion items.
* #993 Added DerefMut and AsMut<[u8]> for `IVec` where it
works similarly to a `Cow`, making a private copy
if the backing `Arc`'s strong count is not 1.
* #1020 The sled wiki has been moved into the documentation
itself, and is accessible through the `doc` module
exported in lib.
## Breaking Changes
* #975 Changed the default `segment_size` from 8m to 512k.
This will result in far smaller database files due
to better file garbage collection granularity.
* #975 deprecated several `Config` options that will be
removed over time.
* #1000 rearranged some transaction-related imports, and
moved them to the `transaction` module away from
the library root to keep the top level docs clean.
* #1015 `TransactionalTree::apply_batch` now accepts
its argument by reference instead of by value.
* `Event` has been changed to make the inner fields
named instead of anonymous.
* #1057 read-only mode has been removed due to not having
the resources to properly keep it tested while
making progress on high priority issues. This may
be correctly implemented in the future if resources
permit.
* The conversion between `Box<[u8]>` and `IVec` has
been temporarily removed. This is re-added in 0.32.1.
# 0.31
## Improvements
* #947 dramatic read and recovery optimizations
* #921 reduced the reliance on locks while
performing multithreaded IO on windows.
* #928 use `sync_file_range` on linux instead
of a full fsync for most writes.
* #946 io_uring support changed to the `rio` crate
* #939 reduced memory consumption during
zstd decompression
## Breaking Changes
* #927 use SQLite-style varints for serializing
`u64`. This dramatically reduces the written
bytes for databases that store small keys and
values.
* #943 use varints for most of the fields in
message headers, causing an additional large
space reduction. combined with #927, these
changes reduce bytes written by 68% for workloads
writing small items.
# 0.30.3
* Documentation-only release
# 0.30.2
## New Features
* Added the `open` function for quickly
opening a database at a path with default
configuration.
# 0.30.1
## Bugfixes
* Fixed an issue where an idle threadpool worker
would spin in a hot loop until work arrived
# 0.30
## Breaking Changes
* Migrated to a new storage format
## Bugfixes
* Fixed a bug where cache was not being evicted.
* Fixed a bug with using transactions with
compression.
# 0.29.2
## New Features
* The `create_new` option has been added
to `Config`, allowing the user to specify
that a database should only be freshly
created, rather than re-opened.
# 0.29.1
## Bugfixes
* Fixed a bug where prefix encoding could be
incorrectly handled when merging nodes together.
# 0.29
## New Features
* The `Config::open` method has been added to give
`Config` a similar feel to std's `fs::OpenOptions`.
The `Config::build` and `Db::start` methods are
now deprecated in favor of calling `Config::open`
directly.
* A `checksum` method has been added to Tree and Db
for use in verifying backups and migrations.
* Transactions may now involve up to 69 different
tables. Nice.
* The `TransactionError::Abort` variant has had
a generic member added that can be returned
as a way to return information from a
manually-aborted transaction. An `abort` helper
function has been added to reduce the boiler-
plate required to return aborted results.
## Breaking Changes
* The `ConfigBuilder` structure has been removed
in favor of a simplified `Config` structure
with the same functionality.
* The way that sled versions are detected at
initialization time is now independent of serde.
* The `cas` method is deprecated in favor of the new
`compare_and_swap` method which now returns the
proposed value that failed to be applied.
* Tree nodes now have constant prefix encoding
lengths.
* The `io_buf_size` configurable renamed to
`segment_size`.
* The `io_buf_size` configurable method has been
removed from ConfigBuilder. This can be manually
set by setting the attribute directly on the
ConfigBuilder, but this is discouraged.
Additionally, this must now be a power of 2.
* The `page_consolidation_threshold` method has been
removed from ConfigBuilder, and this is now
a constant of 10.
# 0.28
## Breaking Changes
* `Iter` no longer has a lifetime parameter.
* `Db::open_tree` now returns a `Tree` instead of
an `Arc`. `Tree` now has an inner type that
uses an `Arc`, so you don't need to think about it.
## Bug Fixes
* A bug with prefix encoding has been fixed that
led to nodes with keys longer than 256 bytes
being stored incorrectly, which led to them
being inaccessible and also leading to infinite
loops during iteration.
* Several cases of incorrect unsafe code were removed
from the sled crate. No bugs are known to have been
encountered, but they may have resulted in
incorrect optimizations in future refactors.
# 0.27
## Breaking Changes
* `Event::Set` has been renamed to `Event::Insert` and
`Event::Del` has been renamed to `Event::Remove`. These
names better align with the methods of BTreeMap from
the standard library.
## Bug Fixes
* A deadlock was possible in very high write volume
situations when the segment accountant lock was
taken by all IO threads while a task was blocked
trying to submit a file truncation request to the
threadpool while holding the segment accountant lock.
## New Features
* `flush_async` has been added to perform time-intensive
flushing in an asynchronous manner, returning a Future.
# 0.26.1
## Improvements
* std::thread is no longer used on platforms other than
linux, macos, and windows, which increases portability.
# 0.26
## New Features
* Transactions! You may now call `Tree::transaction` and
perform reads, writes, and deletes within a provided
closure with a `TransactionalTree` argument. This
closure may be called multiple times if the transaction
encounters a concurrent update in the process of its
execution. Transactions may also be used on tuples of
`Tree` objects, where the closure will then be
parameterized on `TransactionalTree` instances providing
access to each of the provided `Tree` instances. This
allows you to atomically read and modify multiple
`Tree` instances in a single atomic operation.
These transactions are serializable, fully ACID,
and optimistic.
* `Tree::apply_batch` allows you to apply a `Batch`
* `TransactionalTree::apply_batch` allow you to
apply a `Batch` from within a transaction.
## Breaking Changes
* `Tree::batch` has been removed. Now you can directly
create a `Batch` with `Batch::default()` and then apply
it to a `Tree` with `Tree::apply_batch` or during a
transaction using `TransactionalTree::apply_batch`.
This facilitates multi-`Tree` batches via transactions.
* `Event::Merge` has been removed, and `Tree::merge` will
now send a complete `Event::Set` item to be distributed
to all listening subscribers.
================================================
FILE: CONTRIBUTING.md
================================================
# Welcome to the Project :)
* Don't be a jerk - here's our [code of conduct](./code-of-conduct.md).
We have a track record of defending our community from harm.
There are at least three great ways to contribute to sled:
* [financial contribution](https://github.com/sponsors/spacejam)
* coding
* conversation
#### Coding Considerations:
Please don't waste your time or ours by implementing things that
we do not want to introduce and maintain. Please discuss in an
issue or on chat before submitting a PR with:
* public API changes
* new functionality of any sort
* additional unsafe code
* significant refactoring
The above changes are unlikely to be merged or receive
timely attention without prior discussion.
PRs that generally require less coordination beforehand:
* Anything addressing a correctness issue.
* Better docs: whatever you find confusing!
* Small code changes with big performance implications, substantiated with [responsibly-gathered metrics](https://sled.rs/perf#experiment-checklist).
* FFI submodule changes: these are generally less well maintained than the Rust core, and benefit more from public assistance.
* Generally any new kind of test that avoids biases inherent in the others.
#### All PRs block on failing tests!
sled has intense testing, including crash tests, multi-threaded tests with
delay injection, a variety of mechanically-generated tests that combine fault
injection with concurrency in interesting ways, cross-compilation and minimum
supported Rust version checks, LLVM sanitizers, and more. It can sometimes be
challenging to understand why something is failing these intense tests.
For better understanding test failures, please:
1. read the failing test name and output log for clues
1. try to reproduce the failed test locally by running its associated command from the [test script](https://github.com/spacejam/sled/blob/main/.github/workflows/test.yml)
1. If it is not clear why your test is failing, feel free to request help with understanding it either on discord or requesting help on the PR, and we will do our best to help.
Want to help sled but don't have time for individual contributions? Contribute via [GitHub Sponsors](https://github.com/sponsors/spacejam) to support the people pushing the project forward!
================================================
FILE: Cargo.toml
================================================
[package]
name = "sled"
version = "1.0.0-alpha.124"
edition = "2024"
authors = ["Tyler Neely "]
documentation = "https://docs.rs/sled/"
description = "Lightweight high-performance pure-rust transactional embedded database."
license = "MIT OR Apache-2.0"
homepage = "https://github.com/spacejam/sled"
repository = "https://github.com/spacejam/sled"
keywords = ["redis", "mongo", "sqlite", "lmdb", "rocksdb"]
categories = ["database-implementations", "concurrency", "data-structures", "algorithms", "caching"]
readme = "README.md"
exclude = ["benchmarks", "examples", "bindings", "scripts", "experiments"]
[features]
# initializes allocated memory to 0xa1, writes 0xde to deallocated memory before freeing it
testing-shred-allocator = []
# use a counting global allocator that provides the sled::alloc::{allocated, freed, resident, reset} functions
testing-count-allocator = []
for-internal-testing-only = []
# turn off re-use of object IDs and heap slots, disable tree leaf merges, disable heap file truncation.
monotonic-behavior = []
[profile.release]
debug = true
opt-level = 3
overflow-checks = true
panic = "abort"
[profile.test]
debug = true
overflow-checks = true
panic = "abort"
[dependencies]
bincode = "1.3.3"
cache-advisor = "1.0.16"
concurrent-map = { version = "5.0.31", features = ["serde"] }
crc32fast = "1.3.2"
ebr = "0.2.13"
inline-array = { version = "0.1.13", features = ["serde", "concurrent_map_minimum"] }
fs2 = "0.4.3"
log = "0.4.19"
pagetable = "0.4.5"
parking_lot = { version = "0.12.1", features = ["arc_lock"] }
rayon = "1.7.0"
serde = { version = "1.0", features = ["derive"] }
stack-map = { version = "1.0.5", features = ["serde"] }
zstd = "0.12.4"
fnv = "1.0.7"
fault-injection = "1.0.10"
crossbeam-queue = "0.3.8"
crossbeam-channel = "0.5.8"
tempdir = "0.3.7"
[dev-dependencies]
env_logger = "0.10.0"
num-format = "0.4.4"
# heed = "0.11.0"
# rocksdb = "0.21.0"
# rusqlite = "0.29.0"
# old_sled = { version = "0.34", package = "sled" }
rand = "0.9"
quickcheck = "1.0.3"
rand_distr = "0.5"
libc = "0.2.147"
[[test]]
name = "test_crash_recovery"
path = "tests/test_crash_recovery.rs"
harness = false
================================================
FILE: LICENSE-APACHE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2015 Tyler Neely
Copyright 2016 Tyler Neely
Copyright 2017 Tyler Neely
Copyright 2018 Tyler Neely
Copyright 2019 Tyler Neely
Copyright 2020 Tyler Neely
Copyright 2021 Tyler Neely
Copyright 2022 Tyler Neely
Copyright 2023 Tyler Neely
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: LICENSE-MIT
================================================
Copyright (c) 2015 Tyler Neely
Copyright (c) 2016 Tyler Neely
Copyright (c) 2017 Tyler Neely
Copyright (c) 2018 Tyler Neely
Copyright (c) 2019 Tyler Neely
Copyright (c) 2020 Tyler Neely
Copyright (c) 2021 Tyler Neely
Copyright (c) 2022 Tyler Neely
Copyright (c) 2023 Tyler Neely
Permission is hereby granted, free of charge, to any
person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the
Software without restriction, including without
limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software
is furnished to do so, subject to the following
conditions:
The above copyright notice and this permission notice
shall be included in all copies or substantial portions
of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
================================================
FILE: README.md
================================================
# sled - ~~it's all downhill from here!!!~~
An embedded database.
```rust
let tree = sled::open("/tmp/welcome-to-sled")?;
// insert and get, similar to std's BTreeMap
let old_value = tree.insert("key", "value")?;
assert_eq!(
tree.get(&"key")?,
Some(sled::IVec::from("value")),
);
// range queries
for kv_result in tree.range("key_1".."key_9") {}
// deletion
let old_value = tree.remove(&"key")?;
// atomic compare and swap
tree.compare_and_swap(
"key",
Some("current_value"),
Some("new_value"),
)?;
// block until all operations are stable on disk
// (flush_async also available to get a Future)
tree.flush()?;
```
$${\color{red}This \space README \space is \space out \space of \space sync \space with \space the \space main \space branch \space which \space contains \space a \space large \space in-progress \space rewrite }$$
If you would like to work with structured data without paying expensive deserialization costs, check out the [structured](examples/structured.rs) example!
# features
* [API](https://docs.rs/sled) similar to a threadsafe `BTreeMap<[u8], [u8]>`
* serializable (ACID) [transactions](https://docs.rs/sled/latest/sled/struct.Tree.html#method.transaction)
for atomically reading and writing to multiple keys in multiple keyspaces.
* fully atomic single-key operations, including [compare and swap](https://docs.rs/sled/latest/sled/struct.Tree.html#method.compare_and_swap)
* zero-copy reads
* [write batches](https://docs.rs/sled/latest/sled/struct.Tree.html#method.apply_batch)
* [subscribe to changes on key
prefixes](https://docs.rs/sled/latest/sled/struct.Tree.html#method.watch_prefix)
* [multiple keyspaces](https://docs.rs/sled/latest/sled/struct.Db.html#method.open_tree)
* [merge operators](https://docs.rs/sled/latest/sled/doc/merge_operators/index.html)
* forward and reverse iterators over ranges of items
* a crash-safe monotonic [ID generator](https://docs.rs/sled/latest/sled/struct.Db.html#method.generate_id)
capable of generating 75-125 million unique ID's per second
* [zstd](https://github.com/facebook/zstd) compression (use the
`compression` build feature, disabled by default)
* cpu-scalable lock-free implementation
* flash-optimized log-structured storage
* uses modern b-tree techniques such as prefix encoding and suffix
truncation for reducing the storage costs of long keys with shared
prefixes. If keys are the same length and sequential then the
system can avoid storing 99%+ of the key data in most cases,
essentially acting like a learned index
# expectations, gotchas, advice
* Maybe one of the first things that seems weird is the `IVec` type.
This is an inlinable `Arc`ed slice that makes some things more efficient.
* Durability: **sled automatically fsyncs every 500ms by default**,
which can be configured with the `flush_every_ms` configurable, or you may
call `flush` / `flush_async` manually after operations.
* **Transactions are optimistic** - do not interact with external state
or perform IO from within a transaction closure unless it is
[idempotent](https://en.wikipedia.org/wiki/Idempotent).
* Internal tree node optimizations: sled performs prefix encoding
on long keys with similar prefixes that are grouped together in a range,
as well as suffix truncation to further reduce the indexing costs of
long keys. Nodes will skip potentially expensive length and offset pointers
if keys or values are all the same length (tracked separately, don't worry
about making keys the same length as values), so it may improve space usage
slightly if you use fixed-length keys or values. This also makes it easier
to use [structured access](examples/structured.rs) as well.
* sled does not support multiple open instances for the time being. Please
keep sled open for the duration of your process's lifespan. It's totally
safe and often quite convenient to use a global lazy_static sled instance,
modulo the normal global variable trade-offs. Every operation is threadsafe,
and most are implemented under the hood with lock-free algorithms that avoid
blocking in hot paths.
# performance
* [LSM tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree)-like write performance
with [traditional B+ tree](https://en.wikipedia.org/wiki/B%2B_tree)-like read performance
* over a billion operations in under a minute at 95% read 5% writes on 16 cores on a small dataset
* measure your own workloads rather than relying on some marketing for contrived workloads
# a note on lexicographic ordering and endianness
If you want to store numerical keys in a way that will play nicely with sled's iterators and ordered operations, please remember to store your numerical items in big-endian form. Little endian (the default of many things) will often appear to be doing the right thing until you start working with more than 256 items (more than 1 byte), causing lexicographic ordering of the serialized bytes to diverge from the lexicographic ordering of their deserialized numerical form.
* Rust integral types have built-in `to_be_bytes` and `from_be_bytes` [methods](https://doc.rust-lang.org/std/primitive.u64.html#method.from_be_bytes).
* bincode [can be configured](https://docs.rs/bincode/1.2.0/bincode/struct.Config.html#method.big_endian) to store integral types in big-endian form.
# interaction with async
If your dataset resides entirely in cache (achievable at startup by setting the cache
to a large enough value and performing a full iteration) then all reads and writes are
non-blocking and async-friendly, without needing to use Futures or an async runtime.
To asynchronously suspend your async task on the durability of writes, we support the
[`flush_async` method](https://docs.rs/sled/latest/sled/struct.Tree.html#method.flush_async),
which returns a Future that your async tasks can await the completion of if they require
high durability guarantees and you are willing to pay the latency costs of fsync.
Note that sled automatically tries to sync all data to disk several times per second
in the background without blocking user threads.
We support async subscription to events that happen on key prefixes, because the
`Subscriber` struct implements `Future>`:
```rust
let sled = sled::open("my_db").unwrap();
let mut sub = sled.watch_prefix("");
sled.insert(b"a", b"a").unwrap();
extreme::run(async move {
while let Some(event) = (&mut sub).await {
println!("got event {:?}", event);
}
});
```
# minimum supported Rust version (MSRV)
We support Rust 1.62 and up.
# architecture
lock-free tree on a lock-free pagecache on a lock-free log. the pagecache scatters
partial page fragments across the log, rather than rewriting entire pages at a time
as B+ trees for spinning disks historically have. on page reads, we concurrently
scatter-gather reads across the log to materialize the page from its fragments.
check out the [architectural outlook](https://github.com/spacejam/sled/wiki/sled-architectural-outlook)
for a more detailed overview of where we're at and where we see things going!
# philosophy
1. don't make the user think. the interface should be obvious.
1. don't surprise users with performance traps.
1. don't wake up operators. bring reliability techniques from academia into real-world practice.
1. don't use so much electricity. our data structures should play to modern hardware's strengths.
# known issues, warnings
* if reliability is your primary constraint, use SQLite. sled is beta.
* if storage price performance is your primary constraint, use RocksDB. sled uses too much space sometimes.
* if you have a multi-process workload that rarely writes, use LMDB. sled is architected for use with long-running, highly-concurrent workloads such as stateful services or higher-level databases.
* quite young, should be considered unstable for the time being.
* the on-disk format is going to change in ways that require [manual migrations](https://docs.rs/sled/latest/sled/struct.Db.html#method.export) before the `1.0.0` release!
# priorities
1. A full rewrite of sled's storage subsystem is happening on a modular basis as part of the [komora project](https://github.com/komora-io), in particular the marble storage engine. This will dramatically lower both the disk space usage (space amplification) and garbage collection overhead (write amplification) of sled.
2. The memory layout of tree nodes is being completely rewritten to reduce fragmentation and eliminate serialization costs.
3. The merge operator feature will change into a trigger feature that resembles traditional database triggers, allowing state to be modified as part of the same atomic writebatch that triggered it for retaining serializability with reactive semantics.
# fund feature development
Like what we're doing? Help us out via [GitHub Sponsors](https://github.com/sponsors/spacejam)!
================================================
FILE: RELEASE_CHECKLIST.md
================================================
# Release Checklist
This checklist must be completed before publishing a release of any kind.
Over time, anything in this list that can be turned into an automated test should be, but
there are still some big blind spots.
## API stability
- [ ] rust-flavored semver respected
## Performance
- [ ] micro-benchmark regressions should not happen unless newly discovered correctness criteria demands them
- [ ] mixed point operation latency distribution should narrow over time
- [ ] sequential operation average throughput should increase over time
- [ ] workloads should pass TSAN and ASAN on macOS. Linux should additionally pass LSAN & MSAN.
- [ ] workload write and space amplification thresholds should see no regressions
## Concurrency Audit
- [ ] any new `Guard` objects are dropped inside the rayon threadpool
- [ ] no new EBR `Collector`s, as they destroy causality. These will be optimized in-bulk in the future.
- [ ] no code assumes a recently read page pointer will remain unchanged (transactions may change this if reads are inline)
- [ ] no calls to `rand::thread_rng` from a droppable function (anything in the SegmentAccountant)
## Burn-In
- [ ] fuzz tests should run at least 24 hours each with zero crashes
- [ ] sequential and point workloads run at least 24 hours in constrained docker container without OOM / out of disk
================================================
FILE: SAFETY.md
================================================
# sled safety model
This document applies
[STPA](http://psas.scripts.mit.edu/home/get_file.php?name=STPA_handbook.pdf)-style
hazard analysis to the sled embedded database for the purpose of guiding
design and testing efforts to prevent unacceptable losses.
Outline
* [purpose of analysis](#purpose-of-analysis)
* [losses](#losses)
* [system boundary](#system-boundary)
* [hazards](#hazards)
* [leading indicators](#leading-indicators)
* [constraints](#constraints)
* [model of control structure](#model-of-control-structure)
* [identify unsafe control actions](#identify-unsafe-control-actions)
* [identify loss scenarios][#identify-loss-scenarios)
* [resources for learning more about STAMP, STPA, and CAST](#resources)
# Purpose of Analysis
## Losses
We wish to prevent the following undesirable situations:
* data loss
* inconsistent (non-linearizable) data access
* process crash
* resource exhaustion
## System Boundary
We draw the line between system and environment where we can reasonably
invest our efforts to prevent losses.
Inside the boundary:
* codebase
* put safe control actions into place that prevent losses
* documentation
* show users how to use sled safely
* recommend hardware, kernels, user code
Outside the boundary:
* Direct changes to hardware, kernels, user code
## Hazards
These hazards can result in the above losses:
* data may be lost if
* bugs in the logging system
* `Db::flush` fails to make previous writes durable
* bugs in the GC system
* the old location is overwritten before the defragmented location becomes durable
* bugs in the recovery system
* hardare failures
* consistency violations may be caused by
* transaction concurrency control failure to enforce linearizability (strict serializability)
* non-linearizable lock-free single-key operations
* panic
* of user threads
* IO threads
* flusher & GC thread
* indexing
* unwraps/expects
* failed TryInto/TryFrom + unwrap
* persistent storage exceeding (2 + N concurrent writers) * logical data size
* in-memory cache exceeding the configured cache size
* caused by incorrect calculation of cache
* use-after-free
* data race
* memory leak
* integer overflow
* buffer overrun
* uninitialized memory access
## Constraints
# Models of Control Structures
for each control action we have, consider:
1. what hazards happen when we fail to apply it / it does not exist?
2. what hazards happen when we do apply it
3. what hazards happen when we apply it too early or too late?
4. what hazards happen if we apply it for too long or not long enough?
durability model
* recovery
* LogIter::max_lsn
* return None if last_lsn_in_batch >= self.max_lsn
* batch requirement set to last reservation base + inline len - 1
* reserve bumps
* bump_atomic_lsn(&self.iobufs.max_reserved_lsn, reservation_lsn + inline_buf_len as Lsn - 1);
lock-free linearizability model
transactional linearizability (strict serializability) model
panic model
memory usage model
storage usage model
================================================
FILE: SECURITY.md
================================================
# Security Policy
## Reporting a Vulnerability
sled uses some unsafe functionality in the core lock-free algorithms, and in a few places to more efficiently copy data.
Please contact [Tyler Neely](mailto:tylerneely@gmail.com?subject=sled%20security%20issue) immediately if you find any vulnerability, and I will work with you to fix the issue rapidly and coordinate public disclosure with an expedited release including the fix.
If you are a bug hunter or a person with a security interest, here is my mental model of memory corruption risk in the sled codebase:
1. memory issues relating to the lock-free data structures in their colder failure paths. these have been tested a bit by injecting delays into random places, but this is still an area with elevated risk
1. anywhere the `unsafe` keyword is used
================================================
FILE: art/CREDITS
================================================
original tree logo with face:
https://twitter.com/daiyitastic
anti-transphobia additions:
spacejam
================================================
FILE: code-of-conduct.md
================================================
# Contributor Covenant Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at tylerneely@gmail.com. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
[homepage]: https://www.contributor-covenant.org
================================================
FILE: examples/bench.rs
================================================
use std::path::Path;
use std::sync::Barrier;
use std::thread::scope;
use std::time::{Duration, Instant};
use std::{fs, io};
use num_format::{Locale, ToFormattedString};
use sled::{Config, Db as SledDb};
type Db = SledDb<1024>;
const N_WRITES_PER_THREAD: u32 = 4 * 1024 * 1024;
const MAX_CONCURRENCY: u32 = 4;
const CONCURRENCY: &[usize] = &[/*1, 2, 4,*/ MAX_CONCURRENCY as _];
const BYTES_PER_ITEM: u32 = 8;
trait Databench: Clone + Send {
type READ: AsRef<[u8]>;
const NAME: &'static str;
const PATH: &'static str;
fn open() -> Self;
fn remove_generic(&self, key: &[u8]);
fn insert_generic(&self, key: &[u8], value: &[u8]);
fn get_generic(&self, key: &[u8]) -> Option;
fn flush_generic(&self);
fn print_stats(&self);
}
impl Databench for Db {
type READ = sled::InlineArray;
const NAME: &'static str = "sled 1.0.0-alpha";
const PATH: &'static str = "timing_test.sled-new";
fn open() -> Self {
sled::Config {
path: Self::PATH.into(),
zstd_compression_level: 3,
cache_capacity_bytes: 1024 * 1024 * 1024,
entry_cache_percent: 20,
flush_every_ms: Some(200),
..Config::default()
}
.open()
.unwrap()
}
fn insert_generic(&self, key: &[u8], value: &[u8]) {
self.insert(key, value).unwrap();
}
fn remove_generic(&self, key: &[u8]) {
self.remove(key).unwrap();
}
fn get_generic(&self, key: &[u8]) -> Option {
self.get(key).unwrap()
}
fn flush_generic(&self) {
self.flush().unwrap();
}
fn print_stats(&self) {
dbg!(self.stats());
}
}
/*
impl Databench for old_sled::Db {
type READ = old_sled::IVec;
const NAME: &'static str = "sled 0.34.7";
const PATH: &'static str = "timing_test.sled-old";
fn open() -> Self {
old_sled::open(Self::PATH).unwrap()
}
fn insert_generic(&self, key: &[u8], value: &[u8]) {
self.insert(key, value).unwrap();
}
fn get_generic(&self, key: &[u8]) -> Option {
self.get(key).unwrap()
}
fn flush_generic(&self) {
self.flush().unwrap();
}
}
*/
/*
impl Databench for Arc {
type READ = Vec;
const NAME: &'static str = "rocksdb 0.21.0";
const PATH: &'static str = "timing_test.rocksdb";
fn open() -> Self {
Arc::new(rocksdb::DB::open_default(Self::PATH).unwrap())
}
fn insert_generic(&self, key: &[u8], value: &[u8]) {
self.put(key, value).unwrap();
}
fn get_generic(&self, key: &[u8]) -> Option {
self.get(key).unwrap()
}
fn flush_generic(&self) {
self.flush().unwrap();
}
}
*/
/*
struct Lmdb {
env: heed::Env,
db: heed::Database<
heed::types::UnalignedSlice,
heed::types::UnalignedSlice,
>,
}
impl Clone for Lmdb {
fn clone(&self) -> Lmdb {
Lmdb { env: self.env.clone(), db: self.db.clone() }
}
}
impl Databench for Lmdb {
type READ = Vec;
const NAME: &'static str = "lmdb";
const PATH: &'static str = "timing_test.lmdb";
fn open() -> Self {
let _ = std::fs::create_dir_all(Self::PATH);
let env = heed::EnvOpenOptions::new()
.map_size(1024 * 1024 * 1024)
.open(Self::PATH)
.unwrap();
let db = env.create_database(None).unwrap();
Lmdb { env, db }
}
fn insert_generic(&self, key: &[u8], value: &[u8]) {
let mut wtxn = self.env.write_txn().unwrap();
self.db.put(&mut wtxn, key, value).unwrap();
wtxn.commit().unwrap();
}
fn get_generic(&self, key: &[u8]) -> Option {
let rtxn = self.env.read_txn().unwrap();
let ret = self.db.get(&rtxn, key).unwrap().map(Vec::from);
rtxn.commit().unwrap();
ret
}
fn flush_generic(&self) {
// NOOP
}
}
*/
/*
struct Sqlite {
connection: rusqlite::Connection,
}
impl Clone for Sqlite {
fn clone(&self) -> Sqlite {
Sqlite { connection: rusqlite::Connection::open(Self::PATH).unwrap() }
}
}
impl Databench for Sqlite {
type READ = Vec;
const NAME: &'static str = "sqlite";
const PATH: &'static str = "timing_test.sqlite";
fn open() -> Self {
let connection = rusqlite::Connection::open(Self::PATH).unwrap();
connection
.execute(
"create table if not exists bench (
key integer primary key,
val integer not null
)",
[],
)
.unwrap();
Sqlite { connection }
}
fn insert_generic(&self, key: &[u8], value: &[u8]) {
loop {
let res = self.connection.execute(
"insert or ignore into bench (key, val) values (?1, ?2)",
[
format!("{}", u32::from_be_bytes(key.try_into().unwrap())),
format!(
"{}",
u32::from_be_bytes(value.try_into().unwrap())
),
],
);
if res.is_ok() {
break;
}
}
}
fn get_generic(&self, key: &[u8]) -> Option {
let mut stmt = self
.connection
.prepare("SELECT b.val from bench b WHERE key = ?1")
.unwrap();
let mut rows =
stmt.query([u32::from_be_bytes(key.try_into().unwrap())]).unwrap();
let value = rows.next().unwrap()?;
value.get(0).ok()
}
fn flush_generic(&self) {
// NOOP
}
}
*/
fn allocated() -> usize {
#[cfg(feature = "testing-count-allocator")]
{
return sled::alloc::allocated();
}
0
}
fn freed() -> usize {
#[cfg(feature = "testing-count-allocator")]
{
return sled::alloc::freed();
}
0
}
fn resident() -> usize {
#[cfg(feature = "testing-count-allocator")]
{
return sled::alloc::resident();
}
0
}
fn inserts(store: &D) -> Vec {
println!("{} inserts", D::NAME);
let mut i = 0_u32;
let factory = move || {
i += 1;
(store.clone(), i - 1)
};
let f = |state: (D, u32)| {
let (store, offset) = state;
let start = N_WRITES_PER_THREAD * offset;
let end = N_WRITES_PER_THREAD * (offset + 1);
for i in start..end {
let k: &[u8] = &i.to_be_bytes();
store.insert_generic(k, k);
}
};
let mut ret = vec![];
for concurrency in CONCURRENCY {
let insert_elapsed =
execute_lockstep_concurrent(factory, f, *concurrency);
let flush_timer = Instant::now();
store.flush_generic();
let wps = (N_WRITES_PER_THREAD * *concurrency as u32) as u64
* 1_000_000_u64
/ u64::try_from(insert_elapsed.as_micros().max(1))
.unwrap_or(u64::MAX);
ret.push(InsertStats {
thread_count: *concurrency,
inserts_per_second: wps,
});
println!(
"{} inserts/s with {concurrency} threads over {:?}, then {:?} to flush {}",
wps.to_formatted_string(&Locale::en),
insert_elapsed,
flush_timer.elapsed(),
D::NAME,
);
}
ret
}
fn removes(store: &D) -> Vec {
println!("{} removals", D::NAME);
let mut i = 0_u32;
let factory = move || {
i += 1;
(store.clone(), i - 1)
};
let f = |state: (D, u32)| {
let (store, offset) = state;
let start = N_WRITES_PER_THREAD * offset;
let end = N_WRITES_PER_THREAD * (offset + 1);
for i in start..end {
let k: &[u8] = &i.to_be_bytes();
store.remove_generic(k);
}
};
let mut ret = vec![];
for concurrency in CONCURRENCY {
let remove_elapsed =
execute_lockstep_concurrent(factory, f, *concurrency);
let flush_timer = Instant::now();
store.flush_generic();
let wps = (N_WRITES_PER_THREAD * *concurrency as u32) as u64
* 1_000_000_u64
/ u64::try_from(remove_elapsed.as_micros().max(1))
.unwrap_or(u64::MAX);
ret.push(RemoveStats {
thread_count: *concurrency,
removes_per_second: wps,
});
println!(
"{} removes/s with {concurrency} threads over {:?}, then {:?} to flush {}",
wps.to_formatted_string(&Locale::en),
remove_elapsed,
flush_timer.elapsed(),
D::NAME,
);
}
ret
}
fn gets(store: &D) -> Vec {
println!("{} reads", D::NAME);
let factory = || store.clone();
let f = |store: D| {
let start = 0;
let end = N_WRITES_PER_THREAD * MAX_CONCURRENCY;
for i in start..end {
let k: &[u8] = &i.to_be_bytes();
store.get_generic(k);
}
};
let mut ret = vec![];
for concurrency in CONCURRENCY {
let get_stone_elapsed =
execute_lockstep_concurrent(factory, f, *concurrency);
let rps = (N_WRITES_PER_THREAD * MAX_CONCURRENCY * *concurrency as u32)
as u64
* 1_000_000_u64
/ u64::try_from(get_stone_elapsed.as_micros().max(1))
.unwrap_or(u64::MAX);
ret.push(GetStats { thread_count: *concurrency, gets_per_second: rps });
println!(
"{} gets/s with concurrency of {concurrency}, {:?} total reads {}",
rps.to_formatted_string(&Locale::en),
get_stone_elapsed,
D::NAME
);
}
ret
}
fn execute_lockstep_concurrent<
State: Send,
Factory: FnMut() -> State,
F: Sync + Fn(State),
>(
mut factory: Factory,
f: F,
concurrency: usize,
) -> Duration {
let barrier = &Barrier::new(concurrency + 1);
let f = &f;
scope(|s| {
let mut threads = vec![];
for _ in 0..concurrency {
let state = factory();
let thread = s.spawn(move || {
barrier.wait();
f(state);
});
threads.push(thread);
}
barrier.wait();
let get_stone = Instant::now();
for thread in threads.into_iter() {
thread.join().unwrap();
}
get_stone.elapsed()
})
}
#[derive(Debug, Clone, Copy)]
struct InsertStats {
thread_count: usize,
inserts_per_second: u64,
}
#[derive(Debug, Clone, Copy)]
struct GetStats {
thread_count: usize,
gets_per_second: u64,
}
#[derive(Debug, Clone, Copy)]
struct RemoveStats {
thread_count: usize,
removes_per_second: u64,
}
#[allow(unused)]
#[derive(Debug, Clone)]
struct Stats {
post_insert_disk_space: u64,
post_remove_disk_space: u64,
allocated_memory: usize,
freed_memory: usize,
resident_memory: usize,
insert_stats: Vec,
get_stats: Vec,
remove_stats: Vec,
}
impl Stats {
fn print_report(&self) {
println!(
"bytes on disk after inserts: {}",
self.post_insert_disk_space.to_formatted_string(&Locale::en)
);
println!(
"bytes on disk after removes: {}",
self.post_remove_disk_space.to_formatted_string(&Locale::en)
);
println!(
"bytes in memory: {}",
self.resident_memory.to_formatted_string(&Locale::en)
);
for stats in &self.insert_stats {
println!(
"{} threads {} inserts per second",
stats.thread_count,
stats.inserts_per_second.to_formatted_string(&Locale::en)
);
}
for stats in &self.get_stats {
println!(
"{} threads {} gets per second",
stats.thread_count,
stats.gets_per_second.to_formatted_string(&Locale::en)
);
}
for stats in &self.remove_stats {
println!(
"{} threads {} removes per second",
stats.thread_count,
stats.removes_per_second.to_formatted_string(&Locale::en)
);
}
}
}
fn bench() -> Stats {
let store = D::open();
let insert_stats = inserts(&store);
let before_flush = Instant::now();
store.flush_generic();
println!("final flush took {:?} for {}", before_flush.elapsed(), D::NAME);
let post_insert_disk_space = du(D::PATH.as_ref()).unwrap();
let get_stats = gets(&store);
let remove_stats = removes(&store);
store.print_stats();
Stats {
post_insert_disk_space,
post_remove_disk_space: du(D::PATH.as_ref()).unwrap(),
allocated_memory: allocated(),
freed_memory: freed(),
resident_memory: resident(),
insert_stats,
get_stats,
remove_stats,
}
}
fn du(path: &Path) -> io::Result {
fn recurse(mut dir: fs::ReadDir) -> io::Result {
dir.try_fold(0, |acc, file| {
let file = file?;
let size = match file.metadata()? {
data if data.is_dir() => recurse(fs::read_dir(file.path())?)?,
data => data.len(),
};
Ok(acc + size)
})
}
recurse(fs::read_dir(path)?)
}
fn main() {
let _ = env_logger::try_init();
let new_stats = bench::();
println!(
"raw data size: {}",
(MAX_CONCURRENCY * N_WRITES_PER_THREAD * BYTES_PER_ITEM)
.to_formatted_string(&Locale::en)
);
println!("sled 1.0 space stats:");
new_stats.print_report();
/*
let old_stats = bench::();
dbg!(old_stats);
let new_sled_vs_old_sled_storage_ratio =
new_stats.disk_space as f64 / old_stats.disk_space as f64;
let new_sled_vs_old_sled_allocated_memory_ratio =
new_stats.allocated_memory as f64 / old_stats.allocated_memory as f64;
let new_sled_vs_old_sled_freed_memory_ratio =
new_stats.freed_memory as f64 / old_stats.freed_memory as f64;
let new_sled_vs_old_sled_resident_memory_ratio =
new_stats.resident_memory as f64 / old_stats.resident_memory as f64;
dbg!(new_sled_vs_old_sled_storage_ratio);
dbg!(new_sled_vs_old_sled_allocated_memory_ratio);
dbg!(new_sled_vs_old_sled_freed_memory_ratio);
dbg!(new_sled_vs_old_sled_resident_memory_ratio);
let rocksdb_stats = bench::>();
bench::();
bench::();
*/
/*
let new_sled_vs_rocksdb_storage_ratio =
new_stats.disk_space as f64 / rocksdb_stats.disk_space as f64;
let new_sled_vs_rocksdb_allocated_memory_ratio =
new_stats.allocated_memory as f64 / rocksdb_stats.allocated_memory as f64;
let new_sled_vs_rocksdb_freed_memory_ratio =
new_stats.freed_memory as f64 / rocksdb_stats.freed_memory as f64;
let new_sled_vs_rocksdb_resident_memory_ratio =
new_stats.resident_memory as f64 / rocksdb_stats.resident_memory as f64;
dbg!(new_sled_vs_rocksdb_storage_ratio);
dbg!(new_sled_vs_rocksdb_allocated_memory_ratio);
dbg!(new_sled_vs_rocksdb_freed_memory_ratio);
dbg!(new_sled_vs_rocksdb_resident_memory_ratio);
*/
/*
let scan = Instant::now();
let count = stone.iter().count();
assert_eq!(count as u64, N_WRITES_PER_THREAD);
let scan_elapsed = scan.elapsed();
println!(
"{} scanned items/s, total {:?}",
(N_WRITES_PER_THREAD * 1_000_000) / u64::try_from(scan_elapsed.as_micros().max(1)).unwrap_or(u64::MAX),
scan_elapsed
);
*/
/*
let scan_rev = Instant::now();
let count = stone.range(..).rev().count();
assert_eq!(count as u64, N_WRITES_PER_THREAD);
let scan_rev_elapsed = scan_rev.elapsed();
println!(
"{} reverse-scanned items/s, total {:?}",
(N_WRITES_PER_THREAD * 1_000_000) / u64::try_from(scan_rev_elapsed.as_micros().max(1)).unwrap_or(u64::MAX),
scan_rev_elapsed
);
*/
}
================================================
FILE: fuzz/.gitignore
================================================
target
corpus
artifacts
================================================
FILE: fuzz/Cargo.toml
================================================
[package]
name = "bloodstone-fuzz"
version = "0.0.0"
authors = ["Automatically generated"]
publish = false
edition = "2018"
[package.metadata]
cargo-fuzz = true
[dependencies.libfuzzer-sys]
version = "0.4.0"
features = ["arbitrary-derive"]
[dependencies]
arbitrary = { version = "1.0.3", features = ["derive"] }
tempfile = "3.5.0"
[dependencies.sled]
path = ".."
features = []
# Prevent this from interfering with workspaces
[workspace]
members = ["."]
[[bin]]
name = "fuzz_model"
path = "fuzz_targets/fuzz_model.rs"
test = false
doc = false
================================================
FILE: fuzz/fuzz_targets/fuzz_model.rs
================================================
#![no_main]
#[macro_use]
extern crate libfuzzer_sys;
extern crate arbitrary;
extern crate sled;
use arbitrary::Arbitrary;
use sled::{Config, Db as SledDb, InlineArray};
type Db = SledDb<3>;
const KEYSPACE: u64 = 128;
#[derive(Debug)]
enum Op {
Get { key: InlineArray },
Insert { key: InlineArray, value: InlineArray },
Reboot,
Remove { key: InlineArray },
Cas { key: InlineArray, old: Option, new: Option },
Range { start: InlineArray, end: InlineArray },
}
fn keygen(
u: &mut arbitrary::Unstructured<'_>,
) -> arbitrary::Result {
let key_i: u64 = u.int_in_range(0..=KEYSPACE)?;
Ok(key_i.to_be_bytes().as_ref().into())
}
impl<'a> Arbitrary<'a> for Op {
fn arbitrary(
u: &mut arbitrary::Unstructured<'a>,
) -> arbitrary::Result {
Ok(if u.ratio(1, 2)? {
Op::Insert { key: keygen(u)?, value: keygen(u)? }
} else if u.ratio(1, 2)? {
Op::Get { key: keygen(u)? }
} else if u.ratio(1, 2)? {
Op::Reboot
} else if u.ratio(1, 2)? {
Op::Remove { key: keygen(u)? }
} else if u.ratio(1, 2)? {
Op::Cas {
key: keygen(u)?,
old: if u.ratio(1, 2)? { Some(keygen(u)?) } else { None },
new: if u.ratio(1, 2)? { Some(keygen(u)?) } else { None },
}
} else {
let start = u.int_in_range(0..=KEYSPACE)?;
let end = (start + 1).max(u.int_in_range(0..=KEYSPACE)?);
Op::Range {
start: start.to_be_bytes().as_ref().into(),
end: end.to_be_bytes().as_ref().into(),
}
})
}
}
fuzz_target!(|ops: Vec| {
let tmp_dir = tempfile::TempDir::new().unwrap();
let tmp_path = tmp_dir.path().to_owned();
let config = Config::new().path(tmp_path);
let mut tree: Db = config.open().unwrap();
let mut model = std::collections::BTreeMap::new();
for (_i, op) in ops.into_iter().enumerate() {
match op {
Op::Insert { key, value } => {
assert_eq!(
tree.insert(key.clone(), value.clone()).unwrap(),
model.insert(key, value)
);
}
Op::Get { key } => {
assert_eq!(tree.get(&key).unwrap(), model.get(&key).cloned());
}
Op::Reboot => {
drop(tree);
tree = config.open().unwrap();
}
Op::Remove { key } => {
assert_eq!(tree.remove(&key).unwrap(), model.remove(&key));
}
Op::Range { start, end } => {
let mut model_iter =
model.range::(&start..&end);
let mut tree_iter = tree.range(start..end);
for (k1, v1) in &mut model_iter {
let (k2, v2) = tree_iter
.next()
.expect("None returned from iter when Some expected")
.expect("IO issue encountered");
assert_eq!((k1, v1), (&k2, &v2));
}
assert!(tree_iter.next().is_none());
}
Op::Cas { key, old, new } => {
let succ = if old == model.get(&key).cloned() {
if let Some(n) = &new {
model.insert(key.clone(), n.clone());
} else {
model.remove(&key);
}
true
} else {
false
};
let res = tree
.compare_and_swap(key, old.as_ref(), new)
.expect("hit IO error");
if succ {
assert!(res.is_ok());
} else {
assert!(res.is_err());
}
}
};
for (key, value) in &model {
assert_eq!(tree.get(key).unwrap().unwrap(), value);
}
for kv_res in &tree {
let (key, value) = kv_res.unwrap();
assert_eq!(model.get(&key), Some(&value));
}
}
let mut model_iter = model.iter();
let mut tree_iter = tree.iter();
for (k1, v1) in &mut model_iter {
let (k2, v2) = tree_iter.next().unwrap().unwrap();
assert_eq!((k1, v1), (&k2, &v2));
}
assert!(tree_iter.next().is_none());
});
================================================
FILE: scripts/cgtest.sh
================================================
#!/bin/sh
set -e
cgdelete memory:sledTest || true
cgcreate -g memory:sledTest
echo 100M > /sys/fs/cgroup/memory/sledTest/memory.limit_in_bytes
su $SUDO_USER -c 'cargo build --release --features=testing'
for test in target/release/deps/test*; do
if [[ -x $test ]]
then
echo running test: $test
cgexec -g memory:sledTest $test --test-threads=1
rm $test
fi
done
================================================
FILE: scripts/cross_compile.sh
================================================
#!/bin/sh
set -e
# checks sled's compatibility using several targets
targets="wasm32-wasi wasm32-unknown-unknown aarch64-fuchsia aarch64-linux-android \
i686-linux-android i686-unknown-linux-gnu \
x86_64-linux-android x86_64-fuchsia \
mips-unknown-linux-musl aarch64-apple-ios"
rustup update --no-self-update
RUSTFLAGS="--cfg miri" cargo check
rustup toolchain install 1.62 --no-self-update
cargo clean
rm Cargo.lock
cargo +1.62 check
for target in $targets; do
echo "setting up $target..."
rustup target add $target
echo "checking $target..."
cargo check --target $target
done
================================================
FILE: scripts/execution_explorer.py
================================================
#!/usr/bin/gdb --command
"""
a simple python GDB script for running multithreaded
programs in a way that is "deterministic enough"
to tease out and replay interesting bugs.
Tyler Neely 25 Sept 2017
t@jujit.su
references:
https://sourceware.org/gdb/onlinedocs/gdb/All_002dStop-Mode.html
https://sourceware.org/gdb/onlinedocs/gdb/Non_002dStop-Mode.html
https://sourceware.org/gdb/onlinedocs/gdb/Threads-In-Python.html
https://sourceware.org/gdb/onlinedocs/gdb/Events-In-Python.html
https://blog.0x972.info/index.php?tag=gdb.py
"""
import gdb
import random
###############################################################################
# config #
###############################################################################
# set this to a number for reproducing results or None to explore randomly
seed = 156112673742 # None # 951931004895
# set this to the number of valid threads in the program
# {2, 3} assumes a main thread that waits on 2 workers.
# {1, ... N} assumes all of the first N threads are to be explored
threads_whitelist = {2, 3}
# set this to the file of the binary to explore
filename = "target/debug/binary"
# set this to the place the threads should rendezvous before exploring
entrypoint = "src/main.rs:8"
# set this to after the threads are done
exitpoint = "src/main.rs:12"
# invariant unreachable points that should never be accessed
unreachable = [
"panic_unwind::imp::panic"
]
# set this to the locations you want to test interleavings for
interesting = [
"src/main.rs:8",
"src/main.rs:9"
]
# uncomment this to output the specific commands issued to gdb
gdb.execute("set trace-commands on")
###############################################################################
###############################################################################
class UnreachableBreakpoint(gdb.Breakpoint):
pass
class DoneBreakpoint(gdb.Breakpoint):
pass
class InterestingBreakpoint(gdb.Breakpoint):
pass
class DeterministicExecutor:
def __init__(self, seed=None):
if seed:
print("seeding with", seed)
self.seed = seed
random.seed(seed)
else:
# pick a random new seed if not provided with one
self.reseed()
gdb.execute("file " + filename)
# non-stop is necessary to provide thread-specific
# information when breakpoints are hit.
gdb.execute("set non-stop on")
gdb.execute("set confirm off")
self.ready = set()
self.finished = set()
def reseed(self):
random.seed()
self.seed = random.randrange(1e12)
print("reseeding with", self.seed)
random.seed(self.seed)
def restart(self):
# reset inner state
self.ready = set()
self.finished = set()
# disconnect callbacks
gdb.events.stop.disconnect(self.scheduler_callback)
gdb.events.exited.disconnect(self.exit_callback)
# nuke all breakpoints
gdb.execute("d")
# end execution
gdb.execute("k")
# pick new seed
self.reseed()
self.run()
def rendezvous_callback(self, event):
try:
self.ready.add(event.inferior_thread.num)
if len(self.ready) == len(threads_whitelist):
self.run_schedule()
except Exception as e:
# this will be thrown if breakpoint is not a part of event,
# like when the event was stopped for another reason.
print(e)
def run(self):
gdb.execute("b " + entrypoint)
gdb.events.stop.connect(self.rendezvous_callback)
gdb.events.exited.connect(self.exit_callback)
gdb.execute("r")
def run_schedule(self):
print("running schedule")
gdb.execute("d")
gdb.events.stop.disconnect(self.rendezvous_callback)
gdb.events.stop.connect(self.scheduler_callback)
for bp in interesting:
InterestingBreakpoint(bp)
for bp in unreachable:
UnreachableBreakpoint(bp)
DoneBreakpoint(exitpoint)
self.pick()
def pick(self):
threads = self.runnable_threads()
if not threads:
print("restarting execution after running out of valid threads")
self.restart()
return
thread = random.choice(threads)
gdb.execute("t " + str(thread.num))
gdb.execute("c")
def scheduler_callback(self, event):
if not isinstance(event, gdb.BreakpointEvent):
print("WTF sched callback got", event.__dict__)
return
if isinstance(event.breakpoint, DoneBreakpoint):
self.finished.add(event.inferior_thread.num)
elif isinstance(event.breakpoint, UnreachableBreakpoint):
print("!" * 80)
print("unreachable breakpoint triggered with seed", self.seed)
print("!" * 80)
gdb.events.exited.disconnect(self.exit_callback)
gdb.execute("q")
else:
print("thread", event.inferior_thread.num,
"hit breakpoint at", event.breakpoint.location)
self.pick()
def runnable_threads(self):
threads = gdb.selected_inferior().threads()
def f(it):
return (it.is_valid() and not
it.is_exited() and
it.num in threads_whitelist and
it.num not in self.finished)
good_threads = [it for it in threads if f(it)]
good_threads.sort(key=lambda it: it.num)
return good_threads
def exit_callback(self, event):
try:
if event.exit_code != 0:
print("!" * 80)
print("interesting exit with seed", self.seed)
print("!" * 80)
else:
print("happy exit")
self.restart()
gdb.execute("q")
except Exception as e:
pass
de = DeterministicExecutor(seed)
de.run()
================================================
FILE: scripts/instructions
================================================
#!/bin/sh
# counts instructions for a standard workload
set -e
OUTFILE="cachegrind.stress2.`git describe --always --dirty`-`date +%s`"
rm -rf default.sled || true
cargo build \
--bin=stress2 \
--release
# --tool=callgrind --dump-instr=yes --collect-jumps=yes --simulate-cache=yes \
# --callgrind-out-file="$OUTFILE" \
valgrind \
--tool=cachegrind \
--cachegrind-out-file="$OUTFILE" \
./target/release/stress2 --total-ops=50000 --set-prop=1000000000000 --threads=1
LAST=`ls -t cachegrind.stress2.* | sed -n 2p`
echo "comparing $LAST with new $OUTFILE"
echo "--------------------------------------------------------------------------------"
echo "change since last run:"
echo " Ir I1mr ILmr Dr D1mr DLmr Dw D1mw DLmw"
echo "--------------------------------------------------------------------------------"
cg_diff $LAST $OUTFILE | tail -1
================================================
FILE: scripts/sanitizers.sh
================================================
#!/bin/bash
set -eo pipefail
pushd benchmarks/stress2
rustup toolchain install nightly
rustup toolchain install nightly --component rust-src
rustup update
export SLED_LOCK_FREE_DELAY_INTENSITY=2000
echo "msan"
cargo clean
export RUSTFLAGS="-Zsanitizer=memory -Zsanitizer-memory-track-origins"
cargo +nightly build -Zbuild-std --target x86_64-unknown-linux-gnu
sudo rm -rf default.sled
sudo target/x86_64-unknown-linux-gnu/debug/stress2 --duration=30 --set-prop=100000000 --val-len=1000 --entries=100 --threads=100
sudo target/x86_64-unknown-linux-gnu/debug/stress2 --duration=30 --entries=100
sudo target/x86_64-unknown-linux-gnu/debug/stress2 --duration=30
unset MSAN_OPTIONS
echo "asan"
cargo clean
export RUSTFLAGS="-Z sanitizer=address"
export ASAN_OPTIONS="detect_odr_violation=0"
cargo +nightly build --features=lock_free_delays --target x86_64-unknown-linux-gnu
sudo rm -rf default.sled
sudo target/x86_64-unknown-linux-gnu/debug/stress2 --duration=60
sudo target/x86_64-unknown-linux-gnu/debug/stress2 --duration=6
unset ASAN_OPTIONS
echo "lsan"
cargo clean
export RUSTFLAGS="-Z sanitizer=leak"
cargo +nightly build --features=lock_free_delays --target x86_64-unknown-linux-gnu
sudo rm -rf default.sled
sudo target/x86_64-unknown-linux-gnu/debug/stress2 --duration=60
sudo target/x86_64-unknown-linux-gnu/debug/stress2 --duration=6
echo "tsan"
cargo clean
export RUSTFLAGS="-Z sanitizer=thread"
export TSAN_OPTIONS=suppressions=../../tsan_suppressions.txt
sudo rm -rf default.sled
cargo +nightly run --features=lock_free_delays --target x86_64-unknown-linux-gnu -- --duration=60
cargo +nightly run --features=lock_free_delays --target x86_64-unknown-linux-gnu -- --duration=6
unset RUSTFLAGS
unset TSAN_OPTIONS
================================================
FILE: scripts/shufnice.sh
================================================
#!/bin/sh
while true; do
PID=`pgrep $1`
TIDS=`ls /proc/$PID/task`
TID=`echo $TIDS | tr " " "\n" | shuf -n1`
NICE=$((`shuf -i 0-39 -n 1` - 20))
echo "renicing $TID to $NICE"
renice -n $NICE -p $TID
done
================================================
FILE: scripts/ubuntu_bench
================================================
#!/bin/sh
sudo apt-get update
sudo apt-get install htop dstat build-essential linux-tools-common linux-tools-generic linux-tools-`uname -r`
curl https://sh.rustup.rs -sSf | sh
source $HOME/.cargo/env
cargo install flamegraph
git clone https://github.com/spacejam/sled.git
cd sled
cores=$(grep -c ^processor /proc/cpuinfo)
writers=(($cores / 5 + 1 ))
readers=$(( ($cores / 5 + 1) * 4 ))
cargo build --release --bin=stress2 --features=stress
# we use sudo here to get access to symbols
pushd benchmarks/stress2
cargo flamegraph --release -- --get=$readers --set=$writers
================================================
FILE: src/alloc.rs
================================================
#[cfg(any(
feature = "testing-shred-allocator",
feature = "testing-count-allocator"
))]
pub use alloc::*;
// the memshred feature causes all allocated and deallocated
// memory to be set to a specific non-zero value of 0xa1 for
// uninitialized allocations and 0xde for deallocated memory,
// in the hope that it will cause memory errors to surface
// more quickly.
#[cfg(feature = "testing-shred-allocator")]
mod alloc {
use std::alloc::{Layout, System};
#[global_allocator]
static ALLOCATOR: ShredAllocator = ShredAllocator;
#[derive(Default, Debug, Clone, Copy)]
struct ShredAllocator;
unsafe impl std::alloc::GlobalAlloc for ShredAllocator {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
let ret = System.alloc(layout);
assert_ne!(ret, std::ptr::null_mut());
std::ptr::write_bytes(ret, 0xa1, layout.size());
ret
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
std::ptr::write_bytes(ptr, 0xde, layout.size());
System.dealloc(ptr, layout)
}
}
}
#[cfg(feature = "testing-count-allocator")]
mod alloc {
use std::alloc::{Layout, System};
#[global_allocator]
static ALLOCATOR: CountingAllocator = CountingAllocator;
static ALLOCATED: AtomicUsize = AtomicUsize::new(0);
static FREED: AtomicUsize = AtomicUsize::new(0);
static RESIDENT: AtomicUsize = AtomicUsize::new(0);
fn allocated() -> usize {
ALLOCATED.swap(0, Ordering::Relaxed)
}
fn freed() -> usize {
FREED.swap(0, Ordering::Relaxed)
}
fn resident() -> usize {
RESIDENT.load(Ordering::Relaxed)
}
#[derive(Default, Debug, Clone, Copy)]
struct CountingAllocator;
unsafe impl std::alloc::GlobalAlloc for CountingAllocator {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
let ret = System.alloc(layout);
assert_ne!(ret, std::ptr::null_mut());
ALLOCATED.fetch_add(layout.size(), Ordering::Relaxed);
RESIDENT.fetch_add(layout.size(), Ordering::Relaxed);
std::ptr::write_bytes(ret, 0xa1, layout.size());
ret
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
std::ptr::write_bytes(ptr, 0xde, layout.size());
FREED.fetch_add(layout.size(), Ordering::Relaxed);
RESIDENT.fetch_sub(layout.size(), Ordering::Relaxed);
System.dealloc(ptr, layout)
}
}
}
================================================
FILE: src/block_checker.rs
================================================
use std::collections::BTreeMap;
use std::panic::Location;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{LazyLock, Mutex};
static COUNTER: AtomicU64 = AtomicU64::new(0);
static CHECK_INS: LazyLock = LazyLock::new(|| {
std::thread::spawn(move || {
let mut last_top_10 = Default::default();
loop {
std::thread::sleep(std::time::Duration::from_secs(5));
last_top_10 = CHECK_INS.report(last_top_10);
}
});
BlockChecker::default()
});
type LocationMap = BTreeMap>;
#[derive(Default)]
pub(crate) struct BlockChecker {
state: Mutex,
}
impl BlockChecker {
fn report(&self, last_top_10: LocationMap) -> LocationMap {
let state = self.state.lock().unwrap();
println!("top 10 longest blocking sections:");
let top_10: LocationMap =
state.iter().take(10).map(|(k, v)| (*k, *v)).collect();
for (id, location) in &top_10 {
if last_top_10.contains_key(id) {
println!("id: {}, location: {:?}", id, location);
}
}
top_10
}
fn check_in(&self, location: &'static Location) -> BlockGuard {
let next_id = COUNTER.fetch_add(1, Ordering::Relaxed);
let mut state = self.state.lock().unwrap();
state.insert(next_id, location);
BlockGuard { id: next_id }
}
fn check_out(&self, id: u64) {
let mut state = self.state.lock().unwrap();
state.remove(&id);
}
}
pub(crate) struct BlockGuard {
id: u64,
}
impl Drop for BlockGuard {
fn drop(&mut self) {
CHECK_INS.check_out(self.id)
}
}
#[track_caller]
pub(crate) fn track_blocks() -> BlockGuard {
let caller = Location::caller();
CHECK_INS.check_in(caller)
}
================================================
FILE: src/config.rs
================================================
use std::io;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use fault_injection::{annotate, fallible};
use tempdir::TempDir;
use crate::Db;
macro_rules! builder {
($(($name:ident, $t:ty, $desc:expr)),*) => {
$(
#[doc=$desc]
pub fn $name(mut self, to: $t) -> Self {
self.$name = to;
self
}
)*
}
}
#[derive(Debug, Clone)]
pub struct Config {
/// The base directory for storing the database.
pub path: PathBuf,
/// Cache size in **bytes**. Default is 512mb.
pub cache_capacity_bytes: usize,
/// The percentage of the cache that is dedicated to the
/// scan-resistant entry cache.
pub entry_cache_percent: u8,
/// Start a background thread that flushes data to disk
/// every few milliseconds. Defaults to every 200ms.
pub flush_every_ms: Option,
/// The zstd compression level to use when writing data to disk. Defaults to 3.
pub zstd_compression_level: i32,
/// This is only set to `Some` for objects created via
/// `Config::tmp`, and will remove the storage directory
/// when the final Arc drops.
pub tempdir_deleter: Option>,
/// A float between 0.0 and 1.0 that controls how much fragmentation can
/// exist in a file before GC attempts to recompact it.
pub target_heap_file_fill_ratio: f32,
/// Values larger than this configurable will be stored as separate blob
pub max_inline_value_threshold: usize,
}
impl Default for Config {
fn default() -> Config {
Config {
path: "bloodstone.default".into(),
flush_every_ms: Some(200),
cache_capacity_bytes: 512 * 1024 * 1024,
entry_cache_percent: 20,
zstd_compression_level: 3,
tempdir_deleter: None,
target_heap_file_fill_ratio: 0.9,
max_inline_value_threshold: 4096,
}
}
}
impl Config {
/// Returns a default `Config`
pub fn new() -> Config {
Config::default()
}
/// Returns a config with the `path` initialized to a system
/// temporary directory that will be deleted when this `Config`
/// is dropped.
pub fn tmp() -> io::Result {
let tempdir = fallible!(tempdir::TempDir::new("sled_tmp"));
Ok(Config {
path: tempdir.path().into(),
tempdir_deleter: Some(Arc::new(tempdir)),
..Config::default()
})
}
/// Set the path of the database (builder).
pub fn path>(mut self, path: P) -> Config {
self.path = path.as_ref().to_path_buf();
self
}
builder!(
(flush_every_ms, Option, "Start a background thread that flushes data to disk every few milliseconds. Defaults to every 200ms."),
(cache_capacity_bytes, usize, "Cache size in **bytes**. Default is 512mb."),
(entry_cache_percent, u8, "The percentage of the cache that is dedicated to the scan-resistant entry cache."),
(zstd_compression_level, i32, "The zstd compression level to use when writing data to disk. Defaults to 3."),
(target_heap_file_fill_ratio, f32, "A float between 0.0 and 1.0 that controls how much fragmentation can exist in a file before GC attempts to recompact it."),
(max_inline_value_threshold, usize, "Values larger than this configurable will be stored as separate blob")
);
pub fn open(
&self,
) -> io::Result> {
if LEAF_FANOUT < 3 {
return Err(annotate!(io::Error::new(
io::ErrorKind::Unsupported,
"Db's LEAF_FANOUT const generic must be 3 or greater."
)));
}
Db::open_with_config(self)
}
}
================================================
FILE: src/db.rs
================================================
use std::collections::HashMap;
use std::fmt;
use std::io;
use std::sync::{Arc, mpsc};
use std::time::{Duration, Instant};
use parking_lot::Mutex;
use crate::*;
/// sled 1.0 alpha :)
///
/// One of the main differences between this and sled 0.34 is that
/// `Db` and `Tree` now have a `LEAF_FANOUT` const generic parameter.
/// This parameter is an interesting single-knob performance tunable
/// that allows users to traverse the performance-vs-efficiency
/// trade-off spectrum. The default value of `1024` causes keys and
/// values to be more efficiently compressed when stored on disk,
/// but for larger-than-memory random workloads it may be advantageous
/// to lower `LEAF_FANOUT` to between `16` to `256`, depending on your
/// efficiency requirements. A lower value will also cause contention
/// to be reduced for frequently accessed data. This value cannot be
/// changed after creating the database.
///
/// As an alpha release, please do not expect this to be safe for
/// business-critical use cases. However, if you would like this to
/// serve your business-critical use cases over time, please give it
/// a shot in a low-risk non-production environment and report any
/// issues you encounter in a github issue.
///
/// Note that `Db` implements `Deref` for the default `Tree` (sled's
/// version of namespaces / keyspaces / buckets), but you can create
/// and use others using `Db::open_tree`.
#[derive(Clone)]
pub struct Db {
config: Config,
_shutdown_dropper: Arc>,
cache: ObjectCache,
trees: Arc>>>,
collection_id_allocator: Arc,
collection_name_mapping: Tree,
default_tree: Tree,
was_recovered: bool,
}
impl std::ops::Deref for Db {
type Target = Tree;
fn deref(&self) -> &Tree {
&self.default_tree
}
}
impl IntoIterator for &Db {
type Item = io::Result<(InlineArray, InlineArray)>;
type IntoIter = crate::Iter;
fn into_iter(self) -> Self::IntoIter {
self.iter()
}
}
impl fmt::Debug for Db {
fn fmt(&self, w: &mut fmt::Formatter<'_>) -> fmt::Result {
let alternate = w.alternate();
let mut debug_struct = w.debug_struct(&format!("Db<{}>", LEAF_FANOUT));
if alternate {
debug_struct
.field("global_error", &self.check_error())
.field(
"data",
&format!("{:?}", self.iter().collect::>()),
)
.finish()
} else {
debug_struct.field("global_error", &self.check_error()).finish()
}
}
}
fn flusher(
cache: ObjectCache,
shutdown_signal: mpsc::Receiver>,
flush_every_ms: usize,
) {
let interval = Duration::from_millis(flush_every_ms as _);
let mut last_flush_duration = Duration::default();
let flush = || {
let flush_res_res = std::panic::catch_unwind(|| cache.flush());
match flush_res_res {
Ok(Ok(_)) => {
// don't abort.
return;
}
Ok(Err(flush_failure)) => {
log::error!(
"Db flusher encountered error while flushing: {:?}",
flush_failure
);
cache.set_error(&flush_failure);
}
Err(panicked) => {
log::error!(
"Db flusher panicked while flushing: {:?}",
panicked
);
cache.set_error(&io::Error::other(
"Db flusher panicked while flushing".to_string(),
));
}
}
std::process::abort();
};
loop {
let recv_timeout = interval
.saturating_sub(last_flush_duration)
.max(Duration::from_millis(1));
if let Ok(shutdown_sender) = shutdown_signal.recv_timeout(recv_timeout)
{
flush();
// this is probably unnecessary but it will avoid issues
// if egregious bugs get introduced that trigger it
cache.set_error(&io::Error::other(
"system has been shut down".to_string(),
));
assert!(cache.is_clean());
drop(cache);
if let Err(e) = shutdown_sender.send(()) {
log::error!(
"Db flusher could not ack shutdown to requestor: {e:?}"
);
}
log::debug!(
"flush thread terminating after signalling to requestor"
);
return;
}
let before_flush = Instant::now();
flush();
last_flush_duration = before_flush.elapsed();
}
}
impl Drop for Db {
fn drop(&mut self) {
if self.config.flush_every_ms.is_none() {
if let Err(e) = self.flush() {
log::error!("failed to flush Db on Drop: {e:?}");
}
} else {
// otherwise, it is expected that the flusher thread will
// flush while shutting down the final Db/Tree instance
}
}
}
impl Db {
#[cfg(feature = "for-internal-testing-only")]
fn validate(&self) -> io::Result<()> {
// for each tree, iterate over index, read node and assert low key matches
// and assert first time we've ever seen node ID
let mut ever_seen = std::collections::HashSet::new();
let before = std::time::Instant::now();
#[cfg(feature = "for-internal-testing-only")]
let _b0 = crate::block_checker::track_blocks();
for (_cid, tree) in self.trees.lock().iter() {
let mut hi_none_count = 0;
let mut last_hi = None;
for (low, node) in tree.index.iter() {
// ensure we haven't reused the object_id across Trees
assert!(ever_seen.insert(node.object_id));
let (read_low, node_mu, read_node) =
tree.page_in(&low, self.cache.current_flush_epoch())?;
assert_eq!(read_node.object_id, node.object_id);
assert_eq!(node_mu.leaf.as_ref().unwrap().lo, low);
assert_eq!(read_low, low);
if let Some(hi) = &last_hi {
assert_eq!(hi, &node_mu.leaf.as_ref().unwrap().lo);
}
if let Some(hi) = &node_mu.leaf.as_ref().unwrap().hi {
last_hi = Some(hi.clone());
} else {
assert_eq!(hi_none_count, 0);
hi_none_count += 1;
}
}
// each tree should have exactly one leaf with no max hi key
assert_eq!(hi_none_count, 1);
}
log::debug!(
"{} leaves looking good after {} micros",
ever_seen.len(),
before.elapsed().as_micros()
);
Ok(())
}
pub fn stats(&self) -> Stats {
Stats { cache: self.cache.stats() }
}
pub fn size_on_disk(&self) -> io::Result {
use std::fs::read_dir;
fn recurse(mut dir: std::fs::ReadDir) -> io::Result {
dir.try_fold(0, |acc, file| {
let file = file?;
let size = match file.metadata()? {
data if data.is_dir() => recurse(read_dir(file.path())?)?,
data => data.len(),
};
Ok(acc + size)
})
}
recurse(read_dir(&self.cache.config.path)?)
}
/// Returns `true` if the database was
/// recovered from a previous process.
/// Note that database state is only
/// guaranteed to be present up to the
/// last call to `flush`! Otherwise state
/// is synced to disk periodically if the
/// `Config.sync_every_ms` configuration option
/// is set to `Some(number_of_ms_between_syncs)`
/// or if the IO buffer gets filled to
/// capacity before being rotated.
pub fn was_recovered(&self) -> bool {
self.was_recovered
}
pub fn open_with_config(config: &Config) -> io::Result> {
let (shutdown_tx, shutdown_rx) = mpsc::channel();
let (cache, indices, was_recovered) = ObjectCache::recover(config)?;
let _shutdown_dropper = Arc::new(ShutdownDropper {
shutdown_sender: Mutex::new(shutdown_tx),
cache: Mutex::new(cache.clone()),
});
let mut allocated_collection_ids = fnv::FnvHashSet::default();
let mut trees: HashMap> = indices
.into_iter()
.map(|(collection_id, index)| {
assert!(
allocated_collection_ids.insert(collection_id.0),
"allocated_collection_ids already contained {:?}",
collection_id
);
(
collection_id,
Tree::new(
collection_id,
cache.clone(),
index,
_shutdown_dropper.clone(),
),
)
})
.collect();
let collection_name_mapping =
trees.get(&NAME_MAPPING_COLLECTION_ID).unwrap().clone();
let default_tree = trees.get(&DEFAULT_COLLECTION_ID).unwrap().clone();
for kv_res in collection_name_mapping.iter() {
let (_collection_name, collection_id_buf) = kv_res.unwrap();
let collection_id = CollectionId(u64::from_le_bytes(
collection_id_buf.as_ref().try_into().unwrap(),
));
if trees.contains_key(&collection_id) {
continue;
}
// need to initialize tree leaf for empty collection
assert!(
allocated_collection_ids.insert(collection_id.0),
"allocated_collection_ids already contained {:?}",
collection_id
);
let initial_low_key = InlineArray::default();
let empty_node = cache.allocate_default_node(collection_id);
let index = Index::default();
assert!(index.insert(initial_low_key, empty_node).is_none());
let tree = Tree::new(
collection_id,
cache.clone(),
index,
_shutdown_dropper.clone(),
);
trees.insert(collection_id, tree);
}
let collection_id_allocator =
Arc::new(Allocator::from_allocated(&allocated_collection_ids));
assert_eq!(collection_name_mapping.len()? + 2, trees.len());
let ret = Db {
config: config.clone(),
cache: cache.clone(),
default_tree,
collection_name_mapping,
collection_id_allocator,
trees: Arc::new(Mutex::new(trees)),
_shutdown_dropper,
was_recovered,
};
#[cfg(feature = "for-internal-testing-only")]
ret.validate()?;
if let Some(flush_every_ms) = ret.cache.config.flush_every_ms {
let spawn_res = std::thread::Builder::new()
.name("sled_flusher".into())
.spawn(move || flusher(cache, shutdown_rx, flush_every_ms));
if let Err(e) = spawn_res {
return Err(io::Error::other(format!(
"unable to spawn flusher thread for sled database: {:?}",
e
)));
}
}
Ok(ret)
}
/// A database export method for all collections in the `Db`,
/// for use in sled version upgrades. Can be used in combination
/// with the `import` method below on a database running a later
/// version.
///
/// # Panics
///
/// Panics if any IO problems occur while trying
/// to perform the export.
///
/// # Examples
///
/// If you want to migrate from one version of sled
/// to another, you need to pull in both versions
/// by using version renaming:
///
/// `Cargo.toml`:
///
/// ```toml
/// [dependencies]
/// sled = "0.32"
/// old_sled = { version = "0.31", package = "sled" }
/// ```
///
/// and in your code, remember that old versions of
/// sled might have a different way to open them
/// than the current `sled::open` method:
///
/// ```
/// # use sled as old_sled;
/// # fn main() -> Result<(), Box> {
/// let old = old_sled::open("my_old_db_export")?;
///
/// // may be a different version of sled,
/// // the export type is version agnostic.
/// let new = sled::open("my_new_db_export")?;
///
/// let export = old.export();
/// new.import(export);
///
/// assert_eq!(old.checksum()?, new.checksum()?);
/// # drop(old);
/// # drop(new);
/// # let _ = std::fs::remove_dir_all("my_old_db_export");
/// # let _ = std::fs::remove_dir_all("my_new_db_export");
/// # Ok(()) }
/// ```
pub fn export(
&self,
) -> Vec<(
CollectionType,
CollectionName,
impl Iterator>> + '_,
)> {
let trees = self.trees.lock();
let mut ret = vec![];
for kv_res in self.collection_name_mapping.iter() {
let (collection_name, collection_id_buf) = kv_res.unwrap();
let collection_id = CollectionId(u64::from_le_bytes(
collection_id_buf.as_ref().try_into().unwrap(),
));
let tree = trees.get(&collection_id).unwrap().clone();
ret.push((
b"tree".to_vec(),
collection_name.to_vec(),
tree.iter().map(|kv_opt| {
let kv = kv_opt.unwrap();
vec![kv.0.to_vec(), kv.1.to_vec()]
}),
));
}
ret
}
/// Imports the collections from a previous database.
///
/// # Panics
///
/// Panics if any IO problems occur while trying
/// to perform the import.
///
/// # Examples
///
/// If you want to migrate from one version of sled
/// to another, you need to pull in both versions
/// by using version renaming:
///
/// `Cargo.toml`:
///
/// ```toml
/// [dependencies]
/// sled = "0.32"
/// old_sled = { version = "0.31", package = "sled" }
/// ```
///
/// and in your code, remember that old versions of
/// sled might have a different way to open them
/// than the current `sled::open` method:
///
/// ```
/// # use sled as old_sled;
/// # fn main() -> Result<(), Box> {
/// let old = old_sled::open("my_old_db_import")?;
///
/// // may be a different version of sled,
/// // the export type is version agnostic.
/// let new = sled::open("my_new_db_import")?;
///
/// let export = old.export();
/// new.import(export);
///
/// assert_eq!(old.checksum()?, new.checksum()?);
/// # drop(old);
/// # drop(new);
/// # let _ = std::fs::remove_dir_all("my_old_db_import");
/// # let _ = std::fs::remove_dir_all("my_new_db_import");
/// # Ok(()) }
/// ```
pub fn import(
&self,
export: Vec<(
CollectionType,
CollectionName,
impl Iterator>>,
)>,
) {
for (collection_type, collection_name, collection_iter) in export {
match collection_type {
ref t if t == b"tree" => {
let tree = self
.open_tree(collection_name)
.expect("failed to open new tree during import");
for mut kv in collection_iter {
let v = kv
.pop()
.expect("failed to get value from tree export");
let k = kv
.pop()
.expect("failed to get key from tree export");
let old = tree.insert(k, v).expect(
"failed to insert value during tree import",
);
assert!(
old.is_none(),
"import is overwriting existing data"
);
}
}
other => panic!("unknown collection type {:?}", other),
}
}
}
pub fn contains_tree>(&self, name: V) -> io::Result {
Ok(self.collection_name_mapping.get(name.as_ref())?.is_some())
}
pub fn drop_tree>(&self, name: V) -> io::Result {
let name_ref = name.as_ref();
let trees = self.trees.lock();
let tree = if let Some(collection_id_buf) =
self.collection_name_mapping.get(name_ref)?
{
let collection_id = CollectionId(u64::from_le_bytes(
collection_id_buf.as_ref().try_into().unwrap(),
));
trees.get(&collection_id).unwrap()
} else {
return Ok(false);
};
tree.clear()?;
self.collection_name_mapping.remove(name_ref)?;
Ok(true)
}
/// Open or create a new disk-backed [`Tree`] with its own keyspace,
/// accessible from the `Db` via the provided identifier.
pub fn open_tree>(
&self,
name: V,
) -> io::Result> {
let name_ref = name.as_ref();
let mut trees = self.trees.lock();
if let Some(collection_id_buf) =
self.collection_name_mapping.get(name_ref)?
{
let collection_id = CollectionId(u64::from_le_bytes(
collection_id_buf.as_ref().try_into().unwrap(),
));
let tree = trees.get(&collection_id).unwrap();
return Ok(tree.clone());
}
let collection_id =
CollectionId(self.collection_id_allocator.allocate());
let initial_low_key = InlineArray::default();
let empty_node = self.cache.allocate_default_node(collection_id);
let index = Index::default();
assert!(index.insert(initial_low_key, empty_node).is_none());
let tree = Tree::new(
collection_id,
self.cache.clone(),
index,
self._shutdown_dropper.clone(),
);
self.collection_name_mapping
.insert(name_ref, &collection_id.0.to_le_bytes())?;
trees.insert(collection_id, tree.clone());
Ok(tree)
}
}
/// These types provide the information that allows an entire
/// system to be exported and imported to facilitate
/// major upgrades. It is comprised entirely
/// of standard library types to be forward compatible.
/// NB this definitions are expensive to change, because
/// they impact the migration path.
type CollectionType = Vec;
type CollectionName = Vec;
================================================
FILE: src/event_verifier.rs
================================================
use std::collections::BTreeMap;
use std::sync::Mutex;
use crate::{FlushEpoch, ObjectId};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum State {
Unallocated,
Dirty,
CooperativelySerialized,
AddedToWriteBatch,
Flushed,
CleanPagedIn,
PagedOut,
}
impl State {
fn can_transition_within_epoch_to(&self, next: State) -> bool {
match (self, next) {
(State::Flushed, State::PagedOut) => true,
(State::Flushed, _) => false,
(State::AddedToWriteBatch, State::Flushed) => true,
(State::AddedToWriteBatch, _) => false,
(State::CleanPagedIn, State::AddedToWriteBatch) => false,
(State::CleanPagedIn, State::Flushed) => false,
(State::Dirty, State::AddedToWriteBatch) => true,
(State::CooperativelySerialized, State::AddedToWriteBatch) => true,
(State::CooperativelySerialized, _) => false,
(State::Unallocated, State::AddedToWriteBatch) => true,
(State::Unallocated, _) => false,
(State::Dirty, State::Dirty) => true,
(State::Dirty, State::CooperativelySerialized) => true,
(State::Dirty, State::Unallocated) => true,
(State::Dirty, _) => false,
(State::CleanPagedIn, State::Dirty) => true,
(State::CleanPagedIn, State::PagedOut) => true,
(State::CleanPagedIn, State::CleanPagedIn) => true,
(State::CleanPagedIn, State::Unallocated) => true,
(State::CleanPagedIn, State::CooperativelySerialized) => true,
(State::PagedOut, State::CleanPagedIn) => true,
(State::PagedOut, _) => false,
}
}
fn needs_flush(&self) -> bool {
match self {
State::CleanPagedIn => false,
State::Flushed => false,
State::PagedOut => false,
_ => true,
}
}
}
#[derive(Debug, Default)]
pub(crate) struct EventVerifier {
flush_model:
Mutex>>,
}
impl Drop for EventVerifier {
fn drop(&mut self) {
// assert that nothing is currently Dirty
let flush_model = self.flush_model.lock().unwrap();
for ((oid, _epoch), history) in flush_model.iter() {
if let Some((last_state, _at)) = history.last() {
assert_ne!(
*last_state,
State::Dirty,
"{oid:?} is Dirty when system shutting down"
);
}
}
}
}
impl EventVerifier {
pub(crate) fn mark(
&self,
object_id: ObjectId,
epoch: FlushEpoch,
state: State,
at: &'static str,
) {
if matches!(state, State::PagedOut) {
let dirty_epochs = self.dirty_epochs(object_id);
if !dirty_epochs.is_empty() {
println!("{object_id:?} was paged out while having dirty epochs {dirty_epochs:?}");
self.print_debug_history_for_object(object_id);
println!("{state:?} {epoch:?} {at}");
println!("invalid object state transition");
std::process::abort();
}
}
let mut flush_model = self.flush_model.lock().unwrap();
let history = flush_model.entry((object_id, epoch)).or_default();
if let Some((last_state, _at)) = history.last() {
if !last_state.can_transition_within_epoch_to(state) {
println!(
"object_id {object_id:?} performed \
illegal state transition from {last_state:?} \
to {state:?} at {at} in epoch {epoch:?}."
);
println!("history:");
history.push((state, at));
let active_epochs = flush_model.range(
(object_id, FlushEpoch::MIN)..=(object_id, FlushEpoch::MAX),
);
for ((_oid, epoch), history) in active_epochs {
for (last_state, at) in history {
println!("{last_state:?} {epoch:?} {at}");
}
}
println!("invalid object state transition");
std::process::abort();
}
}
history.push((state, at));
}
/// Returns the FlushEpochs for which this ObjectId has unflushed
/// dirty data for.
fn dirty_epochs(&self, object_id: ObjectId) -> Vec {
let mut dirty_epochs = vec![];
let flush_model = self.flush_model.lock().unwrap();
let active_epochs = flush_model
.range((object_id, FlushEpoch::MIN)..=(object_id, FlushEpoch::MAX));
for ((_oid, epoch), history) in active_epochs {
let (last_state, _at) = history.last().unwrap();
if last_state.needs_flush() {
dirty_epochs.push(*epoch);
}
}
dirty_epochs
}
pub(crate) fn print_debug_history_for_object(&self, object_id: ObjectId) {
let flush_model = self.flush_model.lock().unwrap();
println!("history for object {:?}:", object_id);
let active_epochs = flush_model
.range((object_id, FlushEpoch::MIN)..=(object_id, FlushEpoch::MAX));
for ((_oid, epoch), history) in active_epochs {
for (last_state, at) in history {
println!("{last_state:?} {epoch:?} {at}");
}
}
}
}
================================================
FILE: src/flush_epoch.rs
================================================
use std::num::NonZeroU64;
use std::sync::atomic::{AtomicPtr, AtomicU64, Ordering};
use std::sync::{Arc, Condvar, Mutex};
const SEAL_BIT: u64 = 1 << 63;
const SEAL_MASK: u64 = u64::MAX - SEAL_BIT;
const MIN_EPOCH: u64 = 2;
#[derive(
Debug,
Clone,
Copy,
serde::Serialize,
serde::Deserialize,
PartialOrd,
Ord,
PartialEq,
Eq,
Hash,
)]
pub struct FlushEpoch(NonZeroU64);
impl FlushEpoch {
pub const MIN: FlushEpoch = FlushEpoch(NonZeroU64::MIN);
#[allow(unused)]
pub const MAX: FlushEpoch = FlushEpoch(NonZeroU64::MAX);
pub fn increment(&self) -> FlushEpoch {
FlushEpoch(NonZeroU64::new(self.0.get() + 1).unwrap())
}
pub fn get(&self) -> u64 {
self.0.get()
}
}
impl concurrent_map::Minimum for FlushEpoch {
const MIN: FlushEpoch = FlushEpoch::MIN;
}
#[derive(Debug)]
pub(crate) struct FlushInvariants {
max_flushed_epoch: AtomicU64,
max_flushing_epoch: AtomicU64,
}
impl Default for FlushInvariants {
fn default() -> FlushInvariants {
FlushInvariants {
max_flushed_epoch: (MIN_EPOCH - 1).into(),
max_flushing_epoch: (MIN_EPOCH - 1).into(),
}
}
}
impl FlushInvariants {
pub(crate) fn mark_flushed_epoch(&self, epoch: FlushEpoch) {
let last = self.max_flushed_epoch.swap(epoch.get(), Ordering::SeqCst);
assert_eq!(last + 1, epoch.get());
}
pub(crate) fn mark_flushing_epoch(&self, epoch: FlushEpoch) {
let last = self.max_flushing_epoch.swap(epoch.get(), Ordering::SeqCst);
assert_eq!(last + 1, epoch.get());
}
}
#[derive(Clone, Debug)]
pub(crate) struct Completion {
mu: Arc>,
cv: Arc,
epoch: FlushEpoch,
}
impl Completion {
pub fn epoch(&self) -> FlushEpoch {
self.epoch
}
pub fn new(epoch: FlushEpoch) -> Completion {
Completion { mu: Default::default(), cv: Default::default(), epoch }
}
pub fn wait_for_complete(self) -> FlushEpoch {
let mut mu = self.mu.lock().unwrap();
while !*mu {
mu = self.cv.wait(mu).unwrap();
}
self.epoch
}
pub fn mark_complete(self) {
self.mark_complete_inner(false);
}
fn mark_complete_inner(&self, previously_sealed: bool) {
let mut mu = self.mu.lock().unwrap();
if !previously_sealed {
// TODO reevaluate - assert!(!*mu);
}
log::trace!("marking epoch {:?} as complete", self.epoch);
// it's possible for *mu to already be true due to this being
// immediately dropped in the check_in method when we see that
// the checked-in epoch has already been marked as sealed.
*mu = true;
drop(mu);
self.cv.notify_all();
}
#[cfg(test)]
pub fn is_complete(&self) -> bool {
*self.mu.lock().unwrap()
}
}
pub struct FlushEpochGuard<'a> {
tracker: &'a EpochTracker,
previously_sealed: bool,
}
impl Drop for FlushEpochGuard<'_> {
fn drop(&mut self) {
let rc = self.tracker.rc.fetch_sub(1, Ordering::SeqCst) - 1;
if rc & SEAL_MASK == 0 && (rc & SEAL_BIT) == SEAL_BIT {
crate::debug_delay();
self.tracker
.vacancy_notifier
.mark_complete_inner(self.previously_sealed);
}
}
}
impl FlushEpochGuard<'_> {
pub fn epoch(&self) -> FlushEpoch {
self.tracker.epoch
}
}
#[derive(Debug)]
pub(crate) struct EpochTracker {
epoch: FlushEpoch,
rc: AtomicU64,
vacancy_notifier: Completion,
previous_flush_complete: Completion,
}
#[derive(Clone, Debug)]
pub(crate) struct FlushEpochTracker {
active_ebr: ebr::Ebr, 16, 16>,
inner: Arc,
}
#[derive(Debug)]
pub(crate) struct FlushEpochInner {
counter: AtomicU64,
roll_mu: Mutex<()>,
current_active: AtomicPtr,
}
impl Drop for FlushEpochInner {
fn drop(&mut self) {
let vacancy_mu = self.roll_mu.lock().unwrap();
let old_ptr =
self.current_active.swap(std::ptr::null_mut(), Ordering::SeqCst);
if !old_ptr.is_null() {
//let old: &EpochTracker = &*old_ptr;
unsafe { drop(Box::from_raw(old_ptr)) }
}
drop(vacancy_mu);
}
}
impl Default for FlushEpochTracker {
fn default() -> FlushEpochTracker {
let last = Completion::new(FlushEpoch(NonZeroU64::new(1).unwrap()));
let current_active_ptr = Box::into_raw(Box::new(EpochTracker {
epoch: FlushEpoch(NonZeroU64::new(MIN_EPOCH).unwrap()),
rc: AtomicU64::new(0),
vacancy_notifier: Completion::new(FlushEpoch(
NonZeroU64::new(MIN_EPOCH).unwrap(),
)),
previous_flush_complete: last.clone(),
}));
last.mark_complete();
let current_active = AtomicPtr::new(current_active_ptr);
FlushEpochTracker {
inner: Arc::new(FlushEpochInner {
counter: AtomicU64::new(2),
roll_mu: Mutex::new(()),
current_active,
}),
active_ebr: ebr::Ebr::default(),
}
}
}
impl FlushEpochTracker {
/// Returns the epoch notifier for the previous epoch.
/// Intended to be passed to a flusher that can eventually
/// notify the flush-requesting thread.
pub fn roll_epoch_forward(&self) -> (Completion, Completion, Completion) {
let mut tracker_guard = self.active_ebr.pin();
let vacancy_mu = self.inner.roll_mu.lock().unwrap();
let flush_through = self.inner.counter.fetch_add(1, Ordering::SeqCst);
let flush_through_epoch =
FlushEpoch(NonZeroU64::new(flush_through).unwrap());
let new_epoch = flush_through_epoch.increment();
let forward_flush_notifier = Completion::new(flush_through_epoch);
let new_active = Box::into_raw(Box::new(EpochTracker {
epoch: new_epoch,
rc: AtomicU64::new(0),
vacancy_notifier: Completion::new(new_epoch),
previous_flush_complete: forward_flush_notifier.clone(),
}));
let old_ptr =
self.inner.current_active.swap(new_active, Ordering::SeqCst);
assert!(!old_ptr.is_null());
let (last_flush_complete_notifier, vacancy_notifier) = unsafe {
let old: &EpochTracker = &*old_ptr;
let last = old.rc.fetch_add(SEAL_BIT + 1, Ordering::SeqCst);
assert_eq!(
last & SEAL_BIT,
0,
"epoch {} double-sealed",
flush_through
);
// mark_complete_inner called via drop in a uniform way
//println!("dropping flush epoch guard for epoch {flush_through}");
drop(FlushEpochGuard { tracker: old, previously_sealed: true });
(old.previous_flush_complete.clone(), old.vacancy_notifier.clone())
};
tracker_guard.defer_drop(unsafe { Box::from_raw(old_ptr) });
drop(vacancy_mu);
(last_flush_complete_notifier, vacancy_notifier, forward_flush_notifier)
}
pub fn check_in<'a>(&self) -> FlushEpochGuard<'a> {
let _tracker_guard = self.active_ebr.pin();
loop {
let tracker: &'a EpochTracker =
unsafe { &*self.inner.current_active.load(Ordering::SeqCst) };
let rc = tracker.rc.fetch_add(1, Ordering::SeqCst);
let previously_sealed = rc & SEAL_BIT == SEAL_BIT;
let guard = FlushEpochGuard { tracker, previously_sealed };
if previously_sealed {
// the epoch is already closed, so we must drop the rc
// and possibly notify, which is handled in the guard's
// Drop impl.
drop(guard);
} else {
return guard;
}
}
}
pub fn manually_advance_epoch(&self) {
self.active_ebr.manually_advance_epoch();
}
pub fn current_flush_epoch(&self) -> FlushEpoch {
let current = self.inner.counter.load(Ordering::SeqCst);
FlushEpoch(NonZeroU64::new(current).unwrap())
}
}
#[test]
fn flush_epoch_basic_functionality() {
let epoch_tracker = FlushEpochTracker::default();
for expected in MIN_EPOCH..1_000_000 {
let g1 = epoch_tracker.check_in();
let g2 = epoch_tracker.check_in();
assert_eq!(g1.tracker.epoch.0.get(), expected);
assert_eq!(g2.tracker.epoch.0.get(), expected);
let previous_notifier = epoch_tracker.roll_epoch_forward().1;
assert!(!previous_notifier.is_complete());
drop(g1);
assert!(!previous_notifier.is_complete());
drop(g2);
assert_eq!(previous_notifier.wait_for_complete().0.get(), expected);
}
}
#[cfg(test)]
fn concurrent_flush_epoch_burn_in_inner() {
const N_THREADS: usize = 10;
const N_OPS_PER_THREAD: usize = 3000;
let fa = FlushEpochTracker::default();
let barrier = std::sync::Arc::new(std::sync::Barrier::new(21));
let pt = pagetable::PageTable::::default();
let rolls = || {
let fa = fa.clone();
let barrier = barrier.clone();
let pt = &pt;
move || {
barrier.wait();
for _ in 0..N_OPS_PER_THREAD {
let (previous, this, next) = fa.roll_epoch_forward();
let last_epoch = previous.wait_for_complete().0.get();
assert_eq!(0, pt.get(last_epoch).load(Ordering::Acquire));
let flush_through_epoch = this.wait_for_complete().0.get();
assert_eq!(
0,
pt.get(flush_through_epoch).load(Ordering::Acquire)
);
next.mark_complete();
}
}
};
let check_ins = || {
let fa = fa.clone();
let barrier = barrier.clone();
let pt = &pt;
move || {
barrier.wait();
for _ in 0..N_OPS_PER_THREAD {
let guard = fa.check_in();
let epoch = guard.epoch().0.get();
pt.get(epoch).fetch_add(1, Ordering::SeqCst);
std::thread::yield_now();
pt.get(epoch).fetch_sub(1, Ordering::SeqCst);
drop(guard);
}
}
};
std::thread::scope(|s| {
let mut threads = vec![];
for _ in 0..N_THREADS {
threads.push(s.spawn(rolls()));
threads.push(s.spawn(check_ins()));
}
barrier.wait();
for thread in threads.into_iter() {
thread.join().expect("a test thread crashed unexpectedly");
}
});
for i in 0..N_OPS_PER_THREAD * N_THREADS {
assert_eq!(0, pt.get(i as u64).load(Ordering::Acquire));
}
}
#[test]
fn concurrent_flush_epoch_burn_in() {
for _ in 0..128 {
concurrent_flush_epoch_burn_in_inner();
}
}
================================================
FILE: src/heap.rs
================================================
use std::fmt;
use std::fs;
use std::io::{self, Read};
use std::num::NonZeroU64;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::atomic::{AtomicPtr, AtomicU64, Ordering, fence};
use std::time::{Duration, Instant};
use ebr::{Ebr, Guard};
use fault_injection::{annotate, fallible, maybe};
use fnv::FnvHashSet;
use fs2::FileExt as _;
use parking_lot::{Mutex, RwLock};
use rayon::prelude::*;
use crate::object_location_mapper::{AllocatorStats, ObjectLocationMapper};
use crate::{CollectionId, Config, DeferredFree, MetadataStore, ObjectId};
const WARN: &str = "DO_NOT_PUT_YOUR_FILES_HERE";
pub(crate) const N_SLABS: usize = 78;
const FILE_TARGET_FILL_RATIO: u64 = 80;
const FILE_RESIZE_MARGIN: u64 = 115;
const SLAB_SIZES: [usize; N_SLABS] = [
64, // 0x40
80, // 0x50
96, // 0x60
112, // 0x70
128, // 0x80
160, // 0xa0
192, // 0xc0
224, // 0xe0
256, // 0x100
320, // 0x140
384, // 0x180
448, // 0x1c0
512, // 0x200
640, // 0x280
768, // 0x300
896, // 0x380
1024, // 0x400
1280, // 0x500
1536, // 0x600
1792, // 0x700
2048, // 0x800
2560, // 0xa00
3072, // 0xc00
3584, // 0xe00
4096, // 0x1000
5120, // 0x1400
6144, // 0x1800
7168, // 0x1c00
8192, // 0x2000
10240, // 0x2800
12288, // 0x3000
14336, // 0x3800
16384, // 0x4000
20480, // 0x5000
24576, // 0x6000
28672, // 0x7000
32768, // 0x8000
40960, // 0xa000
49152, // 0xc000
57344, // 0xe000
65536, // 0x10000
98304, // 0x1a000
131072, // 0x20000
163840, // 0x28000
196608,
262144,
393216,
524288,
786432,
1048576,
1572864,
2097152,
3145728,
4194304,
6291456,
8388608,
12582912,
16777216,
25165824,
33554432,
50331648,
67108864,
100663296,
134217728,
201326592,
268435456,
402653184,
536870912,
805306368,
1073741824,
1610612736,
2147483648,
3221225472,
4294967296,
6442450944,
8589934592,
12884901888,
17_179_869_184, // 17gb is max page size as-of now
];
#[derive(Default, Debug, Copy, Clone)]
pub struct WriteBatchStats {
pub heap_bytes_written: u64,
pub heap_files_written_to: u64,
/// Latency inclusive of fsync
pub heap_write_latency: Duration,
/// Latency for fsyncing files
pub heap_sync_latency: Duration,
pub metadata_bytes_written: u64,
pub metadata_write_latency: Duration,
pub truncated_files: u64,
pub truncated_bytes: u64,
pub truncate_latency: Duration,
}
#[derive(Default, Debug, Clone, Copy)]
pub struct HeapStats {
pub allocator: AllocatorStats,
pub write_batch_max: WriteBatchStats,
pub write_batch_sum: WriteBatchStats,
pub truncated_file_bytes: u64,
}
impl WriteBatchStats {
pub(crate) fn max(&self, other: &WriteBatchStats) -> WriteBatchStats {
WriteBatchStats {
heap_bytes_written: self
.heap_bytes_written
.max(other.heap_bytes_written),
heap_files_written_to: self
.heap_files_written_to
.max(other.heap_files_written_to),
heap_write_latency: self
.heap_write_latency
.max(other.heap_write_latency),
heap_sync_latency: self
.heap_sync_latency
.max(other.heap_sync_latency),
metadata_bytes_written: self
.metadata_bytes_written
.max(other.metadata_bytes_written),
metadata_write_latency: self
.metadata_write_latency
.max(other.metadata_write_latency),
truncated_files: self.truncated_files.max(other.truncated_files),
truncated_bytes: self.truncated_bytes.max(other.truncated_bytes),
truncate_latency: self.truncate_latency.max(other.truncate_latency),
}
}
pub(crate) fn sum(&self, other: &WriteBatchStats) -> WriteBatchStats {
use std::ops::Add;
WriteBatchStats {
heap_bytes_written: self
.heap_bytes_written
.add(other.heap_bytes_written),
heap_files_written_to: self
.heap_files_written_to
.add(other.heap_files_written_to),
heap_write_latency: self
.heap_write_latency
.add(other.heap_write_latency),
heap_sync_latency: self
.heap_sync_latency
.add(other.heap_sync_latency),
metadata_bytes_written: self
.metadata_bytes_written
.add(other.metadata_bytes_written),
metadata_write_latency: self
.metadata_write_latency
.add(other.metadata_write_latency),
truncated_files: self.truncated_files.add(other.truncated_files),
truncated_bytes: self.truncated_bytes.add(other.truncated_bytes),
truncate_latency: self.truncate_latency.add(other.truncate_latency),
}
}
}
const fn overhead_for_size(size: usize) -> usize {
if size + 5 <= u8::MAX as usize {
// crc32 + 1 byte frame
5
} else if size + 6 <= u16::MAX as usize {
// crc32 + 2 byte frame
6
} else if size + 8 <= u32::MAX as usize {
// crc32 + 4 byte frame
8
} else {
// crc32 + 8 byte frame
12
}
}
fn slab_for_size(size: usize) -> u8 {
let total_size = size + overhead_for_size(size);
for (idx, slab_size) in SLAB_SIZES.iter().enumerate() {
if *slab_size >= total_size {
return u8::try_from(idx).unwrap();
}
}
u8::MAX
}
pub use inline_array::InlineArray;
#[derive(Debug)]
pub struct ObjectRecovery {
pub object_id: ObjectId,
pub collection_id: CollectionId,
pub low_key: InlineArray,
}
pub struct HeapRecovery {
pub heap: Heap,
pub recovered_nodes: Vec,
pub was_recovered: bool,
}
enum PersistentSettings {
V1 { leaf_fanout: u64 },
}
impl PersistentSettings {
// NB: should only be called with a directory lock already exclusively acquired
fn verify_or_store>(
&self,
path: P,
_directory_lock: &std::fs::File,
) -> io::Result<()> {
let settings_path = path.as_ref().join("durability_cookie");
match std::fs::read(&settings_path) {
Ok(previous_bytes) => {
let previous =
PersistentSettings::deserialize(&previous_bytes)?;
self.check_compatibility(&previous)
}
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
std::fs::write(settings_path, self.serialize())
}
Err(e) => Err(e),
}
}
fn deserialize(buf: &[u8]) -> io::Result {
let mut cursor = buf;
let mut buf = [0_u8; 64];
cursor.read_exact(&mut buf)?;
let version = u16::from_le_bytes([buf[0], buf[1]]);
let crc_actual = (crc32fast::hash(&buf[0..60]) ^ 0xAF).to_le_bytes();
let crc_expected = &buf[60..];
if crc_actual != crc_expected {
return Err(io::Error::new(
io::ErrorKind::InvalidData,
"encountered corrupted settings cookie with mismatched CRC.",
));
}
match version {
1 => {
let leaf_fanout =
u64::from_le_bytes(buf[2..10].try_into().unwrap());
Ok(PersistentSettings::V1 { leaf_fanout })
}
_ => Err(io::Error::new(
io::ErrorKind::InvalidData,
"encountered unknown version number when reading settings cookie",
)),
}
}
fn check_compatibility(
&self,
other: &PersistentSettings,
) -> io::Result<()> {
use PersistentSettings::*;
match (self, other) {
(V1 { leaf_fanout: lf1 }, V1 { leaf_fanout: lf2 }) => {
if lf1 != lf2 {
Err(io::Error::new(
io::ErrorKind::Unsupported,
format!(
"sled was already opened with a LEAF_FANOUT const generic of {}, \
and this may not be changed after initial creation. Please use \
Db::import / Db::export to migrate, if you wish to change the \
system's format.",
lf2
),
))
} else {
Ok(())
}
}
}
}
fn serialize(&self) -> Vec {
// format: 64 bytes in total, with the last 4 being a LE crc32
// first 2 are LE version number
let mut buf = vec![];
match self {
PersistentSettings::V1 { leaf_fanout } => {
// LEAF_FANOUT: 8 bytes LE
let version: [u8; 2] = 1_u16.to_le_bytes();
buf.extend_from_slice(&version);
buf.extend_from_slice(&leaf_fanout.to_le_bytes());
}
}
// zero-pad the buffer
assert!(buf.len() < 60);
buf.resize(60, 0);
let hash: u32 = crc32fast::hash(&buf) ^ 0xAF;
let hash_bytes: [u8; 4] = hash.to_le_bytes();
buf.extend_from_slice(&hash_bytes);
// keep the buffer to 64 bytes for easy parsing over time.
assert_eq!(buf.len(), 64);
buf
}
}
#[derive(Clone, Copy, Debug, PartialEq)]
pub(crate) struct SlabAddress {
slab_id: u8,
slab_slot: [u8; 7],
}
impl SlabAddress {
pub(crate) fn from_slab_slot(slab: u8, slot: u64) -> SlabAddress {
let slot_bytes = slot.to_be_bytes();
assert_eq!(slot_bytes[0], 0);
SlabAddress {
slab_id: slab,
slab_slot: slot_bytes[1..].try_into().unwrap(),
}
}
#[inline]
pub const fn slab(&self) -> u8 {
self.slab_id
}
#[inline]
pub const fn slot(&self) -> u64 {
u64::from_be_bytes([
0,
self.slab_slot[0],
self.slab_slot[1],
self.slab_slot[2],
self.slab_slot[3],
self.slab_slot[4],
self.slab_slot[5],
self.slab_slot[6],
])
}
}
impl From for SlabAddress {
fn from(i: NonZeroU64) -> SlabAddress {
let i = i.get();
let bytes = i.to_be_bytes();
SlabAddress {
slab_id: bytes[0] - 1,
slab_slot: bytes[1..].try_into().unwrap(),
}
}
}
impl From for NonZeroU64 {
fn from(sa: SlabAddress) -> NonZeroU64 {
NonZeroU64::new(u64::from_be_bytes([
sa.slab_id + 1,
sa.slab_slot[0],
sa.slab_slot[1],
sa.slab_slot[2],
sa.slab_slot[3],
sa.slab_slot[4],
sa.slab_slot[5],
sa.slab_slot[6],
]))
.unwrap()
}
}
#[cfg(unix)]
mod sys_io {
use std::io;
use std::os::unix::fs::FileExt;
use super::*;
pub(super) fn read_exact_at(
file: &fs::File,
buf: &mut [u8],
offset: u64,
) -> io::Result<()> {
match maybe!(file.read_exact_at(buf, offset)) {
Ok(r) => Ok(r),
Err(e) => {
// FIXME BUG 3: failed to read 64 bytes at offset 192 from file with len 192
println!(
"failed to read {} bytes at offset {} from file with len {}",
buf.len(),
offset,
file.metadata().unwrap().len(),
);
let _ = dbg!(std::backtrace::Backtrace::force_capture());
Err(e)
}
}
}
pub(super) fn write_all_at(
file: &fs::File,
buf: &[u8],
offset: u64,
) -> io::Result<()> {
maybe!(file.write_all_at(buf, offset))
}
}
#[cfg(windows)]
mod sys_io {
use std::os::windows::fs::FileExt;
use super::*;
pub(super) fn read_exact_at(
file: &fs::File,
mut buf: &mut [u8],
mut offset: u64,
) -> io::Result<()> {
while !buf.is_empty() {
match maybe!(file.seek_read(buf, offset)) {
Ok(0) => break,
Ok(n) => {
let tmp = buf;
buf = &mut tmp[n..];
offset += n as u64;
}
Err(ref e) if e.kind() == io::ErrorKind::Interrupted => {}
Err(e) => return Err(annotate!(e)),
}
}
if !buf.is_empty() {
Err(annotate!(io::Error::new(
io::ErrorKind::UnexpectedEof,
"failed to fill whole buffer"
)))
} else {
Ok(())
}
}
pub(super) fn write_all_at(
file: &fs::File,
mut buf: &[u8],
mut offset: u64,
) -> io::Result<()> {
while !buf.is_empty() {
match maybe!(file.seek_write(buf, offset)) {
Ok(0) => {
return Err(annotate!(io::Error::new(
io::ErrorKind::WriteZero,
"failed to write whole buffer",
)));
}
Ok(n) => {
buf = &buf[n..];
offset += n as u64;
}
Err(ref e) if e.kind() == io::ErrorKind::Interrupted => {}
Err(e) => return Err(annotate!(e)),
}
}
Ok(())
}
}
#[derive(Debug)]
struct Slab {
file: fs::File,
slot_size: usize,
max_live_slot_since_last_truncation: AtomicU64,
}
impl Slab {
fn sync(&self) -> io::Result<()> {
self.file.sync_all()
}
fn read(
&self,
slot: u64,
_guard: &mut Guard<'_, DeferredFree, 16, 16>,
) -> io::Result> {
log::trace!("reading from slot {} in slab {}", slot, self.slot_size);
let mut data = Vec::with_capacity(self.slot_size);
unsafe {
data.set_len(self.slot_size);
}
let whence = self.slot_size as u64 * slot;
maybe!(sys_io::read_exact_at(&self.file, &mut data, whence))?;
let hash_actual: [u8; 4] =
(crc32fast::hash(&data[..self.slot_size - 4]) ^ 0xAF).to_le_bytes();
let hash_expected = &data[self.slot_size - 4..];
if hash_expected != hash_actual {
return Err(annotate!(io::Error::new(
io::ErrorKind::InvalidData,
"crc mismatch - data corruption detected"
)));
}
let len: usize = if self.slot_size <= u8::MAX as usize {
// crc32 + 1 byte frame
usize::from(data[self.slot_size - 5])
} else if self.slot_size <= u16::MAX as usize {
// crc32 + 2 byte frame
let mut size_bytes: [u8; 2] = [0; 2];
size_bytes
.copy_from_slice(&data[self.slot_size - 6..self.slot_size - 4]);
usize::from(u16::from_le_bytes(size_bytes))
} else if self.slot_size <= u32::MAX as usize {
// crc32 + 4 byte frame
let mut size_bytes: [u8; 4] = [0; 4];
size_bytes
.copy_from_slice(&data[self.slot_size - 8..self.slot_size - 4]);
usize::try_from(u32::from_le_bytes(size_bytes)).unwrap()
} else {
// crc32 + 8 byte frame
let mut size_bytes: [u8; 8] = [0; 8];
size_bytes.copy_from_slice(
&data[self.slot_size - 12..self.slot_size - 4],
);
usize::try_from(u64::from_le_bytes(size_bytes)).unwrap()
};
data.truncate(len);
Ok(data)
}
fn write(&self, slot: u64, mut data: Vec) -> io::Result<()> {
let len = data.len();
assert!(len + overhead_for_size(data.len()) <= self.slot_size);
data.resize(self.slot_size, 0);
if self.slot_size <= u8::MAX as usize {
// crc32 + 1 byte frame
data[self.slot_size - 5] = u8::try_from(len).unwrap();
} else if self.slot_size <= u16::MAX as usize {
// crc32 + 2 byte frame
let size_bytes: [u8; 2] = u16::try_from(len).unwrap().to_le_bytes();
data[self.slot_size - 6..self.slot_size - 4]
.copy_from_slice(&size_bytes);
} else if self.slot_size <= u32::MAX as usize {
// crc32 + 4 byte frame
let size_bytes: [u8; 4] = u32::try_from(len).unwrap().to_le_bytes();
data[self.slot_size - 8..self.slot_size - 4]
.copy_from_slice(&size_bytes);
} else {
// crc32 + 8 byte frame
let size_bytes: [u8; 8] = u64::try_from(len).unwrap().to_le_bytes();
data[self.slot_size - 12..self.slot_size - 4]
.copy_from_slice(&size_bytes);
}
let hash: [u8; 4] =
(crc32fast::hash(&data[..self.slot_size - 4]) ^ 0xAF).to_le_bytes();
data[self.slot_size - 4..].copy_from_slice(&hash);
let whence = self.slot_size as u64 * slot;
log::trace!("writing to slot {} in slab {}", slot, self.slot_size);
sys_io::write_all_at(&self.file, &data, whence)
}
}
fn set_error(
global_error: &AtomicPtr<(io::ErrorKind, String)>,
error: &io::Error,
) {
let kind = error.kind();
let reason = error.to_string();
let boxed = Box::new((kind, reason));
let ptr = Box::into_raw(boxed);
if global_error
.compare_exchange(
std::ptr::null_mut(),
ptr,
Ordering::SeqCst,
Ordering::SeqCst,
)
.is_err()
{
// global fatal error already installed, drop this one
unsafe {
drop(Box::from_raw(ptr));
}
}
}
#[derive(Debug)]
pub enum Update {
Store {
object_id: ObjectId,
collection_id: CollectionId,
low_key: InlineArray,
data: Vec,
},
Free {
object_id: ObjectId,
collection_id: CollectionId,
},
}
impl Update {
#[allow(unused)]
pub(crate) fn object_id(&self) -> ObjectId {
match self {
Update::Store { object_id, .. }
| Update::Free { object_id, .. } => *object_id,
}
}
}
#[derive(Debug, PartialOrd, Ord, PartialEq, Eq)]
pub enum UpdateMetadata {
Store {
object_id: ObjectId,
collection_id: CollectionId,
low_key: InlineArray,
location: NonZeroU64,
},
Free {
object_id: ObjectId,
collection_id: CollectionId,
},
}
impl UpdateMetadata {
pub fn object_id(&self) -> ObjectId {
match self {
UpdateMetadata::Store { object_id, .. }
| UpdateMetadata::Free { object_id, .. } => *object_id,
}
}
}
#[derive(Debug, Default, Clone, Copy)]
struct WriteBatchStatTracker {
sum: WriteBatchStats,
max: WriteBatchStats,
}
#[derive(Clone)]
pub struct Heap {
path: PathBuf,
slabs: Arc<[Slab; N_SLABS]>,
table: ObjectLocationMapper,
metadata_store: Arc>,
free_ebr: Ebr,
global_error: Arc>,
#[allow(unused)]
directory_lock: Arc,
stats: Arc>,
truncated_file_bytes: Arc,
}
impl fmt::Debug for Heap {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("Heap")
.field("path", &self.path)
.field("stats", &self.stats())
.finish()
}
}
impl Heap {
pub fn recover(
leaf_fanout: usize,
config: &Config,
) -> io::Result {
let path = &config.path;
log::trace!("recovering Heap at {:?}", path);
let slabs_dir = path.join("slabs");
// TODO NOCOMMIT
let sync_status = std::process::Command::new("sync")
.status()
.map(|status| status.success());
if !matches!(sync_status, Ok(true)) {
log::warn!(
"sync command before recovery failed: {:?}",
sync_status
);
}
// initialize directories if not present
let mut was_recovered = true;
for p in [path, &slabs_dir] {
if let Err(e) = fs::read_dir(p) {
if e.kind() == io::ErrorKind::NotFound {
fallible!(fs::create_dir_all(p));
was_recovered = false;
continue;
}
}
maybe!(fs::File::open(p).and_then(|f| f.sync_all()))?;
}
let _ = fs::File::create(path.join(WARN));
let mut file_lock_opts = fs::OpenOptions::new();
file_lock_opts.create(false).read(false).write(false);
let directory_lock = fallible!(fs::File::open(path));
fallible!(directory_lock.try_lock_exclusive());
maybe!(fs::File::open(&slabs_dir).and_then(|f| f.sync_all()))?;
maybe!(directory_lock.sync_all())?;
let persistent_settings =
PersistentSettings::V1 { leaf_fanout: leaf_fanout as u64 };
persistent_settings.verify_or_store(path, &directory_lock)?;
let (metadata_store, recovered_metadata) =
MetadataStore::recover(path.join("metadata"))?;
let table = ObjectLocationMapper::new(
&recovered_metadata,
config.target_heap_file_fill_ratio,
);
let mut recovered_nodes =
Vec::::with_capacity(recovered_metadata.len());
for update_metadata in recovered_metadata {
match update_metadata {
UpdateMetadata::Store {
object_id,
collection_id,
location: _,
low_key,
} => {
recovered_nodes.push(ObjectRecovery {
object_id,
collection_id,
low_key,
});
}
UpdateMetadata::Free { .. } => {
unreachable!()
}
}
}
let mut slabs = vec![];
let mut slab_opts = fs::OpenOptions::new();
slab_opts.create(true).read(true).write(true);
for slot_size in &SLAB_SIZES {
let slab_path = slabs_dir.join(format!("{}", slot_size));
let file = fallible!(slab_opts.open(slab_path));
slabs.push(Slab {
slot_size: *slot_size,
file,
max_live_slot_since_last_truncation: AtomicU64::new(0),
})
}
maybe!(fs::File::open(&slabs_dir).and_then(|f| f.sync_all()))?;
log::debug!("recovery of Heap at {:?} complete", path);
Ok(HeapRecovery {
heap: Heap {
slabs: Arc::new(slabs.try_into().unwrap()),
path: path.into(),
table,
global_error: metadata_store.get_global_error_arc(),
metadata_store: Arc::new(Mutex::new(metadata_store)),
directory_lock: Arc::new(directory_lock),
free_ebr: Ebr::default(),
truncated_file_bytes: Arc::default(),
stats: Arc::default(),
},
recovered_nodes,
was_recovered,
})
}
pub fn get_global_error_arc(
&self,
) -> Arc> {
self.global_error.clone()
}
fn check_error(&self) -> io::Result<()> {
let err_ptr: *const (io::ErrorKind, String) =
self.global_error.load(Ordering::Acquire);
if err_ptr.is_null() {
Ok(())
} else {
let deref: &(io::ErrorKind, String) = unsafe { &*err_ptr };
Err(io::Error::new(deref.0, deref.1.clone()))
}
}
fn set_error(&self, error: &io::Error) {
set_error(&self.global_error, error);
}
pub fn manually_advance_epoch(&self) {
self.free_ebr.manually_advance_epoch();
}
pub fn stats(&self) -> HeapStats {
let truncated_file_bytes =
self.truncated_file_bytes.load(Ordering::Acquire);
let stats = self.stats.read();
HeapStats {
truncated_file_bytes,
allocator: self.table.stats(),
write_batch_max: stats.max,
write_batch_sum: stats.sum,
}
}
pub fn read(&self, object_id: ObjectId) -> Option>> {
if let Err(e) = self.check_error() {
return Some(Err(e));
}
let mut guard = self.free_ebr.pin();
let slab_address = self.table.get_location_for_object(object_id)?;
let slab = &self.slabs[usize::from(slab_address.slab_id)];
match slab.read(slab_address.slot(), &mut guard) {
Ok(bytes) => Some(Ok(bytes)),
Err(e) => {
let annotated = annotate!(e);
self.set_error(&annotated);
Some(Err(annotated))
}
}
}
pub fn write_batch(
&self,
batch: Vec,
) -> io::Result {
self.check_error()?;
let metadata_store = self.metadata_store.try_lock()
.expect("write_batch called concurrently! major correctness assumpiton violated");
let mut guard = self.free_ebr.pin();
let slabs = &self.slabs;
let table = &self.table;
let heap_bytes_written = AtomicU64::new(0);
let heap_files_used_0_to_63 = AtomicU64::new(0);
let heap_files_used_64_to_127 = AtomicU64::new(0);
let map_closure = |update: Update| match update {
Update::Store { object_id, collection_id, low_key, data } => {
let data_len = data.len();
let slab_id = slab_for_size(data_len);
let slab = &slabs[usize::from(slab_id)];
let new_location = table.allocate_slab_slot(slab_id);
let new_location_nzu: NonZeroU64 = new_location.into();
let complete_durability_pipeline =
maybe!(slab.write(new_location.slot(), data));
if let Err(e) = complete_durability_pipeline {
// can immediately free slot as the
table.free_slab_slot(new_location);
return Err(e);
}
// record stats
heap_bytes_written
.fetch_add(data_len as u64, Ordering::Release);
if slab_id < 64 {
let slab_bit = 0b1 << slab_id;
heap_files_used_0_to_63
.fetch_or(slab_bit, Ordering::Release);
} else {
assert!(slab_id < 128);
let slab_bit = 0b1 << (slab_id - 64);
heap_files_used_64_to_127
.fetch_or(slab_bit, Ordering::Release);
}
Ok(UpdateMetadata::Store {
object_id,
collection_id,
low_key,
location: new_location_nzu,
})
}
Update::Free { object_id, collection_id } => {
Ok(UpdateMetadata::Free { object_id, collection_id })
}
};
let before_heap_write = Instant::now();
let metadata_batch_res: io::Result> =
batch.into_par_iter().map(map_closure).collect();
let before_heap_sync = Instant::now();
fence(Ordering::SeqCst);
for slab_id in 0..N_SLABS {
let dirty = if slab_id < 64 {
let slab_bit = 0b1 << slab_id;
heap_files_used_0_to_63.load(Ordering::Acquire) & slab_bit
== slab_bit
} else {
let slab_bit = 0b1 << (slab_id - 64);
heap_files_used_64_to_127.load(Ordering::Acquire) & slab_bit
== slab_bit
};
if dirty {
self.slabs[slab_id].sync()?;
}
}
let heap_sync_latency = before_heap_sync.elapsed();
let heap_write_latency = before_heap_write.elapsed();
let metadata_batch = match metadata_batch_res {
Ok(mut mb) => {
// TODO evaluate impact : cost ratio of this sort
mb.par_sort_unstable();
mb
}
Err(e) => {
self.set_error(&e);
return Err(e);
}
};
// make metadata durable
let before_metadata_write = Instant::now();
let metadata_bytes_written =
match metadata_store.write_batch(&metadata_batch) {
Ok(metadata_bytes_written) => metadata_bytes_written,
Err(e) => {
self.set_error(&e);
return Err(e);
}
};
let metadata_write_latency = before_metadata_write.elapsed();
// reclaim previous disk locations for future writes
for update_metadata in metadata_batch {
let last_address_opt = match update_metadata {
UpdateMetadata::Store { object_id, location, .. } => {
self.table.insert(object_id, SlabAddress::from(location))
}
UpdateMetadata::Free { object_id, .. } => {
guard.defer_drop(DeferredFree {
allocator: self.table.clone_object_id_allocator_arc(),
freed_slot: object_id.0.get(),
});
self.table.remove(object_id)
}
};
if let Some(last_address) = last_address_opt {
guard.defer_drop(DeferredFree {
allocator: self
.table
.clone_slab_allocator_arc(last_address.slab_id),
freed_slot: last_address.slot(),
});
}
}
// truncate files that are now too fragmented
let before_truncate = Instant::now();
let mut truncated_files = 0;
let mut truncated_bytes = 0;
for (i, max_live_slot) in self.table.get_max_allocated_per_slab() {
let slab = &self.slabs[i];
let last_max = slab
.max_live_slot_since_last_truncation
.fetch_max(max_live_slot, Ordering::SeqCst);
let max_since_last_truncation = last_max.max(max_live_slot);
let currently_occupied_bytes =
(max_live_slot + 1) * slab.slot_size as u64;
let max_occupied_bytes =
(max_since_last_truncation + 1) * slab.slot_size as u64;
let ratio = currently_occupied_bytes * 100 / max_occupied_bytes;
if ratio < FILE_TARGET_FILL_RATIO {
let target_len = if max_live_slot < 16 {
currently_occupied_bytes
} else {
currently_occupied_bytes * FILE_RESIZE_MARGIN / 100
};
assert!(target_len < max_occupied_bytes);
assert!(
target_len >= currently_occupied_bytes,
"target_len of {} is above actual occupied len of {}",
target_len,
currently_occupied_bytes
);
if cfg!(not(feature = "monotonic-behavior")) {
if slab.file.set_len(target_len).is_ok() {
slab.max_live_slot_since_last_truncation
.store(max_live_slot, Ordering::SeqCst);
let file_truncated_bytes =
currently_occupied_bytes.saturating_sub(target_len);
self.truncated_file_bytes
.fetch_add(file_truncated_bytes, Ordering::Release);
truncated_files += 1;
truncated_bytes += file_truncated_bytes;
} else {
// TODO surface stats
}
}
}
}
let truncate_latency = before_truncate.elapsed();
let heap_files_written_to = u64::from(
heap_files_used_0_to_63.load(Ordering::Acquire).count_ones()
+ heap_files_used_64_to_127
.load(Ordering::Acquire)
.count_ones(),
);
let stats = WriteBatchStats {
heap_bytes_written: heap_bytes_written.load(Ordering::Acquire),
heap_files_written_to,
heap_write_latency,
heap_sync_latency,
metadata_bytes_written,
metadata_write_latency,
truncated_files,
truncated_bytes,
truncate_latency,
};
{
let mut stats_tracker = self.stats.write();
stats_tracker.max = stats_tracker.max.max(&stats);
stats_tracker.sum = stats_tracker.sum.sum(&stats);
}
Ok(stats)
}
pub fn heap_object_id_pin(&self) -> ebr::Guard<'_, DeferredFree, 16, 16> {
self.free_ebr.pin()
}
pub fn allocate_object_id(&self) -> ObjectId {
self.table.allocate_object_id()
}
pub(crate) fn objects_to_defrag(&self) -> FnvHashSet {
self.table.objects_to_defrag()
}
}
================================================
FILE: src/id_allocator.rs
================================================
use std::collections::BTreeSet;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::Arc;
use crossbeam_queue::SegQueue;
use fnv::FnvHashSet;
use parking_lot::Mutex;
#[derive(Default, Debug)]
struct FreeSetAndTip {
free_set: BTreeSet,
next_to_allocate: u64,
}
#[derive(Default, Debug)]
pub struct Allocator {
free_and_pending: Mutex,
/// Flat combining.
///
/// A lock free queue of recently freed ids which uses when there is contention on `free_and_pending`.
free_queue: SegQueue,
allocation_counter: AtomicU64,
free_counter: AtomicU64,
}
impl Allocator {
/// Intended primarily for heap slab slot allocators when performing GC.
///
/// If the slab is fragmented beyond the desired fill ratio, this returns
/// the range of offsets (min inclusive, max exclusive) that may be copied
/// into earlier free slots if they are currently occupied in order to
/// achieve the desired fragmentation ratio.
pub fn fragmentation_cutoff(
&self,
desired_ratio: f32,
) -> Option<(u64, u64)> {
let mut free_and_tip = self.free_and_pending.lock();
let next_to_allocate = free_and_tip.next_to_allocate;
if next_to_allocate == 0 {
return None;
}
while let Some(free_id) = self.free_queue.pop() {
free_and_tip.free_set.insert(free_id);
}
let live_objects =
next_to_allocate - free_and_tip.free_set.len() as u64;
let actual_ratio = live_objects as f32 / next_to_allocate as f32;
log::trace!(
"fragmented_slots actual ratio: {actual_ratio}, free len: {}",
free_and_tip.free_set.len()
);
if desired_ratio <= actual_ratio {
return None;
}
// calculate theoretical cut-off point, return everything past that
let min = (live_objects as f32 / desired_ratio) as u64;
let max = next_to_allocate;
assert!(min < max);
Some((min, max))
}
pub fn from_allocated(allocated: &FnvHashSet) -> Allocator {
let mut heap = BTreeSet::::default();
let max = allocated.iter().copied().max();
for i in 0..max.unwrap_or(0) {
if !allocated.contains(&i) {
heap.insert(i);
}
}
let free_and_pending = Mutex::new(FreeSetAndTip {
free_set: heap,
next_to_allocate: max.map(|m| m + 1).unwrap_or(0),
});
Allocator {
free_and_pending,
free_queue: SegQueue::default(),
allocation_counter: 0.into(),
free_counter: 0.into(),
}
}
pub fn max_allocated(&self) -> Option {
let next = self.free_and_pending.lock().next_to_allocate;
if next == 0 {
None
} else {
Some(next - 1)
}
}
pub fn allocate(&self) -> u64 {
self.allocation_counter.fetch_add(1, Ordering::Relaxed);
let mut free_and_tip = self.free_and_pending.lock();
while let Some(free_id) = self.free_queue.pop() {
free_and_tip.free_set.insert(free_id);
}
compact(&mut free_and_tip);
let pop_attempt = free_and_tip.free_set.pop_first();
if let Some(id) = pop_attempt {
id
} else {
let ret = free_and_tip.next_to_allocate;
free_and_tip.next_to_allocate += 1;
ret
}
}
pub fn free(&self, id: u64) {
if cfg!(not(feature = "monotonic-behavior")) {
self.free_counter.fetch_add(1, Ordering::Relaxed);
if let Some(mut free) = self.free_and_pending.try_lock() {
while let Some(free_id) = self.free_queue.pop() {
free.free_set.insert(free_id);
}
free.free_set.insert(id);
compact(&mut free);
} else {
self.free_queue.push(id);
}
}
}
/// Returns the counters for allocated, free
pub fn counters(&self) -> (u64, u64) {
(
self.allocation_counter.load(Ordering::Acquire),
self.free_counter.load(Ordering::Acquire),
)
}
}
fn compact(free: &mut FreeSetAndTip) {
let next = &mut free.next_to_allocate;
while *next > 1 && free.free_set.contains(&(*next - 1)) {
free.free_set.remove(&(*next - 1));
*next -= 1;
}
}
pub struct DeferredFree {
pub allocator: Arc,
pub freed_slot: u64,
}
impl Drop for DeferredFree {
fn drop(&mut self) {
self.allocator.free(self.freed_slot)
}
}
================================================
FILE: src/leaf.rs
================================================
use crate::*;
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub(crate) struct Leaf {
pub lo: InlineArray,
pub hi: Option,
pub prefix_length: usize,
data: stack_map::StackMap,
pub in_memory_size: usize,
pub mutation_count: u64,
#[serde(skip)]
pub dirty_flush_epoch: Option,
#[serde(skip)]
pub page_out_on_flush: Option,
#[serde(skip)]
pub deleted: Option,
#[serde(skip)]
pub max_unflushed_epoch: Option,
}
impl Leaf {
pub(crate) fn empty() -> Leaf {
Leaf {
lo: InlineArray::default(),
hi: None,
prefix_length: 0,
data: stack_map::StackMap::default(),
// this does not need to be marked as dirty until it actually
// receives inserted data
dirty_flush_epoch: None,
in_memory_size: std::mem::size_of::>(),
mutation_count: 0,
page_out_on_flush: None,
deleted: None,
max_unflushed_epoch: None,
}
}
pub(crate) const fn is_empty(&self) -> bool {
self.data.is_empty()
}
pub(crate) fn set_dirty_epoch(&mut self, epoch: FlushEpoch) {
assert!(self.deleted.is_none());
if let Some(current_epoch) = self.dirty_flush_epoch {
assert!(current_epoch <= epoch);
}
if self.page_out_on_flush < Some(epoch) {
self.page_out_on_flush = None;
}
self.dirty_flush_epoch = Some(epoch);
}
fn prefix(&self) -> &[u8] {
assert!(self.deleted.is_none());
&self.lo[..self.prefix_length]
}
pub(crate) fn get(&self, key: &[u8]) -> Option<&InlineArray> {
assert!(self.deleted.is_none());
assert!(key.starts_with(self.prefix()));
let prefixed_key = &key[self.prefix_length..];
self.data.get(prefixed_key)
}
pub(crate) fn insert(
&mut self,
key: InlineArray,
value: InlineArray,
) -> Option {
assert!(self.deleted.is_none());
assert!(key.starts_with(self.prefix()));
let prefixed_key = key[self.prefix_length..].into();
self.data.insert(prefixed_key, value)
}
pub(crate) fn remove(&mut self, key: &[u8]) -> Option {
assert!(self.deleted.is_none());
let prefix = self.prefix();
assert!(key.starts_with(prefix));
let partial_key = &key[self.prefix_length..];
self.data.remove(partial_key)
}
pub(crate) fn merge_from(&mut self, other: &mut Self) {
assert!(self.is_empty());
self.hi = other.hi.clone();
let new_prefix_len = if let Some(hi) = &self.hi {
self.lo.iter().zip(hi.iter()).take_while(|(l, r)| l == r).count()
} else {
0
};
assert_eq!(self.lo[..new_prefix_len], other.lo[..new_prefix_len]);
// self.prefix_length is not read because it's expected to be
// initialized here.
self.prefix_length = new_prefix_len;
if self.prefix() == other.prefix() {
self.data = std::mem::take(&mut other.data);
return;
}
assert!(
self.prefix_length < other.prefix_length,
"self: {:?} other: {:?}",
self,
other
);
let unshifted_key_amount = other.prefix_length - self.prefix_length;
let unshifted_prefix = &other.lo
[other.prefix_length - unshifted_key_amount..other.prefix_length];
for (k, v) in other.data.iter() {
let mut unshifted_key =
Vec::with_capacity(unshifted_prefix.len() + k.len());
unshifted_key.extend_from_slice(unshifted_prefix);
unshifted_key.extend_from_slice(k);
self.data.insert(unshifted_key.into(), v.clone());
}
assert_eq!(other.data.len(), self.data.len());
#[cfg(feature = "for-internal-testing-only")]
assert_eq!(
self.iter().collect::>(),
other.iter().collect::>(),
"self: {:#?} \n other: {:#?}\n",
self,
other
);
}
pub(crate) fn iter(
&self,
) -> impl Iterator {
let prefix = self.prefix();
self.data.iter().map(|(k, v)| {
let mut unshifted_key = Vec::with_capacity(prefix.len() + k.len());
unshifted_key.extend_from_slice(prefix);
unshifted_key.extend_from_slice(k);
(unshifted_key.into(), v.clone())
})
}
pub(crate) fn serialize(&self, zstd_compression_level: i32) -> Vec {
let mut ret = vec![];
let mut zstd_enc =
zstd::stream::Encoder::new(&mut ret, zstd_compression_level)
.unwrap();
bincode::serialize_into(&mut zstd_enc, self).unwrap();
zstd_enc.finish().unwrap();
ret
}
pub(crate) fn deserialize(
buf: &[u8],
) -> std::io::Result>> {
let zstd_decoded = zstd::stream::decode_all(buf).unwrap();
let mut leaf: Box> =
bincode::deserialize(&zstd_decoded).unwrap();
// use decompressed buffer length as a cheap proxy for in-memory size for now
leaf.in_memory_size = zstd_decoded.len();
Ok(leaf)
}
fn set_in_memory_size(&mut self) {
self.in_memory_size = std::mem::size_of::>()
+ self.hi.as_ref().map(|h| h.len()).unwrap_or(0)
+ self.lo.len()
+ self.data.iter().map(|(k, v)| k.len() + v.len()).sum::();
}
pub(crate) fn split_if_full(
&mut self,
new_epoch: FlushEpoch,
allocator: &ObjectCache,
collection_id: CollectionId,
) -> Option<(InlineArray, Object)> {
if self.data.is_full() {
let original_len = self.data.len();
let old_prefix_len = self.prefix_length;
// split
let split_offset = if self.lo.is_empty() {
// split left-most shard almost at the beginning for
// optimizing downward-growing workloads
1
} else if self.hi.is_none() {
// split right-most shard almost at the end for
// optimizing upward-growing workloads
self.data.len() - 2
} else {
self.data.len() / 2
};
let data = self.data.split_off(split_offset);
let left_max = &self.data.last().unwrap().0;
let right_min = &data.first().unwrap().0;
// suffix truncation attempts to shrink the split key
// so that shorter keys bubble up into the index
let splitpoint_length = right_min
.iter()
.zip(left_max.iter())
.take_while(|(a, b)| a == b)
.count()
+ 1;
let mut split_vec =
Vec::with_capacity(self.prefix_length + splitpoint_length);
split_vec.extend_from_slice(self.prefix());
split_vec.extend_from_slice(&right_min[..splitpoint_length]);
let split_key = InlineArray::from(split_vec);
let rhs_id = allocator.allocate_object_id(new_epoch);
log::trace!(
"split leaf {:?} at split key: {:?} into new {:?} at {:?}",
self.lo,
split_key,
rhs_id,
new_epoch,
);
let mut rhs = Leaf {
dirty_flush_epoch: Some(new_epoch),
hi: self.hi.clone(),
lo: split_key.clone(),
prefix_length: 0,
in_memory_size: 0,
data,
mutation_count: 0,
page_out_on_flush: None,
deleted: None,
max_unflushed_epoch: None,
};
rhs.shorten_keys_after_split(old_prefix_len);
rhs.set_in_memory_size();
self.hi = Some(split_key.clone());
self.shorten_keys_after_split(old_prefix_len);
self.set_in_memory_size();
assert_eq!(self.hi.as_ref().unwrap(), &split_key);
assert_eq!(rhs.lo, &split_key);
assert_eq!(rhs.data.len() + self.data.len(), original_len);
let rhs_node = Object {
object_id: rhs_id,
collection_id,
low_key: split_key.clone(),
inner: Arc::new(RwLock::new(CacheBox {
leaf: Some(Box::new(rhs)),
logged_index: BTreeMap::default(),
})),
};
return Some((split_key, rhs_node));
}
None
}
pub(crate) fn shorten_keys_after_split(&mut self, old_prefix_len: usize) {
let Some(hi) = self.hi.as_ref() else { return };
let new_prefix_len =
self.lo.iter().zip(hi.iter()).take_while(|(l, r)| l == r).count();
assert_eq!(self.lo[..new_prefix_len], hi[..new_prefix_len]);
// self.prefix_length is not read because it's expected to be
// initialized here.
self.prefix_length = new_prefix_len;
if new_prefix_len == old_prefix_len {
return;
}
assert!(
new_prefix_len > old_prefix_len,
"expected new prefix length of {} to be greater than the pre-split prefix length of {} for node {:?}",
new_prefix_len,
old_prefix_len,
self
);
let key_shift = new_prefix_len - old_prefix_len;
for (k, v) in std::mem::take(&mut self.data).iter() {
self.data.insert(k[key_shift..].into(), v.clone());
}
}
}
================================================
FILE: src/lib.rs
================================================
// 1.0 blockers
//
// bugs
// * page-out needs to be deferred until after any flush of the dirty epoch
// * need to remove max_unflushed_epoch after flushing it
// * can't send reliable page-out request backwards from 7->6
// * re-locking every mutex in a writebatch feels bad
// * need to signal stability status forward
// * maybe we already are
// * can make dirty_flush_epoch atomic and CAS it to 0 after flush
// * can change dirty_flush_epoch to unflushed_epoch
// * can always set mutation_count to max dirty flush epoch
// * this feels nice, we can lazily update a global stable flushed counter
// * can get rid of dirty_flush_epoch and page_out_on_flush?
// * or at least dirty_flush_epoch
// * dirty_flush_epoch really means "hasn't yet been cooperatively serialized @ F.E."
// * interesting metrics:
// * whether dirty for some epoch
// * whether cooperatively serialized for some epoch
// * whether fully flushed for some epoch
// * clean -> dirty -> {maybe coop} -> flushed
// * for page-out, we only care if it's stable or if we need to add it to
// a page-out priority queue
// * page-out doesn't seem to happen as expected
//
// reliability
// TODO make all writes wrapped in a Tearable wrapper that splits writes
// and can possibly crash based on a counter.
// TODO test concurrent drop_tree when other threads are still using it
// TODO list trees test for recovering empty collections
// TODO set explicit max key and value sizes w/ corresponding heap
// TODO add failpoints to writepath
//
// performance
// TODO handle prefix encoding
// TODO (minor) remove cache access for removed node in merge function
// TODO index+log hybrid - tinylsm key -> object location
//
// features
// TODO multi-collection batch
//
// misc
// TODO skim inlining output of RUSTFLAGS="-Cremark=all -Cdebuginfo=1"
//
// ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1.0 cutoff ~~~~~~~~~~~~~~~~~~~~~~~~~~~
//
// post-1.0 improvements
//
// reliability
// TODO bug hiding: if the crash_iter test panics, the test doesn't fail as expected
// TODO event log assertion for testing heap location bidirectional referential integrity,
// particularly in the object location mapper.
// TODO ensure nothing "from the future" gets copied into earlier epochs during GC
// TODO collection_id on page_in checks - it needs to be pinned w/ heap's EBR?
// TODO put aborts behind feature flags for hard crashes
// TODO re-enable transaction tests in test_tree.rs
//
// performance
// TODO force writers to flush when some number of dirty epochs have built up
// TODO serialize flush batch in parallel
// TODO concurrent serialization of NotYetSerialized dirty objects
// TODO make the Arc`,
//! but with several additional capabilities for
//! assisting creators of stateful systems.
//!
//! It is fully thread-safe, and all operations are
//! atomic. Multiple `Tree`s with isolated keyspaces
//! are supported with the
//! [`Db::open_tree`](struct.Db.html#method.open_tree) method.
//!
//! `sled` is built by experienced database engineers
//! who think users should spend less time tuning and
//! working against high-friction APIs. Expect
//! significant ergonomic and performance improvements
//! over time. Most surprises are bugs, so please
//! [let us know](mailto:tylerneely@gmail.com?subject=sled%20sucks!!!)
//! if something is high friction.
//!
//! # Examples
//!
//! ```
//! # let _ = std::fs::remove_dir_all("my_db");
//! let db: sled::Db = sled::open("my_db").unwrap();
//!
//! // insert and get
//! db.insert(b"yo!", b"v1");
//! assert_eq!(&db.get(b"yo!").unwrap().unwrap(), b"v1");
//!
//! // Atomic compare-and-swap.
//! db.compare_and_swap(
//! b"yo!", // key
//! Some(b"v1"), // old value, None for not present
//! Some(b"v2"), // new value, None for delete
//! )
//! .unwrap();
//!
//! // Iterates over key-value pairs, starting at the given key.
//! let scan_key: &[u8] = b"a non-present key before yo!";
//! let mut iter = db.range(scan_key..);
//! assert_eq!(&iter.next().unwrap().unwrap().0, b"yo!");
//! assert!(iter.next().is_none());
//!
//! db.remove(b"yo!");
//! assert!(db.get(b"yo!").unwrap().is_none());
//!
//! let other_tree: sled::Tree = db.open_tree(b"cool db facts").unwrap();
//! other_tree.insert(
//! b"k1",
//! &b"a Db acts like a Tree due to implementing Deref"[..]
//! ).unwrap();
//! # let _ = std::fs::remove_dir_all("my_db");
//! ```
#[cfg(feature = "for-internal-testing-only")]
mod block_checker;
mod config;
mod db;
mod flush_epoch;
mod heap;
mod id_allocator;
mod leaf;
mod metadata_store;
mod object_cache;
mod object_location_mapper;
mod tree;
#[cfg(any(
feature = "testing-shred-allocator",
feature = "testing-count-allocator"
))]
pub mod alloc;
#[cfg(feature = "for-internal-testing-only")]
mod event_verifier;
#[inline]
fn debug_delay() {
#[cfg(debug_assertions)]
{
let rand =
std::time::SystemTime::UNIX_EPOCH.elapsed().unwrap().as_nanos();
if rand % 128 > 100 {
for _ in 0..rand % 16 {
std::thread::yield_now();
}
}
}
}
pub use crate::config::Config;
pub use crate::db::Db;
pub use crate::tree::{Batch, Iter, Tree};
pub use inline_array::InlineArray;
const NAME_MAPPING_COLLECTION_ID: CollectionId = CollectionId(0);
const DEFAULT_COLLECTION_ID: CollectionId = CollectionId(1);
const INDEX_FANOUT: usize = 64;
const EBR_LOCAL_GC_BUFFER_SIZE: usize = 128;
use std::collections::BTreeMap;
use std::num::NonZeroU64;
use std::ops::Bound;
use std::sync::Arc;
use parking_lot::RwLock;
use crate::flush_epoch::{
FlushEpoch, FlushEpochGuard, FlushEpochTracker, FlushInvariants,
};
use crate::heap::{
HeapStats, ObjectRecovery, SlabAddress, Update, WriteBatchStats,
};
use crate::id_allocator::{Allocator, DeferredFree};
use crate::leaf::Leaf;
// These are public so that they can be easily crash tested in external
// binaries. They are hidden because there are zero guarantees around their
// API stability or functionality.
#[doc(hidden)]
pub use crate::heap::{Heap, HeapRecovery};
#[doc(hidden)]
pub use crate::metadata_store::MetadataStore;
#[doc(hidden)]
pub use crate::object_cache::{CacheStats, Dirty, FlushStats, ObjectCache};
/// Opens a `Db` with a default configuration at the
/// specified path. This will create a new storage
/// directory at the specified path if it does
/// not already exist. You can use the `Db::was_recovered`
/// method to determine if your database was recovered
/// from a previous instance.
pub fn open>(path: P) -> std::io::Result {
Config::new().path(path).open()
}
#[derive(Debug, Copy, Clone)]
pub struct Stats {
pub cache: CacheStats,
}
/// Compare and swap result.
///
/// It returns `Ok(Ok(()))` if operation finishes successfully and
/// - `Ok(Err(CompareAndSwapError(current, proposed)))` if operation failed
/// to setup a new value. `CompareAndSwapError` contains current and
/// proposed values.
/// - `Err(Error::Unsupported)` if the database is opened in read-only mode.
/// otherwise.
pub type CompareAndSwapResult = std::io::Result<
std::result::Result,
>;
type Index = concurrent_map::ConcurrentMap<
InlineArray,
Object,
INDEX_FANOUT,
EBR_LOCAL_GC_BUFFER_SIZE,
>;
/// Compare and swap error.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct CompareAndSwapError {
/// The current value which caused your CAS to fail.
pub current: Option,
/// Returned value that was proposed unsuccessfully.
pub proposed: Option,
}
/// Compare and swap success.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct CompareAndSwapSuccess {
/// The current value which was successfully installed.
pub new_value: Option,
/// Returned value that was previously stored.
pub previous_value: Option,
}
impl std::fmt::Display for CompareAndSwapError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "Compare and swap conflict")
}
}
impl std::error::Error for CompareAndSwapError {}
#[derive(
Debug,
Clone,
Copy,
serde::Serialize,
serde::Deserialize,
PartialOrd,
Ord,
PartialEq,
Eq,
Hash,
)]
pub struct ObjectId(NonZeroU64);
impl ObjectId {
fn new(from: u64) -> Option {
NonZeroU64::new(from).map(ObjectId)
}
}
impl std::ops::Deref for ObjectId {
type Target = u64;
fn deref(&self) -> &u64 {
let self_ref: &NonZeroU64 = &self.0;
// NonZeroU64 is repr(transparent) where it wraps a u64
// so it is guaranteed to match the binary layout. This
// makes it safe to cast a reference to one as a reference
// to the other like this.
let self_ptr: *const NonZeroU64 = self_ref as *const _;
let reference: *const u64 = self_ptr as *const u64;
unsafe { &*reference }
}
}
impl concurrent_map::Minimum for ObjectId {
const MIN: ObjectId = ObjectId(NonZeroU64::MIN);
}
#[derive(
Debug,
Clone,
Copy,
serde::Serialize,
serde::Deserialize,
PartialOrd,
Ord,
PartialEq,
Eq,
Hash,
)]
pub struct CollectionId(u64);
impl concurrent_map::Minimum for CollectionId {
const MIN: CollectionId = CollectionId(u64::MIN);
}
#[derive(Debug, Clone)]
struct CacheBox {
leaf: Option>>,
#[allow(unused)]
logged_index: BTreeMap,
}
#[allow(unused)]
#[derive(Debug, Clone)]
struct LogValue {
location: SlabAddress,
value: Option,
}
#[derive(Debug, Clone)]
pub struct Object {
object_id: ObjectId,
collection_id: CollectionId,
low_key: InlineArray,
inner: Arc>>,
}
impl PartialEq for Object {
fn eq(&self, other: &Self) -> bool {
self.object_id == other.object_id
}
}
/// Stored on `Db` and `Tree` in an Arc, so that when the
/// last "high-level" struct is dropped, the flusher thread
/// is cleaned up.
struct ShutdownDropper {
shutdown_sender: parking_lot::Mutex<
std::sync::mpsc::Sender>,
>,
cache: parking_lot::Mutex>,
}
impl Drop for ShutdownDropper {
fn drop(&mut self) {
let (tx, rx) = std::sync::mpsc::channel();
log::debug!("sending shutdown signal to flusher");
if self.shutdown_sender.lock().send(tx).is_ok() {
if let Err(e) = rx.recv() {
log::error!("failed to shut down flusher thread: {:?}", e);
} else {
log::debug!("flush thread successfully terminated");
}
} else {
log::debug!(
"failed to shut down flusher, manually flushing ObjectCache"
);
let cache = self.cache.lock();
if let Err(e) = cache.flush() {
log::error!(
"Db flusher encountered error while flushing: {:?}",
e
);
cache.set_error(&e);
}
}
}
}
fn map_bound U>(bound: Bound, f: F) -> Bound {
match bound {
Bound::Unbounded => Bound::Unbounded,
Bound::Included(x) => Bound::Included(f(x)),
Bound::Excluded(x) => Bound::Excluded(f(x)),
}
}
const fn _assert_public_types_send_sync() {
use std::fmt::Debug;
const fn _assert_send() {}
const fn _assert_send_sync() {}
/*
_assert_send::();
_assert_send_sync::();
_assert_send_sync::();
_assert_send_sync::();
*/
_assert_send::();
_assert_send_sync::();
_assert_send_sync::();
_assert_send_sync::();
_assert_send_sync::();
_assert_send_sync::();
}
================================================
FILE: src/metadata_store.rs
================================================
use std::collections::BTreeSet;
use std::fs;
use std::io::{self, Read, Write};
use std::num::NonZeroU64;
use std::path::{Path, PathBuf};
use std::sync::{
Arc,
atomic::{AtomicPtr, AtomicU64, Ordering},
};
use crossbeam_channel::{Receiver, Sender, bounded, unbounded};
use fault_injection::{annotate, fallible, maybe};
use fnv::FnvHashMap;
use inline_array::InlineArray;
use parking_lot::Mutex;
use rayon::prelude::*;
use zstd::stream::read::Decoder as ZstdDecoder;
use zstd::stream::write::Encoder as ZstdEncoder;
use crate::{CollectionId, ObjectId, heap::UpdateMetadata};
const WARN: &str = "DO_NOT_PUT_YOUR_FILES_HERE";
const TMP_SUFFIX: &str = ".tmp";
const LOG_PREFIX: &str = "log";
const SNAPSHOT_PREFIX: &str = "snapshot";
const ZSTD_LEVEL: i32 = 3;
// NB: intentionally does not implement Clone, and
// the Inner::drop code relies on this invariant for
// now so that we don't free the global error until
// all high-level structs are dropped. This is not
// hard to change over time though, just a current
// invariant.
pub struct MetadataStore {
inner: Inner,
is_shut_down: bool,
}
impl Drop for MetadataStore {
fn drop(&mut self) {
if self.is_shut_down {
return;
}
self.shutdown_inner();
self.is_shut_down = true;
}
}
struct MetadataRecovery {
recovered: Vec,
id_for_next_log: u64,
snapshot_size: u64,
}
struct LogAndStats {
file: fs::File,
bytes_written: u64,
log_sequence_number: u64,
}
enum WorkerMessage {
Shutdown(Sender<()>),
LogReadyToCompact { log_and_stats: LogAndStats },
}
fn get_compactions(
rx: &mut Receiver,
) -> Result, Option>> {
let mut ret = vec![];
match rx.recv() {
Ok(WorkerMessage::Shutdown(tx)) => {
return Err(Some(tx));
}
Ok(WorkerMessage::LogReadyToCompact { log_and_stats }) => {
ret.push(log_and_stats.log_sequence_number);
}
Err(e) => {
log::error!(
"metadata store worker thread unable to receive message, unexpected shutdown: {e:?}"
);
return Err(None);
}
}
// scoop up any additional logs that have built up while we were busy compacting
loop {
match rx.try_recv() {
Ok(WorkerMessage::Shutdown(tx)) => {
tx.send(()).unwrap();
return Err(Some(tx));
}
Ok(WorkerMessage::LogReadyToCompact { log_and_stats }) => {
ret.push(log_and_stats.log_sequence_number);
}
Err(_timeout) => return Ok(ret),
}
}
}
fn worker(
mut rx: Receiver,
mut last_snapshot_lsn: u64,
inner: Inner,
) {
loop {
if let Err(error) = check_error(&inner.global_error) {
drop(inner);
log::error!(
"compaction thread terminating after global error set to {:?}",
error
);
return;
}
match get_compactions(&mut rx) {
Ok(log_ids) => {
assert_eq!(log_ids[0], last_snapshot_lsn + 1);
let write_res = read_snapshot_and_apply_logs(
&inner.storage_directory,
log_ids.into_iter().collect(),
Some(last_snapshot_lsn),
&inner.directory_lock,
);
match write_res {
Err(e) => {
set_error(&inner.global_error, &e);
log::error!(
"log compactor thread encountered error: {:?} - setting global fatal error and shutting down compactions",
e
);
return;
}
Ok(recovery) => {
inner
.snapshot_size
.store(recovery.snapshot_size, Ordering::SeqCst);
last_snapshot_lsn =
recovery.id_for_next_log.checked_sub(1).unwrap();
}
}
}
Err(Some(tx)) => {
drop(inner);
if let Err(e) = tx.send(()) {
log::error!(
"log compactor failed to send shutdown ack to system: {e:?}"
);
}
return;
}
Err(None) => {
return;
}
}
}
}
fn set_error(
global_error: &AtomicPtr<(io::ErrorKind, String)>,
error: &io::Error,
) {
let kind = error.kind();
let reason = error.to_string();
let boxed = Box::new((kind, reason));
let ptr = Box::into_raw(boxed);
if global_error
.compare_exchange(
std::ptr::null_mut(),
ptr,
Ordering::SeqCst,
Ordering::SeqCst,
)
.is_err()
{
// global fatal error already installed, drop this one
unsafe {
drop(Box::from_raw(ptr));
}
}
}
fn check_error(
global_error: &AtomicPtr<(io::ErrorKind, String)>,
) -> io::Result<()> {
let err_ptr: *const (io::ErrorKind, String) =
global_error.load(Ordering::Acquire);
if err_ptr.is_null() {
Ok(())
} else {
let deref: &(io::ErrorKind, String) = unsafe { &*err_ptr };
Err(io::Error::new(deref.0, deref.1.clone()))
}
}
#[derive(Clone)]
struct Inner {
global_error: Arc>,
active_log: Arc>,
snapshot_size: Arc,
storage_directory: PathBuf,
directory_lock: Arc,
worker_outbox: Sender,
}
impl Drop for Inner {
fn drop(&mut self) {
// NB: this is the only place where the global error should be
// reclaimed in the whole sled codebase, as this Inner is only held
// by the background writer and the heap (in an Arc) so when this
// drop happens, it's because the whole system is going down, not
// because any particular Db instance that may have been cloned
// by a thread is dropping.
let error_ptr =
self.global_error.swap(std::ptr::null_mut(), Ordering::Acquire);
if !error_ptr.is_null() {
unsafe {
drop(Box::from_raw(error_ptr));
}
}
}
}
impl MetadataStore {
pub fn get_global_error_arc(
&self,
) -> Arc> {
self.inner.global_error.clone()
}
fn shutdown_inner(&mut self) {
let (tx, rx) = bounded(1);
if self.inner.worker_outbox.send(WorkerMessage::Shutdown(tx)).is_ok() {
let _ = rx.recv();
}
self.set_error(&io::Error::other(
"system has been shut down".to_string(),
));
self.is_shut_down = true;
}
fn check_error(&self) -> io::Result<()> {
check_error(&self.inner.global_error)
}
fn set_error(&self, error: &io::Error) {
set_error(&self.inner.global_error, error);
}
/// Returns the writer handle `MetadataStore`, a sorted array of metadata, and a sorted array
/// of free keys.
pub fn recover>(
storage_directory: P,
) -> io::Result<(
// Metadata writer
MetadataStore,
// Metadata - node id, value, user data
Vec,
)> {
use fs2::FileExt;
// TODO NOCOMMIT
let sync_status = std::process::Command::new("sync")
.status()
.map(|status| status.success());
if !matches!(sync_status, Ok(true)) {
log::warn!(
"sync command before recovery failed: {:?}",
sync_status
);
}
let path = storage_directory.as_ref();
// initialize directories if not present
if let Err(e) = fs::read_dir(path) {
if e.kind() == io::ErrorKind::NotFound {
fallible!(fs::create_dir_all(path));
}
}
let _ = fs::File::create(path.join(WARN));
let directory_lock = fallible!(fs::File::open(path));
fallible!(directory_lock.sync_all());
fallible!(directory_lock.try_lock_exclusive());
let recovery =
MetadataStore::recover_inner(&storage_directory, &directory_lock)?;
let new_log = LogAndStats {
log_sequence_number: recovery.id_for_next_log,
bytes_written: 0,
file: fallible!(fs::File::create(log_path(
path,
recovery.id_for_next_log
))),
};
let (tx, rx) = unbounded();
let inner = Inner {
snapshot_size: Arc::new(recovery.snapshot_size.into()),
storage_directory: path.into(),
directory_lock: Arc::new(directory_lock),
global_error: Default::default(),
active_log: Arc::new(Mutex::new(new_log)),
worker_outbox: tx,
};
let worker_inner = inner.clone();
let spawn_res = std::thread::Builder::new()
.name("sled_flusher".into())
.spawn(move || {
worker(
rx,
recovery.id_for_next_log.checked_sub(1).unwrap(),
worker_inner,
)
});
if let Err(e) = spawn_res {
return Err(io::Error::other(format!(
"unable to spawn metadata compactor thread for sled database: {:?}",
e
)));
}
Ok((MetadataStore { inner, is_shut_down: false }, recovery.recovered))
}
/// Returns the recovered mappings, the id for the next log file, the highest allocated object id, and the set of free ids
fn recover_inner>(
storage_directory: P,
directory_lock: &fs::File,
) -> io::Result {
let path = storage_directory.as_ref();
log::debug!("opening MetadataStore at {:?}", path);
let (log_ids, snapshot_id_opt) = enumerate_logs_and_snapshot(path)?;
read_snapshot_and_apply_logs(
path,
log_ids,
snapshot_id_opt,
directory_lock,
)
}
/// Write a batch of metadata. `None` for the second half of the outer tuple represents a
/// deletion. Returns the bytes written.
pub fn write_batch(&self, batch: &[UpdateMetadata]) -> io::Result {
self.check_error()?;
let batch_bytes = serialize_batch(batch);
let ret = batch_bytes.len() as u64;
let mut log = self.inner.active_log.lock();
if let Err(e) = maybe!(log.file.write_all(&batch_bytes)) {
self.set_error(&e);
return Err(e);
}
if let Err(e) = maybe!(log.file.sync_all())
.and_then(|_| self.inner.directory_lock.sync_all())
{
self.set_error(&e);
return Err(e);
}
log.bytes_written += batch_bytes.len() as u64;
if log.bytes_written
> self.inner.snapshot_size.load(Ordering::Acquire).max(64 * 1024)
{
let next_offset = log.log_sequence_number + 1;
let next_path =
log_path(&self.inner.storage_directory, next_offset);
// open new log
let mut next_log_file_opts = fs::OpenOptions::new();
next_log_file_opts.create(true).read(true).write(true);
let next_log_file = match maybe!(next_log_file_opts.open(next_path))
{
Ok(nlf) => nlf,
Err(e) => {
self.set_error(&e);
return Err(e);
}
};
let next_log_and_stats = LogAndStats {
file: next_log_file,
log_sequence_number: next_offset,
bytes_written: 0,
};
// replace log
let old_log_and_stats =
std::mem::replace(&mut *log, next_log_and_stats);
// send to snapshot writer
self.inner
.worker_outbox
.send(WorkerMessage::LogReadyToCompact {
log_and_stats: old_log_and_stats,
})
.expect("unable to send log to compact to worker");
}
Ok(ret)
}
}
fn serialize_batch(batch: &[UpdateMetadata]) -> Vec {
// we initialize the vector to contain placeholder bytes for the frame length
let batch_bytes = 0_u64.to_le_bytes().to_vec();
// write format:
// 6 byte LE frame length (in bytes, not items)
// 2 byte crc of the frame length
// payload:
// zstd encoded 8 byte LE key
// zstd encoded 8 byte LE value
// repeated for each kv pair
// LE encoded crc32 of length + payload raw bytes, XOR 0xAF to make non-zero in empty case
let mut batch_encoder = ZstdEncoder::new(batch_bytes, ZSTD_LEVEL).unwrap();
for update_metadata in batch {
match update_metadata {
UpdateMetadata::Store {
object_id,
collection_id,
low_key,
location,
} => {
batch_encoder
.write_all(&object_id.0.get().to_le_bytes())
.unwrap();
batch_encoder
.write_all(&collection_id.0.to_le_bytes())
.unwrap();
batch_encoder.write_all(&location.get().to_le_bytes()).unwrap();
let low_key_len: u64 = low_key.len() as u64;
batch_encoder.write_all(&low_key_len.to_le_bytes()).unwrap();
batch_encoder.write_all(low_key).unwrap();
}
UpdateMetadata::Free { object_id, collection_id } => {
batch_encoder
.write_all(&object_id.0.get().to_le_bytes())
.unwrap();
batch_encoder
.write_all(&collection_id.0.to_le_bytes())
.unwrap();
// heap location
batch_encoder.write_all(&0_u64.to_le_bytes()).unwrap();
// metadata len
batch_encoder.write_all(&0_u64.to_le_bytes()).unwrap();
}
}
}
let mut batch_bytes = batch_encoder.finish().unwrap();
let batch_len = batch_bytes.len().checked_sub(8).unwrap();
batch_bytes[..8].copy_from_slice(&batch_len.to_le_bytes());
assert_eq!(&[0, 0], &batch_bytes[6..8]);
let len_hash: [u8; 2] =
(crc32fast::hash(&batch_bytes[..6]) as u16).to_le_bytes();
batch_bytes[6..8].copy_from_slice(&len_hash);
let hash: u32 = crc32fast::hash(&batch_bytes) ^ 0xAF;
let hash_bytes: [u8; 4] = hash.to_le_bytes();
batch_bytes.extend_from_slice(&hash_bytes);
batch_bytes
}
fn read_frame(
file: &mut fs::File,
reusable_frame_buffer: &mut Vec,
) -> io::Result> {
let mut frame_size_with_crc_buf: [u8; 8] = [0; 8];
// TODO only break if UnexpectedEof, otherwise propagate
fallible!(file.read_exact(&mut frame_size_with_crc_buf));
let expected_len_hash_buf =
[frame_size_with_crc_buf[6], frame_size_with_crc_buf[7]];
let actual_len_hash_buf: [u8; 2] =
(crc32fast::hash(&frame_size_with_crc_buf[..6]) as u16).to_le_bytes();
// clear crc bytes before turning into usize
let mut frame_size_buf = frame_size_with_crc_buf;
frame_size_buf[6] = 0;
frame_size_buf[7] = 0;
if actual_len_hash_buf != expected_len_hash_buf {
return Err(annotate!(io::Error::new(
io::ErrorKind::InvalidData,
"corrupt frame length"
)));
}
let len_u64: u64 = u64::from_le_bytes(frame_size_buf);
let len: usize = usize::try_from(len_u64).unwrap();
reusable_frame_buffer.clear();
reusable_frame_buffer.reserve(len + 12);
unsafe {
reusable_frame_buffer.set_len(len + 12);
}
reusable_frame_buffer[..8].copy_from_slice(&frame_size_with_crc_buf);
fallible!(file.read_exact(&mut reusable_frame_buffer[8..]));
let crc_actual = crc32fast::hash(&reusable_frame_buffer[..len + 8]) ^ 0xAF;
let crc_recorded = u32::from_le_bytes([
reusable_frame_buffer[len + 8],
reusable_frame_buffer[len + 9],
reusable_frame_buffer[len + 10],
reusable_frame_buffer[len + 11],
]);
if crc_actual != crc_recorded {
log::warn!("encountered incorrect crc for batch in log");
return Err(annotate!(io::Error::new(
io::ErrorKind::InvalidData,
"crc mismatch for read of batch frame",
)));
}
let mut ret = vec![];
let mut decoder = ZstdDecoder::new(&reusable_frame_buffer[8..len + 8])
.expect("failed to create zstd decoder");
let mut object_id_buf: [u8; 8] = [0; 8];
let mut collection_id_buf: [u8; 8] = [0; 8];
let mut location_buf: [u8; 8] = [0; 8];
let mut low_key_len_buf: [u8; 8] = [0; 8];
let mut low_key_buf = vec![];
loop {
let first_read_res = decoder
.read_exact(&mut object_id_buf)
.and_then(|_| decoder.read_exact(&mut collection_id_buf))
.and_then(|_| decoder.read_exact(&mut location_buf))
.and_then(|_| decoder.read_exact(&mut low_key_len_buf));
if let Err(e) = first_read_res {
if e.kind() != io::ErrorKind::UnexpectedEof {
return Err(e);
} else {
break;
}
}
let object_id_u64 = u64::from_le_bytes(object_id_buf);
let object_id = if let Some(object_id) = ObjectId::new(object_id_u64) {
object_id
} else {
return Err(annotate!(io::Error::new(
io::ErrorKind::InvalidData,
"corrupt object ID 0 somehow passed crc check"
)));
};
let collection_id = CollectionId(u64::from_le_bytes(collection_id_buf));
let location = u64::from_le_bytes(location_buf);
let low_key_len_raw = u64::from_le_bytes(low_key_len_buf);
let low_key_len = usize::try_from(low_key_len_raw).unwrap();
low_key_buf.reserve(low_key_len);
unsafe {
low_key_buf.set_len(low_key_len);
}
decoder
.read_exact(&mut low_key_buf)
.expect("we expect reads from crc-verified buffers to succeed");
if let Some(location_nzu) = NonZeroU64::new(location) {
let low_key = InlineArray::from(&*low_key_buf);
ret.push(UpdateMetadata::Store {
object_id,
collection_id,
location: location_nzu,
low_key,
});
} else {
ret.push(UpdateMetadata::Free { object_id, collection_id });
}
}
Ok(ret)
}
// returns the deduplicated data in this log, along with an optional offset where a
// final torn write occurred.
fn read_log(
directory_path: &Path,
lsn: u64,
) -> io::Result> {
log::trace!("reading log {lsn}");
let mut ret = FnvHashMap::default();
let mut file = fallible!(fs::File::open(log_path(directory_path, lsn)));
let mut reusable_frame_buffer: Vec = vec![];
while let Ok(frame) = read_frame(&mut file, &mut reusable_frame_buffer) {
for update_metadata in frame {
ret.insert(update_metadata.object_id(), update_metadata);
}
}
log::trace!("recovered {} items in log {}", ret.len(), lsn);
Ok(ret)
}
/// returns the data from the snapshot as well as the size of the snapshot
fn read_snapshot(
directory_path: &Path,
lsn: u64,
) -> io::Result<(FnvHashMap, u64)> {
log::trace!("reading snapshot {lsn}");
let mut reusable_frame_buffer: Vec = vec![];
let mut file =
fallible!(fs::File::open(snapshot_path(directory_path, lsn, false)));
let size = fallible!(file.metadata()).len();
let raw_frame = read_frame(&mut file, &mut reusable_frame_buffer)?;
let frame: FnvHashMap = raw_frame
.into_iter()
.map(|update_metadata| (update_metadata.object_id(), update_metadata))
.collect();
log::trace!("recovered {} items in snapshot {}", frame.len(), lsn);
Ok((frame, size))
}
fn log_path(directory_path: &Path, id: u64) -> PathBuf {
directory_path.join(format!("{LOG_PREFIX}_{:016x}", id))
}
fn snapshot_path(directory_path: &Path, id: u64, temporary: bool) -> PathBuf {
if temporary {
directory_path
.join(format!("{SNAPSHOT_PREFIX}_{:016x}{TMP_SUFFIX}", id))
} else {
directory_path.join(format!("{SNAPSHOT_PREFIX}_{:016x}", id))
}
}
fn enumerate_logs_and_snapshot(
directory_path: &Path,
) -> io::Result<(BTreeSet, Option)> {
let mut logs = BTreeSet::new();
let mut snapshot: Option = None;
for dir_entry_res in fallible!(fs::read_dir(directory_path)) {
let dir_entry = fallible!(dir_entry_res);
let file_name = if let Ok(f) = dir_entry.file_name().into_string() {
f
} else {
log::warn!(
"skipping unexpected file with non-unicode name {:?}",
dir_entry.file_name()
);
continue;
};
if file_name.ends_with(TMP_SUFFIX) {
log::warn!("removing incomplete snapshot rewrite {file_name:?}");
fallible!(fs::remove_file(directory_path.join(file_name)));
} else if file_name.starts_with(LOG_PREFIX) {
let start = LOG_PREFIX.len() + 1;
let stop = start + 16;
if let Ok(id) = u64::from_str_radix(&file_name[start..stop], 16) {
logs.insert(id);
} else {
todo!()
}
} else if file_name.starts_with(SNAPSHOT_PREFIX) {
let start = SNAPSHOT_PREFIX.len() + 1;
let stop = start + 16;
if let Ok(id) = u64::from_str_radix(&file_name[start..stop], 16) {
if let Some(snap_id) = snapshot {
if snap_id < id {
log::warn!(
"removing stale snapshot {id} that is superceded by snapshot {id}"
);
if let Err(e) = fs::remove_file(&file_name) {
log::warn!(
"failed to remove stale snapshot file {:?}: {:?}",
file_name,
e
);
}
snapshot = Some(id);
}
} else {
snapshot = Some(id);
}
} else {
todo!()
}
}
}
let snap_id = snapshot.unwrap_or(0);
for stale_log_id in logs.range(..=snap_id) {
let file_name = log_path(directory_path, *stale_log_id);
log::warn!(
"removing stale log {file_name:?} that is contained within snapshot {snap_id}"
);
fallible!(fs::remove_file(file_name));
}
logs.retain(|l| *l > snap_id);
Ok((logs, snapshot))
}
fn read_snapshot_and_apply_logs(
path: &Path,
log_ids: BTreeSet,
snapshot_id_opt: Option,
locked_directory: &fs::File,
) -> io::Result {
let (snapshot_tx, snapshot_rx) = bounded(1);
if let Some(snapshot_id) = snapshot_id_opt {
let path: PathBuf = path.into();
rayon::spawn(move || {
let snap_res = read_snapshot(&path, snapshot_id)
.map(|(snapshot, _snapshot_len)| snapshot);
snapshot_tx.send(snap_res).unwrap();
});
} else {
snapshot_tx.send(Ok(Default::default())).unwrap();
}
let mut max_log_id = snapshot_id_opt.unwrap_or(0);
let log_data_res: io::Result<
Vec<(u64, FnvHashMap)>,
> = (&log_ids) //.iter().collect::>())
.into_par_iter()
.map(move |log_id| {
if let Some(snapshot_id) = snapshot_id_opt {
assert!(*log_id > snapshot_id);
}
let log_data = read_log(path, *log_id)?;
Ok((*log_id, log_data))
})
.collect();
let mut recovered: FnvHashMap =
snapshot_rx.recv().unwrap()?;
log::trace!("recovered snapshot contains {recovered:?}");
for (log_id, log_datum) in log_data_res? {
max_log_id = max_log_id.max(log_id);
for (object_id, update_metadata) in log_datum {
if matches!(update_metadata, UpdateMetadata::Store { .. }) {
recovered.insert(object_id, update_metadata);
} else {
let previous = recovered.remove(&object_id);
if previous.is_none() {
log::trace!(
"recovered a Free for {object_id:?} without a preceeding Store"
);
}
}
}
}
let mut recovered: Vec = recovered.into_values().collect();
recovered.par_sort_unstable();
// write fresh snapshot with recovered data
let new_snapshot_data = serialize_batch(&recovered);
let snapshot_size = new_snapshot_data.len() as u64;
let new_snapshot_tmp_path = snapshot_path(path, max_log_id, true);
log::trace!("writing snapshot to {new_snapshot_tmp_path:?}");
let mut snapshot_file_opts = fs::OpenOptions::new();
snapshot_file_opts.create(true).read(false).write(true);
let mut snapshot_file =
fallible!(snapshot_file_opts.open(&new_snapshot_tmp_path));
fallible!(snapshot_file.write_all(&new_snapshot_data));
drop(new_snapshot_data);
fallible!(snapshot_file.sync_all());
let new_snapshot_path = snapshot_path(path, max_log_id, false);
log::trace!("renaming written snapshot to {new_snapshot_path:?}");
fallible!(fs::rename(new_snapshot_tmp_path, new_snapshot_path));
fallible!(locked_directory.sync_all());
for log_id in &log_ids {
let log_path = log_path(path, *log_id);
fallible!(fs::remove_file(log_path));
}
if let Some(old_snapshot_id) = snapshot_id_opt {
let old_snapshot_path = snapshot_path(path, old_snapshot_id, false);
fallible!(fs::remove_file(old_snapshot_path));
}
Ok(MetadataRecovery {
recovered,
id_for_next_log: max_log_id + 1,
snapshot_size,
})
}
================================================
FILE: src/object_cache.rs
================================================
use std::cell::RefCell;
use std::collections::HashMap;
use std::io;
use std::sync::Arc;
use std::sync::atomic::{AtomicPtr, AtomicU64, Ordering};
use std::time::{Duration, Instant};
use cache_advisor::CacheAdvisor;
use concurrent_map::{ConcurrentMap, Minimum};
use fault_injection::annotate;
use inline_array::InlineArray;
use parking_lot::RwLock;
use crate::*;
#[derive(Debug, Copy, Clone)]
pub struct CacheStats {
pub cache_hits: u64,
pub cache_misses: u64,
pub cache_hit_ratio: f32,
pub max_read_io_latency_us: u64,
pub sum_read_io_latency_us: u64,
pub deserialization_latency_max_us: u64,
pub deserialization_latency_sum_us: u64,
pub heap: HeapStats,
pub flush_max: FlushStats,
pub flush_sum: FlushStats,
pub compacted_heap_slots: u64,
pub tree_leaves_merged: u64,
}
#[derive(Default, Debug, Clone, Copy)]
pub struct FlushStats {
pub pre_block_on_previous_flush: Duration,
pub pre_block_on_current_quiescence: Duration,
pub serialization_latency: Duration,
pub compute_defrag_latency: Duration,
pub storage_latency: Duration,
pub post_write_eviction_latency: Duration,
pub objects_flushed: u64,
pub write_batch: WriteBatchStats,
}
impl FlushStats {
pub fn sum(&self, other: &FlushStats) -> FlushStats {
use std::ops::Add;
FlushStats {
pre_block_on_previous_flush: self
.pre_block_on_previous_flush
.add(other.pre_block_on_previous_flush),
pre_block_on_current_quiescence: self
.pre_block_on_current_quiescence
.add(other.pre_block_on_current_quiescence),
compute_defrag_latency: self
.compute_defrag_latency
.add(other.compute_defrag_latency),
serialization_latency: self
.serialization_latency
.add(other.serialization_latency),
storage_latency: self.storage_latency.add(other.storage_latency),
post_write_eviction_latency: self
.post_write_eviction_latency
.add(other.post_write_eviction_latency),
objects_flushed: self.objects_flushed.add(other.objects_flushed),
write_batch: self.write_batch.sum(&other.write_batch),
}
}
pub fn max(&self, other: &FlushStats) -> FlushStats {
FlushStats {
pre_block_on_previous_flush: self
.pre_block_on_previous_flush
.max(other.pre_block_on_previous_flush),
pre_block_on_current_quiescence: self
.pre_block_on_current_quiescence
.max(other.pre_block_on_current_quiescence),
compute_defrag_latency: self
.compute_defrag_latency
.max(other.compute_defrag_latency),
serialization_latency: self
.serialization_latency
.max(other.serialization_latency),
storage_latency: self.storage_latency.max(other.storage_latency),
post_write_eviction_latency: self
.post_write_eviction_latency
.max(other.post_write_eviction_latency),
objects_flushed: self.objects_flushed.max(other.objects_flushed),
write_batch: self.write_batch.max(&other.write_batch),
}
}
}
#[derive(Clone, Debug, PartialEq)]
pub enum Dirty {
NotYetSerialized {
low_key: InlineArray,
node: Object,
collection_id: CollectionId,
},
CooperativelySerialized {
object_id: ObjectId,
collection_id: CollectionId,
low_key: InlineArray,
data: Arc>,
mutation_count: u64,
},
MergedAndDeleted {
object_id: ObjectId,
collection_id: CollectionId,
},
}
impl Dirty {
pub fn is_final_state(&self) -> bool {
match self {
Dirty::NotYetSerialized { .. } => false,
Dirty::CooperativelySerialized { .. } => true,
Dirty::MergedAndDeleted { .. } => true,
}
}
}
#[derive(Debug, Default, Clone, Copy)]
struct FlushStatTracker {
count: u64,
sum: FlushStats,
max: FlushStats,
}
#[derive(Debug, Default)]
pub(crate) struct ReadStatTracker {
pub cache_hits: AtomicU64,
pub cache_misses: AtomicU64,
pub max_read_io_latency_us: AtomicU64,
pub sum_read_io_latency_us: AtomicU64,
pub max_deserialization_latency_us: AtomicU64,
pub sum_deserialization_latency_us: AtomicU64,
}
#[derive(Clone)]
pub struct ObjectCache {
pub config: Config,
global_error: Arc>,
pub object_id_index: ConcurrentMap<
ObjectId,
Object,
INDEX_FANOUT,
EBR_LOCAL_GC_BUFFER_SIZE,
>,
heap: Heap,
cache_advisor: RefCell,
flush_epoch: FlushEpochTracker,
dirty: ConcurrentMap<(FlushEpoch, ObjectId), Dirty, 4>,
compacted_heap_slots: Arc,
pub(super) tree_leaves_merged: Arc,
#[cfg(feature = "for-internal-testing-only")]
pub(super) event_verifier: Arc,
invariants: Arc,
flush_stats: Arc>,
pub(super) read_stats: Arc,
}
impl std::panic::RefUnwindSafe
for ObjectCache
{
}
impl ObjectCache {
/// Returns the recovered ObjectCache, the tree indexes, and a bool signifying whether the system
/// was recovered or not
pub fn recover(
config: &Config,
) -> io::Result<(
ObjectCache,
HashMap>,
bool,
)> {
let HeapRecovery { heap, recovered_nodes, was_recovered } =
Heap::recover(LEAF_FANOUT, config)?;
let (object_id_index, indices) = initialize(&recovered_nodes, &heap);
// validate recovery
for ObjectRecovery { object_id, collection_id, low_key } in
recovered_nodes
{
let index = indices.get(&collection_id).unwrap();
let node = index.get(&low_key).unwrap();
assert_eq!(node.object_id, object_id);
}
if config.cache_capacity_bytes < 256 {
log::debug!(
"Db configured to have Config.cache_capacity_bytes \
of under 256, so we will use the minimum of 256 bytes instead"
);
}
if config.entry_cache_percent > 80 {
log::debug!(
"Db configured to have Config.entry_cache_percent\
of over 80%, so we will clamp it to the maximum of 80% instead"
);
}
let pc = ObjectCache {
config: config.clone(),
object_id_index,
cache_advisor: RefCell::new(CacheAdvisor::new(
config.cache_capacity_bytes.max(256),
config.entry_cache_percent.min(80),
)),
global_error: heap.get_global_error_arc(),
heap,
dirty: Default::default(),
flush_epoch: Default::default(),
#[cfg(feature = "for-internal-testing-only")]
event_verifier: Arc::default(),
compacted_heap_slots: Arc::default(),
tree_leaves_merged: Arc::default(),
invariants: Arc::default(),
flush_stats: Arc::default(),
read_stats: Arc::default(),
};
Ok((pc, indices, was_recovered))
}
pub fn is_clean(&self) -> bool {
self.dirty.is_empty()
}
pub fn read(&self, object_id: ObjectId) -> Option>> {
match self.heap.read(object_id) {
Some(Ok(buf)) => Some(Ok(buf)),
Some(Err(e)) => Some(Err(annotate!(e))),
None => None,
}
}
pub fn stats(&self) -> CacheStats {
let flush_stats = { *self.flush_stats.read() };
let cache_hits = self.read_stats.cache_hits.load(Ordering::Acquire);
let cache_misses = self.read_stats.cache_misses.load(Ordering::Acquire);
let cache_hit_ratio =
cache_hits as f32 / (cache_hits + cache_misses).max(1) as f32;
CacheStats {
cache_hits,
cache_misses,
cache_hit_ratio,
compacted_heap_slots: self
.compacted_heap_slots
.load(Ordering::Acquire),
tree_leaves_merged: self.tree_leaves_merged.load(Ordering::Acquire),
heap: self.heap.stats(),
flush_max: flush_stats.max,
flush_sum: flush_stats.sum,
deserialization_latency_max_us: self
.read_stats
.max_deserialization_latency_us
.load(Ordering::Acquire),
deserialization_latency_sum_us: self
.read_stats
.sum_deserialization_latency_us
.load(Ordering::Acquire),
max_read_io_latency_us: self
.read_stats
.max_read_io_latency_us
.load(Ordering::Acquire),
sum_read_io_latency_us: self
.read_stats
.sum_read_io_latency_us
.load(Ordering::Acquire),
}
}
pub fn check_error(&self) -> io::Result<()> {
let err_ptr: *const (io::ErrorKind, String) =
self.global_error.load(Ordering::Acquire);
if err_ptr.is_null() {
Ok(())
} else {
let deref: &(io::ErrorKind, String) = unsafe { &*err_ptr };
Err(io::Error::new(deref.0, deref.1.clone()))
}
}
pub fn set_error(&self, error: &io::Error) {
let kind = error.kind();
let reason = error.to_string();
let boxed = Box::new((kind, reason));
let ptr = Box::into_raw(boxed);
if self
.global_error
.compare_exchange(
std::ptr::null_mut(),
ptr,
Ordering::SeqCst,
Ordering::SeqCst,
)
.is_err()
{
// global fatal error already installed, drop this one
unsafe {
drop(Box::from_raw(ptr));
}
}
}
pub fn allocate_default_node(
&self,
collection_id: CollectionId,
) -> Object {
let object_id = self.allocate_object_id(FlushEpoch::MIN);
let node = Object {
object_id,
collection_id,
low_key: InlineArray::default(),
inner: Arc::new(RwLock::new(CacheBox {
leaf: Some(Box::new(Leaf::empty())),
logged_index: BTreeMap::default(),
})),
};
self.object_id_index.insert(object_id, node.clone());
node
}
pub fn allocate_object_id(
&self,
#[allow(unused)] flush_epoch: FlushEpoch,
) -> ObjectId {
let object_id = self.heap.allocate_object_id();
#[cfg(feature = "for-internal-testing-only")]
{
self.event_verifier.mark(
object_id,
flush_epoch,
event_verifier::State::CleanPagedIn,
concat!(file!(), ':', line!(), ":allocated"),
);
}
object_id
}
pub fn current_flush_epoch(&self) -> FlushEpoch {
self.flush_epoch.current_flush_epoch()
}
pub fn check_into_flush_epoch(&self) -> FlushEpochGuard {
self.flush_epoch.check_in()
}
pub fn install_dirty(
&self,
flush_epoch: FlushEpoch,
object_id: ObjectId,
dirty: Dirty,
) {
// dirty can transition from:
// None -> NotYetSerialized
// None -> MergedAndDeleted
// None -> CooperativelySerialized
//
// NotYetSerialized -> MergedAndDeleted
// NotYetSerialized -> CooperativelySerialized
//
// if the new Dirty is final, we must assert that
// we are transitioning from None or NotYetSerialized.
//
// if the new Dirty is not final, we must assert
// that the old value is also not final.
let last_dirty_opt = self.dirty.insert((flush_epoch, object_id), dirty);
if let Some(last_dirty) = last_dirty_opt {
assert!(
!last_dirty.is_final_state(),
"tried to install another Dirty marker for a node that is already
finalized for this flush epoch. \nflush_epoch: {:?}\nlast: {:?}",
flush_epoch, last_dirty,
);
}
}
// NB: must not be called while holding a leaf lock - which also means
// that no two LeafGuards can be held concurrently in the same scope due to
// this being called in the destructor.
pub fn mark_access_and_evict(
&self,
accessed_object_id: ObjectId,
size: usize,
#[allow(unused)] flush_epoch: FlushEpoch,
) -> io::Result<()> {
let mut ca = self.cache_advisor.borrow_mut();
let to_evict = ca.accessed_reuse_buffer(*accessed_object_id, size);
let mut not_found = 0;
for (node_to_evict, _rough_size) in to_evict {
let object_id =
if let Some(object_id) = ObjectId::new(*node_to_evict) {
object_id
} else {
unreachable!("object ID must never have been 0");
};
if accessed_object_id == object_id {
// TODO our own object was evicted, so
// set page out after current epoch (or just page out if clean?)
continue;
}
let node = if let Some(n) = self.object_id_index.get(&object_id) {
if *n.object_id != *node_to_evict {
continue;
}
n
} else {
not_found += 1;
continue;
};
let mut write = node.inner.write();
if write.leaf.is_none() {
// already paged out
continue;
}
let leaf: &mut Leaf = write.leaf.as_mut().unwrap();
if let Some(dirty_epoch) = leaf.dirty_flush_epoch {
// We can't page out this leaf until it has been
// flushed, because its changes are not yet durable.
leaf.page_out_on_flush =
leaf.page_out_on_flush.max(Some(dirty_epoch));
} else if let Some(max_unflushed_epoch) = leaf.max_unflushed_epoch {
leaf.page_out_on_flush =
leaf.page_out_on_flush.max(Some(max_unflushed_epoch));
} else {
#[cfg(feature = "for-internal-testing-only")]
{
self.event_verifier.mark(
node.object_id,
flush_epoch,
event_verifier::State::PagedOut,
concat!(file!(), ':', line!(), ":page-out"),
);
}
write.leaf = None;
}
}
if not_found > 0 {
log::trace!(
"during cache eviction, did not find {} nodes that we were trying to evict",
not_found
);
}
Ok(())
}
pub fn heap_object_id_pin(&self) -> ebr::Guard<'_, DeferredFree, 16, 16> {
self.heap.heap_object_id_pin()
}
pub fn flush(&self) -> io::Result {
let mut write_batch = vec![];
log::trace!("advancing epoch");
let (
previous_flush_complete_notifier,
this_vacant_notifier,
forward_flush_notifier,
) = self.flush_epoch.roll_epoch_forward();
let before_previous_block = Instant::now();
log::trace!(
"waiting for previous flush of {:?} to complete",
previous_flush_complete_notifier.epoch()
);
let previous_epoch =
previous_flush_complete_notifier.wait_for_complete();
let pre_block_on_previous_flush = before_previous_block.elapsed();
let before_current_quiescence = Instant::now();
log::trace!(
"waiting for our epoch {:?} to become vacant",
this_vacant_notifier.epoch()
);
assert_eq!(previous_epoch.increment(), this_vacant_notifier.epoch());
let flush_through_epoch: FlushEpoch =
this_vacant_notifier.wait_for_complete();
let pre_block_on_current_quiescence =
before_current_quiescence.elapsed();
self.invariants.mark_flushing_epoch(flush_through_epoch);
let mut objects_to_defrag = self.heap.objects_to_defrag();
let flush_boundary = (flush_through_epoch.increment(), ObjectId::MIN);
let mut evict_after_flush = vec![];
let before_serialization = Instant::now();
for ((dirty_epoch, dirty_object_id), dirty_value_initial_read) in
self.dirty.range(..flush_boundary)
{
objects_to_defrag.remove(&dirty_object_id);
let dirty_value = self
.dirty
.remove(&(dirty_epoch, dirty_object_id))
.expect("violation of flush responsibility");
if let Dirty::NotYetSerialized { .. } = &dirty_value {
assert_eq!(dirty_value_initial_read, dirty_value);
}
// drop is necessary to increase chance of Arc strong count reaching 1
// while taking ownership of the value
drop(dirty_value_initial_read);
assert_eq!(dirty_epoch, flush_through_epoch);
match dirty_value {
Dirty::MergedAndDeleted { object_id, collection_id } => {
assert_eq!(object_id, dirty_object_id);
log::trace!(
"MergedAndDeleted for {:?}, adding None to write_batch",
object_id
);
write_batch.push(Update::Free { object_id, collection_id });
#[cfg(feature = "for-internal-testing-only")]
{
self.event_verifier.mark(
object_id,
dirty_epoch,
event_verifier::State::AddedToWriteBatch,
concat!(
file!(),
':',
line!(),
":flush-merged-and-deleted"
),
);
}
}
Dirty::CooperativelySerialized {
object_id: _,
collection_id,
low_key,
mutation_count: _,
mut data,
} => {
Arc::make_mut(&mut data);
let data = Arc::into_inner(data).unwrap();
write_batch.push(Update::Store {
object_id: dirty_object_id,
collection_id,
low_key,
data,
});
#[cfg(feature = "for-internal-testing-only")]
{
self.event_verifier.mark(
dirty_object_id,
dirty_epoch,
event_verifier::State::AddedToWriteBatch,
concat!(
file!(),
':',
line!(),
":flush-cooperative"
),
);
}
}
Dirty::NotYetSerialized { low_key, collection_id, node } => {
assert_eq!(low_key, node.low_key);
assert_eq!(
dirty_object_id, node.object_id,
"mismatched node ID for NotYetSerialized with low key {:?}",
low_key
);
let mut lock = node.inner.write();
let leaf_ref: &mut Leaf = if let Some(
lock_ref,
) =
lock.leaf.as_mut()
{
lock_ref
} else {
#[cfg(feature = "for-internal-testing-only")]
self.event_verifier
.print_debug_history_for_object(dirty_object_id);
panic!(
"failed to get lock for node that was NotYetSerialized, low key {:?} id {:?}",
low_key, node.object_id
);
};
assert_eq!(leaf_ref.lo, low_key);
let data = if leaf_ref.dirty_flush_epoch
== Some(flush_through_epoch)
{
if let Some(deleted_at) = leaf_ref.deleted {
#[cfg(feature = "for-internal-testing-only")]
if deleted_at <= flush_through_epoch {
println!(
"{dirty_object_id:?} deleted at {deleted_at:?} \
but we are flushing at {flush_through_epoch:?}"
);
self.event_verifier
.print_debug_history_for_object(
dirty_object_id,
);
}
assert!(deleted_at > flush_through_epoch);
}
leaf_ref.max_unflushed_epoch =
leaf_ref.dirty_flush_epoch.take();
#[cfg(feature = "for-internal-testing-only")]
{
self.event_verifier.mark(
dirty_object_id,
dirty_epoch,
event_verifier::State::AddedToWriteBatch,
concat!(
file!(),
':',
line!(),
":flush-serialize"
),
);
}
leaf_ref.serialize(self.config.zstd_compression_level)
} else {
// Here we expect that there was a benign data race and that another thread
// mutated the leaf after encountering it being dirty for our epoch, after
// storing a CooperativelySerialized in the dirty map.
let dirty_value_2_opt =
self.dirty.remove(&(dirty_epoch, dirty_object_id));
if let Some(Dirty::CooperativelySerialized {
low_key: low_key_2,
mutation_count: _,
mut data,
collection_id: ci2,
object_id: ni2,
}) = dirty_value_2_opt
{
assert_eq!(node.object_id, ni2);
assert_eq!(node.object_id, dirty_object_id);
assert_eq!(low_key, low_key_2);
assert_eq!(node.low_key, low_key);
assert_eq!(collection_id, ci2);
Arc::make_mut(&mut data);
#[cfg(feature = "for-internal-testing-only")]
{
self.event_verifier.mark(
dirty_object_id,
dirty_epoch,
event_verifier::State::AddedToWriteBatch,
concat!(
file!(),
':',
line!(),
":flush-laggy-cooperative"
),
);
}
Arc::into_inner(data).unwrap()
} else {
log::error!(
"violation of flush responsibility for second read \
of expected cooperative serialization. leaf in question's \
dirty_flush_epoch is {:?}, our expected key was {:?}. node.deleted: {:?}",
leaf_ref.dirty_flush_epoch,
(dirty_epoch, dirty_object_id),
leaf_ref.deleted,
);
#[cfg(feature = "for-internal-testing-only")]
self.event_verifier.print_debug_history_for_object(
dirty_object_id,
);
unreachable!(
"a leaf was expected to be cooperatively serialized but it was not available. \
violation of flush responsibility for second read \
of expected cooperative serialization. leaf in question's \
dirty_flush_epoch is {:?}, our expected key was {:?}. node.deleted: {:?}",
leaf_ref.dirty_flush_epoch,
(dirty_epoch, dirty_object_id),
leaf_ref.deleted,
);
}
};
write_batch.push(Update::Store {
object_id: dirty_object_id,
collection_id,
low_key,
data,
});
if leaf_ref.page_out_on_flush == Some(flush_through_epoch) {
// page_out_on_flush is set to false
// on page-in due to serde(skip)
evict_after_flush.push(node.clone());
}
}
}
}
if !objects_to_defrag.is_empty() {
log::debug!(
"objects to defrag (after flush loop): {}",
objects_to_defrag.len()
);
self.compacted_heap_slots
.fetch_add(objects_to_defrag.len() as u64, Ordering::Relaxed);
}
let before_compute_defrag = Instant::now();
if cfg!(not(feature = "monotonic-behavior")) {
let mut object_not_found = 0;
for fragmented_object_id in objects_to_defrag {
let object_opt =
self.object_id_index.get(&fragmented_object_id);
let object = if let Some(object) = object_opt {
object
} else {
object_not_found += 1;
continue;
};
if let Some(ref inner) = object.inner.read().leaf {
if let Some(dirty) = inner.dirty_flush_epoch {
assert!(dirty > flush_through_epoch);
// This object will be rewritten anyway when its dirty epoch gets flushed
continue;
}
}
let data = match self.read(fragmented_object_id) {
Some(Ok(data)) => data,
Some(Err(e)) => {
let annotated = annotate!(e);
log::error!(
"failed to read object during GC: {annotated:?}"
);
continue;
}
None => {
log::error!(
"failed to read object during GC: object not found"
);
continue;
}
};
write_batch.push(Update::Store {
object_id: fragmented_object_id,
collection_id: object.collection_id,
low_key: object.low_key,
data,
});
}
if object_not_found > 0 {
log::debug!(
"{} objects not found while defragmenting",
object_not_found
);
}
}
let compute_defrag_latency = before_compute_defrag.elapsed();
let serialization_latency = before_serialization.elapsed();
let before_storage = Instant::now();
let objects_flushed = write_batch.len() as u64;
#[cfg(feature = "for-internal-testing-only")]
let write_batch_object_ids: Vec =
write_batch.iter().map(Update::object_id).collect();
let write_batch_stats = if objects_flushed > 0 {
let write_batch_stats = self.heap.write_batch(write_batch)?;
log::trace!(
"marking {flush_through_epoch:?} as flushed - \
{objects_flushed} objects written, {write_batch_stats:?}",
);
write_batch_stats
} else {
WriteBatchStats::default()
};
let storage_latency = before_storage.elapsed();
#[cfg(feature = "for-internal-testing-only")]
{
for update_object_id in write_batch_object_ids {
self.event_verifier.mark(
update_object_id,
flush_through_epoch,
event_verifier::State::Flushed,
concat!(file!(), ':', line!(), ":flush-finished"),
);
}
}
log::trace!(
"marking the forward flush notifier that {:?} is flushed",
flush_through_epoch
);
self.invariants.mark_flushed_epoch(flush_through_epoch);
forward_flush_notifier.mark_complete();
let before_eviction = Instant::now();
for node_to_evict in evict_after_flush {
// NB: since we dropped this leaf and lock after we marked its
// node in evict_after_flush, it's possible that it may have
// been written to afterwards.
let mut lock = node_to_evict.inner.write();
let leaf = lock.leaf.as_mut().unwrap();
if let Some(dirty_epoch) = leaf.dirty_flush_epoch {
if dirty_epoch != flush_through_epoch {
continue;
}
} else {
continue;
}
#[cfg(feature = "for-internal-testing-only")]
{
self.event_verifier.mark(
node_to_evict.object_id,
flush_through_epoch,
event_verifier::State::PagedOut,
concat!(file!(), ':', line!(), ":page-out-after-flush"),
);
}
lock.leaf = None;
}
let post_write_eviction_latency = before_eviction.elapsed();
// kick forward the low level epoch-based reclamation systems
// because this operation can cause a lot of garbage to build
// up, and this speeds up its reclamation.
self.flush_epoch.manually_advance_epoch();
self.heap.manually_advance_epoch();
let ret = FlushStats {
pre_block_on_current_quiescence,
pre_block_on_previous_flush,
serialization_latency,
storage_latency,
post_write_eviction_latency,
objects_flushed,
write_batch: write_batch_stats,
compute_defrag_latency,
};
let mut flush_stats = self.flush_stats.write();
flush_stats.count += 1;
flush_stats.max = flush_stats.max.max(&ret);
flush_stats.sum = flush_stats.sum.sum(&ret);
assert_eq!(self.dirty.range(..flush_boundary).count(), 0);
Ok(ret)
}
}
fn initialize(
recovered_nodes: &[ObjectRecovery],
heap: &Heap,
) -> (
ConcurrentMap<
ObjectId,
Object
