Showing preview only (1,122K chars total). Download the full file or copy to clipboard to get everything.
Repository: dgraph-io/badger
Branch: main
Commit: 796cb85f662c
Files: 136
Total size: 1.1 MB
Directory structure:
gitextract_mc7da0i7/
├── .github/
│ ├── CODEOWNERS
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ ├── config.yml
│ │ └── feature_request.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ ├── renovate.json
│ └── workflows/
│ ├── cd-badger.yml
│ ├── ci-badger-bank-tests-nightly.yml
│ ├── ci-badger-bank-tests.yml
│ ├── ci-badger-tests.yml
│ ├── ci-dgraph-tests.yml
│ └── trunk.yml
├── .gitignore
├── .trunk/
│ ├── .gitignore
│ ├── configs/
│ │ ├── .checkov.yaml
│ │ ├── .markdownlint.json
│ │ ├── .prettierrc
│ │ ├── .shellcheckrc
│ │ ├── .yamllint.yaml
│ │ └── svgo.config.mjs
│ └── trunk.yaml
├── .vscode/
│ ├── extensions.json
│ └── settings.json
├── CHANGELOG.md
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── Makefile
├── README.md
├── SECURITY.md
├── VERSIONING.md
├── backup.go
├── backup_test.go
├── batch.go
├── batch_test.go
├── changes.sh
├── compaction.go
├── contrib/
│ └── RELEASE.md
├── db.go
├── db2_test.go
├── db_test.go
├── dir_aix.go
├── dir_other.go
├── dir_plan9.go
├── dir_unix.go
├── dir_windows.go
├── discard.go
├── discard_test.go
├── doc.go
├── docs/
│ ├── design.md
│ ├── encryption-at-rest.md
│ ├── index.md
│ ├── quickstart.md
│ └── troubleshooting.md
├── errors.go
├── fb/
│ ├── BlockOffset.go
│ ├── TableIndex.go
│ ├── flatbuffer.fbs
│ ├── gen.sh
│ └── install_flatbuffers.sh
├── go.mod
├── go.sum
├── histogram.go
├── histogram_test.go
├── integration/
│ └── testgc/
│ ├── .gitignore
│ └── main.go
├── iterator.go
├── iterator_test.go
├── key_registry.go
├── key_registry_test.go
├── level_handler.go
├── levels.go
├── levels_test.go
├── logger.go
├── logger_test.go
├── managed_db.go
├── managed_db_test.go
├── manifest.go
├── manifest_test.go
├── memtable.go
├── merge.go
├── merge_test.go
├── metrics_test.go
├── options/
│ └── options.go
├── options.go
├── options_test.go
├── pb/
│ ├── badgerpb4.pb.go
│ ├── badgerpb4.proto
│ ├── gen.sh
│ └── protos_test.go
├── publisher.go
├── publisher_test.go
├── skl/
│ ├── README.md
│ ├── arena.go
│ ├── skl.go
│ └── skl_test.go
├── stream.go
├── stream_test.go
├── stream_writer.go
├── stream_writer_test.go
├── structs.go
├── structs_test.go
├── table/
│ ├── README.md
│ ├── builder.go
│ ├── builder_test.go
│ ├── iterator.go
│ ├── merge_iterator.go
│ ├── merge_iterator_test.go
│ ├── table.go
│ └── table_test.go
├── test.sh
├── test_extensions.go
├── trie/
│ ├── trie.go
│ └── trie_test.go
├── txn.go
├── txn_test.go
├── util.go
├── value.go
├── value_test.go
├── watermark_edge_test.go
└── y/
├── bloom.go
├── bloom_test.go
├── checksum.go
├── checksum_test.go
├── encrypt.go
├── encrypt_test.go
├── error.go
├── error_test.go
├── file_dsync.go
├── file_nodsync.go
├── iterator.go
├── metrics.go
├── watermark.go
├── y.go
├── y_test.go
└── zstd.go
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/CODEOWNERS
================================================
# CODEOWNERS info: https://help.github.com/en/articles/about-code-owners
# Owners are automatically requested for review for PRs that changes code
# that they own.
* @dgraph-io/maintainers
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.md
================================================
---
name: Bug report
about: Create a report to help us improve
title: ""
labels: bug
assignees: ""
---
## Describe the bug
A clear and concise description of what the bug is.
## To Reproduce
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
## Expected behavior
A clear and concise description of what you expected to happen.
## Screenshots
If applicable, add screenshots to help explain your problem.
## Environment
- OS: [e.g. macOS, Windows, Ubuntu]
- Language [e.g. AssemblyScript, Go]
- Version [e.g. v0.xx]
## Additional context
Add any other context about the problem here.
================================================
FILE: .github/ISSUE_TEMPLATE/config.yml
================================================
blank_issues_enabled: false
contact_links:
- name: Badger Community Support
url: https://github.com/orgs/dgraph-io/discussions
about: Please ask and answer questions here
================================================
FILE: .github/ISSUE_TEMPLATE/feature_request.md
================================================
---
name: Feature request
about: Suggest an idea for this project
title: ""
labels: ""
assignees: ""
---
## Is your feature request related to a problem? Please describe
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
## Describe the solution you'd like
A clear and concise description of what you want to happen.
## Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
## Additional context
Add any other context or screenshots about the feature request here.
================================================
FILE: .github/PULL_REQUEST_TEMPLATE.md
================================================
**Description**
Please explain the changes you made here.
**Checklist**
- [ ] Code compiles correctly and linting passes locally
- [ ] Tests added for new functionality, or regression tests for bug fixes added as applicable
**Instructions**
- The PR title should follow the [Conventional Commits](https://www.conventionalcommits.org/)
syntax, leading with `fix:`, `feat:`, `chore:`, `ci:`, etc.
- The description should briefly explain what the PR is about. In the case of a bugfix, describe or
link to the bug.
- In the checklist section, check the boxes in that are applicable, using `[x]` syntax.
- If not applicable, remove the entire line. Only leave the box unchecked if you intend to come
back and check the box later.
- Delete the `Instructions` line and everything below it, to indicate you have read and are
following these instructions. 🙂
Thank you for your contribution to Badger!
================================================
FILE: .github/renovate.json
================================================
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["local>dgraph-io/renovate-config"],
"rangeStrategy": "widen",
"packageRules": [
{ "matchManagers": ["gomod"], "matchPackageNames": ["go"], "enabled": false },
{ "matchManagers": ["gomod"], "matchDepNames": ["go"], "enabled": false }
],
"ignoreDeps": ["go"]
}
================================================
FILE: .github/workflows/cd-badger.yml
================================================
name: cd-badger
on:
workflow_dispatch:
inputs:
releasetag:
description: releasetag
required: true
type: string
permissions:
contents: read
jobs:
badger-build-amd64:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
ref: "${{ github.event.inputs.releasetag }}"
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
- name: Set Badger Release Version
run: |
#!/bin/bash
GIT_TAG_NAME='${{ github.event.inputs.releasetag }}'
if [[ "$GIT_TAG_NAME" == "v"* ]];
then
echo "this is a release tag"
else
echo "this is NOT a release tag"
exit 1
fi
BADGER_RELEASE_VERSION='${{ github.event.inputs.releasetag }}'
echo "making a new release for "$BADGER_RELEASE_VERSION
echo "BADGER_RELEASE_VERSION=$BADGER_RELEASE_VERSION" >> $GITHUB_ENV
- name: Fetch dependencies
run: sudo apt-get update && sudo apt-get -y install build-essential
- name: Build badger linux/amd64
run: make badger
- name: Generate SHA for Linux Build
run:
cd badger && sha256sum badger-linux-amd64 | cut -c-64 > badger-checksum-linux-amd64.sha256
- name: Tar Archive for Linux Build
run: cd badger && tar -zcvf badger-linux-amd64.tar.gz badger-linux-amd64
- name: Upload Badger Binary Build Artifacts
uses: actions/upload-artifact@v4
with:
name: badger-linux-amd64-${{ github.run_id }}-${{ github.job }}
path: |
badger/badger-checksum-linux-amd64.sha256
badger/badger-linux-amd64.tar.gz
badger-build-arm64:
runs-on: ubuntu-24.04-arm
steps:
- uses: actions/checkout@v5
with:
ref: "${{ github.event.inputs.releasetag }}"
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
- name: Set Badger Release Version
run: |
#!/bin/bash
GIT_TAG_NAME='${{ github.event.inputs.releasetag }}'
if [[ "$GIT_TAG_NAME" == "v"* ]];
then
echo "this is a release tag"
else
echo "this is NOT a release tag"
exit 1
fi
BADGER_RELEASE_VERSION='${{ github.event.inputs.releasetag }}'
echo "making a new release for "$BADGER_RELEASE_VERSION
echo "BADGER_RELEASE_VERSION=$BADGER_RELEASE_VERSION" >> $GITHUB_ENV
- name: Fetch dependencies
run: sudo apt-get -y install build-essential
- name: Build badger linux/arm64
run: make badger
- name: Generate SHA for Linux Build
run:
cd badger && sha256sum badger-linux-arm64 | cut -c-64 > badger-checksum-linux-arm64.sha256
- name: Tar Archive for Linux Build
run: cd badger && tar -zcvf badger-linux-arm64.tar.gz badger-linux-arm64
- name: List Artifacts
run: ls -al badger/
- name: Upload Badger Binary Build Artifacts
uses: actions/upload-artifact@v4
with:
name: badger-linux-arm64-${{ github.run_id }}-${{ github.job }}
path: |
badger/badger-checksum-linux-arm64.sha256
badger/badger-linux-arm64.tar.gz
================================================
FILE: .github/workflows/ci-badger-bank-tests-nightly.yml
================================================
name: ci-badger-bank-tests-nightly
on:
push:
paths-ignore:
- "**.md"
- docs/**
- images/**
branches:
- main
- release/v*
schedule:
- cron: 1 3 * * *
permissions:
contents: read
jobs:
badger-bank:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Setup Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
- name: Install Dependencies
run: make dependency
- name: Install jemalloc
run: make jemalloc
- name: Install Badger
run: cd badger && go install --race --tags=jemalloc .
- name: Run Badger Bank Test
run: |
#!/bin/bash -x
set -o pipefail
# get 16 random bytes from /dev/urandom
hexdump -vn16 -e'4/4 "%08X" 1 "\n"' /dev/urandom > badgerkey16bytes
badger bank test --dir=. --encryption-key "badgerkey16bytes" -d=4h 2>&1 | tee badgerbanktest.log | grep -v 'Moved $5'
if [ $? -ne 0 ]; then
if grep -qi 'data race' badgerbanktest.log; then
echo "Detected data race via grep..."
cat badgerbanktest.log | grep -v 'Moved $5'
else
echo "No data race detected via grep. Assuming txn violation..."
tail -1000 badgerbanktest.log
badger bank disect --dir=. --decryption-key "badgerkey16bytes"
fi
exit 1
fi
echo 'Bank test finished with no issues.'
================================================
FILE: .github/workflows/ci-badger-bank-tests.yml
================================================
name: ci-badger-bank-tests
on:
workflow_dispatch: # allows manual trigger from GitHub
pull_request:
paths-ignore:
- "**.md"
- docs/**
- images/**
branches:
- main
- release/v*
permissions:
contents: read
jobs:
badger-bank:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Setup Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
- name: Install Dependencies
run: make dependency
- name: Install jemalloc
run: make jemalloc
- name: Install Badger
run: cd badger && go install --race --tags=jemalloc .
- name: Run Badger Bank Test
run: |
#!/bin/bash
mkdir bank && cd bank
badger bank test -v --dir=. -d=20m
================================================
FILE: .github/workflows/ci-badger-tests.yml
================================================
name: ci-badger-tests
on:
workflow_dispatch: # allows manual trigger from GitHub
pull_request:
paths-ignore:
- "**.md"
- docs/**
- images/**
- contrib/**
branches:
- main
- release/v*
permissions:
contents: read
jobs:
cross-compile:
runs-on: ubuntu-latest
strategy:
matrix:
include:
- goos: linux
goarch: amd64
- goos: linux
goarch: arm64
- goos: darwin
goarch: amd64
- goos: darwin
goarch: arm64
- goos: windows
goarch: amd64
- goos: aix
goarch: ppc64
- goos: plan9
goarch: amd64
steps:
- uses: actions/checkout@v5
- name: Setup Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
- name: Cross-compile for ${{ matrix.goos }}/${{ matrix.goarch }}
env:
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
run: go build ./...
badger-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- name: Setup Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
- name: Install Dependencies
run: make dependency
- name: Run Badger Tests
run: make test
================================================
FILE: .github/workflows/ci-dgraph-tests.yml
================================================
name: ci-dgraph-tests
on:
push:
paths-ignore:
- "**.md"
- docs/**
- images/**
branches:
- main
permissions:
contents: read
jobs:
dgraph-tests:
runs-on: ubuntu-latest
steps:
- name: Checkout Dgraph repo
uses: actions/checkout@v5
with:
repository: dgraph-io/dgraph
ref: main
- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
- name: Install gotestsum
run: go install gotest.tools/gotestsum@latest
- name: Fetch latest Badger version
run: |
go get github.com/dgraph-io/badger/v4@main
- name: Set up Node
uses: actions/setup-node@v5
with:
node-version: 16 || 22
- name: Install protobuf-compiler
run: sudo apt update && sudo apt install -y protobuf-compiler
- name: Check protobuf
run: |
cd ./protos
go mod tidy
make regenerate
git diff --exit-code -- .
- name: Make Linux Build and Docker Image
run: make docker-image
- name: Build Test Binary
run: |
#!/bin/bash
# build the test binary
cd t; go build .
- name: Clean Up Environment
run: |
#!/bin/bash
# clean cache
go clean -testcache
# clean up docker containers before test execution
cd t; ./t -r
- name: Run Unit Tests
run: |
#!/bin/bash
# go env settings
export GOPATH=~/go
# move the binary
cp dgraph/dgraph ~/go/bin/dgraph
# run the tests
cd t; ./t --pkg=edgraph,posting,worker,query
# clean up docker containers after test execution
./t -r
# sleep
sleep 5
================================================
FILE: .github/workflows/trunk.yml
================================================
name: Trunk Code Quality
on:
pull_request:
branches: main
permissions:
contents: read
actions: write
checks: write
jobs:
trunk-code-quality:
name: Trunk Code Quality
uses: dgraph-io/.github/.github/workflows/trunk.yml@main
================================================
FILE: .gitignore
================================================
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
badger/badger-*
# Test binary, build with `go test -c`
*.test
badger-test*/
# Output of the go coverage tool
*.out
#darwin
.DS_Store
================================================
FILE: .trunk/.gitignore
================================================
*out
*logs
*actions
*notifications
*tools
plugins
user_trunk.yaml
user.yaml
tmp
================================================
FILE: .trunk/configs/.checkov.yaml
================================================
skip-check:
- CKV_GHA_7
================================================
FILE: .trunk/configs/.markdownlint.json
================================================
{
"line-length": { "line_length": 150, "tables": false },
"no-inline-html": false,
"no-bare-urls": false,
"no-space-in-emphasis": false,
"no-emphasis-as-heading": false,
"first-line-heading": false
}
================================================
FILE: .trunk/configs/.prettierrc
================================================
{
"semi": false,
"proseWrap": "always",
"printWidth": 100
}
================================================
FILE: .trunk/configs/.shellcheckrc
================================================
enable=all
source-path=SCRIPTDIR
disable=SC2154
# If you're having issues with shellcheck following source, disable the errors via:
# disable=SC1090
# disable=SC1091
================================================
FILE: .trunk/configs/.yamllint.yaml
================================================
rules:
quoted-strings:
required: only-when-needed
extra-allowed: ["{|}"]
key-duplicates: {}
octal-values:
forbid-implicit-octal: true
================================================
FILE: .trunk/configs/svgo.config.mjs
================================================
export default {
plugins: [
{
name: "preset-default",
params: {
overrides: {
removeViewBox: false, // https://github.com/svg/svgo/issues/1128
sortAttrs: true,
removeOffCanvasPaths: true,
},
},
},
],
}
================================================
FILE: .trunk/trunk.yaml
================================================
# This file controls the behavior of Trunk: https://docs.trunk.io/cli
# To learn more about the format of this file, see https://docs.trunk.io/reference/trunk-yaml
version: 0.1
cli:
version: 1.25.0
plugins:
sources:
- id: trunk
ref: v1.7.4
uri: https://github.com/trunk-io/plugins
runtimes:
enabled:
- go@1.25.5
lint:
ignore:
- linters: [ALL]
paths:
- pb/*.pb.go
enabled:
- golangci-lint2@2.4.0
- trivy@0.64.1
- renovate@41.76.0
- actionlint@1.7.7
- checkov@3.2.461
- git-diff-check
- gofmt@1.20.4
- golangci-lint@1.64.8
- markdownlint@0.45.0
- osv-scanner@2.0.3
- oxipng@9.1.5
- prettier@3.6.2
- shellcheck@0.10.0
- shfmt@3.6.0
- svgo@4.0.0
- taplo@0.9.3
- trufflehog@3.90.5
- yamllint@1.37.1
actions:
enabled:
- trunk-announce
- trunk-check-pre-push
- trunk-fmt-pre-commit
- trunk-upgrade-available
================================================
FILE: .vscode/extensions.json
================================================
{
"recommendations": ["trunk.io"]
}
================================================
FILE: .vscode/settings.json
================================================
{
"editor.formatOnSave": true,
"editor.defaultFormatter": "trunk.io",
"editor.trimAutoWhitespace": true,
"trunk.autoInit": false
}
================================================
FILE: CHANGELOG.md
================================================
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
## [4.9.1] - 2026-02-04
**Fixed**
- fix(aix): add aix directory synchronization support (#2115)
- fix: correct the comment on value size in skl.node (#2250)
**Tests**
- test: add checksum tests for package y (#2246)
**Chores**
- chore(ci): update arm runner label (#2248)
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.9.0...v4.9.1
## [4.9.0] - 2025-12-15
**Fixed**
- fix(y): y.SafeCopy shall always return empty slice rather than nil (#2245)
> **WARNING** SafeCopy now returns an empty slice rather than nil. For those using our `y` utility
> package, this could be a breaking change. This has implications for empty slices stored in
> badger, specifically, upon retrieval the value stored with the key will be equal to what was set
> (an empty []byte). See #2067 for more details.
- fix: test.sh error (#2225)
- fix: typo of abandoned (#2222)
**Docs**
- add doc for encryption at rest (#2240)
- move docs pages in the repo (#2232)
**Chores**
- chore(ci): restrict Dgraph test to core packages only (#2242)
- chore: update README.md with correct links and badges (#2239)
- chore: change renovate to maintain backwards compatible go version (#2236)
- chore: configure renovate to leave go version as declared (#2235)
- chore(deps): Update actions (major) (#2229)
- chore(deps): Update actions/checkout action to v5 (#2221)
- chore(deps): Update go minor and patch (#2218)
- chore: update the trunk conf file (#2217)
- chore(deps): Update dependency node to v22 (#2219)
- chore(deps): Update go minor and patch (#2212)
**CI**
- move to GitHub Actions runners
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.8.0...v4.8.1
## [4.8.0] - 2025-07-15
**Features**
- feat(stream): Update stream framework with new alternate keyToList function (#2211)
**Fixed**
- fix: crash loop on missing manifest tables (#2198)
**Chores**
- chore(deps): Update module golang.org/x/sys to v0.34.0 (#2210)
- chore(deps): Update go minor and patch (#2208)
- chore(deps): Update go minor and patch (#2204)
- chore(deps): Update go minor and patch (#2202)
- chore(deps): Update go minor and patch (#2200)
- chore(deps): Update module golang.org/x/sys to v0.33.0 (#2195)
- chore(deps): Update go minor and patch (#2189)
- Compile with jemalloc v5.3.0 (#2191)
**CI**
- Update trunk.yml
- move Trunk to action
**Docs**
- docs: add new badge (#2194)
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.7.0...v4.8.0
## [4.7.0] - 2025-04-08
**Chores**
- chore(deps): remove dependency on github.com/pkg/errors (#2184)
- chore(deps): Update go minor and patch (#2187)
- chore(deps): Update go minor and patch (#2181)
- chore(deps): Update module golang.org/x/sys to v0.31.0 (#2179)
**Fixed**
- fix broken badge (#2186)
**Docs**
- Update README.md
- doc: add Blink Labs projects to the using Badger list (#2183)
- doc: add FlowG to "Projects Using Badger" section of the README (#2180)
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.6.0...v4.7.0
## [4.6.0] - 2025-02-26
**Chores**
- chore(deps): Migrate from OpenCensus to OpenTelemetry (#2169)
- chore(deps): Update go minor and patch (#2177)
- chore(deps): Update module github.com/spf13/cobra to v1.9.0 (#2174)
- chore: add editor config
- update .gitignore (#2176)
**Fixed**
- fix: remove accidentally uploaded binary `badger-darwin-arm64` (#2175)
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.5.2...v4.6.0
## [4.5.2] - 2025-02-14
**Chores**
- chore(deps): Update go minor and patch (#2168)
- chore(deps): bump minimum Go support to 1.22 (#2171)
- chore: migrate docs to centralized docs repo (#2166)
- chore: align repo conventions (#2158)
- chore(deps): bump the patch group with 2 updates (#2156)
- chore(deps): bump github.com/google/flatbuffers from 24.12.23+incompatible to 25.1.21+incompatible
(#2153)
- chore(deps): bump golangci/golangci-lint-action from 6.1.1 to 6.2.0 in the actions group (#2154)
- Update renovate.json
- Update trunk.yaml
- enable Trivy
**Fixed**
- update docs link in error message (#2170)
- Revert "Update badgerpb4.pb.go" (#2172)
**Docs**
- Update README.md
- Added my project that uses Badger database (#2157)
- Create SECURITY.md
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.5.1...v4.5.2
## [4.5.1] - 2025-01-21
- chore(deps): bump google.golang.org/protobuf from 1.36.2 to 1.36.3 in the patch group (#2150)
- bump github.com/dgraph-io/ristretto/v2 from 2.0.1 to 2.1.0 in the minor group (#2151)
- feat(info): print total size of listed keys (#2149)
- chore(deps): bump google.golang.org/protobuf from 1.36.1 to 1.36.2 in the patch group (#2146)
- chore(deps): bump the minor group with 2 updates (#2147)
- fix(info): print Total BloomFilter Size with totalBloomFilter instead of totalIndex (#2145)
- chore(deps): bump the minor group with 2 updates (#2141)
- chore(deps): bump google.golang.org/protobuf from 1.36.0 to 1.36.1 in the patch group (#2140)
- chore(deps): bump google.golang.org/protobuf from 1.35.2 to 1.36.0 in the minor group (#2139)
- chore(deps): bump github.com/dgraph-io/ristretto/v2 from 2.0.0 to 2.0.1 in the patch group (#2136)
- chore(deps): bump golang.org/x/net from 0.31.0 to 0.32.0 in the minor group (#2137)
- chore(deps): bump the minor group with 2 updates (#2135)
- docs: Add pagination explanation to docs (#2134)
- Fix build for GOARCH=wasm with GOOS=js or GOOS=wasip1 (#2048)
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.5.0...v4.5.1
## [4.5.0] - 2024-11-29
- fix the cd pipeline by @mangalaman93 in https://github.com/dgraph-io/badger/pull/2127
- chore(deps): bump the minor group with 2 updates by @dependabot in
https://github.com/dgraph-io/badger/pull/2128
- chore(deps): bump github.com/stretchr/testify from 1.9.0 to 1.10.0 in the minor group by
@dependabot in https://github.com/dgraph-io/badger/pull/2130
- upgrade protobuf library by @shivaji-kharse in https://github.com/dgraph-io/badger/pull/2131
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.4.0...v4.5.0
## [4.4.0] - 2024-10-26
- retract v4.3.0 due to #2121 and #2113, upgrade to Go v1.23, use ristretto v2 in
https://github.com/dgraph-io/badger/pull/2122
- Allow stream custom maxsize per batch in https://github.com/dgraph-io/badger/pull/2063
- chore(deps): bump github.com/klauspost/compress from 1.17.10 to 1.17.11 in the patch group in
https://github.com/dgraph-io/badger/pull/2120
- fix: sentinel errors should not have stack traces in https://github.com/dgraph-io/badger/pull/2042
- chore(deps): bump the minor group with 2 updates in https://github.com/dgraph-io/badger/pull/2119
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.3.1...v4.4.0
## [4.3.1] - 2024-10-06
- chore: update docs links by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2097
- chore(deps): bump golang.org/x/sys from 0.24.0 to 0.25.0 in the minor group by @dependabot in
https://github.com/dgraph-io/badger/pull/2100
- chore(deps): bump golang.org/x/net from 0.28.0 to 0.29.0 in the minor group by @dependabot in
https://github.com/dgraph-io/badger/pull/2106
- fix: fix reverse iterator broken by seek by @harshil-goel in
https://github.com/dgraph-io/badger/pull/2109
- chore(deps): bump github.com/klauspost/compress from 1.17.9 to 1.17.10 in the patch group by
@dependabot in https://github.com/dgraph-io/badger/pull/2114
- chore(deps): bump github.com/dgraph-io/ristretto from 0.1.2-0.20240116140435-c67e07994f91 to 1.0.0
by @dependabot in https://github.com/dgraph-io/badger/pull/2112
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.3.0...v4.3.1
## [4.3.0] - 2024-08-29
> **Warning** The tag v4.3.0 has been retracted due to an issue go.sum. Use v4.3.1 (see #2121 and
> #2113)
**Fixes**
- chore(changelog): add a missed entry in CHANGELOG for v4.2.0 by @mangalaman93 in
https://github.com/dgraph-io/badger/pull/1988
- update README with project KVS using badger by @tauraamui in
https://github.com/dgraph-io/badger/pull/1989
- fix edge case for watermark when index is zero by @mangalaman93 in
https://github.com/dgraph-io/badger/pull/1999
- upgrade spf13/cobra to version v1.7.0 by @mangalaman93 in
https://github.com/dgraph-io/badger/pull/2001
- chore: update readme by @joshua-goldstein in https://github.com/dgraph-io/badger/pull/2011
- perf: upgrade compress package test and benchmark. by @siddhant2001 in
https://github.com/dgraph-io/badger/pull/2009
- fix(Transactions): Fix resource consumption on empty write transaction by @Zach-Johnson in
https://github.com/dgraph-io/badger/pull/2018
- chore(deps): bump golang.org/x/net from 0.7.0 to 0.17.0 by @dependabot in
https://github.com/dgraph-io/badger/pull/2017
- perf(compactor): optimize allocations: use buffer for priorities by @deff7 in
https://github.com/dgraph-io/badger/pull/2006
- fix(Transaction): discard empty transactions on CommitWith by @Wondertan in
https://github.com/dgraph-io/badger/pull/2031
- fix(levelHandler): use lock for levelHandler sort tables instead of rlock by @xgzlucario in
https://github.com/dgraph-io/badger/pull/2034
- Docs: update README with project LLS using badger by @Boc-chi-no in
https://github.com/dgraph-io/badger/pull/2032
- chore: MaxTableSize has been renamed to BaseTableSize by @mitar in
https://github.com/dgraph-io/badger/pull/2038
- Update CODEOWNERS by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2043
- Chore(): add Stale Action by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2070
- Update ristretto and refactor for use of generics by @paralin in
https://github.com/dgraph-io/badger/pull/2047
- chore: Remove obsolete comment by @mitar in https://github.com/dgraph-io/badger/pull/2039
- chore(Docs): Update jQuery 3.2.1 to 3.7.1 by @kokizzu in
https://github.com/dgraph-io/badger/pull/2023
- chore(deps): bump the go_modules group with 3 updates by @dependabot in
https://github.com/dgraph-io/badger/pull/2074
- docs(): update docs path by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2076
- perf: fix operation in seek by @harshil-goel in https://github.com/dgraph-io/badger/pull/2077
- Add lakeFS to README.md by @N-o-Z in https://github.com/dgraph-io/badger/pull/2078
- chore(): add Dependabot by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2080
- chore(deps): bump golangci/golangci-lint-action from 4 to 6 by @dependabot in
https://github.com/dgraph-io/badger/pull/2083
- chore(deps): bump actions/upload-artifact from 3 to 4 by @dependabot in
https://github.com/dgraph-io/badger/pull/2081
- chore(deps): bump github/codeql-action from 2 to 3 by @dependabot in
https://github.com/dgraph-io/badger/pull/2082
- chore(deps): bump the minor group with 7 updates by @dependabot in
https://github.com/dgraph-io/badger/pull/2089
- Action Manager by @madhu72 in https://github.com/dgraph-io/badger/pull/2050
- chore(deps): bump golang.org/x/sys from 0.23.0 to 0.24.0 in the minor group by @dependabot in
https://github.com/dgraph-io/badger/pull/2091
- chore(deps): bump github.com/golang/protobuf from 1.5.3 to 1.5.4 in the patch group by @dependabot
in https://github.com/dgraph-io/badger/pull/2090
- chore: fix some comments by @dufucun in https://github.com/dgraph-io/badger/pull/2092
- chore(deps): bump github.com/google/flatbuffers from 1.12.1 to 24.3.25+incompatible by @dependabot
in https://github.com/dgraph-io/badger/pull/2084
**CI**
- ci: change cron frequency to fix ghost jobs by @joshua-goldstein in
https://github.com/dgraph-io/badger/pull/2010
- fix(CI): Update to pull_request trigger by @ryanfoxtyler in
https://github.com/dgraph-io/badger/pull/2056
- ci/cd optimization by @ryanfoxtyler in https://github.com/dgraph-io/badger/pull/2051
- fix(cd): fixed cd pipeline by @harshil-goel in https://github.com/dgraph-io/badger/pull/2093
- fix(cd): change name by @harshil-goel in https://github.com/dgraph-io/badger/pull/2094
- fix(cd): added more debug things to cd by @harshil-goel in
https://github.com/dgraph-io/badger/pull/2095
- fix(cd): removing some debug items by @harshil-goel in
https://github.com/dgraph-io/badger/pull/2096
**Full Changelog**: https://github.com/dgraph-io/badger/compare/v4.2.0...v4.3.0
## [4.2.0] - 2023-08-03
**Breaking**
- feat(metrics): fix and update metrics in badger (#1948)
- fix(metrics): remove badger version in the metrics name (#1982)
**Fixed**
- fix(db): avoid panic in parallel reads after closing DB (#1987)
- fix(logging): fix direct access to logger (#1980)
- fix(sec): bump google.golang.org/grpc from 1.20.1 to 1.53.0 (#1977)
- fix(sec): update gopkg.in/yaml.v2 package (#1969)
- fix(test): fix flakiness of TestPersistLFDiscardStats (#1963)
- fix(stream): setup oracle correctly in stream writer (#1968) (#1904)
- fix(stream): improve incremental stream writer (#1901)
- fix(test): improve the params in BenchmarkDbGrowth (#1967)
- fix(sync): sync active memtable and value log on Db.Sync (#1847) (#1953)
- fix(test): handle draining of closed channel, speed up test. (#1957)
- fix(test): fix table checksum test. Test on uncompressed. (#1952)
- fix(level): change split key range right key to use ts=0 (#1932)
- fix(test): the new test case PagebufferReader5 introduced an error. (#1936)
- fix(test): add missing unlock in TestPersistLFDiscardStats (#1924)
- fix(PageBufferReader): should conform to io.Reader interface (#1935)
- fix(publisher): publish updates after persistence in WAL (#1917)
**CI**
- chore(ci): split off coverage workflow (#1944)
- chore(ci): adding trivy scanning workflow (#1925)
## [4.1.0] - 2023-03-30
This release adds support for incremental stream writer. We also do some cleanup in the docs and
resolve some CI issues for community PR's. We resolve high and medium CVE's and fix
[#1833](https://github.com/dgraph-io/badger/issues/1833).
**Features**
- feat(stream): add support for incremental stream writer (#1722) (#1874)
**Fixes**
- chore: upgrade xxhash from v1.1.0 to v2.1.2 (#1910) (fixes
[#1833](https://github.com/dgraph-io/badger/issues/1833))
**Security**
- chore(deps): bump golang.org/x/net from 0.0.0-20201021035429-f5854403a974 to 0.7.0 (#1885)
**CVEs**
- [CVE-2021-31525](https://github.com/dgraph-io/badger/security/dependabot/7)
- [CVE-2022-41723](https://github.com/dgraph-io/badger/security/dependabot/4)
- [CVE-2022-27664](https://github.com/dgraph-io/badger/security/dependabot/5)
- [CVE-2021-33194](https://github.com/dgraph-io/badger/security/dependabot/9)
- [CVE-2022-41723](https://github.com/dgraph-io/badger/security/dependabot/13)
- [CVE-2021-33194](https://github.com/dgraph-io/badger/security/dependabot/16)
- [CVE-2021-38561](https://github.com/dgraph-io/badger/security/dependabot/8)
**Chores**
- fix(docs): update README (#1915)
- cleanup sstable file after tests (#1912)
- chore(ci): add dgraph regression tests (#1908)
- docs: fix the default value in docs (#1909)
- chore: update URL for unsupported manifest version error (#1905)
- docs(README): add raft-badger to projects using badger (#1902)
- sync the docs with README with projects using badger (#1903)
- fix: update code comments for WithNumCompactors (#1900)
- docs: add loggie to projects using badger (#1882)
- chore(memtable): refactor code for memtable flush (#1866)
- resolve coveralls issue for community PR's (#1892, #1894, #1896)
## [4.0.1] - 2023-02-28
We issue a follow up release in order to resolve a bug in subscriber. We also generate updated
protobufs for Badger v4.
**Fixed**
- fix(pb): fix generated protos #1888
- fix(publisher): initialize the atomic variable #1889
**Chores**
- chore(cd): tag based deployments #1887
- chore(ci): fail fast when testing #1890
## [4.0.0] - 2023-02-27
> **Warning** The tag v4.0.0 has been retracted due to a bug in publisher. Use v4.0.1 (see #1889)
This release fixes a bug in the maxHeaderSize parameter that could lead to panics. We introduce an
external magic number to keep track of external dependencies. We bump up the minimum required Go
version to 1.19. No changes were made to the format of data on disk. This is a major release because
we are making a switch to SemVer in order to make it easier for the community to understand when
breaking API and data format changes are made.
**Fixed**
- fix: update maxHeaderSize #1877
- feat(externalMagic): Introduce external magic number (#1745) #1852
- fix(bench): bring in benchmark fixes from main #1863
**Chores**
- upgrade go to 1.19 #1868
- enable linters (gosimple, govet, lll, unused, staticcheck, errcheck, ineffassign, gofmt) #1871
#1870 #1876
- remove dependency on io/ioutil #1879
- various doc and comment fixes #1857
- moving from CalVer to SemVer
## [3.2103.5] - 2022-12-15
We release Badger CLI tool binaries for amd64 and now arm64. This release does not involve any core
code changes to Badger. We add a CD job for building Badger for arm64.
## [3.2103.4] - 2022-11-04
**Fixed**
- fix(manifest): fix manifest corruption due to race condition in concurrent compactions (#1756)
**Chores**
- We bring the release branch to parity with main by updating the CI/CD jobs, Readme, Codeowners, PR
and issue templates, etc.
## [3.2103.3] - 2022-10-14
**Remarks**
- This is a minor patch release that fixes arm64 related issues. The issues in the `z` package in
Ristretto were resolved in Ristretto v0.1.1.
**Fixed**
- fix(arm64): bump ristretto v0.1.0 --> v0.1.1 (#1806)
## [3.2103.2] - 2021-10-07
**Fixed**
- fix(compact): close vlog after the compaction at L0 has completed (#1752)
- fix(builder): put the upper limit on reallocation (#1748)
- deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
- fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
- fix(pubsub): avoid deadlock in publisher and subscriber (#1749) (#1751)
## [3.2103.1] - 2021-07-08
**Fixed**
- fix(compaction): copy over the file ID when building tables #1713
- fix: Fix conflict detection for managed DB (#1716)
- fix(pendingWrites): don't skip the pending entries with version=0 (#1721)
**Features**
- feat(zstd): replace datadog's zstd with Klauspost's zstd (#1709)
## [3.2103.0] - 2021-06-02
**Breaking**
- Subscribe: Add option to subscribe with holes in prefixes. (#1658)
**Fixed**
- fix(compaction): Remove compaction backoff mechanism (#1686)
- Add a name to mutexes to make them unexported (#1678)
- fix(merge-operator): don't read the deleted keys (#1675)
- fix(discard): close the discard stats file on db close (#1672)
- fix(iterator): fix iterator when data does not exist in read only mode (#1670)
- fix(badger): Do not reuse variable across badger commands (#1624)
- fix(dropPrefix): check properly if the key is present in a table (#1623)
**Performance**
- Opt(Stream): Optimize how we deduce key ranges for iteration (#1687)
- Increase value threshold from 1 KB to 1 MB (#1664)
- opt(DropPrefix): check if there exist some data to drop before dropping prefixes (#1621)
**Features**
- feat(options): allow special handling and checking when creating options from superflag (#1688)
- overwrite default Options from SuperFlag string (#1663)
- Support SinceTs in iterators (#1653)
- feat(info): Add a flag to parse and print DISCARD file (#1662)
- feat(vlog): making vlog threshold dynamic 6ce3b7c (#1635)
- feat(options): add NumGoroutines option for default Stream.numGo (#1656)
- feat(Trie): Working prefix match with holes (#1654)
- feat: add functionality to ban a prefix (#1638)
- feat(compaction): Support Lmax to Lmax compaction (#1615)
**New APIs**
- Badger.DB
- BanNamespace
- BannedNamespaces
- Ranges
- Badger.Options
- FromSuperFlag
- WithNumGoRoutines
- WithNamespaceOffset
- WithVLogPercentile
- Badger.Trie
- AddMatch
- DeleteMatch
- Badger.Table
- StaleDataSize
- Badger.Table.Builder
- AddStaleKey
- Badger.InitDiscardStats
**Removed APIs**
- Badger.DB
- KeySplits
- Badger.Options
- SkipVlog
### Changed APIs
- Badger.DB
- Subscribe
- Badger.Options
- WithValueThreshold
## [3.2011.1] - 2021-01-22
**Fixed**
- Fix(compaction): Set base level correctly after stream (#1651)
- Fix: update ristretto and use filepath (#1652)
- Fix(badger): Do not reuse variable across badger commands (#1650)
- Fix(build): fix 32-bit build (#1646)
- Fix(table): always sync SST to disk (#1645)
## [3.2011.0] - 2021-01-15
This release is not backward compatible with Badger v2.x.x
**Breaking**:
- opt(compactions): Improve compaction performance (#1574)
- Change how Badger handles WAL (#1555)
- feat(index): Use flatbuffers instead of protobuf (#1546)
**Fixed**:
- Fix(GC): Set bits correctly for moved keys (#1619)
- Fix(tableBuilding): reduce scope of valuePointer (#1617)
- Fix(compaction): fix table size estimation on compaction (#1613)
- Fix(OOM): Reuse pb.KVs in Stream (#1609)
- Fix race condition in L0StallMs variable (#1605)
- Fix(stream): Stop produceKVs on error (#1604)
- Fix(skiplist): Remove z.Buffer from skiplist (#1600)
- Fix(readonly): fix the file opening mode (#1592)
- Fix: Disable CompactL0OnClose by default (#1586)
- Fix(compaction): Don't drop data when split overlaps with top tables (#1587)
- Fix(subcompaction): Close builder before throttle.Done (#1582)
- Fix(table): Add onDisk size (#1569)
- Fix(Stream): Only send done markers if told to do so
- Fix(value log GC): Fix a bug which caused value log files to not be GCed.
- Fix segmentation fault when cache sizes are small. (#1552)
- Fix(builder): Too many small tables when compression is enabled (#1549)
- Fix integer overflow error when building for 386 (#1541)
- Fix(writeBatch): Avoid deadlock in commit callback (#1529)
- Fix(db): Handle nil logger (#1534)
- Fix(maxVersion): Use choosekey instead of KeyToList (#1532)
- Fix(Backup/Restore): Keep all versions (#1462)
- Fix(build): Fix nocgo builds. (#1493)
- Fix(cleanup): Avoid truncating in value.Open on error (#1465)
- Fix(compaction): Don't use cache for table compaction (#1467)
- Fix(compaction): Use separate compactors for L0, L1 (#1466)
- Fix(options): Do not implicitly enable cache (#1458)
- Fix(cleanup): Do not close cache before compaction (#1464)
- Fix(replay): Update head for LSM entires also (#1456)
- fix(levels): Cleanup builder resources on building an empty table (#1414)
**Performance**
- perf(GC): Remove move keys (#1539)
- Keep the cheaper parts of the index within table struct. (#1608)
- Opt(stream): Use z.Buffer to stream data (#1606)
- opt(builder): Use z.Allocator for building tables (#1576)
- opt(memory): Use z.Calloc for allocating KVList (#1563)
- opt: Small memory usage optimizations (#1562)
- KeySplits checks tables and memtables when number of splits is small. (#1544)
- perf: Reduce memory usage by better struct packing (#1528)
- perf(tableIterator): Don't do next on NewIterator (#1512)
- Improvements: Manual Memory allocation via Calloc (#1459)
- Various bug fixes: Break up list and run DropAll func (#1439)
- Add a limit to the size of the batches sent over a stream. (#1412)
- Commit does not panic after Finish, instead returns an error (#1396)
- levels: Compaction incorrectly drops some delete markers (#1422)
- Remove vlog file if bootstrap, syncDir or mmap fails (#1434)
**Features**:
- Use opencensus for tracing (#1566)
- Export functions from Key Registry (#1561)
- Allow sizes of block and index caches to be updated. (#1551)
- Add metric for number of tables being compacted (#1554)
- feat(info): Show index and bloom filter size (#1543)
- feat(db): Add db.MaxVersion API (#1526)
- Expose DB options in Badger. (#1521)
- Feature: Add a Calloc based Buffer (#1471)
- Add command to stream contents of DB into another DB. (#1463)
- Expose NumAlloc metrics via expvar (#1470)
- Support fully disabling the bloom filter (#1319)
- Add --enc-key flag in badger info tool (#1441)
**New APIs**
- Badger.DB
- CacheMaxCost (#1551)
- Levels (#1574)
- LevelsToString (#1574)
- Opts (#1521)
- Badger.Options
- WithBaseLevelSize (#1574)
- WithBaseTableSize (#1574)
- WithMemTableSize (#1574)
- Badger.KeyRegistry
- DataKey (#1561)
- LatestDataKey (#1561)
**Removed APIs**
- Badger.Options
- WithKeepL0InMemory (#1555)
- WithLevelOneSize (#1574)
- WithLoadBloomsOnOpen (#1555)
- WithLogRotatesToFlush (#1574)
- WithMaxTableSize (#1574)
- WithTableLoadingMode (#1555)
- WithTruncate (#1555)
- WithValueLogLoadingMode (#1555)
## [2.2007.4] - 2021-08-25
**Fixed**
- Fix build on Plan 9 (#1451) (#1508) (#1738)
**Features**
- feat(zstd): backport replacement of DataDog's zstd with Klauspost's zstd (#1736)
## [2.2007.3] - 2021-07-21
**Fixed**
- fix(maxVersion): Use choosekey instead of KeyToList (#1532) #1533
- fix(flatten): Add --num_versions flag (#1518) #1520
- fix(build): Fix integer overflow on 32-bit architectures #1558
- fix(pb): avoid protobuf warning due to common filename (#1519)
**Features**
- Add command to stream contents of DB into another DB. (#1486)
**New APIs**
- DB.StreamDB
- DB.MaxVersion
## [2.2007.2] - 2020-08-31
**Fixed**
- Compaction: Use separate compactors for L0, L1 (#1466)
- Rework Block and Index cache (#1473)
- Add IsClosed method (#1478)
- Cleanup: Avoid truncating in vlog.Open on error (#1465)
- Cleanup: Do not close cache before compactions (#1464)
**New APIs**
- Badger.DB
- BlockCacheMetrics (#1473)
- IndexCacheMetrics (#1473)
- Badger.Option
- WithBlockCacheSize (#1473)
- WithIndexCacheSize (#1473)
**Removed APIs** [Breaking Changes]
- Badger.DB
- DataCacheMetrics (#1473)
- BfCacheMetrics (#1473)
- Badger.Option
- WithMaxCacheSize (#1473)
- WithMaxBfCacheSize (#1473)
- WithKeepBlockIndicesInCache (#1473)
- WithKeepBlocksInCache (#1473)
## [2.2007.1] - 2020-08-19
**Fixed**
- Remove vlog file if bootstrap, syncDir or mmap fails (#1434)
- levels: Compaction incorrectly drops some delete markers (#1422)
- Replay: Update head for LSM entires also (#1456)
## [2.2007.0] - 2020-08-10
**Fixed**
- Add a limit to the size of the batches sent over a stream. (#1412)
- Fix Sequence generates duplicate values (#1281)
- Fix race condition in DoesNotHave (#1287)
- Fail fast if cgo is disabled and compression is ZSTD (#1284)
- Proto: make badger/v2 compatible with v1 (#1293)
- Proto: Rename dgraph.badger.v2.pb to badgerpb2 (#1314)
- Handle duplicates in ManagedWriteBatch (#1315)
- Ensure `bitValuePointer` flag is cleared for LSM entry values written to LSM (#1313)
- DropPrefix: Return error on blocked writes (#1329)
- Confirm `badgerMove` entry required before rewrite (#1302)
- Drop move keys when its key prefix is dropped (#1331)
- Iterator: Always add key to txn.reads (#1328)
- Restore: Account for value size as well (#1358)
- Compaction: Expired keys and delete markers are never purged (#1354)
- GC: Consider size of value while rewriting (#1357)
- Force KeepL0InMemory to be true when InMemory is true (#1375)
- Rework DB.DropPrefix (#1381)
- Update head while replaying value log (#1372)
- Avoid panic on multiple closer.Signal calls (#1401)
- Return error if the vlog writes exceeds more than 4GB (#1400)
**Performance**
- Clean up transaction oracle as we go (#1275)
- Use cache for storing block offsets (#1336)
**Features**
- Support disabling conflict detection (#1344)
- Add leveled logging (#1249)
- Support entry version in Write batch (#1310)
- Add Write method to batch write (#1321)
- Support multiple iterators in read-write transactions (#1286)
**New APIs**
- Badger.DB
- NewManagedWriteBatch (#1310)
- DropPrefix (#1381)
- Badger.Option
- WithDetectConflicts (#1344)
- WithKeepBlockIndicesInCache (#1336)
- WithKeepBlocksInCache (#1336)
- Badger.WriteBatch
- DeleteAt (#1310)
- SetEntryAt (#1310)
- Write (#1321)
### Changes to Default Options
- DefaultOptions: Set KeepL0InMemory to false (#1345)
- Increase default valueThreshold from 32B to 1KB (#1346)
### Deprecated
- Badger.Option
- WithEventLogging (#1203)
### Reverts
This sections lists the changes which were reverted because of non-reproducible crashes.
- Compress/Encrypt Blocks in the background (#1227)
## [2.0.3] - 2020-03-24
**Fixed**
- Add support for watching nil prefix in subscribe API (#1246)
**Performance**
- Compress/Encrypt Blocks in the background (#1227)
- Disable cache by default (#1257)
**Features**
- Add BypassDirLock option (#1243)
- Add separate cache for bloomfilters (#1260)
**New APIs**
- badger.DB
- BfCacheMetrics (#1260)
- DataCacheMetrics (#1260)
- badger.Options
- WithBypassLockGuard (#1243)
- WithLoadBloomsOnOpen (#1260)
- WithMaxBfCacheSize (#1260)
## [2.0.2] - 2020-03-02
**Fixed**
- Cast sz to uint32 to fix compilation on 32 bit. (#1175)
- Fix checkOverlap in compaction. (#1166)
- Avoid sync in inmemory mode. (#1190)
- Support disabling the cache completely. (#1185)
- Add support for caching bloomfilters. (#1204)
- Fix int overflow for 32bit. (#1216)
- Remove the 'this entry should've caught' log from value.go. (#1170)
- Rework concurrency semantics of valueLog.maxFid. (#1187)
**Performance**
- Use fastRand instead of locked-rand in skiplist. (#1173)
- Improve write stalling on level 0 and 1. (#1186)
- Disable compression and set ZSTD Compression Level to 1. (#1191)
## [2.0.1] - 2020-01-02
**New APIs**
- badger.Options
- WithInMemory (f5b6321)
- WithZSTDCompressionLevel (3eb4e72)
- Badger.TableInfo
- EstimatedSz (f46f8ea)
**Features**
- Introduce in-memory mode in badger. (#1113)
**Fixed**
- Limit manifest's change set size. (#1119)
- Cast idx to uint32 to fix compilation on i386. (#1118)
- Fix request increment ref bug. (#1121)
- Fix windows dataloss issue. (#1134)
- Fix VerifyValueChecksum checks. (#1138)
- Fix encryption in stream writer. (#1146)
- Fix segmentation fault in vlog.Read. (header.Decode) (#1150)
- Fix merge iterator duplicates issue. (#1157)
**Performance**
- Set level 15 as default compression level in Zstd. (#1111)
- Optimize createTable in stream_writer.go. (#1132)
## [2.0.0] - 2019-11-12
**New APIs**
- badger.DB
- NewWriteBatchAt (7f43769)
- CacheMetrics (b9056f1)
- badger.Options
- WithMaxCacheSize (b9056f1)
- WithEventLogging (75c6a44)
- WithBlockSize (1439463)
- WithBloomFalsePositive (1439463)
- WithKeepL0InMemory (ee70ff2)
- WithVerifyValueChecksum (ee70ff2)
- WithCompression (5f3b061)
- WithEncryptionKey (a425b0e)
- WithEncryptionKeyRotationDuration (a425b0e)
- WithChecksumVerificationMode (7b4083d)
**Features**
- Data cache to speed up lookups and iterations. (#1066)
- Data compression. (#1013)
- Data encryption-at-rest. (#1042)
**Fixed**
- Fix deadlock when flushing discard stats. (#976)
- Set move key's expiresAt for keys with TTL. (#1006)
- Fix unsafe usage in Decode. (#1097)
- Fix race condition on db.orc.nextTxnTs. (#1101)
- Fix level 0 GC dataloss bug. (#1090)
- Fix deadlock in discard stats. (#1070)
- Support checksum verification for values read from vlog. (#1052)
- Store entire L0 in memory. (#963)
- Fix table.Smallest/Biggest and iterator Prefix bug. (#997)
- Use standard proto functions for Marshal/Unmarshal and Size. (#994)
- Fix boundaries on GC batch size. (#987)
- VlogSize to store correct directory name to expvar.Map. (#956)
- Fix transaction too big issue in restore. (#957)
- Fix race condition in updateDiscardStats. (#973)
- Cast results of len to uint32 to fix compilation in i386 arch. (#961)
- Making the stream writer APIs goroutine-safe. (#959)
- Fix prefix bug in key iterator and allow all versions. (#950)
- Drop discard stats if we can't unmarshal it. (#936)
- Fix race condition in flushDiscardStats function. (#921)
- Ensure rewrite in vlog is within transactional limits. (#911)
- Fix discard stats moved by GC bug. (#929)
- Fix busy-wait loop in Watermark. (#920)
**Performance**
- Introduce fast merge iterator. (#1080)
- Binary search based table picker. (#983)
- Flush vlog buffer if it grows beyond threshold. (#1067)
- Introduce StreamDone in Stream Writer. (#1061)
- Performance Improvements to block iterator. (#977)
- Prevent unnecessary safecopy in iterator parseKV. (#971)
- Use pointers instead of binary encoding. (#965)
- Reuse block iterator inside table iterator. (#972)
- [breaking/format] Remove vlen from entry header. (#945)
- Replace FarmHash with AESHash for Oracle conflicts. (#952)
- [breaking/format] Optimize Bloom filters. (#940)
- [breaking/format] Use varint for header encoding (without header length). (#935)
- Change file picking strategy in compaction. (#894)
- [breaking/format] Block level changes. (#880)
- [breaking/format] Add key-offset index to the end of SST table. (#881)
## [1.6.0] - 2019-07-01
This is a release including almost 200 commits, so expect many changes - some of them not backward
compatible.
Regarding backward compatibility in Badger versions, you might be interested on reading
[VERSIONING.md](VERSIONING.md).
_Note_: The hashes in parentheses correspond to the commits that impacted the given feature.
**New APIs**
- badger.DB
- DropPrefix (291295e)
- Flatten (7e41bba)
- KeySplits (4751ef1)
- MaxBatchCount (b65e2a3)
- MaxBatchSize (b65e2a3)
- PrintKeyValueHistogram (fd59907)
- Subscribe (26128a7)
- Sync (851e462)
- badger.DefaultOptions() and badger.LSMOnlyOptions() (91ce687)
- badger.Options.WithX methods
- badger.Entry (e9447c9)
- NewEntry
- WithMeta
- WithDiscard
- WithTTL
- badger.Item
- KeySize (fd59907)
- ValueSize (5242a99)
- badger.IteratorOptions
- PickTable (7d46029, 49a49e3)
- Prefix (7d46029)
- badger.Logger (fbb2778)
- badger.Options
- CompactL0OnClose (7e41bba)
- Logger (3f66663)
- LogRotatesToFlush (2237832)
- badger.Stream (14cbd89, 3258067)
- badger.StreamWriter (7116e16)
- badger.TableInfo.KeyCount (fd59907)
- badger.TableManifest (2017987)
- badger.Tx.NewKeyIterator (49a49e3)
- badger.WriteBatch (6daccf9, 7e78e80)
**Modified APIs**
**Breaking**
- badger.DefaultOptions and badger.LSMOnlyOptions are now functions rather than variables (91ce687)
- badger.Item.Value now receives a function that returns an error (439fd46)
- badger.Txn.Commit doesn't receive any params now (6daccf9)
- badger.DB.Tables now receives a boolean (76b5341)
**Features**
- badger.LSMOptions changed values (799c33f)
- badger.DB.NewIterator now allows multiple iterators per RO txn (41d9656)
- badger.Options.TableLoadingMode's new default is options.MemoryMap (6b97bac)
**Removed APIs**
- badger.ManagedDB (d22c0e8)
- badger.Options.DoNotCompact (7e41bba)
- badger.Txn.SetWithX (e9447c9)
**Tools**
- badger bank disect (13db058)
- badger bank test (13db058) --mmap (03870e3)
- badger fill (7e41bba)
- badger flatten (7e41bba)
- badger info --histogram (fd59907) --history --lookup --show-keys --show-meta --with-prefix
(09e9b63) --show-internal (fb2eed9)
- badger benchmark read (239041e)
- badger benchmark write (6d3b67d)
## [1.5.5] - 2019-06-20
- Introduce support for Go Modules
## [1.5.3] - 2018-07-11
Bug Fixes:
- Fix a panic caused due to item.vptr not copying over vs.Value, when looking for a move key.
## [1.5.2] - 2018-06-19
Bug Fixes:
- Fix the way move key gets generated.
- If a transaction has unclosed, or multiple iterators running simultaneously, throw a panic. Every
iterator must be properly closed. At any point in time, only one iterator per transaction can be
running. This is to avoid bugs in a transaction data structure which is thread unsafe.
- _Warning: This change might cause panics in user code. Fix is to properly close your iterators,
and only have one running at a time per transaction._
## [1.5.1] - 2018-06-04
Bug Fixes:
- Fix for infinite yieldItemValue recursion. #503
- Fix recursive addition of `badgerMove` prefix.
https://github.com/dgraph-io/badger/commit/2e3a32f0ccac3066fb4206b28deb39c210c5266f
- Use file size based window size for sampling, instead of fixing it to 10MB. #501
Cleanup:
- Clarify comments and documentation.
- Move badger tool one directory level up.
## [1.5.0] - 2018-05-08
- Introduce `NumVersionsToKeep` option. This option is used to discard many versions of the same
key, which saves space.
- Add a new `SetWithDiscard` method, which would indicate that all the older versions of the key are
now invalid. Those versions would be discarded during compactions.
- Value log GC moves are now bound to another keyspace to ensure latest versions of data are always
at the top in LSM tree.
- Introduce `ValueLogMaxEntries` to restrict the number of key-value pairs per value log file. This
helps bound the time it takes to garbage collect one file.
## [1.4.0] - 2018-05-04
- Make mmap-ing of value log optional.
- Run GC multiple times, based on recorded discard statistics.
- Add MergeOperator.
- Force compact L0 on clsoe (#439).
- Add truncate option to warn about data loss (#452).
- Discard key versions during compaction (#464).
- Introduce new `LSMOnlyOptions`, to make Badger act like a typical LSM based DB.
Bug fix:
- (Temporary) Check max version across all tables in Get (removed in next release).
- Update commit and read ts while loading from backup.
- Ensure all transaction entries are part of the same value log file.
- On commit, run unlock callbacks before doing writes (#413).
- Wait for goroutines to finish before closing iterators (#421).
## [1.3.0] - 2017-12-12
- Add `DB.NextSequence()` method to generate monotonically increasing integer sequences.
- Add `DB.Size()` method to return the size of LSM and value log files.
- Tweaked mmap code to make Windows 32-bit builds work.
- Tweaked build tags on some files to make iOS builds work.
- Fix `DB.PurgeOlderVersions()` to not violate some constraints.
## [1.2.0] - 2017-11-30
- Expose a `Txn.SetEntry()` method to allow setting the key-value pair and all the metadata at the
same time.
## [1.1.1] - 2017-11-28
- Fix bug where txn.Get was returing key deleted in same transaction.
- Fix race condition while decrementing reference in oracle.
- Update doneCommit in the callback for CommitAsync.
- Iterator see writes of current txn.
## [1.1.0] - 2017-11-13
- Create Badger directory if it does not exist when `badger.Open` is called.
- Added `Item.ValueCopy()` to avoid deadlocks in long-running iterations
- Fixed 64-bit alignment issues to make Badger run on Arm v7
## [1.0.1] - 2017-11-06
- Fix an uint16 overflow when resizing key slice
[4.9.0]: https://github.com/dgraph-io/badger/compare/v4.8.0...v4.9.0
[4.8.0]: https://github.com/dgraph-io/badger/compare/v4.7.0...v4.8.0
[4.7.0]: https://github.com/dgraph-io/badger/compare/v4.6.0...v4.7.0
[4.6.0]: https://github.com/dgraph-io/badger/compare/v4.5.2...v4.6.0
[4.5.2]: https://github.com/dgraph-io/badger/compare/v4.5.1...v4.5.2
[4.5.1]: https://github.com/dgraph-io/badger/compare/v4.5.0...v4.5.1
[4.5.0]: https://github.com/dgraph-io/badger/compare/v4.4.0...v4.5.0
[4.4.0]: https://github.com/dgraph-io/badger/compare/v4.3.1...v4.4.0
[4.3.1]: https://github.com/dgraph-io/badger/compare/v4.3.0...v4.3.1
[4.3.0]: https://github.com/dgraph-io/badger/compare/v4.2.0...v4.3.0
[4.2.0]: https://github.com/dgraph-io/badger/compare/v4.1.0...v4.2.0
[4.1.0]: https://github.com/dgraph-io/badger/compare/v4.0.1...v4.1.0
[4.0.1]: https://github.com/dgraph-io/badger/compare/v4.0.0...v4.0.1
[4.0.0]: https://github.com/dgraph-io/badger/compare/v3.2103.5...v4.0.0
[3.2103.5]: https://github.com/dgraph-io/badger/compare/v3.2103.4...v3.2103.5
[3.2103.4]: https://github.com/dgraph-io/badger/compare/v3.2103.3...v3.2103.4
[3.2103.3]: https://github.com/dgraph-io/badger/compare/v3.2103.2...v3.2103.3
[3.2103.2]: https://github.com/dgraph-io/badger/compare/v3.2103.1...v3.2103.2
[3.2103.1]: https://github.com/dgraph-io/badger/compare/v3.2103.0...v3.2103.1
[3.2103.0]: https://github.com/dgraph-io/badger/compare/v3.2011.1...v3.2103.0
[3.2011.1]: https://github.com/dgraph-io/badger/compare/v3.2011.0...v3.2011.1
[3.2011.0]: https://github.com/dgraph-io/badger/compare/v2.2007.4...v3.2011.0
[2.2007.4]: https://github.com/dgraph-io/badger/compare/v2.2007.3...v2.2007.4
[2.2007.3]: https://github.com/dgraph-io/badger/compare/v2.2007.2...v2.2007.3
[2.2007.2]: https://github.com/dgraph-io/badger/compare/v2.2007.1...v2.2007.2
[2.2007.1]: https://github.com/dgraph-io/badger/compare/v2.2007.0...v2.2007.1
[2.2007.0]: https://github.com/dgraph-io/badger/compare/v2.0.3...v2.2007.0
[2.0.3]: https://github.com/dgraph-io/badger/compare/v2.0.2...v2.0.3
[2.0.2]: https://github.com/dgraph-io/badger/compare/v2.0.1...v2.0.2
[2.0.1]: https://github.com/dgraph-io/badger/compare/v2.0.0...v2.0.1
[2.0.0]: https://github.com/dgraph-io/badger/compare/v1.6.0...v2.0.0
[1.6.0]: https://github.com/dgraph-io/badger/compare/v1.5.5...v1.6.0
[1.5.5]: https://github.com/dgraph-io/badger/compare/v1.5.3...v1.5.5
[1.5.3]: https://github.com/dgraph-io/badger/compare/v1.5.2...v1.5.3
[1.5.2]: https://github.com/dgraph-io/badger/compare/v1.5.1...v1.5.2
[1.5.1]: https://github.com/dgraph-io/badger/compare/v1.5.0...v1.5.1
[1.5.0]: https://github.com/dgraph-io/badger/compare/v1.4.0...v1.5.0
[1.4.0]: https://github.com/dgraph-io/badger/compare/v1.3.0...v1.4.0
[1.3.0]: https://github.com/dgraph-io/badger/compare/v1.2.0...v1.3.0
[1.2.0]: https://github.com/dgraph-io/badger/compare/v1.1.1...v1.2.0
[1.1.1]: https://github.com/dgraph-io/badger/compare/v1.1.0...v1.1.1
[1.1.0]: https://github.com/dgraph-io/badger/compare/v1.0.1...v1.1.0
[1.0.1]: https://github.com/dgraph-io/badger/compare/v1.0.0...v1.0.1
================================================
FILE: CODE_OF_CONDUCT.md
================================================
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our community a
harassment-free experience for everyone, regardless of age, body size, visible or invisible
disability, ethnicity, sex characteristics, gender identity and expression, level of experience,
education, socio-economic status, nationality, personal appearance, race, religion, or sexual
identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and
healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our community include:
- Demonstrating empathy and kindness toward other people
- Being respectful of differing opinions, viewpoints, and experiences
- Giving and gracefully accepting constructive feedback
- Accepting responsibility and apologizing to those affected by our mistakes, and learning from the
experience
- Focusing on what is best not just for us as individuals, but for the overall community
Examples of unacceptable behavior include:
- The use of sexualized language or imagery, and sexual attention or advances of any kind
- Trolling, insulting or derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others' private information, such as a physical or email address, without their
explicit permission
- Other conduct which could reasonably be considered inappropriate in a professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior
and will take appropriate and fair corrective action in response to any behavior that they deem
inappropriate, threatening, offensive, or harmful.
Community leaders have the right and responsibility to remove, edit, or reject comments, commits,
code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and
will communicate reasons for moderation decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when an individual is
officially representing the community in public spaces. Examples of representing our community
include using an official e-mail address, posting via an official social media account, or acting as
an appointed representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community
leaders responsible for enforcement at dgraph-admin@istaridigital.com. All complaints will be
reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of any
incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining the consequences for
any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or
unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing clarity around the
nature of the violation and an explanation of why the behavior was inappropriate. A public apology
may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series of actions.
**Consequence**: A warning with consequences for continued behavior. No interaction with the people
involved, including unsolicited interaction with those enforcing the Code of Conduct, for a
specified period of time. This includes avoiding interactions in community spaces as well as
external channels like social media. Violating these terms may lead to a temporary or permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including sustained inappropriate
behavior.
**Consequence**: A temporary ban from any sort of interaction or public communication with the
community for a specified period of time. No public or private interaction with the people involved,
including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this
period. Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community standards, including
sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement
of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.0, available at
https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity).
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at
https://www.contributor-covenant.org/translations.
================================================
FILE: CONTRIBUTING.md
================================================
# Contribution Guide
- [Before you get started](#before-you-get-started)
- [Code of Conduct](#code-of-conduct)
- [Your First Contribution](#your-first-contribution)
- [Find a good first topic](#find-a-good-first-topic)
- [Setting up your development environment](#setting-up-your-development-environment)
- [Fork the project](#fork-the-project)
- [Clone the project](#clone-the-project)
- [New branch for a new code](#new-branch-for-a-new-code)
- [Test](#test)
- [Commit and push](#commit-and-push)
- [Create a Pull Request](#create-a-pull-request)
- [Sign the CLA](#sign-the-cla)
- [Get a code review](#get-a-code-review)
## Before you get started
### Code of Conduct
Please make sure to read and observe our [Code of Conduct](./CODE_OF_CONDUCT.md).
## Your First Contribution
### Find a good first topic
You can start by finding an existing issue with the
[good first issue](https://github.com/dgraph-io/badger/labels/good%20first%20issue) or
[help wanted](https://github.com/dgraph-io/badger/labels/help%20wanted) labels. These issues are
well suited for new contributors.
## Setting up your development environment
- [Install Go 1.25.0 or above](https://golang.org/doc/install).
- Install
[trunk](https://docs.trunk.io/code-quality/overview/getting-started/install#install-the-launcher).
Our CI uses trunk to lint and check code, having it installed locally will save you time.
### Fork the project
- Visit https://github.com/dgraph-io/badger
- Click the `Fork` button (top right) to create a fork of the repository
### Clone the project
```sh
git clone https://github.com/$GITHUB_USER/badger
cd badger
git remote add upstream git@github.com:dgraph-io/badger.git
# Never push to the upstream master
git remote set-url --push upstream no_push
```
### New branch for a new code
Get your local master up to date:
```sh
git fetch upstream
git checkout master
git rebase upstream/master
```
Create a new branch from the master:
```sh
git checkout -b my_new_feature
```
And now you can finally add your changes to project.
### Test
Build and run all tests:
```sh
./test.sh
```
### Commit and push
Commit your changes:
```sh
git commit
```
When the changes are ready to review:
```sh
git push origin my_new_feature
```
### Create a Pull Request
Just open `https://github.com/$GITHUB_USER/badger/pull/new/my_new_feature` and fill the PR
description.
### Sign the CLA
Click the **Sign in with Github to agree** button to sign the CLA.
[An example](https://cla-assistant.io/dgraph-io/badger?pullRequest=1377).
### Get a code review
If your pull request (PR) is opened, it will be assigned to one or more reviewers. Those reviewers
will do a code review.
To address review comments, you should commit the changes to the same branch of the PR on your fork.
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
================================================
FILE: Makefile
================================================
#
# SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.
# SPDX-License-Identifier: Apache-2.0
#
USER_ID = $(shell id -u)
HAS_JEMALLOC = $(shell test -f /usr/local/lib/libjemalloc.a && echo "jemalloc")
JEMALLOC_URL = "https://github.com/jemalloc/jemalloc/releases/download/5.3.0/jemalloc-5.3.0.tar.bz2"
.PHONY: all badger test jemalloc dependency
badger: jemalloc
@echo "Compiling Badger binary..."
@$(MAKE) -C badger badger
@echo "Badger binary located in badger directory."
test: jemalloc
@echo "Running Badger tests..."
@./test.sh
jemalloc:
@if [ -z "$(HAS_JEMALLOC)" ] ; then \
mkdir -p /tmp/jemalloc-temp && cd /tmp/jemalloc-temp ; \
echo "Downloading jemalloc..." ; \
curl -s -L ${JEMALLOC_URL} -o jemalloc.tar.bz2 ; \
tar xjf ./jemalloc.tar.bz2 ; \
cd jemalloc-5.3.0 ; \
./configure --with-jemalloc-prefix='je_' --with-malloc-conf='background_thread:true,metadata_thp:auto'; \
make ; \
if [ "$(USER_ID)" -eq "0" ]; then \
make install ; \
else \
echo "==== Need sudo access to install jemalloc" ; \
sudo make install ; \
fi \
fi
dependency:
@echo "Installing dependencies..."
@sudo apt-get update
@sudo apt-get -y install \
ca-certificates \
curl \
gnupg \
lsb-release \
build-essential \
protobuf-compiler \
================================================
FILE: README.md
================================================
# BadgerDB
[](https://pkg.go.dev/github.com/dgraph-io/badger/v4)
[](https://goreportcard.com/report/github.com/dgraph-io/badger/v4)
[](https://sourcegraph.com/github.com/dgraph-io/badger?badge)
[](https://github.com/dgraph-io/badger/actions/workflows/ci-badger-tests.yml)
[](https://github.com/dgraph-io/badger/actions/workflows/ci-badger-bank-tests.yml)
[](https://github.com/dgraph-io/badger/actions/workflows/ci-badger-bank-tests-nightly.yml)
BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the
underlying database for [Dgraph](https://github.com/dgraph-io/dgraph), a fast, distributed graph
database. It's meant to be a performant alternative to non-Go-based key-value stores like RocksDB.
## Project Status
Badger is stable and is being used to serve data sets worth hundreds of terabytes. Badger supports
concurrent ACID transactions with serializable snapshot isolation (SSI) guarantees. A Jepsen-style
bank test runs nightly for 8h, with `--race` flag and ensures the maintenance of transactional
guarantees. Badger has also been tested to work with filesystem level anomalies, to ensure
persistence and consistency. Badger is being used by a number of projects which includes Dgraph,
Jaeger Tracing, UsenetExpress, and many more.
The list of projects using Badger can be found [here](#projects-using-badger).
Please consult the [Changelog] for more detailed information on releases.
Note: Badger is built with go 1.23 and we refrain from bumping this version to minimize downstream
effects of those using Badger in applications built with older versions of Go.
[Changelog]: https://github.com/dgraph-io/badger/blob/main/CHANGELOG.md
## Table of Contents
- [BadgerDB](#badgerdb)
- [Project Status](#project-status)
- [Table of Contents](#table-of-contents)
- [Getting Started](#getting-started)
- [Installing](#installing)
- [Installing Badger Command Line Tool](#installing-badger-command-line-tool)
- [Choosing a version](#choosing-a-version)
- [Badger Documentation](#badger-documentation)
- [Resources](#resources)
- [Blog Posts](#blog-posts)
- [Design](#design)
- [Comparisons](#comparisons)
- [Benchmarks](#benchmarks)
- [Projects Using Badger](#projects-using-badger)
- [Contributing](#contributing)
- [Contact](#contact)
## Getting Started
### Installing
To start using Badger, install Go 1.23 or above. Badger v3 and above needs go modules. From your
project, run the following command
```sh
go get github.com/dgraph-io/badger/v4
```
This will retrieve the library.
#### Installing Badger Command Line Tool
Badger provides a CLI tool which can perform certain operations like offline backup/restore. To
install the Badger CLI, retrieve the repository and checkout the desired version. Then run
```sh
cd badger
go install .
```
This will install the badger command line utility into your $GOBIN path.
## Badger Documentation
Badger Documentation is available at [https://badger.dgraph.io](https://badger.dgraph.io)
## Resources
### Blog Posts
1. [Introducing Badger: A fast key-value store written natively in Go](https://hypermode.com/blog/badger/)
2. [Make Badger crash resilient with ALICE](https://hypermode.com/blog/alice/)
3. [Badger vs LMDB vs BoltDB: Benchmarking key-value databases in Go](https://hypermode.com/blog/badger-lmdb-boltdb/)
4. [Concurrent ACID Transactions in Badger](https://hypermode.com/blog/badger-txn/)
## Design
Badger was written with these design goals in mind:
- Write a key-value database in pure Go.
- Use latest research to build the fastest KV database for data sets spanning terabytes.
- Optimize for SSDs.
Badger’s design is based on a paper titled _[WiscKey: Separating Keys from Values in SSD-conscious
Storage][wisckey]_.
[wisckey]: https://www.usenix.org/system/files/conference/fast16/fast16-papers-lu.pdf
### Comparisons
| Feature | Badger | RocksDB | BoltDB |
| ----------------------------- | ------------------------------------------ | ----------------------------- | --------- |
| Design | LSM tree with value log | LSM tree only | B+ tree |
| High Read throughput | Yes | No | Yes |
| High Write throughput | Yes | Yes | No |
| Designed for SSDs | Yes (with latest research <sup>1</sup>) | Not specifically <sup>2</sup> | No |
| Embeddable | Yes | Yes | Yes |
| Sorted KV access | Yes | Yes | Yes |
| Pure Go (no Cgo) | Yes | No | Yes |
| Transactions | Yes, ACID, concurrent with SSI<sup>3</sup> | Yes (but non-ACID) | Yes, ACID |
| Snapshots | Yes | Yes | Yes |
| TTL support | Yes | Yes | No |
| 3D access (key-value-version) | Yes<sup>4</sup> | No | No |
<sup>1</sup> The [WISCKEY paper][wisckey] (on which Badger is based) saw big wins with separating
values from keys, significantly reducing the write amplification compared to a typical LSM tree.
<sup>2</sup> RocksDB is an SSD optimized version of LevelDB, which was designed specifically for
rotating disks. As such RocksDB's design isn't aimed at SSDs.
<sup>3</sup> SSI: Serializable Snapshot Isolation. For more details, see the blog post
[Concurrent ACID Transactions in Badger](https://hypermode.com/blog/badger-txn/)
<sup>4</sup> Badger provides direct access to value versions via its Iterator API. Users can also
specify how many versions to keep per key via Options.
### Benchmarks
We have run comprehensive benchmarks against RocksDB, Bolt and LMDB. The benchmarking code, and the
detailed logs for the benchmarks can be found in the [badger-bench] repo. More explanation,
including graphs can be found the blog posts (linked above).
[badger-bench]: https://github.com/dgraph-io/badger-bench
## Projects Using Badger
Below is a list of known projects that use Badger:
- [Dgraph](https://github.com/dgraph-io/dgraph) - Distributed graph database.
- [Jaeger](https://github.com/jaegertracing/jaeger) - Distributed tracing platform.
- [go-ipfs](https://github.com/ipfs/go-ipfs) - Go client for the InterPlanetary File System (IPFS),
a new hypermedia distribution protocol.
- [Riot](https://github.com/go-ego/riot) - An open-source, distributed search engine.
- [emitter](https://github.com/emitter-io/emitter) - Scalable, low latency, distributed pub/sub
broker with message storage, uses MQTT, gossip and badger.
- [OctoSQL](https://github.com/cube2222/octosql) - Query tool that allows you to join, analyse and
transform data from multiple databases using SQL.
- [Dkron](https://dkron.io/) - Distributed, fault tolerant job scheduling system.
- [smallstep/certificates](https://github.com/smallstep/certificates) - Step-ca is an online
certificate authority for secure, automated certificate management.
- [Sandglass](https://github.com/celrenheit/sandglass) - distributed, horizontally scalable,
persistent, time sorted message queue.
- [TalariaDB](https://github.com/grab/talaria) - Grab's Distributed, low latency time-series
database.
- [Sloop](https://github.com/salesforce/sloop) - Salesforce's Kubernetes History Visualization
Project.
- [Usenet Express](https://usenetexpress.com/) - Serving over 300TB of data with Badger.
- [gorush](https://github.com/appleboy/gorush) - A push notification server written in Go.
- [0-stor](https://github.com/zero-os/0-stor) - Single device object store.
- [Dispatch Protocol](https://github.com/dispatchlabs/disgo) - Blockchain protocol for distributed
application data analytics.
- [GarageMQ](https://github.com/valinurovam/garagemq) - AMQP server written in Go.
- [RedixDB](https://alash3al.github.io/redix/) - A real-time persistent key-value store with the
same redis protocol.
- [BBVA](https://github.com/BBVA/raft-badger) - Raft backend implementation using BadgerDB for
Hashicorp raft.
- [Fantom](https://github.com/Fantom-foundation/go-lachesis) - aBFT Consensus platform for
distributed applications.
- [decred](https://github.com/decred/dcrdata) - An open, progressive, and self-funding
cryptocurrency with a system of community-based governance integrated into its blockchain.
- [OpenNetSys](https://github.com/opennetsys/c3-go) - Create useful dApps in any software language.
- [HoneyTrap](https://github.com/honeytrap/honeytrap) - An extensible and opensource system for
running, monitoring and managing honeypots.
- [Insolar](https://github.com/insolar/insolar) - Enterprise-ready blockchain platform.
- [IoTeX](https://github.com/iotexproject/iotex-core) - The next generation of the decentralized
network for IoT powered by scalability- and privacy-centric blockchains.
- [go-sessions](https://github.com/kataras/go-sessions) - The sessions manager for Go net/http and
fasthttp.
- [Babble](https://github.com/mosaicnetworks/babble) - BFT Consensus platform for distributed
applications.
- [Tormenta](https://github.com/jpincas/tormenta) - Embedded object-persistence layer / simple JSON
database for Go projects.
- [BadgerHold](https://github.com/timshannon/badgerhold) - An embeddable NoSQL store for querying Go
types built on Badger
- [Goblero](https://github.com/didil/goblero) - Pure Go embedded persistent job queue backed by
BadgerDB
- [Surfline](https://www.surfline.com) - Serving global wave and weather forecast data with Badger.
- [Cete](https://github.com/mosuka/cete) - Simple and highly available distributed key-value store
built on Badger. Makes it easy bringing up a cluster of Badger with Raft consensus algorithm by
hashicorp/raft.
- [Volument](https://volument.com/) - A new take on website analytics backed by Badger.
- [KVdb](https://kvdb.io/) - Hosted key-value store and serverless platform built on top of Badger.
- [Terminotes](https://gitlab.com/asad-awadia/terminotes) - Self hosted notes storage and search
server - storage powered by BadgerDB
- [Pyroscope](https://github.com/pyroscope-io/pyroscope) - Open source continuous profiling platform
built with BadgerDB
- [Veri](https://github.com/bgokden/veri) - A distributed feature store optimized for Search and
Recommendation tasks.
- [bIter](https://github.com/MikkelHJuul/bIter) - A library and Iterator interface for working with
the `badger.Iterator`, simplifying from-to, and prefix mechanics.
- [ld](https://github.com/MikkelHJuul/ld) - (Lean Database) A very simple gRPC-only key-value
database, exposing BadgerDB with key-range scanning semantics.
- [Souin](https://github.com/darkweak/Souin) - A RFC compliant HTTP cache with lot of other features
based on Badger for the storage. Compatible with all existing reverse-proxies.
- [Xuperchain](https://github.com/xuperchain/xupercore) - A highly flexible blockchain architecture
with great transaction performance.
- [m2](https://github.com/qichengzx/m2) - A simple http key/value store based on the raft protocol.
- [chaindb](https://github.com/ChainSafe/chaindb) - A blockchain storage layer used by
[Gossamer](https://chainsafe.github.io/gossamer/), a Go client for the
[Polkadot Network](https://polkadot.network/).
- [vxdb](https://github.com/vitalvas/vxdb) - Simple schema-less Key-Value NoSQL database with
simplest API interface.
- [Opacity](https://github.com/opacity/storage-node) - Backend implementation for the Opacity
storage project
- [Vephar](https://github.com/vaccovecrana/vephar) - A minimal key/value store using hashicorp-raft
for cluster coordination and Badger for data storage.
- [gowarcserver](https://github.com/nlnwa/gowarcserver) - Open-source server for warc files. Can be
used in conjunction with pywb
- [flow-go](https://github.com/onflow/flow-go) - A fast, secure, and developer-friendly blockchain
built to support the next generation of games, apps and the digital assets that power them.
- [Wrgl](https://www.wrgl.co) - A data version control system that works like Git but specialized to
store and diff CSV.
- [Loggie](https://github.com/loggie-io/loggie) - A lightweight, cloud-native data transfer agent
and aggregator.
- [raft-badger](https://github.com/rfyiamcool/raft-badger) - raft-badger implements LogStore and
StableStore Interface of hashcorp/raft. it is used to store raft log and metadata of
hashcorp/raft.
- [DVID](https://github.com/janelia-flyem/dvid) - A dataservice for branched versioning of a variety
of data types. Originally created for large-scale brain reconstructions in Connectomics.
- [KVS](https://github.com/tauraamui/kvs) - A library for making it easy to persist, load and query
full structs into BadgerDB, using an ownership hierarchy model.
- [LLS](https://github.com/Boc-chi-no/LLS) - LLS is an efficient URL Shortener that can be used to
shorten links and track link usage. Support for BadgerDB and MongoDB. Improved performance by more
than 30% when using BadgerDB
- [lakeFS](https://github.com/treeverse/lakeFS) - lakeFS is an open-source data version control that
transforms your object storage to Git-like repositories. lakeFS uses BadgerDB for its underlying
local metadata KV store implementation
- [Goptivum](https://github.com/smegg99/Goptivum) - Goptivum is a better frontend and API for the
Vulcan Optivum schedule program
- [ActionManager](https://mftlabs.io/actionmanager) - A dynamic entity manager based on rjsf schema
and badger db
- [MightyMap](https://github.com/thisisdevelopment/mightymap) - Mightymap: Conveys both robustness
and high capability, fitting for a powerful concurrent map.
- [FlowG](https://github.com/link-society/flowg) - A low-code log processing facility
- [Bluefin](https://github.com/blinklabs-io/bluefin) - Bluefin is a TUNA Proof of Work miner for the
Fortuna smart contract on the Cardano blockchain
- [cDNSd](https://github.com/blinklabs-io/cdnsd) - A Cardano blockchain backed DNS server daemon
- [Dingo](https://github.com/blinklabs-io/dingo) - A Cardano blockchain data node
If you are using Badger in a project please send a pull request to add it to the list.
### Platform Compatibility
Badger uses OS-specific implementations for directory locking and `fsync` operations. On
**POSIX-compliant systems** (Linux, macOS, BSD), these work as expected.
For **non-POSIX platforms**, be aware of potential limitations:
| Platform | File | Notes |
| -------- | ---------------- | -------------------------------------------------------------------- |
| AIX | `dir_aix.go` | Directory `fsync` not supported; durability on crash may be affected |
| Windows | `dir_windows.go` | Uses different locking mechanism |
| Plan9 | `dir_plan9.go` | No file locking support |
| WASM/JS | `dir_other.go` | No file locking support |
If you encounter issues on these platforms, review the corresponding `dir_*.go` source file for
implementation details.
## Contributing
If you're interested in contributing to Badger see [CONTRIBUTING](./CONTRIBUTING.md).
## Contact
- Please use [Github issues](https://github.com/dgraph-io/badger/issues) for filing bugs.
- Please use [Discussions](https://github.com/orgs/dgraph-io/discussions) for questions,
discussions, and feature requests.
================================================
FILE: SECURITY.md
================================================
# Reporting Security Concerns
We take the security of Badger very seriously. If you believe you have found a security
vulnerability in Badger, we encourage you to let us know right away.
We will investigate all legitimate reports and do our best to quickly fix the problem. Please report
any issues or vulnerabilities via GitHub Security Advisories instead of posting a public issue in
GitHub. You can also send security communications to dgraph-admin@istaridigital.com.
Please include the version identifier and details on how the vulnerability can be exploited.
================================================
FILE: VERSIONING.md
================================================
# Serialization Versioning: Semantic Versioning for databases
Semantic Versioning, commonly known as SemVer, is a great idea that has been very widely adopted as
a way to decide how to name software versions. The whole concept is very well summarized on
semver.org with the following lines:
> Given a version number MAJOR.MINOR.PATCH, increment the:
>
> 1. MAJOR version when you make incompatible API changes,
> 2. MINOR version when you add functionality in a backwards-compatible manner, and
> 3. PATCH version when you make backwards-compatible bug fixes.
>
> Additional labels for pre-release and build metadata are available as extensions to the
> MAJOR.MINOR.PATCH format.
Unfortunately, API changes are not the most important changes for libraries that serialize data for
later consumption. For these libraries, such as BadgerDB, changes to the API are much easier to
handle than change to the data format used to store data on disk.
## Serialization Version specification
Serialization Versioning, like Semantic Versioning, uses 3 numbers and also calls them
MAJOR.MINOR.PATCH, but the semantics of the numbers are slightly modified:
Given a version number MAJOR.MINOR.PATCH, increment the:
- MAJOR version when you make changes that require a transformation of the dataset before it can be
used again.
- MINOR version when old datasets are still readable but the API might have changed in
backwards-compatible or incompatible ways.
- PATCH version when you make backwards-compatible bug fixes.
Additional labels for pre-release and build metadata are available as extensions to the
MAJOR.MINOR.PATCH format.
Following this naming strategy, migration from v1.x to v2.x requires a migration strategy for your
existing dataset, and as such has to be carefully planned. Migrations in between different minor
versions (e.g. v1.5.x and v1.6.x) might break your build, as the API _might_ have changed, but once
your code compiles there's no need for any data migration. Lastly, changes in between two different
patch versions should never break your build or dataset.
For more background on our decision to adopt Serialization Versioning, read the blog post [Semantic
Versioning, Go Modules, and Databases][blog] and the original proposal on [this comment on Dgraph's
Discuss forum][discuss].
[blog]: https://open.dgraph.io/post/serialization-versioning/
[discuss]: https://discuss.dgraph.io/t/go-modules-on-badger-and-dgraph/4662/7
================================================
FILE: backup.go
================================================
/*
* SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.
* SPDX-License-Identifier: Apache-2.0
*/
package badger
import (
"bufio"
"bytes"
"context"
"encoding/binary"
"fmt"
"io"
"google.golang.org/protobuf/proto"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// flushThreshold determines when a buffer will be flushed. When performing a
// backup/restore, the entries will be batched up until the total size of batch
// is more than flushThreshold or entry size (without the value size) is more
// than the maxBatchSize.
const flushThreshold = 100 << 20
// Backup dumps a protobuf-encoded list of all entries in the database into the
// given writer, that are newer than or equal to the specified version. It
// returns a timestamp (version) indicating the version of last entry that is
// dumped, which after incrementing by 1 can be passed into later invocation to
// generate incremental backup of entries that have been added/modified since
// the last invocation of DB.Backup().
// DB.Backup is a wrapper function over Stream.Backup to generate full and
// incremental backups of the DB. For more control over how many goroutines are
// used to generate the backup, or if you wish to backup only a certain range
// of keys, use Stream.Backup directly.
func (db *DB) Backup(w io.Writer, since uint64) (uint64, error) {
stream := db.NewStream()
stream.LogPrefix = "DB.Backup"
stream.SinceTs = since
return stream.Backup(w, since)
}
// Backup dumps a protobuf-encoded list of all entries in the database into the
// given writer, that are newer than or equal to the specified version. It returns a
// timestamp(version) indicating the version of last entry that was dumped, which
// after incrementing by 1 can be passed into a later invocation to generate an
// incremental dump of entries that have been added/modified since the last
// invocation of Stream.Backup().
//
// This can be used to backup the data in a database at a given point in time.
func (stream *Stream) Backup(w io.Writer, since uint64) (uint64, error) {
stream.KeyToList = func(key []byte, itr *Iterator) (*pb.KVList, error) {
list := &pb.KVList{}
a := itr.Alloc
for ; itr.Valid(); itr.Next() {
item := itr.Item()
if !bytes.Equal(item.Key(), key) {
return list, nil
}
if item.Version() < since {
return nil, fmt.Errorf("Backup: Item Version: %d less than sinceTs: %d",
item.Version(), since)
}
var valCopy []byte
if !item.IsDeletedOrExpired() {
// No need to copy value, if item is deleted or expired.
err := item.Value(func(val []byte) error {
valCopy = a.Copy(val)
return nil
})
if err != nil {
stream.db.opt.Errorf("Key [%x, %d]. Error while fetching value [%v]\n",
item.Key(), item.Version(), err)
return nil, err
}
}
// clear txn bits
meta := item.meta &^ (bitTxn | bitFinTxn)
kv := y.NewKV(a)
*kv = pb.KV{
Key: a.Copy(item.Key()),
Value: valCopy,
UserMeta: a.Copy([]byte{item.UserMeta()}),
Version: item.Version(),
ExpiresAt: item.ExpiresAt(),
Meta: a.Copy([]byte{meta}),
}
list.Kv = append(list.Kv, kv)
switch {
case item.DiscardEarlierVersions():
// If we need to discard earlier versions of this item, add a delete
// marker just below the current version.
list.Kv = append(list.Kv, &pb.KV{
Key: item.KeyCopy(nil),
Version: item.Version() - 1,
Meta: []byte{bitDelete},
})
return list, nil
case item.IsDeletedOrExpired():
return list, nil
}
}
return list, nil
}
var maxVersion uint64
stream.Send = func(buf *z.Buffer) error {
list, err := BufferToKVList(buf)
if err != nil {
return err
}
out := list.Kv[:0]
for _, kv := range list.Kv {
if maxVersion < kv.Version {
maxVersion = kv.Version
}
if !kv.StreamDone {
// Don't pick stream done changes.
out = append(out, kv)
}
}
list.Kv = out
return writeTo(list, w)
}
if err := stream.Orchestrate(context.Background()); err != nil {
return 0, err
}
return maxVersion, nil
}
func writeTo(list *pb.KVList, w io.Writer) error {
if err := binary.Write(w, binary.LittleEndian, uint64(proto.Size(list))); err != nil {
return err
}
buf, err := proto.Marshal(list)
if err != nil {
return err
}
_, err = w.Write(buf)
return err
}
// KVLoader is used to write KVList objects in to badger. It can be used to restore a backup.
type KVLoader struct {
db *DB
throttle *y.Throttle
entries []*Entry
entriesSize int64
totalSize int64
}
// NewKVLoader returns a new instance of KVLoader.
func (db *DB) NewKVLoader(maxPendingWrites int) *KVLoader {
return &KVLoader{
db: db,
throttle: y.NewThrottle(maxPendingWrites),
entries: make([]*Entry, 0, db.opt.maxBatchCount),
}
}
// Set writes the key-value pair to the database.
func (l *KVLoader) Set(kv *pb.KV) error {
var userMeta, meta byte
if len(kv.UserMeta) > 0 {
userMeta = kv.UserMeta[0]
}
if len(kv.Meta) > 0 {
meta = kv.Meta[0]
}
e := &Entry{
Key: y.KeyWithTs(kv.Key, kv.Version),
Value: kv.Value,
UserMeta: userMeta,
ExpiresAt: kv.ExpiresAt,
meta: meta,
}
estimatedSize := e.estimateSizeAndSetThreshold(l.db.valueThreshold())
// Flush entries if inserting the next entry would overflow the transactional limits.
if int64(len(l.entries))+1 >= l.db.opt.maxBatchCount ||
l.entriesSize+estimatedSize >= l.db.opt.maxBatchSize ||
l.totalSize >= flushThreshold {
if err := l.send(); err != nil {
return err
}
}
l.entries = append(l.entries, e)
l.entriesSize += estimatedSize
l.totalSize += estimatedSize + int64(len(e.Value))
return nil
}
func (l *KVLoader) send() error {
if err := l.throttle.Do(); err != nil {
return err
}
if err := l.db.batchSetAsync(l.entries, func(err error) {
l.throttle.Done(err)
}); err != nil {
return err
}
l.entries = make([]*Entry, 0, l.db.opt.maxBatchCount)
l.entriesSize = 0
l.totalSize = 0
return nil
}
// Finish is meant to be called after all the key-value pairs have been loaded.
func (l *KVLoader) Finish() error {
if len(l.entries) > 0 {
if err := l.send(); err != nil {
return err
}
}
return l.throttle.Finish()
}
// Load reads a protobuf-encoded list of all entries from a reader and writes
// them to the database. This can be used to restore the database from a backup
// made by calling DB.Backup(). If more complex logic is needed to restore a badger
// backup, the KVLoader interface should be used instead.
//
// DB.Load() should be called on a database that is not running any other
// concurrent transactions while it is running.
func (db *DB) Load(r io.Reader, maxPendingWrites int) error {
br := bufio.NewReaderSize(r, 16<<10)
unmarshalBuf := make([]byte, 1<<10)
ldr := db.NewKVLoader(maxPendingWrites)
for {
var sz uint64
err := binary.Read(br, binary.LittleEndian, &sz)
if err == io.EOF {
break
} else if err != nil {
return err
}
if cap(unmarshalBuf) < int(sz) {
unmarshalBuf = make([]byte, sz)
}
if _, err = io.ReadFull(br, unmarshalBuf[:sz]); err != nil {
return err
}
list := &pb.KVList{}
if err := proto.Unmarshal(unmarshalBuf[:sz], list); err != nil {
return err
}
for _, kv := range list.Kv {
if err := ldr.Set(kv); err != nil {
return err
}
// Update nextTxnTs, memtable stores this
// timestamp in badger head when flushed.
if kv.Version >= db.orc.nextTxnTs {
db.orc.nextTxnTs = kv.Version + 1
}
}
}
if err := ldr.Finish(); err != nil {
return err
}
db.orc.txnMark.Done(db.orc.nextTxnTs - 1)
return nil
}
================================================
FILE: backup_test.go
================================================
/*
* SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.
* SPDX-License-Identifier: Apache-2.0
*/
package badger
import (
"bytes"
"fmt"
"math/rand"
"os"
"path/filepath"
"reflect"
"strconv"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/dgraph-io/badger/v4/pb"
)
func TestBackupRestore1(t *testing.T) {
dir, err := os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(dir)
db, err := Open(getTestOptions(dir))
require.NoError(t, err)
// Write some stuff
entries := []struct {
key []byte
val []byte
userMeta byte
version uint64
}{
{key: []byte("answer1"), val: []byte("42"), version: 1},
{key: []byte("answer2"), val: []byte("43"), userMeta: 1, version: 2},
}
err = db.Update(func(txn *Txn) error {
e := entries[0]
err := txn.SetEntry(NewEntry(e.key, e.val).WithMeta(e.userMeta))
if err != nil {
return err
}
return nil
})
require.NoError(t, err)
err = db.Update(func(txn *Txn) error {
e := entries[1]
err := txn.SetEntry(NewEntry(e.key, e.val).WithMeta(e.userMeta))
if err != nil {
return err
}
return nil
})
require.NoError(t, err)
// Use different directory.
dir, err = os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(dir)
bak, err := os.CreateTemp(dir, "badgerbak")
require.NoError(t, err)
_, err = db.Backup(bak, 0)
require.NoError(t, err)
require.NoError(t, bak.Close())
require.NoError(t, db.Close())
db, err = Open(getTestOptions(dir))
require.NoError(t, err)
defer db.Close()
bak, err = os.Open(bak.Name())
require.NoError(t, err)
defer bak.Close()
require.NoError(t, db.Load(bak, 16))
err = db.View(func(txn *Txn) error {
opts := DefaultIteratorOptions
opts.AllVersions = true
it := txn.NewIterator(opts)
defer it.Close()
var count int
for it.Rewind(); it.Valid(); it.Next() {
item := it.Item()
val, err := item.ValueCopy(nil)
if err != nil {
return err
}
t.Logf("Got entry: %v\n", item.Version())
require.Equal(t, entries[count].key, item.Key())
require.Equal(t, entries[count].val, val)
require.Equal(t, entries[count].version, item.Version())
require.Equal(t, entries[count].userMeta, item.UserMeta())
count++
}
require.Equal(t, count, 2)
return nil
})
require.NoError(t, err)
require.Equal(t, 3, int(db.orc.nextTs()))
}
func TestBackupRestore2(t *testing.T) {
tmpdir, err := os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(tmpdir)
s1Path := filepath.Join(tmpdir, "test1")
s2Path := filepath.Join(tmpdir, "test2")
s3Path := filepath.Join(tmpdir, "test3")
db1, err := Open(getTestOptions(s1Path))
require.NoError(t, err)
defer db1.Close()
key1 := []byte("key1")
key2 := []byte("key2")
rawValue := []byte("NotLongValue")
N := byte(251)
err = db1.Update(func(tx *Txn) error {
if err := tx.SetEntry(NewEntry(key1, rawValue)); err != nil {
return err
}
return tx.SetEntry(NewEntry(key2, rawValue))
})
require.NoError(t, err)
for i := byte(1); i < N; i++ {
err = db1.Update(func(tx *Txn) error {
if err := tx.SetEntry(NewEntry(append(key1, i), rawValue)); err != nil {
return err
}
return tx.SetEntry(NewEntry(append(key2, i), rawValue))
})
require.NoError(t, err)
}
var backup bytes.Buffer
_, err = db1.Backup(&backup, 0)
require.NoError(t, err)
fmt.Println("backup1 length:", backup.Len())
db2, err := Open(getTestOptions(s2Path))
require.NoError(t, err)
defer db2.Close()
err = db2.Load(&backup, 16)
require.NoError(t, err)
// Check nextTs is correctly set.
require.Equal(t, db1.orc.nextTs(), db2.orc.nextTs())
for i := byte(1); i < N; i++ {
err = db2.View(func(tx *Txn) error {
k := append(key1, i)
item, err := tx.Get(k)
if err != nil {
if err == ErrKeyNotFound {
return fmt.Errorf("Key %q has been not found, but was set\n", k)
}
return err
}
v, err := item.ValueCopy(nil)
if err != nil {
return err
}
if !reflect.DeepEqual(v, rawValue) {
return fmt.Errorf("Values not match, got %v, expected %v", v, rawValue)
}
return nil
})
require.NoError(t, err)
}
for i := byte(1); i < N; i++ {
err = db2.Update(func(tx *Txn) error {
if err := tx.SetEntry(NewEntry(append(key1, i), rawValue)); err != nil {
return err
}
return tx.SetEntry(NewEntry(append(key2, i), rawValue))
})
require.NoError(t, err)
}
backup.Reset()
_, err = db2.Backup(&backup, 0)
require.NoError(t, err)
fmt.Println("backup2 length:", backup.Len())
db3, err := Open(getTestOptions(s3Path))
require.NoError(t, err)
defer db3.Close()
err = db3.Load(&backup, 16)
require.NoError(t, err)
// Check nextTs is correctly set.
require.Equal(t, db2.orc.nextTs(), db3.orc.nextTs())
for i := byte(1); i < N; i++ {
err = db3.View(func(tx *Txn) error {
k := append(key1, i)
item, err := tx.Get(k)
if err != nil {
if err == ErrKeyNotFound {
return fmt.Errorf("Key %q has been not found, but was set\n", k)
}
return err
}
v, err := item.ValueCopy(nil)
if err != nil {
return err
}
if !reflect.DeepEqual(v, rawValue) {
return fmt.Errorf("Values not match, got %v, expected %v", v, rawValue)
}
return nil
})
require.NoError(t, err)
}
}
var randSrc = rand.NewSource(time.Now().UnixNano())
func createEntries(n int) []*pb.KV {
entries := make([]*pb.KV, n)
for i := 0; i < n; i++ {
entries[i] = &pb.KV{
Key: []byte(fmt.Sprint("key", i)),
Value: []byte{1},
UserMeta: []byte{0},
Meta: []byte{0},
}
}
return entries
}
func populateEntries(db *DB, entries []*pb.KV) error {
return db.Update(func(txn *Txn) error {
var err error
for i, e := range entries {
if err = txn.SetEntry(NewEntry(e.Key, e.Value)); err != nil {
return err
}
entries[i].Version = 1
}
return nil
})
}
func TestBackup(t *testing.T) {
test := func(t *testing.T, db *DB) {
var bb bytes.Buffer
N := 1000
entries := createEntries(N)
require.NoError(t, populateEntries(db, entries))
_, err := db.Backup(&bb, 0)
require.NoError(t, err)
err = db.View(func(txn *Txn) error {
opts := DefaultIteratorOptions
it := txn.NewIterator(opts)
defer it.Close()
var count int
for it.Rewind(); it.Valid(); it.Next() {
item := it.Item()
idx, err := strconv.Atoi(string(item.Key())[3:])
if err != nil {
return err
}
if idx > N || !bytes.Equal(entries[idx].Key, item.Key()) {
return fmt.Errorf("%s: %s", string(item.Key()), ErrKeyNotFound)
}
count++
}
if N != count {
return fmt.Errorf("wrong number of items: %d expected, %d actual", N, count)
}
return nil
})
require.NoError(t, err)
}
t.Run("disk mode", func(t *testing.T) {
tmpdir, err := os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(tmpdir)
opt := DefaultOptions(filepath.Join(tmpdir, "backup0"))
runBadgerTest(t, &opt, func(t *testing.T, db *DB) {
test(t, db)
})
})
t.Run("InMemory mode", func(t *testing.T) {
opt := DefaultOptions("")
opt.InMemory = true
runBadgerTest(t, &opt, func(t *testing.T, db *DB) {
test(t, db)
})
})
}
func TestBackupRestore3(t *testing.T) {
var bb bytes.Buffer
tmpdir, err := os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(tmpdir)
N := 1000
entries := createEntries(N)
var db1NextTs uint64
// backup
{
db1, err := Open(DefaultOptions(filepath.Join(tmpdir, "backup1")))
require.NoError(t, err)
defer db1.Close()
require.NoError(t, populateEntries(db1, entries))
_, err = db1.Backup(&bb, 0)
require.NoError(t, err)
db1NextTs = db1.orc.nextTs()
require.NoError(t, db1.Close())
}
require.True(t, len(entries) == N)
require.True(t, bb.Len() > 0)
// restore
db2, err := Open(DefaultOptions(filepath.Join(tmpdir, "restore1")))
require.NoError(t, err)
defer db2.Close()
require.NotEqual(t, db1NextTs, db2.orc.nextTs())
require.NoError(t, db2.Load(&bb, 16))
require.Equal(t, db1NextTs, db2.orc.nextTs())
// verify
err = db2.View(func(txn *Txn) error {
opts := DefaultIteratorOptions
it := txn.NewIterator(opts)
defer it.Close()
var count int
for it.Rewind(); it.Valid(); it.Next() {
item := it.Item()
idx, err := strconv.Atoi(string(item.Key())[3:])
if err != nil {
return err
}
if idx > N || !bytes.Equal(entries[idx].Key, item.Key()) {
return fmt.Errorf("%s: %s", string(item.Key()), ErrKeyNotFound)
}
count++
}
if N != count {
return fmt.Errorf("wrong number of items: %d expected, %d actual", N, count)
}
return nil
})
require.NoError(t, err)
}
func TestBackupLoadIncremental(t *testing.T) {
tmpdir, err := os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(tmpdir)
N := 100
entries := createEntries(N)
updates := make(map[int]byte)
var bb bytes.Buffer
var db1NextTs uint64
// backup
{
db1, err := Open(DefaultOptions(filepath.Join(tmpdir, "backup2")))
require.NoError(t, err)
defer db1.Close()
require.NoError(t, populateEntries(db1, entries))
since, err := db1.Backup(&bb, 0)
require.NoError(t, err)
ints := rand.New(randSrc).Perm(N)
// pick 10 items to mark as deleted.
err = db1.Update(func(txn *Txn) error {
for _, i := range ints[:10] {
if err := txn.Delete(entries[i].Key); err != nil {
return err
}
updates[i] = bitDelete
}
return nil
})
require.NoError(t, err)
since, err = db1.Backup(&bb, since)
require.NoError(t, err)
// pick 5 items to mark as expired.
err = db1.Update(func(txn *Txn) error {
for _, i := range (ints)[10:15] {
entry := NewEntry(entries[i].Key, entries[i].Value).WithTTL(-time.Hour)
if err := txn.SetEntry(entry); err != nil {
return err
}
updates[i] = bitDelete // expired
}
return nil
})
require.NoError(t, err)
since, err = db1.Backup(&bb, since)
require.NoError(t, err)
// pick 5 items to mark as discard.
err = db1.Update(func(txn *Txn) error {
for _, i := range ints[15:20] {
entry := NewEntry(entries[i].Key, entries[i].Value).WithDiscard()
if err := txn.SetEntry(entry); err != nil {
return err
}
updates[i] = bitDiscardEarlierVersions
}
return nil
})
require.NoError(t, err)
_, err = db1.Backup(&bb, since)
require.NoError(t, err)
db1NextTs = db1.orc.nextTs()
require.NoError(t, db1.Close())
}
require.True(t, len(entries) == N)
require.True(t, bb.Len() > 0)
// restore
db2, err := Open(getTestOptions(filepath.Join(tmpdir, "restore2")))
require.NoError(t, err)
defer db2.Close()
require.NotEqual(t, db1NextTs, db2.orc.nextTs())
require.NoError(t, db2.Load(&bb, 16))
require.Equal(t, db1NextTs, db2.orc.nextTs())
// verify
actual := make(map[int]byte)
err = db2.View(func(txn *Txn) error {
opts := DefaultIteratorOptions
opts.AllVersions = true
it := txn.NewIterator(opts)
defer it.Close()
var count int
for it.Rewind(); it.Valid(); it.Next() {
item := it.Item()
idx, err := strconv.Atoi(string(item.Key())[3:])
if err != nil {
return err
}
if item.IsDeletedOrExpired() {
_, ok := updates[idx]
if !ok {
return fmt.Errorf("%s: not expected to be updated but it is",
string(item.Key()))
}
actual[idx] = item.meta
count++
continue
}
}
if len(updates) != count {
return fmt.Errorf("mismatched updated items: %d expected, %d actual",
len(updates), count)
}
return nil
})
require.NoError(t, err, "%v %v", updates, actual)
}
func TestBackupBitClear(t *testing.T) {
dir, err := os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(dir)
opt := getTestOptions(dir)
opt.ValueThreshold = 10 // This is important
db, err := Open(opt)
require.NoError(t, err)
key := []byte("foo")
val := []byte(fmt.Sprintf("%0100d", 1))
require.Greater(t, int64(len(val)), db.valueThreshold())
err = db.Update(func(txn *Txn) error {
e := NewEntry(key, val)
// Value > valueTheshold so bitValuePointer will be set.
return txn.SetEntry(e)
})
require.NoError(t, err)
// Use different directory.
dir, err = os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(dir)
bak, err := os.CreateTemp(dir, "badgerbak")
require.NoError(t, err)
_, err = db.Backup(bak, 0)
require.NoError(t, err)
require.NoError(t, bak.Close())
oldValue := db.orc.nextTs()
require.NoError(t, db.Close())
opt = getTestOptions(dir)
opt.ValueThreshold = 200 // This is important.
db, err = Open(opt)
require.NoError(t, err)
defer db.Close()
bak, err = os.Open(bak.Name())
require.NoError(t, err)
defer bak.Close()
require.NoError(t, db.Load(bak, 16))
// Ensure nextTs is still the same.
require.Equal(t, oldValue, db.orc.nextTs())
require.NoError(t, db.View(func(txn *Txn) error {
e, err := txn.Get(key)
require.NoError(t, err)
v, err := e.ValueCopy(nil)
require.NoError(t, err)
require.Equal(t, val, v)
return nil
}))
}
================================================
FILE: batch.go
================================================
/*
* SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.
* SPDX-License-Identifier: Apache-2.0
*/
package badger
import (
"errors"
"fmt"
"sync"
"sync/atomic"
"google.golang.org/protobuf/proto"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
// WriteBatch holds the necessary info to perform batched writes.
type WriteBatch struct {
sync.Mutex
txn *Txn
db *DB
throttle *y.Throttle
err atomic.Value
isManaged bool
commitTs uint64
finished bool
}
// NewWriteBatch creates a new WriteBatch. This provides a way to conveniently do a lot of writes,
// batching them up as tightly as possible in a single transaction and using callbacks to avoid
// waiting for them to commit, thus achieving good performance. This API hides away the logic of
// creating and committing transactions. Due to the nature of SSI guaratees provided by Badger,
// blind writes can never encounter transaction conflicts (ErrConflict).
func (db *DB) NewWriteBatch() *WriteBatch {
if db.opt.managedTxns {
panic("cannot use NewWriteBatch in managed mode. Use NewWriteBatchAt instead")
}
return db.newWriteBatch(false)
}
func (db *DB) newWriteBatch(isManaged bool) *WriteBatch {
return &WriteBatch{
db: db,
isManaged: isManaged,
txn: db.newTransaction(true, isManaged),
throttle: y.NewThrottle(16),
}
}
// SetMaxPendingTxns sets a limit on maximum number of pending transactions while writing batches.
// This function should be called before using WriteBatch. Default value of MaxPendingTxns is
// 16 to minimise memory usage.
func (wb *WriteBatch) SetMaxPendingTxns(max int) {
wb.throttle = y.NewThrottle(max)
}
// Cancel function must be called if there's a chance that Flush might not get
// called. If neither Flush or Cancel is called, the transaction oracle would
// never get a chance to clear out the row commit timestamp map, thus causing an
// unbounded memory consumption. Typically, you can call Cancel as a defer
// statement right after NewWriteBatch is called.
//
// Note that any committed writes would still go through despite calling Cancel.
func (wb *WriteBatch) Cancel() {
wb.Lock()
defer wb.Unlock()
wb.finished = true
if err := wb.throttle.Finish(); err != nil {
wb.db.opt.Errorf("WatchBatch.Cancel error while finishing: %v", err)
}
wb.txn.Discard()
}
func (wb *WriteBatch) callback(err error) {
// sync.WaitGroup is thread-safe, so it doesn't need to be run inside wb.Lock.
defer wb.throttle.Done(err)
if err == nil {
return
}
if err := wb.Error(); err != nil {
return
}
wb.err.Store(err)
}
func (wb *WriteBatch) writeKV(kv *pb.KV) error {
e := Entry{Key: kv.Key, Value: kv.Value}
if len(kv.UserMeta) > 0 {
e.UserMeta = kv.UserMeta[0]
}
y.AssertTrue(kv.Version != 0)
e.version = kv.Version
return wb.handleEntry(&e)
}
func (wb *WriteBatch) Write(buf *z.Buffer) error {
wb.Lock()
defer wb.Unlock()
err := buf.SliceIterate(func(s []byte) error {
kv := &pb.KV{}
if err := proto.Unmarshal(s, kv); err != nil {
return err
}
return wb.writeKV(kv)
})
return err
}
func (wb *WriteBatch) WriteList(kvList *pb.KVList) error {
wb.Lock()
defer wb.Unlock()
for _, kv := range kvList.Kv {
if err := wb.writeKV(kv); err != nil {
return err
}
}
return nil
}
// SetEntryAt is the equivalent of Txn.SetEntry but it also allows setting version for the entry.
// SetEntryAt can be used only in managed mode.
func (wb *WriteBatch) SetEntryAt(e *Entry, ts uint64) error {
if !wb.db.opt.managedTxns {
return errors.New("SetEntryAt can only be used in managed mode. Use SetEntry instead")
}
e.version = ts
return wb.SetEntry(e)
}
// Should be called with lock acquired.
func (wb *WriteBatch) handleEntry(e *Entry) error {
if err := wb.txn.SetEntry(e); err != ErrTxnTooBig {
return err
}
// Txn has reached it's zenith. Commit now.
if cerr := wb.commit(); cerr != nil {
return cerr
}
// This time the error must not be ErrTxnTooBig, otherwise, we make the
// error permanent.
if err := wb.txn.SetEntry(e); err != nil {
wb.err.Store(err)
return err
}
return nil
}
// SetEntry is the equivalent of Txn.SetEntry.
func (wb *WriteBatch) SetEntry(e *Entry) error {
wb.Lock()
defer wb.Unlock()
return wb.handleEntry(e)
}
// Set is equivalent of Txn.Set().
func (wb *WriteBatch) Set(k, v []byte) error {
e := &Entry{Key: k, Value: v}
return wb.SetEntry(e)
}
// DeleteAt is equivalent of Txn.Delete but accepts a delete timestamp.
func (wb *WriteBatch) DeleteAt(k []byte, ts uint64) error {
e := Entry{Key: k, meta: bitDelete, version: ts}
return wb.SetEntry(&e)
}
// Delete is equivalent of Txn.Delete.
func (wb *WriteBatch) Delete(k []byte) error {
wb.Lock()
defer wb.Unlock()
if err := wb.txn.Delete(k); err != ErrTxnTooBig {
return err
}
if err := wb.commit(); err != nil {
return err
}
if err := wb.txn.Delete(k); err != nil {
wb.err.Store(err)
return err
}
return nil
}
// Caller to commit must hold a write lock.
func (wb *WriteBatch) commit() error {
if err := wb.Error(); err != nil {
return err
}
if wb.finished {
return y.ErrCommitAfterFinish
}
if err := wb.throttle.Do(); err != nil {
wb.err.Store(err)
return err
}
wb.txn.CommitWith(wb.callback)
wb.txn = wb.db.newTransaction(true, wb.isManaged)
wb.txn.commitTs = wb.commitTs
return wb.Error()
}
// Flush must be called at the end to ensure that any pending writes get committed to Badger. Flush
// returns any error stored by WriteBatch.
func (wb *WriteBatch) Flush() error {
wb.Lock()
err := wb.commit()
if err != nil {
wb.Unlock()
return err
}
wb.finished = true
wb.txn.Discard()
wb.Unlock()
if err := wb.throttle.Finish(); err != nil {
if wb.Error() != nil {
return fmt.Errorf("wb.err: %w err: %w", wb.Error(), err)
}
return err
}
return wb.Error()
}
// Error returns any errors encountered so far. No commits would be run once an error is detected.
func (wb *WriteBatch) Error() error {
// If the interface conversion fails, the err will be nil.
err, _ := wb.err.Load().(error)
return err
}
================================================
FILE: batch_test.go
================================================
/*
* SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.
* SPDX-License-Identifier: Apache-2.0
*/
package badger
import (
"fmt"
"os"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/dgraph-io/badger/v4/y"
)
func TestWriteBatch(t *testing.T) {
key := func(i int) []byte {
return []byte(fmt.Sprintf("%10d", i))
}
val := func(i int) []byte {
return []byte(fmt.Sprintf("%128d", i))
}
test := func(t *testing.T, db *DB) {
wb := db.NewWriteBatch()
defer wb.Cancel()
// Sanity check for SetEntryAt.
require.Error(t, wb.SetEntryAt(&Entry{}, 12))
N, M := 50000, 1000
start := time.Now()
for i := 0; i < N; i++ {
require.NoError(t, wb.Set(key(i), val(i)))
}
for i := 0; i < M; i++ {
require.NoError(t, wb.Delete(key(i)))
}
require.NoError(t, wb.Flush())
t.Logf("Time taken for %d writes (w/ test options): %s\n", N+M, time.Since(start))
err := db.View(func(txn *Txn) error {
itr := txn.NewIterator(DefaultIteratorOptions)
defer itr.Close()
i := M
for itr.Rewind(); itr.Valid(); itr.Next() {
item := itr.Item()
require.Equal(t, string(key(i)), string(item.Key()))
valcopy, err := item.ValueCopy(nil)
require.NoError(t, err)
require.Equal(t, val(i), valcopy)
i++
}
require.Equal(t, N, i)
return nil
})
require.NoError(t, err)
}
t.Run("disk mode", func(t *testing.T) {
opt := getTestOptions("")
// Set value threshold to 32 bytes otherwise write batch will generate
// too many files and we will crash with too many files open error.
opt.ValueThreshold = 32
runBadgerTest(t, &opt, func(t *testing.T, db *DB) {
test(t, db)
})
t.Logf("Disk mode done\n")
})
t.Run("InMemory mode", func(t *testing.T) {
t.Skipf("TODO(ibrahim): Please fix this")
opt := getTestOptions("")
opt.InMemory = true
db, err := Open(opt)
require.NoError(t, err)
test(t, db)
t.Logf("Disk mode done\n")
require.NoError(t, db.Close())
})
}
// This test ensures we don't end up in deadlock in case of empty writebatch.
func TestEmptyWriteBatch(t *testing.T) {
t.Run("normal mode", func(t *testing.T) {
runBadgerTest(t, nil, func(t *testing.T, db *DB) {
wb := db.NewWriteBatch()
require.NoError(t, wb.Flush())
wb = db.NewWriteBatch()
require.NoError(t, wb.Flush())
wb = db.NewWriteBatch()
// Flush commits inner txn and sets a new one instead.
// Thus we need to save it to check if it was discarded.
txn := wb.txn
require.NoError(t, wb.Flush())
// check that flushed txn was discarded and marked as read.
require.True(t, txn.discarded)
})
})
t.Run("managed mode", func(t *testing.T) {
opt := getTestOptions("")
opt.managedTxns = true
runBadgerTest(t, &opt, func(t *testing.T, db *DB) {
t.Run("WriteBatchAt", func(t *testing.T) {
wb := db.NewWriteBatchAt(2)
require.NoError(t, wb.Flush())
wb = db.NewWriteBatchAt(208)
require.NoError(t, wb.Flush())
wb = db.NewWriteBatchAt(31)
require.NoError(t, wb.Flush())
})
t.Run("ManagedWriteBatch", func(t *testing.T) {
wb := db.NewManagedWriteBatch()
require.NoError(t, wb.Flush())
wb = db.NewManagedWriteBatch()
require.NoError(t, wb.Flush())
wb = db.NewManagedWriteBatch()
require.NoError(t, wb.Flush())
})
})
})
}
// This test ensures we don't panic during flush.
// See issue: https://github.com/dgraph-io/badger/issues/1394
func TestFlushPanic(t *testing.T) {
t.Run("flush after flush", func(t *testing.T) {
runBadgerTest(t, nil, func(t *testing.T, db *DB) {
wb := db.NewWriteBatch()
wb.Flush()
require.Error(t, y.ErrCommitAfterFinish, wb.Flush())
})
})
t.Run("flush after cancel", func(t *testing.T) {
runBadgerTest(t, nil, func(t *testing.T, db *DB) {
wb := db.NewWriteBatch()
wb.Cancel()
require.Error(t, y.ErrCommitAfterFinish, wb.Flush())
})
})
}
func TestBatchErrDeadlock(t *testing.T) {
dir, err := os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(dir)
opt := DefaultOptions(dir)
db, err := OpenManaged(opt)
require.NoError(t, err)
wb := db.NewManagedWriteBatch()
require.NoError(t, wb.SetEntryAt(&Entry{Key: []byte("foo")}, 0))
require.Error(t, wb.Flush())
require.NoError(t, db.Close())
}
================================================
FILE: changes.sh
================================================
#!/bin/bash
set -e
GHORG=${GHORG:-dgraph-io}
GHREPO=${GHREPO:-badger}
cat <<EOF
This description was generated using this script:
\`\`\`sh
$(cat "$0")
\`\`\`
Invoked as:
$(echo GHORG="${GHORG}" GHREPO="${GHREPO}" $(basename "$0") ${@:1})
EOF
git log --oneline --reverse ${@:1} |
sed -E "s/^(\S{7}\s)//g" |
sed -E "s/([\s|\(| ])#([0-9]+)/\1${GHORG}\/${GHREPO}#\2/g"
================================================
FILE: compaction.go
================================================
/*
* SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.
* SPDX-License-Identifier: Apache-2.0
*/
package badger
import (
"bytes"
"fmt"
"log"
"math"
"sync"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
)
type keyRange struct {
left []byte
right []byte
inf bool
size int64 // size is used for Key splits.
}
func (r keyRange) isEmpty() bool {
return len(r.left) == 0 && len(r.right) == 0 && !r.inf
}
var infRange = keyRange{inf: true}
func (r keyRange) String() string {
return fmt.Sprintf("[left=%x, right=%x, inf=%v]", r.left, r.right, r.inf)
}
func (r keyRange) equals(dst keyRange) bool {
return bytes.Equal(r.left, dst.left) &&
bytes.Equal(r.right, dst.right) &&
r.inf == dst.inf
}
func (r *keyRange) extend(kr keyRange) {
// TODO(ibrahim): Is this needed?
if kr.isEmpty() {
return
}
if r.isEmpty() {
*r = kr
}
if len(r.left) == 0 || y.CompareKeys(kr.left, r.left) < 0 {
r.left = kr.left
}
if len(r.right) == 0 || y.CompareKeys(kr.right, r.right) > 0 {
r.right = kr.right
}
if kr.inf {
r.inf = true
}
}
func (r keyRange) overlapsWith(dst keyRange) bool {
// Empty keyRange always overlaps.
if r.isEmpty() {
return true
}
// TODO(ibrahim): Do you need this?
// Empty dst doesn't overlap with anything.
if dst.isEmpty() {
return false
}
if r.inf || dst.inf {
return true
}
// [dst.left, dst.right] ... [r.left, r.right]
// If my left is greater than dst right, we have no overlap.
if y.CompareKeys(r.left, dst.right) > 0 {
return false
}
// [r.left, r.right] ... [dst.left, dst.right]
// If my right is less than dst left, we have no overlap.
if y.CompareKeys(r.right, dst.left) < 0 {
return false
}
// We have overlap.
return true
}
// getKeyRange returns the smallest and the biggest in the list of tables.
// TODO(naman): Write a test for this. The smallest and the biggest should
// be the smallest of the leftmost table and the biggest of the right most table.
func getKeyRange(tables ...*table.Table) keyRange {
if len(tables) == 0 {
return keyRange{}
}
smallest := tables[0].Smallest()
biggest := tables[0].Biggest()
for i := 1; i < len(tables); i++ {
if y.CompareKeys(tables[i].Smallest(), smallest) < 0 {
smallest = tables[i].Smallest()
}
if y.CompareKeys(tables[i].Biggest(), biggest) > 0 {
biggest = tables[i].Biggest()
}
}
// We pick all the versions of the smallest and the biggest key. Note that version zero would
// be the rightmost key, considering versions are default sorted in descending order.
return keyRange{
left: y.KeyWithTs(y.ParseKey(smallest), math.MaxUint64),
right: y.KeyWithTs(y.ParseKey(biggest), 0),
}
}
type levelCompactStatus struct {
ranges []keyRange
delSize int64
}
func (lcs *levelCompactStatus) debug() string {
var b bytes.Buffer
for _, r := range lcs.ranges {
b.WriteString(r.String())
}
return b.String()
}
func (lcs *levelCompactStatus) overlapsWith(dst keyRange) bool {
for _, r := range lcs.ranges {
if r.overlapsWith(dst) {
return true
}
}
return false
}
func (lcs *levelCompactStatus) remove(dst keyRange) bool {
final := lcs.ranges[:0]
var found bool
for _, r := range lcs.ranges {
if !r.equals(dst) {
final = append(final, r)
} else {
found = true
}
}
lcs.ranges = final
return found
}
type compactStatus struct {
sync.RWMutex
levels []*levelCompactStatus
tables map[uint64]struct{}
}
func (cs *compactStatus) overlapsWith(level int, this keyRange) bool {
cs.RLock()
defer cs.RUnlock()
thisLevel := cs.levels[level]
return thisLevel.overlapsWith(this)
}
func (cs *compactStatus) delSize(l int) int64 {
cs.RLock()
defer cs.RUnlock()
return cs.levels[l].delSize
}
type thisAndNextLevelRLocked struct{}
// compareAndAdd will check whether we can run this compactDef. That it doesn't overlap with any
// other running compaction. If it can be run, it would store this run in the compactStatus state.
func (cs *compactStatus) compareAndAdd(_ thisAndNextLevelRLocked, cd compactDef) bool {
cs.Lock()
defer cs.Unlock()
tl := cd.thisLevel.level
y.AssertTruef(tl < len(cs.levels), "Got level %d. Max levels: %d", tl, len(cs.levels))
thisLevel := cs.levels[cd.thisLevel.level]
nextLevel := cs.levels[cd.nextLevel.level]
if thisLevel.overlapsWith(cd.thisRange) {
return false
}
if nextLevel.overlapsWith(cd.nextRange) {
return false
}
// Check whether this level really needs compaction or not. Otherwise, we'll end up
// running parallel compactions for the same level.
// Update: We should not be checking size here. Compaction priority already did the size checks.
// Here we should just be executing the wish of others.
thisLevel.ranges = append(thisLevel.ranges, cd.thisRange)
nextLevel.ranges = append(nextLevel.ranges, cd.nextRange)
thisLevel.delSize += cd.thisSize
for _, t := range append(cd.top, cd.bot...) {
cs.tables[t.ID()] = struct{}{}
}
return true
}
func (cs *compactStatus) delete(cd compactDef) {
cs.Lock()
defer cs.Unlock()
tl := cd.thisLevel.level
y.AssertTruef(tl < len(cs.levels), "Got level %d. Max levels: %d", tl, len(cs.levels))
thisLevel := cs.levels[cd.thisLevel.level]
nextLevel := cs.levels[cd.nextLevel.level]
thisLevel.delSize -= cd.thisSize
found := thisLevel.remove(cd.thisRange)
// The following check makes sense only if we're compacting more than one
// table. In case of the max level, we might rewrite a single table to
// remove stale data.
if cd.thisLevel != cd.nextLevel && !cd.nextRange.isEmpty() {
found = nextLevel.remove(cd.nextRange) && found
}
if !found {
this := cd.thisRange
next := cd.nextRange
fmt.Printf("Looking for: %s in this level %d.\n", this, tl)
fmt.Printf("This Level:\n%s\n", thisLevel.debug())
fmt.Println()
fmt.Printf("Looking for: %s in next level %d.\n", next, cd.nextLevel.level)
fmt.Printf("Next Level:\n%s\n", nextLevel.debug())
log.Fatal("keyRange not found")
}
for _, t := range append(cd.top, cd.bot...) {
_, ok := cs.tables[t.ID()]
y.AssertTrue(ok)
delete(cs.tables, t.ID())
}
}
================================================
FILE: contrib/RELEASE.md
================================================
# Badger Release Process
This document outlines the steps needed to build and push a new release of Badger.
1. Have a team member "at-the-ready" with github `writer` access (you'll need them to approve PRs).
1. Create a new branch (prepare-for-release-vXX.X.X, for instance).
1. Update dependencies in `go.mod` for Ristretto, if required.
1. Update the CHANGELOG.md. Opus 4.5 does a great job of doing this. Example prompt:
`I'm releasing vXX.X.X off the main branch, add a new entry for this release. Conform to the`
`"Keep a Changelog" format, use past entries as a formatting guide. Run the trunk linter.`
1. Validate the version does not have storage incompatibilities with the previous version. If so,
add a warning to the CHANGELOG.md that export/import of data will need to be run as part of the
upgrade process.
1. Commit and push your changes. Create a PR and have a team member approve it.
1. Once your "prepare for release branch" is merged into main, on the github
[releases page](https://github.com/dgraph-io/badger/releases), create a new draft release.
1. Start the deployment workflow from the
[CD workflow page](https://github.com/dgraph-io/badger/actions/workflows/cd-badger.yml).
The CD workflow handles the building and copying of release artifacts to the releases area.
1. For all major and minor releases (non-patches), create a release branch. In order to easily
backport fixes to the release branch, create a release branch from the tag head. For instance, if
we're releasing v4.9.0, create a branch called `release/v4.9` from the tag head (ensure you're on
the main branch from which you created the tag):
```sh
git checkout main
git pull origin main
git checkout -b release/v4.9
git push origin release/v4.9
```
1. Splash the "go index" cache to ensure that the latest release is available to the public with:
```sh
go list -m github.com/dgraph-io/badger/v4@vX.X.X
```
1. If needed, create a new announcement thread in the
[Discussions](https://github.com/orgs/dgraph-io/discussions) forum for the release.
================================================
FILE: db.go
================================================
/*
* SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.
* SPDX-License-Identifier: Apache-2.0
*/
package badger
import (
"bytes"
"context"
"encoding/binary"
"errors"
"expvar"
"fmt"
"math"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"sync/atomic"
"time"
humanize "github.com/dustin/go-humanize"
"github.com/dgraph-io/badger/v4/fb"
"github.com/dgraph-io/badger/v4/options"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/skl"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2"
"github.com/dgraph-io/ristretto/v2/z"
)
var (
badgerPrefix = []byte("!badger!") // Prefix for internal keys used by badger.
txnKey = []byte("!badger!txn") // For indicating end of entries in txn.
bannedNsKey = []byte("!badger!banned") // For storing the banned namespaces.
)
type closers struct {
updateSize *z.Closer
compactors *z.Closer
memtable *z.Closer
writes *z.Closer
valueGC *z.Closer
pub *z.Closer
cacheHealth *z.Closer
}
type lockedKeys struct {
sync.RWMutex
keys map[uint64]struct{}
}
func (lk *lockedKeys) add(key uint64) {
lk.Lock()
defer lk.Unlock()
lk.keys[key] = struct{}{}
}
func (lk *lockedKeys) has(key uint64) bool {
lk.RLock()
defer lk.RUnlock()
_, ok := lk.keys[key]
return ok
}
func (lk *lockedKeys) all() []uint64 {
lk.RLock()
defer lk.RUnlock()
keys := make([]uint64, 0, len(lk.keys))
for key := range lk.keys {
keys = append(keys, key)
}
return keys
}
// DB provides the various functions required to interact with Badger.
// DB is thread-safe.
type DB struct {
testOnlyDBExtensions
lock sync.RWMutex // Guards list of inmemory tables, not individual reads and writes.
dirLockGuard *directoryLockGuard
// nil if Dir and ValueDir are the same
valueDirGuard *directoryLockGuard
closers closers
mt *memTable // Our latest (actively written) in-memory table
imm []*memTable // Add here only AFTER pushing to flushChan.
// Initialized via openMemTables.
nextMemFid int
opt Options
manifest *manifestFile
lc *levelsController
vlog valueLog
writeCh chan *request
flushChan chan *memTable // For flushing memtables.
closeOnce sync.Once // For closing DB only once.
blockWrites atomic.Int32
isClosed atomic.Uint32
orc *oracle
bannedNamespaces *lockedKeys
threshold *vlogThreshold
pub *publisher
registry *KeyRegistry
blockCache *ristretto.Cache[[]byte, *table.Block]
indexCache *ristretto.Cache[uint64, *fb.TableIndex]
allocPool *z.AllocatorPool
}
const (
kvWriteChCapacity = 1000
)
func checkAndSetOptions(opt *Options) error {
// It's okay to have zero compactors which will disable all compactions but
// we cannot have just one compactor otherwise we will end up with all data
// on level 2.
if opt.NumCompactors == 1 {
return errors.New("Cannot have 1 compactor. Need at least 2")
}
if opt.InMemory && (opt.Dir != "" || opt.ValueDir != "") {
return errors.New("Cannot use badger in Disk-less mode with Dir or ValueDir set")
}
opt.maxBatchSize = (15 * opt.MemTableSize) / 100
opt.maxBatchCount = opt.maxBatchSize / int64(skl.MaxNodeSize)
// This is the maximum value, vlogThreshold can have if dynamic thresholding is enabled.
opt.maxValueThreshold = math.Min(maxValueThreshold, float64(opt.maxBatchSize))
if opt.VLogPercentile < 0.0 || opt.VLogPercentile > 1.0 {
return errors.New("vlogPercentile must be within range of 0.0-1.0")
}
// We are limiting opt.ValueThreshold to maxValueThreshold for now.
if opt.ValueThreshold > maxValueThreshold {
return fmt.Errorf("Invalid ValueThreshold, must be less or equal to %d",
maxValueThreshold)
}
// If ValueThreshold is greater than opt.maxBatchSize, we won't be able to push any data using
// the transaction APIs. Transaction batches entries into batches of size opt.maxBatchSize.
if opt.ValueThreshold > opt.maxBatchSize {
return fmt.Errorf("Valuethreshold %d greater than max batch size of %d. Either "+
"reduce opt.ValueThreshold or increase opt.BaseTableSize.",
opt.ValueThreshold, opt.maxBatchSize)
}
// ValueLogFileSize should be strictly LESS than 2<<30 otherwise we will
// overflow the uint32 when we mmap it in OpenMemtable.
if !(opt.ValueLogFileSize < 2<<30 && opt.ValueLogFileSize >= 1<<20) {
return ErrValueLogSize
}
if opt.ReadOnly {
// Do not perform compaction in read only mode.
opt.CompactL0OnClose = false
}
needCache := (opt.Compression != options.None) || (len(opt.EncryptionKey) > 0)
if needCache && opt.BlockCacheSize == 0 {
panic("BlockCacheSize should be set since compression/encryption are enabled")
}
return nil
}
// Open returns a new DB object.
func Open(opt Options) (*DB, error) {
if err := checkAndSetOptions(&opt); err != nil {
return nil, err
}
var dirLockGuard, valueDirLockGuard *directoryLockGuard
// Create directories and acquire lock on it only if badger is not running in InMemory mode.
// We don't have any directories/files in InMemory mode so we don't need to acquire
// any locks on them.
if !opt.InMemory {
if err := createDirs(opt); err != nil {
return nil, err
}
var err error
if !opt.BypassLockGuard {
dirLockGuard, err = acquireDirectoryLock(opt.Dir, lockFile, opt.ReadOnly)
if err != nil {
return nil, err
}
defer func() {
if dirLockGuard != nil {
_ = dirLockGuard.release()
}
}()
absDir, err := filepath.Abs(opt.Dir)
if err != nil {
return nil, err
}
absValueDir, err := filepath.Abs(opt.ValueDir)
if err != nil {
return nil, err
}
if absValueDir != absDir {
valueDirLockGuard, err = acquireDirectoryLock(opt.ValueDir, lockFile, opt.ReadOnly)
if err != nil {
return nil, err
}
defer func() {
if valueDirLockGuard != nil {
_ = valueDirLockGuard.release()
}
}()
}
}
}
manifestFile, manifest, err := openOrCreateManifestFile(opt)
if err != nil {
return nil, err
}
defer func() {
if manifestFile != nil {
_ = manifestFile.close()
}
}()
db := &DB{
imm: make([]*memTable, 0, opt.NumMemtables),
flushChan: make(chan *memTable, opt.NumMemtables),
writeCh: make(chan *request, kvWriteChCapacity),
opt: opt,
manifest: manifestFile,
dirLockGuard: dirLockGuard,
valueDirGuard: valueDirLockGuard,
orc: newOracle(opt),
pub: newPublisher(),
allocPool: z.NewAllocatorPool(8),
bannedNamespaces: &lockedKeys{keys: make(map[uint64]struct{})},
threshold: initVlogThreshold(&opt),
}
db.syncChan = opt.syncChan
// Cleanup all the goroutines started by badger in case of an error.
defer func() {
if err != nil {
opt.Errorf("Received err: %v. Cleaning up...", err)
db.cleanup()
db = nil
}
}()
if opt.BlockCacheSize > 0 {
numInCache := opt.BlockCacheSize / int64(opt.BlockSize)
if numInCache == 0 {
// Make the value of this variable at least one since the cache requires
// the number of counters to be greater than zero.
numInCache = 1
}
config := ristretto.Config[[]byte, *table.Block]{
NumCounters: numInCache * 8,
MaxCost: opt.BlockCacheSize,
BufferItems: 64,
Metrics: true,
OnExit: table.BlockEvictHandler,
}
db.blockCache, err = ristretto.NewCache[[]byte, *table.Block](&config)
if err != nil {
return nil, y.Wrap(err, "failed to create data cache")
}
}
if opt.IndexCacheSize > 0 {
// Index size is around 5% of the table size.
indexSz := int64(float64(opt.MemTableSize) * 0.05)
numInCache := opt.IndexCacheSize / indexSz
if numInCache == 0 {
// Make the value of this variable at least one since the cache requires
// the number of counters to be greater than zero.
numInCache = 1
}
config := ristretto.Config[uint64, *fb.TableIndex]{
NumCounters: numInCache * 8,
MaxCost: opt.IndexCacheSize,
BufferItems: 64,
Metrics: true,
}
db.indexCache, err = ristretto.NewCache(&config)
if err != nil {
return nil, y.Wrap(err, "failed to create bf cache")
}
}
db.closers.cacheHealth = z.NewCloser(1)
go db.monitorCache(db.closers.cacheHealth)
if db.opt.InMemory {
db.opt.SyncWrites = false
// If badger is running in memory mode, push everything into the LSM Tree.
db.opt.ValueThreshold = math.MaxInt32
}
krOpt := KeyRegistryOptions{
ReadOnly: opt.ReadOnly,
Dir: opt.Dir,
EncryptionKey: opt.EncryptionKey,
EncryptionKeyRotationDuration: opt.EncryptionKeyRotationDuration,
InMemory: opt.InMemory,
}
if db.registry, err = OpenKeyRegistry(krOpt); err != nil {
return db, err
}
db.calculateSize()
db.closers.updateSize = z.NewCloser(1)
go db.updateSize(db.closers.updateSize)
if err := db.openMemTables(db.opt); err != nil {
return nil, y.Wrapf(err, "while opening memtables")
}
if !db.opt.ReadOnly {
if db.mt, err = db.newMemTable(); err != nil {
return nil, y.Wrapf(err, "cannot create memtable")
}
}
// newLevelsController potentially loads files in directory.
if db.lc, err = newLevelsController(db, &manifest); err != nil {
return db, err
}
// Initialize vlog struct.
db.vlog.init(db)
if !opt.ReadOnly {
db.closers.compactors = z.NewCloser(1)
db.lc.startCompact(db.closers.compactors)
db.closers.memtable = z.NewCloser(1)
go func() {
db.flushMemtable(db.closers.memtable) // Need levels controller to be up.
}()
// Flush them to disk asap.
for _, mt := range db.imm {
db.flushChan <- mt
}
}
// We do increment nextTxnTs below. So, no need to do it here.
db.orc.nextTxnTs = db.MaxVersion()
db.opt.Infof("Set nextTxnTs to %d", db.orc.nextTxnTs)
if err = db.vlog.open(db); err != nil {
return db, y.Wrapf(err, "During db.vlog.open")
}
// Let's advance nextTxnTs to one more than whatever we observed via
// replaying the logs.
db.orc.txnMark.Done(db.orc.nextTxnTs)
// In normal mode, we must update readMark so older versions of keys can be removed during
// compaction when run in offline mode via the flatten tool.
db.orc.readMark.Done(db.orc.nextTxnTs)
db.orc.incrementNextTs()
go db.threshold.listenForValueThresholdUpdate()
if err := db.initBannedNamespaces(); err != nil {
return db, fmt.Errorf("While setting banned keys: %w", err)
}
db.closers.writes = z.NewCloser(1)
go db.doWrites(db.closers.writes)
if !db.opt.InMemory {
db.closers.valueGC = z.NewCloser(1)
go db.vlog.waitOnGC(db.closers.valueGC)
}
db.closers.pub = z.NewCloser(1)
go db.pub.listenForUpdates(db.closers.pub)
valueDirLockGuard = nil
dirLockGuard = nil
manifestFile = nil
return db, nil
}
// initBannedNamespaces retrieves the banned namespaces from the DB and updates in-memory structure.
func (db *DB) initBannedNamespaces() error {
if db.opt.NamespaceOffset < 0 {
return nil
}
return db.View(func(txn *Txn) error {
iopts := DefaultIteratorOptions
iopts.Prefix = bannedNsKey
iopts.PrefetchValues = false
iopts.InternalAccess = true
itr := txn.NewIterator(iopts)
defer itr.Close()
for itr.Rewind(); itr.Valid(); itr.Next() {
key := y.BytesToU64(itr.Item().Key()[len(bannedNsKey):])
db.bannedNamespaces.add(key)
}
return nil
})
}
func (db *DB) MaxVersion() uint64 {
var maxVersion uint64
update := func(a uint64) {
if a > maxVersion {
maxVersion = a
}
}
db.lock.Lock()
// In read only mode, we do not create new mem table.
if !db.opt.ReadOnly {
update(db.mt.maxVersion)
}
for _, mt := range db.imm {
update(mt.maxVersion)
}
db.lock.Unlock()
for _, ti := range db.Tables() {
update(ti.MaxVersion)
}
return maxVersion
}
func (db *DB) monitorCache(c *z.Closer) {
defer c.Done()
count := 0
analyze := func(name string, metrics *ristretto.Metrics) {
// If the mean life expectancy is less than 10 seconds, the cache
// might be too small.
le := metrics.LifeExpectancySeconds()
if le == nil {
return
}
lifeTooShort := le.Count > 0 && float64(le.Sum)/float64(le.Count) < 10
hitRatioTooLow := metrics.Ratio() > 0 && metrics.Ratio() < 0.4
if lifeTooShort && hitRatioTooLow {
db.opt.Warningf("%s might be too small. Metrics: %s\n", name, metrics)
db.opt.Warningf("Cache life expectancy (in seconds): %+v\n", le)
} else if le.Count > 1000 && count%5 == 0 {
db.opt.Infof("%s metrics: %s\n", name, metrics)
}
}
ticker := time.NewTicker(1 * time.Minute)
defer ticker.Stop()
for {
select {
case <-c.HasBeenClosed():
return
case <-ticker.C:
}
analyze("Block cache", db.BlockCacheMetrics())
analyze("Index cache", db.IndexCacheMetrics())
count++
}
}
// cleanup stops all the goroutines started by badger. This is used in open to
// cleanup goroutines in case of an error.
func (db *DB) cleanup() {
db.stopMemoryFlush()
db.stopCompactions()
db.blockCache.Close()
db.indexCache.Close()
if db.closers.updateSize != nil {
db.closers.updateSize.Signal()
}
if db.closers.valueGC != nil {
db.closers.valueGC.Signal()
}
if db.closers.writes != nil {
db.closers.writes.Signal()
}
if db.closers.pub != nil {
db.closers.pub.Signal()
}
db.orc.Stop()
// Do not use vlog.Close() here. vlog.Close truncates the files. We don't
// want to truncate files unless the user has specified the truncate flag.
}
// BlockCacheMetrics returns the metrics for the underlying block cache.
func (db *DB) BlockCacheMetrics() *ristretto.Metrics {
if db.blockCache != nil {
return db.blockCache.Metrics
}
return nil
}
// IndexCacheMetrics returns the metrics for the underlying index cache.
func (db *DB) IndexCacheMetrics() *ristretto.Metrics {
if db.indexCache != nil {
return db.indexCache.Metrics
}
return nil
}
// Close closes a DB. It's crucial to call it to ensure all the pending updates make their way to
// disk. Calling DB.Close() multiple times would still only close the DB once.
func (db *DB) Close() error {
var err error
db.closeOnce.Do(func() {
err = db.close()
})
return err
}
// IsClosed denotes if the badger DB is closed or not. A DB instance should not
// be used after closing it.
func (db *DB) IsClosed() bool {
return db.isClosed.Load() == 1
}
func (db *DB) close() (err error) {
defer db.allocPool.Release()
db.opt.Debugf("Closing database")
db.opt.Infof("Lifetime L0 stalled for: %s\n", time.Duration(db.lc.l0stallsMs.Load()))
db.blockWrites.Store(1)
db.isClosed.Store(1)
if !db.opt.InMemory {
// Stop value GC first.
db.closers.valueGC.SignalAndWait()
}
// Stop writes next.
db.closers.writes.SignalAndWait()
// Don't accept any more write.
close(db.writeCh)
db.closers.pub.SignalAndWait()
db.closers.cacheHealth.Signal()
// Make sure that block writer is done pushing stuff into memtable!
// Otherwise, you will have a race condition: we are trying to flush memtables
// and remove them completely, while the block / memtable writer is still
// trying to push stuff into the memtable. This will also resolve the value
// offset problem: as we push into memtable, we update value offsets there.
if db.mt != nil {
if db.mt.sl.Empty() {
// Remove the memtable if empty.
db.mt.DecrRef()
} else {
db.opt.Debugf("Flushing memtable")
for {
pushedMemTable := func() bool {
db.lock.Lock()
defer db.lock.Unlock()
y.AssertTrue(db.mt != nil)
select {
case db.flushChan <- db.mt:
db.imm = append(db.imm, db.mt) // Flusher will attempt to remove this from s.imm.
db.mt = nil // Will segfault if we try writing!
db.opt.Debugf("pushed to flush chan\n")
return true
default:
// If we fail to push, we need to unlock and wait for a short while.
// The flushing operation needs to update s.imm. Otherwise, we have a
// deadlock.
// TODO: Think about how to do this more cleanly, maybe without any locks.
}
return false
}()
if pushedMemTable {
break
}
time.Sleep(10 * time.Millisecond)
}
}
}
db.stopMemoryFlush()
db.stopCompactions()
// Force Compact L0
// We don't need to care about cstatus since no parallel compaction is running.
if db.opt.CompactL0OnClose {
err := db.lc.doCompact(173, compactionPriority{level: 0, score: 1.73})
switch err {
case errFillTables:
// This error only means that there might be enough tables to do a compaction. So, we
// should not report it to the end user to avoid confusing them.
case nil:
db.opt.Debugf("Force compaction on level 0 done")
default:
db.opt.Warningf("While forcing compaction on level 0: %v", err)
}
}
// Now close the value log.
if vlogErr := db.vlog.Close(); vlogErr != nil {
err = y.Wrap(vlogErr, "DB.Close")
}
db.opt.Infof(db.LevelsToString())
if lcErr := db.lc.close(); err == nil {
err = y.Wrap(lcErr, "DB.Close")
}
db.opt.Debugf("Waiting for closer")
db.closers.updateSize.SignalAndWait()
db.orc.Stop()
db.blockCache.Close()
db.indexCache.Close()
db.threshold.close()
if db.opt.InMemory {
return
}
if db.dirLockGuard != nil {
if guardErr := db.dirLockGuard.release(); err == nil {
err = y.Wrap(guardErr, "DB.Close")
}
}
if db.valueDirGuard != nil {
if guardErr := db.valueDirGuard.release(); err == nil {
err = y.Wrap(guardErr, "DB.Close")
}
}
if manifestErr := db.manifest.close(); err == nil {
err = y.Wrap(manifestErr, "DB.Close")
}
if registryErr := db.registry.Close(); err == nil {
err = y.Wrap(registryErr, "DB.Close")
}
// Fsync directories to ensure that lock file, and any other removed files whose directory
// we haven't specifically fsynced, are guaranteed to have their directory entry removal
// persisted to disk.
if syncErr := db.syncDir(db.opt.Dir); err == nil {
err = y.Wrap(syncErr, "DB.Close")
}
if syncErr := db.syncDir(db.opt.ValueDir); err == nil {
err = y.Wrap(syncErr, "DB.Close")
}
return err
}
// VerifyChecksum verifies checksum for all tables on all levels.
// This method can be used to verify checksum, if opt.ChecksumVerificationMode is NoVerification.
func (db *DB) VerifyChecksum() error {
return db.lc.verifyChecksum()
}
const (
lockFile = "LOCK"
)
// Sync syncs database content to disk. This function provides
// more control to user to sync data whenever required.
func (db *DB) Sync() error {
/**
Make an attempt to sync both the logs, the active memtable's WAL and the vLog (1847).
Cases:
- All_ok :: If both the logs sync successfully.
- Entry_Lost :: If an entry with a value pointer was present in the active memtable's WAL,
:: and the WAL was synced but there was an error in syncing the vLog.
:: The entry will be considered lost and this case will need to be handled during recovery.
- Entries_Lost :: If there were errors in syncing both the logs, multiple entries would be lost.
- Entries_Lost :: If the active memtable's WAL is not synced but the vLog is synced, it will
:: result in entries being lost because recovery of the active memtable is done from its WAL.
:: Check `UpdateSkipList` in memtable.go.
- Nothing_lost :: If an entry with its value was present in the active memtable's WAL, and the WAL was synced,
:: but there was an error in syncing the vLog.
:: Nothing is lost for this very specific entry because the entry is completely present in the memtable's WAL.
- Partially_lost :: If entries were written partially in either of the logs,
:: the logs will be truncated during recovery.
:: As a result of truncation, some entries might be lost.
:: Assume that 4KB of data is to be synced and invoking `Sync` results only in syncing 3KB
:: of data and then the machine shuts down or the disk failure happens,
:: this will result in partial writes. [[This case needs verification]]
*/
db.lock.RLock()
memtableSyncError := db.mt.SyncWAL()
db.lock.RUnlock()
vLogSyncError := db.vlog.sync()
return y.CombineErrors(memtableSyncError, vLogSyncError)
}
// getMemtables returns the current memtables and get references.
func (db *DB) getMemTables() ([]*memTable, func()) {
db.lock.RLock()
defer db.lock.RUnlock()
var tables []*memTable
// Mutable memtable does not exist in read-only mode.
if !db.opt.ReadOnly {
// Get mutable memtable.
tables = append(tables, db.mt)
db.mt.IncrRef()
}
// Get immutable memtables.
last := len(db.imm) - 1
for i := range db.imm {
tables = append(tables, db.imm[last-i])
db.imm[last-i].IncrRef()
}
return tables, func() {
for _, tbl := range tables {
tbl.DecrRef()
}
}
}
// get returns the value in memtable or disk for given key.
// Note that value will include meta byte.
//
// IMPORTANT: We should never write an entry with an older timestamp for the same key, We need to
// maintain this invariant to search for the latest value of a key, or else we need to search in all
// tables and find the max version among them. To maintain this invariant, we also need to ensure
// that all versions of a key are always present in the same table from level 1, because compaction
// can push any table down.
//
// Update(23/09/2020) - We have dropped the move key implementation. Earlier we
// were inserting move keys to fix the invalid value pointers but we no longer
// do that. For every get("fooX") call where X is the version, we will search
// for "fooX" in all the levels of the LSM tree. This is expensive but it
// removes the overhead of handling move keys completely.
func (db *DB) get(key []byte) (y.ValueStruct, error) {
if db.IsClosed() {
return y.ValueStruct{}, ErrDBClosed
}
tables, decr := db.getMemTables() // Lock should be released.
defer decr()
var maxVs y.ValueStruct
version := y.ParseTs(key)
y.NumGetsAdd(db.opt.MetricsEnabled, 1)
for i := 0; i < len(tables); i++ {
vs := tables[i].sl.Get(key)
y.NumMemtableGetsAdd(db.opt.MetricsEnabled, 1)
if vs.Meta == 0 && vs.Value == nil {
continue
}
// Found the required version of the key, return immediately.
if vs.Version == version {
y.NumGetsWithResultsAdd(db.opt.MetricsEnabled, 1)
return vs, nil
}
if maxVs.Version < vs.Version {
maxVs = vs
}
}
return db.lc.get(key, maxVs, 0)
}
var requestPool = sync.Pool{
New: func() interface{} {
return new(request)
},
}
func (db *DB) writeToLSM(b *request) error {
// We should check the length of b.Prts and b.Entries only when badger is not
// running in InMemory mode. In InMemory mode, we don't write anything to the
// value log and that's why the length of b.Ptrs will always be zero.
if !db.opt.InMemory && len(b.Ptrs) != len(b.Entries) {
return fmt.Errorf("Ptrs and Entries don't match: %+v", b)
}
for i, entry := range b.Entries {
var err error
if entry.skipVlogAndSetThreshold(db.valueThreshold()) {
// Will include deletion / tombstone case.
err = db.mt.Put(entry.Key,
y.ValueStruct{
Value: entry.Value,
// Ensure value pointer flag is removed. Otherwise, the value will fail
// to be retrieved during iterator prefetch. `bitValuePointer` is only
// known to be set in write to LSM when the entry is loaded from a backup
// with lower ValueThreshold and its value was stored in the value log.
Meta: entry.meta &^ bitValuePointer,
UserMeta: entry.UserMeta,
ExpiresAt: entry.ExpiresAt,
})
} else {
// Write pointer to Memtable.
err = db.mt.Put(entry.Key,
y.ValueStruct{
Value: b.Ptrs[i].Encode(),
Meta: entry.meta | bitValuePointer,
UserMeta: entry.UserMeta,
ExpiresAt: entry.ExpiresAt,
})
}
if err != nil {
return y.Wrapf(err, "while writing to memTable")
}
}
if db.opt.SyncWrites {
return db.mt.SyncWAL()
}
return nil
}
// writeRequests is called serially by only one goroutine.
func (db *DB) writeRequests(reqs []*request) error {
if len(reqs) == 0 {
return nil
}
done := func(err error) {
for _, r := range reqs {
r.Err = err
r.Wg.Done()
}
}
db.opt.Debugf("writeRequests called. Writing to value log")
err := db.vlog.write(reqs)
if err != nil {
done(err)
return err
}
db.opt.Debugf("Writing to memtable")
var count int
for _, b := range reqs {
if len(b.Entries) == 0 {
continue
}
count += len(b.Entries)
var i uint64
var err error
for err = db.ensureRoomForWrite(); err == errNoRoom; err = db.ensureRoomForWrite() {
i++
if i%100 == 0 {
db.opt.Debugf("Making room for writes")
}
// We need to poll a bit because both hasRoomForWrite and the flusher need access to s.imm.
// When flushChan is full and you are blocked there, and the flusher is trying to update s.imm,
// you will get a deadlock.
time.Sleep(10 * time.Millisecond)
}
if err != nil {
done(err)
return y.Wrap(err, "writeRequests")
}
if err := db.writeToLSM(b); err != nil {
done(err)
return y.Wrap(err, "writeRequests")
}
}
db.opt.Debugf("Sending updates to subscribers")
db.pub.sendUpdates(reqs)
done(nil)
db.opt.Debugf("%d entries written", count)
return nil
}
func (db *DB) sendToWriteCh(entries []*Entry) (*request, error) {
if db.blockWrites.Load() == 1 {
return nil, ErrBlockedWrites
}
var count, size int64
for _, e := range entries {
size += e.estimateSizeAndSetThreshold(db.valueThreshold())
count++
}
y.NumBytesWrittenUserAdd(db.opt.MetricsEnabled, size)
if count >= db.opt.maxBatchCount || size >= db.opt.maxBatchSize {
return nil, ErrTxnTooBig
}
// We can only service one request because we need each txn to be stored in a contiguous section.
// Txns should not interleave among other txns or rewrites.
req := requestPool.Get().(*request)
req.reset()
req.Entries = entries
req.Wg.Add(1)
req.IncrRef() // for db write
db.writeCh <- req // Handled in doWrites.
y.NumPutsAdd(db.opt.MetricsEnabled, int64(len(entries)))
return req, nil
}
func (db *DB) doWrites(lc *z.Closer) {
defer lc.Done()
pendingCh := make(chan struct{}, 1)
writeRequests := func(reqs []*request) {
if err := db.writeRequests(reqs); err != nil {
db.opt.Errorf("writeRequests: %v", err)
}
<-pendingCh
}
// This variable tracks the number of pending writes.
reqLen := new(expvar.Int)
y.PendingWritesSet(db.opt.MetricsEnabled, db.opt.Dir, reqLen)
reqs := make([]*request, 0, 10)
for {
var r *request
select {
case r = <-db.writeCh:
case <-lc.HasBeenClosed():
goto closedCase
}
for {
reqs = append(reqs, r)
reqLen.Set(int64(len(reqs)))
if len(reqs) >= 3*kvWriteChCapacity {
pendingCh <- struct{}{} // blocking.
goto writeCase
}
select {
// Either push to pending, or continue to pick from writeCh.
case r = <-db.writeCh:
case pendingCh <- struct{}{}:
goto writeCase
case <-lc.HasBeenClosed():
goto closedCase
}
}
closedCase:
// All the pending request are drained.
// Don't close the writeCh, because it has be used in several places.
for {
select {
case r = <-db.writeCh:
reqs = append(reqs, r)
default:
pendingCh <- struct{}{} // Push to pending before doing a write.
writeRequests(reqs)
return
}
}
writeCase:
go writeRequests(reqs)
reqs = make([]*request, 0, 10)
reqLen.Set(0)
}
}
// batchSet applies a list of badger.Entry. If a request level error occurs it
// will be returned.
//
// Check(kv.BatchSet(entries))
func (db *DB) batchSet(entries []*Entry) error {
req, err := db.sendToWriteCh(entries)
if err != nil {
return err
}
return req.Wait()
}
// batchSetAsync is the asynchronous version of batchSet. It accepts a callback
// function which is called when all the sets are complete. If a request level
// error occurs, it will be passed back via the callback.
//
// err := kv.BatchSetAsync(entries, func(err error)) {
// Check(err)
// }
func (db *DB) batchSetAsync(entries []*Entry, f func(error)) error {
req, err := db.sendToWriteCh(entries)
if err != nil {
return err
}
go func() {
err := req.Wait()
// Write is complete. Let's call the callback function now.
f(err)
}()
return nil
}
var errNoRoom = errors.New("No room for write")
// ensureRoomForWrite is always called serially.
func (db *DB) ensureRoomForWrite() error {
var err error
db.lock.Lock()
defer db.lock.Unlock()
y.AssertTrue(db.mt != nil) // A nil mt indicates that DB is being closed.
if !db.mt.isFull() {
return nil
}
select {
case db.flushChan <- db.mt:
db.opt.Debugf("Flushing memtable, mt.size=%d size of flushChan: %d\n",
db.mt.sl.MemSize(), len(db.flushChan))
// We manage to push this task. Let's modify imm.
db.imm = append(db.imm, db.mt)
db.mt, err = db.newMemTable()
if err != nil {
return y.Wrapf(err, "cannot create new mem table")
}
// New memtable is empty. We certainly have room.
return nil
default:
// We need to do this to unlock and allow the flusher to modify imm.
return errNoRoom
}
}
func arenaSize(opt Options) int64 {
return opt.MemTableSize + opt.maxBatchSize + opt.maxBatchCount*int64(skl.MaxNodeSize)
}
// buildL0Table builds a new table from the memtable.
func buildL0Table(iter y.Iterator, dropPrefixes [][]byte, bopts table.Options) *table.Builder {
defer iter.Close()
b := table.NewTableBuilder(bopts)
for iter.Rewind(); iter.Valid(); iter.Next() {
if len(dropPrefixes) > 0 && hasAnyPrefixes(iter.Key(), dropPrefixes) {
continue
}
vs := iter.Value()
var vp valuePointer
if vs.Meta&bitValuePointer > 0 {
vp.Decode(vs.Value)
}
b.Add(iter.Key(), iter.Value(), vp.Len)
}
return b
}
// handleMemTableFlush must be run serially.
func (db *DB) handleMemTableFlush(mt *memTable, dropPrefixes [][]byte) error {
bopts := buildTableOptions(db)
itr := mt.sl.NewUniIterator(false)
builder := buildL0Table(itr, nil, bopts)
defer builder.Close()
// buildL0Table can return nil if the none of the items in the skiplist are
// added to the builder. This can happen when drop prefix is set and all
// the items are skipped.
if builder.Empty() {
builder.Finish()
return nil
}
fileID := db.lc.reserveFileID()
var tbl *table.Table
var err error
if db.opt.InMemory {
data := builder.Finish()
tbl, err = table.OpenInMemoryTable(data, fileID, &bopts)
} else {
tbl, err = table.CreateTable(table.NewFilename(fileID, db.opt.Dir), builder)
}
if err != nil {
return y.Wrap(err, "error while creating table")
}
// We own a ref on tbl.
err = db.lc.addLevel0Table(tbl) // This will incrRef
_ = tbl.DecrRef() // Releases our ref.
return err
}
// flushMemtable must keep running until we send it an empty memtable. If there
// are errors during handling the memtable flush, we'll retry indefinitely.
func (db *DB) flushMemtable(lc *z.Closer) {
defer lc.Done()
for mt := range db.flushChan {
if mt == nil {
continue
}
for {
if err := db.handleMemTableFlush(mt, nil); err != nil {
// Encountered error. Retry indefinitely.
db.opt.Errorf("error flushing memtable to disk: %v, retrying", err)
time.Sleep(time.Second)
continue
}
// Update s.imm. Need a lock.
db.lock.Lock()
// This is a single-threaded operation. mt corresponds to the head of
// db.imm list. Once we flush it, we advance db.imm. The next mt
// which would arrive here would match db.imm[0], because we acquire a
// lock over DB when pushing to flushChan.
// TODO: This logic is dirty AF. Any change and this could easily break.
y.AssertTrue(mt == db.imm[0])
db.imm = db.imm[1:]
mt.DecrRef() // Return memory.
// unlock
db.lock.Unlock()
break
}
}
}
func exists(path string) (bool, error) {
_, err := os.Stat(path)
if err == nil {
return true, nil
}
if os.IsNotExist(err) {
return false, nil
}
return true, err
}
// This function does a filewalk, calculates the size of vlog and sst files and stores it in
// y.LSMSize and y.VlogSize.
func (db *DB) calculateSize() {
if db.opt.InMemory {
return
}
newInt := func(val int64) *expvar.Int {
v := new(expvar.Int)
v.Add(val)
return v
}
totalSize := func(dir string) (int64, int64) {
var lsmSize, vlogSize int64
err := filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
ext := filepath.Ext(path)
switch ext {
case ".sst":
lsmSize += info.Size()
case ".vlog":
vlogSize += info.Size()
}
return nil
})
if err != nil {
db.opt.Debugf("Got error while calculating total size of directory: %s", dir)
}
return lsmSize, vlogSize
}
lsmSize, vlogSize := totalSize(db.opt.Dir)
y.LSMSizeSet(db.opt.MetricsEnabled, db.opt.Dir, newInt(lsmSize))
// If valueDir is different from dir, we'd have to do another walk.
if db.opt.ValueDir != db.opt.Dir {
_, vlogSize = totalSize(db.opt.ValueDir)
}
y.VlogSizeSet(db.opt.MetricsEnabled, db.opt.ValueDir, newInt(vlogSize))
}
func (db *DB) updateSize(lc *z.Closer) {
defer lc.Done()
if db.opt.InMemory {
return
}
metricsTicker := time.NewTicker(time.Minute)
defer metricsTicker.Stop()
for {
select {
case <-metricsTicker.C:
db.calculateSize()
case <-lc.HasBeenClosed():
return
}
}
}
// RunValueLogGC triggers a value log garbage collection.
//
// It picks value log files to perform GC based on statistics that are collected
// during compactions. If no such statistics are available, then log files are
// picked in random order. The process stops as soon as the first log file is
// encountered which does not result in garbage collection.
//
// When a log file is picked, it is first sampled. If the sample shows that we
// can discard at least discardRatio space of that file, it would be rewritten.
//
// If a call to RunValueLogGC results in no rewrites, then an ErrNoRewrite is
// thrown indicating that the call resulted in no file rewrites.
//
// We recommend setting discardRatio to 0.5, thus indicating that a file be
// rewritten if half the space can be discarded. This results in a lifetime
// value log write amplification of 2 (1 from original write + 0.5 rewrite +
// 0.25 + 0.125 + ... = 2). Setting it to higher value would result in fewer
// space reclaims, while setting it to a lower value would result in more space
// reclaims at the cost of increased activity on the LSM tree. discardRatio
// must be in the range (0.0, 1.0), both endpoints excluded, otherwise an
// ErrInvalidRequest is returned.
//
// Only one GC is allowed at a time. If another value log GC is running, or DB
// has been closed, this would return an ErrRejected.
//
// Note: Every time GC is run, it would produce a spike of activity on the LSM
// tree.
func (db *DB) RunValueLogGC(discardRatio float64) error {
if db.opt.InMemory {
return ErrGCInMemoryMode
}
if discardRatio >= 1.0 || discardRatio <= 0.0 {
return ErrInvalidRequest
}
// Pick a log file and run GC
return db.vlog.runGC(discardRatio)
}
// Size returns the size of lsm and value log files in bytes. It can be used to decide how often to
// call RunValueLogGC.
func (db *DB) Size() (lsm, vlog int64) {
if y.LSMSizeGet(db.opt.MetricsEnabled, db.opt.Dir) == nil {
lsm, vlog = 0, 0
return
}
lsm = y.LSMSizeGet(db.opt.MetricsEnabled, db.opt.Dir).(*expvar.Int).Value()
vlog = y.VlogSizeGet(db.opt.MetricsEnabled, db.opt.ValueDir).(*expvar.Int).Value()
return
}
// Sequence represents a Badger sequence.
type Sequence struct {
lock sync.Mutex
db *DB
key []byte
next uint64
leased uint64
bandwidth uint64
}
// Next would return the next integer in the sequence, updating the lease by running a transaction
// if needed.
func (seq *Sequence) Next() (uint64, error) {
seq.lock.Lock()
defer seq.lock.Unlock()
if seq.next >= seq.leased {
if err := seq.updateLease(); err != nil {
return 0, err
}
}
val := seq.next
seq.next++
return val, nil
}
// Release the leased sequence to avoid wasted integers. This should be done right
// before closing the associated DB. However it is valid to use the sequence after
// it was released, causing a new lease with full bandwidth.
func (seq *Sequence) Release() error {
seq.lock.Lock()
defer seq.lock.Unlock()
err := seq.db.Update(func(txn *Txn) error {
item, err := txn.Get(seq.key)
if err != nil {
return err
}
var num uint64
if err := item.Value(func(v []byte) error {
num = binary.BigEndian.Uint64(v)
return nil
}); err != nil {
return err
}
if num == seq.leased {
var buf [8]byte
binary.BigEndian.PutUint64(buf[:], seq.next)
return txn.SetEntry(NewEntry(seq.key, buf[:]))
}
return nil
})
if err != nil {
return err
}
seq.leased = seq.next
return nil
}
func (seq *Sequence) updateLease() error {
return seq.db.Update(func(txn *Txn) error {
item, err := txn.Get(seq.key)
switch {
case err == ErrKeyNotFound:
seq.next = 0
case err != nil:
return err
default:
var num uint64
if err := item.Value(func(v []byte) error {
num = binary.BigEndian.Uint64(v)
return nil
}); err != nil {
return err
}
seq.next = num
}
lease := seq.next + seq.bandwidth
var buf [8]byte
binary.BigEndian.PutUint64(buf[:], lease)
if err = txn.SetEntry(NewEntry(seq.key, buf[:])); err != nil {
return err
}
seq.leased = lease
return nil
})
}
// GetSequence would initiate a new sequence object, generating it from the stored lease, if
// available, in the database. Sequence can be used to get a list of monotonically increasing
// integers. Multiple sequences can be created by providing different keys. Bandwidth sets the
// size of the lease, determining how many Next() requests can be served from memory.
//
// GetSequence is not supported on ManagedDB. Calling this would result in a panic.
func (db *DB) GetSequence(key []byte, bandwidth uint64) (*Sequence, error) {
if db.opt.managedTxns {
panic("Cannot use GetSequence with managedDB=true.")
}
switch {
case len(key) == 0:
return nil, ErrEmptyKey
case bandwidth == 0:
return nil, ErrZeroBandwidth
}
seq := &Sequence{
db: db,
key: key,
next: 0,
leased: 0,
bandwidth: bandwidth,
}
err := seq.updateLease()
return seq, err
}
// Tables gets the TableInfo objects from the level controller. If withKeysCount
// is true, TableInfo objects also contain counts of keys for the tables.
func (db *DB) Tables() []TableInfo {
return db.lc.getTableInfo()
}
// Levels gets the LevelInfo.
func (db *DB) Levels() []LevelInfo {
return db.lc.getLevelInfo()
}
// EstimateSize can be used to get rough estimate of data size for a given prefix.
func (db *DB) EstimateSize(prefix []byte) (uint64, uint64) {
var onDiskSize, uncompressedSize uint64
tables := db.Tables()
for _, ti := range tables {
if bytes.HasPrefix(ti.Left, prefix) && bytes.HasPrefix(ti.Right, prefix) {
onDiskSize += uint64(ti.OnDiskSize)
uncompressedSize += uint64(ti.UncompressedSize)
}
}
return onDiskSize, uncompressedSize
}
// Ranges can be used to get rough key ranges to divide up iteration over the DB. The ranges here
// would consider the prefix, but would not necessarily start or end with the prefix. In fact, the
// first range would have nil as left key, and the last range would have nil as the right key.
func (db *DB) Ranges(prefix []byte, numRanges int) []*keyRange {
var splits []string
tables := db.Tables()
// We just want table ranges here and not keys count.
for _, ti := range tables {
// We don't use ti.Left, because that has a tendency to store !badger keys. Skip over tables
// at upper levels. Only choose tables from the last level.
if ti.Level != db.opt.MaxLevels-1 {
continue
}
if bytes.HasPrefix(ti.Right, prefix) {
splits = append(splits, string(ti.Right))
}
}
// If the number of splits is low, look at the offsets inside the
// tables to generate more splits.
if len(splits) < 32 {
numTables := len(tables)
if numTables == 0 {
numTables = 1
}
numPerTable := 32 / numTables
if numPerTable == 0 {
numPerTable = 1
}
splits = db.lc.keySplits(numPerTable, prefix)
}
// If the number of splits is still < 32, then look at the memtables.
if len(splits) < 32 {
maxPerSplit := 10000
mtSplits := func(mt *memTable) {
if mt == nil {
return
}
count := 0
iter := mt.sl.NewIterator()
for iter.SeekToFirst(); iter.Valid(); iter.Next() {
if count%maxPerSplit == 0 {
// Add a split every maxPerSplit keys.
if bytes.HasPrefix(iter.Key(), prefix) {
splits = append(splits, string(iter.Key()))
}
}
count += 1
}
_ = iter.Close()
}
db.lock.Lock()
defer db.lock.Unlock()
var memTables []*memTable
memTables = append(memTables, db.imm...)
for _, mt := range memTables {
mtSplits(mt)
}
mtSplits(db.mt)
}
// We have our splits now. Let's convert them to ranges.
sort.Strings(splits)
var ranges []*keyRange
var start []byte
for _, key := range splits {
ranges = append(ranges, &keyRange{left: start, right: y.SafeCopy(nil, []byte(key))})
start = y.SafeCopy(nil, []byte(key))
}
ranges = append(ranges, &keyRange{left: start})
// Figure out the approximate table size this range has to deal with.
for _, t := range tables {
tr := keyRange{left: t.Left, right: t.Right}
for _, r := range ranges {
if len(r.left) == 0 || len(r.right) == 0 {
continue
}
if r.overlapsWith(tr) {
r.size += int64(t.UncompressedSize)
}
}
}
var total int64
for _, r := range ranges {
total += r.size
}
if total == 0 {
return ranges
}
// Figure out the average size, so we know how to bin the ranges together.
avg := total / int64(numRanges)
var out []*keyRange
var i int
for i < len(ranges) {
r := ranges[i]
cur := &keyRange{left: r.left, size: r.size, right: r.right}
i++
for ; i < len(ranges); i++ {
next := ranges[i]
if cur.size+next.size > avg {
break
}
cur.right = next.right
cur.size += next.size
}
out = append(out, cur)
}
return out
}
// MaxBatchCount returns max possible entries in batch
func (db *DB) MaxBatchCount() int64 {
return db.opt.maxBatchCount
}
// MaxBatchSize returns max possible batch size
func (db *DB) MaxBatchSize() int64 {
return db.opt.maxBatchSize
}
func (db *DB) stopMemoryFlush() {
// Stop memtable flushes.
if db.closers.memtable != nil {
close(db.flushChan)
db.closers.memtable.SignalAndWait()
}
}
func (db *DB) stopCompactions() {
// Stop compactions.
if db.closers.compactors != nil {
db.closers.compactors.SignalAndWait()
}
}
func (db *DB) startCompactions() {
// Resume compactions.
if db.closers.compactors != nil {
db.closers.compactors = z.NewCloser(1)
db.lc.startCompact(db.closers.compactors)
}
}
func (db *DB) startMemoryFlush() {
// Start memory fluhser.
if db.closers.memtable != nil {
db.flushChan = make(chan *memTable, db.opt.NumMemtables)
db.closers.memtable = z.NewCloser(1)
go func() {
db.flushMemtable(db.closers.memtable)
}()
}
}
// Flatten can be used to force compactions on the LSM tree so all the tables fall on the same
// level. This ensures that all the versions of keys are colocated and not split across multiple
// levels, which is necessary after a restore from backup. During Flatten, live compactions are
// stopped. Ideally, no writes are going on during Flatten. Otherwise, it would create competition
// between flattening the tree and new tables being created at level zero.
func (db *DB) Flatten(workers int) error {
db.stopCompactions()
defer db.startCompactions()
compactAway := func(cp compactionPriority) error {
db.opt.Infof("Attempting to compact with %+v\n", cp)
errCh := make(chan error, 1)
for i := 0; i < workers; i++ {
go func() {
errCh <- db.lc.doCompact(175, cp)
}()
}
var success int
var rerr error
for i := 0; i < workers; i++ {
err := <-errCh
if err != nil {
rerr = err
db.opt.Warningf("While running doCompact with %+v. Error: %v\n", cp, err)
} else {
success++
}
}
if success == 0 {
return rerr
}
// We could do at least one successful compaction. So, we'll consider this a success.
db.opt.Infof("%d compactor(s) succeeded. One or more tables from level %d compacted.\n",
success, cp.level)
return nil
}
hbytes := func(sz int64) string {
return humanize.IBytes(uint64(sz))
}
t := db.lc.levelTargets()
for {
db.opt.Infof("\n")
var levels []int
for i, l := range db.lc.levels {
sz := l.getTotalSize()
db.opt.Infof("Level: %d. %8s Size. %8s Max.\n",
i, hbytes(l.getTotalSize()), hbytes(t.targetSz[i]))
if sz > 0 {
levels = append(levels, i)
}
}
if len(levels) <= 1 {
prios := db.lc.pickCompactLevels(nil)
if len(prios) == 0 || prios[0].score <= 1.0 {
db.opt.Infof("All tables consolidated into one level. Flattening done.\n")
return nil
}
if err := compactAway(prios[0]); err != nil {
return err
}
continue
}
// Create an artificial compaction priority, to ensure that we compact the level.
cp := compactionPriority{level: levels[0], score: 1.71}
if err := compactAway(cp); err != nil {
return err
}
}
}
func (db *DB) blockWrite() error {
// Stop accepting new writes.
if !db.blockWrites.CompareAndSwap(0, 1) {
return ErrBlockedWrites
}
// Make all pending writes finish. The following will also close writeCh.
db.closers.writes.SignalAndWait()
db.opt.Infof("Writes flushed. Stopping compactions now...")
return nil
}
func (db *DB) unblockWrite() {
db.closers.writes = z.NewCloser(1)
go db.doWrites(db.closers.writes)
// Resume writes.
db.blockWrites.Store(0)
}
func (db *DB) prepareToDrop() (func(), error) {
if db.opt.ReadOnly {
panic("Attempting to drop data in read-only mode.")
}
// In order prepare for drop, we need to block the incoming writes and
// write it to db. Then, flush all the pending memtable. So that, we
// don't miss any entries.
if err := db.blockWrite(); err != nil {
return func() {}, err
}
reqs := make([]*request, 0, 10)
for {
select {
case r := <-db.writeCh:
reqs = append(reqs, r)
default:
if err := db.writeRequests(reqs); err != nil {
db.opt.Errorf("writeRequests: %v", err)
}
db.stopMemoryFlush()
return func() {
db.opt.Infof("Resuming writes")
db.startMemoryFlush()
db.unblockWrite()
}, nil
}
}
}
// DropAll would drop all the data stored in Badger. It does this in the following way.
// - Stop accepting new writes.
// - Pause memtable flushes and compactions.
// - Pick all tables from all levels, create a changeset to delete all these
// tables and apply it to manifest.
// - Pick all log files from value log, and delete all of them. Restart value log files from zero.
// - Resume memtable flushes and compactions.
//
// NOTE: DropAll is resilient to concurrent writes, but not to reads. It is up to the user to not do
// any reads while DropAll is going on, otherwise they may result in panics. Ideally, both reads and
// writes are paused before running DropAll, and resumed after it is finished.
func (db *DB) DropAll() error {
f, err := db.dropAll()
if f != nil {
f()
}
return err
}
func (db *DB) dropAll() (func(), error) {
db.opt.Infof("DropAll called. Blocking writes...")
f, err := db.prepareToDrop()
if err != nil {
return f, err
}
// prepareToDrop will stop all the incoming write and flushes any pending memtables.
// Before we drop, we'll stop the compaction because anyways all the data are going to
// be deleted.
db.stopCompactions()
resume := func() {
db.startCompactions()
f()
}
// Block all foreign interactions with memory tables.
db.lock.Lock()
defer db.lock.Unlock()
// Remove inmemory tables. Calling DecrRef for safety. Not sure if they're absolutely needed.
db.mt.DecrRef()
for _, mt := range db.imm {
mt.DecrRef()
}
db.imm = db.imm[:0]
db.mt, err = db.newMemTable() // Set it up for future writes.
if err != nil {
return resume, y.Wrapf(err, "cannot open new memtable")
}
num, err := db.lc.dropTree()
if err != nil {
return resume, err
}
db.opt.Infof("Deleted %d SSTables. Now deleting value logs...\n", num)
num, err = db.vlog.dropAll()
if err != nil {
return resume, err
}
db.lc.nextFileID.Store(1)
db.opt.Infof("Deleted %d value log files. DropAll done.\n", num)
db.blockCache.Clear()
db.indexCache.Clear()
db.threshold.Clear(db.opt)
return resume, nil
}
// DropPrefix would drop all the keys with the provided prefix. It does this in the following way:
// - Stop accepting new writes.
// - Stop memtable flushes before acquiring lock. Because we're acquiring lock here
// and memtable flush stalls for lock, which leads to deadlock
// - Flush out all memtables, skipping over keys with the given prefix, Kp.
// - Write out the value log header to memtables when flushing, so we don't accidentally bring Kp
// back after a restart.
// - Stop compaction.
// - Compact L0->L1, skipping over Kp.
// - Compact rest of the levels, Li->Li, picking tables which have Kp.
// - Resume memtable flushes, compactions and writes.
func (db *DB) DropPrefix(prefixes ...[]byte) error {
if len(prefixes) == 0 {
return nil
}
db.opt.Infof("DropPrefix called for %s", prefixes)
f, err := db.prepareToDrop()
if err != nil {
return err
}
defer f()
var filtered [][]byte
if filtered, err = db.filterPrefixesToDrop(prefixes); err != nil {
return err
}
// If there is no prefix for which the data already exist, do not do anything.
if len(filtered) == 0 {
db.opt.Infof("No prefixes to drop")
return nil
}
// Block all foreign interactions with memory tables.
db.lock.Lock()
defer db.lock.Unlock()
db.imm = append(db.imm, db.mt)
for _, memtable := range db.imm {
if memtable.sl.Empty() {
memtable.DecrRef()
continue
}
db.opt.Debugf("Flushing memtable")
if err := db.handleMemTableFlush(memtable, filtered); err != nil {
db.opt.Errorf("While trying to flush memtable: %v", err)
return err
}
memtable.DecrRef()
}
db.stopCompactions()
defer db.startCompactions()
db.imm = db.imm[:0]
db.mt, err = db.newMemTable()
if err != nil {
return y.Wrapf(err, "cannot create new mem table")
}
// Drop prefixes from the levels.
if err := db.lc.dropPrefixes(filtered); err != nil {
return err
}
db.opt.Infof("DropPrefix done")
return nil
}
func (db *DB) filterPrefixesToDrop(prefixes [][]byte) ([][]byte, error) {
var filtered [][]byte
for _, prefix := range prefixes {
err := db.View(func(txn *Txn) error {
iopts := DefaultIteratorOptions
iopts.Prefix = prefix
iopts.PrefetchValues = false
itr := txn.NewIterator(iopts)
defer itr.Close()
itr.Rewind()
if itr.ValidForPrefix(prefix) {
filtered = append(filtered, prefix)
}
return nil
})
if err != nil {
return filtered, err
}
}
return filtered, nil
}
// Checks if the key is banned. Returns the respective error if the key belongs to any of the banned
// namepspaces. Else it returns nil.
func (db *DB) isBanned(key []byte) error {
if db.opt.NamespaceOffset < 0 {
return nil
}
if len(key) <= db.opt.NamespaceOffset+8 {
return nil
}
if db.bannedNamespaces.has(y.BytesToU64(key[db.opt.NamespaceOffset:])) {
return ErrBannedKey
}
return nil
}
// BanNamespace bans a namespace. Read/write to keys belonging to any of such namespace is denied.
func (db *DB) BanNamespace(ns uint64) error {
if db.opt.NamespaceOffset < 0 {
return ErrNamespaceMode
}
db.opt.Infof("Banning namespace: %d", ns)
// First set the banned namespaces in DB and then update the in-memory structure.
key := y.KeyWithTs(append(bannedNsKey, y.U64ToBytes(ns)...), 1)
entry := []*Entry{{
Key: key,
Value: nil,
}}
req, err := db.sendToWriteCh(entry)
if err != nil {
return err
}
if err := req.Wait(); err != nil {
return err
}
db.bannedNamespaces.add(ns)
return nil
}
// BannedNamespaces returns the list of prefixes banned for DB.
func (db *DB) BannedNamespaces() []uint64 {
return db.bannedNamespaces.all()
}
// KVList contains a list of key-value pairs.
type KVList = pb.KVList
// Subscribe can be used to watch key changes for the given key prefixes and the ignore string.
// At least one prefix should be passed, or an error will be returned.
// You can use an empty prefix to monitor all changes to the DB.
// Ignore string is the byte ranges for which prefix matching will be ignored.
// For example: ignore = "2-3", and prefix = "abc" will match for keys "abxxc", "abdfc" etc.
// This function blocks until the given context is done or an error occurs.
// The given function will be called with a new KVList containing the modified keys and the
// corresponding values.
func (db *DB) Subscribe(ctx context.Context, cb func(kv *KVList) error, matches []pb.Match) error {
if cb == nil {
return ErrNilCallback
}
c := z.NewCloser(1)
s, err := db.pub.newSubscriber(c, matches)
if err != nil {
return y.Wrapf(err, "while creating a new subscriber")
}
slurp := func(batch *pb.KVList) error {
for {
select {
case kvs := <-s.sendCh:
batch.Kv = append(batch.Kv, kvs.Kv...)
default:
if len(batch.GetKv()) > 0 {
return cb(batch)
}
return nil
}
}
}
drain := func() {
for {
select {
case _, ok := <-s.sendCh:
if !ok {
// Channel is closed.
return
}
default:
return
}
}
}
for {
select {
case <-c.HasBeenClosed():
// No need to delete here. Closer will be called only while
// closing DB. Subscriber will be deleted by cleanSubscribers.
err := slurp(new(pb.KVList))
// Drain if any pending updates.
c.Done()
return err
case <-ctx.Done():
c.Done()
s.active.Store(0)
drain()
db.pub.deleteSubscriber(s.id)
// Delete the subscriber to avoid further updates.
return ctx.Err()
case batch := <-s.sendCh:
err := slurp(batch)
if err != nil {
c.Done()
s.active.Store(0)
drain()
// Delete the subscriber if there is an error by the callback.
db.pub.deleteSubscriber(s.id)
return err
}
}
}
}
func (db *DB) syncDir(dir string) error {
if db.opt.InMemory {
return nil
}
return syncDir(dir)
}
func createDirs(opt Options) error {
for _, path := range []string{opt.Dir, opt.ValueDir} {
dirExists, err := exists(path)
if err != nil {
return y.Wrapf(err, "Invalid Dir: %q", path)
}
if !dirExists {
if opt.ReadOnly {
return fmt.Errorf("Cannot find directory %q for read-only open", path)
}
// Try to create the directory
err = os.MkdirAll(path, 0700)
if err != nil {
return y.Wrapf(err, "Error Creating Dir: %q", path)
}
}
}
return nil
}
// Stream the contents of this DB to a new DB with options outOptions that will be
// created in outDir.
func (db *DB) StreamDB(outOptions Options) error {
outDir := outOptions.Dir
// Open output DB.
outDB, err := OpenManaged(outOptions)
if err != nil {
return y.Wrapf(err, "cannot open out DB at %s", outDir)
}
defer outDB.Close()
writer := outDB.NewStreamWriter()
if err := writer.Prepare(); err != nil {
return y.Wrapf(err, "cannot create stream writer in out DB at %s", outDir)
}
// Stream contents of DB to the output DB.
stream := db.NewStreamAt(math.MaxUint64)
stream.LogPrefix = fmt.Sprintf("Streaming DB to new DB at %s", outDir)
stream.Send = func(buf *z.Buffer) error {
return writer.Write(buf)
}
if err := stream.Orchestrate(context.Background()); err != nil {
return y.Wrapf(err, "cannot stream DB to out DB at %s", outDir)
}
if err := writer.Flush(); err != nil {
return y.Wrapf(err, "cannot flush writer")
}
return nil
}
// Opts returns a copy of the DB options.
func (db *DB) Opts() Options {
return db.opt
}
type CacheType int
const (
BlockCache CacheType = iota
IndexCache
)
// CacheMaxCost updates the max cost of the given cache (either block or index cache).
// The call will have an effect only if the DB was created with the cache. Otherwise it is
// a no-op. If you pass a negative value, the function will return the current value
// without updating it.
func (db *DB) CacheMaxCost(cache CacheType, maxCost int64) (int64, error) {
if db == nil {
return 0, nil
}
if maxCost < 0 {
switch cache {
case BlockCache:
return db.blockCache.MaxCost(), nil
case IndexCache:
return db.indexCache.MaxCost(), nil
default:
return 0, errors.New("invalid cache type")
}
}
switch cache {
case BlockCache:
db.blockCache.UpdateMaxCost(maxCost)
return maxCost, nil
case IndexCache:
db.indexCache.UpdateMaxCost(maxCost)
return maxCost, nil
default:
return 0, errors.New("invalid cache type")
}
}
func (db *DB) LevelsToString() string {
levels := db.Levels()
h := func(sz int64) string {
return humanize.IBytes(uint64(sz))
}
base := func(b bool) string {
if b {
return "B"
}
return " "
}
var b strings.Builder
b.WriteRune('\n')
for _, li := range levels {
b.WriteString(fmt.Sprintf(
"Level %d [%s]: NumTables: %02d. Size: %s of %s. Score: %.2f->%.2f"+
" StaleData: %s Target FileSize: %s\n",
li.Level, base(li.IsBaseLevel), li.NumTables,
h(li.Size), h(li.TargetSize), li.Score, li.Adjusted, h(li.StaleDatSize),
h(li.TargetFileSize)))
}
b.WriteString("Level Done\n")
return b.String()
}
================================================
FILE: db2_test.go
================================================
/*
* SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.
* SPDX-License-Identifier: Apache-2.0
*/
package badger
import (
"bytes"
"context"
"encoding/binary"
"flag"
"fmt"
"log"
"math"
"math/rand"
"os"
"path/filepath"
"regexp"
"runtime"
"sync"
"testing"
"time"
"github.com/stretchr/testify/require"
"github.com/dgraph-io/badger/v4/options"
"github.com/dgraph-io/badger/v4/pb"
"github.com/dgraph-io/badger/v4/table"
"github.com/dgraph-io/badger/v4/y"
"github.com/dgraph-io/ristretto/v2/z"
)
func TestTruncateVlogWithClose(t *testing.T) {
key := func(i int) []byte {
return []byte(fmt.Sprintf("%d%10d", i, i))
}
data := func(l int) []byte {
m := make([]byte, l)
_, err := rand.Read(m)
require.NoError(t, err)
return m
}
dir, err := os.MkdirTemp("", "badger-test")
require.NoError(t, err)
defer removeDir(dir)
opt := getTestOptions(dir)
opt.SyncWrites = true
opt.ValueThreshold = 1 // Force all reads from value log.
db, err := Open(opt)
require.NoError(t, err)
err = db.Update(func(txn *Txn) error {
return txn.SetEntry(NewEntry(key(0), data(4055)))
})
require.NoError(t, err)
// Close the DB.
require.NoError(t, db.Close())
// We start value logs at 1.
require.NoError(t, os.Truncate(filepath.Join(dir, "000001.vlog"), 4090))
// Reopen and write some new data.
db, err = Open(opt)
require.NoError(t, err)
for i := 0; i < 32; i++ {
err := db.Update(func(txn *Txn) error {
return txn.SetEntry(NewEntry(key(i), data(10)))
})
require.NoError(t, err)
}
// Read it back to ensure that we can read it now.
for i := 0; i < 32; i++ {
err := db.View(func(txn *Txn) error {
item, err := txn.Get(key(i))
require.NoError(t, err)
val := getItemValue(t, item)
require.Equal(t, 10, len(val))
return nil
})
require.NoError(t, err)
}
require.NoError(t, db.Close())
// Reopen and read the data again.
db, err = Open(opt)
require.NoError(t, err)
for i := 0; i < 32; i++ {
err := db.View(func(txn *Txn) error {
item, err := txn.Get(key(i))
require.NoError(t, err, "key: %s", key(i))
val := getItemValue(t, item)
require.Equal(t, 10, len(val))
return nil
})
require.NoError(t, err)
}
require.NoError(t, db.Close())
}
var manual = flag.Bool("manual", false, "Set when manually running some tests.")
// Badger dir to be used for performing db.Open benchmark.
var benchDir = flag.String("benchdir", "", "Set when running db.Open benchmark")
// The following 3 TruncateVlogNoClose tests should be run one after another.
// None of these close the DB, simulating a crash. They should be run with a
// script, which truncates the value log to 4090, lining up with the end of the
// first entry in the txn. At <4090, it would cause the entry to be truncated
// immediately, at >4090, same thing.
func TestTruncateVlogNoClose(t *testing.T) {
if !*manua
gitextract_mc7da0i7/
├── .github/
│ ├── CODEOWNERS
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ ├── config.yml
│ │ └── feature_request.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ ├── renovate.json
│ └── workflows/
│ ├── cd-badger.yml
│ ├── ci-badger-bank-tests-nightly.yml
│ ├── ci-badger-bank-tests.yml
│ ├── ci-badger-tests.yml
│ ├── ci-dgraph-tests.yml
│ └── trunk.yml
├── .gitignore
├── .trunk/
│ ├── .gitignore
│ ├── configs/
│ │ ├── .checkov.yaml
│ │ ├── .markdownlint.json
│ │ ├── .prettierrc
│ │ ├── .shellcheckrc
│ │ ├── .yamllint.yaml
│ │ └── svgo.config.mjs
│ └── trunk.yaml
├── .vscode/
│ ├── extensions.json
│ └── settings.json
├── CHANGELOG.md
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── Makefile
├── README.md
├── SECURITY.md
├── VERSIONING.md
├── backup.go
├── backup_test.go
├── batch.go
├── batch_test.go
├── changes.sh
├── compaction.go
├── contrib/
│ └── RELEASE.md
├── db.go
├── db2_test.go
├── db_test.go
├── dir_aix.go
├── dir_other.go
├── dir_plan9.go
├── dir_unix.go
├── dir_windows.go
├── discard.go
├── discard_test.go
├── doc.go
├── docs/
│ ├── design.md
│ ├── encryption-at-rest.md
│ ├── index.md
│ ├── quickstart.md
│ └── troubleshooting.md
├── errors.go
├── fb/
│ ├── BlockOffset.go
│ ├── TableIndex.go
│ ├── flatbuffer.fbs
│ ├── gen.sh
│ └── install_flatbuffers.sh
├── go.mod
├── go.sum
├── histogram.go
├── histogram_test.go
├── integration/
│ └── testgc/
│ ├── .gitignore
│ └── main.go
├── iterator.go
├── iterator_test.go
├── key_registry.go
├── key_registry_test.go
├── level_handler.go
├── levels.go
├── levels_test.go
├── logger.go
├── logger_test.go
├── managed_db.go
├── managed_db_test.go
├── manifest.go
├── manifest_test.go
├── memtable.go
├── merge.go
├── merge_test.go
├── metrics_test.go
├── options/
│ └── options.go
├── options.go
├── options_test.go
├── pb/
│ ├── badgerpb4.pb.go
│ ├── badgerpb4.proto
│ ├── gen.sh
│ └── protos_test.go
├── publisher.go
├── publisher_test.go
├── skl/
│ ├── README.md
│ ├── arena.go
│ ├── skl.go
│ └── skl_test.go
├── stream.go
├── stream_test.go
├── stream_writer.go
├── stream_writer_test.go
├── structs.go
├── structs_test.go
├── table/
│ ├── README.md
│ ├── builder.go
│ ├── builder_test.go
│ ├── iterator.go
│ ├── merge_iterator.go
│ ├── merge_iterator_test.go
│ ├── table.go
│ └── table_test.go
├── test.sh
├── test_extensions.go
├── trie/
│ ├── trie.go
│ └── trie_test.go
├── txn.go
├── txn_test.go
├── util.go
├── value.go
├── value_test.go
├── watermark_edge_test.go
└── y/
├── bloom.go
├── bloom_test.go
├── checksum.go
├── checksum_test.go
├── encrypt.go
├── encrypt_test.go
├── error.go
├── error_test.go
├── file_dsync.go
├── file_nodsync.go
├── iterator.go
├── metrics.go
├── watermark.go
├── y.go
├── y_test.go
└── zstd.go
SYMBOL INDEX (1493 symbols across 86 files)
FILE: backup.go
constant flushThreshold (line 27) | flushThreshold = 100 << 20
method Backup (line 39) | func (db *DB) Backup(w io.Writer, since uint64) (uint64, error) {
method Backup (line 54) | func (stream *Stream) Backup(w io.Writer, since uint64) (uint64, error) {
function writeTo (line 139) | func writeTo(list *pb.KVList, w io.Writer) error {
type KVLoader (line 152) | type KVLoader struct
method Set (line 170) | func (l *KVLoader) Set(kv *pb.KV) error {
method send (line 200) | func (l *KVLoader) send() error {
method Finish (line 217) | func (l *KVLoader) Finish() error {
method NewKVLoader (line 161) | func (db *DB) NewKVLoader(maxPendingWrites int) *KVLoader {
method Load (line 233) | func (db *DB) Load(r io.Reader, maxPendingWrites int) error {
FILE: backup_test.go
function TestBackupRestore1 (line 24) | func TestBackupRestore1(t *testing.T) {
function TestBackupRestore2 (line 108) | func TestBackupRestore2(t *testing.T) {
function createEntries (line 237) | func createEntries(n int) []*pb.KV {
function populateEntries (line 250) | func populateEntries(db *DB, entries []*pb.KV) error {
function TestBackup (line 263) | func TestBackup(t *testing.T) {
function TestBackupRestore3 (line 315) | func TestBackupRestore3(t *testing.T) {
function TestBackupLoadIncremental (line 377) | func TestBackupLoadIncremental(t *testing.T) {
function TestBackupBitClear (line 497) | func TestBackupBitClear(t *testing.T) {
FILE: batch.go
type WriteBatch (line 22) | type WriteBatch struct
method SetMaxPendingTxns (line 58) | func (wb *WriteBatch) SetMaxPendingTxns(max int) {
method Cancel (line 69) | func (wb *WriteBatch) Cancel() {
method callback (line 79) | func (wb *WriteBatch) callback(err error) {
method writeKV (line 91) | func (wb *WriteBatch) writeKV(kv *pb.KV) error {
method Write (line 101) | func (wb *WriteBatch) Write(buf *z.Buffer) error {
method WriteList (line 115) | func (wb *WriteBatch) WriteList(kvList *pb.KVList) error {
method SetEntryAt (line 128) | func (wb *WriteBatch) SetEntryAt(e *Entry, ts uint64) error {
method handleEntry (line 137) | func (wb *WriteBatch) handleEntry(e *Entry) error {
method SetEntry (line 155) | func (wb *WriteBatch) SetEntry(e *Entry) error {
method Set (line 162) | func (wb *WriteBatch) Set(k, v []byte) error {
method DeleteAt (line 168) | func (wb *WriteBatch) DeleteAt(k []byte, ts uint64) error {
method Delete (line 174) | func (wb *WriteBatch) Delete(k []byte) error {
method commit (line 192) | func (wb *WriteBatch) commit() error {
method Flush (line 211) | func (wb *WriteBatch) Flush() error {
method Error (line 233) | func (wb *WriteBatch) Error() error {
method NewWriteBatch (line 39) | func (db *DB) NewWriteBatch() *WriteBatch {
method newWriteBatch (line 46) | func (db *DB) newWriteBatch(isManaged bool) *WriteBatch {
FILE: batch_test.go
function TestWriteBatch (line 19) | func TestWriteBatch(t *testing.T) {
function TestEmptyWriteBatch (line 87) | func TestEmptyWriteBatch(t *testing.T) {
function TestFlushPanic (line 129) | func TestFlushPanic(t *testing.T) {
function TestBatchErrDeadlock (line 146) | func TestBatchErrDeadlock(t *testing.T) {
FILE: compaction.go
type keyRange (line 19) | type keyRange struct
method isEmpty (line 26) | func (r keyRange) isEmpty() bool {
method String (line 32) | func (r keyRange) String() string {
method equals (line 36) | func (r keyRange) equals(dst keyRange) bool {
method extend (line 42) | func (r *keyRange) extend(kr keyRange) {
method overlapsWith (line 61) | func (r keyRange) overlapsWith(dst keyRange) bool {
function getKeyRange (line 92) | func getKeyRange(tables ...*table.Table) keyRange {
type levelCompactStatus (line 115) | type levelCompactStatus struct
method debug (line 120) | func (lcs *levelCompactStatus) debug() string {
method overlapsWith (line 128) | func (lcs *levelCompactStatus) overlapsWith(dst keyRange) bool {
method remove (line 137) | func (lcs *levelCompactStatus) remove(dst keyRange) bool {
type compactStatus (line 151) | type compactStatus struct
method overlapsWith (line 157) | func (cs *compactStatus) overlapsWith(level int, this keyRange) bool {
method delSize (line 165) | func (cs *compactStatus) delSize(l int) int64 {
method compareAndAdd (line 175) | func (cs *compactStatus) compareAndAdd(_ thisAndNextLevelRLocked, cd c...
method delete (line 204) | func (cs *compactStatus) delete(cd compactDef) {
type thisAndNextLevelRLocked (line 171) | type thisAndNextLevelRLocked struct
FILE: db.go
type closers (line 42) | type closers struct
type lockedKeys (line 52) | type lockedKeys struct
method add (line 57) | func (lk *lockedKeys) add(key uint64) {
method has (line 63) | func (lk *lockedKeys) has(key uint64) bool {
method all (line 70) | func (lk *lockedKeys) all() []uint64 {
type DB (line 82) | type DB struct
method initBannedNamespaces (line 396) | func (db *DB) initBannedNamespaces() error {
method MaxVersion (line 415) | func (db *DB) MaxVersion() uint64 {
method monitorCache (line 437) | func (db *DB) monitorCache(c *z.Closer) {
method cleanup (line 475) | func (db *DB) cleanup() {
method BlockCacheMetrics (line 501) | func (db *DB) BlockCacheMetrics() *ristretto.Metrics {
method IndexCacheMetrics (line 509) | func (db *DB) IndexCacheMetrics() *ristretto.Metrics {
method Close (line 518) | func (db *DB) Close() error {
method IsClosed (line 528) | func (db *DB) IsClosed() bool {
method close (line 532) | func (db *DB) close() (err error) {
method VerifyChecksum (line 663) | func (db *DB) VerifyChecksum() error {
method Sync (line 673) | func (db *DB) Sync() error {
method getMemTables (line 709) | func (db *DB) getMemTables() ([]*memTable, func()) {
method get (line 749) | func (db *DB) get(key []byte) (y.ValueStruct, error) {
method writeToLSM (line 784) | func (db *DB) writeToLSM(b *request) error {
method writeRequests (line 828) | func (db *DB) writeRequests(reqs []*request) error {
method sendToWriteCh (line 883) | func (db *DB) sendToWriteCh(entries []*Entry) (*request, error) {
method doWrites (line 910) | func (db *DB) doWrites(lc *z.Closer) {
method batchSet (line 978) | func (db *DB) batchSet(entries []*Entry) error {
method batchSetAsync (line 994) | func (db *DB) batchSetAsync(entries []*Entry, f func(error)) error {
method ensureRoomForWrite (line 1010) | func (db *DB) ensureRoomForWrite() error {
method handleMemTableFlush (line 1063) | func (db *DB) handleMemTableFlush(mt *memTable, dropPrefixes [][]byte)...
method flushMemtable (line 1097) | func (db *DB) flushMemtable(lc *z.Closer) {
method calculateSize (line 1143) | func (db *DB) calculateSize() {
method updateSize (line 1183) | func (db *DB) updateSize(lc *z.Closer) {
method RunValueLogGC (line 1229) | func (db *DB) RunValueLogGC(discardRatio float64) error {
method Size (line 1243) | func (db *DB) Size() (lsm, vlog int64) {
method GetSequence (line 1349) | func (db *DB) GetSequence(key []byte, bandwidth uint64) (*Sequence, er...
method Tables (line 1373) | func (db *DB) Tables() []TableInfo {
method Levels (line 1378) | func (db *DB) Levels() []LevelInfo {
method EstimateSize (line 1383) | func (db *DB) EstimateSize(prefix []byte) (uint64, uint64) {
method Ranges (line 1398) | func (db *DB) Ranges(prefix []byte, numRanges int) []*keyRange {
method MaxBatchCount (line 1512) | func (db *DB) MaxBatchCount() int64 {
method MaxBatchSize (line 1517) | func (db *DB) MaxBatchSize() int64 {
method stopMemoryFlush (line 1521) | func (db *DB) stopMemoryFlush() {
method stopCompactions (line 1529) | func (db *DB) stopCompactions() {
method startCompactions (line 1536) | func (db *DB) startCompactions() {
method startMemoryFlush (line 1544) | func (db *DB) startMemoryFlush() {
method Flatten (line 1560) | func (db *DB) Flatten(workers int) error {
method blockWrite (line 1628) | func (db *DB) blockWrite() error {
method unblockWrite (line 1640) | func (db *DB) unblockWrite() {
method prepareToDrop (line 1648) | func (db *DB) prepareToDrop() (func(), error) {
method DropAll (line 1688) | func (db *DB) DropAll() error {
method dropAll (line 1696) | func (db *DB) dropAll() (func(), error) {
method DropPrefix (line 1754) | func (db *DB) DropPrefix(prefixes ...[]byte) error {
method filterPrefixesToDrop (line 1807) | func (db *DB) filterPrefixesToDrop(prefixes [][]byte) ([][]byte, error) {
method isBanned (line 1831) | func (db *DB) isBanned(key []byte) error {
method BanNamespace (line 1845) | func (db *DB) BanNamespace(ns uint64) error {
method BannedNamespaces (line 1868) | func (db *DB) BannedNamespaces() []uint64 {
method Subscribe (line 1883) | func (db *DB) Subscribe(ctx context.Context, cb func(kv *KVList) error...
method syncDir (line 1950) | func (db *DB) syncDir(dir string) error {
method StreamDB (line 1979) | func (db *DB) StreamDB(outOptions Options) error {
method Opts (line 2010) | func (db *DB) Opts() Options {
method CacheMaxCost (line 2025) | func (db *DB) CacheMaxCost(cache CacheType, maxCost int64) (int64, err...
method LevelsToString (line 2053) | func (db *DB) LevelsToString() string {
constant kvWriteChCapacity (line 122) | kvWriteChCapacity = 1000
function checkAndSetOptions (line 125) | func checkAndSetOptions(opt *Options) error {
function Open (line 177) | func Open(opt Options) (*DB, error) {
constant lockFile (line 668) | lockFile = "LOCK"
function arenaSize (line 1038) | func arenaSize(opt Options) int64 {
function buildL0Table (line 1043) | func buildL0Table(iter y.Iterator, dropPrefixes [][]byte, bopts table.Op...
function exists (line 1130) | func exists(path string) (bool, error) {
type Sequence (line 1254) | type Sequence struct
method Next (line 1265) | func (seq *Sequence) Next() (uint64, error) {
method Release (line 1281) | func (seq *Sequence) Release() error {
method updateLease (line 1313) | func (seq *Sequence) updateLease() error {
function createDirs (line 1957) | func createDirs(opt Options) error {
type CacheType (line 2014) | type CacheType
constant BlockCache (line 2017) | BlockCache CacheType = iota
constant IndexCache (line 2018) | IndexCache
FILE: db2_test.go
function TestTruncateVlogWithClose (line 34) | func TestTruncateVlogWithClose(t *testing.T) {
function TestTruncateVlogNoClose (line 115) | func TestTruncateVlogNoClose(t *testing.T) {
function TestTruncateVlogNoClose2 (line 135) | func TestTruncateVlogNoClose2(t *testing.T) {
function TestTruncateVlogNoClose3 (line 167) | func TestTruncateVlogNoClose3(t *testing.T) {
function TestBigKeyValuePairs (line 194) | func TestBigKeyValuePairs(t *testing.T) {
function TestPushValueLogLimit (line 285) | func TestPushValueLogLimit(t *testing.T) {
function BenchmarkDBOpen (line 335) | func BenchmarkDBOpen(b *testing.B) {
function TestBigValues (line 351) | func TestBigValues(t *testing.T) {
function TestCompactionFilePicking (line 415) | func TestCompactionFilePicking(t *testing.T) {
function addToManifest (line 482) | func addToManifest(t *testing.T, db *DB, tab *table.Table, level uint32,...
function createTableWithRange (line 495) | func createTableWithRange(t *testing.T, db *DB, start, end int) *table.T...
function TestReadSameVlog (line 514) | func TestReadSameVlog(t *testing.T) {
function TestL0GCBug (line 566) | func TestL0GCBug(t *testing.T) {
function TestWindowsDataLoss (line 659) | func TestWindowsDataLoss(t *testing.T) {
function TestDropPrefixWithNoData (line 735) | func TestDropPrefixWithNoData(t *testing.T) {
function TestDropAllDropPrefix (line 759) | func TestDropAllDropPrefix(t *testing.T) {
function TestIsClosed (line 811) | func TestIsClosed(t *testing.T) {
function TestMaxVersion (line 842) | func TestMaxVersion(t *testing.T) {
function TestTxnReadTs (line 909) | func TestTxnReadTs(t *testing.T) {
function TestKeyCount (line 930) | func TestKeyCount(t *testing.T) {
function TestAssertValueLogIsNotWrittenToOnStartup (line 1052) | func TestAssertValueLogIsNotWrittenToOnStartup(t *testing.T) {
FILE: db_test.go
function waitForMessage (line 35) | func waitForMessage(ch chan string, expected string, count int, timeout ...
type summary (line 69) | type summary struct
method getSummary (line 73) | func (s *levelsController) getSummary() *summary {
method getSummary (line 83) | func (s *levelHandler) getSummary(sum *summary) {
method validate (line 91) | func (s *DB) validate() error { return s.lc.validate() }
function getTestOptions (line 93) | func getTestOptions(dir string) Options {
function getItemValue (line 100) | func getItemValue(t *testing.T, item *Item) (val []byte) {
function txnSet (line 119) | func txnSet(t *testing.T, kv *DB, key []byte, val []byte, meta byte) {
function txnDelete (line 125) | func txnDelete(t *testing.T, kv *DB, key []byte) {
function runBadgerTest (line 132) | func runBadgerTest(t *testing.T, opts *Options, test func(t *testing.T, ...
function TestReverseIterator (line 156) | func TestReverseIterator(t *testing.T) {
function TestWrite (line 194) | func TestWrite(t *testing.T) {
function TestUpdateAndView (line 202) | func TestUpdateAndView(t *testing.T) {
function TestConcurrentWrite (line 238) | func TestConcurrentWrite(t *testing.T) {
function TestGet (line 289) | func TestGet(t *testing.T) {
function TestGetAfterDelete (line 353) | func TestGetAfterDelete(t *testing.T) {
function TestTxnTooBig (line 369) | func TestTxnTooBig(t *testing.T) {
function TestForceCompactL0 (line 400) | func TestForceCompactL0(t *testing.T) {
function TestStreamDB (line 436) | func TestStreamDB(t *testing.T) {
function dirSize (line 485) | func dirSize(path string) (int64, error) {
function BenchmarkDbGrowth (line 512) | func BenchmarkDbGrowth(b *testing.B) {
function TestGetMore (line 589) | func TestGetMore(t *testing.T) {
function TestExistsMore (line 694) | func TestExistsMore(t *testing.T) {
function TestIterate2Basic (line 770) | func TestIterate2Basic(t *testing.T) {
function TestLoad (line 845) | func TestLoad(t *testing.T) {
function TestIterateDeleted (line 935) | func TestIterateDeleted(t *testing.T) {
function TestIterateParallel (line 985) | func TestIterateParallel(t *testing.T) {
function TestDeleteWithoutSyncWrite (line 1079) | func TestDeleteWithoutSyncWrite(t *testing.T) {
function TestPidFile (line 1110) | func TestPidFile(t *testing.T) {
function TestInvalidKey (line 1119) | func TestInvalidKey(t *testing.T) {
function TestIteratorPrefetchSize (line 1148) | func TestIteratorPrefetchSize(t *testing.T) {
function TestSetIfAbsentAsync (line 1192) | func TestSetIfAbsentAsync(t *testing.T) {
function TestGetSetRace (line 1235) | func TestGetSetRace(t *testing.T) {
function TestDiscardVersionsBelow (line 1283) | func TestDiscardVersionsBelow(t *testing.T) {
function TestExpiry (line 1340) | func TestExpiry(t *testing.T) {
function TestExpiryImproperDBClose (line 1385) | func TestExpiryImproperDBClose(t *testing.T) {
function randBytes (line 1449) | func randBytes(n int) []byte {
function TestLargeKeys (line 1471) | func TestLargeKeys(t *testing.T) {
function TestCreateDirs (line 1512) | func TestCreateDirs(t *testing.T) {
function TestGetSetDeadlock (line 1524) | func TestGetSetDeadlock(t *testing.T) {
function TestWriteDeadlock (line 1567) | func TestWriteDeadlock(t *testing.T) {
function TestSequence (line 1620) | func TestSequence(t *testing.T) {
function TestSequence_Release (line 1671) | func TestSequence_Release(t *testing.T) {
function TestTestSequence2 (line 1717) | func TestTestSequence2(t *testing.T) {
function TestReadOnly (line 1744) | func TestReadOnly(t *testing.T) {
function TestLSMOnly (line 1816) | func TestLSMOnly(t *testing.T) {
function TestMinReadTs (line 1852) | func TestMinReadTs(t *testing.T) {
function TestGoroutineLeak (line 1894) | func TestGoroutineLeak(t *testing.T) {
function ExampleOpen (line 1944) | func ExampleOpen() {
function ExampleTxn_NewIterator (line 1999) | func ExampleTxn_NewIterator() {
function TestSyncForRace (line 2055) | func TestSyncForRace(t *testing.T) {
function TestSyncForNoErrors (line 2103) | func TestSyncForNoErrors(t *testing.T) {
function TestSyncForReadingTheEntriesThatWereSynced (line 2129) | func TestSyncForReadingTheEntriesThatWereSynced(t *testing.T) {
function TestForceFlushMemtable (line 2164) | func TestForceFlushMemtable(t *testing.T) {
function TestVerifyChecksum (line 2202) | func TestVerifyChecksum(t *testing.T) {
function TestMain (line 2254) | func TestMain(m *testing.M) {
function removeDir (line 2259) | func removeDir(dir string) {
function TestWriteInemory (line 2265) | func TestWriteInemory(t *testing.T) {
function TestMinCacheSize (line 2292) | func TestMinCacheSize(t *testing.T) {
function TestUpdateMaxCost (line 2305) | func TestUpdateMaxCost(t *testing.T) {
function TestOpenDBReadOnly (line 2335) | func TestOpenDBReadOnly(t *testing.T) {
function TestBannedPrefixes (line 2412) | func TestBannedPrefixes(t *testing.T) {
function TestIterateWithBanned (line 2494) | func TestIterateWithBanned(t *testing.T) {
function TestBannedAtZeroOffset (line 2597) | func TestBannedAtZeroOffset(t *testing.T) {
function TestCompactL0OnClose (line 2637) | func TestCompactL0OnClose(t *testing.T) {
function TestCloseDBWhileReading (line 2666) | func TestCloseDBWhileReading(t *testing.T) {
FILE: dir_aix.go
type directoryLockGuard (line 26) | type directoryLockGuard struct
method release (line 105) | func (guard *directoryLockGuard) release() error {
type aixFlock (line 35) | type aixFlock struct
function acquireDirectoryLock (line 48) | func acquireDirectoryLock(dirPath string, pidFileName string, readOnly b...
function openDir (line 134) | func openDir(path string) (*os.File, error) { return os.Open(path) }
function syncDir (line 139) | func syncDir(dir string) error {
FILE: dir_other.go
type directoryLockGuard (line 21) | type directoryLockGuard struct
method release (line 64) | func (guard *directoryLockGuard) release() error {
function acquireDirectoryLock (line 33) | func acquireDirectoryLock(dirPath string, pidFileName string, readOnly b...
function openDir (line 81) | func openDir(path string) (*os.File, error) { return os.Open(path) }
function syncDir (line 86) | func syncDir(dir string) error {
FILE: dir_plan9.go
type directoryLockGuard (line 19) | type directoryLockGuard struct
method release (line 79) | func (guard *directoryLockGuard) release() error {
function acquireDirectoryLock (line 29) | func acquireDirectoryLock(dirPath string, pidFileName string, readOnly b...
function openDir (line 93) | func openDir(path string) (*os.File, error) { return os.Open(path) }
function syncDir (line 98) | func syncDir(dir string) error {
function isLocked (line 130) | func isLocked(err error) bool {
FILE: dir_unix.go
type directoryLockGuard (line 23) | type directoryLockGuard struct
method release (line 74) | func (guard *directoryLockGuard) release() error {
function acquireDirectoryLock (line 35) | func acquireDirectoryLock(dirPath string, pidFileName string, readOnly b...
function openDir (line 91) | func openDir(path string) (*os.File, error) { return os.Open(path) }
function syncDir (line 96) | func syncDir(dir string) error {
FILE: dir_windows.go
constant FILE_ATTRIBUTE_TEMPORARY (line 26) | FILE_ATTRIBUTE_TEMPORARY = 0x00000100
constant FILE_FLAG_DELETE_ON_CLOSE (line 27) | FILE_FLAG_DELETE_ON_CLOSE = 0x04000000
function openDir (line 30) | func openDir(path string) (*os.File, error) {
function openDirWin (line 38) | func openDirWin(path string) (fd syscall.Handle, err error) {
type directoryLockGuard (line 54) | type directoryLockGuard struct
method release (line 93) | func (g *directoryLockGuard) release() error {
function acquireDirectoryLock (line 60) | func acquireDirectoryLock(dirPath string, pidFileName string, readOnly b...
function syncDir (line 100) | func syncDir(dir string) error { return nil }
FILE: discard.go
type discardStats (line 21) | type discardStats struct
method Len (line 59) | func (lf *discardStats) Len() int {
method Less (line 62) | func (lf *discardStats) Less(i, j int) bool {
method Swap (line 65) | func (lf *discardStats) Swap(i, j int) {
method get (line 75) | func (lf *discardStats) get(offset int) uint64 {
method set (line 78) | func (lf *discardStats) set(offset int, val uint64) {
method zeroOut (line 83) | func (lf *discardStats) zeroOut() {
method maxSlot (line 88) | func (lf *discardStats) maxSlot() int {
method Update (line 95) | func (lf *discardStats) Update(fidu uint32, discard int64) int64 {
method Iterate (line 137) | func (lf *discardStats) Iterate(f func(fid, stats uint64)) {
method MaxDiscard (line 145) | func (lf *discardStats) MaxDiscard() (uint32, int64) {
constant discardFname (line 29) | discardFname string = "DISCARD"
function InitDiscardStats (line 31) | func InitDiscardStats(opt Options) (*discardStats, error) {
FILE: discard_test.go
function TestDiscardStats (line 15) | func TestDiscardStats(t *testing.T) {
function TestReloadDiscardStats (line 45) | func TestReloadDiscardStats(t *testing.T) {
FILE: errors.go
constant ValueThresholdLimit (line 15) | ValueThresholdLimit = math.MaxUint16 - 16 + 1
FILE: fb/BlockOffset.go
type BlockOffset (line 9) | type BlockOffset struct
method Init (line 20) | func (rcv *BlockOffset) Init(buf []byte, i flatbuffers.UOffsetT) {
method Table (line 25) | func (rcv *BlockOffset) Table() flatbuffers.Table {
method Key (line 29) | func (rcv *BlockOffset) Key(j int) byte {
method KeyLength (line 38) | func (rcv *BlockOffset) KeyLength() int {
method KeyBytes (line 46) | func (rcv *BlockOffset) KeyBytes() []byte {
method MutateKey (line 54) | func (rcv *BlockOffset) MutateKey(j int, n byte) bool {
method Offset (line 63) | func (rcv *BlockOffset) Offset() uint32 {
method MutateOffset (line 71) | func (rcv *BlockOffset) MutateOffset(n uint32) bool {
method Len (line 75) | func (rcv *BlockOffset) Len() uint32 {
method MutateLen (line 83) | func (rcv *BlockOffset) MutateLen(n uint32) bool {
function GetRootAsBlockOffset (line 13) | func GetRootAsBlockOffset(buf []byte, offset flatbuffers.UOffsetT) *Bloc...
function BlockOffsetStart (line 87) | func BlockOffsetStart(builder *flatbuffers.Builder) {
function BlockOffsetAddKey (line 90) | func BlockOffsetAddKey(builder *flatbuffers.Builder, key flatbuffers.UOf...
function BlockOffsetStartKeyVector (line 93) | func BlockOffsetStartKeyVector(builder *flatbuffers.Builder, numElems in...
function BlockOffsetAddOffset (line 96) | func BlockOffsetAddOffset(builder *flatbuffers.Builder, offset uint32) {
function BlockOffsetAddLen (line 99) | func BlockOffsetAddLen(builder *flatbuffers.Builder, len uint32) {
function BlockOffsetEnd (line 102) | func BlockOffsetEnd(builder *flatbuffers.Builder) flatbuffers.UOffsetT {
FILE: fb/TableIndex.go
type TableIndex (line 9) | type TableIndex struct
method Init (line 20) | func (rcv *TableIndex) Init(buf []byte, i flatbuffers.UOffsetT) {
method Table (line 25) | func (rcv *TableIndex) Table() flatbuffers.Table {
method Offsets (line 29) | func (rcv *TableIndex) Offsets(obj *BlockOffset, j int) bool {
method OffsetsLength (line 41) | func (rcv *TableIndex) OffsetsLength() int {
method BloomFilter (line 49) | func (rcv *TableIndex) BloomFilter(j int) byte {
method BloomFilterLength (line 58) | func (rcv *TableIndex) BloomFilterLength() int {
method BloomFilterBytes (line 66) | func (rcv *TableIndex) BloomFilterBytes() []byte {
method MutateBloomFilter (line 74) | func (rcv *TableIndex) MutateBloomFilter(j int, n byte) bool {
method MaxVersion (line 83) | func (rcv *TableIndex) MaxVersion() uint64 {
method MutateMaxVersion (line 91) | func (rcv *TableIndex) MutateMaxVersion(n uint64) bool {
method KeyCount (line 95) | func (rcv *TableIndex) KeyCount() uint32 {
method MutateKeyCount (line 103) | func (rcv *TableIndex) MutateKeyCount(n uint32) bool {
method UncompressedSize (line 107) | func (rcv *TableIndex) UncompressedSize() uint32 {
method MutateUncompressedSize (line 115) | func (rcv *TableIndex) MutateUncompressedSize(n uint32) bool {
method OnDiskSize (line 119) | func (rcv *TableIndex) OnDiskSize() uint32 {
method MutateOnDiskSize (line 127) | func (rcv *TableIndex) MutateOnDiskSize(n uint32) bool {
method StaleDataSize (line 131) | func (rcv *TableIndex) StaleDataSize() uint32 {
method MutateStaleDataSize (line 139) | func (rcv *TableIndex) MutateStaleDataSize(n uint32) bool {
function GetRootAsTableIndex (line 13) | func GetRootAsTableIndex(buf []byte, offset flatbuffers.UOffsetT) *Table...
function TableIndexStart (line 143) | func TableIndexStart(builder *flatbuffers.Builder) {
function TableIndexAddOffsets (line 146) | func TableIndexAddOffsets(builder *flatbuffers.Builder, offsets flatbuff...
function TableIndexStartOffsetsVector (line 149) | func TableIndexStartOffsetsVector(builder *flatbuffers.Builder, numElems...
function TableIndexAddBloomFilter (line 152) | func TableIndexAddBloomFilter(builder *flatbuffers.Builder, bloomFilter ...
function TableIndexStartBloomFilterVector (line 155) | func TableIndexStartBloomFilterVector(builder *flatbuffers.Builder, numE...
function TableIndexAddMaxVersion (line 158) | func TableIndexAddMaxVersion(builder *flatbuffers.Builder, maxVersion ui...
function TableIndexAddKeyCount (line 161) | func TableIndexAddKeyCount(builder *flatbuffers.Builder, keyCount uint32) {
function TableIndexAddUncompressedSize (line 164) | func TableIndexAddUncompressedSize(builder *flatbuffers.Builder, uncompr...
function TableIndexAddOnDiskSize (line 167) | func TableIndexAddOnDiskSize(builder *flatbuffers.Builder, onDiskSize ui...
function TableIndexAddStaleDataSize (line 170) | func TableIndexAddStaleDataSize(builder *flatbuffers.Builder, staleDataS...
function TableIndexEnd (line 173) | func TableIndexEnd(builder *flatbuffers.Builder) flatbuffers.UOffsetT {
FILE: histogram.go
method PrintHistogram (line 16) | func (db *DB) PrintHistogram(keyPrefix []byte) {
type histogramData (line 29) | type histogramData struct
method Update (line 79) | func (histogram *histogramData) Update(value int64) {
method printHistogram (line 127) | func (histogram histogramData) printHistogram() {
type sizeHistogram (line 39) | type sizeHistogram struct
function newSizeHistogram (line 45) | func newSizeHistogram() *sizeHistogram {
function createHistogramBins (line 69) | func createHistogramBins(minExponent, maxExponent uint32) []int64 {
method buildHistogram (line 108) | func (db *DB) buildHistogram(keyPrefix []byte) *sizeHistogram {
FILE: histogram_test.go
function TestBuildKeyValueSizeHistogram (line 14) | func TestBuildKeyValueSizeHistogram(t *testing.T) {
FILE: integration/testgc/main.go
type testSuite (line 28) | type testSuite struct
method write (line 41) | func (s *testSuite) write(db *badger.DB) error {
method read (line 70) | func (s *testSuite) read(db *badger.DB) error {
function encoded (line 35) | func encoded(i uint64) []byte {
function main (line 104) | func main() {
FILE: iterator.go
type prefetchStatus (line 22) | type prefetchStatus
constant prefetched (line 25) | prefetched prefetchStatus = iota + 1
type Item (line 30) | type Item struct
method String (line 49) | func (item *Item) String() string {
method Key (line 57) | func (item *Item) Key() []byte {
method KeyCopy (line 64) | func (item *Item) KeyCopy(dst []byte) []byte {
method Version (line 69) | func (item *Item) Version() uint64 {
method Value (line 83) | func (item *Item) Value(fn func(val []byte) error) error {
method ValueCopy (line 110) | func (item *Item) ValueCopy(dst []byte) ([]byte, error) {
method hasValue (line 120) | func (item *Item) hasValue() bool {
method IsDeletedOrExpired (line 129) | func (item *Item) IsDeletedOrExpired() bool {
method DiscardEarlierVersions (line 135) | func (item *Item) DiscardEarlierVersions() bool {
method yieldItemValue (line 139) | func (item *Item) yieldItemValue() ([]byte, func(), error) {
method prefetchValue (line 197) | func (item *Item) prefetchValue() {
method EstimatedSize (line 216) | func (item *Item) EstimatedSize() int64 {
method KeySize (line 230) | func (item *Item) KeySize() int64 {
method ValueSize (line 238) | func (item *Item) ValueSize() int64 {
method UserMeta (line 256) | func (item *Item) UserMeta() byte {
method ExpiresAt (line 262) | func (item *Item) ExpiresAt() uint64 {
function runCallback (line 191) | func runCallback(cb func()) {
type list (line 267) | type list struct
method push (line 272) | func (l *list) push(i *Item) {
method pop (line 283) | func (l *list) pop() *Item {
type IteratorOptions (line 304) | type IteratorOptions struct
method compareToPrefix (line 323) | func (opt *IteratorOptions) compareToPrefix(key []byte) int {
method pickTable (line 332) | func (opt *IteratorOptions) pickTable(t table.TableInterface) bool {
method pickTables (line 356) | func (opt *IteratorOptions) pickTables(all []*table.Table) []*table.Ta...
type Iterator (line 426) | type Iterator struct
method newItem (line 506) | func (it *Iterator) newItem() *Item {
method Item (line 516) | func (it *Iterator) Item() *Item {
method Valid (line 523) | func (it *Iterator) Valid() bool {
method ValidForPrefix (line 535) | func (it *Iterator) ValidForPrefix(prefix []byte) bool {
method Close (line 540) | func (it *Iterator) Close() {
method Next (line 570) | func (it *Iterator) Next() {
method parseItem (line 607) | func (it *Iterator) parseItem() bool {
method fill (line 696) | func (it *Iterator) fill(item *Item) {
method prefetch (line 726) | func (it *Iterator) prefetch() {
method Seek (line 749) | func (it *Iterator) Seek(key []byte) {
method Rewind (line 783) | func (it *Iterator) Rewind() {
method NewIterator (line 459) | func (txn *Txn) NewIterator(opt IteratorOptions) *Iterator {
method NewKeyIterator (line 496) | func (txn *Txn) NewKeyIterator(key []byte, opt IteratorOptions) *Iterator {
function isDeletedOrExpired (line 590) | func isDeletedOrExpired(meta byte, expiresAt uint64) bool {
function hasPrefix (line 717) | func hasPrefix(it *Iterator) bool {
FILE: iterator_test.go
type tableMock (line 25) | type tableMock struct
method Smallest (line 29) | func (tm *tableMock) Smallest() []byte { return tm.left }
method Biggest (line 30) | func (tm *tableMock) Biggest() []byte { return tm.right }
method DoesNotHave (line 31) | func (tm *tableMock) DoesNotHave(hash uint32) bool { return false }
method MaxVersion (line 32) | func (tm *tableMock) MaxVersion() uint64 { return math.MaxUi...
function TestPickTables (line 34) | func TestPickTables(t *testing.T) {
function TestPickSortTables (line 65) | func TestPickSortTables(t *testing.T) {
function TestIterateSinceTs (line 117) | func TestIterateSinceTs(t *testing.T) {
function TestIterateSinceTsWithPendingWrites (line 153) | func TestIterateSinceTsWithPendingWrites(t *testing.T) {
function TestIteratePrefix (line 171) | func TestIteratePrefix(t *testing.T) {
function TestIteratorReadOnlyWithNoData (line 285) | func TestIteratorReadOnlyWithNoData(t *testing.T) {
function BenchmarkIteratePrefixSingleKey (line 350) | func BenchmarkIteratePrefixSingleKey(b *testing.B) {
FILE: key_registry.go
constant KeyRegistryFileName (line 27) | KeyRegistryFileName = "KEYREGISTRY"
constant KeyRegistryRewriteFileName (line 29) | KeyRegistryRewriteFileName = "REWRITE-KEYREGISTRY"
type KeyRegistry (line 36) | type KeyRegistry struct
method DataKey (line 288) | func (kr *KeyRegistry) DataKey(id uint64) (*pb.DataKey, error) {
method LatestDataKey (line 305) | func (kr *KeyRegistry) LatestDataKey() (*pb.DataKey, error) {
method Close (line 372) | func (kr *KeyRegistry) Close() error {
type KeyRegistryOptions (line 45) | type KeyRegistryOptions struct
function newKeyRegistry (line 54) | func newKeyRegistry(opt KeyRegistryOptions) *KeyRegistry {
function OpenKeyRegistry (line 64) | func OpenKeyRegistry(opt KeyRegistryOptions) (*KeyRegistry, error) {
type keyRegistryIterator (line 122) | type keyRegistryIterator struct
method next (line 163) | func (kri *keyRegistryIterator) next() (*pb.DataKey, error) {
function newKeyRegistryIterator (line 131) | func newKeyRegistryIterator(fp *os.File, encryptionKey []byte) (*keyRegi...
function validRegistry (line 140) | func validRegistry(fp *os.File, encryptionKey []byte) error {
function readKeyRegistry (line 201) | func readKeyRegistry(fp *os.File, opt KeyRegistryOptions) (*KeyRegistry,...
function WriteKeyRegistry (line 239) | func WriteKeyRegistry(reg *KeyRegistry, opt KeyRegistryOptions) error {
function storeDataKey (line 380) | func storeDataKey(buf *bytes.Buffer, storageKey []byte, k *pb.DataKey) e...
FILE: key_registry_test.go
function getRegistryTestOptions (line 16) | func getRegistryTestOptions(dir string, key []byte) KeyRegistryOptions {
function TestBuildRegistry (line 23) | func TestBuildRegistry(t *testing.T) {
function TestRewriteRegistry (line 55) | func TestRewriteRegistry(t *testing.T) {
function TestMismatch (line 81) | func TestMismatch(t *testing.T) {
function TestEncryptionAndDecryption (line 106) | func TestEncryptionAndDecryption(t *testing.T) {
function TestKeyRegistryInMemory (line 129) | func TestKeyRegistryInMemory(t *testing.T) {
FILE: level_handler.go
type levelHandler (line 17) | type levelHandler struct
method isLastLevel (line 34) | func (s *levelHandler) isLastLevel() bool {
method getTotalStaleSize (line 38) | func (s *levelHandler) getTotalStaleSize() int64 {
method getTotalSize (line 44) | func (s *levelHandler) getTotalSize() int64 {
method initTables (line 51) | func (s *levelHandler) initTables(tables []*table.Table) {
method deleteTables (line 77) | func (s *levelHandler) deleteTables(toDel []*table.Table) error {
method replaceTables (line 104) | func (s *levelHandler) replaceTables(toDel, toAdd []*table.Table) error {
method addTable (line 145) | func (s *levelHandler) addTable(t *table.Table) {
method sortTables (line 156) | func (s *levelHandler) sortTables() {
method tryAddLevel0Table (line 183) | func (s *levelHandler) tryAddLevel0Table(t *table.Table) bool {
method addSize (line 201) | func (s *levelHandler) addSize(t *table.Table) {
method subtractSize (line 207) | func (s *levelHandler) subtractSize(t *table.Table) {
method numTables (line 211) | func (s *levelHandler) numTables() int {
method close (line 217) | func (s *levelHandler) close() error {
method getTableForKey (line 230) | func (s *levelHandler) getTableForKey(key []byte) ([]*table.Table, fun...
method get (line 266) | func (s *levelHandler) get(key []byte) (y.ValueStruct, error) {
method appendIterators (line 298) | func (s *levelHandler) appendIterators(iters []y.Iterator, opt *Iterat...
method overlappingTables (line 331) | func (s *levelHandler) overlappingTables(_ levelHandlerRLocked, kr key...
function decrRefs (line 165) | func decrRefs(tables []*table.Table) error {
function newLevelHandler (line 174) | func newLevelHandler(db *DB, level int) *levelHandler {
type levelHandlerRLocked (line 326) | type levelHandlerRLocked struct
FILE: levels.go
type levelsController (line 32) | type levelsController struct
method cleanupLevels (line 195) | func (s *levelsController) cleanupLevels() error {
method dropTree (line 208) | func (s *levelsController) dropTree() (int, error) {
method dropPrefixes (line 255) | func (s *levelsController) dropPrefixes(prefixes [][]byte) error {
method startCompact (line 344) | func (s *levelsController) startCompact(lc *z.Closer) {
method levelTargets (line 371) | func (s *levelsController) levelTargets() targets {
method runCompactor (line 428) | func (s *levelsController) runCompactor(id int, lc *z.Closer) {
method lastLevel (line 532) | func (s *levelsController) lastLevel() *levelHandler {
method pickCompactLevels (line 540) | func (s *levelsController) pickCompactLevels(priosBuffer []compactionP...
method checkOverlap (line 615) | func (s *levelsController) checkOverlap(tables []*table.Table, lev int...
method subcompact (line 638) | func (s *levelsController) subcompact(it y.Iterator, kr keyRange, cd c...
method compactBuildTables (line 868) | func (s *levelsController) compactBuildTables(
method addSplits (line 1048) | func (s *levelsController) addSplits(cd *compactDef) {
method fillTablesL0ToL0 (line 1107) | func (s *levelsController) fillTablesL0ToL0(cd *compactDef) bool {
method fillTablesL0ToLbase (line 1175) | func (s *levelsController) fillTablesL0ToLbase(cd *compactDef) bool {
method fillTablesL0 (line 1237) | func (s *levelsController) fillTablesL0(cd *compactDef) bool {
method sortByStaleDataSize (line 1246) | func (s *levelsController) sortByStaleDataSize(tables []*table.Table, ...
method sortByHeuristic (line 1258) | func (s *levelsController) sortByHeuristic(tables []*table.Table, cd *...
method fillMaxLevelTables (line 1270) | func (s *levelsController) fillMaxLevelTables(tables []*table.Table, c...
method fillTables (line 1354) | func (s *levelsController) fillTables(cd *compactDef) bool {
method runCompactDef (line 1405) | func (s *levelsController) runCompactDef(id, l int, cd compactDef) (er...
method doCompact (line 1507) | func (s *levelsController) doCompact(id int, p compactionPriority) err...
method addLevel0Table (line 1559) | func (s *levelsController) addLevel0Table(t *table.Table) error {
method close (line 1591) | func (s *levelsController) close() error {
method get (line 1599) | func (s *levelsController) get(key []byte, maxVs y.ValueStruct, startL...
method appendIterators (line 1646) | func (s *levelsController) appendIterators(
method getTableInfo (line 1671) | func (s *levelsController) getTableInfo() (result []TableInfo) {
method getLevelInfo (line 1713) | func (s *levelsController) getLevelInfo() []LevelInfo {
method verifyChecksum (line 1738) | func (s *levelsController) verifyChecksum() error {
method keySplits (line 1767) | func (s *levelsController) keySplits(numPerTable int, prefix []byte) [...
function revertToManifest (line 46) | func revertToManifest(kv *DB, mf *Manifest, idMap map[uint64]struct{}) e...
function newLevelsController (line 68) | func newLevelsController(db *DB, mf *Manifest) (*levelsController, error) {
function closeAllTables (line 187) | func closeAllTables(tables [][]*table.Table) {
type targets (line 352) | type targets struct
type compactionPriority (line 524) | type compactionPriority struct
function buildChangeSet (line 962) | func buildChangeSet(cd *compactDef, newTables []*table.Table) pb.Manifes...
function hasAnyPrefixes (line 980) | func hasAnyPrefixes(s []byte, listOfPrefixes [][]byte) bool {
function containsPrefix (line 990) | func containsPrefix(table *table.Table, prefix []byte) bool {
function containsAnyPrefixes (line 1018) | func containsAnyPrefixes(table *table.Table, listOfPrefixes [][]byte) bo...
type compactDef (line 1028) | type compactDef struct
method lockLevels (line 1090) | func (cd *compactDef) lockLevels() {
method unlockLevels (line 1095) | func (cd *compactDef) unlockLevels() {
method allTables (line 1100) | func (cd *compactDef) allTables() []*table.Table {
function tablesToString (line 1495) | func tablesToString(tables []*table.Table) []string {
function appendIteratorsReversed (line 1636) | func appendIteratorsReversed(out []y.Iterator, th []*table.Table, opt in...
type TableInfo (line 1657) | type TableInfo struct
type LevelInfo (line 1701) | type LevelInfo struct
FILE: levels_test.go
function createAndOpen (line 26) | func createAndOpen(db *DB, td []keyValVersion, level int) {
type keyValVersion (line 57) | type keyValVersion struct
function TestCheckOverlap (line 64) | func TestCheckOverlap(t *testing.T) {
function getAllAndCheck (line 122) | func getAllAndCheck(t *testing.T, db *DB, expected []keyValVersion) {
function TestCompaction (line 152) | func TestCompaction(t *testing.T) {
function TestCompactionTwoVersions (line 444) | func TestCompactionTwoVersions(t *testing.T) {
function TestCompactionAllVersions (line 501) | func TestCompactionAllVersions(t *testing.T) {
function TestDiscardTs (line 585) | func TestDiscardTs(t *testing.T) {
function TestDiscardFirstVersion (line 691) | func TestDiscardFirstVersion(t *testing.T) {
function TestL1Stall (line 746) | func TestL1Stall(t *testing.T) {
function createEmptyTable (line 795) | func createEmptyTable(db *DB) *table.Table {
function TestL0Stall (line 814) | func TestL0Stall(t *testing.T) {
function TestLevelGet (line 867) | func TestLevelGet(t *testing.T) {
function TestKeyVersions (line 962) | func TestKeyVersions(t *testing.T) {
function TestSameLevel (line 1044) | func TestSameLevel(t *testing.T) {
function TestTableContainsPrefix (line 1117) | func TestTableContainsPrefix(t *testing.T) {
function TestFillTableCleanup (line 1158) | func TestFillTableCleanup(t *testing.T) {
function TestStaleDataCleanup (line 1236) | func TestStaleDataCleanup(t *testing.T) {
FILE: logger.go
type Logger (line 14) | type Logger interface
method Errorf (line 23) | func (opt *Options) Errorf(format string, v ...interface{}) {
method Infof (line 31) | func (opt *Options) Infof(format string, v ...interface{}) {
method Warningf (line 39) | func (opt *Options) Warningf(format string, v ...interface{}) {
method Debugf (line 47) | func (opt *Options) Debugf(format string, v ...interface{}) {
type loggingLevel (line 54) | type loggingLevel
constant DEBUG (line 57) | DEBUG loggingLevel = iota
constant INFO (line 58) | INFO
constant WARNING (line 59) | WARNING
constant ERROR (line 60) | ERROR
type defaultLog (line 63) | type defaultLog struct
method Errorf (line 72) | func (l *defaultLog) Errorf(f string, v ...interface{}) {
method Warningf (line 78) | func (l *defaultLog) Warningf(f string, v ...interface{}) {
method Infof (line 84) | func (l *defaultLog) Infof(f string, v ...interface{}) {
method Debugf (line 90) | func (l *defaultLog) Debugf(f string, v ...interface{}) {
function defaultLogger (line 68) | func defaultLogger(level loggingLevel) *defaultLog {
FILE: logger_test.go
type mockLogger (line 15) | type mockLogger struct
method Errorf (line 19) | func (l *mockLogger) Errorf(f string, v ...interface{}) {
method Infof (line 23) | func (l *mockLogger) Infof(f string, v ...interface{}) {
method Warningf (line 27) | func (l *mockLogger) Warningf(f string, v ...interface{}) {
method Debugf (line 31) | func (l *mockLogger) Debugf(f string, v ...interface{}) {
function TestDbLog (line 36) | func TestDbLog(t *testing.T) {
function TestNoDbLog (line 49) | func TestNoDbLog(t *testing.T) {
FILE: managed_db.go
function OpenManaged (line 13) | func OpenManaged(opts Options) (*DB, error) {
method NewTransactionAt (line 23) | func (db *DB) NewTransactionAt(readTs uint64, update bool) *Txn {
method NewWriteBatchAt (line 34) | func (db *DB) NewWriteBatchAt(commitTs uint64) *WriteBatch {
method NewManagedWriteBatch (line 44) | func (db *DB) NewManagedWriteBatch() *WriteBatch {
method CommitAt (line 58) | func (txn *Txn) CommitAt(commitTs uint64, callback func(error)) error {
method SetDiscardTs (line 73) | func (db *DB) SetDiscardTs(ts uint64) {
FILE: managed_db_test.go
function val (line 25) | func val(large bool) []byte {
function numKeys (line 36) | func numKeys(db *DB) int {
function numKeysManaged (line 51) | func numKeysManaged(db *DB, readTs uint64) int {
function TestDropAllManaged (line 65) | func TestDropAllManaged(t *testing.T) {
function TestDropAll (line 110) | func TestDropAll(t *testing.T) {
function TestDropAllTwice (line 146) | func TestDropAllTwice(t *testing.T) {
function TestDropAllWithPendingTxn (line 189) | func TestDropAllWithPendingTxn(t *testing.T) {
function TestDropReadOnly (line 260) | func TestDropReadOnly(t *testing.T) {
function TestWriteAfterClose (line 293) | func TestWriteAfterClose(t *testing.T) {
function TestDropAllRace (line 319) | func TestDropAllRace(t *testing.T) {
function TestDropPrefix (line 383) | func TestDropPrefix(t *testing.T) {
function TestDropPrefixWithPendingTxn (line 435) | func TestDropPrefixWithPendingTxn(t *testing.T) {
function TestDropPrefixReadOnly (line 507) | func TestDropPrefixReadOnly(t *testing.T) {
function TestDropPrefixRace (line 540) | func TestDropPrefixRace(t *testing.T) {
function TestWriteBatchManagedMode (line 606) | func TestWriteBatchManagedMode(t *testing.T) {
function TestWriteBatchManaged (line 652) | func TestWriteBatchManaged(t *testing.T) {
function TestWriteBatchDuplicate (line 699) | func TestWriteBatchDuplicate(t *testing.T) {
function TestZeroDiscardStats (line 780) | func TestZeroDiscardStats(t *testing.T) {
FILE: manifest.go
type Manifest (line 36) | type Manifest struct
method asChanges (line 100) | func (m *Manifest) asChanges() []*pb.ManifestChange {
method clone (line 108) | func (m *Manifest) clone(opt Options) Manifest {
function createManifest (line 46) | func createManifest() Manifest {
type levelManifest (line 56) | type levelManifest struct
type TableManifest (line 62) | type TableManifest struct
type manifestFile (line 70) | type manifestFile struct
method close (line 186) | func (mf *manifestFile) close() error {
method addChanges (line 197) | func (mf *manifestFile) addChanges(changesParam []*pb.ManifestChange, ...
method rewrite (line 308) | func (mf *manifestFile) rewrite() error {
constant ManifestFilename (line 92) | ManifestFilename = "MANIFEST"
constant manifestRewriteFilename (line 93) | manifestRewriteFilename = "MANIFEST-REWRITE"
constant manifestDeletionsRewriteThreshold (line 94) | manifestDeletionsRewriteThreshold = 10000
constant manifestDeletionsRatio (line 95) | manifestDeletionsRatio = 10
function openOrCreateManifestFile (line 117) | func openOrCreateManifestFile(opt Options) (
function helpOpenOrCreateManifestFile (line 126) | func helpOpenOrCreateManifestFile(dir string, readOnly bool, extMagic ui...
constant badgerMagicVersion (line 239) | badgerMagicVersion = 8
function helpRewrite (line 241) | func helpRewrite(dir string, m *Manifest, extMagic uint16) (*os.File, in...
type countingReader (line 324) | type countingReader struct
method Read (line 329) | func (r *countingReader) Read(p []byte) (n int, err error) {
method ReadByte (line 335) | func (r *countingReader) ReadByte() (b byte, err error) {
function ReplayManifestFile (line 353) | func ReplayManifestFile(fp *os.File, extMagic uint16, opt Options) (Mani...
function applyManifestChange (line 429) | func applyManifestChange(build *Manifest, tc *pb.ManifestChange, opt Opt...
function applyChangeSet (line 465) | func applyChangeSet(build *Manifest, changeSet *pb.ManifestChangeSet, op...
function newCreateChange (line 474) | func newCreateChange(
function newDeleteChange (line 487) | func newDeleteChange(id uint64) *pb.ManifestChange {
FILE: manifest_test.go
function TestManifestBasic (line 26) | func TestManifestBasic(t *testing.T) {
function helpTestManifestFileCorruption (line 61) | func helpTestManifestFileCorruption(t *testing.T, off int64, errorConten...
function TestManifestMagic (line 88) | func TestManifestMagic(t *testing.T) {
function TestManifestVersion (line 92) | func TestManifestVersion(t *testing.T) {
function TestManifestChecksum (line 96) | func TestManifestChecksum(t *testing.T) {
function key (line 100) | func key(prefix string, i int) string {
function buildTable (line 106) | func buildTable(t *testing.T, keyValues [][]string, bopts table.Options)...
function TestOverlappingKeyRangeError (line 136) | func TestOverlappingKeyRangeError(t *testing.T) {
function TestManifestRewrite (line 196) | func TestManifestRewrite(t *testing.T) {
function TestConcurrentManifestCompaction (line 243) | func TestConcurrentManifestCompaction(t *testing.T) {
FILE: memtable.go
type memTable (line 34) | type memTable struct
method SyncWAL (line 155) | func (mt *memTable) SyncWAL() error {
method isFull (line 159) | func (mt *memTable) isFull() bool {
method Put (line 170) | func (mt *memTable) Put(key []byte, value y.ValueStruct) error {
method UpdateSkipList (line 201) | func (mt *memTable) UpdateSkipList() error {
method IncrRef (line 216) | func (mt *memTable) IncrRef() {
method DecrRef (line 221) | func (mt *memTable) DecrRef() {
method replayFunction (line 225) | func (mt *memTable) replayFunction(opt Options) func(Entry, valuePoint...
method openMemTables (line 43) | func (db *DB) openMemTables(opt Options) error {
constant memFileExt (line 95) | memFileExt string = ".mem"
method openMemTable (line 97) | func (db *DB) openMemTable(fid, flags int) (*memTable, error) {
method newMemTable (line 137) | func (db *DB) newMemTable() (*memTable, error) {
method mtFilePath (line 151) | func (db *DB) mtFilePath(fid int) string {
type logFile (line 249) | type logFile struct
method Truncate (line 267) | func (lf *logFile) Truncate(end int64) error {
method encodeEntry (line 283) | func (lf *logFile) encodeEntry(buf *bytes.Buffer, e *Entry, offset uin...
method writeEntry (line 324) | func (lf *logFile) writeEntry(buf *bytes.Buffer, e *Entry, opt Options...
method decodeEntry (line 337) | func (lf *logFile) decodeEntry(buf []byte, offset uint32) (*Entry, err...
method decryptKV (line 360) | func (lf *logFile) decryptKV(buf []byte, offset uint32) ([]byte, error) {
method keyID (line 365) | func (lf *logFile) keyID() uint64 {
method encryptionEnabled (line 373) | func (lf *logFile) encryptionEnabled() bool {
method read (line 378) | func (lf *logFile) read(p valuePointer) (buf []byte, err error) {
method generateIV (line 399) | func (lf *logFile) generateIV(offset uint32) []byte {
method doneWriting (line 408) | func (lf *logFile) doneWriting(offset uint32) error {
method iterate (line 433) | func (lf *logFile) iterate(readOnly bool, offset uint32, fn logEntry) ...
method zeroNextEntry (line 533) | func (lf *logFile) zeroNextEntry() {
method open (line 537) | func (lf *logFile) open(path string, flags int, fsize int64) error {
method bootstrap (line 584) | func (lf *logFile) bootstrap() error {
FILE: merge.go
type MergeOperator (line 18) | type MergeOperator struct
method iterateAndMerge (line 51) | func (op *MergeOperator) iterateAndMerge() (newVal []byte, latest uint...
method compact (line 95) | func (op *MergeOperator) compact() error {
method runCompactions (line 120) | func (op *MergeOperator) runCompactions(dur time.Duration) {
method Add (line 142) | func (op *MergeOperator) Add(val []byte) error {
method Get (line 152) | func (op *MergeOperator) Get() ([]byte, error) {
method Stop (line 168) | func (op *MergeOperator) Stop() {
type MergeFunc (line 31) | type MergeFunc
method GetMergeOperator (line 36) | func (db *DB) GetMergeOperator(key []byte,
FILE: merge_test.go
function TestGetMergeOperator (line 17) | func TestGetMergeOperator(t *testing.T) {
function uint64ToBytes (line 165) | func uint64ToBytes(i uint64) []byte {
function bytesToUint64 (line 171) | func bytesToUint64(b []byte) uint64 {
function add (line 176) | func add(existing, latest []byte) []byte {
FILE: metrics_test.go
function clearAllMetrics (line 16) | func clearAllMetrics() {
function TestWriteMetrics (line 32) | func TestWriteMetrics(t *testing.T) {
function TestVlogMetrics (line 77) | func TestVlogMetrics(t *testing.T) {
function TestReadMetrics (line 122) | func TestReadMetrics(t *testing.T) {
FILE: options.go
type Options (line 31) | type Options struct
method FromSuperFlag (line 297) | func (opt Options) FromSuperFlag(superflag string) Options {
method WithDir (line 355) | func (opt Options) WithDir(val string) Options {
method WithValueDir (line 365) | func (opt Options) WithValueDir(val string) Options {
method WithSyncWrites (line 379) | func (opt Options) WithSyncWrites(val bool) Options {
method WithNumVersionsToKeep (line 389) | func (opt Options) WithNumVersionsToKeep(val int) Options {
method WithNumGoroutines (line 397) | func (opt Options) WithNumGoroutines(val int) Options {
method WithReadOnly (line 410) | func (opt Options) WithReadOnly(val bool) Options {
method WithMetricsEnabled (line 425) | func (opt Options) WithMetricsEnabled(val bool) Options {
method WithLogger (line 435) | func (opt Options) WithLogger(val Logger) Options {
method WithLoggingLevel (line 446) | func (opt Options) WithLoggingLevel(val loggingLevel) Options {
method WithBaseTableSize (line 456) | func (opt Options) WithBaseTableSize(val int64) Options {
method WithLevelSizeMultiplier (line 469) | func (opt Options) WithLevelSizeMultiplier(val int) Options {
method WithMaxLevels (line 479) | func (opt Options) WithMaxLevels(val int) Options {
method WithValueThreshold (line 491) | func (opt Options) WithValueThreshold(val int64) Options {
method WithVLogPercentile (line 508) | func (opt Options) WithVLogPercentile(t float64) Options {
method WithNumMemtables (line 518) | func (opt Options) WithNumMemtables(val int) Options {
method WithMemTableSize (line 528) | func (opt Options) WithMemTableSize(val int64) Options {
method WithBloomFalsePositive (line 544) | func (opt Options) WithBloomFalsePositive(val float64) Options {
method WithBlockSize (line 555) | func (opt Options) WithBlockSize(val int) Options {
method WithNumLevelZeroTables (line 563) | func (opt Options) WithNumLevelZeroTables(val int) Options {
method WithNumLevelZeroTablesStall (line 572) | func (opt Options) WithNumLevelZeroTablesStall(val int) Options {
method WithBaseLevelSize (line 580) | func (opt Options) WithBaseLevelSize(val int64) Options {
method WithValueLogFileSize (line 588) | func (opt Options) WithValueLogFileSize(val int64) Options {
method WithValueLogMaxEntries (line 598) | func (opt Options) WithValueLogMaxEntries(val uint32) Options {
method WithNumCompactors (line 607) | func (opt Options) WithNumCompactors(val int) Options {
method WithCompactL0OnClose (line 616) | func (opt Options) WithCompactL0OnClose(val bool) Options {
method WithEncryptionKey (line 624) | func (opt Options) WithEncryptionKey(key []byte) Options {
method WithEncryptionKeyRotationDuration (line 636) | func (opt Options) WithEncryptionKeyRotationDuration(d time.Duration) ...
method WithCompression (line 646) | func (opt Options) WithCompression(cType options.CompressionType) Opti...
method WithVerifyValueChecksum (line 656) | func (opt Options) WithVerifyValueChecksum(val bool) Options {
method WithChecksumVerificationMode (line 667) | func (opt Options) WithChecksumVerificationMode(cvMode options.Checksu...
method WithBlockCacheSize (line 682) | func (opt Options) WithBlockCacheSize(size int64) Options {
method WithInMemory (line 691) | func (opt Options) WithInMemory(b bool) Options {
method WithZSTDCompressionLevel (line 715) | func (opt Options) WithZSTDCompressionLevel(cLevel int) Options {
method WithBypassLockGuard (line 728) | func (opt Options) WithBypassLockGuard(b bool) Options {
method WithIndexCacheSize (line 746) | func (opt Options) WithIndexCacheSize(size int64) Options {
method WithDetectConflicts (line 760) | func (opt Options) WithDetectConflicts(b bool) Options {
method WithNamespaceOffset (line 770) | func (opt Options) WithNamespaceOffset(offset int) Options {
method WithExternalMagic (line 777) | func (opt Options) WithExternalMagic(magic uint16) Options {
method getFileFlags (line 782) | func (opt Options) getFileFlags() int {
function DefaultOptions (line 123) | func DefaultOptions(path string) Options {
function buildTableOptions (line 180) | func buildTableOptions(db *DB) table.Options {
constant maxValueThreshold (line 201) | maxValueThreshold = (1 << 20)
function LSMOnlyOptions (line 208) | func LSMOnlyOptions(path string) Options {
function parseCompression (line 224) | func parseCompression(cStr string) (options.CompressionType, int, error) {
function generateSuperFlag (line 252) | func generateSuperFlag(options Options) string {
FILE: options/options.go
type ChecksumVerificationMode (line 9) | type ChecksumVerificationMode
constant NoVerification (line 13) | NoVerification ChecksumVerificationMode = iota
constant OnTableRead (line 15) | OnTableRead
constant OnBlockRead (line 17) | OnBlockRead
constant OnTableAndBlockRead (line 20) | OnTableAndBlockRead
type CompressionType (line 24) | type CompressionType
constant None (line 28) | None CompressionType = 0
constant Snappy (line 30) | Snappy CompressionType = 1
constant ZSTD (line 32) | ZSTD CompressionType = 2
FILE: options_test.go
function TestOptions (line 15) | func TestOptions(t *testing.T) {
function optionsEqual (line 51) | func optionsEqual(o1, o2 Options) bool {
FILE: pb/badgerpb4.pb.go
constant _ (line 24) | _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion)
constant _ (line 26) | _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20)
type EncryptionAlgo (line 29) | type EncryptionAlgo
method Enum (line 45) | func (x EncryptionAlgo) Enum() *EncryptionAlgo {
method String (line 51) | func (x EncryptionAlgo) String() string {
method Descriptor (line 55) | func (EncryptionAlgo) Descriptor() protoreflect.EnumDescriptor {
method Type (line 59) | func (EncryptionAlgo) Type() protoreflect.EnumType {
method Number (line 63) | func (x EncryptionAlgo) Number() protoreflect.EnumNumber {
method EnumDescriptor (line 68) | func (EncryptionAlgo) EnumDescriptor() ([]byte, []int) {
constant EncryptionAlgo_aes (line 32) | EncryptionAlgo_aes EncryptionAlgo = 0
type ManifestChange_Operation (line 72) | type ManifestChange_Operation
method Enum (line 91) | func (x ManifestChange_Operation) Enum() *ManifestChange_Operation {
method String (line 97) | func (x ManifestChange_Operation) String() string {
method Descriptor (line 101) | func (ManifestChange_Operation) Descriptor() protoreflect.EnumDescript...
method Type (line 105) | func (ManifestChange_Operation) Type() protoreflect.EnumType {
method Number (line 109) | func (x ManifestChange_Operation) Number() protoreflect.EnumNumber {
method EnumDescriptor (line 114) | func (ManifestChange_Operation) EnumDescriptor() ([]byte, []int) {
constant ManifestChange_CREATE (line 75) | ManifestChange_CREATE ManifestChange_Operation = 0
constant ManifestChange_DELETE (line 76) | ManifestChange_DELETE ManifestChange_Operation = 1
type Checksum_Algorithm (line 118) | type Checksum_Algorithm
method Enum (line 137) | func (x Checksum_Algorithm) Enum() *Checksum_Algorithm {
method String (line 143) | func (x Checksum_Algorithm) String() string {
method Descriptor (line 147) | func (Checksum_Algorithm) Descriptor() protoreflect.EnumDescriptor {
method Type (line 151) | func (Checksum_Algorithm) Type() protoreflect.EnumType {
method Number (line 155) | func (x Checksum_Algorithm) Number() protoreflect.EnumNumber {
method EnumDescriptor (line 160) | func (Checksum_Algorithm) EnumDescriptor() ([]byte, []int) {
constant Checksum_CRC32C (line 121) | Checksum_CRC32C Checksum_Algorithm = 0
constant Checksum_XXHash64 (line 122) | Checksum_XXHash64 Checksum_Algorithm = 1
type KV (line 164) | type KV struct
method Reset (line 181) | func (x *KV) Reset() {
method String (line 190) | func (x *KV) String() string {
method ProtoMessage (line 194) | func (*KV) ProtoMessage() {}
method ProtoReflect (line 196) | func (x *KV) ProtoReflect() protoreflect.Message {
method Descriptor (line 209) | func (*KV) Descriptor() ([]byte, []int) {
method GetKey (line 213) | func (x *KV) GetKey() []byte {
method GetValue (line 220) | func (x *KV) GetValue() []byte {
method GetUserMeta (line 227) | func (x *KV) GetUserMeta() []byte {
method GetVersion (line 234) | func (x *KV) GetVersion() uint64 {
method GetExpiresAt (line 241) | func (x *KV) GetExpiresAt() uint64 {
method GetMeta (line 248) | func (x *KV) GetMeta() []byte {
method GetStreamId (line 255) | func (x *KV) GetStreamId() uint32 {
method GetStreamDone (line 262) | func (x *KV) GetStreamDone() bool {
type KVList (line 269) | type KVList struct
method Reset (line 279) | func (x *KVList) Reset() {
method String (line 288) | func (x *KVList) String() string {
method ProtoMessage (line 292) | func (*KVList) ProtoMessage() {}
method ProtoReflect (line 294) | func (x *KVList) ProtoReflect() protoreflect.Message {
method Descriptor (line 307) | func (*KVList) Descriptor() ([]byte, []int) {
method GetKv (line 311) | func (x *KVList) GetKv() []*KV {
method GetAllocRef (line 318) | func (x *KVList) GetAllocRef() uint64 {
type ManifestChangeSet (line 325) | type ManifestChangeSet struct
method Reset (line 334) | func (x *ManifestChangeSet) Reset() {
method String (line 343) | func (x *ManifestChangeSet) String() string {
method ProtoMessage (line 347) | func (*ManifestChangeSet) ProtoMessage() {}
method ProtoReflect (line 349) | func (x *ManifestChangeSet) ProtoReflect() protoreflect.Message {
method Descriptor (line 362) | func (*ManifestChangeSet) Descriptor() ([]byte, []int) {
method GetChanges (line 366) | func (x *ManifestChangeSet) GetChanges() []*ManifestChange {
type ManifestChange (line 373) | type ManifestChange struct
method Reset (line 386) | func (x *ManifestChange) Reset() {
method String (line 395) | func (x *ManifestChange) String() string {
method ProtoMessage (line 399) | func (*ManifestChange) ProtoMessage() {}
method ProtoReflect (line 401) | func (x *ManifestChange) ProtoReflect() protoreflect.Message {
method Descriptor (line 414) | func (*ManifestChange) Descriptor() ([]byte, []int) {
method GetId (line 418) | func (x *ManifestChange) GetId() uint64 {
method GetOp (line 425) | func (x *ManifestChange) GetOp() ManifestChange_Operation {
method GetLevel (line 432) | func (x *ManifestChange) GetLevel() uint32 {
method GetKeyId (line 439) | func (x *ManifestChange) GetKeyId() uint64 {
method GetEncryptionAlgo (line 446) | func (x *ManifestChange) GetEncryptionAlgo() EncryptionAlgo {
method GetCompression (line 453) | func (x *ManifestChange) GetCompression() uint32 {
type Checksum (line 460) | type Checksum struct
method Reset (line 469) | func (x *Checksum) Reset() {
method String (line 478) | func (x *Checksum) String() string {
method ProtoMessage (line 482) | func (*Checksum) ProtoMessage() {}
method ProtoReflect (line 484) | func (x *Checksum) ProtoReflect() protoreflect.Message {
method Descriptor (line 497) | func (*Checksum) Descriptor() ([]byte, []int) {
method GetAlgo (line 501) | func (x *Checksum) GetAlgo() Checksum_Algorithm {
method GetSum (line 508) | func (x *Checksum) GetSum() uint64 {
type DataKey (line 515) | type DataKey struct
method Reset (line 526) | func (x *DataKey) Reset() {
method String (line 535) | func (x *DataKey) String() string {
method ProtoMessage (line 539) | func (*DataKey) ProtoMessage() {}
method ProtoReflect (line 541) | func (x *DataKey) ProtoReflect() protoreflect.Message {
method Descriptor (line 554) | func (*DataKey) Descriptor() ([]byte, []int) {
method GetKeyId (line 558) | func (x *DataKey) GetKeyId() uint64 {
method GetData (line 565) | func (x *DataKey) GetData() []byte {
method GetIv (line 572) | func (x *DataKey) GetIv() []byte {
method GetCreatedAt (line 579) | func (x *DataKey) GetCreatedAt() int64 {
type Match (line 586) | type Match struct
method Reset (line 595) | func (x *Match) Reset() {
method String (line 604) | func (x *Match) String() string {
method ProtoMessage (line 608) | func (*Match) ProtoMessage() {}
method ProtoReflect (line 610) | func (x *Match) ProtoReflect() protoreflect.Message {
method Descriptor (line 623) | func (*Match) Descriptor() ([]byte, []int) {
method GetPrefix (line 627) | func (x *Match) GetPrefix() []byte {
method GetIgnoreBytes (line 634) | func (x *Match) GetIgnoreBytes() string {
function file_badgerpb4_proto_rawDescGZIP (line 715) | func file_badgerpb4_proto_rawDescGZIP() []byte {
function init (line 749) | func init() { file_badgerpb4_proto_init() }
function file_badgerpb4_proto_init (line 750) | func file_badgerpb4_proto_init() {
FILE: pb/protos_test.go
function Exec (line 15) | func Exec(argv ...string) error {
function TestProtosRegenerate (line 24) | func TestProtosRegenerate(t *testing.T) {
FILE: publisher.go
type subscriber (line 18) | type subscriber struct
type publisher (line 28) | type publisher struct
method listenForUpdates (line 45) | func (p *publisher) listenForUpdates(c *z.Closer) {
method publishUpdates (line 71) | func (p *publisher) publishUpdates(reqs requests) {
method newSubscriber (line 109) | func (p *publisher) newSubscriber(c *z.Closer, matches []pb.Match) (su...
method cleanSubscribers (line 135) | func (p *publisher) cleanSubscribers() {
method deleteSubscriber (line 147) | func (p *publisher) deleteSubscriber(id uint64) {
method sendUpdates (line 158) | func (p *publisher) sendUpdates(reqs requests) {
method noOfSubscribers (line 165) | func (p *publisher) noOfSubscribers() int {
function newPublisher (line 36) | func newPublisher() *publisher {
FILE: publisher_test.go
function TestPublisherDeadlock (line 25) | func TestPublisherDeadlock(t *testing.T) {
function TestPublisherOrdering (line 88) | func TestPublisherOrdering(t *testing.T) {
function TestMultiplePrefix (line 127) | func TestMultiplePrefix(t *testing.T) {
FILE: skl/arena.go
constant offsetSize (line 16) | offsetSize = int(unsafe.Sizeof(uint32(0)))
constant nodeAlign (line 22) | nodeAlign = int(unsafe.Sizeof(uint64(0))) - 1
type Arena (line 26) | type Arena struct
method size (line 40) | func (s *Arena) size() int64 {
method putNode (line 46) | func (s *Arena) putNode(height int) uint32 {
method putVal (line 67) | func (s *Arena) putVal(v y.ValueStruct) uint32 {
method putKey (line 78) | func (s *Arena) putKey(key []byte) uint32 {
method getNode (line 94) | func (s *Arena) getNode(offset uint32) *node {
method getKey (line 103) | func (s *Arena) getKey(offset uint32, size uint16) []byte {
method getVal (line 109) | func (s *Arena) getVal(offset uint32, size uint32) (ret y.ValueStruct) {
method getNodeOffset (line 116) | func (s *Arena) getNodeOffset(nd *node) uint32 {
function newArena (line 32) | func newArena(n int64) *Arena {
FILE: skl/skl.go
constant maxHeight (line 34) | maxHeight = 20
constant heightIncrease (line 35) | heightIncrease = math.MaxUint32 / 3
constant MaxNodeSize (line 39) | MaxNodeSize = int(unsafe.Sizeof(node{}))
type node (line 41) | type node struct
method getValueOffset (line 127) | func (s *node) getValueOffset() (uint32, uint32) {
method key (line 132) | func (s *node) key(arena *Arena) []byte {
method setValue (line 136) | func (s *node) setValue(arena *Arena, v y.ValueStruct) {
method getNextOffset (line 142) | func (s *node) getNextOffset(h int) uint32 {
method casNextOffset (line 146) | func (s *node) casNextOffset(h int, old, val uint32) bool {
type Skiplist (line 65) | type Skiplist struct
method IncrRef (line 74) | func (s *Skiplist) IncrRef() {
method DecrRef (line 79) | func (s *Skiplist) DecrRef() {
method randomHeight (line 157) | func (s *Skiplist) randomHeight() int {
method getNext (line 165) | func (s *Skiplist) getNext(nd *node, height int) *node {
method findNear (line 175) | func (s *Skiplist) findNear(key []byte, less bool, allowEqual bool) (*...
method findSpliceForLevel (line 247) | func (s *Skiplist) findSpliceForLevel(key []byte, before *node, level ...
method getHeight (line 268) | func (s *Skiplist) getHeight() int32 {
method Put (line 273) | func (s *Skiplist) Put(key []byte, v y.ValueStruct) {
method Empty (line 338) | func (s *Skiplist) Empty() bool {
method findLast (line 344) | func (s *Skiplist) findLast() *node {
method Get (line 365) | func (s *Skiplist) Get(key []byte) y.ValueStruct {
method NewIterator (line 383) | func (s *Skiplist) NewIterator() *Iterator {
method MemSize (line 390) | func (s *Skiplist) MemSize() int64 { return s.arena.size() }
method NewUniIterator (line 467) | func (s *Skiplist) NewUniIterator(reversed bool) *UniIterator {
function newNode (line 96) | func newNode(arena *Arena, key []byte, v y.ValueStruct, height int) *node {
function encodeValue (line 107) | func encodeValue(valOffset uint32, valSize uint32) uint64 {
function decodeValue (line 111) | func decodeValue(value uint64) (valOffset uint32, valSize uint32) {
function NewSkiplist (line 118) | func NewSkiplist(arenaSize int64) *Skiplist {
type Iterator (line 394) | type Iterator struct
method Close (line 400) | func (s *Iterator) Close() error {
method Valid (line 406) | func (s *Iterator) Valid() bool { return s.n != nil }
method Key (line 409) | func (s *Iterator) Key() []byte {
method Value (line 414) | func (s *Iterator) Value() y.ValueStruct {
method ValueUint64 (line 420) | func (s *Iterator) ValueUint64() uint64 {
method Next (line 425) | func (s *Iterator) Next() {
method Prev (line 431) | func (s *Iterator) Prev() {
method Seek (line 437) | func (s *Iterator) Seek(target []byte) {
method SeekForPrev (line 442) | func (s *Iterator) SeekForPrev(target []byte) {
method SeekToFirst (line 448) | func (s *Iterator) SeekToFirst() {
method SeekToLast (line 454) | func (s *Iterator) SeekToLast() {
type UniIterator (line 461) | type UniIterator struct
method Next (line 475) | func (s *UniIterator) Next() {
method Rewind (line 484) | func (s *UniIterator) Rewind() {
method Seek (line 493) | func (s *UniIterator) Seek(key []byte) {
method Key (line 502) | func (s *UniIterator) Key() []byte { return s.iter.Key() }
method Value (line 505) | func (s *UniIterator) Value() y.ValueStruct { return s.iter.Value() }
method Valid (line 508) | func (s *UniIterator) Valid() bool { return s.iter.Valid() }
method Close (line 511) | func (s *UniIterator) Close() error { return s.iter.Close() }
FILE: skl/skl_test.go
constant arenaSize (line 23) | arenaSize = 1 << 20
method valid (line 25) | func (s *Skiplist) valid() bool { return s.arena != nil }
function newValue (line 27) | func newValue(v int) []byte {
function length (line 32) | func length(s *Skiplist) int {
function TestEmpty (line 42) | func TestEmpty(t *testing.T) {
function TestBasic (line 77) | func TestBasic(t *testing.T) {
function TestConcurrentBasic (line 121) | func TestConcurrentBasic(t *testing.T) {
function TestConcurrentBasicBigValues (line 151) | func TestConcurrentBasicBigValues(t *testing.T) {
function TestOneKey (line 186) | func TestOneKey(t *testing.T) {
function TestFindNear (line 221) | func TestFindNear(t *testing.T) {
function TestIteratorNext (line 327) | func TestIteratorNext(t *testing.T) {
function TestIteratorPrev (line 351) | func TestIteratorPrev(t *testing.T) {
function TestIteratorSeek (line 375) | func TestIteratorSeek(t *testing.T) {
function randomKey (line 440) | func randomKey(rng *rand.Rand) []byte {
function BenchmarkReadWrite (line 451) | func BenchmarkReadWrite(b *testing.B) {
function BenchmarkReadWriteMap (line 479) | func BenchmarkReadWriteMap(b *testing.B) {
function BenchmarkWrite (line 509) | func BenchmarkWrite(b *testing.B) {
FILE: stream.go
constant batchSize (line 24) | batchSize = 16 << 20
type Stream (line 36) | type Stream struct
method SendDoneMarkers (line 104) | func (st *Stream) SendDoneMarkers(done bool) {
method ToList (line 110) | func (st *Stream) ToList(key []byte, itr *Iterator) (*pb.KVList, error) {
method produceRanges (line 153) | func (st *Stream) produceRanges(ctx context.Context) {
method produceKVs (line 173) | func (st *Stream) produceKVs(ctx context.Context, threadId int) error {
method streamKVs (line 313) | func (st *Stream) streamKVs(ctx context.Context) error {
method Orchestrate (line 415) | func (st *Stream) Orchestrate(ctx context.Context) error {
method newStream (line 479) | func (db *DB) newStream() *Stream {
method NewStream (line 489) | func (db *DB) NewStream() *Stream {
method NewStreamAt (line 497) | func (db *DB) NewStreamAt(readTs uint64) *Stream {
function BufferToKVList (line 506) | func BufferToKVList(buf *z.Buffer) (*pb.KVList, error) {
function KVToBuffer (line 519) | func KVToBuffer(kv *pb.KV, buf *z.Buffer) {
FILE: stream_test.go
function keyWithPrefix (line 25) | func keyWithPrefix(prefix string, k int) []byte {
function keyToInt (line 29) | func keyToInt(k []byte) (string, int) {
function value (line 36) | func value(k int) []byte {
type collector (line 40) | type collector struct
method Send (line 44) | func (c *collector) Send(buf *z.Buffer) error {
function TestStream (line 61) | func TestStream(t *testing.T) {
function TestStreamMaxSize (line 163) | func TestStreamMaxSize(t *testing.T) {
function TestStreamWithThreadId (line 224) | func TestStreamWithThreadId(t *testing.T) {
function TestBigStream (line 270) | func TestBigStream(t *testing.T) {
function TestStreamCustomKeyToList (line 327) | func TestStreamCustomKeyToList(t *testing.T) {
FILE: stream_writer.go
type StreamWriter (line 33) | type StreamWriter struct
method Prepare (line 61) | func (sw *StreamWriter) Prepare() error {
method PrepareIncremental (line 74) | func (sw *StreamWriter) PrepareIncremental() error {
method Write (line 131) | func (sw *StreamWriter) Write(buf *z.Buffer) error {
method Flush (line 252) | func (sw *StreamWriter) Flush() error {
method Cancel (line 314) | func (sw *StreamWriter) Cancel() {
method newWriter (line 353) | func (sw *StreamWriter) newWriter(streamID uint32) (*sortedWriter, err...
method NewStreamWriter (line 47) | func (db *DB) NewStreamWriter() *StreamWriter {
type sortedWriter (line 339) | type sortedWriter struct
method handleRequests (line 373) | func (w *sortedWriter) handleRequests() {
method Add (line 417) | func (w *sortedWriter) Add(key []byte, vs y.ValueStruct) error {
method send (line 442) | func (w *sortedWriter) send(done bool) error {
method Done (line 463) | func (w *sortedWriter) Done() error {
method createTable (line 474) | func (w *sortedWriter) createTable(builder *table.Builder) error {
FILE: stream_writer_test.go
function getSortedKVList (line 24) | func getSortedKVList(valueSize, listSize int) *z.Buffer {
function TestStreamWriter1 (line 42) | func TestStreamWriter1(t *testing.T) {
function TestStreamWriter2 (line 93) | func TestStreamWriter2(t *testing.T) {
function TestStreamWriter3 (line 155) | func TestStreamWriter3(t *testing.T) {
function TestStreamWriter4 (line 251) | func TestStreamWriter4(t *testing.T) {
function TestStreamWriter5 (line 278) | func TestStreamWriter5(t *testing.T) {
function TestStreamWriter6 (line 316) | func TestStreamWriter6(t *testing.T) {
function TestStreamWriterCancel (line 362) | func TestStreamWriterCancel(t *testing.T) {
function TestStreamDone (line 394) | func TestStreamDone(t *testing.T) {
function TestSendOnClosedStream (line 428) | func TestSendOnClosedStream(t *testing.T) {
function TestSendOnClosedStream2 (line 478) | func TestSendOnClosedStream2(t *testing.T) {
function TestStreamWriterEncrypted (line 525) | func TestStreamWriterEncrypted(t *testing.T) {
function TestStreamWriterWithLargeValue (line 571) | func TestStreamWriterWithLargeValue(t *testing.T) {
function TestStreamWriterIncremental (line 593) | func TestStreamWriterIncremental(t *testing.T) {
FILE: structs.go
type valuePointer (line 15) | type valuePointer struct
method Less (line 23) | func (p valuePointer) Less(o valuePointer) bool {
method IsZero (line 33) | func (p valuePointer) IsZero() bool {
method Encode (line 38) | func (p valuePointer) Encode() []byte {
method Decode (line 46) | func (p *valuePointer) Decode(b []byte) {
constant vptrSize (line 21) | vptrSize = unsafe.Sizeof(valuePointer{})
type header (line 54) | type header struct
method Encode (line 74) | func (h header) Encode(out []byte) int {
method Decode (line 85) | func (h *header) Decode(buf []byte) int {
method DecodeFrom (line 100) | func (h *header) DecodeFrom(reader *hashReader) (int, error) {
constant maxHeaderSize (line 65) | maxHeaderSize = 22
type Entry (line 129) | type Entry struct
method isZero (line 143) | func (e *Entry) isZero() bool {
method estimateSizeAndSetThreshold (line 147) | func (e *Entry) estimateSizeAndSetThreshold(threshold int64) int64 {
method skipVlogAndSetThreshold (line 159) | func (e *Entry) skipVlogAndSetThreshold(threshold int64) bool {
method print (line 167) | func (e Entry) print(prefix string) {
method WithMeta (line 187) | func (e *Entry) WithMeta(meta byte) *Entry {
method WithDiscard (line 198) | func (e *Entry) WithDiscard() *Entry {
method WithTTL (line 205) | func (e *Entry) WithTTL(dur time.Duration) *Entry {
method withMergeBit (line 212) | func (e *Entry) withMergeBit() *Entry {
function NewEntry (line 177) | func NewEntry(key, value []byte) *Entry {
FILE: structs_test.go
function TestLargeEncode (line 17) | func TestLargeEncode(t *testing.T) {
function TestNumFieldsHeader (line 23) | func TestNumFieldsHeader(t *testing.T) {
FILE: table/builder.go
constant KB (line 29) | KB = 1024
constant MB (line 30) | MB = KB * 1024
constant padding (line 34) | padding = 256
type header (line 37) | type header struct
method Encode (line 45) | func (h header) Encode() []byte {
method Decode (line 52) | func (h *header) Decode(buf []byte) {
constant headerSize (line 42) | headerSize = uint16(unsafe.Sizeof(header{}))
type bblock (line 60) | type bblock struct
type Builder (line 68) | type Builder struct
method allocate (line 88) | func (b *Builder) allocate(need int) []byte {
method append (line 109) | func (b *Builder) append(data []byte) {
method handleBlock (line 158) | func (b *Builder) handleBlock() {
method Close (line 191) | func (b *Builder) Close() {
method Empty (line 196) | func (b *Builder) Empty() bool { return len(b.keyHashes) == 0 }
method keyDiff (line 199) | func (b *Builder) keyDiff(newKey []byte) []byte {
method addHelper (line 209) | func (b *Builder) addHelper(key []byte, v y.ValueStruct, vpLen uint32) {
method finishBlock (line 262) | func (b *Builder) finishBlock() {
method shouldFinishBlock (line 294) | func (b *Builder) shouldFinishBlock(key []byte, value y.ValueStruct) b...
method AddStaleKey (line 325) | func (b *Builder) AddStaleKey(key []byte, v y.ValueStruct, valueLen ui...
method Add (line 332) | func (b *Builder) Add(key []byte, value y.ValueStruct, valueLen uint32) {
method addInternal (line 336) | func (b *Builder) addInternal(key []byte, value y.ValueStruct, valueLe...
method ReachedCapacity (line 358) | func (b *Builder) ReachedCapacity() bool {
method Finish (line 389) | func (b *Builder) Finish() []byte {
method Done (line 418) | func (b *Builder) Done() buildData {
method calculateChecksum (line 454) | func (b *Builder) calculateChecksum(data []byte) []byte {
method DataKey (line 477) | func (b *Builder) DataKey() *pb.DataKey {
method Opts (line 481) | func (b *Builder) Opts() *Options {
method encrypt (line 487) | func (b *Builder) encrypt(data []byte) ([]byte, error) {
method shouldEncrypt (line 505) | func (b *Builder) shouldEncrypt() bool {
method compressData (line 510) | func (b *Builder) compressData(data []byte) ([]byte, error) {
method buildIndex (line 526) | func (b *Builder) buildIndex(bloom []byte) ([]byte, uint32) {
method writeBlockOffsets (line 564) | func (b *Builder) writeBlockOffsets(builder *fbs.Builder) ([]fbs.UOffs...
method writeBlockOffset (line 577) | func (b *Builder) writeBlockOffset(
constant maxAllocatorInitialSz (line 114) | maxAllocatorInitialSz = 256 << 20
function NewTableBuilder (line 117) | func NewTableBuilder(opts Options) *Builder {
function maxEncodedLen (line 148) | func maxEncodedLen(ctype options.CompressionType, sz int) int {
type buildData (line 397) | type buildData struct
method Copy (line 405) | func (bd *buildData) Copy(dst []byte) int {
FILE: table/builder_test.go
function TestTableIndex (line 24) | func TestTableIndex(t *testing.T) {
function TestInvalidCompression (line 131) | func TestInvalidCompression(t *testing.T) {
function BenchmarkBuilder (line 149) | func BenchmarkBuilder(b *testing.B) {
function TestBloomfilter (line 226) | func TestBloomfilter(t *testing.T) {
function TestEmptyBuilder (line 272) | func TestEmptyBuilder(t *testing.T) {
FILE: table/iterator.go
type blockIterator (line 18) | type blockIterator struct
method setBlock (line 35) | func (itr *blockIterator) setBlock(b *Block) {
method setIdx (line 53) | func (itr *blockIterator) setIdx(i int) {
method Valid (line 106) | func (itr *blockIterator) Valid() bool {
method Error (line 110) | func (itr *blockIterator) Error() error {
method Close (line 114) | func (itr *blockIterator) Close() {
method seek (line 124) | func (itr *blockIterator) seek(key []byte, whence int) {
method seekToFirst (line 147) | func (itr *blockIterator) seekToFirst() {
method seekToLast (line 152) | func (itr *blockIterator) seekToLast() {
method next (line 156) | func (itr *blockIterator) next() {
method prev (line 160) | func (itr *blockIterator) prev() {
type Iterator (line 165) | type Iterator struct
method Close (line 184) | func (itr *Iterator) Close() error {
method reset (line 189) | func (itr *Iterator) reset() {
method Valid (line 195) | func (itr *Iterator) Valid() bool {
method useCache (line 199) | func (itr *Iterator) useCache() bool {
method seekToFirst (line 203) | func (itr *Iterator) seekToFirst() {
method seekToLast (line 222) | func (itr *Iterator) seekToLast() {
method seekHelper (line 241) | func (itr *Iterator) seekHelper(blockIdx int, key []byte) {
method seekFrom (line 256) | func (itr *Iterator) seekFrom(key []byte, whence int) {
method seek (line 298) | func (itr *Iterator) seek(key []byte) {
method seekForPrev (line 303) | func (itr *Iterator) seekForPrev(key []byte) {
method next (line 311) | func (itr *Iterator) next() {
method prev (line 342) | func (itr *Iterator) prev() {
method Key (line 374) | func (itr *Iterator) Key() []byte {
method Value (line 379) | func (itr *Iterator) Value() (ret y.ValueStruct) {
method ValueCopy (line 386) | func (itr *Iterator) ValueCopy() (ret y.ValueStruct) {
method Next (line 393) | func (itr *Iterator) Next() {
method Rewind (line 402) | func (itr *Iterator) Rewind() {
method Seek (line 411) | func (itr *Iterator) Seek(key []byte) {
method NewIterator (line 177) | func (t *Table) NewIterator(opt int) *Iterator {
type ConcatIterator (line 426) | type ConcatIterator struct
method setIdx (line 453) | func (s *ConcatIterator) setIdx(idx int) {
method Rewind (line 466) | func (s *ConcatIterator) Rewind() {
method Valid (line 479) | func (s *ConcatIterator) Valid() bool {
method Key (line 484) | func (s *ConcatIterator) Key() []byte {
method Value (line 489) | func (s *ConcatIterator) Value() y.ValueStruct {
method Seek (line 494) | func (s *ConcatIterator) Seek(key []byte) {
method Next (line 517) | func (s *ConcatIterator) Next() {
method Close (line 541) | func (s *ConcatIterator) Close() error {
function NewConcatIterator (line 435) | func NewConcatIterator(tbls []*Table, opt int) *ConcatIterator {
FILE: table/merge_iterator.go
type MergeIterator (line 16) | type MergeIterator struct
method fix (line 87) | func (mi *MergeIterator) fix() {
method bigger (line 122) | func (mi *MergeIterator) bigger() *node {
method swapSmall (line 129) | func (mi *MergeIterator) swapSmall() {
method Next (line 141) | func (mi *MergeIterator) Next() {
method setCurrent (line 152) | func (mi *MergeIterator) setCurrent() {
method Rewind (line 157) | func (mi *MergeIterator) Rewind() {
method Seek (line 165) | func (mi *MergeIterator) Seek(key []byte) {
method Valid (line 173) | func (mi *MergeIterator) Valid() bool {
method Key (line 178) | func (mi *MergeIterator) Key() []byte {
method Value (line 183) | func (mi *MergeIterator) Value() y.ValueStruct {
method Close (line 188) | func (mi *MergeIterator) Close() error {
type node (line 25) | type node struct
method setIterator (line 37) | func (n *node) setIterator(iter y.Iterator) {
method setKey (line 45) | func (n *node) setKey() {
method next (line 65) | func (n *node) next() {
method rewind (line 77) | func (n *node) rewind() {
method seek (line 82) | func (n *node) seek(key []byte) {
function NewMergeIterator (line 198) | func NewMergeIterator(iters []y.Iterator, reverse bool) y.Iterator {
FILE: table/merge_iterator_test.go
type SimpleIterator (line 17) | type SimpleIterator struct
method Close (line 28) | func (s *SimpleIterator) Close() error { closeCount++; return nil }
method Next (line 30) | func (s *SimpleIterator) Next() {
method Rewind (line 38) | func (s *SimpleIterator) Rewind() {
method Seek (line 46) | func (s *SimpleIterator) Seek(key []byte) {
method Key (line 60) | func (s *SimpleIterator) Key() []byte { return s.keys[s.idx] }
method Value (line 61) | func (s *SimpleIterator) Value() y.ValueStruct {
method Valid (line 68) | func (s *SimpleIterator) Valid() bool {
function newSimpleIterator (line 74) | func newSimpleIterator(keys []string, vals []string, reversed bool) *Sim...
function getAll (line 90) | func getAll(it y.Iterator) ([]string, []string) {
function closeAndCheck (line 101) | func closeAndCheck(t *testing.T, it y.Iterator, expected int) {
function TestSimpleIterator (line 107) | func TestSimpleIterator(t *testing.T) {
function reversed (line 119) | func reversed(a []string) []string {
function TestMergeSingle (line 127) | func TestMergeSingle(t *testing.T) {
function TestMergeSingleReversed (line 139) | func TestMergeSingleReversed(t *testing.T) {
function TestMergeMore (line 151) | func TestMergeMore(t *testing.T) {
function TestMergeIteratorNested (line 215) | func TestMergeIteratorNested(t *testing.T) {
function TestMergeIteratorSeek (line 228) | func TestMergeIteratorSeek(t *testing.T) {
function TestMergeIteratorSeekReversed (line 241) | func TestMergeIteratorSeekReversed(t *testing.T) {
function TestMergeIteratorSeekInvalid (line 254) | func TestMergeIteratorSeekInvalid(t *testing.T) {
function TestMergeIteratorSeekInvalidReversed (line 265) | func TestMergeIteratorSeekInvalidReversed(t *testing.T) {
function TestMergeIteratorDuplicate (line 276) | func TestMergeIteratorDuplicate(t *testing.T) {
function TestMergeDuplicates (line 330) | func TestMergeDuplicates(t *testing.T) {
FILE: table/table.go
constant fileSuffix (line 36) | fileSuffix = ".sst"
constant intSize (line 37) | intSize = int(unsafe.Sizeof(int(0)))
type Options (line 40) | type Options struct
type TableInterface (line 79) | type TableInterface interface
type Table (line 87) | type Table struct
method cheapIndex (line 120) | func (t *Table) cheapIndex() *cheapIndex {
method offsetsLength (line 123) | func (t *Table) offsetsLength() int { return t.cheapIndex().OffsetsLen...
method MaxVersion (line 126) | func (t *Table) MaxVersion() uint64 { return t.cheapIndex().MaxVersion }
method BloomFilterSize (line 129) | func (t *Table) BloomFilterSize() int { return t.cheapIndex().BloomFil...
method UncompressedSize (line 132) | func (t *Table) UncompressedSize() uint32 { return t.cheapIndex().Unco...
method KeyCount (line 135) | func (t *Table) KeyCount() uint32 { return t.cheapIndex().KeyCount }
method OnDiskSize (line 139) | func (t *Table) OnDiskSize() uint32 { return t.cheapIndex().OnDiskSize }
method CompressionType (line 142) | func (t *Table) CompressionType() options.CompressionType {
method IncrRef (line 147) | func (t *Table) IncrRef() {
method DecrRef (line 152) | func (t *Table) DecrRef() error {
method initBiggestAndSmallest (line 334) | func (t *Table) initBiggestAndSmallest() error {
method read (line 407) | func (t *Table) read(off, sz int) ([]byte, error) {
method readNoFail (line 411) | func (t *Table) readNoFail(off, sz int) []byte {
method initIndex (line 419) | func (t *Table) initIndex() (*fb.BlockOffset, error) {
method KeySplits (line 477) | func (t *Table) KeySplits(n int, prefix []byte) []string {
method fetchIndex (line 502) | func (t *Table) fetchIndex() *fb.TableIndex {
method offsets (line 520) | func (t *Table) offsets(ko *fb.BlockOffset, i int) bool {
method block (line 527) | func (t *Table) block(idx int, useCache bool) (*Block, error) {
method blockCacheKey (line 627) | func (t *Table) blockCacheKey(idx int) []byte {
method indexKey (line 640) | func (t *Table) indexKey() uint64 {
method IndexSize (line 645) | func (t *Table) IndexSize() int {
method Size (line 650) | func (t *Table) Size() int64 { return int64(t.tableSize) }
method StaleDataSize (line 653) | func (t *Table) StaleDataSize() uint32 { return t.fetchIndex().StaleDa...
method Smallest (line 656) | func (t *Table) Smallest() []byte { return t.smallest }
method Biggest (line 659) | func (t *Table) Biggest() []byte { return t.biggest }
method Filename (line 662) | func (t *Table) Filename() string { return t.Fd.Name() }
method ID (line 665) | func (t *Table) ID() uint64 { return t.id }
method DoesNotHave (line 669) | func (t *Table) DoesNotHave(hash uint32) bool {
method readTableIndex (line 685) | func (t *Table) readTableIndex() (*fb.TableIndex, error) {
method VerifyChecksum (line 700) | func (t *Table) VerifyChecksum() error {
method shouldDecrypt (line 725) | func (t *Table) shouldDecrypt() bool {
method KeyID (line 730) | func (t *Table) KeyID() uint64 {
method decrypt (line 739) | func (t *Table) decrypt(data []byte, viaCalloc bool) ([]byte, error) {
method decompress (line 785) | func (t *Table) decompress(b *Block) error {
type cheapIndex (line 111) | type cheapIndex struct
function BlockEvictHandler (line 170) | func BlockEvictHandler(b *Block) {
type Block (line 174) | type Block struct
method incrRef (line 189) | func (b *Block) incrRef() bool {
method decrRef (line 212) | func (b *Block) decrRef() {
method size (line 232) | func (b *Block) size() int64 {
method verifyCheckSum (line 237) | func (b *Block) verifyCheckSum() error {
function CreateTable (line 245) | func CreateTable(fname string, builder *Builder) (*Table, error) {
function OpenTable (line 268) | func OpenTable(mf *z.MmapFile, opts Options) (*Table, error) {
function OpenInMemoryTable (line 313) | func OpenInMemoryTable(data []byte, id uint64, opt *Options) (*Table, er...
function ParseFileID (line 758) | func ParseFileID(name string) (uint64, bool) {
function IDToFilename (line 774) | func IDToFilename(id uint64) string {
function NewFilename (line 780) | func NewFilename(id uint64, dir string) string {
FILE: table/table_test.go
function key (line 29) | func key(prefix string, i int) string {
function getTestTableOptions (line 33) | func getTestTableOptions() Options {
function buildTestTable (line 42) | func buildTestTable(t *testing.T, prefix string, n int, opts Options) *T...
function buildTable (line 57) | func buildTable(t *testing.T, keyValues [][]string, opts Options) *Table {
function TestTableIterator (line 77) | func TestTableIterator(t *testing.T) {
function TestSeekToFirst (line 98) | func TestSeekToFirst(t *testing.T) {
function TestSeekToLast (line 115) | func TestSeekToLast(t *testing.T) {
function TestSeek (line 137) | func TestSeek(t *testing.T) {
function TestSeekForPrev (line 171) | func TestSeekForPrev(t *testing.T) {
function TestIterateFromStart (line 205) | func TestIterateFromStart(t *testing.T) {
function TestIterateFromEnd (line 231) | func TestIterateFromEnd(t *testing.T) {
function TestTable (line 256) | func TestTable(t *testing.T) {
function TestIterateBackAndForth (line 282) | func TestIterateBackAndForth(t *testing.T) {
function TestUniIterator (line 322) | func TestUniIterator(t *testing.T) {
function TestConcatIteratorOneTable (line 353) | func TestConcatIteratorOneTable(t *testing.T) {
function TestConcatIterator (line 373) | func TestConcatIterator(t *testing.T) {
function TestMergingIterator (line 448) | func TestMergingIterator(t *testing.T) {
function TestMergingIteratorReversed (line 493) | func TestMergingIteratorReversed(t *testing.T) {
function TestMergingIteratorTakeOne (line 542) | func TestMergingIteratorTakeOne(t *testing.T) {
function TestMergingIteratorTakeTwo (line 585) | func TestMergingIteratorTakeTwo(t *testing.T) {
function TestTableBigValues (line 629) | func TestTableBigValues(t *testing.T) {
function TestTableChecksum (line 667) | func TestTableChecksum(t *testing.T) {
function BenchmarkRead (line 704) | func BenchmarkRead(b *testing.B) {
function BenchmarkReadAndBuild (line 721) | func BenchmarkReadAndBuild(b *testing.B) {
function BenchmarkReadMerged (line 746) | func BenchmarkReadMerged(b *testing.B) {
function BenchmarkChecksum (line 791) | func BenchmarkChecksum(b *testing.B) {
function BenchmarkRandomRead (line 813) | func BenchmarkRandomRead(b *testing.B) {
function getTableForBenchmarks (line 840) | func getTableForBenchmarks(b *testing.B, count int, cache *ristretto.Cac...
function TestMain (line 863) | func TestMain(m *testing.M) {
function TestDoesNotHaveRace (line 869) | func TestDoesNotHaveRace(t *testing.T) {
function TestMaxVersion (line 885) | func TestMaxVersion(t *testing.T) {
FILE: test_extensions.go
constant updateDiscardStatsMsg (line 21) | updateDiscardStatsMsg = "updateDiscardStats iteration done"
constant endVLogInitMsg (line 22) | endVLogInitMsg = "End: vlog.init(db)"
type testOnlyOptions (line 27) | type testOnlyOptions struct
type testOnlyDBExtensions (line 36) | type testOnlyDBExtensions struct
method logToSyncChan (line 50) | func (db *DB) logToSyncChan(msg string) {
method captureDiscardStats (line 60) | func (db *DB) captureDiscardStats() {
FILE: trie/trie.go
type node (line 17) | type node struct
method isEmpty (line 23) | func (n *node) isEmpty() bool {
function newNode (line 27) | func newNode() *node {
type Trie (line 35) | type Trie struct
method Add (line 96) | func (t *Trie) Add(prefix []byte, id uint64) {
method AddMatch (line 110) | func (t *Trie) AddMatch(m pb.Match, id uint64) error {
method fix (line 119) | func (t *Trie) fix(m pb.Match, id uint64, op int) error {
method Get (line 173) | func (t *Trie) Get(key []byte) map[uint64]struct{} {
method get (line 178) | func (t *Trie) get(curNode *node, key []byte) map[uint64]struct{} {
method Delete (line 226) | func (t *Trie) Delete(prefix []byte, id uint64) error {
method DeleteMatch (line 230) | func (t *Trie) DeleteMatch(m pb.Match, id uint64) error {
function NewTrie (line 40) | func NewTrie() *Trie {
function parseIgnoreBytes (line 48) | func parseIgnoreBytes(ig string) ([]bool, error) {
constant set (line 115) | set = iota
constant del (line 116) | del
function removeEmpty (line 208) | func removeEmpty(curNode *node) bool {
function numNodes (line 240) | func numNodes(curNode *node) int {
FILE: trie/trie_test.go
function TestGet (line 17) | func TestGet(t *testing.T) {
function TestTrieDelete (line 45) | func TestTrieDelete(t *testing.T) {
function TestParseIgnoreBytes (line 79) | func TestParseIgnoreBytes(t *testing.T) {
function TestPrefixMatchWithHoles (line 93) | func TestPrefixMatchWithHoles(t *testing.T) {
FILE: txn.go
type oracle (line 24) | type oracle struct
method Stop (line 74) | func (o *oracle) Stop() {
method readTs (line 78) | func (o *oracle) readTs() uint64 {
method nextTs (line 97) | func (o *oracle) nextTs() uint64 {
method incrementNextTs (line 103) | func (o *oracle) incrementNextTs() {
method setDiscardTs (line 111) | func (o *oracle) setDiscardTs(ts uint64) {
method discardAtOrBelow (line 118) | func (o *oracle) discardAtOrBelow() uint64 {
method hasConflict (line 128) | func (o *oracle) hasConflict(txn *Txn) bool {
method newCommitTs (line 153) | func (o *oracle) newCommitTs(txn *Txn) (uint64, bool) {
method doneRead (line 190) | func (o *oracle) doneRead(txn *Txn) {
method cleanupCommittedTransactions (line 197) | func (o *oracle) cleanupCommittedTransactions() { // Must be called un...
method doneCommit (line 230) | func (o *oracle) doneCommit(cts uint64) {
type committedTxn (line 51) | type committedTxn struct
function newOracle (line 57) | func newOracle(opt Options) *oracle {
type Txn (line 239) | type Txn struct
method newPendingWritesIterator (line 312) | func (txn *Txn) newPendingWritesIterator(reversed bool) *pendingWrites...
method checkSize (line 335) | func (txn *Txn) checkSize(e *Entry) error {
method modify (line 351) | func (txn *Txn) modify(e *Entry) error {
method Set (line 403) | func (txn *Txn) Set(key, val []byte) error {
method SetEntry (line 412) | func (txn *Txn) SetEntry(e *Entry) error {
method Delete (line 424) | func (txn *Txn) Delete(key []byte) error {
method Get (line 434) | func (txn *Txn) Get(key []byte) (item *Item, rerr error) {
method addReadKey (line 489) | func (txn *Txn) addReadKey(key []byte) {
method Discard (line 508) | func (txn *Txn) Discard() {
method commitAndSend (line 521) | func (txn *Txn) commitAndSend() (func() error, error) {
method commitPrecheck (line 609) | func (txn *Txn) commitPrecheck() error {
method Commit (line 649) | func (txn *Txn) Commit() error {
method CommitWith (line 700) | func (txn *Txn) CommitWith(cb func(error)) {
method ReadTs (line 733) | func (txn *Txn) ReadTs() uint64 {
type pendingWritesIterator (line 260) | type pendingWritesIterator struct
method Next (line 267) | func (pi *pendingWritesIterator) Next() {
method Rewind (line 271) | func (pi *pendingWritesIterator) Rewind() {
method Seek (line 275) | func (pi *pendingWritesIterator) Seek(key []byte) {
method Key (line 286) | func (pi *pendingWritesIterator) Key() []byte {
method Value (line 292) | func (pi *pendingWritesIterator) Value() y.ValueStruct {
method Valid (line 304) | func (pi *pendingWritesIterator) Valid() bool {
method Close (line 308) | func (pi *pendingWritesIterator) Close() error {
function exceedsSize (line 346) | func exceedsSize(prefix string, max int64, key []byte) error {
type txnCb (line 674) | type txnCb struct
function runTxnCallback (line 680) | func runTxnCallback(cb *txnCb) {
method NewTransaction (line 757) | func (db *DB) NewTransaction(update bool) *Txn {
method newTransaction (line 761) | func (db *DB) newTransaction(update, isManaged bool) *Txn {
method View (line 788) | func (db *DB) View(fn func(txn *Txn) error) error {
method Update (line 806) | func (db *DB) Update(fn func(txn *Txn) error) error {
FILE: txn_test.go
function TestTxnSimple (line 24) | func TestTxnSimple(t *testing.T) {
function TestTxnReadAfterWrite (line 47) | func TestTxnReadAfterWrite(t *testing.T) {
function TestTxnCommitAsync (line 88) | func TestTxnCommitAsync(t *testing.T) {
function TestTxnVersions (line 154) | func TestTxnVersions(t *testing.T) {
function TestTxnWriteSkew (line 258) | func TestTxnWriteSkew(t *testing.T) {
function TestTxnIterationEdgeCase (line 326) | func TestTxnIterationEdgeCase(t *testing.T) {
function TestTxnIterationEdgeCase2 (line 416) | func TestTxnIterationEdgeCase2(t *testing.T) {
function TestTxnIterationEdgeCase3 (line 509) | func TestTxnIterationEdgeCase3(t *testing.T) {
function TestIteratorAllVersionsWithDeleted (line 619) | func TestIteratorAllVersionsWithDeleted(t *testing.T) {
function TestIteratorAllVersionsWithDeleted2 (line 682) | func TestIteratorAllVersionsWithDeleted2(t *testing.T) {
function TestManagedDB (line 724) | func TestManagedDB(t *testing.T) {
function TestArmV7Issue311Fix (line 838) | func TestArmV7Issue311Fix(t *testing.T) {
function TestConflict (line 871) | func TestConflict(t *testing.T) {
FILE: util.go
method validate (line 19) | func (s *levelsController) validate() error {
method validate (line 29) | func (s *levelHandler) validate() error {
method reserveFileID (line 81) | func (s *levelsController) reserveFileID() uint64 {
function getIDMap (line 86) | func getIDMap(dir string) map[uint64]struct{} {
function init (line 103) | func init() {
FILE: value.go
constant bitDelete (line 38) | bitDelete byte = 1 << 0
constant bitValuePointer (line 39) | bitValuePointer byte = 1 << 1
constant bitDiscardEarlierVersions (line 40) | bitDiscardEarlierVersions byte = 1 << 2
constant bitMergeEntry (line 42) | bitMergeEntry byte = 1 << 3
constant bitTxn (line 44) | bitTxn byte = 1 << 6
constant bitFinTxn (line 45) | bitFinTxn byte = 1 << 7
constant mi (line 47) | mi int64 = 1 << 20
constant vlogHeaderSize (line 53) | vlogHeaderSize = 20
type logEntry (line 59) | type logEntry
type safeRead (line 61) | type safeRead struct
method Entry (line 109) | func (r *safeRead) Entry(reader io.Reader) (*Entry, error) {
type hashReader (line 71) | type hashReader struct
method Read (line 86) | func (t *hashReader) Read(p []byte) (int, error) {
method ReadByte (line 96) | func (t *hashReader) ReadByte() (byte, error) {
method Sum32 (line 103) | func (t *hashReader) Sum32() uint32 {
function newHashReader (line 77) | func newHashReader(r io.Reader) *hashReader {
method valueThreshold (line 430) | func (db *DB) valueThreshold() int64 {
type valueLog (line 434) | type valueLog struct
method rewrite (line 162) | func (vlog *valueLog) rewrite(f *logFile) error {
method incrIteratorCount (line 355) | func (vlog *valueLog) incrIteratorCount() {
method iteratorCount (line 359) | func (vlog *valueLog) iteratorCount() int {
method decrIteratorCount (line 363) | func (vlog *valueLog) decrIteratorCount() error {
method deleteLogFile (line 386) | func (vlog *valueLog) deleteLogFile(lf *logFile) error {
method dropAll (line 398) | func (vlog *valueLog) dropAll() (int, error) {
method fpath (line 458) | func (vlog *valueLog) fpath(fid uint32) string {
method populateFilesMap (line 462) | func (vlog *valueLog) populateFilesMap() error {
method createVlogFile (line 498) | func (vlog *valueLog) createVlogFile() (*logFile, error) {
method init (line 533) | func (vlog *valueLog) init(db *DB) {
method open (line 551) | func (vlog *valueLog) open(db *DB) error {
method Close (line 616) | func (vlog *valueLog) Close() error {
method sortedFids (line 645) | func (vlog *valueLog) sortedFids() []uint32 {
method sync (line 719) | func (vlog *valueLog) sync() error {
method woffset (line 742) | func (vlog *valueLog) woffset() uint32 {
method validateWrites (line 749) | func (vlog *valueLog) validateWrites(reqs []*request) error {
method write (line 782) | func (vlog *valueLog) write(reqs []*request) error {
method getFileRLocked (line 902) | func (vlog *valueLog) getFileRLocked(vp valuePointer) (*logFile, error) {
method Read (line 930) | func (vlog *valueLog) Read(vp valuePointer, _ *y.Slice) ([]byte, func(...
method getUnlockCallback (line 971) | func (vlog *valueLog) getUnlockCallback(lf *logFile) func() {
method readValueBytes (line 980) | func (vlog *valueLog) readValueBytes(vp valuePointer) ([]byte, *logFil...
method pickLog (line 992) | func (vlog *valueLog) pickLog(discardRatio float64) *logFile {
method doRunGC (line 1054) | func (vlog *valueLog) doRunGC(lf *logFile) error {
method waitOnGC (line 1066) | func (vlog *valueLog) waitOnGC(lc *z.Closer) {
method runGC (line 1076) | func (vlog *valueLog) runGC(discardRatio float64) error {
method updateDiscardStats (line 1094) | func (vlog *valueLog) updateDiscardStats(stats map[uint32]int64) {
function vlogFilePath (line 454) | func vlogFilePath(dirPath string, fid uint32) string {
function errFile (line 527) | func errFile(err error, path string, msg string) error {
type request (line 662) | type request struct
method reset (line 672) | func (req *request) reset() {
method IncrRef (line 680) | func (req *request) IncrRef() {
method DecrRef (line 684) | func (req *request) DecrRef() {
method Wait (line 693) | func (req *request) Wait() error {
type requests (line 700) | type requests
method DecrRef (line 702) | func (reqs requests) DecrRef() {
method IncrRef (line 708) | func (reqs requests) IncrRef() {
function estimateRequestSize (line 773) | func estimateRequestSize(req *request) uint64 {
function discardEntry (line 1035) | func discardEntry(e Entry, vs y.ValueStruct, db *DB) bool {
type vlogThreshold (line 1106) | type vlogThreshold struct
method Clear (line 1150) | func (v *vlogThreshold) Clear(opt Options) {
method update (line 1155) | func (v *vlogThreshold) update(sizes []int64) {
method close (line 1159) | func (v *vlogThreshold) close() {
method listenForValueThresholdUpdate (line 1163) | func (v *vlogThreshold) listenForValueThresholdUpdate() {
function initVlogThreshold (line 1117) | func initVlogThreshold(opt *Options) *vlogThreshold {
FILE: value_test.go
function TestDynamicValueThreshold (line 25) | func TestDynamicValueThreshold(t *testing.T) {
function TestValueBasic (line 67) | func TestValueBasic(t *testing.T) {
function TestValueGCManaged (line 127) | func TestValueGCManaged(t *testing.T) {
function TestValueGC (line 196) | func TestValueGC(t *testing.T) {
function TestValueGC2 (line 249) | func TestValueGC2(t *testing.T) {
function TestValueGC3 (line 326) | func TestValueGC3(t *testing.T) {
function TestValueGC4 (line 401) | func TestValueGC4(t *testing.T) {
function TestPersistLFDiscardStats (line 475) | func TestPersistLFDiscardStats(t *testing.T) {
function TestValueChecksums (line 544) | func TestValueChecksums(t *testing.T) {
function TestPartialAppendToWAL (line 627) | func TestPartialAppendToWAL(t *testing.T) {
function TestReadOnlyOpenWithPartialAppendToWAL (line 689) | func TestReadOnlyOpenWithPartialAppendToWAL(t *testing.T) {
function TestValueLogTrigger (line 727) | func TestValueLogTrigger(t *testing.T) {
function createMemFile (line 765) | func createMemFile(t *testing.T, entries []*Entry) ([]byte, uint32) {
function TestPenultimateMemCorruption (line 792) | func TestPenultimateMemCorruption(t *testing.T) {
function checkKeys (line 858) | func checkKeys(t *testing.T, kv *DB, keys [][]byte) {
type testHelper (line 871) | type testHelper struct
method key (line 877) | func (th *testHelper) key(i int) []byte {
method value (line 880) | func (th *testHelper) value() []byte {
method writeRange (line 890) | func (th *testHelper) writeRange(from, to int) {
method readRange (line 899) | func (th *testHelper) readRange(from, to int) {
function TestBug578 (line 919) | func TestBug578(t *testing.T) {
function BenchmarkReadWrite (line 951) | func BenchmarkReadWrite(b *testing.B) {
function TestValueLogTruncate (line 1018) | func TestValueLogTruncate(t *testing.T) {
function TestSafeEntry (line 1058) | func TestSafeEntry(t *testing.T) {
function TestValueEntryChecksum (line 1075) | func TestValueEntryChecksum(t *testing.T) {
function TestValidateWrite (line 1155) | func TestValidateWrite(t *testing.T) {
function TestValueLogMeta (line 1216) | func TestValueLogMeta(t *testing.T) {
function TestFirstVlogFile (line 1260) | func TestFirstVlogFile(t *testing.T) {
FILE: watermark_edge_test.go
function TestWaterMarkEdgeCase (line 18) | func TestWaterMarkEdgeCase(t *testing.T) {
function doWork (line 46) | func doWork(db *DB, i int) error {
function generateRandomBytes (line 80) | func generateRandomBytes() []byte {
function getValue (line 88) | func getValue(txn *Txn, key string) {
function setValue (line 96) | func setValue(txn *Txn, key, value string) {
function delay (line 102) | func delay() {
FILE: y/bloom.go
type Filter (line 10) | type Filter
method MayContainKey (line 12) | func (f Filter) MayContainKey(k []byte) bool {
method MayContain (line 18) | func (f Filter) MayContain(h uint32) bool {
function NewFilter (line 45) | func NewFilter(keys []uint32, bitsPerKey int) Filter {
function BloomBitsPerKey (line 51) | func BloomBitsPerKey(numEntries int, fp float64) int {
function appendFilter (line 57) | func appendFilter(buf []byte, keys []uint32, bitsPerKey int) []byte {
function extend (line 95) | func extend(b []byte, n int) (overall, trailer []byte) {
function Hash (line 117) | func Hash(b []byte) uint32 {
FILE: y/bloom_test.go
method String (line 11) | func (f Filter) String() string {
function TestSmallBloomFilter (line 25) | func TestSmallBloomFilter(t *testing.T) {
function TestBloomFilter (line 56) | func TestBloomFilter(t *testing.T) {
function TestHash (line 127) | func TestHash(t *testing.T) {
FILE: y/checksum.go
function CalculateChecksum (line 21) | func CalculateChecksum(data []byte, ct pb.Checksum_Algorithm) uint64 {
function VerifyChecksum (line 33) | func VerifyChecksum(data []byte, expected *pb.Checksum) error {
FILE: y/checksum_test.go
function TestCalculateChecksum_CRC32C (line 12) | func TestCalculateChecksum_CRC32C(t *testing.T) {
function TestCalculateChecksum_XXHash64 (line 24) | func TestCalculateChecksum_XXHash64(t *testing.T) {
function TestVerifyChecksum_Success (line 31) | func TestVerifyChecksum_Success(t *testing.T) {
function TestVerifyChecksum_Mismatch (line 40) | func TestVerifyChecksum_Mismatch(t *testing.T) {
function TestCalculateChecksum_UnsupportedAlgoPanics (line 48) | func TestCalculateChecksum_UnsupportedAlgoPanics(t *testing.T) {
FILE: y/encrypt.go
function XORBlock (line 19) | func XORBlock(dst, src, key, iv []byte) error {
function XORBlockAllocate (line 29) | func XORBlockAllocate(src, key, iv []byte) ([]byte, error) {
function XORBlockStream (line 40) | func XORBlockStream(w io.Writer, src, key, iv []byte) error {
function GenerateIV (line 52) | func GenerateIV() ([]byte, error) {
FILE: y/encrypt_test.go
function TestXORBlock (line 16) | func TestXORBlock(t *testing.T) {
FILE: y/error.go
function Check (line 29) | func Check(err error) {
function Check2 (line 36) | func Check2(_ interface{}, err error) {
function AssertTrue (line 41) | func AssertTrue(b bool) {
function AssertTruef (line 48) | func AssertTruef(b bool, format string, args ...interface{}) {
function Wrap (line 55) | func Wrap(err error, msg string) error {
function Wrapf (line 66) | func Wrapf(err error, format string, args ...interface{}) error {
function CombineErrors (line 70) | func CombineErrors(one, other error) error {
FILE: y/error_test.go
function TestCombineWithBothErrorsPresent (line 15) | func TestCombineWithBothErrorsPresent(t *testing.T) {
function TestCombineErrorsWithOneErrorPresent (line 20) | func TestCombineErrorsWithOneErrorPresent(t *testing.T) {
function TestCombineErrorsWithOtherErrorPresent (line 25) | func TestCombineErrorsWithOtherErrorPresent(t *testing.T) {
function TestCombineErrorsWithBothErrorsAsNil (line 30) | func TestCombineErrorsWithBothErrorsAsNil(t *testing.T) {
FILE: y/file_dsync.go
function init (line 13) | func init() {
FILE: y/file_nodsync.go
function init (line 13) | func init() {
FILE: y/iterator.go
type ValueStruct (line 15) | type ValueStruct struct
method EncodedSize (line 36) | func (v *ValueStruct) EncodedSize() uint32 {
method Decode (line 43) | func (v *ValueStruct) Decode(b []byte) {
method Encode (line 52) | func (v *ValueStruct) Encode(b []byte) uint32 {
method EncodeTo (line 63) | func (v *ValueStruct) EncodeTo(buf *bytes.Buffer) {
function sizeVarint (line 24) | func sizeVarint(x uint64) (n int) {
type Iterator (line 74) | type Iterator interface
FILE: y/metrics.go
constant BADGER_METRIC_PREFIX (line 13) | BADGER_METRIC_PREFIX = "badger_"
function init (line 67) | func init() {
function NumIteratorsCreatedAdd (line 98) | func NumIteratorsCreatedAdd(enabled bool, val int64) {
function NumGetsWithResultsAdd (line 102) | func NumGetsWithResultsAdd(enabled bool, val int64) {
function NumReadsVlogAdd (line 106) | func NumReadsVlogAdd(enabled bool, val int64) {
function NumBytesWrittenUserAdd (line 110) | func NumBytesWrittenUserAdd(enabled bool, val int64) {
function NumWritesVlogAdd (line 114) | func NumWritesVlogAdd(enabled bool, val int64) {
function NumBytesReadsVlogAdd (line 118) | func NumBytesReadsVlogAdd(enabled bool, val int64) {
function NumBytesReadsLSMAdd (line 122) | func NumBytesReadsLSMAdd(enabled bool, val int64) {
function NumBytesWrittenVlogAdd (line 126) | func NumBytesWrittenVlogAdd(enabled bool, val int64) {
function NumBytesWrittenToL0Add (line 130) | func NumBytesWrittenToL0Add(enabled bool, val int64) {
function NumBytesCompactionWrittenAdd (line 134) | func NumBytesCompactionWrittenAdd(enabled bool, key string, val int64) {
function NumGetsAdd (line 138) | func NumGetsAdd(enabled bool, val int64) {
function NumPutsAdd (line 142) | func NumPutsAdd(enabled bool, val int64) {
function NumMemtableGetsAdd (line 146) | func NumMemtableGetsAdd(enabled bool, val int64) {
function NumCompactionTablesAdd (line 150) | func NumCompactionTablesAdd(enabled bool, val int64) {
function LSMSizeSet (line 154) | func LSMSizeSet(enabled bool, key string, val expvar.Var) {
function VlogSizeSet (line 158) | func VlogSizeSet(enabled bool, key string, val expvar.Var) {
function PendingWritesSet (line 162) | func PendingWritesSet(enabled bool, key string, val expvar.Var) {
function NumLSMBloomHitsAdd (line 166) | func NumLSMBloomHitsAdd(enabled bool, key string, val int64) {
function NumLSMGetsAdd (line 170) | func NumLSMGetsAdd(enabled bool, key string, val int64) {
function LSMSizeGet (line 174) | func LSMSizeGet(enabled bool, key string) expvar.Var {
function VlogSizeGet (line 178) | func VlogSizeGet(enabled bool, key string) expvar.Var {
function addInt (line 182) | func addInt(enabled bool, metric *expvar.Int, val int64) {
function addToMap (line 190) | func addToMap(enabled bool, metric *expvar.Map, key string, val int64) {
function storeToMap (line 198) | func storeToMap(enabled bool, metric *expvar.Map, key string, val expvar...
function getFromMap (line 206) | func getFromMap(enabled bool, metric *expvar.Map, key string) expvar.Var {
FILE: y/watermark.go
type uint64Heap (line 16) | type uint64Heap
method Len (line 18) | func (u uint64Heap) Len() int { return len(u) }
method Less (line 19) | func (u uint64Heap) Less(i, j int) bool { return u[i] < u[j] }
method Swap (line 20) | func (u uint64Heap) Swap(i, j int) { u[i], u[j] = u[j], u[i] }
method Push (line 21) | func (u *uint64Heap) Push(x interface{}) { *u = append(*u, x.(uint64)) }
method Pop (line 22) | func (u *uint64Heap) Pop() interface{} {
type mark (line 33) | type mark struct
type WaterMark (line 51) | type WaterMark struct
method Init (line 59) | func (w *WaterMark) Init(closer *z.Closer) {
method Begin (line 65) | func (w *WaterMark) Begin(index uint64) {
method BeginMany (line 71) | func (w *WaterMark) BeginMany(indices []uint64) {
method Done (line 77) | func (w *WaterMark) Done(index uint64) {
method DoneMany (line 82) | func (w *WaterMark) DoneMany(indices []uint64) {
method DoneUntil (line 88) | func (w *WaterMark) DoneUntil() uint64 {
method SetDoneUntil (line 94) | func (w *WaterMark) SetDoneUntil(val uint64) {
method LastIndex (line 99) | func (w *WaterMark) LastIndex() uint64 {
method WaitForMark (line 104) | func (w *WaterMark) WaitForMark(ctx context.Context, index uint64) err...
method process (line 127) | func (w *WaterMark) process(closer *z.Closer) {
FILE: y/y.go
type Flags (line 37) | type Flags
constant Sync (line 43) | Sync Flags = 1 << iota
constant ReadOnly (line 45) | ReadOnly
function OpenExistingFile (line 57) | func OpenExistingFile(filename string, flags Flags) (*os.File, error) {
function CreateSyncedFile (line 70) | func CreateSyncedFile(filename string, sync bool) (*os.File, error) {
function OpenSyncedFile (line 79) | func OpenSyncedFile(filename string, sync bool) (*os.File, error) {
function OpenTruncFile (line 88) | func OpenTruncFile(filename string, sync bool) (*os.File, error) {
function SafeCopy (line 97) | func SafeCopy(a, src []byte) []byte {
function Copy (line 106) | func Copy(a []byte) []byte {
function KeyWithTs (line 113) | func KeyWithTs(key []byte, ts uint64) []byte {
function ParseTs (line 121) | func ParseTs(key []byte) uint64 {
function CompareKeys (line 132) | func CompareKeys(key1, key2 []byte) int {
function ParseKey (line 140) | func ParseKey(key []byte) []byte {
function SameKey (line 149) | func SameKey(src, dst []byte) bool {
type Slice (line 158) | type Slice struct
method Resize (line 164) | func (s *Slice) Resize(sz int) []byte {
function FixedDuration (line 173) | func FixedDuration(d time.Duration) string {
type Throttle (line 187) | type Throttle struct
method Do (line 206) | func (t *Throttle) Do() error {
method Done (line 222) | func (t *Throttle) Done(err error) {
method Finish (line 237) | func (t *Throttle) Finish() error {
function NewThrottle (line 196) | func NewThrottle(max int) *Throttle {
function U16ToBytes (line 254) | func U16ToBytes(v uint16) []byte {
function BytesToU16 (line 261) | func BytesToU16(b []byte) uint16 {
function U32ToBytes (line 266) | func U32ToBytes(v uint32) []byte {
function BytesToU32 (line 273) | func BytesToU32(b []byte) uint32 {
function U32SliceToBytes (line 278) | func U32SliceToBytes(u32s []uint32) []byte {
function BytesToU32Slice (line 291) | func BytesToU32Slice(b []byte) []uint32 {
function U64ToBytes (line 304) | func U64ToBytes(v uint64) []byte {
function BytesToU64 (line 311) | func BytesToU64(b []byte) uint64 {
function U64SliceToBytes (line 316) | func U64SliceToBytes(u64s []uint64) []byte {
function BytesToU64Slice (line 329) | func BytesToU64Slice(b []byte) []uint64 {
type page (line 342) | type page struct
type PageBuffer (line 351) | type PageBuffer struct
method Write (line 367) | func (b *PageBuffer) Write(data []byte) (int, error) {
method WriteByte (line 389) | func (b *PageBuffer) WriteByte(data byte) error {
method Len (line 395) | func (b *PageBuffer) Len() int {
method pageForOffset (line 400) | func (b *PageBuffer) pageForOffset(offset int) (int, int) {
method Truncate (line 420) | func (b *PageBuffer) Truncate(n int) {
method Bytes (line 430) | func (b *PageBuffer) Bytes() []byte {
method WriteTo (line 441) | func (b *PageBuffer) WriteTo(w io.Writer) (int64, error) {
method NewReaderAt (line 455) | func (b *PageBuffer) NewReaderAt(offset int) *PageBufferReader {
function NewPageBuffer (line 359) | func NewPageBuffer(pageSize int) *PageBuffer {
type PageBufferReader (line 466) | type PageBufferReader struct
method Read (line 473) | func (r *PageBufferReader) Read(p []byte) (int, error) {
constant kvsz (line 510) | kvsz = int(unsafe.Sizeof(pb.KV{}))
function NewKV (line 512) | func NewKV(alloc *z.Allocator) *pb.KV {
function IBytesToString (line 524) | func IBytesToString(size uint64, precision int) string {
type RateMonitor (line 538) | type RateMonitor struct
method Capture (line 557) | func (rm *RateMonitor) Capture(sent uint64) {
method Rate (line 571) | func (rm *RateMonitor) Rate() uint64 {
function NewRateMonitor (line 546) | func NewRateMonitor(numSamples int) *RateMonitor {
constant minRate (line 553) | minRate = 0.0001
FILE: y/y_test.go
function BenchmarkBuffer (line 24) | func BenchmarkBuffer(b *testing.B) {
function TestPageBuffer (line 49) | func TestPageBuffer(t *testing.T) {
function TestBufferWrite (line 75) | func TestBufferWrite(t *testing.T) {
function TestPagebufferTruncate (line 98) | func TestPagebufferTruncate(t *testing.T) {
function TestPagebufferReader (line 126) | func TestPagebufferReader(t *testing.T) {
function TestPagebufferReader2 (line 174) | func TestPagebufferReader2(t *testing.T) {
function TestPagebufferReader3 (line 203) | func TestPagebufferReader3(t *testing.T) {
function TestPagebufferReader4 (line 244) | func TestPagebufferReader4(t *testing.T) {
function TestPagebufferReader5 (line 269) | func TestPagebufferReader5(t *testing.T) {
function TestSizeVarintForZero (line 285) | func TestSizeVarintForZero(t *testing.T) {
function TestEncodedSize (line 290) | func TestEncodedSize(t *testing.T) {
function TestAllocatorReuse (line 306) | func TestAllocatorReuse(t *testing.T) {
function TestSafeCopy_Issue2067 (line 332) | func TestSafeCopy_Issue2067(t *testing.T) {
FILE: y/zstd.go
function ZSTDDecompress (line 22) | func ZSTDDecompress(dst, src []byte) ([]byte, error) {
function ZSTDCompress (line 32) | func ZSTDCompress(dst, src []byte, compressionLevel int) ([]byte, error) {
function ZSTDCompressBound (line 46) | func ZSTDCompressBound(srcSize int) int {
Condensed preview — 136 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,202K chars).
[
{
"path": ".github/CODEOWNERS",
"chars": 189,
"preview": "# CODEOWNERS info: https://help.github.com/en/articles/about-code-owners\n# Owners are automatically requested for review"
},
{
"path": ".github/ISSUE_TEMPLATE/bug_report.md",
"chars": 649,
"preview": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: \"\"\nlabels: bug\nassignees: \"\"\n---\n\n## Describe the "
},
{
"path": ".github/ISSUE_TEMPLATE/config.yml",
"chars": 181,
"preview": "blank_issues_enabled: false\ncontact_links:\n - name: Badger Community Support\n url: https://github.com/orgs/dgraph-io"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.md",
"chars": 593,
"preview": "---\nname: Feature request\nabout: Suggest an idea for this project\ntitle: \"\"\nlabels: \"\"\nassignees: \"\"\n---\n\n## Is your fea"
},
{
"path": ".github/PULL_REQUEST_TEMPLATE.md",
"chars": 911,
"preview": "**Description**\n\nPlease explain the changes you made here.\n\n**Checklist**\n\n- [ ] Code compiles correctly and linting pas"
},
{
"path": ".github/renovate.json",
"chars": 357,
"preview": "{\n \"$schema\": \"https://docs.renovatebot.com/renovate-schema.json\",\n \"extends\": [\"local>dgraph-io/renovate-config\"],\n "
},
{
"path": ".github/workflows/cd-badger.yml",
"chars": 3364,
"preview": "name: cd-badger\n\non:\n workflow_dispatch:\n inputs:\n releasetag:\n description: releasetag\n required"
},
{
"path": ".github/workflows/ci-badger-bank-tests-nightly.yml",
"chars": 1515,
"preview": "name: ci-badger-bank-tests-nightly\n\non:\n push:\n paths-ignore:\n - \"**.md\"\n - docs/**\n - images/**\n "
},
{
"path": ".github/workflows/ci-badger-bank-tests.yml",
"chars": 811,
"preview": "name: ci-badger-bank-tests\n\non:\n workflow_dispatch: # allows manual trigger from GitHub\n pull_request:\n paths-ignor"
},
{
"path": ".github/workflows/ci-badger-tests.yml",
"chars": 1361,
"preview": "name: ci-badger-tests\n\non:\n workflow_dispatch: # allows manual trigger from GitHub\n pull_request:\n paths-ignore:\n "
},
{
"path": ".github/workflows/ci-dgraph-tests.yml",
"chars": 1842,
"preview": "name: ci-dgraph-tests\n\non:\n push:\n paths-ignore:\n - \"**.md\"\n - docs/**\n - images/**\n branches:\n "
},
{
"path": ".github/workflows/trunk.yml",
"chars": 247,
"preview": "name: Trunk Code Quality\non:\n pull_request:\n branches: main\n\npermissions:\n contents: read\n actions: write\n checks"
},
{
"path": ".gitignore",
"chars": 205,
"preview": "# Binaries for programs and plugins\n*.exe\n*.exe~\n*.dll\n*.so\n*.dylib\nbadger/badger-*\n\n# Test binary, build with `go test "
},
{
"path": ".trunk/.gitignore",
"chars": 80,
"preview": "*out\n*logs\n*actions\n*notifications\n*tools\nplugins\nuser_trunk.yaml\nuser.yaml\ntmp\n"
},
{
"path": ".trunk/configs/.checkov.yaml",
"chars": 26,
"preview": "skip-check:\n - CKV_GHA_7\n"
},
{
"path": ".trunk/configs/.markdownlint.json",
"chars": 212,
"preview": "{\n \"line-length\": { \"line_length\": 150, \"tables\": false },\n \"no-inline-html\": false,\n \"no-bare-urls\": false,\n \"no-sp"
},
{
"path": ".trunk/configs/.prettierrc",
"chars": 66,
"preview": "{\n \"semi\": false,\n \"proseWrap\": \"always\",\n \"printWidth\": 100\n}\n"
},
{
"path": ".trunk/configs/.shellcheckrc",
"chars": 167,
"preview": "enable=all\nsource-path=SCRIPTDIR\ndisable=SC2154\n\n# If you're having issues with shellcheck following source, disable the"
},
{
"path": ".trunk/configs/.yamllint.yaml",
"chars": 152,
"preview": "rules:\n quoted-strings:\n required: only-when-needed\n extra-allowed: [\"{|}\"]\n key-duplicates: {}\n octal-values:\n"
},
{
"path": ".trunk/configs/svgo.config.mjs",
"chars": 277,
"preview": "export default {\n plugins: [\n {\n name: \"preset-default\",\n params: {\n overrides: {\n removeV"
},
{
"path": ".trunk/trunk.yaml",
"chars": 943,
"preview": "# This file controls the behavior of Trunk: https://docs.trunk.io/cli\n# To learn more about the format of this file, see"
},
{
"path": ".vscode/extensions.json",
"chars": 38,
"preview": "{\n \"recommendations\": [\"trunk.io\"]\n}\n"
},
{
"path": ".vscode/settings.json",
"chars": 139,
"preview": "{\n \"editor.formatOnSave\": true,\n \"editor.defaultFormatter\": \"trunk.io\",\n \"editor.trimAutoWhitespace\": true,\n \"trunk."
},
{
"path": "CHANGELOG.md",
"chars": 41649,
"preview": "# Changelog\n\nAll notable changes to this project will be documented in this file.\n\nThe format is based on [Keep a Change"
},
{
"path": "CODE_OF_CONDUCT.md",
"chars": 5225,
"preview": "# Contributor Covenant Code of Conduct\n\n## Our Pledge\n\nWe as members, contributors, and leaders pledge to make participa"
},
{
"path": "CONTRIBUTING.md",
"chars": 2808,
"preview": "# Contribution Guide\n\n- [Before you get started](#before-you-get-started)\n - [Code of Conduct](#code-of-conduct)\n- [You"
},
{
"path": "LICENSE",
"chars": 10173,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "Makefile",
"chars": 1301,
"preview": "#\n# SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n# SPDX-License-Identifier: Apache-2.0\n#\n\nUSER_ID = $(s"
},
{
"path": "README.md",
"chars": 16652,
"preview": "# BadgerDB\n\n[](https://pkg.go.dev/github.com"
},
{
"path": "SECURITY.md",
"chars": 567,
"preview": "# Reporting Security Concerns\n\nWe take the security of Badger very seriously. If you believe you have found a security\nv"
},
{
"path": "VERSIONING.md",
"chars": 2451,
"preview": "# Serialization Versioning: Semantic Versioning for databases\n\nSemantic Versioning, commonly known as SemVer, is a great"
},
{
"path": "backup.go",
"chars": 7709,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "backup_test.go",
"chars": 12978,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "batch.go",
"chars": 6125,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "batch_test.go",
"chars": 4243,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "changes.sh",
"chars": 374,
"preview": "#!/bin/bash\n\nset -e\nGHORG=${GHORG:-dgraph-io}\nGHREPO=${GHREPO:-badger}\ncat <<EOF\nThis description was generated using th"
},
{
"path": "compaction.go",
"chars": 6076,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "contrib/RELEASE.md",
"chars": 2101,
"preview": "# Badger Release Process\n\nThis document outlines the steps needed to build and push a new release of Badger.\n\n1. Have a "
},
{
"path": "db.go",
"chars": 57241,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "db2_test.go",
"chars": 29579,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "db_test.go",
"chars": 68886,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "dir_aix.go",
"chars": 3933,
"preview": "//go:build aix\n// +build aix\n\n/*\n * SPDX-FileCopyrightText: © 2017-2026 Istari Digital, Inc.\n * SPDX-License-Identifier:"
},
{
"path": "dir_other.go",
"chars": 2862,
"preview": "//go:build js || wasip1\n// +build js wasip1\n\n/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-Lice"
},
{
"path": "dir_plan9.go",
"chars": 4483,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "dir_unix.go",
"chars": 3127,
"preview": "//go:build !windows && !plan9 && !js && !wasip1 && !aix\n// +build !windows,!plan9,!js,!wasip1,!aix\n\n/*\n * SPDX-FileCopyr"
},
{
"path": "dir_windows.go",
"chars": 3415,
"preview": "//go:build windows\n// +build windows\n\n/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Ide"
},
{
"path": "discard.go",
"chars": 3593,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "discard_test.go",
"chars": 1459,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "doc.go",
"chars": 1219,
"preview": "/*\nPackage badger implements an embeddable, simple and fast key-value database,\nwritten in pure Go. It is designed to be"
},
{
"path": "docs/design.md",
"chars": 3444,
"preview": "# Design\n\nWe wrote Badger with these design goals in mind:\n\n- Write a key-value database in pure Go\n- Use latest researc"
},
{
"path": "docs/encryption-at-rest.md",
"chars": 5324,
"preview": "# Encryption at Rest in Dgraph and Badger\n\nBadger provides encryption at rest using AES encryption, enabling compliance "
},
{
"path": "docs/index.md",
"chars": 597,
"preview": "# Badger\n\n## What is Badger?\n\nBadgerDB is an embeddable, persistent, and fast key-value (KV) database written in pure Go"
},
{
"path": "docs/quickstart.md",
"chars": 24221,
"preview": "# Quickstart\n\n## Prerequisites\n\n- [Go](https://go.dev/doc/install) - v1.23 or higher\n- Text editor - we recommend [VS Co"
},
{
"path": "docs/troubleshooting.md",
"chars": 6802,
"preview": "# Troubleshooting\n\n## Writes are getting stuck\n\n**Update: with the new `Value(func(v []byte))` API, this deadlock can no"
},
{
"path": "errors.go",
"chars": 5143,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "fb/BlockOffset.go",
"chars": 2504,
"preview": "// Code generated by the FlatBuffers compiler. DO NOT EDIT.\n\npackage fb\n\nimport (\n\tflatbuffers \"github.com/google/flatbu"
},
{
"path": "fb/TableIndex.go",
"chars": 4558,
"preview": "// Code generated by the FlatBuffers compiler. DO NOT EDIT.\n\npackage fb\n\nimport (\n\tflatbuffers \"github.com/google/flatbu"
},
{
"path": "fb/flatbuffer.fbs",
"chars": 422,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\nnamespace fb;"
},
{
"path": "fb/gen.sh",
"chars": 285,
"preview": "#!/usr/bin/env bash\n\nset -e\n\n## Install flatc if not present\n## ref. https://google.github.io/flatbuffers/flatbuffers_gu"
},
{
"path": "fb/install_flatbuffers.sh",
"chars": 1086,
"preview": "#!/usr/bin/env bash\n\nset -e\n\ninstall_mac() {\n\tcommand -v brew >/dev/null ||\n\t\t{\n\t\t\techo \"[ERROR]: 'brew' command not not"
},
{
"path": "go.mod",
"chars": 1159,
"preview": "module github.com/dgraph-io/badger/v4\n\ngo 1.23.0\n\ntoolchain go1.25.0\n\nrequire (\n\tgithub.com/cespare/xxhash/v2 v2.3.0\n\tgi"
},
{
"path": "go.sum",
"chars": 5648,
"preview": "github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=\ngithub.com/cespare/xxhash/v2 v2.3.0/"
},
{
"path": "histogram.go",
"chars": 4498,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "histogram_test.go",
"chars": 2924,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "integration/testgc/.gitignore",
"chars": 8,
"preview": "/testgc\n"
},
{
"path": "integration/testgc/main.go",
"chars": 4755,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage main\n"
},
{
"path": "iterator.go",
"chars": 21875,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "iterator_test.go",
"chars": 11817,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "key_registry.go",
"chars": 12702,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "key_registry_test.go",
"chars": 4269,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "level_handler.go",
"chars": 9446,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "levels.go",
"chars": 53684,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "levels_test.go",
"chars": 42447,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "logger.go",
"chars": 2004,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "logger_test.go",
"chars": 1355,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "managed_db.go",
"chars": 2532,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "managed_db_test.go",
"chars": 21567,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "manifest.go",
"chars": 14260,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "manifest_test.go",
"chars": 7218,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "memtable.go",
"chars": 17218,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "merge.go",
"chars": 4536,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "merge_test.go",
"chars": 4842,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "metrics_test.go",
"chars": 5770,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "options/options.go",
"chars": 1088,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage optio"
},
{
"path": "options.go",
"chars": 28453,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "options_test.go",
"chars": 2203,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "pb/badgerpb4.pb.go",
"chars": 26551,
"preview": "//\n// SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n// SPDX-License-Identifier: Apache-2.0\n\n// Use protos/gen"
},
{
"path": "pb/badgerpb4.proto",
"chars": 1578,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\n// Use protos"
},
{
"path": "pb/gen.sh",
"chars": 237,
"preview": "#!/bin/bash\n\n# Run this script from its directory, so that badgerpb4.proto is where it's expected to\n# be.\n\ngo install g"
},
{
"path": "pb/protos_test.go",
"chars": 661,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage pb\n\ni"
},
{
"path": "publisher.go",
"chars": 3421,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "publisher_test.go",
"chars": 4371,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "skl/README.md",
"chars": 4821,
"preview": "This is much better than `skiplist` and `slist`.\n\n```sh\nBenchmarkReadWrite/frac_0-8 3000000 537 ns/op"
},
{
"path": "skl/arena.go",
"chars": 3492,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage skl\n\n"
},
{
"path": "skl/skl.go",
"chars": 14466,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\n/*\nAdapted fr"
},
{
"path": "skl/skl_test.go",
"chars": 14845,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage skl\n\n"
},
{
"path": "stream.go",
"chars": 15889,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "stream_test.go",
"chars": 9528,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "stream_writer.go",
"chars": 14440,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "stream_writer_test.go",
"chars": 23374,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "structs.go",
"chars": 6254,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "structs_test.go",
"chars": 660,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "table/README.md",
"chars": 3885,
"preview": "Size of table is 123,217,667 bytes for all benchmarks.\n\n# BenchmarkRead\n\n```sh\n$ go test -bench ^BenchmarkRead$ -run ^$ "
},
{
"path": "table/builder.go",
"chars": 18218,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage table"
},
{
"path": "table/builder_test.go",
"chars": 6999,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage table"
},
{
"path": "table/iterator.go",
"chars": 13241,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage table"
},
{
"path": "table/merge_iterator.go",
"chars": 4624,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage table"
},
{
"path": "table/merge_iterator_test.go",
"chars": 11348,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage table"
},
{
"path": "table/table.go",
"chars": 24371,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage table"
},
{
"path": "table/table_test.go",
"chars": 24134,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage table"
},
{
"path": "test.sh",
"chars": 4047,
"preview": "#!/bin/bash\n\nset -eo pipefail\n\ngo version\n\n# Check if Github Actions is running\nif [ \"$CI\" = \"true\" ]; then\n\t# Enable co"
},
{
"path": "test_extensions.go",
"chars": 2619,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "trie/trie.go",
"chars": 5506,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage trie\n"
},
{
"path": "trie/trie_test.go",
"chars": 4377,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage trie\n"
},
{
"path": "txn.go",
"chars": 23457,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "txn_test.go",
"chars": 24471,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "util.go",
"chars": 2446,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "value.go",
"chars": 33152,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "value_test.go",
"chars": 33384,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "watermark_edge_test.go",
"chars": 2106,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage badge"
},
{
"path": "y/bloom.go",
"chars": 4474,
"preview": "// Copyright 2013 The LevelDB-Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// "
},
{
"path": "y/bloom_test.go",
"chars": 3217,
"preview": "// Copyright 2013 The LevelDB-Go Authors. All rights reserved.\n// Use of this source code is governed by a BSD-style\n// "
},
{
"path": "y/checksum.go",
"chars": 1044,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
},
{
"path": "y/checksum_test.go",
"chars": 1637,
"preview": "package y\n\nimport (\n\t\"hash/crc32\"\n\t\"testing\"\n\n\t\"github.com/cespare/xxhash/v2\"\n\t\"github.com/dgraph-io/badger/v4/pb\"\n\t\"git"
},
{
"path": "y/encrypt.go",
"chars": 1217,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
},
{
"path": "y/encrypt_test.go",
"chars": 1242,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
},
{
"path": "y/error.go",
"chars": 2168,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\n//"
},
{
"path": "y/error_test.go",
"chars": 844,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
},
{
"path": "y/file_dsync.go",
"chars": 332,
"preview": "//go:build !dragonfly && !freebsd && !windows && !plan9 && !js && !wasip1\n// +build !dragonfly,!freebsd,!windows,!plan9,"
},
{
"path": "y/file_nodsync.go",
"chars": 282,
"preview": "//go:build dragonfly || freebsd || windows || plan9\n// +build dragonfly freebsd windows plan9\n\n/*\n * SPDX-FileCopyrightT"
},
{
"path": "y/iterator.go",
"chars": 1892,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
},
{
"path": "y/metrics.go",
"chars": 6266,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
},
{
"path": "y/watermark.go",
"chars": 6780,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
},
{
"path": "y/y.go",
"chars": 15077,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
},
{
"path": "y/y_test.go",
"chars": 10241,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
},
{
"path": "y/zstd.go",
"chars": 1346,
"preview": "/*\n * SPDX-FileCopyrightText: © 2017-2025 Istari Digital, Inc.\n * SPDX-License-Identifier: Apache-2.0\n */\n\npackage y\n\nim"
}
]
About this extraction
This page contains the full source code of the dgraph-io/badger GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 136 files (1.1 MB), approximately 329.0k tokens, and a symbol index with 1493 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.