Repository: sudeep9/mojo
Branch: main
Commit: 730a25eeb9e8
Files: 53
Total size: 125.9 KB
Directory structure:
gitextract_2dkkx48z/
├── .gitignore
├── Cargo.toml
├── LICENSE
├── README.md
├── build.sh
├── crates/
│ ├── mojo-cli/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── buckets.rs
│ │ ├── commit.rs
│ │ ├── diff.rs
│ │ ├── iget.rs
│ │ ├── iview.rs
│ │ ├── main.rs
│ │ ├── state.rs
│ │ └── truncate.rs
│ ├── mojofs/
│ │ ├── .gitignore
│ │ ├── Cargo.toml
│ │ ├── meson.build
│ │ ├── src/
│ │ │ ├── error.rs
│ │ │ ├── kvfile.rs
│ │ │ ├── lib.rs
│ │ │ ├── native_file.rs
│ │ │ ├── open_options.rs
│ │ │ ├── vfs.rs
│ │ │ └── vfsfile.rs
│ │ └── tests/
│ │ └── mojofs_test.rs
│ ├── mojoio/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── error.rs
│ │ ├── lib.rs
│ │ └── nix.rs
│ └── mojokv/
│ ├── .gitignore
│ ├── Cargo.toml
│ └── src/
│ ├── bmap.rs
│ ├── bucket.rs
│ ├── error.rs
│ ├── index/
│ │ ├── mem.rs
│ │ └── mod.rs
│ ├── keymap.rs
│ ├── lib.rs
│ ├── state.rs
│ ├── store.rs
│ ├── utils.rs
│ └── value.rs
├── docs/
│ ├── design.md
│ ├── source.md
│ └── user-guide.md
├── meson.build
├── sqlite-ext/
│ ├── mojo.c
│ └── mojofs.h
└── test-scripts/
├── commands.py
├── perftest.py
├── test.sql
├── test2.sql
└── testdb.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
target
log
log*
*.log
*.dylib
.vscode
*.db
*.swp
*-journal
*-wal
build
Cargo.lock
testdbs
__pycache__
.DS_Store
================================================
FILE: Cargo.toml
================================================
[workspace]
members = [
"crates/mojofs",
"crates/mojokv",
"crates/mojoio",
"crates/mojo-cli",
]
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2022 Sudeep Jathar
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# MojoFS
MojoFS is a versioning, userspace filesystem for sqlite DB. It is tailor made for sqlite and not supposed to be used as a general purpose fs.
The main feature of the fs is versioning/snapshotting. Only one version is writable and all the old versions are immutable.
- [MojoFS](#mojofs)
- [License](#license)
- [Development status](#development-status)
- [Build](#build)
- [Usage](#usage)
- [Create database using mojo and insert few records](#create-database-using-mojo-and-insert-few-records)
- [Select data](#select-data)
- [Commit the database](#commit-the-database)
- [Write to active version=2](#write-to-active-version2)
- [Read old version=1](#read-old-version1)
- [Read active version=2:](#read-active-version2)
- [Docs](#docs)
- [Source map](#source-map)
- [Limits](#limits)
- [Testing](#testing)
- [Performance](#performance)
- [Road to v1.0](#road-to-v10)
# License
I have changed it to MIT. See the [LICENSE](/LICENSE) file.
# Development status
|Item|Value|
|-------|------|
|Quality|pre-alpha|
|Maintenance|active|
# Build
At present only mac/linux is supported. Windows is not supported only because I do not have windows machine. This may change in future.
The build expects the following in the environment:
1. Meson Build + Ninja (see [here](https://mesonbuild.com/Getting-meson.html))
2. C compiler (gcc or clang)
3. Rust version v1.59+
4. sqlite headers + libraries
5. Optional: python3 for testing (most versions should be ok)
```
git clone github.com/sudeep9/mojo
cd mojo
./build.sh release
```
Following artifacts will be in `build` dir:
* `build/libmojo.dylib` (`.so` extension in linux) => The sqlite extension/vfs
* `build/mojo-cli` => mojo cli tool to manage the file system
# Usage
All the examples below uses sqlite3 binary. However, you can use any bindings of sqlite.
## Create database using mojo and insert few records
```
rm -fR a.db
sqlite3 <<EOF
pragma page_size = 4096;
.load ./build/libmojo
.open 'file:a.db?vfs=mojo&pagesz=4096'
create table if not exists test (n int);
insert into test values(1);
insert into test values(2);
EOF
```
The `.load` loads the extension and registers the mojo vfs.
The `.open` creates the database `a.db`. The mojofs creates a dir `a.db` instead of a file.
The `pagesz=4096` is the page size which the fs will use.
This needs to be same as the page size used by sqlite.
I am not aware of any way this can be detected automatically by VFS layer.
Unfortunately its upto the user to ensure that these values are consistent.
If these values mismatch the database is bound to be corrupted.
## Select data
```
sqlite3 <<EOF
pragma page_size = 4096;
.load ./build/libmojo
.open 'file:a.db?vfs=mojo&pagesz=4096'
select * from test;
EOF
```
Output:
```
1
2
```
## Commit the database
```
./build/mojo-cli ./a.db commit
```
Output:
```
active version before commit: 1
active version after commit: 2
```
New databases always start with version=1. The commit advances the version number. Here the version=1 is immutable and version=2 is now writable. Lets write again
## Write to active version=2
```
sqlite3 <<EOF
pragma page_size = 4096;
.load ./build/libmojo
.open 'file:a.db?vfs=mojo&pagesz=4096'
insert into test values(3);
insert into test values(4);
EOF
```
## Read old version=1
```
sqlite3 <<EOF
pragma page_size = 4096;
.load ./build/libmojo
.open 'file:a.db?vfs=mojo&pagesz=4096&ver=1&mode=ro'
select * from test;
EOF
```
We specify the `ver=1` and `mode=ro` (i.e. readonly)
Output:
```
1
2
```
## Read active version=2:
```
sqlite3 <<EOF
pragma page_size = 4096;
.load ./build/libmojo
.open 'file:a.db?vfs=mojo&pagesz=4096'
select * from test;
EOF
```
Output:
```
1
2
3
4
```
# Docs
* design doc => [here](https://github.com/sudeep9/mojo/blob/main/docs/design.md)
* user-guide => [here](https://github.com/sudeep9/mojo/blob/main/docs/user-guide.md)
### Source map
* All the rust code is under `crates` folder
* All the docs are under `docs` folder
* The `sqlite-ext` folder has C code which is compiled down to shared lib
* The `test-scripts` has various assorted test scripts which includes perf & black-box test
Crate wise details are at: [here](https://github.com/sudeep9/mojo/blob/main/docs/source.md)
# Limits
For a page size = 4KB (which is the default for sqlite for some years now) following are the limits:
* Max sqlite db file size (KB) = `pow(2,32) * 4` = 17179869184 KB or 16TB
16TB is logical size i.e. file size reported by stat like call in any version. Since there could be multiple versions of the file, the total size of all such versions taken together can exceed 16TB.
* Max version num = `pow(2,24)` = 16777216 or 16 million
To put 16M versions in perspective, even if you create a version every 1 min, it will take ~31.92 years to reach the max version.
Note: All these limits are actually artificial to keep the memory usage reasonable. In future, these will be tunable and also have the ability to baseline the versions.
# Testing
I wanted to use sqlite [test harness](https://www.sqlite.org/th3.html) but it requires license. Quoting from the test harness link:
> SQLite itself is in the public domain and can be used for any purpose. But TH3 is proprietary and requires a license.
Instead there is `testdb.py` for black box testing and `perftest.py` for perf tests.
At present the `testdb.py` tests combinations of the following:
```
page_sizes = [4096]
journal_modes = ["OFF", "WAL", "MEMORY", "DELETE", "TRUNCATE", "PERSIST"]
vacuum_modes = ["NONE", "FULL", "INCREMENTAL"]
```
For each of the combination, there are about ~11 subtests so in all 18 x 11 = 198 tests.
These are early days and off-course there is a long way to go.
To run the full suite:
```
python3 testdb.py build/libmojo full
```
# Performance
About `10_000_000` rows are inserted and then for reading we select the rows and get the row count.
Finally it updates half the rows.
To run the perf test:
```
MOJOKV_CLI=build/mojo-cli python3 perftest.py ./build/libmojo
```
Output on 2018/19 macbook:
```
Running perf for: insert
vfs=std time elapsed (s): 20.524982929229736
vfs=mojo time elapsed (s): 21.202611923217773
Mojo takes 1.033 times than std vfs
------------------------
Running perf for: update rows
vfs=std time elapsed (s): 2.871242046356201
vfs=mojo time elapsed (s): 2.439574956893921
Mojo takes 0.85 times than std vfs
------------------------
Running perf for: select
select iter count: 10000000
vfs=std time elapsed (s): 8.659775018692017
select iter count: 10000000
vfs=mojo time elapsed (s): 5.907814025878906
Mojo takes 0.682 times than std vfs
------------------------
Running perf for: row count
row count: 0
vfs=std time elapsed (s): 2.96425199508667
row count: 0
vfs=mojo time elapsed (s): 1.5106308460235596
Mojo takes 0.51 times than std vfs
------------------------
```
The writes being only `1.033` times worse is in line with my expectations. However, I am investigating why the reads are so better with mojo.
My guess as of now is that in the standard default vfs at https://github.com/sqlite/sqlite/blob/master/src/os_unix.c does not use pread, whereas mojo uses pread. Lack of pread results into two system call i.e. seek + read. This ***might*** explain the perf difference. This need further confirmation though.
See the comment and code in the c file above:
```
** ... Since SQLite does not define USE_PREAD
** in any form by default, we will not attempt to define _XOPEN_SOURCE.
** See tickets #2741 and #2681.
```
In seekAndRead function:
```
#if defined(USE_PREAD)
got = osPread(id->h, pBuf, cnt, offset);
SimulateIOError( got = -1 );
#elif defined(USE_PREAD64)
got = osPread64(id->h, pBuf, cnt, offset);
SimulateIOError( got = -1 );
#else
newOffset = lseek(id->h, offset, SEEK_SET);
SimulateIOError( newOffset = -1 );
if( newOffset<0 ){
storeLastErrno((unixFile*)id, errno);
return -1;
}
got = osRead(id->h, pBuf, cnt);
```
# Road to v1.0
It needs atleast the following:
- [ ] Top-notch unit & black box test coverage
- [ ] Ease of use e.g debugability, add more mojo-cli admin commands
- [ ] Ability to diff the versions
- [ ] Ability to delete versions
- [ ] Ability to merge versions (not like git merge)
- [ ] Ability to recover from corrupted fs.
- [ ] Stabilize on-disk format
- [ ] User guide
A lot of the above needs to be clearly defined.
================================================
FILE: build.sh
================================================
#!/usr/bin/env bash
export BUILD_PROFILE=debug
export CARGO_OPT=""
export MESON_BUILD_PROFILE="debug"
export MESON_BUILD_DIR="build"
if [ $# -gt 0 ]; then
if [ $1 = "release" ]; then
export BUILD_PROFILE=release
export CARGO_OPT="--release"
export MESON_BUILD_PROFILE="release"
fi
fi
echo "BUILD_PROFILE=$BUILD_PROFILE"
echo "-------------------------------"
echo " [Rust build starting]"
echo "-------------------------------"
cargo build $CARGO_OPT
if [ $? -ne 0 ]; then
echo "Error: Cargo build failed with $?"
exit $?
fi
mkdir -p $MESON_BUILD_DIR
rm -fR $MESON_BUILD_DIR/*
cp target/$BUILD_PROFILE/mojo-cli $MESON_BUILD_DIR/.
if [ $? -ne 0 ]; then
echo "Copying failed"
exit $?
fi
echo "-------------------------------"
echo " [C build starting]"
echo "-------------------------------"
meson setup --buildtype=$MESON_BUILD_PROFILE $MESON_BUILD_DIR
if [ $? -ne 0 ]; then
echo "Error: meson reconfigure failed with $?"
exit $?
fi
meson compile -C $MESON_BUILD_DIR
if [ $? -ne 0 ]; then
echo "Error: meson compile failed with $?"
exit $?
fi
echo ">> The build artifacts are at the dir: $MESON_BUILD_DIR"
if [ "$MOJO_TEST" != "" ]; then
echo "-------------------------------"
echo " [Mojo test starting]"
echo "-------------------------------"
export MOJOKV_CLI=./target/$BUILD_PROFILE/mojo-cli
python3 test-scripts/testdb.py $MESON_BUILD_DIR/mojo
if [ $? -ne 0 ]; then
echo "Error: mojo test failed with $?"
exit 1
fi
fi
================================================
FILE: crates/mojo-cli/Cargo.toml
================================================
[package]
name = "mojo-cli"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
mojokv = {path="../mojokv"}
log = "0.4.17"
env_logger = "0.9.0"
clap = {version="3.2.6", features=["derive"] }
anyhow = "1.0.58"
================================================
FILE: crates/mojo-cli/src/buckets.rs
================================================
use anyhow::Error;
use mojokv::BucketMap;
pub fn cmd(kvpath: &std::path::Path, ver: u32) -> Result<(), Error> {
let bmap = BucketMap::load(kvpath, ver)?;
for (bucket_name, ver) in bmap.map()?.iter() {
println!("{} -> {}", bucket_name, ver);
}
Ok(())
}
================================================
FILE: crates/mojo-cli/src/commit.rs
================================================
use anyhow::Error;
use mojokv::Store;
pub fn cmd(kvpath: &std::path::Path) -> Result<(), Error> {
let st = Store::writable(&kvpath, false, None, None)?;
println!("active version before commit: {}", st.active_ver());
let new_ver = st.commit()?;
println!("active version after commit: {}", new_ver);
Ok(())
}
================================================
FILE: crates/mojo-cli/src/diff.rs
================================================
use anyhow::Error;
use mojokv::KVStore;
pub fn cmd(kvpath: &std::path::Path, fver: u32, tver: u32) -> Result<(), Error> {
if fver >= tver {
return Err(Error::msg("'from' version cannot be greater than or equal 'to' version"));
}
let state = KVStore::load_state(kvpath)?;
let f_index = KVStore::load_index(kvpath, fver)?;
let t_index = KVStore::load_index(kvpath, tver)?;
let mut key = 0u32;
for (slot_index, t_slot) in t_index.kmap.slot_map.iter().enumerate() {
let f_slot = &f_index.kmap.slot_map[slot_index];
match (f_slot, t_slot) {
(Some(fs), Some(ts)) => {
if fs.len() != ts.len() {
return Err(Error::msg(format!("Slot length mismatch {} {}", fs.len(), ts.len())));
}
for (j,(fv, tv)) in std::iter::zip(fs, ts).enumerate() {
if fv.get_ver() != tv.get_ver() {
println!("M k={} fv={} tv={} fo={} to={}",
key+j as u32,
fv.get_ver(),
tv.get_ver(),
fv.get_off(),
tv.get_off());
}
}
},
(Some(_fs), None) => {
println!("D {} -> {} deleted", key, key + state.pps);
},
(None, Some(_ts)) => {
println!("A {} -> {} added", key, key + state.pps);
},
(None, None) => {},
}
key += state.pps;
}
Ok(())
}
================================================
FILE: crates/mojo-cli/src/iget.rs
================================================
use anyhow::Error;
use mojokv::{Store,BucketOpenMode};
pub fn cmd(kvpath: &std::path::Path, bucket: &str, ver: u32, key: u32) -> Result<(), Error> {
let st = Store::readonly(&kvpath, ver)?;
let b = st.open(bucket, BucketOpenMode::Read)?;
println!("Max key: {}", b.max_key());
match b.get_key(key)? {
Some(val) => {
println!("value: {:?}", val);
},
None => {
println!("Key not found")
}
}
Ok(())
}
================================================
FILE: crates/mojo-cli/src/iview.rs
================================================
use anyhow::Error;
use mojokv::{Store};
pub fn cmd(kvpath: &std::path::Path, name: &str, ver: u32, additional: bool, keys: bool) -> Result<(), Error> {
let st = Store::readonly(kvpath, ver)?;
let ret = st.get_index(name)?; //Bucket::load_index(&kvpath, name, ver)?;
if ret.is_none() {
println!("Bucket {} does not exists", name);
}
let (uncomp_sz, comp_sz, i) = ret.unwrap();
let h = i.header();
let st = if additional {
Some(Store::load_state(kvpath)?)
}else{
None
};
println!("Format version : {}", h.format_ver);
println!("Minimum version : {}", h.min_ver);
println!("Maximum version : {}", h.max_ver);
println!("Active version : {}", h.active_ver);
println!("Pages per slot : {}", h.pps);
println!("Maximum key : {}", h.max_key);
println!("Compressed size : {}", comp_sz);
println!("Uncompressed size : {}", uncomp_sz);
if let Some(st) = st {
println!("----------------------");
println!("Logical size : {}", st.page_size() * (h.max_key + 1) as u32);
}
if keys {
println!("----------------------");
println!("keys");
for (key, val) in i.iter(0, 0) {
println!(" {} {:?}", key, val);
}
}
Ok(())
}
================================================
FILE: crates/mojo-cli/src/main.rs
================================================
mod iget;
mod iview;
mod state;
mod commit;
mod buckets;
use anyhow::Error;
use clap::{Parser, Subcommand};
#[derive(Parser)]
#[clap(author, version, about, long_about = None)]
#[clap(propagate_version = true)]
struct Cli {
kvpath: std::path::PathBuf,
#[clap(subcommand)]
command: Commands,
}
#[derive(Subcommand)]
enum Commands {
/// View index value for a key
#[clap(name="iget")]
MemIndexGet{
#[clap(value_parser)]
bucket: String,
#[clap(value_parser)]
ver: u32,
#[clap(value_parser)]
key: u32,
},
/// View index value for a key
#[clap(name="iview")]
MemIndexView{
#[clap(value_parser)]
bucket: String,
#[clap(value_parser)]
ver: u32,
#[clap(short, action)]
additional: bool,
#[clap(short, action)]
keys: bool
},
/// View the current kv state
#[clap(name="state")]
State{
/// Print additional internal numbers
#[clap(short, action)]
additional: bool
},
/// Commit the store
#[clap(name="commit")]
Commit{
},
/// List buckets
#[clap(name="buckets")]
Buckets{
/// Version of the bmap
#[clap(value_parser)]
ver: u32,
},
}
fn main() -> Result<(), Error> {
env_logger::init();
let cli = Cli::parse();
// You can check for the existence of subcommands, and if found use their
// matches just as you would the top level cmd
match &cli.command {
Commands::MemIndexGet{bucket, ver, key} => {
iget::cmd(&cli.kvpath, bucket.as_str(), *ver, *key)?;
},
Commands::MemIndexView{bucket, ver, additional, keys} => {
iview::cmd(&cli.kvpath, bucket.as_str(), *ver, *additional, *keys)?;
},
Commands::State{additional} => {
state::cmd(&cli.kvpath, *additional)?;
},
Commands::Commit{} => {
commit::cmd(&cli.kvpath)?;
},
Commands::Buckets{ver} => {
buckets::cmd(&cli.kvpath, *ver)?;
},
}
Ok(())
}
================================================
FILE: crates/mojo-cli/src/state.rs
================================================
use anyhow::Error;
use mojokv::{self, Store};
use std::mem::size_of;
pub fn cmd(kvpath: &std::path::Path, additional: bool) -> Result<(), Error> {
let st = Store::load_state(kvpath)?;
println!("Format version : {}", st.format_ver());
println!("Minimum version : {}", st.min_ver());
println!("Active version : {}", st.active_ver());
println!("Pages per slot : {}", st.pps());
println!("Page size : {}", st.page_size());
println!("File header len : {}", st.file_page_sz());
if additional {
println!("----------------------------");
println!("Size of KVStore : {} bytes", size_of::<Store>());
println!("Size of MemIndex : {} bytes", size_of::<mojokv::index::mem::MemIndex>());
println!("Size of KeyMap : {} bytes", size_of::<mojokv::KeyMap>());
println!("Size of Value : {} bytes", size_of::<mojokv::Value>());
println!("Size of Slot : {} bytes", size_of::<mojokv::Slot>());
}
Ok(())
}
================================================
FILE: crates/mojo-cli/src/truncate.rs
================================================
use anyhow::Error;
use mojokv::KVStore;
pub fn cmd(kvpath: &std::path::Path, sz: usize) -> Result<(), Error> {
let st = KVStore::load_state(&kvpath)?;
if sz % (st.page_size() as usize) != 0 {
return Err(Error::msg(format!("Error: truncate size is not multiple of page sz ({})", st.page_size())));
}
let mut store = KVStore::writable(kvpath, st.page_size(), Some(st.pps()))?;
store.truncate(sz)?;
store.sync()?;
Ok(())
}
================================================
FILE: crates/mojofs/.gitignore
================================================
testfs
================================================
FILE: crates/mojofs/Cargo.toml
================================================
[package]
name = "mojofs"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
crate-type = ["staticlib", "lib"]
[dev-dependencies]
anyhow = "1.0"
env_logger = "0.9.0"
[dependencies]
libsqlite3-sys = {version = "0.24.2", features = ["bundled"]}
thiserror = "1.0.31"
parking_lot = "0.12.1"
mojokv = {path = "../mojokv"}
nix = "0.24"
log = "0.4.17"
env_logger = "0.9.0"
================================================
FILE: crates/mojofs/meson.build
================================================
project('cmojo', 'c')
if get_option('buildtype') == 'release'
mojo_lib_dir = meson.source_root() + '/../target/release'
else
mojo_lib_dir = meson.source_root() + '/../target/debug'
endif
message(mojo_lib_dir)
mojo_lib = meson.get_compiler('c').find_library('mojofs',
static: true,
dirs: [mojo_lib_dir])
shared_library('cmojo',
sources:['cmojo.c'],
dependencies: [mojo_lib])
================================================
FILE: crates/mojofs/src/error.rs
================================================
pub const MOJOFS_ERR_NOT_IMPL: i32 = 1;
pub const MOJOFS_ERR_IO: i32 = 2;
pub const MOJOFS_ERR_NIX: i32 = 3;
pub const MOJOFS_ERR_UTF8: i32 = 4;
pub const MOJOFS_ERR_MOJOKV: i32 = 5;
pub const MOJOFS_ERR_URL_PARSE: i32 = 6;
pub const MOJOFS_ERR_INT_PARSE: i32 = 7;
pub const MOJOFS_ERR_LARGE_PAGE: i32 = 8;
pub const MOJOFS_ERR_ARG_VER_MISSING: i32 = 9;
pub const MOJOFS_ERR_ARG_PAGESZ_MISSING: i32 = 10;
pub const MOJOFS_ERR_ARG_PPS_MISSING: i32 = 11;
#[derive(thiserror::Error, Debug)]
pub struct Error {
pub code: i32,
pub msg: String,
}
impl Error {
pub fn new(code: i32, msg: String) -> Self {
Error {
code,
msg,
}
}
pub fn not_impl() -> Self {
Error::new(MOJOFS_ERR_NOT_IMPL, "Not implemented".to_owned())
}
}
impl From<std::io::Error> for Error {
fn from(err: std::io::Error) -> Self {
Error {
code: MOJOFS_ERR_IO,
msg: format!("{:?}", err),
}
}
}
impl From<nix::Error> for Error {
fn from(err: nix::Error) -> Self {
Error {
code: MOJOFS_ERR_NIX,
msg: err.to_string(),
}
}
}
impl From<std::str::Utf8Error> for Error {
fn from(err: std::str::Utf8Error) -> Self {
Error {
code: MOJOFS_ERR_UTF8,
msg: err.to_string(),
}
}
}
impl From<std::num::ParseIntError> for Error {
fn from(err: std::num::ParseIntError) -> Self {
Error {
code: MOJOFS_ERR_INT_PARSE,
msg: err.to_string(),
}
}
}
impl From<mojokv::Error> for Error {
fn from(err: mojokv::Error) -> Self {
Error {
code: MOJOFS_ERR_MOJOKV,
msg: err.to_string(),
}
}
}
impl std::fmt::Display for Error {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "({}, {})", self.code, self.msg)
}
}
================================================
FILE: crates/mojofs/src/kvfile.rs
================================================
use mojokv::Bucket;
use crate::Error;
pub struct KVFile {
pub bucket: Bucket,
opt: KVFileOpt,
}
#[derive(Clone)]
pub struct KVFileOpt {
pub page_sz: u32,
pub pps: u32,
pub ver: u32,
}
impl KVFile {
pub fn open(bucket: Bucket, opt: KVFileOpt) -> Result<Self, Error> {
Ok(KVFile{
bucket,
opt,
})
}
pub fn pread(&self, buf: &mut [u8], off: i64) -> Result<usize, Error> {
log::debug!("kv pread o={}, blen={}", off, buf.len());
if buf.len() > self.opt.page_sz as usize {
return Err(Error::new(crate::MOJOFS_ERR_LARGE_PAGE,
format!("buf larger ({}) than page size", buf.len())));
}
let page_off = off % self.opt.page_sz as i64;
let key = off / self.opt.page_sz as i64;
let n = match self.bucket.get(key as u32, page_off as u64, buf) {
Ok(n) => n,
Err(err) => {
if let mojokv::Error::KeyNotFoundErr(_) = err {
0
}else{
return Err(err.into());
}
}
};
if n<buf.len() {
log::debug!("after kv pread o={}, blen={} n={}", off, buf.len(), n);
let _ = &mut buf[n..].fill(0);
}
Ok(n)
}
fn pwrite_page(&mut self, key: u32, page_off: u32, buf: &[u8]) -> Result<(), Error> {
log::debug!("kv pwrite page key={}, po={} blen={}", key, page_off, buf.len());
self.bucket.put(key, page_off as u64, buf)?;
Ok(())
}
pub fn pwrite(&mut self, off: i64, buf: &[u8]) -> Result<(), Error> {
log::debug!("kv pwrite o={}, blen={}", off, buf.len());
let mut po = off % self.opt.page_sz as i64;
let mut key = off / self.opt.page_sz as i64;
let mut s = 0usize;
let buflen = buf.len();
while s < buflen {
let e = (buflen-s).min(self.opt.page_sz as usize - po as usize);
self.pwrite_page(key as u32, po as u32, &buf[s..s+e])?;
s += e;
po = 0;
key += self.opt.page_sz as i64;
}
Ok(())
}
pub fn close(self) -> Result<(), Error> {
self.bucket.close()?;
Ok(())
}
pub fn sync(&mut self) -> Result<(), Error> {
self.bucket.sync()?;
Ok(())
}
pub fn filesize(&self) -> Result<u64, Error> {
Ok(self.bucket.logical_size())
}
pub fn truncate(&mut self, new_sz: u64) -> Result<(), Error> {
log::debug!("kv truncate {}", new_sz);
self.bucket.truncate(new_sz as usize)?;
Ok(())
}
}
================================================
FILE: crates/mojofs/src/lib.rs
================================================
pub mod vfs;
pub mod error;
pub mod open_options;
pub mod vfsfile;
mod native_file;
mod kvfile;
use std::ffi::CStr;
use std::collections::HashMap;
use libsqlite3_sys::{
self,
sqlite3_file,
sqlite3_vfs};
use std::ffi::c_void;
use std::os::raw::{c_int, c_char};
pub use error::*;
pub use vfs::{VFS,AccessCheck};
pub use vfsfile::VFSFile;
pub use open_options::*;
#[repr(C)]
pub struct MojoFile {
base: sqlite3_file,
custom_file: *mut c_void,
vfs: *mut sqlite3_vfs,
}
#[no_mangle]
pub extern "C" fn mojo_create() -> *mut sqlite3_vfs {
let mut name_buf = Vec::from("mojo".as_bytes());
name_buf.push(0);
let fs_name_c = name_buf.as_ptr();
std::mem::forget(name_buf);
let fs = Box::new(vfs::VFS::default());
let p_app_data = Box::into_raw(fs) as *mut c_void;
//println!("p_app_data={:?}", p_app_data);
//let sql_vfs = unsafe {
// let raw_ptr = sqlite3_malloc64(std::mem::size_of::<sqlite3_vfs>() as u64);
// raw_ptr as *mut sqlite3_vfs
//};
let vfs = Box::into_raw(Box::new(sqlite3_vfs{
iVersion: 1,
szOsFile: (std::mem::size_of::<MojoFile>()) as i32,
mxPathname: 512,
pNext: std::ptr::null_mut(),
zName: fs_name_c as *const i8,
pAppData: p_app_data,
xOpen: Some(mojo_open),
xDelete: Some(mojo_delete),
xAccess: Some(mojo_access),
xFullPathname: Some(mojo_fullname),
xDlOpen: Some(mojo_dlopen),
xDlError: Some(mojo_dlerror),
xDlSym: Some(mojo_dlsym),
xDlClose: Some(mojo_dlclose),
xRandomness: Some(mojo_randomness),
xSleep: Some(mojo_sleep),
xCurrentTime: Some(mojo_current_time),
xCurrentTimeInt64: Some(mojo_current_time64),
xGetLastError: Some(mojo_getlasterr),
xSetSystemCall: None,
xGetSystemCall: None,
xNextSystemCall: None,
}));
vfs
}
#[no_mangle]
extern "C" fn mojo_open(vfs: *mut sqlite3_vfs, zname: *const c_char, file: *mut sqlite3_file, flags: c_int, out_flags: *mut c_int) -> c_int {
let fs = getfs(vfs);
let opt = open_options::OpenOptions::from_flags(flags as i32).unwrap();
let mut out_opt = opt.clone();
let file_str = if zname.is_null() {
""
}else{
let file_rs = unsafe{std::ffi::CStr::from_ptr(zname)};
match file_rs.to_str() {
Ok(file_str) => file_str,
Err(_err) => {
println!("mojo_open error in filepath");
return libsqlite3_sys::SQLITE_CANTOPEN
},
}
};
if opt.kind.is_main() {
let query_map = match extract_query_params(zname) {
Ok(map) => map,
Err(_err) => {
return libsqlite3_sys::SQLITE_CANTOPEN
}
};
if let Err(err) = fs.init(file_str, &query_map, opt.clone()) {
log::error!("mojo_open init path={} err = {:?}", file_str, err);
return libsqlite3_sys::SQLITE_CANTOPEN
}
}
match fs.open(file_str, opt, &mut out_opt) {
Ok(vfs_file) => {
let mojo_file = unsafe {(file as *mut MojoFile).as_mut().unwrap()};
let io_methods = Box::into_raw(Box::new(libsqlite3_sys::sqlite3_io_methods{
iVersion: 1,
xClose: Some(mojo_close),
xRead: Some(mojo_read),
xWrite: Some(mojo_write),
xTruncate: Some(mojo_truncate),
xSync: Some(mojo_sync),
xFileSize: Some(mojo_filesize),
xLock: Some(mojo_lock),
xUnlock: Some(mojo_unlock),
xCheckReservedLock: Some(mojo_check_reserved_lock),
xFileControl: Some(mojo_file_control),
xSectorSize: Some(mojo_sector_size),
xDeviceCharacteristics: Some(mojo_device_char),
xShmMap: None,
xShmLock: None,
xShmBarrier: None,
xShmUnmap: None,
xFetch: None,
xUnfetch: None,
}));
mojo_file.base.pMethods = io_methods as *const libsqlite3_sys::sqlite3_io_methods;
mojo_file.custom_file = Box::into_raw(vfs_file) as *mut c_void;
mojo_file.vfs = vfs;
},
Err(err) => {
log::error!("mojo_open path={} err = {:?}", file_str, err);
return libsqlite3_sys::SQLITE_CANTOPEN;
}
}
unsafe {
if !out_flags.is_null() {
*out_flags = out_opt.flags;
}
};
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_close(sfile: *mut sqlite3_file) -> c_int {
unsafe {
let mojo_file = (sfile as *mut MojoFile).as_ref().unwrap();
let vfs_file = Box::from_raw(mojo_file.custom_file as *mut VFSFile);
let fs = getfs(mojo_file.vfs);
match fs.close(*vfs_file) {
Ok(_) => {},
Err(_err) => {
return libsqlite3_sys::SQLITE_IOERR_CLOSE
}
}
}
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_read(sfile: *mut sqlite3_file, ptr: *mut c_void, n: i32, off: i64) -> c_int {
let file = get_file(sfile);
let buf = unsafe{ std::slice::from_raw_parts_mut(ptr as *mut u8, n as usize)};
let rc = match file.pread(off as u64, buf) {
Ok(n) => {
//let m = n.min(20);
//log::debug!("after read n={} {:?}", n, &buf[..m]);
if n == buf.len() {
libsqlite3_sys::SQLITE_OK
}else{
//println!("short read");
libsqlite3_sys::SQLITE_IOERR_SHORT_READ
}
}
Err(err) => {
log::error!("mojo_read id={} off={} blen={} err={:?}", file.id(), off, buf.len(), err);
libsqlite3_sys::SQLITE_IOERR_READ
},
};
rc
}
#[no_mangle]
extern "C" fn mojo_write(sfile: *mut sqlite3_file, ptr: *const c_void, n: i32, off: i64) -> c_int {
let file = get_file_mut(sfile);
let buf = unsafe{ std::slice::from_raw_parts(ptr as *const u8, n as usize)};
let rc = match file.pwrite(off as u64, buf) {
Ok(_) => libsqlite3_sys::SQLITE_OK,
Err(err) => {
log::error!("mojo_write id={} off={} blen={} err={:?}", file.id(), off, buf.len(), err);
libsqlite3_sys::SQLITE_IOERR_WRITE
},
};
rc
}
#[no_mangle]
extern "C" fn mojo_truncate(sfile: *mut sqlite3_file, new_sz: i64) -> c_int {
let file = get_file_mut(sfile);
let rc = match file.truncate(new_sz as u64) {
Ok(_) => libsqlite3_sys::SQLITE_OK,
Err(err) => {
log::error!("mojo_truncate id={} new_sz={} err={:?}", file.id(), new_sz, err);
libsqlite3_sys::SQLITE_IOERR_WRITE
},
};
rc
}
#[no_mangle]
extern "C" fn mojo_sync(sfile: *mut sqlite3_file, flags: i32) -> c_int {
let file = get_file_mut(sfile);
let rc = match file.sync(flags) {
Ok(_) => libsqlite3_sys::SQLITE_OK,
Err(err) => {
log::error!("mojo_sync id={} err={:?}", file.id(), err);
libsqlite3_sys::SQLITE_IOERR_WRITE
},
};
rc
}
#[no_mangle]
extern "C" fn mojo_filesize(sfile: *mut sqlite3_file, out_sz: *mut i64) -> c_int {
let file = get_file(sfile);
let rc = match file.filesize() {
Ok(sz) => {
unsafe{*out_sz = sz as i64;}
libsqlite3_sys::SQLITE_OK
}
Err(_) => libsqlite3_sys::SQLITE_IOERR_WRITE,
};
rc
}
#[no_mangle]
extern "C" fn mojo_access(vfs: *mut sqlite3_vfs, zname: *const c_char, flags: c_int, resout: *mut c_int) -> c_int {
let path = match c_to_path(zname) {
Ok(path) => path,
Err(_) => {
return libsqlite3_sys::SQLITE_IOERR_CONVPATH;
}
};
let access_req = if flags == libsqlite3_sys::SQLITE_ACCESS_EXISTS {
AccessCheck::Exists
}else{
AccessCheck::ReadWrite
};
let fs = getfs(vfs);
match fs.access(&path, access_req) {
Ok(status) => {
unsafe{*resout = if status {1}else{0}}
},
Err(_) => {
return libsqlite3_sys::SQLITE_IOERR_ACCESS;
}
}
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_delete(vfs: *mut sqlite3_vfs, zname: *const c_char, _syncdir: c_int) -> c_int {
let path = match c_to_path(zname) {
Ok(path) => path,
Err(_) => {
return libsqlite3_sys::SQLITE_IOERR_DELETE;
}
};
let fs = getfs(vfs);
match fs.delete(&path) {
Ok(_) => {
libsqlite3_sys::SQLITE_OK
}
Err(err) => {
log::error!("mojo_delete path={:?} err={:?}", path, err);
libsqlite3_sys::SQLITE_IOERR_DELETE
}
}
}
#[no_mangle]
extern "C" fn mojo_fullname(vfs: *mut sqlite3_vfs, zname: *const c_char, _nout: c_int, _zout: *mut c_char) -> c_int {
let fs = getfs(vfs);
let file_rs = unsafe{std::ffi::CStr::from_ptr(zname)};
let file_str = match file_rs.to_str() {
Ok(file_str) => file_str,
Err(_err) => {
log::error!("mojo_fullname error in filepath");
return libsqlite3_sys::SQLITE_CANTOPEN;
}
};
let _ = fs.fullpath(file_str);
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_dlopen(_arg1: *mut sqlite3_vfs, _zfilename: *const c_char) -> *mut c_void {
std::ptr::null_mut()
}
#[no_mangle]
extern "C" fn mojo_dlerror(_arg1: *mut sqlite3_vfs, _nbyte: c_int, _zerrmsg: *mut c_char) {
}
//extern "C" fn (_arg1: *mut sqlite3_vfs, _arg2: *mut c_void, _zSymbol: *const c_char)
#[no_mangle]
extern "C" fn mojo_dlsym(_arg1: *mut sqlite3_vfs, _arg2: *mut c_void, _zsymbol: *const c_char)
-> Option<unsafe extern "C" fn(*mut sqlite3_vfs, *mut c_void, *const i8)> {
None
}
#[no_mangle]
extern "C" fn mojo_dlclose(_arg1: *mut sqlite3_vfs, _arg2: *mut c_void) {
}
#[no_mangle]
extern "C" fn mojo_randomness(_arg1: *mut sqlite3_vfs, _nbyte: c_int, _zout: *mut c_char) -> c_int {
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_sleep(_arg1: *mut sqlite3_vfs, microseconds: c_int) -> c_int {
std::thread::sleep(std::time::Duration::from_micros(microseconds as u64));
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_current_time(_arg1: *mut sqlite3_vfs, p: *mut f64) -> c_int {
let now = (std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap()).as_secs() as f64;
unsafe {
*p = (2440587.5 + now / 864.0e5) * 864.0e5;
}
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_current_time64(_arg1: *mut sqlite3_vfs, p: *mut i64) -> c_int {
let now = (std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap()).as_secs() as f64;
unsafe {
*p = ((2440587.5 + now / 864.0e5) * 864.0e5) as i64;
}
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_getlasterr(_arg1: *mut sqlite3_vfs, _arg2: c_int, _arg3: *mut c_char) -> c_int {
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_lock(_sfile: *mut sqlite3_file, _flags: c_int) -> c_int {
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_unlock(_sfile: *mut sqlite3_file, _flags: c_int) -> c_int {
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_check_reserved_lock(_sfile: *mut sqlite3_file, res_out: *mut c_int) -> c_int {
unsafe{*res_out = 0;}
libsqlite3_sys::SQLITE_OK
}
#[no_mangle]
extern "C" fn mojo_file_control(_sfile: *mut sqlite3_file, _op: c_int, _arg: *mut c_void) -> c_int {
libsqlite3_sys::SQLITE_NOTFOUND
}
#[no_mangle]
extern "C" fn mojo_sector_size(_sfile: *mut sqlite3_file) -> c_int {
0
}
#[no_mangle]
extern "C" fn mojo_device_char(_sfile: *mut sqlite3_file) -> c_int {
libsqlite3_sys::SQLITE_OK
}
fn getfs(vfs: *mut sqlite3_vfs) -> &'static mut VFS {
let fs = unsafe{
((*vfs).pAppData as *mut VFS).as_mut().unwrap()
};
fs
}
fn get_file_mut(sfile: *mut sqlite3_file) -> &'static mut VFSFile {
let file = unsafe{
let mojo_file = (sfile as *mut MojoFile).as_mut().unwrap();
(mojo_file.custom_file as *mut VFSFile).as_mut().unwrap()
};
file
}
fn get_file(sfile: *mut sqlite3_file) -> &'static VFSFile {
let file = unsafe{
let mojo_file = (sfile as *mut MojoFile).as_ref().unwrap();
(mojo_file.custom_file as *mut VFSFile).as_ref().unwrap()
};
file
}
fn c_to_path(cpath: *const c_char) -> Result<std::path::PathBuf, Error> {
let file_rs = unsafe{std::ffi::CStr::from_ptr(cpath)};
let file_str = file_rs.to_str()?;
Ok(std::path::PathBuf::from(file_str))
}
#[no_mangle]
pub extern "C" fn mojofs_init_log() {
env_logger::init();
}
fn extract_query_params(filepath: *const c_char) -> Result<HashMap<String, String>, Error> {
let mut map = HashMap::new();
if filepath.is_null() {
return Ok(map);
}
let mut itr: *const c_char = filepath;
let mut parse_key = true;
let mut key: &str = "dummy";
let mut value: &str;
unsafe {
while key.len() > 0 {
while *itr != 0 {
itr = itr.add(1);
}
itr = itr.add(1);
let s = CStr::from_ptr(itr).to_str()?;
if parse_key {
key = s;
parse_key = false;
}else{
value = s;
parse_key = true;
map.insert(key.to_owned(), value.to_owned());
}
}
}
Ok(map)
}
================================================
FILE: crates/mojofs/src/native_file.rs
================================================
use std::path::{Path, PathBuf};
use nix::fcntl::{self, OFlag};
use nix::sys::stat::Mode;
use crate::open_options::*;
use crate::Error;
pub struct NativeFile {
path: PathBuf,
fd: i32,
opt: OpenOptions,
}
impl NativeFile {
pub fn open(path: &Path, opt: &OpenOptions) -> Result<Self, Error> {
let open_flags = match opt.access {
OpenAccess::Read => {
OFlag::O_RDONLY
},
OpenAccess::Write => {
OFlag::O_RDWR
},
OpenAccess::Create => {
OFlag::O_CREAT|OFlag::O_RDWR
},
OpenAccess::CreateNew => {
OFlag::O_CREAT|OFlag::O_RDWR|OFlag::O_EXCL
}
};
let file_perm = Mode::all();
let fd = fcntl::open(path, open_flags, file_perm)?;
Ok(NativeFile{
path: path.to_owned(),
opt: opt.clone(),
fd,
})
}
pub fn pread(&self, buf: &mut [u8], off: i64) -> Result<usize, Error> {
log::debug!("native pread fd={} o={}, blen={}", self.fd, off, buf.len());
let mut i = 0;
while i<buf.len() {
let n = nix::sys::uio::pread(self.fd, &mut buf[i..], off + i as i64)?;
if n == 0 {
break;
}
i += n;
}
if i<buf.len() {
let _ = &mut buf[i..].fill(0);
}
Ok(i)
}
pub fn pwrite(&mut self, off: i64, buf: &[u8]) -> Result<(), Error> {
log::debug!("native pwrite fd={} o={}, blen={}", self.fd, off, buf.len());
let mut i=0;
while i<buf.len() {
let n = nix::sys::uio::pwrite(self.fd, &buf[i..], off + i as i64)?;
i += n;
}
Ok(())
}
pub fn close(self) -> Result<(), Error> {
nix::unistd::close(self.fd)?;
if self.opt.delete_on_close {
nix::unistd::unlink(&self.path)?;
}
Ok(())
}
pub fn sync(&mut self) -> Result<(), Error> {
nix::unistd::fsync(self.fd)?;
Ok(())
}
pub fn filesize(&self) -> Result<u64, Error> {
let st = nix::sys::stat::fstat(self.fd)?;
Ok(st.st_size as u64)
}
pub fn truncate(&mut self, new_sz: u64) -> Result<(), Error> {
log::debug!("truncate id={} {}", self.fd, new_sz);
nix::unistd::ftruncate(self.fd, new_sz as i64)?;
Ok(())
}
}
================================================
FILE: crates/mojofs/src/open_options.rs
================================================
use libsqlite3_sys as ffi;
#[derive(Debug, Clone, PartialEq)]
pub struct OpenOptions {
pub flags: i32,
/// The object type that is being opened.
pub kind: OpenKind,
/// The access an object is opened with.
pub access: OpenAccess,
/// The file should be deleted when it is closed.
pub delete_on_close: bool,
}
/// The object type that is being opened.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum OpenKind {
MainDb,
MainJournal,
TempDb,
TempJournal,
TransientDb,
SubJournal,
SuperJournal,
Wal,
}
impl OpenKind {
pub fn is_main(&self) -> bool {
*self == OpenKind::MainDb
}
}
/// The access an object is opened with.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum OpenAccess {
/// Read access.
Read,
/// Write access (includes read access).
Write,
/// Create the file if it does not exist (includes write and read access).
Create,
/// Create the file, but throw if it it already exist (includes write and read access).
CreateNew,
}
impl OpenOptions {
pub fn from_flags(flags: i32) -> Option<Self> {
Some(OpenOptions {
flags,
kind: OpenKind::from_flags(flags)?,
access: OpenAccess::from_flags(flags)?,
delete_on_close: flags & ffi::SQLITE_OPEN_DELETEONCLOSE > 0,
})
}
}
impl OpenKind {
fn from_flags(flags: i32) -> Option<Self> {
match flags {
flags if flags & ffi::SQLITE_OPEN_MAIN_DB > 0 => Some(Self::MainDb),
flags if flags & ffi::SQLITE_OPEN_MAIN_JOURNAL > 0 => Some(Self::MainJournal),
flags if flags & ffi::SQLITE_OPEN_TEMP_DB > 0 => Some(Self::TempDb),
flags if flags & ffi::SQLITE_OPEN_TEMP_JOURNAL > 0 => Some(Self::TempJournal),
flags if flags & ffi::SQLITE_OPEN_TRANSIENT_DB > 0 => Some(Self::TransientDb),
flags if flags & ffi::SQLITE_OPEN_SUBJOURNAL > 0 => Some(Self::SubJournal),
flags if flags & ffi::SQLITE_OPEN_SUPER_JOURNAL > 0 => Some(Self::SuperJournal),
flags if flags & ffi::SQLITE_OPEN_WAL > 0 => Some(Self::Wal),
_ => None,
}
}
}
impl OpenAccess {
fn from_flags(flags: i32) -> Option<Self> {
match flags {
flags
if (flags & ffi::SQLITE_OPEN_CREATE > 0)
&& (flags & ffi::SQLITE_OPEN_EXCLUSIVE > 0) =>
{
Some(Self::CreateNew)
}
flags if flags & ffi::SQLITE_OPEN_CREATE > 0 => Some(Self::Create),
flags if flags & ffi::SQLITE_OPEN_READWRITE > 0 => Some(Self::Write),
flags if flags & ffi::SQLITE_OPEN_READONLY > 0 => Some(Self::Read),
_ => None,
}
}
}
================================================
FILE: crates/mojofs/src/vfs.rs
================================================
use std::path::{PathBuf, Path};
use crate::{error, Error};
use crate::open_options::*;
use std::collections::HashMap;
use crate::kvfile::{KVFile, KVFileOpt};
use crate::vfsfile::FileImpl;
use mojokv::{Store, BucketOpenMode};
use crate::vfsfile::VFSFile;
#[derive(Debug)]
pub enum AccessCheck {
Exists,
ReadWrite,
}
#[derive(Default)]
pub struct VFS {
store: Option<Store>,
file_counter: usize,
fopt: FSOptions,
}
impl VFS {
pub fn name(&self) -> String {
return "mojo".to_owned()
}
pub fn fs_options(&self) -> FSOptions {
self.fopt.clone()
}
pub fn active_ver(&self) -> u32 {
self.store.as_ref().unwrap().active_ver()
}
pub fn init(&mut self, root_path: &str, params: &HashMap<String, String>, opt: OpenOptions) -> Result<(), Error> {
log::debug!("init: root_path={} params={:?} opt={:?}", root_path, params, opt);
self.fopt = FSOptions::parse(params)?;
let root_path = Path::new(root_path);
if opt.access == OpenAccess::Read {
self.store = Some(Store::readonly(root_path, self.fopt.ver)?);
log::debug!("store opened in readonly mode");
}else{
self.store = Some(Store::writable(root_path, true, Some(self.fopt.pagesz), Some(self.fopt.pps))?);
log::debug!("store opened writable mode");
}
Ok(())
}
pub fn open(&mut self, filepath: &str, opt: OpenOptions, _out_opt: &mut OpenOptions) -> Result<Box<VFSFile>, Error> {
log::debug!("open: file={} opt={:?}", filepath, opt);
self.file_counter += 1;
let id = self.file_counter;
let file_path = if filepath.len() == 0{
std::path::PathBuf::from(format!("mojo.tmp.{}", id))
}else{
std::path::PathBuf::from(filepath)
};
let store = self.store.as_mut().unwrap();
let bucket_name = Self::bucket_name(&file_path);
let bmode = if let OpenAccess::Read = opt.access {
BucketOpenMode::Read
}else{
BucketOpenMode::Write
};
let b = store.open(bucket_name, bmode)?;
let kvfileopt = self.fopt.to_kvfile_opt();
let f = KVFile::open(b, kvfileopt)?;
let fimpl = FileImpl::KV(f);
log::debug!("open: file={} id={} done", filepath, id);
Ok(Box::new(VFSFile::new(id, bucket_name, opt, fimpl)))
}
pub fn fullpath(&mut self, filepath: &str) -> Result<PathBuf, Error> {
log::debug!("fullpath filepath={}", filepath);
let filepath_rs = std::path::Path::new(filepath);
if filepath_rs.is_absolute() {
Ok(filepath_rs.to_owned())
}else{
let mut cwd = std::env::current_dir()?;
cwd.push(filepath_rs);
Ok(cwd)
}
}
fn bucket_name(path: &Path) -> &str {
path.file_name().unwrap().to_str().unwrap()
}
//TODO: add sync dir
pub fn delete(&mut self, path: &std::path::Path) -> Result<(), Error> {
log::debug!("delete path={:?}", path);
let name = Self::bucket_name(path);
let store = self.store.as_mut().unwrap();
store.delete(name)?;
Ok(())
}
pub fn access(&self, path: &std::path::Path, req: AccessCheck) -> Result<bool, Error> {
log::debug!("access path={:?} req={:?}", path, req);
let name = Self::bucket_name(path);
let store = self.store.as_ref().unwrap();
let status = store.exists(name);
log::debug!("access path={:?} status={}", path, status);
Ok(status)
}
pub fn close(&mut self, f: VFSFile) -> Result<(), Error> {
let fid = f.id();
log::debug!("close id={}", f.id());
let bucket_name = f.bucket.clone();
let opt = f.opt();
f.close()?;
let store = self.store.as_mut().unwrap();
if opt.delete_on_close {
log::debug!("close_on_delete is set for id={}", fid);
store.delete(bucket_name.as_str())?;
}
Ok(())
}
pub fn commit(&mut self) -> Result<(), Error> {
let store = self.store.as_mut().unwrap();
store.commit()?;
Ok(())
}
}
#[derive(Default, Clone)]
pub struct FSOptions {
pub ver: u32,
pub pagesz: u32,
pub pps: u32,
}
impl FSOptions {
fn parse(map: &HashMap<String, String>) -> Result<FSOptions, Error> {
let mut opt = FSOptions {ver: 0, pagesz: 4096, pps: 0};
opt.ver = match map.get("ver") {
Some(s) => s.parse()?,
None => 1
};
let s = map.get("pagesz").ok_or(Error::new(error::MOJOFS_ERR_ARG_PAGESZ_MISSING,
"fs arg page size missing".to_owned()))?;
opt.pagesz = s.parse()?;
opt.pps = match map.get("pps") {
Some(s) => s.parse()?,
None => 65536
};
Ok(opt)
}
fn to_kvfile_opt(&self) -> KVFileOpt {
KVFileOpt {
ver: self.ver,
page_sz: self.pagesz,
pps: self.pps,
}
}
}
================================================
FILE: crates/mojofs/src/vfsfile.rs
================================================
use crate::error::Error;
use crate::native_file::NativeFile;
use crate::kvfile::KVFile;
use crate::open_options::OpenOptions;
pub enum FileImpl {
Reg(NativeFile),
KV(KVFile)
}
pub struct VFSFile {
pub bucket: String,
id: usize,
fimpl: FileImpl,
opt: OpenOptions,
}
impl VFSFile {
pub fn new(id: usize, name: &str, opt: OpenOptions, fimpl: FileImpl) -> Self {
VFSFile{
bucket: name.to_owned(),
id,
fimpl,
opt,
}
}
pub fn id(&self) -> usize {
self.id
}
pub fn opt(&self) -> OpenOptions {
self.opt.clone()
}
pub fn pread(&self, off: u64, buf: &mut [u8]) -> Result<usize, Error> {
log::debug!("pread id={} o={}, blen={}", self.id, off, buf.len());
let nread = match &self.fimpl {
FileImpl::Reg(f) => {
f.pread(buf, off as i64)?
},
FileImpl::KV(f) => {
f.pread(buf, off as i64)?
}
};
Ok(nread)
}
pub fn pwrite(&mut self, off: u64, buf: &[u8]) -> Result<(), Error> {
log::debug!("pwrite id={} o={}, blen={}", self.id, off, buf.len());
match &mut self.fimpl {
FileImpl::Reg(f) => {
f.pwrite(off as i64, buf)?;
},
FileImpl::KV(f) => {
f.pwrite(off as i64, buf)?;
}
};
Ok(())
}
pub fn close(self) -> Result<(), Error> {
log::debug!("file close id={}", self.id);
match self.fimpl {
FileImpl::Reg(f) => {
f.close()?;
},
FileImpl::KV(f) => {
f.close()?;
}
};
Ok(())
}
pub fn sync(&mut self, flags: i32) -> Result<(), Error> {
log::debug!("sync id={} flags={}", self.id, flags);
match &mut self.fimpl {
FileImpl::Reg(f) => {
f.sync()?;
},
FileImpl::KV(f) => {
f.sync()?;
}
};
Ok(())
}
pub fn filesize(&self) -> Result<u64, Error> {
let sz = match &self.fimpl {
FileImpl::Reg(f) => {
f.filesize()?
},
FileImpl::KV(f) => {
f.filesize()?
}
};
Ok(sz)
}
pub fn truncate(&mut self, new_sz: u64) -> Result<(), Error> {
log::debug!("truncate id={} {}", self.id, new_sz);
match &mut self.fimpl {
FileImpl::Reg(f) => {
f.truncate(new_sz)?;
},
FileImpl::KV(f) => {
f.truncate(new_sz)?;
}
};
Ok(())
}
pub fn lock(&mut self, _flag: i32) -> Result<(), Error> {
Ok(())
}
pub fn unlock(&mut self, _flag: i32) -> Result<(), Error> {
Ok(())
}
pub fn check_reserved_lock(&self) -> Result<i32, Error> {
Ok(0)
}
pub fn file_control(&mut self, _op: i32) -> Result<(), Error> {
Ok(())
}
pub fn sector_size(&self) -> Result<i32, Error> {
Ok(0)
}
pub fn device_char(&self) -> Result<(), Error> {
Ok(())
}
}
================================================
FILE: crates/mojofs/tests/mojofs_test.rs
================================================
use std::path::Path;
use anyhow::Error;
use std::collections::HashMap;
use mojofs::{self, VFS, VFSFile};
fn remove_fs(rootpath: &Path) -> Result<(), Error> {
if let Err(err) = std::fs::remove_dir_all(rootpath) {
if err.kind() != std::io::ErrorKind::NotFound {
return Err(err.into());
}
}
Ok(())
}
fn setup() -> Result<String, Error> {
let path = Path::new("./testfs");
remove_fs(path)?;
Ok(path.to_owned().to_str().unwrap().to_owned())
}
fn default_params(pagesz: u32) -> HashMap<String, String> {
let mut h = HashMap::new();
h.insert("ver".to_owned(), "1".to_owned());
h.insert("pagesz".to_owned(), format!("{}", pagesz));
h.insert("pps".to_owned(), "65536".to_owned());
h
}
fn read_test(a: &mut VFSFile, nitems: usize, pagesz: u64, f: fn(usize) -> usize) -> Result<(), Error> {
let mut buf = [0u8; 8];
for i in 0usize..nitems {
let off = i as u64 * pagesz;
let n = f(i);
a.pread(off, &mut buf)?;
assert_eq!(n, usize::from_be_bytes(buf));
}
Ok(())
}
fn write_test(a: &mut VFSFile, nitems: usize, pagesz: u64, f: fn(usize) -> usize) -> Result<(), Error> {
for i in 0usize..nitems {
let off = i as u64 * pagesz;
let n = f(i);
a.pwrite(off, &n.to_be_bytes())?;
}
a.sync(1)?;
Ok(())
}
fn write_read(a: &mut VFSFile, nitems: usize, pagesz: u64, f: fn(usize) -> usize) -> Result<(), Error> {
write_test(a, nitems, pagesz, f)?;
read_test(a, nitems, pagesz, f)?;
Ok(())
}
#[test]
fn rw_same_version() -> Result<(), Error> {
env_logger::init();
let fspath = setup()?;
let mut fs_uri_opt = default_params(8);
let mut opt = mojofs::OpenOptions::from_flags(326).unwrap();
let nitems = 10;
{
let mut fs = VFS::default();
fs.init(&fspath, &fs_uri_opt, opt.clone())?;
let fsopt = fs.fs_options();
let mut a = fs.open("a", opt.clone(), &mut opt)?;
assert_eq!(fsopt.pagesz, 8);
write_read(&mut a, nitems, fsopt.pagesz as u64, |n| n)?;
assert_eq!(a.filesize()?, (fsopt.pagesz as u64) * nitems as u64);
a.close()?;
fs.commit()?;
assert_eq!(fs.active_ver(), 2);
let mut a = fs.open("a", opt.clone(), &mut opt)?;
write_read(&mut a, nitems, fsopt.pagesz as u64, |n| n+10)?;
assert_eq!(a.filesize()?, (fsopt.pagesz as u64) * nitems as u64);
a.close()?;
fs.commit()?;
assert_eq!(fs.active_ver(), 3);
}
{
let mut fs = VFS::default();
opt.access = mojofs::OpenAccess::Read;
fs_uri_opt.insert("ver".to_owned(), "1".to_owned());
fs.init(&fspath, &fs_uri_opt, opt.clone())?;
let fsopt = fs.fs_options();
let mut a = fs.open("a", opt.clone(), &mut opt)?;
read_test(&mut a, nitems, fsopt.pagesz as u64, |n| n)?;
}
{
let mut fs = VFS::default();
opt.access = mojofs::OpenAccess::Read;
fs_uri_opt.insert("ver".to_owned(), "2".to_owned());
fs.init(&fspath, &fs_uri_opt, opt.clone())?;
let fsopt = fs.fs_options();
let mut a = fs.open("a", opt.clone(), &mut opt)?;
read_test(&mut a, nitems, fsopt.pagesz as u64, |n| n+10)?;
}
Ok(())
}
================================================
FILE: crates/mojoio/Cargo.toml
================================================
[package]
name = "mojoio"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
nix = "0.24"
log = "0.4.17"
thiserror = "1.0.31"
parking_lot = {version = "0.12", features=["serde"]}
================================================
FILE: crates/mojoio/src/error.rs
================================================
#[derive(thiserror::Error, Debug)]
pub enum Error {
#[error("io error")]
IoErr(#[from] std::io::Error),
#[error("Unix error")]
NixErr(#[from] nix::Error),
#[error("Parse int error")]
ParseIntErr(#[from] std::num::ParseIntError),
#[error("Unknown error `{0}`")]
UnknownStr(String),
#[error("UTF8 error")]
UTF8Err(#[from] std::str::Utf8Error),
#[error("Unknown error")]
Unknown,
}
================================================
FILE: crates/mojoio/src/lib.rs
================================================
pub mod nix;
mod error;
pub use error::Error;
pub const BUFFER_MAGIC: &[u8] = b"mojo";
pub const PAGE_HEADER_LEN: usize = 8;
pub fn add(left: usize, right: usize) -> usize {
left + right
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn it_works() {
let result = add(2, 2);
assert_eq!(result, 4);
}
}
================================================
FILE: crates/mojoio/src/nix.rs
================================================
use std::path::Path;
use nix::fcntl::{self, OFlag};
use nix::sys::stat::Mode;
//use crate::value;
use crate::Error;
pub struct NixFile {
file_fd: i32,
curr_off: u64,
page_header_buf: [u8; crate::PAGE_HEADER_LEN],
page_header: PageHeader,
}
impl NixFile {
pub fn open(filepath: &Path, _file_no: u32) -> Result<Self, Error> {
let open_flags = OFlag::O_CREAT|OFlag::O_RDWR;
let file_perm = Mode::all();
let file_fd = fcntl::open(filepath, open_flags, file_perm)?;
let curr_off = nix::unistd::lseek(file_fd, 0, nix::unistd::Whence::SeekEnd)? as u64;
log::debug!("open path={:?} fd={}", filepath, file_fd);
Ok(NixFile {
file_fd,
curr_off,
page_header_buf: [0; crate::PAGE_HEADER_LEN],
page_header: PageHeader::new(),
})
}
pub fn close(&mut self) -> Result<(), Error> {
log::debug!("close fd={}", self.file_fd);
nix::unistd::close(self.file_fd)?;
Ok(())
}
pub fn write_buf_at(&mut self, off: u64, block_no: u32, buf: &[u8]) -> Result<(), Error> {
self.page_header.block_no = block_no;
self.page_header.encode(&mut self.page_header_buf);
let header_io = std::io::IoSlice::new(&self.page_header_buf);
let buf_io = std::io::IoSlice::new(buf);
let io_bufs = [header_io, buf_io];
log::debug!("file write at fd={} off={} {}", self.file_fd, off, buf.len());
let n = nix::sys::uio::pwritev(self.file_fd, &io_bufs, off as i64)?;
if n < header_io.len() + buf_io.len() {
return Err(Error::UnknownStr("vectored write did not write all data".to_owned()));
}
Ok(())
}
pub fn write_buf(&mut self, block_no: u32, poff: u64, buf: &[u8]) -> Result<u64, Error> {
self.write_buf_at(self.curr_off, block_no, buf)?;
let page_off = self.curr_off;
//let page_off = self.curr_off;
self.curr_off += buf.len() as u64 + NixFile::header_len() as u64;
self.curr_off += poff;
//self.curr_off += buf.len() as u64;
Ok(page_off)
}
pub fn header_len() -> usize {
return crate::PAGE_HEADER_LEN;
}
fn read_all_at(&self, off: u64, buf: &mut [u8]) -> Result<usize, Error> {
let mut i = 0;
while i < buf.len() {
//TODO: Should n==0 be handled?
let n = nix::sys::uio::pread(self.file_fd, &mut buf[i..], off as i64 + i as i64)?;
if n == 0 {
break;
}
i += n;
}
Ok(i)
}
pub fn read_buf_at(&self, off: u64, buf: &mut [u8]) -> Result<usize, Error> {
log::debug!("file read at fd={} off={} {}", self.file_fd, off, buf.len());
let n = self.read_all_at(off, buf)?;
Ok(n)
}
pub fn sync(&self) -> Result<(), Error> {
log::debug!("sync fd={}", self.file_fd);
nix::unistd::fsync(self.file_fd)?;
Ok(())
}
}
struct PageHeader {
magic: &'static [u8],
block_no: u32,
}
impl PageHeader {
fn new() -> PageHeader {
PageHeader {
magic: crate::BUFFER_MAGIC,
block_no: 0,
}
}
pub fn encode(&mut self, buf: &mut [u8; crate::PAGE_HEADER_LEN]) {
let _ = &buf[..4].copy_from_slice(self.magic);
let _ = &buf[4..8].copy_from_slice(&self.block_no.to_le_bytes());
//let _ = &buf[12..13].copy_from_slice(&self.flags.to_le_bytes());
//let _ = &buf[13..17].copy_from_slice(&self.file_no.to_le_bytes());
}
/*
pub fn decode(buf: &[u8]) -> Result<PageHeader, Error> {
let magic = &buf[..4];
if magic != BUFFER_MAGIC {
return Err(Error::UnknownStr("Invalid buffer magic".to_string()));
}
let mut tmp_buf = [0u8; 8];
tmp_buf.copy_from_slice(&buf[4..12]);
let block_no = u64::from_be_bytes(tmp_buf);
let mut tmp_buf = [0u8; 2];
tmp_buf.copy_from_slice(&buf[12..14]);
let size = u16::from_be_bytes(tmp_buf);
tmp_buf.copy_from_slice(&buf[12..14]);
let flags = buf[15];
let mut tmp_buf = [0u8; 4];
tmp_buf.copy_from_slice(&buf[15..19]);
let file_no = u32::from_be_bytes(tmp_buf);
Ok(PageHeader{
magic: BUFFER_MAGIC,
block_no,
size,
flags,
file_no,
})
}
*/
}
================================================
FILE: crates/mojokv/.gitignore
================================================
/target
/Cargo.lock
*.db
log*
*.log
================================================
FILE: crates/mojokv/Cargo.toml
================================================
[package]
name = "mojokv"
version = "0.2.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
mojoio = {path="../mojoio"}
modular-bitfield = "0.11"
lz4 = "1.23.3"
zstd = "0.11.2"
thiserror = "1.0.31"
log = "0.4.17"
env_logger = "0.9.0"
rand = "0.8.5"
clap = {version="3.2.6", features=["derive"] }
anyhow = "1.0.58"
parking_lot = {version = "0.12", features=["serde"]}
serde = { version = "1.0", features = ["derive", "rc"] }
serde_json = "1.0"
rmp-serde = "1.1.0"
fslock = "0.2.1"
rustc-hash = "1.1.0"
================================================
FILE: crates/mojokv/src/bmap.rs
================================================
use std::path::{Path, PathBuf};
use std::collections::HashMap;
use std::sync::Arc;
use crate::bucket::Bucket;
use crate::Error;
use parking_lot::RwLock;
use serde::{Serialize, Deserialize};
#[derive(Clone, Default, Debug, Serialize, Deserialize)]
pub struct BucketMap {
map: Arc<RwLock<HashMap<String, u32>>>,
}
impl BucketMap {
pub fn add(&self, name: &str, ver: u32) {
log::debug!("add name={} ver={} {:?}", name, ver, self.map);
let mut map = self.map.write();
//self.buckets.insert(name.to_owned(), b);
map.insert(name.to_owned(), ver);
}
pub fn exists(&self, name: &str) -> bool {
let map = self.map.read();
map.contains_key(name)
}
pub fn get(&self, name: &str) -> Option<u32>{
log::debug!("get name={}", name);
let map = self.map.read();
map.get(name).map(|v| *v)
}
pub fn delete(&self, root_path: &Path, name: &str, ver: u32) -> Result<(), Error> {
log::debug!("delete name={} {:?}", name, self.map);
let mut map = self.map.write();
map.remove(name);
Bucket::delete_ver(root_path, name, ver)?;
Ok(())
}
pub fn map(&self) -> Result<HashMap<String, u32>, Error> {
let map = self.map.read();
Ok(map.clone())
}
pub fn serialize_to_path(&self, path: &Path) -> Result<(), Error> {
let buf = serde_json::to_vec(&self)?;
log::debug!("serializing bmap={:?}", std::str::from_utf8(&buf));
crate::utils::write_file(path, &buf)?;
Ok(())
}
pub fn deserialize_from_path(path: &Path) -> Result<Self, Error> {
let mut buf = Vec::new();
crate::utils::load_file(path, &mut buf)?;
let map = serde_json::from_slice(&buf)?;
Ok(map)
}
fn bmap_path(root_path: &Path, ver: u32) -> PathBuf {
root_path.join(&format!("mojo.bmap.{}", ver))
}
pub fn load(root_path: &Path, ver: u32) -> Result<Self, Error> {
let bmap_path = Self::bmap_path(root_path, ver);
log::debug!("loading bmap from path={:?}", bmap_path);
let bmap = Self::deserialize_from_path(&bmap_path)?;
Ok(bmap)
}
}
================================================
FILE: crates/mojokv/src/bucket.rs
================================================
use std::collections::HashSet;
use std::path::{Path, PathBuf};
use crate::{Error, BucketMap};
use mojoio::nix::NixFile;
use crate::index::mem::MemIndex;
use crate::value::Value;
use crate::state::State;
pub struct BucketInner {
name: String,
root_path: PathBuf,
index: MemIndex,
file_page_sz: usize,
fmap: FileMap,
is_dirty: bool,
is_modified: bool,
is_closed: bool,
active_ver: u32,
}
impl BucketInner {
fn active_file(&mut self, ver: u32) -> &mut NixFile {
self.fmap.file_mut(ver)
}
fn sync_index(&mut self, ver: u32) -> Result<(), Error> {
let non_ref_vers =self.index.update_min_max_ver();
log::debug!("closing versions={:?} as they are no longer referenced", non_ref_vers);
self.fmap.close_versions(&non_ref_vers, self.active_ver)?;
let index_path = Bucket::index_path(&self.root_path, self.name.as_str(), ver);
log::debug!("syncing index ver={} {:?}", ver, index_path);
self.index.serialize_to_path(&index_path)?;
log::debug!("syncing index ver={} done", ver);
Ok(())
}
}
pub struct Bucket {
state: State,
//inner: Arc<RwLock<BucketInner>>,
inner: BucketInner,
bmap: BucketMap,
is_write: bool,
}
impl Bucket {
fn with_inner(state: State, inner: BucketInner, bmap: BucketMap) -> Self {
Bucket {
state,
//inner: Arc::new(RwLock::new(inner)),
inner,
bmap,
is_write: false,
}
}
pub fn set_writable(&mut self) {
self.is_write = true
}
pub fn readonly(root_path: &Path, name: &str, ver: u32, state: State, bmap: BucketMap) -> Result<Bucket, Error> {
log::debug!("bucket name={} readonly at ver={}", name, ver);
let b = Self::load(root_path, name, state, bmap, ver)?;
Ok(b)
}
fn index_path(rootpath: &Path, name: &str, ver: u32) -> PathBuf {
rootpath.join(&format!("{}_i.{}", name, ver))
}
pub fn get_key(&self, key: u32) -> Result<Option<Value>, Error> {
//let inner = self.inner.read();
Ok(self.inner.index.get(key)?.map(|v| v.clone()))
}
pub fn max_key(&self) -> isize {
//let inner = self.inner.read();
self.inner.index.max_key()
}
pub fn is_modified(&self) -> bool {
//let inner = self.inner.read();
self.inner.is_modified
}
pub fn writable(root_path: &Path, name: &str, state: State, bmap: BucketMap, load_ver: u32) -> Result<Bucket, Error> {
log::debug!("mojo initing bucket pps={}", state.pps());
let aver = state.active_ver();
let index_path = Self::index_path(root_path, name, load_ver);
let mut b = if index_path.exists() {
log::debug!("bucket index for version={} exists", load_ver);
Self::load(root_path, name, state, bmap, aver)?
}else{
log::debug!("creating new bucket at ver={}", aver);
let mut b = Self::new(root_path, name, state, bmap)?;
b.sync()?;
b
};
b.set_writable();
log::debug!("mojo state={:?}", b.state);
Ok(b)
}
pub fn load(root_path: &Path, name: &str, state: State, bmap: BucketMap, ver: u32) -> Result<Self, Error> {
log::debug!("loading bucket={} version={}", name, ver);
if ver < state.min_ver() || ver > state.active_ver() {
return Err(Error::VersionNotFoundErr(ver));
}
let (_, _, mut index) = Self::load_index(root_path, name, ver)?;
let fmap = FileMap::init(root_path, name, &index.header().vset, state.active_ver())?;
index.set_active_ver(state.active_ver());
let file_page_sz = state.page_size() as usize + NixFile::header_len();
let inner = BucketInner {
name: name.to_owned(),
root_path: root_path.to_owned(),
index,
file_page_sz,
fmap,
is_dirty: false,
is_modified: false,
is_closed: false,
active_ver: state.active_ver(),
};
log::debug!("mojo load version done");
Ok(Bucket::with_inner(state, inner, bmap))
}
pub fn load_index(root_path: &Path, name: &str, ver: u32) -> Result<(usize, usize, MemIndex), Error> {
let index_path = Self::index_path(root_path, name, ver);
log::debug!("loading index={:?} for name={} at ver={}", index_path, name, ver);
if !index_path.exists() {
return Err(Error::BucketNotAtVerErr(name.to_owned(), ver));
}
let index = MemIndex::deserialize_from_path(&index_path)?;
Ok(index)
}
pub fn new(root_path: &Path, name: &str, state: State, bmap: BucketMap) -> Result<Self, Error> {
log::debug!("creating new bucket name={} at ver={}", name, state.active_ver());
let _ = std::fs::create_dir_all(root_path)?;
let index = MemIndex::new(state.pps() as usize);
let fmap = FileMap::init(root_path, name, &index.header().vset, state.active_ver())?;
let mut inner = BucketInner {
name: name.to_owned(),
root_path: root_path.to_owned(),
index,
file_page_sz: state.page_size() as usize + NixFile::header_len(),
fmap,
is_dirty: false,
is_modified: false,
is_closed: false,
active_ver: state.active_ver(),
};
inner.index.set_active_ver(state.active_ver());
let b = Bucket::with_inner(state, inner, bmap);
Ok(b)
}
pub fn logical_size(&self) -> u64 {
//let inner = self.inner.read();
(self.state.page_size() as isize * (self.inner.index.max_key() + 1)) as u64
}
pub fn close(mut self) -> Result<(), Error> {
//let mut inner = self.inner.write();
if self.inner.is_closed {
return Ok(())
}
self.inner.fmap.close()?;
self.inner.is_closed = true;
Ok(())
}
pub fn truncate(&mut self, new_sz: usize) -> Result<(), Error> {
let _ = self.state.commit_lock.read();
//let mut inner = self.inner.write();
log::debug!("truncate bucket={} new_sz={}", self.inner.name, new_sz);
let pages = new_sz/(self.state.page_size() as usize);
//let real_sz = pages * self.file_page_sz;
self.inner.index.truncate(pages as u32)?;
self.inner.is_modified = true;
//TODO: Delete blocks from file
//self.active_file().truncate(real_sz)?;
Ok(())
}
fn put_at(&mut self, key: u32, page_off: u64, buf: &[u8], val: &Value) -> Result<(), Error> {
let mut off = val.get_off() as u64;
off = off * self.inner.file_page_sz as u64;
off += page_off;
let file = self.inner.active_file(self.state.active_ver());
file.write_buf_at(off, key, buf)?;
Ok(())
}
pub fn put(&mut self, key: u32, page_off: u64, buf: &[u8]) -> Result<(), Error> {
if !self.is_write {
return Err(Error::BucketNotWritableErr);
}
if self.inner.active_ver < self.state.active_ver() {
return Err(Error::VerNotWritable(self.inner.active_ver, self.state.active_ver()));
}
let _ = self.state.commit_lock.read();
log::debug!("store put aver={} key={}, buflen={}", self.state.active_ver(), key, buf.len());
let val_opt = self.get_value_opt(key)?.map(|v| v.clone());
match val_opt {
Some(val) => {
//let mut inner = self.inner.write();
log::debug!("store put value exists value={:?}", val);
if val.get_ver() == self.state.active_ver() {
self.put_at(key, page_off, buf, &val)?;
self.inner.index.put(key, val.get_off())?;
}else{
let file = self.inner.active_file(self.state.active_ver());
let write_off = file.write_buf(key, page_off, buf)?;
let block_no = (write_off/(self.inner.file_page_sz as u64)) as u32;
log::debug!("bucket put was done at block_no={} old value={:?}", block_no, val);
self.inner.index.put(key, block_no)?;
}
self.inner.is_dirty = true;
self.inner.is_modified = true;
},
None => {
//let mut inner = self.inner.write();
let file = self.inner.active_file(self.state.active_ver());
let write_off = file.write_buf(key, page_off, buf)?;
let block_no = (write_off/(self.inner.file_page_sz as u64)) as u32;
self.inner.index.put(key, block_no)?;
self.inner.is_dirty = true;
self.inner.is_modified = true;
log::debug!("store put value not present. value={:?}", block_no);
}
}
Ok(())
}
pub fn get(&self, key: u32, page_off: u64, out_buf: &mut [u8]) -> Result<usize, Error> {
//let inner = self.inner.read();
let value = self.get_value(key)?;
let mut read_off = (value.get_off() as u64) * (self.inner.file_page_sz as u64);
read_off += NixFile::header_len() as u64 + page_off;
let read_ver = value.get_ver();
log::debug!("get name={} key={} value: {:?}", self.inner.name, key, value);
let file = self.inner.fmap.file(read_ver);
let n = file.read_buf_at(read_off, out_buf)?;
log::debug!("get name={} key={} n={}", self.inner.name, key, n);
Ok(n)
}
fn get_value_opt(&self, key: u32) -> Result<Option<Value>, Error> {
//let inner = self.inner.read();
match self.inner.index.get(key)? {
None => {
log::debug!("get_value_opt no slot key={}", key);
return Ok(None)
}
Some(val) => {
if !val.is_allocated() {
log::debug!("get_value_opt allocated key={}", key);
return Ok(None)
}else{
return Ok(Some(val.clone()))
}
}
}
}
fn get_value(&self, key: u32) -> Result<Value, Error> {
self.get_value_opt(key)?.ok_or(Error::KeyNotFoundErr(key))
}
pub (crate) fn sync_no_commit_lock(&mut self) -> Result<(), Error> {
if !self.is_write {
return Err(Error::StoreNotWritableErr);
}
//let mut inner = self.inner.write();
log::debug!("syncing bucket={} at ver={}", self.inner.name, self.state.active_ver());
self.bmap.add(&self.inner.name, self.state.active_ver());
self.inner.active_file(self.state.active_ver()).sync()?;
self.inner.sync_index(self.state.active_ver())?;
self.inner.is_dirty = false;
log::debug!("syncing done");
Ok(())
}
pub fn sync(&mut self) -> Result<(), Error> {
let _ = self.state.commit_lock.read();
self.sync_no_commit_lock()
}
pub fn delete_ver(root_path: &Path, name: &str, ver: u32) -> Result<(), Error> {
log::debug!("Deleting bucket name={} ver={}", name, ver);
let index_path = Self::index_path(root_path, name, ver);
log::debug!("removing index file={:?}", index_path);
std::fs::remove_file(index_path)?;
let data_path = FileMap::data_path(&root_path, name, ver);
log::debug!("removing data file={:?}", data_path);
std::fs::remove_file(data_path)?;
Ok(())
}
}
struct FileMap {
fmap: rustc_hash::FxHashMap<u32,NixFile>,
}
impl FileMap {
fn init(root_path: &Path, name: &str, vset: &HashSet<u32>, aver: u32) -> Result<Self, Error> {
//let active_file = Self::open_active_file(root_path, name, active_ver)?;
log::debug!("fmap initing for name={} with vset={:?}", name, vset);
let mut fmap = FileMap {
fmap: rustc_hash::FxHashMap::default(),
};
for ver in vset.iter() {
if *ver != aver {
fmap.add_file(root_path, name, *ver)?;
}
}
fmap.add_file(root_path, name, aver)?;
Ok(fmap)
}
fn close(&mut self) -> Result<(), Error> {
for (_v, f) in &mut self.fmap {
f.close()?;
}
Ok(())
}
fn close_versions(&mut self, vlist: &Vec<u32>, aver: u32) -> Result<(), Error> {
for v in vlist {
if *v == aver {
continue
}
if let Some(mut f) = self.fmap.remove(v) {
f.close()?;
}
}
Ok(())
}
fn data_path(root_path: &Path, name: &str, ver: u32) -> PathBuf {
root_path.join(format!("{}_d.{}", name, ver))
}
fn add_file(&mut self, root_path: &Path, name: &str, ver: u32) -> Result<(), Error> {
let ver_path = Self::data_path(root_path, name, ver);
log::debug!("adding new file: {:?}", ver_path);
let f = NixFile::open(&ver_path, ver)?;
self.fmap.insert(ver, f);
Ok(())
}
fn file_mut(&mut self, ver: u32) -> &mut NixFile {
self.fmap.get_mut(&ver).expect(&format!("write ver={} not found", ver))
}
fn file(&self, ver: u32) -> &NixFile {
&self.fmap.get(&ver).expect(&format!("read ver={} not found", ver))
}
}
================================================
FILE: crates/mojokv/src/error.rs
================================================
#[derive(thiserror::Error, Debug)]
pub enum Error {
#[error("io error")]
IoErr(#[from] std::io::Error),
#[error("mojo file error")]
MojoFileErr(#[from] mojoio::Error),
#[error("Bucket {0} not found at ver={1}")]
BucketNotAtVerErr(String, u32),
#[error("Bucket not writable")]
BucketNotWritableErr,
#[error("Version no longer writable bucket ver={0} active ver={1}")]
VerNotWritable(u32, u32),
#[error("Store not found")]
StoreNotFoundErr,
#[error("Store not writable")]
StoreNotWritableErr,
#[error("Missing arguments")]
MissingArgsErr,
#[error("Commit lock could not be acquired")]
CommitLockedErr,
#[error("Only single version exists")]
SingleVersionErr,
#[error("Json serialization error")]
SerdeJsonErr(#[from] serde_json::Error),
#[error("rmp encode error")]
RmpEncodeErr(#[from] rmp_serde::encode::Error),
#[error("rmp decode error")]
RmpDecodeErr(#[from] rmp_serde::decode::Error),
#[error("Key {0} not found")]
KeyNotFoundErr(u32),
#[error("Key {0} not multiple of page size")]
KeyNotMultipleErr(u32),
#[error("Version {0} not found")]
VersionNotFoundErr(u32),
#[error("Parse int error")]
ParseIntErr(#[from] std::num::ParseIntError),
#[error("Unknown error `{0}`")]
UnknownStr(String),
#[error("UTF8 error")]
UTF8Err(#[from] std::str::Utf8Error),
#[error("Unknown error")]
Unknown,
}
================================================
FILE: crates/mojokv/src/index/mem.rs
================================================
use std::io::Write;
use serde::Deserialize;
use serde::Serialize;
use crate::value::Value;
use crate::keymap::KeyMap;
use crate::Error;
use crate::utils;
use super::IndexHeader;
//TODO: Reserve some space for additional data
#[derive(Serialize, Deserialize)]
pub struct MemIndex {
header: IndexHeader,
pub kmap: KeyMap
}
impl MemIndex {
pub fn new(pps: usize) -> Self {
MemIndex {
header: IndexHeader::new(pps),
kmap: KeyMap::new(pps),
}
}
pub fn header(&self) -> &IndexHeader {
&self.header
}
fn key_map(&self) -> &KeyMap {
&self.kmap
}
pub fn set_active_ver(&mut self, ver: u32) {
self.header.vset.insert(ver);
self.header.active_ver = ver;
}
pub fn active_ver(&self) -> u32 {
self.header.active_ver
}
pub fn max_key(&self) -> isize {
self.header.max_key
}
pub fn update_min_max_ver(&mut self) -> Vec<u32> {
let (min_ver, max_ver, vset) = self.kmap.get_min_max_ver();
self.header.min_ver = min_ver;
self.header.max_ver = max_ver;
let non_ref_vers: Vec<u32> = self.header.vset.difference(&vset).map(|n| *n).collect();
non_ref_vers
}
pub fn put(&mut self, key: u32, off: u32) -> Result<(), Error> {
let mut val = Value::new();
val.put_off(off);
val.put_ver(self.header.active_ver);
log::debug!("index put val:{:?}", val);
self.header.max_key = self.header.max_key.max(key as isize);
self.kmap.put(key, val);
Ok(())
}
pub fn get(&self, key: u32) -> Result<Option<&Value>, Error> {
Ok(self.kmap.get(key))
}
pub fn truncate(&mut self, key: u32) -> Result<(), Error> {
self.kmap.truncate(key);
self.header.max_key = key as isize -1;
Ok(())
}
pub fn iter<'a>(&'a self, from_key: u32, to_key: u32) -> Box<dyn Iterator<Item=(u32, &'a Value)> + 'a > {
let itr = MemIndexIterator {
key: from_key,
to_key,
index: self
};
Box::new(itr)
}
pub fn serialize_to_path(&self, filepath: &std::path::Path) -> Result<(), Error> {
let tmp_buf = rmp_serde::to_vec(&self)?;
let cbuf = zstd::bulk::compress(&tmp_buf, 3)?;
let mut f = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.open(filepath)?;
let cap_buf = tmp_buf.len().to_le_bytes();
f.write_all(&cap_buf)?;
f.write_all(&cbuf)?;
f.sync_data()?;
Ok(())
}
pub fn deserialize_from_path(filepath: &std::path::Path) -> Result<(usize, usize, MemIndex), Error> {
let mut b = Vec::new();
utils::load_file(filepath, &mut b)?;
let cap = usize::from_le_bytes([b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7]]);
let buf = zstd::bulk::decompress(&b[8..], cap)?;
let index = rmp_serde::from_slice(&buf)?;
Ok((cap, b.len(), index))
}
}
pub struct MemIndexIterator<'a> {
index: &'a MemIndex,
key: u32,
to_key: u32
}
impl<'a> MemIndexIterator<'a> {
pub fn new(from_key: u32, to_key: u32, index: &'a MemIndex) -> Self {
MemIndexIterator {
index,
key: from_key,
to_key,
}
}
}
impl<'a> Iterator for MemIndexIterator<'a> {
type Item = (u32, &'a Value);
fn next(&mut self) -> Option<Self::Item> {
if self.to_key > 0 && self.key >= self.to_key {
return None;
}
let kmap_index = self.key/self.index.header.pps as u32;
if kmap_index as usize >= self.index.key_map().slot_map.len() {
return None;
}
let slot_map = &self.index.key_map().slot_map[kmap_index as usize];
let ret = match slot_map {
Some(map) => {
let slot_index = (self.key as usize) %self.index.header.pps;
if slot_index >= map.len() {
return None;
}
let val = &map[slot_index];
if val.is_allocated() {
Some((self.key, val))
}else{
None
}
},
None => None
};
self.key += 1;
ret
}
}
================================================
FILE: crates/mojokv/src/index/mod.rs
================================================
pub mod mem;
use std::collections::HashSet;
use crate::Error;
use crate::value::Value;
use serde::{Serialize, Deserialize};
pub const MOJO_INDEX_MAGIC: &'static str = "mojo_index";
#[derive(Debug, Serialize, Deserialize)]
pub struct IndexHeader {
pub magic: String,
pub format_ver: u32,
pub min_ver: u32,
pub max_ver: u32,
pub vset: HashSet<u32>,
pub active_ver: u32,
pub max_key: isize,
pub pps: usize,
}
impl IndexHeader {
pub fn new(pps: usize) -> Self {
let mut vset = HashSet::new();
vset.insert(1);
IndexHeader {
magic: MOJO_INDEX_MAGIC.to_owned(),
format_ver: 1,
min_ver: 1,
max_ver: 1,
vset,
active_ver: 1,
pps,
max_key: -1,
}
}
}
pub trait Index {
fn put(&mut self, key: u32, off: u32) -> Result<(), Error>;
fn get(&self, key: u32) -> Result<Option<&Value>, Error>;
fn truncate(&mut self, key: u32) -> Result<(), Error>;
}
pub trait IndexSerde {
fn serialize<I: Index, W: std::io::Write>(idx: &I, w: &mut W) -> Result<(), Error>;
fn deserialize<I: Index, R: std::io::Read>(idx: &I, r: &mut R) -> Result<I, Error>;
}
================================================
FILE: crates/mojokv/src/keymap.rs
================================================
use std::collections::HashSet;
use crate::value::{Value, Slot};
use serde::{Serialize, Deserialize};
#[derive(Serialize, Deserialize)]
pub struct KeyMap {
pub slot_map: Vec<Slot>,
pps: usize
}
impl KeyMap {
//TODO: remove this flag
pub fn new(pps: usize) -> Self {
KeyMap {
slot_map: Vec::new(),
pps,
}
}
fn alloc_value_arr(pps: usize) -> Vec<Value> {
let v = vec![Value::new(); pps];
v
}
pub fn get_min_max_ver(&self) -> (u32, u32, HashSet<u32>) {
let mut set = HashSet::new();
let (mut min_ver, mut max_ver) = (u32::MAX,0);
for slot in self.slot_map.iter() {
if let Some(slot) = slot {
for val in slot.iter() {
let v = val.get_ver();
if v == 0 {
break;
}
set.insert(v);
min_ver = min_ver.min(v);
max_ver = max_ver.max(v);
}
}
}
(min_ver, max_ver, set)
}
pub fn put(&mut self, key: u32, val: Value) {
let slot = (key as usize)/self.pps;
if slot >= self.slot_map.len() {
log::debug!("KeyMap put key={}, value= {:?} slot={} kmaplen={}", key, val, slot, self.slot_map.len());
for _ in 0..(slot - self.slot_map.len() + 1) {
self.slot_map.push(None);
}
}
let val_arr = self.slot_map[slot].get_or_insert_with(||{
KeyMap::alloc_value_arr(self.pps)
});
let slot_key = key % (self.pps as u32);
log::debug!("KeyMap put slot_key={}", slot_key);
val_arr[slot_key as usize] = val;
}
pub fn get(&self, key: u32) -> Option<&Value> {
let slot = key/self.pps as u32;
log::debug!("KeyMap get key={}, slot={}, kmaplen={}", key, slot, self.slot_map.len());
if slot as usize >= self.slot_map.len() {
return None;
}
self.slot_map[slot as usize].as_ref().map(|val_arr|{
let slot_key = key % self.pps as u32;
&val_arr[slot_key as usize]
})
}
pub fn truncate(&mut self, key: u32) {
let slot = key/self.pps as u32;
self.slot_map.truncate((slot+1) as usize);
let slot_key = key % (self.pps as u32);
if let Some(slot_vec) = self.slot_map[slot as usize].as_mut() {
for i in slot_key as usize ..slot_vec.len() {
slot_vec[i].deallocate();
}
}
}
}
================================================
FILE: crates/mojokv/src/lib.rs
================================================
//#![feature(write_all_vectored)]
pub mod index;
mod bucket;
mod error;
mod value;
mod state;
mod keymap;
mod utils;
mod store;
mod bmap;
pub use error::Error;
pub use bucket::Bucket;
pub use bmap::BucketMap;
pub use keymap::KeyMap;
pub use value::{Value, Slot};
pub use store::{Store, BucketOpenMode};
//TODO: Pass pps from single place
================================================
FILE: crates/mojokv/src/state.rs
================================================
use crate::Error;
use mojoio::nix::NixFile;
use crate::utils;
use std::sync::Arc;
use parking_lot::RwLock;
use serde::{Serialize, Deserialize};
#[derive(Debug, Serialize, Deserialize)]
pub struct StateInner {
pub format_ver: u32,
pub min_ver: u32,
pub max_ver: u32,
pub active_ver: u32,
pub pps: u32,
pub page_sz: u32,
pub file_header_len: u32,
pub file_page_sz: u32,
//TODO: add timestamp
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct State {
inner: Arc<RwLock<StateInner>>,
#[serde(skip)]
pub commit_lock: Arc<RwLock<bool>>,
}
impl State {
pub fn new(page_sz: u32, pps: u32) -> Self {
let inner = StateInner {
format_ver: 1,
min_ver: 1,
max_ver: 1,
active_ver: 1,
pps,
page_sz,
file_header_len: NixFile::header_len() as u32,
file_page_sz: page_sz + NixFile::header_len() as u32,
};
let state = State {
inner: Arc::new(RwLock::new(inner)),
commit_lock: Arc::new(RwLock::new(false)),
};
state
}
pub fn format_ver(&self) -> u32 {
let inner = self.inner.read();
inner.format_ver
}
pub fn active_ver(&self) -> u32 {
let inner = self.inner.read();
inner.active_ver
}
pub fn page_size(&self) -> u32 {
let inner = self.inner.read();
inner.page_sz
}
pub fn file_page_sz(&self) -> u32 {
let inner = self.inner.read();
inner.file_page_sz
}
pub fn pps(&self) -> u32 {
let inner = self.inner.read();
inner.pps
}
pub fn min_ver(&self) -> u32 {
let inner = self.inner.read();
inner.min_ver
}
pub fn max_ver(&self) -> u32 {
let inner = self.inner.read();
inner.max_ver
}
pub fn advance_ver(&self) -> u32 {
let mut inner = self.inner.write();
inner.active_ver += 1;
inner.max_ver = inner.max_ver.max(inner.active_ver);
inner.active_ver
}
pub fn serialize_to_path(&self, filepath: &std::path::Path) -> Result<(), Error> {
let buf = rmp_serde::to_vec_named(&self)?;
utils::write_file(filepath, &buf)?;
Ok(())
}
pub fn deserialize_from_path(filepath: &std::path::Path) -> Result<State, Error> {
let mut buf = Vec::new();
utils::load_file(filepath, &mut buf)?;
let state = rmp_serde::from_slice(&buf)?;
Ok(state)
}
}
================================================
FILE: crates/mojokv/src/store.rs
================================================
use std::path::{Path, PathBuf};
use std::sync::Arc;
use crate::{Error, utils};
use crate::state::State;
use crate::bucket::Bucket;
use crate::bmap::BucketMap;
use crate::index::mem::MemIndex;
use parking_lot::RwLock;
use fslock::LockFile;
struct StoreInner {
root_path: PathBuf,
state: State,
is_write: bool,
bmap: BucketMap,
}
pub struct Store {
inner: Arc<RwLock<StoreInner>>,
}
impl Store {
pub fn exists(&self, name: &str) -> bool {
let inner = self.inner.read();
inner.bmap.exists(name)
}
pub fn open(&self, name: &str, mode: BucketOpenMode) -> Result<Bucket, Error> {
let mut inner = self.inner.write();
log::debug!("store bucket open name={} mode writable={} store is write: {}", name, mode.is_write(), inner.is_write);
if !inner.is_write && mode.is_write() {
return Err(Error::StoreNotWritableErr);
}
let mut b = match inner.bmap.get(name) {
Some(v) => {
log::debug!("Bucket name={} exists at ver={}", name, v);
Bucket::load(&inner.root_path, name, inner.state.clone(), inner.bmap.clone(), v)?
},
None => {
log::debug!("Bucket name={} does not exists", name);
if !inner.is_write {
return Err(Error::StoreNotWritableErr);
}
Bucket::new(&inner.root_path, name, inner.state.clone(), inner.bmap.clone())?
}
};
if inner.is_write && mode.is_write() {
log::debug!("setting bucket={} to writable", name);
b.set_writable();
b.sync()?;
}
if mode.is_write() {
inner.sync_bmap()?;
}
Ok(b)
}
pub fn delete(&self, name: &str) -> Result<(), Error> {
let mut inner = self.inner.write();
let aver = inner.state.active_ver();
inner.bmap.delete(&inner.root_path, name, aver)?;
inner.sync_bmap()
}
pub fn commit(&self) -> Result<u32, Error> {
let mut inner = self.inner.write();
log::debug!("committing store ver={}", inner.state.active_ver());
let _ = inner.state.commit_lock.write();
log::debug!("about to acquire commit file lock ver={}", inner.state.active_ver());
let mut commit_lock_file = Self::create_lock_file(&inner.root_path)?;
if !commit_lock_file.try_lock_with_pid()? {
return Err(Error::CommitLockedErr);
}
let new_ver = inner.state.advance_ver();
inner.sync_state()?;
inner.sync_bmap()?;
log::debug!("committing store done");
Ok(new_ver)
}
pub fn active_ver(&self) -> u32 {
let inner = self.inner.read();
inner.state.active_ver()
}
pub fn load_state(rootpath: &Path) -> Result<State, Error> {
let state_path = rootpath.join("mojo.state");
log::debug!("loading state from {:?}", state_path);
let state = State::deserialize_from_path(&state_path)?;
Ok(state)
}
pub fn readonly(root_path: &Path, ver: u32) -> Result<Self, Error> {
log::debug!("opening store in readonly mode at ver={}", ver);
let state = Self::load_state(root_path)?;
Self::load_store(root_path, state, ver)
}
pub fn writable(rootpath: &Path, create: bool, page_sz: Option<u32>, pps: Option<u32>) -> Result<Store, Error> {
let init_path = rootpath.join("mojo.init");
if create && (page_sz.is_none() || pps.is_none()) {
log::debug!("Missing mandatory params page_sz:{:?} pps:{:?}", page_sz, pps);
return Err(Error::MissingArgsErr);
}
let store = if !init_path.exists() {
if !create {
return Err(Error::StoreNotFoundErr);
}
log::debug!("Store does not exists. Initing now");
let mut store = Store::new(rootpath, page_sz.unwrap(), pps.unwrap())?;
store.init()?;
log::debug!("Store init successfull");
store
}else{
let state = Self::load_state(rootpath)?;
let aver = state.active_ver();
Self::load_store(rootpath, state, aver)?
};
{
let mut inner = store.inner.write();
inner.is_write = true;
}
Ok(store)
}
fn load_store(root_path: &Path, state: State, ver: u32) -> Result<Store, Error> {
log::debug!("loading store at ver={}", ver);
let bmap = BucketMap::load(root_path, ver)?;
let inner = StoreInner {
root_path: root_path.to_owned(),
state,
is_write: false,
bmap,
};
let store = Store {inner: Arc::new(RwLock::new(inner))};
Ok(store)
}
pub fn get_index(&self, name: &str) -> Result<Option<(usize, usize, MemIndex)>, Error> {
let inner = self.inner.read();
match inner.bmap.get(name) {
Some(v) => {
let ret = Bucket::load_index(&inner.root_path, name, v)?;
Ok(Some(ret))
},
None => {
log::debug!("Bucket name={} does not exists", name);
return Ok(None)
}
}
}
fn new(root_path: &Path, page_sz: u32, pps: u32) -> Result<Self, Error> {
let state = State::new(page_sz, pps);
let inner = StoreInner {
root_path: root_path.to_owned(),
state,
is_write: false,
bmap: BucketMap::default(),
};
let store = Store {
inner: Arc::new(RwLock::new(inner)),
};
Ok(store)
}
fn init(&mut self) -> Result<(), Error> {
let mut inner = self.inner.write();
std::fs::create_dir_all(&inner.root_path)?;
inner.sync()?;
let touch_file = inner.root_path.join("mojo.init");
utils::touch_file(&touch_file)?;
Ok(())
}
fn create_lock_file(root_path: &Path) -> Result<LockFile, Error> {
let lock_path = root_path.join("mojo.lock");
log::debug!("creating lock file: {:?}", lock_path);
Ok(LockFile::open(&lock_path)?)
}
}
impl StoreInner {
fn sync(&mut self) -> Result<(), Error> {
self.sync_state()?;
self.sync_bmap()?;
Ok(())
}
fn sync_bmap(&mut self) -> Result<(), Error> {
log::debug!("syncing bmap at ver={}", self.state.active_ver());
let bmap_path = self.root_path.join(&format!("mojo.bmap.{}", self.state.active_ver()));
self.bmap.serialize_to_path(&bmap_path)?;
Ok(())
}
fn sync_state(&mut self) -> Result<(), Error> {
let file_path = self.root_path.join("mojo.state");
log::debug!("syncing state ver={} {:?}", self.state.active_ver(), file_path);
self.state.serialize_to_path(&file_path)?;
log::debug!("syncing state done");
Ok(())
}
}
#[derive(Clone, Debug, PartialEq)]
pub enum BucketOpenMode {
Read,
Write,
}
impl BucketOpenMode {
pub fn is_write(&self) -> bool {
*self == Self::Write
}
}
================================================
FILE: crates/mojokv/src/utils.rs
================================================
use std::path::Path;
use std::io::{Read, Write};
use crate::Error;
pub fn load_file(path: &Path, buf: &mut Vec<u8>) -> Result<(), Error> {
let mut f = std::fs::OpenOptions::new().read(true).open(path)?;
f.read_to_end(buf)?;
Ok(())
}
pub fn write_file(path: &Path, buf: &[u8]) -> Result<(), Error> {
let mut f = std::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.open(path)?;
f.write_all(buf)?;
f.sync_data()?;
Ok(())
}
pub fn touch_file(path: &Path) -> Result<(), Error> {
log::debug!("creating init file: {:?}", path);
let _ = std::fs::File::create(path)?;
log::debug!("creating init file done");
Ok(())
}
/*
pub fn read_le_u32<R: std::io::Read>(r: &mut R) -> Result<u32, Error> {
let mut buf = [0u8; 4];
r.read_exact(&mut buf)?;
Ok(u32::from_le_bytes(buf))
}
pub fn read_le_isize<R: std::io::Read>(r: &mut R) -> Result<isize, Error> {
let mut buf = [0u8; std::mem::size_of::<isize>()];
r.read_exact(&mut buf)?;
Ok(isize::from_le_bytes(buf))
}
pub fn read_le_usize<R: std::io::Read>(r: &mut R) -> Result<usize, Error> {
let mut buf = [0u8; std::mem::size_of::<isize>()];
r.read_exact(&mut buf)?;
Ok(usize::from_le_bytes(buf))
}
*/
================================================
FILE: crates/mojokv/src/value.rs
================================================
use modular_bitfield::{bitfield, specifiers::*};
use serde::{Serialize, Deserialize};
use serde::de::Visitor;
pub type Slot = Option<Vec<Value>>;
#[derive(Clone, Copy)]
#[bitfield]
pub struct Value {
off: B32,
ver: B24
}
impl std::fmt::Debug for Value {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {
f.write_fmt(format_args!("o={},v={}", self.off(), self.ver()))
}
}
impl Value {
pub fn is_allocated(&self) -> bool {
self.ver() > 0
}
pub fn deallocate(&mut self) {
self.set_off(0);
self.set_ver(0);
}
pub fn put_off(&mut self, off: u32) {
self.set_off(off);
}
pub fn get_off(&self) -> u32 {
self.off()
}
pub fn put_ver(&mut self, v: u32) {
self.set_ver(v);
}
pub fn get_ver(&self) -> u32 {
self.ver() as u32
}
}
impl Serialize for Value {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: serde::Serializer {
serializer.serialize_bytes(&self.into_bytes())
}
}
struct ValueVisitor {}
impl<'de> Visitor<'de> for ValueVisitor {
type Value = Value;
fn visit_bytes<E>(self, v: &[u8]) -> Result<Self::Value, E>
where
E: serde::de::Error, {
if v.len() != 7 {
return Err(serde::de::Error::invalid_length(v.len(), &self));
}
Ok(Value::from_bytes([v[0], v[1], v[2], v[3], v[4], v[5], v[6]]))
}
fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {
formatter.write_str("byte array of 7 bytes")
}
}
impl<'de> Deserialize<'de> for Value {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: serde::Deserializer<'de> {
deserializer.deserialize_bytes(ValueVisitor{})
}
}
/*
pub fn serialize_valuearr<W: std::io::Write>(val_opt: &Option<Vec<Value>>, w: &mut W) -> Result<(), Error> {
match val_opt {
Some(val) => {
let n_items = val.len() as u32;
w.write_all(&n_items.to_le_bytes())?;
for elem in val.iter() {
rmp_serde::encode::write(w, elem)?;
}
},
None => {
w.write_all(&0u32.to_le_bytes())?;
}
}
Ok(())
}
pub fn deserialize_valuearr<R: std::io::Read>(r: &mut R, pps: usize) -> Result<Option<Vec<Value>>, Error> {
let mut tmp_buf = [0u8; 4];
r.read_exact(&mut tmp_buf)?;
let count = u32::from_le_bytes(tmp_buf);
if count == 0 {
Ok(None)
}else{
if count as usize != pps {
return Err(Error::UnknownStr("Less number of values than expected".to_owned()));
}
let mut tmp_vec = Vec::new();
for _ in 0..count {
let val = rmp_serde::decode::from_read(r)?;
tmp_vec.push(val);
}
Ok(Some(tmp_vec))
}
}
*/
================================================
FILE: docs/design.md
================================================
- [Design choices](#design-choices)
- [Design](#design)
- [Layers](#layers)
- [MojoKV](#mojokv)
- [Index](#index)
- [MojoFS](#mojofs)
# Design choices
Mojofs is a versioning file-system for sqlite database. It is a completely tailor made for sqlite and is not
to be used as a general purpose file-system. This allows the fs to make certain assumptions which in turn drives design choices.
Following are the assumptions about any sqlite fs, which I think are reasonable:
* Small number of files
* Large files
* Very flat folder structure
* Read/write in terms of fixed page size (or there multiple) dominates compared to any random offset.
This influences the way we store namespace (i.e. folder/file names), the index (i.e. mapping between pages and offsets) and internal decoupling (e.g mojofs itself doesn't do much but instead uses mojokv as storage layer)
# Design
## Layers
The layers of mojofs are:
Sqlite -> Mojofs extension lib -> mojofs -> mojokv
The sqlite could be the cli (i.e. sqlite3 binary) or sqlite C API or any of its bindings.
The fs is developed as both an sqlite extension and vfs, which is compiled down to a shared library.
This shared library is loaded as extension which then registers the VFS=mojo with the sqlite.
Mojofs implements the sqlite's [VFS interface](https://www.sqlite.org/vfs.html) which asks file system like apis to be implemented.
The fs uses mojokv (Mojo Key-Value store) to actually store the data. The KV has a notion of 'bucket' which the fs creates for each new file asked by sqlite. All the buckets i.e files taken together are versioned.
The MojoKV is tailor made for the needs of sqlite and as such is not a general purpose key-value store.
## MojoKV
MojoKV is the core storage layer which handles the index and actual data files.
The KV has a notion of bucket, on which read/write happens.
Each bucket has an index which is a mapping of (Page No) => (New Page No, version)
### Index
The write api at [File IO methods](https://www.sqlite.org/c3ref/io_methods.html) looks like below:
```
int (*xWrite)(sqlite3_file*, const void*, int iAmt, sqlite3_int64 iOfst);
```
The `sqlite3_file*` is the file handle, `const void*, int iAmt` are the pointer to data and its length. The `iOfst` is the offset at which data needs to be written. Mojofs has versions/snapshots and the sqlite API does not know anything about it. This means that when write is called, mojofs needs to know to which version the data should be written.
The sqlite divides the file into pages. This is configurable when db is created for the first time, but assume 4KB for this document. We can logically think of a file and its versions as below:
| |1|2|3|4|
|------|-|-|-|-|
|Page 0|1|2|3|4|
|Page 1|1|2| |4|
|Page 2|1| | | |
|Page 3|1| |3|4|
|Page 4|1| |3| |
The file above has 5 pages of 4KB each and are depicted as rows. The columns are the version numbers.
The value for page 0 and version 2 is 2. This means that the page 0 was modified in version 2.
For page 2 and version 2 the value is empty and it means the page was not modified from previous version.
When the file is created new, naturally all the pages will be marked as version 1.
When page 2 and version=3 needs to be read, it actually needs to be fetch the page from version=1.
Essentially the page no should map to a certain location on disk.
For simple, unversioned file, the page number translates to an offset in file i.e. page no x page size.
But for versioned file, the mapping of between the tuple (Pg No, Version) => \<Location where page is stored\> is needed.
This nicely yields itself to be stored in a key-value store.
The actual data-structure used is `Vec<Option<Vec<Slot>>>`. The entire bucket/file is split into slots and the inner/nested vector (`Vec<Slot>`) is called slot map. Each slot corresponds to a page of sqlite. A slot is a tuple of (physical page number, version). There is a maximum number of slots controlled by a tunable called `pages per slot`.
The version 3 for the file in the above example is represented as below (with `pages per slot` = 2)
Assume page size=4KB.
```
Slot 0:
index 0 = (0,3)
index 1 = (1,2)
Slot 1:
index 0 = (2,1)
index 1 = (2,3)
Slot 2:
index 0 = (2,3)
```
The outer vector consists of 3 vectors, one each corresponding to one slot. The key (which is a page no) is converted to slot vector and index within each slot vector. So the writing and reading from from index is O(1).
The value is a tuple (physical page no in a version file, version). Since for each version a separate file is created, the page number in the tuple is the physical page no and version in which the page was last modified/created.
## MojoFS
Each sqlite database is created as a directory instead of a single file.
For each file name = F and version = V the mojokv stores the file with name F.V.
The filename (=F) is chosen by the fs.
So for a given sqlite db 'a.db' being created for the first time the fs will create the following on disk:
```
sudeep@local-3 mojo % tree ./a.db
./a.db
├── a.db_d.1
├── a.db_i.1
├── mojo.bmap.1
├── mojo.init
└── mojo.state
...
```
So the fs creates the dir = `a.db`. The sqlite issues open call for the main db file i.e. test.db.
The fs adds `_d` (d=data) to the name creates `a.db_d.1`. The `.1` is the version.
The `a.db_i.1` is the index file, which is internal to the kv. This has the mapping of (page no, version) => Physical offset.
When a version 2 is created it will create a file: `a.db_d.2`. This file will contain only those pages which were modified in that version.
As a result it will also create the index file `a.db_i.2`.
The `mojo.*` files are files created/for the mojokv:
`mojo.init` => Presence of this file indicates that filesystem was properly initialized.
`mojo.state` => It stores the current state of the filesystem, like how many versions, current version, etc
`mojo.bmap.1` => Bucket map. This stores namespace of the mojokv.
================================================
FILE: docs/source.md
================================================
Whether you want to just read or contribute to mojo, this document describes the code to get you started.
*__Note__*: This is not a design document.
## Bird's eye view
* All the rust code is under `crates` folder
* All the docs are under `docs` folder
* The `sqlite-ext` folder has C code which is compiled down to shared lib
* The `test-scripts` has various assorted test scripts which includes perf & black-box test
## Crates
### mojokv
This is the KV which powers the mojofs.
* `store.rs` has the main store object. Buckets are "opened" using a store object
* `bucket.rs` has the bucket object. A bucket has get & put methods. Each bucket has an index.
* `index/mem.rs` has the `MemIndex` which is in-memory index which has the mapping `offset -> (physical offset, version)`. Each index has KeyMap.
* `keymap.rs` The index is split into slots and a vector such slots are wrapped in KeyMap.
* `state.rs` has the state object which reflects the current state of the kv
### mojoio
Abstracts out the notion of file. This is the code which does the actual IO. It will have different implementations including remote KV store.
* `nix.rs` implements unix based file
### mojofs
Mojofs is the filesystem which is powered by mojokv. Each user file in fs maps to a bucket in mojokv.
* `vfs.rs` has FS like operations like `open`, `delete`, `access`, etc
* `kvfile.rs` has file like object which is implemented using mojokv, hence the name.
* `native_file.rs` is the regular passthrough file object (uses std read/write)
* `vfsfile.rs` has the object VFSFile which either is a kvfile or nativefile. At present everything is kvfile. The native file will be used for transient/temp files which does not need versioning. This is an optimization.
* `lib.rs` has vfs functions needed by sqlite e.g. `fn mojo_read(sfile: *mut sqlite3_file, ptr: *mut c_void, n: i32, off: i64)`
### mojo-cli
This a CLI utility to control the mojokv. Each command in mojo-cli maps to a `*.rs` file. Example: `commit` command can be found in `commit.rs`
================================================
FILE: docs/user-guide.md
================================================
This document assumes that you have followed the build instructions and/or have the libmojo shared library.
- [Opening/Creating the database](#openingcreating-the-database)
- [Committing database](#committing-database)
- [Committing MojoFS vs Committing Database](#committing-mojofs-vs-committing-database)
- [Reading old version](#reading-old-version)
## Opening/Creating the database
Mojofs is compiled down to a shared library.
It should be loaded as extension in sqlite before it can be used.
```shell
sqlite3 <<EOF
pragma page_size = 4096;
.load ./path/to/libmojo
.open 'file:a.db?vfs=mojo&pagesz=4096'
EOF
```
In python3 sqlite binding:
```python
import sqlite3
mojo_lib="<path/to/libmojo>"
db_path="a.db"
con = sqlite3.connect(":memory:")
con.enable_load_extension(True)
con.execute(f"select load_extension('{mojo_lib}')")
con.execute("pragma page_size=4096")
con.enable_load_extension(False)
con.close()
conn_str = f"file:{db_path}?vfs=mojo&pagesz=4096"
conn = sqlite3.Connection(conn_str, uri=True)
```
Let's decode the URI in the open `.open 'file:a.db?vfs=mojo&pagesz=4096'`:
* `a.db` => Name of the database
* `vfs=mojo` => Name of the MojoFS
* `pagesz=4096` => Page size used by the fs.
This should same as the page size in the pragma `pragma page_size = 4096`
Once set, the page size cannot be changed.
When the database is created for the first time, it starts with version=1.
Version numbers are ever incrementing and the highest version number is writable
and old versions are read-only.
The fs has to be commited to make the current active version as read-only.
## Committing database
The `mojo-cli` is the tool for administration of the mojofs. To commit the database `a.db`:
```shell
mojo-cli ./a.db commit
```
Commit advances version number by 1. So if current active version=1 then after committing, a new version=2
will be created and version=1 will be read-only now.
Committing FS is really a cheap operation. It only manipulates the metadata of the FS and no data movement
is involved.
## Committing MojoFS vs Committing Database
Committing the fs is different than committing the database. You can continue to use the database
in the active version as long as you want. This means multiple transactions can be initiated with
all the DB commits and rollbacks, all in a single version.
Committing the database makes sqlite issue `fsync()` call which makes the data written to disk durable.
It is recommended to commit the fs when databases is just committed/rolled-backed.
The FS is not aware any uncommitted transactions and committing the fs midway can cause undefined behaviour.
## Reading old version
Pass the `ver=<num>` and `mode=ro` to open it in readonly mode:
```
.open 'file:a.db?vfs=mojo&pagesz=4096&ver=2&mode=ro'
```
================================================
FILE: meson.build
================================================
project('mojo', 'c', default_options: ['c_std=c11'], version: '0.1.0')
compiler = meson.get_compiler('c')
mojokv_lib_path = meson.source_root() + '/target/debug'
if get_option('buildtype') == 'release'
mojokv_lib_path = meson.source_root() + '/target/release'
endif
mojofs_lib = compiler.find_library('mojofs',
dirs: [mojokv_lib_path])
mojofs_dep = declare_dependency(dependencies: [mojofs_lib])
sqlite_dep = dependency('sqlite3')
shared_library('mojo',
name_prefix: '',
sources: ['sqlite-ext/mojo.c'],
include_directories: ['sqlite-ext'],
dependencies: [mojofs_dep, sqlite_dep])
================================================
FILE: sqlite-ext/mojo.c
================================================
#include <stdio.h>
#include <sqlite3ext.h>
SQLITE_EXTENSION_INIT1
#include <assert.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/file.h>
#include <sys/param.h>
#include <unistd.h>
#include <time.h>
#include <errno.h>
#include <fcntl.h>
#include <mojofs.h>
/*
** The maximum pathname length supported by this VFS.
*/
#define MAXPATHNAME 512
/*
** Argument zPath points to a nul-terminated string containing a file path.
** If zPath is an absolute path, then it is copied as is into the output
** buffer. Otherwise, if it is a relative path, then the equivalent full
** path is written to the output buffer.
**
** This function assumes that paths are UNIX style. Specifically, that:
**
** 1. Path components are separated by a '/'. and
** 2. Full paths begin with a '/' character.
*/
static int mojo_fullpath_name(
sqlite3_vfs *pVfs, /* VFS */
const char *zPath, /* Input path (possibly a relative path) */
int nPathOut, /* Size of output buffer in bytes */
char *zPathOut /* Pointer to output buffer */
){
char zDir[MAXPATHNAME+1];
if( zPath[0]=='/' ){
zDir[0] = '\0';
}else{
if( getcwd(zDir, sizeof(zDir))==0 ) return SQLITE_IOERR;
}
zDir[MAXPATHNAME] = '\0';
sqlite3_snprintf(nPathOut, zPathOut, "%s/%s", zDir, zPath);
zPathOut[nPathOut-1] = '\0';
return SQLITE_OK;
}
int sqlite3_mojo_init(sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi){
int rc = SQLITE_OK;
SQLITE_EXTENSION_INIT2(pApi);
mojofs_init_log();
sqlite3_vfs *vfs = mojo_create();
vfs->xFullPathname = mojo_fullpath_name;
rc = sqlite3_vfs_register(vfs, 0);
//printf("extension mojo called with rc=%d\n", rc);
fflush(stdout);
return rc;
}
================================================
FILE: sqlite-ext/mojofs.h
================================================
#ifndef __mojo_h
#define __mojo_h
#include <sqlite3.h>
typedef struct MojoFile {
sqlite3_vfs base;
void* custom_file;
} MojoFile;
sqlite3_vfs* mojo_create();
void mojofs_init_log();
#endif
================================================
FILE: test-scripts/commands.py
================================================
"""Multiple commands for testing"""
import subprocess
import sqlite3
MOJOKV_CLI=None
class TestConfig:
"""Config for test db"""
def __init__(self, page_sz=4096, journal_mode="WAL", vac_mode="NONE"):
self.page_sz = page_sz
self.journal_mode = journal_mode
self.vac_mode = vac_mode
def __repr__(self):
return f"page_sz={self.page_sz} journal={self.journal_mode}"
def vacuum(cur):
"""Vacuum"""
cur.execute("vacuum")
def opendb(cfg, db_path, ver="1", mode=""):
"""Open sqlite db using cfg"""
if mode == "":
conn_str = f"file:{db_path}?vfs=mojo&ver={ver}&pagesz={cfg.page_sz}&pps=65536"
else:
conn_str = f"file:{db_path}?vfs=mojo&mode=ro&ver={ver}&pagesz={cfg.page_sz}&pps=65536"
conn = sqlite3.Connection(conn_str, uri=True)
if mode != "ro":
conn.execute(f"PRAGMA page_size={cfg.page_sz}")
conn.execute(f"PRAGMA journal_mode={cfg.journal_mode}")
conn.execute(f"PRAGMA auto_vacuum={cfg.vac_mode}")
return conn
def mkdir(dir_path):
""""Make dir"""
subprocess.run(["mkdir", "-p", dir_path], check=True, capture_output=True)
def commit_version(dbpath):
'''commit_version commits version for given dbpath'''
subprocess.run([MOJOKV_CLI, dbpath, "commit"], check=True, capture_output=True)
def create_table_person(cur):
"""Create table Person"""
cur.execute("""create table if not exists person(
name text primary key,
age integer,
id integer
)""")
cur.execute("create index if not exists person_idx_1 on person(id)")
def get_row_count(cur, table):
"""Get row count of table"""
row = cur.execute(f"select count(*) from {table}").fetchone()
if row is None or row[0] is None:
return 0
return int(row[0])
def table_person_count(conn):
"""Get row count of Person"""
cur = conn.cursor()
row = cur.execute("select count(*) from person").fetchone()
return int(row[0])
def get_max_id_person(cur):
"""get max id"""
row = cur.execute("select max(id) from person").fetchone()
if row is None or row[0] is None:
return 0
return int(row[0])
def insert_table_person(cur, count):
"""Insert into person table"""
max_id = get_max_id_person(cur)
for n in range(max_id+1, max_id+count+1):
name = f"name-{n}"
cur.execute("insert into person(name,age,id) values(?,?,?)", [name,n,n])
def delete_table_person(cur, from_id, to_id):
"""Delete from person table"""
cur.execute(f"delete from person where id>={from_id} and id <= {to_id}")
def drop_table_person(cur):
""""Drop table person"""
cur.execute("drop table person")
def copy_table_person(cur, new_table):
""""Copy table from person table"""
cur.execute(f"create table if not exists {new_table} as select name,age,id from person")
cur.execute(f"create index if not exists {new_table}_idx_1 on {new_table}(id)")
================================================
FILE: test-scripts/perftest.py
================================================
"""" Perf test for mojo filesystem """
import sqlite3
import os
import shutil
import time
import sys
def rm_dir(path):
'''Remove dir. Ignore file not found error'''
try:
if os.path.isdir(path):
rm_fr(path)
else:
os.unlink(path)
except FileNotFoundError:
pass
def rm_fr(path):
'''Equivalent of rm -fr'''
journal_path = path + "-journal"
wal_path = path + "-wal"
rm_dir(journal_path)
rm_dir(wal_path)
if not os.path.exists(path):
return
if os.path.isfile(path) or os.path.islink(path):
os.unlink(path)
else:
shutil.rmtree(path)
def _mojo_conn_str(db_path, ver="1", mode=""):
"""open database"""
if mode == "":
conn_str = f"file:{db_path}?vfs=mojo&ver={ver}&pagesz=4096&pps=65536"
else:
conn_str = f"file:{db_path}?vfs=mojo&mode=ro&ver={ver}&pagesz=4096&pps=65536"
return conn_str
def _std_conn_str(db_path, mode=""):
if mode == "":
conn_str = f"file:{db_path}"
else:
conn_str = f"file:{db_path}?mode=ro"
return conn_str
def open_db(db_path, vfs="mojo", ver="1", mode=""):
"""Open database"""
if vfs == "mojo":
conn_str=_mojo_conn_str(db_path, ver=ver, mode=mode)
else:
conn_str=_std_conn_str(db_path, mode=mode)
conn = sqlite3.Connection(conn_str, uri=True)
if mode != "ro":
conn.execute("PRAGMA page_size=4096")
return conn
def create_table(conn):
"""create table"""
conn.execute("create table test(s text primary key)")
def insert_rows(conn, row_count, ver, suffix=""):
"""create table"""
cur = conn.cursor()
for i in range(row_count):
val = f"{ver}-text-{i}{suffix}"
cur.execute("insert into test values(?)", (val,))
def count_rows(conn, condition=""):
"""count rows"""
cur = conn.cursor()
if condition == "":
row = cur.execute("select count(*) from test").fetchone()
else:
row = cur.execute(f"select count(*) from test where {condition}").fetchone()
return int(row[0])
def select_rows(conn, condition=""):
"""select rows"""
cur = conn.cursor()
if condition == "":
rows = cur.execute("select * from test")
else:
rows = cur.execute(f"select * from test where {condition}")
count = 0
for _r in rows:
count += 1
return count
def update_text_rows(conn):
"""Update rows"""
key = "odd-update-text"
conn.execute("update test set s = ? where s like 'odd%'",(key,))
conn.commit()
def load_extension(mojo_lib):
"""load_ext"""
print("using libpath =", mojo_lib)
con = sqlite3.connect(":memory:")
# enable extension loading
con.enable_load_extension(True)
con.execute(f"select load_extension('{mojo_lib}')")
con.execute("pragma page_size=4096")
con.enable_load_extension(False)
con.close()
ROW_COUNT=10000000
def perf_insert(conn):
""""Perf insert"""
start = time.time()
create_table(conn)
insert_rows(conn, ROW_COUNT, "1")
conn.commit()
end = time.time()
return end-start
def perf_select(conn):
""""Perf select"""
start = time.time()
count = select_rows(conn)
end = time.time()
print("select iter count:", count)
return end-start
def perf_count_rows(conn):
""""Perf select"""
start = time.time()
count = count_rows(conn, "s like '%abc%'")
print("row count:", count)
end = time.time()
return end-start
def perf_update_rows(conn):
""""Perf update rows"""
start = time.time()
update_text_rows(conn)
end = time.time()
return end-start
if __name__ == '__main__':
if len(sys.argv[1:]) < 1:
print("Error: missing extension library path")
sys.exit(1)
ext_path = sys.argv[1]
load_extension(ext_path)
STD_DBPATH="./perf-std.db"
MOJO_DBPATH="./perf-mojo.db"
try:
rm_fr(STD_DBPATH)
rm_fr(MOJO_DBPATH)
perf_list=[
("insert", perf_insert),
("update rows", perf_update_rows),
("select", perf_select),
("row count", perf_count_rows),
]
for desc, perf_fn in perf_list:
print("Running perf for:", desc)
e = []
for dbpath, vfs in [(STD_DBPATH, "std"), (MOJO_DBPATH, "mojo")]:
dbconn = open_db(dbpath, vfs=vfs)
elapsed = perf_fn(dbconn)
e.append(elapsed)
print(f"\tvfs={vfs} time elapsed (s):", elapsed)
dbconn.close()
ratio = round(e[1]/e[0], 3)
print(f"\tMojo takes {ratio} times than std vfs")
print("------------------------")
except Exception as e:
raise e
finally:
rm_fr(STD_DBPATH)
rm_fr(MOJO_DBPATH)
================================================
FILE: test-scripts/test.sql
================================================
pragma page_size = 4096;
.load ./build/libmojo
.open 'file:a.db?vfs=mojo&pagesz=4096'
create table if not exists test (
n int
);
insert into test values (1);
insert into test values (2);
insert into test values (3);
================================================
FILE: test-scripts/test2.sql
================================================
pragma page_size = 4096;
.load ./build/libmojo
.open 'file:./testdbs/a_1.db?vfs=mojo&pagesz=4096&mode=ro&ver=2'
select count(*), max(id) from person;
================================================
FILE: test-scripts/testdb.py
================================================
"""" Tests for mojo filesystem """
import os
import sys
import unittest
import sqlite3
import shutil
import commands as c
MOJOKV_CLI=None
class TestConfig:
'''Config for test db'''
def __init__(self, page_sz=4096, journal_mode="WAL", vac_mode="NONE", use_tx=True):
self.page_sz = page_sz
self.journal_mode = journal_mode
self.vac_mode = vac_mode
self.use_tx = use_tx
def __repr__(self):
return f"page_sz={self.page_sz} journal={self.journal_mode}"
def rm_dir(path):
'''Remove dir. Ignore file not found error'''
try:
if os.path.isdir(path):
rm_fr(path)
else:
os.unlink(path)
except FileNotFoundError:
pass
def rm_fr(path):
'''Equivalent of rm -fr'''
journal_path = path + "-journal"
wal_path = path + "-wal"
rm_dir(journal_path)
rm_dir(wal_path)
if not os.path.exists(path):
return
if os.path.isfile(path) or os.path.islink(path):
os.unlink(path)
else:
shutil.rmtree(path)
class MojoWritableTest(unittest.TestCase):
'''MojoWritableTest'''
def __init__(self, cfg, dbpath, *args, **kargs):
self.cfg = cfg
self.db_conn = None
self.db_path = dbpath
super(MojoWritableTest, self).__init__(*args, **kargs)
def setUp(self):
rm_fr(self.db_path)
def tearDown(self):
if self.db_conn:
self.db_conn.close()
def _subtest_name(self, name):
return f"{name}: {self.cfg} {self.db_path}"
def begin(self, cur):
"""begin"""
if self.cfg.use_tx:
cur.execute("begin")
def commit(self, cur):
"""commit"""
if self.cfg.use_tx:
cur.execute("commit")
def rollback(self, cur):
"""rollback"""
if self.cfg.use_tx:
cur.execute("rollback")
def test_db_use(self):
'''Tests the general usage of the database '''
db_conn = c.opendb(self.cfg, self.db_path)
ins_row_count = 100
with self.subTest(self._subtest_name("create table")):
c.create_table_person(db_conn)
with self.subTest(self._subtest_name("insert rows v1")):
c.insert_table_person(db_conn, ins_row_count)
db_conn.commit()
count = c.table_person_count(db_conn)
self.assertEqual(ins_row_count, count)
db_conn.close()
### Commit ver=1
c.commit_version(self.db_path)
db_conn = c.opendb(self.cfg, self.db_path)
### active ver=2
with self.subTest(self._subtest_name("insert rows v2")):
self.assertEqual(ins_row_count, c.table_person_count(db_conn))
c.insert_table_person(db_conn, ins_row_count)
db_conn.commit()
self.assertEqual(ins_row_count*2, c.table_person_count(db_conn))
#db_conn.close()
### Commit ver=2
db_conn.close()
c.commit_version(self.db_path)
db_conn = c.opendb(self.cfg, self.db_path)
### active ver=3
with self.subTest(self._subtest_name("copy table")):
self.assertEqual(ins_row_count*2, c.table_person_count(db_conn))
c.copy_table_person(db_conn, "person_2")
db_conn.commit()
self.assertEqual(ins_row_count*2, c.get_row_count(db_conn, "person_2"))
with self.subTest(self._subtest_name("read v1")):
db_v1 = c.opendb(self.cfg, self.db_path, mode="ro", ver="1")
self.assertEqual(ins_row_count, c.table_person_count(db_v1))
db_v1.close()
db_conn.close()
### Commit ver=3
c.commit_version(self.db_path)
db_conn = c.opendb(self.cfg, self.db_path)
### active ver=4
with self.subTest(self._subtest_name("delete rows in v3")):
c.delete_table_person(db_conn, 0, 10000)
db_conn.commit()
self.assertEqual(0, c.table_person_count(db_conn))
db_conn.close()
### Commit ver=4
c.commit_version(self.db_path)
db_conn = c.opendb(self.cfg, self.db_path)
### active ver=5
with self.subTest(self._subtest_name("vacuum")):
c.vacuum(db_conn)
db_conn.commit()
self.assertEqual(0, c.table_person_count(db_conn))
db_v3 = c.opendb(self.cfg, self.db_path, mode="ro", ver="4")
self.assertEqual(0, c.table_person_count(db_v3))
db_v3.close()
db_v2 = c.opendb(self.cfg, self.db_path, mode="ro", ver="2")
self.assertEqual(ins_row_count*2, c.table_person_count(db_v2))
db_v2.close()
db_conn.close()
### Commit ver=5
c.commit_version(self.db_path)
def load_extension(mojo_lib):
"""load_extension"""
print("using libpath =", mojo_lib)
con = sqlite3.connect(":memory:")
# enable extension loading
con.enable_load_extension(True)
con.execute(f"select load_extension('{mojo_lib}')")
con.execute("pragma page_size=4096")
con.enable_load_extension(False)
con.close()
def create_suite(full_mode):
''' Test suite for mojo '''
suite = unittest.TestSuite()
if not full_mode:
page_sizes = [4096]
journal_modes = ["WAL"]
vacuum_modes = ["INCREMENTAL"]
use_tx = [False]
else:
page_sizes = [4096]
journal_modes = ["OFF", "WAL", "MEMORY", "DELETE", "TRUNCATE", "PERSIST"]
vacuum_modes = ["NONE", "FULL", "INCREMENTAL"]
use_tx = [False]
dbid = 0
for page_sz in page_sizes:
for journal_mode in journal_modes:
for vac_mode in vacuum_modes:
for tx in use_tx:
cfg = TestConfig(page_sz=page_sz,
journal_mode=journal_mode,
vac_mode=vac_mode,
use_tx=tx)
dbid += 1
dbpath = f"./testdbs/a_{dbid}.db"
suite.addTest(MojoWritableTest(cfg, dbpath, 'test_db_use'))
return suite
if __name__ == '__main__':
if len(sys.argv[1:]) < 1:
print("Error: missing extension library path")
sys.exit(1)
if len(sys.argv[2:]) >= 1 and sys.argv[2] == "full":
FULL = True
else:
FULL = False
MOJOKV_CLI=os.getenv("MOJOKV_CLI")
if not MOJOKV_CLI:
MOJOKV_CLI="./build/mojo-cli"
c.MOJOKV_CLI = MOJOKV_CLI
ext_path = sys.argv[1]
load_extension(ext_path)
rm_fr("./testdbs/*")
c.mkdir("./testdbs")
runner = unittest.TextTestRunner(failfast=True)
runner.run(create_suite(FULL))
#conn = sqlite3.connect("file:a.db?vfs=mojo&ver=1&pagesz=4096", uri=True)
#conn = sqlite3.connect("a.db")
gitextract_2dkkx48z/
├── .gitignore
├── Cargo.toml
├── LICENSE
├── README.md
├── build.sh
├── crates/
│ ├── mojo-cli/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── buckets.rs
│ │ ├── commit.rs
│ │ ├── diff.rs
│ │ ├── iget.rs
│ │ ├── iview.rs
│ │ ├── main.rs
│ │ ├── state.rs
│ │ └── truncate.rs
│ ├── mojofs/
│ │ ├── .gitignore
│ │ ├── Cargo.toml
│ │ ├── meson.build
│ │ ├── src/
│ │ │ ├── error.rs
│ │ │ ├── kvfile.rs
│ │ │ ├── lib.rs
│ │ │ ├── native_file.rs
│ │ │ ├── open_options.rs
│ │ │ ├── vfs.rs
│ │ │ └── vfsfile.rs
│ │ └── tests/
│ │ └── mojofs_test.rs
│ ├── mojoio/
│ │ ├── Cargo.toml
│ │ └── src/
│ │ ├── error.rs
│ │ ├── lib.rs
│ │ └── nix.rs
│ └── mojokv/
│ ├── .gitignore
│ ├── Cargo.toml
│ └── src/
│ ├── bmap.rs
│ ├── bucket.rs
│ ├── error.rs
│ ├── index/
│ │ ├── mem.rs
│ │ └── mod.rs
│ ├── keymap.rs
│ ├── lib.rs
│ ├── state.rs
│ ├── store.rs
│ ├── utils.rs
│ └── value.rs
├── docs/
│ ├── design.md
│ ├── source.md
│ └── user-guide.md
├── meson.build
├── sqlite-ext/
│ ├── mojo.c
│ └── mojofs.h
└── test-scripts/
├── commands.py
├── perftest.py
├── test.sql
├── test2.sql
└── testdb.py
SYMBOL INDEX (326 symbols across 35 files)
FILE: crates/mojo-cli/src/buckets.rs
function cmd (line 4) | pub fn cmd(kvpath: &std::path::Path, ver: u32) -> Result<(), Error> {
FILE: crates/mojo-cli/src/commit.rs
function cmd (line 4) | pub fn cmd(kvpath: &std::path::Path) -> Result<(), Error> {
FILE: crates/mojo-cli/src/diff.rs
function cmd (line 4) | pub fn cmd(kvpath: &std::path::Path, fver: u32, tver: u32) -> Result<(),...
FILE: crates/mojo-cli/src/iget.rs
function cmd (line 4) | pub fn cmd(kvpath: &std::path::Path, bucket: &str, ver: u32, key: u32) -...
FILE: crates/mojo-cli/src/iview.rs
function cmd (line 4) | pub fn cmd(kvpath: &std::path::Path, name: &str, ver: u32, additional: b...
FILE: crates/mojo-cli/src/main.rs
type Cli (line 14) | struct Cli {
type Commands (line 22) | enum Commands {
function main (line 71) | fn main() -> Result<(), Error> {
FILE: crates/mojo-cli/src/state.rs
function cmd (line 5) | pub fn cmd(kvpath: &std::path::Path, additional: bool) -> Result<(), Err...
FILE: crates/mojo-cli/src/truncate.rs
function cmd (line 4) | pub fn cmd(kvpath: &std::path::Path, sz: usize) -> Result<(), Error> {
FILE: crates/mojofs/src/error.rs
constant MOJOFS_ERR_NOT_IMPL (line 2) | pub const MOJOFS_ERR_NOT_IMPL: i32 = 1;
constant MOJOFS_ERR_IO (line 3) | pub const MOJOFS_ERR_IO: i32 = 2;
constant MOJOFS_ERR_NIX (line 4) | pub const MOJOFS_ERR_NIX: i32 = 3;
constant MOJOFS_ERR_UTF8 (line 5) | pub const MOJOFS_ERR_UTF8: i32 = 4;
constant MOJOFS_ERR_MOJOKV (line 6) | pub const MOJOFS_ERR_MOJOKV: i32 = 5;
constant MOJOFS_ERR_URL_PARSE (line 7) | pub const MOJOFS_ERR_URL_PARSE: i32 = 6;
constant MOJOFS_ERR_INT_PARSE (line 8) | pub const MOJOFS_ERR_INT_PARSE: i32 = 7;
constant MOJOFS_ERR_LARGE_PAGE (line 9) | pub const MOJOFS_ERR_LARGE_PAGE: i32 = 8;
constant MOJOFS_ERR_ARG_VER_MISSING (line 10) | pub const MOJOFS_ERR_ARG_VER_MISSING: i32 = 9;
constant MOJOFS_ERR_ARG_PAGESZ_MISSING (line 11) | pub const MOJOFS_ERR_ARG_PAGESZ_MISSING: i32 = 10;
constant MOJOFS_ERR_ARG_PPS_MISSING (line 12) | pub const MOJOFS_ERR_ARG_PPS_MISSING: i32 = 11;
type Error (line 15) | pub struct Error {
method new (line 21) | pub fn new(code: i32, msg: String) -> Self {
method not_impl (line 28) | pub fn not_impl() -> Self {
method from (line 34) | fn from(err: std::io::Error) -> Self {
method from (line 43) | fn from(err: nix::Error) -> Self {
method from (line 52) | fn from(err: std::str::Utf8Error) -> Self {
method from (line 61) | fn from(err: std::num::ParseIntError) -> Self {
method from (line 70) | fn from(err: mojokv::Error) -> Self {
method fmt (line 79) | fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
FILE: crates/mojofs/src/kvfile.rs
type KVFile (line 5) | pub struct KVFile {
method open (line 19) | pub fn open(bucket: Bucket, opt: KVFileOpt) -> Result<Self, Error> {
method pread (line 26) | pub fn pread(&self, buf: &mut [u8], off: i64) -> Result<usize, Error> {
method pwrite_page (line 56) | fn pwrite_page(&mut self, key: u32, page_off: u32, buf: &[u8]) -> Resu...
method pwrite (line 64) | pub fn pwrite(&mut self, off: i64, buf: &[u8]) -> Result<(), Error> {
method close (line 83) | pub fn close(self) -> Result<(), Error> {
method sync (line 88) | pub fn sync(&mut self) -> Result<(), Error> {
method filesize (line 93) | pub fn filesize(&self) -> Result<u64, Error> {
method truncate (line 97) | pub fn truncate(&mut self, new_sz: u64) -> Result<(), Error> {
type KVFileOpt (line 11) | pub struct KVFileOpt {
FILE: crates/mojofs/src/lib.rs
type MojoFile (line 26) | pub struct MojoFile {
function mojo_create (line 33) | pub extern "C" fn mojo_create() -> *mut sqlite3_vfs {
function mojo_open (line 78) | extern "C" fn mojo_open(vfs: *mut sqlite3_vfs, zname: *const c_char, fil...
function mojo_close (line 157) | extern "C" fn mojo_close(sfile: *mut sqlite3_file) -> c_int {
function mojo_read (line 173) | extern "C" fn mojo_read(sfile: *mut sqlite3_file, ptr: *mut c_void, n: i...
function mojo_write (line 199) | extern "C" fn mojo_write(sfile: *mut sqlite3_file, ptr: *const c_void, n...
function mojo_truncate (line 215) | extern "C" fn mojo_truncate(sfile: *mut sqlite3_file, new_sz: i64) -> c_...
function mojo_sync (line 230) | extern "C" fn mojo_sync(sfile: *mut sqlite3_file, flags: i32) -> c_int {
function mojo_filesize (line 245) | extern "C" fn mojo_filesize(sfile: *mut sqlite3_file, out_sz: *mut i64) ...
function mojo_access (line 260) | extern "C" fn mojo_access(vfs: *mut sqlite3_vfs, zname: *const c_char, f...
function mojo_delete (line 288) | extern "C" fn mojo_delete(vfs: *mut sqlite3_vfs, zname: *const c_char, _...
function mojo_fullname (line 309) | extern "C" fn mojo_fullname(vfs: *mut sqlite3_vfs, zname: *const c_char,...
function mojo_dlopen (line 327) | extern "C" fn mojo_dlopen(_arg1: *mut sqlite3_vfs, _zfilename: *const c_...
function mojo_dlerror (line 332) | extern "C" fn mojo_dlerror(_arg1: *mut sqlite3_vfs, _nbyte: c_int, _zerr...
function mojo_dlsym (line 337) | extern "C" fn mojo_dlsym(_arg1: *mut sqlite3_vfs, _arg2: *mut c_void, _z...
function mojo_dlclose (line 343) | extern "C" fn mojo_dlclose(_arg1: *mut sqlite3_vfs, _arg2: *mut c_void) {
function mojo_randomness (line 347) | extern "C" fn mojo_randomness(_arg1: *mut sqlite3_vfs, _nbyte: c_int, _z...
function mojo_sleep (line 352) | extern "C" fn mojo_sleep(_arg1: *mut sqlite3_vfs, microseconds: c_int) -...
function mojo_current_time (line 358) | extern "C" fn mojo_current_time(_arg1: *mut sqlite3_vfs, p: *mut f64) ->...
function mojo_current_time64 (line 368) | extern "C" fn mojo_current_time64(_arg1: *mut sqlite3_vfs, p: *mut i64) ...
function mojo_getlasterr (line 378) | extern "C" fn mojo_getlasterr(_arg1: *mut sqlite3_vfs, _arg2: c_int, _ar...
function mojo_lock (line 383) | extern "C" fn mojo_lock(_sfile: *mut sqlite3_file, _flags: c_int) -> c_i...
function mojo_unlock (line 388) | extern "C" fn mojo_unlock(_sfile: *mut sqlite3_file, _flags: c_int) -> c...
function mojo_check_reserved_lock (line 393) | extern "C" fn mojo_check_reserved_lock(_sfile: *mut sqlite3_file, res_ou...
function mojo_file_control (line 399) | extern "C" fn mojo_file_control(_sfile: *mut sqlite3_file, _op: c_int, _...
function mojo_sector_size (line 404) | extern "C" fn mojo_sector_size(_sfile: *mut sqlite3_file) -> c_int {
function mojo_device_char (line 409) | extern "C" fn mojo_device_char(_sfile: *mut sqlite3_file) -> c_int {
function getfs (line 413) | fn getfs(vfs: *mut sqlite3_vfs) -> &'static mut VFS {
function get_file_mut (line 421) | fn get_file_mut(sfile: *mut sqlite3_file) -> &'static mut VFSFile {
function get_file (line 430) | fn get_file(sfile: *mut sqlite3_file) -> &'static VFSFile {
function c_to_path (line 439) | fn c_to_path(cpath: *const c_char) -> Result<std::path::PathBuf, Error> {
function mojofs_init_log (line 447) | pub extern "C" fn mojofs_init_log() {
function extract_query_params (line 452) | fn extract_query_params(filepath: *const c_char) -> Result<HashMap<Strin...
FILE: crates/mojofs/src/native_file.rs
type NativeFile (line 9) | pub struct NativeFile {
method open (line 16) | pub fn open(path: &Path, opt: &OpenOptions) -> Result<Self, Error> {
method pread (line 42) | pub fn pread(&self, buf: &mut [u8], off: i64) -> Result<usize, Error> {
method pwrite (line 60) | pub fn pwrite(&mut self, off: i64, buf: &[u8]) -> Result<(), Error> {
method close (line 71) | pub fn close(self) -> Result<(), Error> {
method sync (line 80) | pub fn sync(&mut self) -> Result<(), Error> {
method filesize (line 85) | pub fn filesize(&self) -> Result<u64, Error> {
method truncate (line 90) | pub fn truncate(&mut self, new_sz: u64) -> Result<(), Error> {
FILE: crates/mojofs/src/open_options.rs
type OpenOptions (line 4) | pub struct OpenOptions {
method from_flags (line 54) | pub fn from_flags(flags: i32) -> Option<Self> {
type OpenKind (line 19) | pub enum OpenKind {
method is_main (line 31) | pub fn is_main(&self) -> bool {
method from_flags (line 65) | fn from_flags(flags: i32) -> Option<Self> {
type OpenAccess (line 38) | pub enum OpenAccess {
method from_flags (line 81) | fn from_flags(flags: i32) -> Option<Self> {
FILE: crates/mojofs/src/vfs.rs
type AccessCheck (line 13) | pub enum AccessCheck {
type VFS (line 19) | pub struct VFS {
method name (line 26) | pub fn name(&self) -> String {
method fs_options (line 30) | pub fn fs_options(&self) -> FSOptions {
method active_ver (line 34) | pub fn active_ver(&self) -> u32 {
method init (line 38) | pub fn init(&mut self, root_path: &str, params: &HashMap<String, Strin...
method open (line 54) | pub fn open(&mut self, filepath: &str, opt: OpenOptions, _out_opt: &mu...
method fullpath (line 84) | pub fn fullpath(&mut self, filepath: &str) -> Result<PathBuf, Error> {
method bucket_name (line 97) | fn bucket_name(path: &Path) -> &str {
method delete (line 102) | pub fn delete(&mut self, path: &std::path::Path) -> Result<(), Error> {
method access (line 112) | pub fn access(&self, path: &std::path::Path, req: AccessCheck) -> Resu...
method close (line 123) | pub fn close(&mut self, f: VFSFile) -> Result<(), Error> {
method commit (line 141) | pub fn commit(&mut self) -> Result<(), Error> {
type FSOptions (line 151) | pub struct FSOptions {
method parse (line 158) | fn parse(map: &HashMap<String, String>) -> Result<FSOptions, Error> {
method to_kvfile_opt (line 178) | fn to_kvfile_opt(&self) -> KVFileOpt {
FILE: crates/mojofs/src/vfsfile.rs
type FileImpl (line 7) | pub enum FileImpl {
type VFSFile (line 12) | pub struct VFSFile {
method new (line 21) | pub fn new(id: usize, name: &str, opt: OpenOptions, fimpl: FileImpl) -...
method id (line 30) | pub fn id(&self) -> usize {
method opt (line 34) | pub fn opt(&self) -> OpenOptions {
method pread (line 38) | pub fn pread(&self, off: u64, buf: &mut [u8]) -> Result<usize, Error> {
method pwrite (line 54) | pub fn pwrite(&mut self, off: u64, buf: &[u8]) -> Result<(), Error> {
method close (line 69) | pub fn close(self) -> Result<(), Error> {
method sync (line 84) | pub fn sync(&mut self, flags: i32) -> Result<(), Error> {
method filesize (line 99) | pub fn filesize(&self) -> Result<u64, Error> {
method truncate (line 112) | pub fn truncate(&mut self, new_sz: u64) -> Result<(), Error> {
method lock (line 127) | pub fn lock(&mut self, _flag: i32) -> Result<(), Error> {
method unlock (line 131) | pub fn unlock(&mut self, _flag: i32) -> Result<(), Error> {
method check_reserved_lock (line 135) | pub fn check_reserved_lock(&self) -> Result<i32, Error> {
method file_control (line 139) | pub fn file_control(&mut self, _op: i32) -> Result<(), Error> {
method sector_size (line 143) | pub fn sector_size(&self) -> Result<i32, Error> {
method device_char (line 147) | pub fn device_char(&self) -> Result<(), Error> {
FILE: crates/mojofs/tests/mojofs_test.rs
function remove_fs (line 6) | fn remove_fs(rootpath: &Path) -> Result<(), Error> {
function setup (line 15) | fn setup() -> Result<String, Error> {
function default_params (line 21) | fn default_params(pagesz: u32) -> HashMap<String, String> {
function read_test (line 31) | fn read_test(a: &mut VFSFile, nitems: usize, pagesz: u64, f: fn(usize) -...
function write_test (line 42) | fn write_test(a: &mut VFSFile, nitems: usize, pagesz: u64, f: fn(usize) ...
function write_read (line 53) | fn write_read(a: &mut VFSFile, nitems: usize, pagesz: u64, f: fn(usize) ...
function rw_same_version (line 61) | fn rw_same_version() -> Result<(), Error> {
FILE: crates/mojoio/src/error.rs
type Error (line 3) | pub enum Error {
FILE: crates/mojoio/src/lib.rs
constant BUFFER_MAGIC (line 6) | pub const BUFFER_MAGIC: &[u8] = b"mojo";
constant PAGE_HEADER_LEN (line 7) | pub const PAGE_HEADER_LEN: usize = 8;
function add (line 10) | pub fn add(left: usize, right: usize) -> usize {
function it_works (line 19) | fn it_works() {
FILE: crates/mojoio/src/nix.rs
type NixFile (line 9) | pub struct NixFile {
method open (line 17) | pub fn open(filepath: &Path, _file_no: u32) -> Result<Self, Error> {
method close (line 34) | pub fn close(&mut self) -> Result<(), Error> {
method write_buf_at (line 40) | pub fn write_buf_at(&mut self, off: u64, block_no: u32, buf: &[u8]) ->...
method write_buf (line 58) | pub fn write_buf(&mut self, block_no: u32, poff: u64, buf: &[u8]) -> R...
method header_len (line 70) | pub fn header_len() -> usize {
method read_all_at (line 74) | fn read_all_at(&self, off: u64, buf: &mut [u8]) -> Result<usize, Error> {
method read_buf_at (line 87) | pub fn read_buf_at(&self, off: u64, buf: &mut [u8]) -> Result<usize, E...
method sync (line 93) | pub fn sync(&self) -> Result<(), Error> {
type PageHeader (line 101) | struct PageHeader {
method new (line 107) | fn new() -> PageHeader {
method encode (line 114) | pub fn encode(&mut self, buf: &mut [u8; crate::PAGE_HEADER_LEN]) {
FILE: crates/mojokv/src/bmap.rs
type BucketMap (line 11) | pub struct BucketMap {
method add (line 16) | pub fn add(&self, name: &str, ver: u32) {
method exists (line 23) | pub fn exists(&self, name: &str) -> bool {
method get (line 28) | pub fn get(&self, name: &str) -> Option<u32>{
method delete (line 34) | pub fn delete(&self, root_path: &Path, name: &str, ver: u32) -> Result...
method map (line 45) | pub fn map(&self) -> Result<HashMap<String, u32>, Error> {
method serialize_to_path (line 51) | pub fn serialize_to_path(&self, path: &Path) -> Result<(), Error> {
method deserialize_from_path (line 58) | pub fn deserialize_from_path(path: &Path) -> Result<Self, Error> {
method bmap_path (line 66) | fn bmap_path(root_path: &Path, ver: u32) -> PathBuf {
method load (line 70) | pub fn load(root_path: &Path, ver: u32) -> Result<Self, Error> {
FILE: crates/mojokv/src/bucket.rs
type BucketInner (line 10) | pub struct BucketInner {
method active_file (line 23) | fn active_file(&mut self, ver: u32) -> &mut NixFile {
method sync_index (line 27) | fn sync_index(&mut self, ver: u32) -> Result<(), Error> {
type Bucket (line 41) | pub struct Bucket {
method with_inner (line 50) | fn with_inner(state: State, inner: BucketInner, bmap: BucketMap) -> Se...
method set_writable (line 60) | pub fn set_writable(&mut self) {
method readonly (line 64) | pub fn readonly(root_path: &Path, name: &str, ver: u32, state: State, ...
method index_path (line 71) | fn index_path(rootpath: &Path, name: &str, ver: u32) -> PathBuf {
method get_key (line 75) | pub fn get_key(&self, key: u32) -> Result<Option<Value>, Error> {
method max_key (line 80) | pub fn max_key(&self) -> isize {
method is_modified (line 85) | pub fn is_modified(&self) -> bool {
method writable (line 90) | pub fn writable(root_path: &Path, name: &str, state: State, bmap: Buck...
method load (line 113) | pub fn load(root_path: &Path, name: &str, state: State, bmap: BucketMa...
method load_index (line 142) | pub fn load_index(root_path: &Path, name: &str, ver: u32) -> Result<(u...
method new (line 155) | pub fn new(root_path: &Path, name: &str, state: State, bmap: BucketMap...
method logical_size (line 180) | pub fn logical_size(&self) -> u64 {
method close (line 185) | pub fn close(mut self) -> Result<(), Error> {
method truncate (line 196) | pub fn truncate(&mut self, new_sz: usize) -> Result<(), Error> {
method put_at (line 212) | fn put_at(&mut self, key: u32, page_off: u64, buf: &[u8], val: &Value)...
method put (line 223) | pub fn put(&mut self, key: u32, page_off: u64, buf: &[u8]) -> Result<(...
method get (line 273) | pub fn get(&self, key: u32, page_off: u64, out_buf: &mut [u8]) -> Resu...
method get_value_opt (line 290) | fn get_value_opt(&self, key: u32) -> Result<Option<Value>, Error> {
method get_value (line 309) | fn get_value(&self, key: u32) -> Result<Value, Error> {
method sync_no_commit_lock (line 313) | pub (crate) fn sync_no_commit_lock(&mut self) -> Result<(), Error> {
method sync (line 331) | pub fn sync(&mut self) -> Result<(), Error> {
method delete_ver (line 337) | pub fn delete_ver(root_path: &Path, name: &str, ver: u32) -> Result<()...
type FileMap (line 354) | struct FileMap {
method init (line 359) | fn init(root_path: &Path, name: &str, vset: &HashSet<u32>, aver: u32) ...
method close (line 378) | fn close(&mut self) -> Result<(), Error> {
method close_versions (line 385) | fn close_versions(&mut self, vlist: &Vec<u32>, aver: u32) -> Result<()...
method data_path (line 397) | fn data_path(root_path: &Path, name: &str, ver: u32) -> PathBuf {
method add_file (line 401) | fn add_file(&mut self, root_path: &Path, name: &str, ver: u32) -> Resu...
method file_mut (line 411) | fn file_mut(&mut self, ver: u32) -> &mut NixFile {
method file (line 415) | fn file(&self, ver: u32) -> &NixFile {
FILE: crates/mojokv/src/error.rs
type Error (line 3) | pub enum Error {
FILE: crates/mojokv/src/index/mem.rs
type MemIndex (line 15) | pub struct MemIndex {
method new (line 21) | pub fn new(pps: usize) -> Self {
method header (line 28) | pub fn header(&self) -> &IndexHeader {
method key_map (line 32) | fn key_map(&self) -> &KeyMap {
method set_active_ver (line 36) | pub fn set_active_ver(&mut self, ver: u32) {
method active_ver (line 41) | pub fn active_ver(&self) -> u32 {
method max_key (line 45) | pub fn max_key(&self) -> isize {
method update_min_max_ver (line 49) | pub fn update_min_max_ver(&mut self) -> Vec<u32> {
method put (line 58) | pub fn put(&mut self, key: u32, off: u32) -> Result<(), Error> {
method get (line 69) | pub fn get(&self, key: u32) -> Result<Option<&Value>, Error> {
method truncate (line 73) | pub fn truncate(&mut self, key: u32) -> Result<(), Error> {
method iter (line 79) | pub fn iter<'a>(&'a self, from_key: u32, to_key: u32) -> Box<dyn Itera...
method serialize_to_path (line 89) | pub fn serialize_to_path(&self, filepath: &std::path::Path) -> Result<...
method deserialize_from_path (line 107) | pub fn deserialize_from_path(filepath: &std::path::Path) -> Result<(us...
type MemIndexIterator (line 121) | pub struct MemIndexIterator<'a> {
function new (line 128) | pub fn new(from_key: u32, to_key: u32, index: &'a MemIndex) -> Self {
type Item (line 138) | type Item = (u32, &'a Value);
method next (line 140) | fn next(&mut self) -> Option<Self::Item> {
FILE: crates/mojokv/src/index/mod.rs
constant MOJO_INDEX_MAGIC (line 8) | pub const MOJO_INDEX_MAGIC: &'static str = "mojo_index";
type IndexHeader (line 11) | pub struct IndexHeader {
method new (line 23) | pub fn new(pps: usize) -> Self {
type Index (line 40) | pub trait Index {
method put (line 41) | fn put(&mut self, key: u32, off: u32) -> Result<(), Error>;
method get (line 42) | fn get(&self, key: u32) -> Result<Option<&Value>, Error>;
method truncate (line 43) | fn truncate(&mut self, key: u32) -> Result<(), Error>;
type IndexSerde (line 46) | pub trait IndexSerde {
method serialize (line 47) | fn serialize<I: Index, W: std::io::Write>(idx: &I, w: &mut W) -> Resul...
method deserialize (line 48) | fn deserialize<I: Index, R: std::io::Read>(idx: &I, r: &mut R) -> Resu...
FILE: crates/mojokv/src/keymap.rs
type KeyMap (line 7) | pub struct KeyMap {
method new (line 14) | pub fn new(pps: usize) -> Self {
method alloc_value_arr (line 21) | fn alloc_value_arr(pps: usize) -> Vec<Value> {
method get_min_max_ver (line 26) | pub fn get_min_max_ver(&self) -> (u32, u32, HashSet<u32>) {
method put (line 47) | pub fn put(&mut self, key: u32, val: Value) {
method get (line 65) | pub fn get(&self, key: u32) -> Option<&Value> {
method truncate (line 78) | pub fn truncate(&mut self, key: u32) {
FILE: crates/mojokv/src/state.rs
type StateInner (line 10) | pub struct StateInner {
type State (line 24) | pub struct State {
method new (line 33) | pub fn new(page_sz: u32, pps: u32) -> Self {
method format_ver (line 54) | pub fn format_ver(&self) -> u32 {
method active_ver (line 59) | pub fn active_ver(&self) -> u32 {
method page_size (line 64) | pub fn page_size(&self) -> u32 {
method file_page_sz (line 69) | pub fn file_page_sz(&self) -> u32 {
method pps (line 74) | pub fn pps(&self) -> u32 {
method min_ver (line 79) | pub fn min_ver(&self) -> u32 {
method max_ver (line 84) | pub fn max_ver(&self) -> u32 {
method advance_ver (line 89) | pub fn advance_ver(&self) -> u32 {
method serialize_to_path (line 97) | pub fn serialize_to_path(&self, filepath: &std::path::Path) -> Result<...
method deserialize_from_path (line 105) | pub fn deserialize_from_path(filepath: &std::path::Path) -> Result<Sta...
FILE: crates/mojokv/src/store.rs
type StoreInner (line 11) | struct StoreInner {
method sync (line 210) | fn sync(&mut self) -> Result<(), Error> {
method sync_bmap (line 216) | fn sync_bmap(&mut self) -> Result<(), Error> {
method sync_state (line 227) | fn sync_state(&mut self) -> Result<(), Error> {
type Store (line 17) | pub struct Store {
method exists (line 22) | pub fn exists(&self, name: &str) -> bool {
method open (line 27) | pub fn open(&self, name: &str, mode: BucketOpenMode) -> Result<Bucket,...
method delete (line 63) | pub fn delete(&self, name: &str) -> Result<(), Error> {
method commit (line 71) | pub fn commit(&self) -> Result<u32, Error> {
method active_ver (line 93) | pub fn active_ver(&self) -> u32 {
method load_state (line 98) | pub fn load_state(rootpath: &Path) -> Result<State, Error> {
method readonly (line 105) | pub fn readonly(root_path: &Path, ver: u32) -> Result<Self, Error> {
method writable (line 111) | pub fn writable(rootpath: &Path, create: bool, page_sz: Option<u32>, p...
method load_store (line 143) | fn load_store(root_path: &Path, state: State, ver: u32) -> Result<Stor...
method get_index (line 159) | pub fn get_index(&self, name: &str) -> Result<Option<(usize, usize, Me...
method new (line 174) | fn new(root_path: &Path, page_sz: u32, pps: u32) -> Result<Self, Error> {
method init (line 191) | fn init(&mut self) -> Result<(), Error> {
method create_lock_file (line 202) | fn create_lock_file(root_path: &Path) -> Result<LockFile, Error> {
type BucketOpenMode (line 241) | pub enum BucketOpenMode {
method is_write (line 247) | pub fn is_write(&self) -> bool {
FILE: crates/mojokv/src/utils.rs
function load_file (line 6) | pub fn load_file(path: &Path, buf: &mut Vec<u8>) -> Result<(), Error> {
function write_file (line 12) | pub fn write_file(path: &Path, buf: &[u8]) -> Result<(), Error> {
function touch_file (line 25) | pub fn touch_file(path: &Path) -> Result<(), Error> {
FILE: crates/mojokv/src/value.rs
type Slot (line 6) | pub type Slot = Option<Vec<Value>>;
type Value (line 10) | pub struct Value {
method fmt (line 16) | fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt:...
method is_allocated (line 22) | pub fn is_allocated(&self) -> bool {
method deallocate (line 26) | pub fn deallocate(&mut self) {
method put_off (line 31) | pub fn put_off(&mut self, off: u32) {
method get_off (line 35) | pub fn get_off(&self) -> u32 {
method put_ver (line 39) | pub fn put_ver(&mut self, v: u32) {
method get_ver (line 43) | pub fn get_ver(&self) -> u32 {
method deserialize (line 80) | fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
method serialize (line 50) | fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
type ValueVisitor (line 57) | struct ValueVisitor {}
type Value (line 60) | type Value = Value;
method visit_bytes (line 62) | fn visit_bytes<E>(self, v: &[u8]) -> Result<Self::Value, E>
method expecting (line 73) | fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::...
FILE: sqlite-ext/mojo.c
function mojo_fullpath_name (line 34) | static int mojo_fullpath_name(
function sqlite3_mojo_init (line 54) | int sqlite3_mojo_init(sqlite3 *db, char **pzErrMsg, const sqlite3_api_ro...
FILE: sqlite-ext/mojofs.h
type MojoFile (line 6) | typedef struct MojoFile {
FILE: test-scripts/commands.py
class TestConfig (line 8) | class TestConfig:
method __init__ (line 11) | def __init__(self, page_sz=4096, journal_mode="WAL", vac_mode="NONE"):
method __repr__ (line 16) | def __repr__(self):
function vacuum (line 19) | def vacuum(cur):
function opendb (line 23) | def opendb(cfg, db_path, ver="1", mode=""):
function mkdir (line 40) | def mkdir(dir_path):
function commit_version (line 44) | def commit_version(dbpath):
function create_table_person (line 48) | def create_table_person(cur):
function get_row_count (line 58) | def get_row_count(cur, table):
function table_person_count (line 65) | def table_person_count(conn):
function get_max_id_person (line 71) | def get_max_id_person(cur):
function insert_table_person (line 78) | def insert_table_person(cur, count):
function delete_table_person (line 87) | def delete_table_person(cur, from_id, to_id):
function drop_table_person (line 92) | def drop_table_person(cur):
function copy_table_person (line 97) | def copy_table_person(cur, new_table):
FILE: test-scripts/perftest.py
function rm_dir (line 9) | def rm_dir(path):
function rm_fr (line 19) | def rm_fr(path):
function _mojo_conn_str (line 35) | def _mojo_conn_str(db_path, ver="1", mode=""):
function _std_conn_str (line 44) | def _std_conn_str(db_path, mode=""):
function open_db (line 51) | def open_db(db_path, vfs="mojo", ver="1", mode=""):
function create_table (line 65) | def create_table(conn):
function insert_rows (line 69) | def insert_rows(conn, row_count, ver, suffix=""):
function count_rows (line 77) | def count_rows(conn, condition=""):
function select_rows (line 88) | def select_rows(conn, condition=""):
function update_text_rows (line 102) | def update_text_rows(conn):
function load_extension (line 109) | def load_extension(mojo_lib):
function perf_insert (line 123) | def perf_insert(conn):
function perf_select (line 133) | def perf_select(conn):
function perf_count_rows (line 142) | def perf_count_rows(conn):
function perf_update_rows (line 151) | def perf_update_rows(conn):
FILE: test-scripts/test.sql
type test (line 6) | create table if not exists test (
FILE: test-scripts/testdb.py
class TestConfig (line 12) | class TestConfig:
method __init__ (line 15) | def __init__(self, page_sz=4096, journal_mode="WAL", vac_mode="NONE", ...
method __repr__ (line 21) | def __repr__(self):
function rm_dir (line 24) | def rm_dir(path):
function rm_fr (line 34) | def rm_fr(path):
class MojoWritableTest (line 50) | class MojoWritableTest(unittest.TestCase):
method __init__ (line 53) | def __init__(self, cfg, dbpath, *args, **kargs):
method setUp (line 59) | def setUp(self):
method tearDown (line 62) | def tearDown(self):
method _subtest_name (line 66) | def _subtest_name(self, name):
method begin (line 69) | def begin(self, cur):
method commit (line 74) | def commit(self, cur):
method rollback (line 79) | def rollback(self, cur):
method test_db_use (line 84) | def test_db_use(self):
function load_extension (line 167) | def load_extension(mojo_lib):
function create_suite (line 179) | def create_suite(full_mode):
Condensed preview — 53 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (137K chars).
[
{
"path": ".gitignore",
"chars": 111,
"preview": "target\nlog\nlog*\n*.log\n*.dylib\n.vscode\n*.db\n*.swp\n*-journal\n*-wal\nbuild\nCargo.lock\ntestdbs\n__pycache__\n.DS_Store"
},
{
"path": "Cargo.toml",
"chars": 113,
"preview": "[workspace]\n\n\nmembers = [\n \"crates/mojofs\",\n \"crates/mojokv\",\n \"crates/mojoio\",\n \"crates/mojo-cli\",\n]"
},
{
"path": "LICENSE",
"chars": 1070,
"preview": "MIT License\n\nCopyright (c) 2022 Sudeep Jathar\n\nPermission is hereby granted, free of charge, to any person obtaining a c"
},
{
"path": "README.md",
"chars": 8424,
"preview": "# MojoFS\n\nMojoFS is a versioning, userspace filesystem for sqlite DB. It is tailor made for sqlite and not supposed to b"
},
{
"path": "build.sh",
"chars": 1548,
"preview": "#!/usr/bin/env bash\n\nexport BUILD_PROFILE=debug\nexport CARGO_OPT=\"\"\nexport MESON_BUILD_PROFILE=\"debug\"\nexport MESON_BUIL"
},
{
"path": "crates/mojo-cli/Cargo.toml",
"chars": 305,
"preview": "[package]\nname = \"mojo-cli\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rus"
},
{
"path": "crates/mojo-cli/src/buckets.rs",
"chars": 281,
"preview": "use anyhow::Error;\nuse mojokv::BucketMap;\n\npub fn cmd(kvpath: &std::path::Path, ver: u32) -> Result<(), Error> {\n let"
},
{
"path": "crates/mojo-cli/src/commit.rs",
"chars": 328,
"preview": "use anyhow::Error;\nuse mojokv::Store;\n\npub fn cmd(kvpath: &std::path::Path) -> Result<(), Error> {\n let st = Store::w"
},
{
"path": "crates/mojo-cli/src/diff.rs",
"chars": 1582,
"preview": "use anyhow::Error;\nuse mojokv::KVStore;\n\npub fn cmd(kvpath: &std::path::Path, fver: u32, tver: u32) -> Result<(), Error>"
},
{
"path": "crates/mojo-cli/src/iget.rs",
"chars": 478,
"preview": "use anyhow::Error;\nuse mojokv::{Store,BucketOpenMode};\n\npub fn cmd(kvpath: &std::path::Path, bucket: &str, ver: u32, key"
},
{
"path": "crates/mojo-cli/src/iview.rs",
"chars": 1312,
"preview": "use anyhow::Error;\nuse mojokv::{Store};\n\npub fn cmd(kvpath: &std::path::Path, name: &str, ver: u32, additional: bool, ke"
},
{
"path": "crates/mojo-cli/src/main.rs",
"chars": 2117,
"preview": "\nmod iget;\nmod iview;\nmod state;\nmod commit;\nmod buckets;\n\nuse anyhow::Error;\nuse clap::{Parser, Subcommand};\n\n#[derive("
},
{
"path": "crates/mojo-cli/src/state.rs",
"chars": 991,
"preview": "use anyhow::Error;\nuse mojokv::{self, Store};\nuse std::mem::size_of;\n\npub fn cmd(kvpath: &std::path::Path, additional: b"
},
{
"path": "crates/mojo-cli/src/truncate.rs",
"chars": 458,
"preview": "use anyhow::Error;\nuse mojokv::KVStore;\n\npub fn cmd(kvpath: &std::path::Path, sz: usize) -> Result<(), Error> {\n let "
},
{
"path": "crates/mojofs/.gitignore",
"chars": 6,
"preview": "testfs"
},
{
"path": "crates/mojofs/Cargo.toml",
"chars": 456,
"preview": "[package]\nname = \"mojofs\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-"
},
{
"path": "crates/mojofs/meson.build",
"chars": 402,
"preview": "project('cmojo', 'c')\n\nif get_option('buildtype') == 'release'\n mojo_lib_dir = meson.source_root() + '/../target/rele"
},
{
"path": "crates/mojofs/src/error.rs",
"chars": 1917,
"preview": "\npub const MOJOFS_ERR_NOT_IMPL: i32 = 1;\npub const MOJOFS_ERR_IO: i32 = 2;\npub const MOJOFS_ERR_NIX: i32 = 3;\npub const "
},
{
"path": "crates/mojofs/src/kvfile.rs",
"chars": 2644,
"preview": "\nuse mojokv::Bucket;\nuse crate::Error;\n\npub struct KVFile {\n pub bucket: Bucket,\n opt: KVFileOpt,\n}\n\n#[derive(Clon"
},
{
"path": "crates/mojofs/src/lib.rs",
"chars": 13584,
"preview": "\npub mod vfs;\npub mod error;\npub mod open_options;\npub mod vfsfile;\nmod native_file;\nmod kvfile;\n\nuse std::ffi::CStr;\nus"
},
{
"path": "crates/mojofs/src/native_file.rs",
"chars": 2452,
"preview": "\nuse std::path::{Path, PathBuf};\nuse nix::fcntl::{self, OFlag};\nuse nix::sys::stat::Mode;\n\nuse crate::open_options::*;\nu"
},
{
"path": "crates/mojofs/src/open_options.rs",
"chars": 2763,
"preview": "use libsqlite3_sys as ffi;\n\n#[derive(Debug, Clone, PartialEq)]\npub struct OpenOptions {\n pub flags: i32,\n\n /// The"
},
{
"path": "crates/mojofs/src/vfs.rs",
"chars": 5063,
"preview": "\nuse std::path::{PathBuf, Path};\nuse crate::{error, Error};\nuse crate::open_options::*;\nuse std::collections::HashMap;\nu"
},
{
"path": "crates/mojofs/src/vfsfile.rs",
"chars": 3244,
"preview": "use crate::error::Error;\n\nuse crate::native_file::NativeFile;\nuse crate::kvfile::KVFile;\nuse crate::open_options::OpenOp"
},
{
"path": "crates/mojofs/tests/mojofs_test.rs",
"chars": 3289,
"preview": "use std::path::Path;\nuse anyhow::Error;\nuse std::collections::HashMap;\nuse mojofs::{self, VFS, VFSFile};\n\nfn remove_fs(r"
},
{
"path": "crates/mojoio/Cargo.toml",
"chars": 276,
"preview": "[package]\nname = \"mojoio\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-"
},
{
"path": "crates/mojoio/src/error.rs",
"chars": 432,
"preview": "\n#[derive(thiserror::Error, Debug)]\npub enum Error {\n #[error(\"io error\")]\n IoErr(#[from] std::io::Error),\n\n #["
},
{
"path": "crates/mojoio/src/lib.rs",
"chars": 345,
"preview": "pub mod nix;\nmod error;\n\npub use error::Error;\n\npub const BUFFER_MAGIC: &[u8] = b\"mojo\";\npub const PAGE_HEADER_LEN: usiz"
},
{
"path": "crates/mojoio/src/nix.rs",
"chars": 4432,
"preview": "use std::path::Path;\n\nuse nix::fcntl::{self, OFlag};\nuse nix::sys::stat::Mode;\n\n//use crate::value;\nuse crate::Error;\n\np"
},
{
"path": "crates/mojokv/.gitignore",
"chars": 36,
"preview": "/target\n/Cargo.lock\n*.db\nlog*\n*.log\n"
},
{
"path": "crates/mojokv/Cargo.toml",
"chars": 583,
"preview": "[package]\nname = \"mojokv\"\nversion = \"0.2.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-"
},
{
"path": "crates/mojokv/src/bmap.rs",
"chars": 2182,
"preview": "\nuse std::path::{Path, PathBuf};\nuse std::collections::HashMap;\nuse std::sync::Arc;\nuse crate::bucket::Bucket;\nuse crate"
},
{
"path": "crates/mojokv/src/bucket.rs",
"chars": 13441,
"preview": "\nuse std::collections::HashSet;\nuse std::path::{Path, PathBuf};\nuse crate::{Error, BucketMap};\nuse mojoio::nix::NixFile;"
},
{
"path": "crates/mojokv/src/error.rs",
"chars": 1473,
"preview": "\n#[derive(thiserror::Error, Debug)]\npub enum Error {\n #[error(\"io error\")]\n IoErr(#[from] std::io::Error),\n\n #["
},
{
"path": "crates/mojokv/src/index/mem.rs",
"chars": 4364,
"preview": "use std::io::Write;\n\nuse serde::Deserialize;\nuse serde::Serialize;\n\nuse crate::value::Value;\nuse crate::keymap::KeyMap;\n"
},
{
"path": "crates/mojokv/src/index/mod.rs",
"chars": 1220,
"preview": "pub mod mem;\nuse std::collections::HashSet;\n\nuse crate::Error;\nuse crate::value::Value;\nuse serde::{Serialize, Deseriali"
},
{
"path": "crates/mojokv/src/keymap.rs",
"chars": 2581,
"preview": "\nuse std::collections::HashSet;\nuse crate::value::{Value, Slot};\nuse serde::{Serialize, Deserialize};\n\n#[derive(Serializ"
},
{
"path": "crates/mojokv/src/lib.rs",
"chars": 341,
"preview": "//#![feature(write_all_vectored)]\n\npub mod index;\nmod bucket;\nmod error;\nmod value;\nmod state;\nmod keymap;\nmod utils;\nmo"
},
{
"path": "crates/mojokv/src/state.rs",
"chars": 2536,
"preview": "\nuse crate::Error;\nuse mojoio::nix::NixFile;\nuse crate::utils;\nuse std::sync::Arc;\nuse parking_lot::RwLock;\nuse serde::{"
},
{
"path": "crates/mojokv/src/store.rs",
"chars": 7184,
"preview": "use std::path::{Path, PathBuf};\nuse std::sync::Arc;\nuse crate::{Error, utils};\nuse crate::state::State;\nuse crate::bucke"
},
{
"path": "crates/mojokv/src/utils.rs",
"chars": 1274,
"preview": "use std::path::Path;\nuse std::io::{Read, Write};\n\nuse crate::Error;\n\npub fn load_file(path: &Path, buf: &mut Vec<u8>) ->"
},
{
"path": "crates/mojokv/src/value.rs",
"chars": 2942,
"preview": "\nuse modular_bitfield::{bitfield, specifiers::*};\nuse serde::{Serialize, Deserialize};\nuse serde::de::Visitor;\n\npub type"
},
{
"path": "docs/design.md",
"chars": 5972,
"preview": "- [Design choices](#design-choices)\n- [Design](#design)\n - [Layers](#layers)\n - [MojoKV](#mojokv)\n - [Index](#index"
},
{
"path": "docs/source.md",
"chars": 2038,
"preview": "\nWhether you want to just read or contribute to mojo, this document describes the code to get you started.\n\n*__Note__*: "
},
{
"path": "docs/user-guide.md",
"chars": 2787,
"preview": "\nThis document assumes that you have followed the build instructions and/or have the libmojo shared library.\n\n- [Opening"
},
{
"path": "meson.build",
"chars": 610,
"preview": "project('mojo', 'c', default_options: ['c_std=c11'], version: '0.1.0')\n\ncompiler = meson.get_compiler('c')\n\nmojokv_lib_p"
},
{
"path": "sqlite-ext/mojo.c",
"chars": 1777,
"preview": "#include <stdio.h>\n\n#include <sqlite3ext.h>\nSQLITE_EXTENSION_INIT1\n\n#include <assert.h>\n#include <string.h>\n#include <sy"
},
{
"path": "sqlite-ext/mojofs.h",
"chars": 200,
"preview": "#ifndef __mojo_h\n#define __mojo_h\n\n#include <sqlite3.h>\n\ntypedef struct MojoFile {\n sqlite3_vfs base;\n void* custo"
},
{
"path": "test-scripts/commands.py",
"chars": 2947,
"preview": "\"\"\"Multiple commands for testing\"\"\"\n\nimport subprocess\nimport sqlite3\n\nMOJOKV_CLI=None\n\nclass TestConfig:\n \"\"\"Config "
},
{
"path": "test-scripts/perftest.py",
"chars": 4836,
"preview": "\"\"\"\" Perf test for mojo filesystem \"\"\"\n\nimport sqlite3\nimport os\nimport shutil\nimport time\nimport sys\n\ndef rm_dir(path):"
},
{
"path": "test-scripts/test.sql",
"chars": 223,
"preview": "\npragma page_size = 4096;\n.load ./build/libmojo\n.open 'file:a.db?vfs=mojo&pagesz=4096'\n\ncreate table if not exists test "
},
{
"path": "test-scripts/test2.sql",
"chars": 151,
"preview": "\npragma page_size = 4096;\n.load ./build/libmojo\n.open 'file:./testdbs/a_1.db?vfs=mojo&pagesz=4096&mode=ro&ver=2'\n\nselect"
},
{
"path": "test-scripts/testdb.py",
"chars": 6772,
"preview": "\"\"\"\" Tests for mojo filesystem \"\"\"\n\nimport os\nimport sys\nimport unittest\nimport sqlite3\nimport shutil\nimport commands as"
}
]
About this extraction
This page contains the full source code of the sudeep9/mojo GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 53 files (125.9 KB), approximately 36.1k tokens, and a symbol index with 326 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.