[ { "path": ".gitignore", "content": "target\nlog\nlog*\n*.log\n*.dylib\n.vscode\n*.db\n*.swp\n*-journal\n*-wal\nbuild\nCargo.lock\ntestdbs\n__pycache__\n.DS_Store" }, { "path": "Cargo.toml", "content": "[workspace]\n\n\nmembers = [\n \"crates/mojofs\",\n \"crates/mojokv\",\n \"crates/mojoio\",\n \"crates/mojo-cli\",\n]" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2022 Sudeep Jathar\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "# MojoFS\n\nMojoFS is a versioning, userspace filesystem for sqlite DB. It is tailor made for sqlite and not supposed to be used as a general purpose fs.\n\nThe main feature of the fs is versioning/snapshotting. Only one version is writable and all the old versions are immutable.\n\n- [MojoFS](#mojofs)\n- [License](#license)\n- [Development status](#development-status)\n- [Build](#build)\n- [Usage](#usage)\n - [Create database using mojo and insert few records](#create-database-using-mojo-and-insert-few-records)\n - [Select data](#select-data)\n - [Commit the database](#commit-the-database)\n - [Write to active version=2](#write-to-active-version2)\n - [Read old version=1](#read-old-version1)\n - [Read active version=2:](#read-active-version2)\n- [Docs](#docs)\n - [Source map](#source-map)\n- [Limits](#limits)\n- [Testing](#testing)\n- [Performance](#performance)\n- [Road to v1.0](#road-to-v10)\n\n# License\n\nI have changed it to MIT. See the [LICENSE](/LICENSE) file.\n\n# Development status\n\n\n|Item|Value|\n|-------|------|\n|Quality|pre-alpha|\n|Maintenance|active|\n\n# Build\n\nAt present only mac/linux is supported. Windows is not supported only because I do not have windows machine. This may change in future.\n\nThe build expects the following in the environment:\n\n1. Meson Build + Ninja (see [here](https://mesonbuild.com/Getting-meson.html))\n2. C compiler (gcc or clang)\n3. Rust version v1.59+\n4. sqlite headers + libraries\n5. Optional: python3 for testing (most versions should be ok)\n\n```\ngit clone github.com/sudeep9/mojo\ncd mojo\n./build.sh release\n```\n\nFollowing artifacts will be in `build` dir:\n\n* `build/libmojo.dylib` (`.so` extension in linux) => The sqlite extension/vfs \n* `build/mojo-cli` => mojo cli tool to manage the file system\n\n# Usage\n\nAll the examples below uses sqlite3 binary. However, you can use any bindings of sqlite.\n\n## Create database using mojo and insert few records\n\n```\nrm -fR a.db\nsqlite3 < [here](https://github.com/sudeep9/mojo/blob/main/docs/design.md)\n* user-guide => [here](https://github.com/sudeep9/mojo/blob/main/docs/user-guide.md)\n\n### Source map\n\n* All the rust code is under `crates` folder\n* All the docs are under `docs` folder\n* The `sqlite-ext` folder has C code which is compiled down to shared lib\n* The `test-scripts` has various assorted test scripts which includes perf & black-box test\n\nCrate wise details are at: [here](https://github.com/sudeep9/mojo/blob/main/docs/source.md)\n\n# Limits\n\nFor a page size = 4KB (which is the default for sqlite for some years now) following are the limits:\n\n* Max sqlite db file size (KB) = `pow(2,32) * 4` = 17179869184 KB or 16TB \n\n 16TB is logical size i.e. file size reported by stat like call in any version. Since there could be multiple versions of the file, the total size of all such versions taken together can exceed 16TB.\n\n* Max version num = `pow(2,24)` = 16777216 or 16 million\n\n To put 16M versions in perspective, even if you create a version every 1 min, it will take ~31.92 years to reach the max version.\n\nNote: All these limits are actually artificial to keep the memory usage reasonable. In future, these will be tunable and also have the ability to baseline the versions.\n\n# Testing\n\nI wanted to use sqlite [test harness](https://www.sqlite.org/th3.html) but it requires license. Quoting from the test harness link:\n\n> SQLite itself is in the public domain and can be used for any purpose. But TH3 is proprietary and requires a license.\n\nInstead there is `testdb.py` for black box testing and `perftest.py` for perf tests.\nAt present the `testdb.py` tests combinations of the following:\n\n``` \npage_sizes = [4096]\njournal_modes = [\"OFF\", \"WAL\", \"MEMORY\", \"DELETE\", \"TRUNCATE\", \"PERSIST\"]\nvacuum_modes = [\"NONE\", \"FULL\", \"INCREMENTAL\"]\n```\n\nFor each of the combination, there are about ~11 subtests so in all 18 x 11 = 198 tests.\nThese are early days and off-course there is a long way to go.\n\nTo run the full suite:\n\n```\npython3 testdb.py build/libmojo full\n```\n\n\n# Performance\n\nAbout `10_000_000` rows are inserted and then for reading we select the rows and get the row count.\nFinally it updates half the rows.\n\nTo run the perf test:\n\n```\nMOJOKV_CLI=build/mojo-cli python3 perftest.py ./build/libmojo\n```\n\nOutput on 2018/19 macbook:\n\n```\nRunning perf for: insert\n\tvfs=std time elapsed (s): 20.524982929229736\n\tvfs=mojo time elapsed (s): 21.202611923217773\n\tMojo takes 1.033 times than std vfs\n------------------------\nRunning perf for: update rows\n\tvfs=std time elapsed (s): 2.871242046356201\n\tvfs=mojo time elapsed (s): 2.439574956893921\n\tMojo takes 0.85 times than std vfs\n------------------------\nRunning perf for: select\nselect iter count: 10000000\n\tvfs=std time elapsed (s): 8.659775018692017\nselect iter count: 10000000\n\tvfs=mojo time elapsed (s): 5.907814025878906\n\tMojo takes 0.682 times than std vfs\n------------------------\nRunning perf for: row count\nrow count: 0\n\tvfs=std time elapsed (s): 2.96425199508667\nrow count: 0\n\tvfs=mojo time elapsed (s): 1.5106308460235596\n\tMojo takes 0.51 times than std vfs\n------------------------\n```\n\nThe writes being only `1.033` times worse is in line with my expectations. However, I am investigating why the reads are so better with mojo. \n\nMy guess as of now is that in the standard default vfs at https://github.com/sqlite/sqlite/blob/master/src/os_unix.c does not use pread, whereas mojo uses pread. Lack of pread results into two system call i.e. seek + read. This ***might*** explain the perf difference. This need further confirmation though.\n\nSee the comment and code in the c file above:\n\n```\n** ... Since SQLite does not define USE_PREAD\n** in any form by default, we will not attempt to define _XOPEN_SOURCE.\n** See tickets #2741 and #2681.\n```\n\nIn seekAndRead function:\n\n```\n#if defined(USE_PREAD)\n got = osPread(id->h, pBuf, cnt, offset);\n SimulateIOError( got = -1 );\n#elif defined(USE_PREAD64)\n got = osPread64(id->h, pBuf, cnt, offset);\n SimulateIOError( got = -1 );\n#else\n newOffset = lseek(id->h, offset, SEEK_SET);\n SimulateIOError( newOffset = -1 );\n if( newOffset<0 ){\n storeLastErrno((unixFile*)id, errno);\n return -1;\n }\n got = osRead(id->h, pBuf, cnt);\n```\n\n# Road to v1.0\n\nIt needs atleast the following:\n\n- [ ] Top-notch unit & black box test coverage\n- [ ] Ease of use e.g debugability, add more mojo-cli admin commands\n- [ ] Ability to diff the versions\n- [ ] Ability to delete versions\n- [ ] Ability to merge versions (not like git merge)\n- [ ] Ability to recover from corrupted fs.\n- [ ] Stabilize on-disk format\n- [ ] User guide\n\nA lot of the above needs to be clearly defined.\n" }, { "path": "build.sh", "content": "#!/usr/bin/env bash\n\nexport BUILD_PROFILE=debug\nexport CARGO_OPT=\"\"\nexport MESON_BUILD_PROFILE=\"debug\"\nexport MESON_BUILD_DIR=\"build\"\n\nif [ $# -gt 0 ]; then\n if [ $1 = \"release\" ]; then\n export BUILD_PROFILE=release\n export CARGO_OPT=\"--release\"\n export MESON_BUILD_PROFILE=\"release\"\n fi\nfi\n\necho \"BUILD_PROFILE=$BUILD_PROFILE\"\n\n\necho \"-------------------------------\"\necho \" [Rust build starting]\"\necho \"-------------------------------\"\n\ncargo build $CARGO_OPT\nif [ $? -ne 0 ]; then\n echo \"Error: Cargo build failed with $?\"\n exit $?\nfi\n\nmkdir -p $MESON_BUILD_DIR\n\nrm -fR $MESON_BUILD_DIR/*\n\ncp target/$BUILD_PROFILE/mojo-cli $MESON_BUILD_DIR/.\nif [ $? -ne 0 ]; then\n echo \"Copying failed\"\n exit $?\nfi\n\necho \"-------------------------------\"\necho \" [C build starting]\"\necho \"-------------------------------\"\n\nmeson setup --buildtype=$MESON_BUILD_PROFILE $MESON_BUILD_DIR\nif [ $? -ne 0 ]; then\n echo \"Error: meson reconfigure failed with $?\"\n exit $?\nfi\n\nmeson compile -C $MESON_BUILD_DIR\nif [ $? -ne 0 ]; then\n echo \"Error: meson compile failed with $?\"\n exit $?\nfi\n\necho \">> The build artifacts are at the dir: $MESON_BUILD_DIR\"\n\nif [ \"$MOJO_TEST\" != \"\" ]; then\n echo \"-------------------------------\"\n echo \" [Mojo test starting]\"\n echo \"-------------------------------\"\n\n export MOJOKV_CLI=./target/$BUILD_PROFILE/mojo-cli\n\n python3 test-scripts/testdb.py $MESON_BUILD_DIR/mojo\n if [ $? -ne 0 ]; then\n echo \"Error: mojo test failed with $?\"\n exit 1\n fi\nfi" }, { "path": "crates/mojo-cli/Cargo.toml", "content": "[package]\nname = \"mojo-cli\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nmojokv = {path=\"../mojokv\"}\nlog = \"0.4.17\"\nenv_logger = \"0.9.0\"\nclap = {version=\"3.2.6\", features=[\"derive\"] }\nanyhow = \"1.0.58\"" }, { "path": "crates/mojo-cli/src/buckets.rs", "content": "use anyhow::Error;\nuse mojokv::BucketMap;\n\npub fn cmd(kvpath: &std::path::Path, ver: u32) -> Result<(), Error> {\n let bmap = BucketMap::load(kvpath, ver)?; \n\n for (bucket_name, ver) in bmap.map()?.iter() {\n println!(\"{} -> {}\", bucket_name, ver);\n }\n Ok(())\n}" }, { "path": "crates/mojo-cli/src/commit.rs", "content": "use anyhow::Error;\nuse mojokv::Store;\n\npub fn cmd(kvpath: &std::path::Path) -> Result<(), Error> {\n let st = Store::writable(&kvpath, false, None, None)?;\n\n println!(\"active version before commit: {}\", st.active_ver());\n let new_ver = st.commit()?;\n println!(\"active version after commit: {}\", new_ver);\n Ok(())\n}" }, { "path": "crates/mojo-cli/src/diff.rs", "content": "use anyhow::Error;\nuse mojokv::KVStore;\n\npub fn cmd(kvpath: &std::path::Path, fver: u32, tver: u32) -> Result<(), Error> {\n if fver >= tver {\n return Err(Error::msg(\"'from' version cannot be greater than or equal 'to' version\"));\n }\n\n let state = KVStore::load_state(kvpath)?;\n let f_index = KVStore::load_index(kvpath, fver)?;\n let t_index = KVStore::load_index(kvpath, tver)?;\n\n let mut key = 0u32;\n for (slot_index, t_slot) in t_index.kmap.slot_map.iter().enumerate() {\n let f_slot = &f_index.kmap.slot_map[slot_index];\n\n match (f_slot, t_slot) {\n (Some(fs), Some(ts)) => {\n if fs.len() != ts.len() {\n return Err(Error::msg(format!(\"Slot length mismatch {} {}\", fs.len(), ts.len()))); \n }\n\n for (j,(fv, tv)) in std::iter::zip(fs, ts).enumerate() {\n if fv.get_ver() != tv.get_ver() {\n println!(\"M k={} fv={} tv={} fo={} to={}\",\n key+j as u32, \n fv.get_ver(),\n tv.get_ver(),\n fv.get_off(),\n tv.get_off());\n }\n }\n },\n (Some(_fs), None) => {\n println!(\"D {} -> {} deleted\", key, key + state.pps);\n },\n (None, Some(_ts)) => {\n println!(\"A {} -> {} added\", key, key + state.pps);\n },\n (None, None) => {},\n }\n\n key += state.pps;\n }\n\n Ok(())\n}" }, { "path": "crates/mojo-cli/src/iget.rs", "content": "use anyhow::Error;\nuse mojokv::{Store,BucketOpenMode};\n\npub fn cmd(kvpath: &std::path::Path, bucket: &str, ver: u32, key: u32) -> Result<(), Error> {\n let st = Store::readonly(&kvpath, ver)?;\n let b = st.open(bucket, BucketOpenMode::Read)?;\n\n println!(\"Max key: {}\", b.max_key());\n match b.get_key(key)? {\n Some(val) => {\n println!(\"value: {:?}\", val);\n },\n None => {\n println!(\"Key not found\")\n }\n }\n Ok(())\n}" }, { "path": "crates/mojo-cli/src/iview.rs", "content": "use anyhow::Error;\nuse mojokv::{Store};\n\npub fn cmd(kvpath: &std::path::Path, name: &str, ver: u32, additional: bool, keys: bool) -> Result<(), Error> {\n let st = Store::readonly(kvpath, ver)?;\n let ret = st.get_index(name)?; //Bucket::load_index(&kvpath, name, ver)?;\n\n if ret.is_none() {\n println!(\"Bucket {} does not exists\", name);\n }\n\n let (uncomp_sz, comp_sz, i) = ret.unwrap();\n\n let h = i.header();\n\n let st = if additional {\n Some(Store::load_state(kvpath)?)\n }else{\n None\n };\n\n println!(\"Format version : {}\", h.format_ver);\n println!(\"Minimum version : {}\", h.min_ver);\n println!(\"Maximum version : {}\", h.max_ver);\n println!(\"Active version : {}\", h.active_ver);\n println!(\"Pages per slot : {}\", h.pps);\n println!(\"Maximum key : {}\", h.max_key);\n println!(\"Compressed size : {}\", comp_sz);\n println!(\"Uncompressed size : {}\", uncomp_sz);\n\n if let Some(st) = st {\n println!(\"----------------------\");\n println!(\"Logical size : {}\", st.page_size() * (h.max_key + 1) as u32);\n }\n\n if keys {\n println!(\"----------------------\");\n println!(\"keys\");\n for (key, val) in i.iter(0, 0) {\n println!(\" {} {:?}\", key, val);\n }\n }\n \n\n Ok(())\n}" }, { "path": "crates/mojo-cli/src/main.rs", "content": "\nmod iget;\nmod iview;\nmod state;\nmod commit;\nmod buckets;\n\nuse anyhow::Error;\nuse clap::{Parser, Subcommand};\n\n#[derive(Parser)]\n#[clap(author, version, about, long_about = None)]\n#[clap(propagate_version = true)]\nstruct Cli {\n kvpath: std::path::PathBuf,\n\n #[clap(subcommand)]\n command: Commands,\n}\n\n#[derive(Subcommand)]\nenum Commands {\n /// View index value for a key\n #[clap(name=\"iget\")]\n MemIndexGet{\n #[clap(value_parser)]\n bucket: String,\n\n #[clap(value_parser)]\n ver: u32,\n\n #[clap(value_parser)]\n key: u32,\n },\n /// View index value for a key\n #[clap(name=\"iview\")]\n MemIndexView{\n #[clap(value_parser)]\n bucket: String,\n\n #[clap(value_parser)]\n ver: u32,\n\n #[clap(short, action)]\n additional: bool,\n\n #[clap(short, action)]\n keys: bool\n },\n /// View the current kv state\n #[clap(name=\"state\")]\n State{\n /// Print additional internal numbers\n #[clap(short, action)]\n additional: bool\n },\n /// Commit the store\n #[clap(name=\"commit\")]\n Commit{\n },\n /// List buckets\n #[clap(name=\"buckets\")]\n Buckets{\n\n /// Version of the bmap\n #[clap(value_parser)]\n ver: u32,\n },\n}\n\nfn main() -> Result<(), Error> {\n env_logger::init();\n let cli = Cli::parse();\n\n // You can check for the existence of subcommands, and if found use their\n // matches just as you would the top level cmd\n\n match &cli.command {\n Commands::MemIndexGet{bucket, ver, key} => {\n iget::cmd(&cli.kvpath, bucket.as_str(), *ver, *key)?;\n },\n Commands::MemIndexView{bucket, ver, additional, keys} => {\n iview::cmd(&cli.kvpath, bucket.as_str(), *ver, *additional, *keys)?;\n },\n Commands::State{additional} => {\n state::cmd(&cli.kvpath, *additional)?;\n },\n\n Commands::Commit{} => {\n commit::cmd(&cli.kvpath)?;\n },\n Commands::Buckets{ver} => {\n buckets::cmd(&cli.kvpath, *ver)?;\n },\n }\n\n Ok(())\n}\n" }, { "path": "crates/mojo-cli/src/state.rs", "content": "use anyhow::Error;\nuse mojokv::{self, Store};\nuse std::mem::size_of;\n\npub fn cmd(kvpath: &std::path::Path, additional: bool) -> Result<(), Error> {\n let st = Store::load_state(kvpath)?;\n\n println!(\"Format version : {}\", st.format_ver());\n println!(\"Minimum version : {}\", st.min_ver());\n println!(\"Active version : {}\", st.active_ver());\n println!(\"Pages per slot : {}\", st.pps());\n println!(\"Page size : {}\", st.page_size());\n println!(\"File header len : {}\", st.file_page_sz());\n\n if additional {\n println!(\"----------------------------\");\n println!(\"Size of KVStore : {} bytes\", size_of::());\n println!(\"Size of MemIndex : {} bytes\", size_of::());\n println!(\"Size of KeyMap : {} bytes\", size_of::());\n println!(\"Size of Value : {} bytes\", size_of::());\n println!(\"Size of Slot : {} bytes\", size_of::());\n }\n\n Ok(())\n}" }, { "path": "crates/mojo-cli/src/truncate.rs", "content": "use anyhow::Error;\nuse mojokv::KVStore;\n\npub fn cmd(kvpath: &std::path::Path, sz: usize) -> Result<(), Error> {\n let st = KVStore::load_state(&kvpath)?;\n if sz % (st.page_size() as usize) != 0 {\n return Err(Error::msg(format!(\"Error: truncate size is not multiple of page sz ({})\", st.page_size())));\n }\n\n let mut store = KVStore::writable(kvpath, st.page_size(), Some(st.pps()))?;\n\n store.truncate(sz)?;\n store.sync()?;\n\n Ok(())\n}" }, { "path": "crates/mojofs/.gitignore", "content": "testfs" }, { "path": "crates/mojofs/Cargo.toml", "content": "[package]\nname = \"mojofs\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[lib]\ncrate-type = [\"staticlib\", \"lib\"]\n\n[dev-dependencies]\nanyhow = \"1.0\"\nenv_logger = \"0.9.0\"\n\n[dependencies]\nlibsqlite3-sys = {version = \"0.24.2\", features = [\"bundled\"]}\nthiserror = \"1.0.31\"\nparking_lot = \"0.12.1\"\nmojokv = {path = \"../mojokv\"}\nnix = \"0.24\"\nlog = \"0.4.17\"\nenv_logger = \"0.9.0\"" }, { "path": "crates/mojofs/meson.build", "content": "project('cmojo', 'c')\n\nif get_option('buildtype') == 'release'\n mojo_lib_dir = meson.source_root() + '/../target/release'\nelse\n mojo_lib_dir = meson.source_root() + '/../target/debug'\nendif\n\nmessage(mojo_lib_dir)\n\nmojo_lib = meson.get_compiler('c').find_library('mojofs',\n static: true,\n dirs: [mojo_lib_dir])\n\nshared_library('cmojo',\n sources:['cmojo.c'],\n dependencies: [mojo_lib])\n" }, { "path": "crates/mojofs/src/error.rs", "content": "\npub const MOJOFS_ERR_NOT_IMPL: i32 = 1;\npub const MOJOFS_ERR_IO: i32 = 2;\npub const MOJOFS_ERR_NIX: i32 = 3;\npub const MOJOFS_ERR_UTF8: i32 = 4;\npub const MOJOFS_ERR_MOJOKV: i32 = 5;\npub const MOJOFS_ERR_URL_PARSE: i32 = 6;\npub const MOJOFS_ERR_INT_PARSE: i32 = 7;\npub const MOJOFS_ERR_LARGE_PAGE: i32 = 8;\npub const MOJOFS_ERR_ARG_VER_MISSING: i32 = 9;\npub const MOJOFS_ERR_ARG_PAGESZ_MISSING: i32 = 10;\npub const MOJOFS_ERR_ARG_PPS_MISSING: i32 = 11;\n\n#[derive(thiserror::Error, Debug)]\npub struct Error {\n pub code: i32,\n pub msg: String,\n}\n\nimpl Error {\n pub fn new(code: i32, msg: String) -> Self {\n Error {\n code,\n msg,\n }\n }\n\n pub fn not_impl() -> Self {\n Error::new(MOJOFS_ERR_NOT_IMPL, \"Not implemented\".to_owned())\n }\n}\n\nimpl From for Error {\n fn from(err: std::io::Error) -> Self {\n Error { \n code: MOJOFS_ERR_IO,\n msg: format!(\"{:?}\", err),\n }\n }\n}\n\nimpl From for Error {\n fn from(err: nix::Error) -> Self {\n Error { \n code: MOJOFS_ERR_NIX,\n msg: err.to_string(),\n }\n }\n}\n\nimpl From for Error {\n fn from(err: std::str::Utf8Error) -> Self {\n Error { \n code: MOJOFS_ERR_UTF8,\n msg: err.to_string(),\n }\n }\n}\n\nimpl From for Error {\n fn from(err: std::num::ParseIntError) -> Self {\n Error { \n code: MOJOFS_ERR_INT_PARSE,\n msg: err.to_string(),\n }\n }\n}\n\nimpl From for Error {\n fn from(err: mojokv::Error) -> Self {\n Error { \n code: MOJOFS_ERR_MOJOKV,\n msg: err.to_string(),\n }\n }\n}\n\nimpl std::fmt::Display for Error {\n fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n write!(f, \"({}, {})\", self.code, self.msg)\n }\n}\n" }, { "path": "crates/mojofs/src/kvfile.rs", "content": "\nuse mojokv::Bucket;\nuse crate::Error;\n\npub struct KVFile {\n pub bucket: Bucket,\n opt: KVFileOpt,\n}\n\n#[derive(Clone)]\npub struct KVFileOpt {\n pub page_sz: u32,\n pub pps: u32,\n pub ver: u32,\n}\n\n\nimpl KVFile {\n pub fn open(bucket: Bucket, opt: KVFileOpt) -> Result {\n Ok(KVFile{\n bucket,\n opt,\n })\n }\n\n pub fn pread(&self, buf: &mut [u8], off: i64) -> Result {\n log::debug!(\"kv pread o={}, blen={}\", off, buf.len());\n\n if buf.len() > self.opt.page_sz as usize {\n return Err(Error::new(crate::MOJOFS_ERR_LARGE_PAGE, \n format!(\"buf larger ({}) than page size\", buf.len())));\n }\n\n let page_off = off % self.opt.page_sz as i64;\n let key = off / self.opt.page_sz as i64;\n\n let n = match self.bucket.get(key as u32, page_off as u64, buf) {\n Ok(n) => n,\n Err(err) => {\n if let mojokv::Error::KeyNotFoundErr(_) = err {\n 0\n }else{\n return Err(err.into());\n }\n }\n };\n\n if n Result<(), Error> {\n log::debug!(\"kv pwrite page key={}, po={} blen={}\", key, page_off, buf.len());\n\n self.bucket.put(key, page_off as u64, buf)?;\n\n Ok(())\n }\n\n pub fn pwrite(&mut self, off: i64, buf: &[u8]) -> Result<(), Error> {\n log::debug!(\"kv pwrite o={}, blen={}\", off, buf.len());\n\n let mut po = off % self.opt.page_sz as i64;\n let mut key = off / self.opt.page_sz as i64;\n let mut s = 0usize;\n let buflen = buf.len();\n\n while s < buflen {\n let e = (buflen-s).min(self.opt.page_sz as usize - po as usize);\n self.pwrite_page(key as u32, po as u32, &buf[s..s+e])?;\n s += e;\n po = 0;\n key += self.opt.page_sz as i64;\n }\n\n Ok(())\n }\n\n pub fn close(self) -> Result<(), Error> {\n self.bucket.close()?;\n Ok(())\n }\n\n pub fn sync(&mut self) -> Result<(), Error> {\n self.bucket.sync()?;\n Ok(())\n }\n\n pub fn filesize(&self) -> Result {\n Ok(self.bucket.logical_size())\n }\n\n pub fn truncate(&mut self, new_sz: u64) -> Result<(), Error> {\n log::debug!(\"kv truncate {}\", new_sz);\n self.bucket.truncate(new_sz as usize)?;\n Ok(())\n }\n}\n" }, { "path": "crates/mojofs/src/lib.rs", "content": "\npub mod vfs;\npub mod error;\npub mod open_options;\npub mod vfsfile;\nmod native_file;\nmod kvfile;\n\nuse std::ffi::CStr;\nuse std::collections::HashMap;\n\nuse libsqlite3_sys::{\n self,\n sqlite3_file,\n sqlite3_vfs};\n\nuse std::ffi::c_void;\nuse std::os::raw::{c_int, c_char};\npub use error::*;\npub use vfs::{VFS,AccessCheck};\npub use vfsfile::VFSFile;\npub use open_options::*;\n\n\n#[repr(C)]\npub struct MojoFile {\n base: sqlite3_file,\n custom_file: *mut c_void,\n vfs: *mut sqlite3_vfs,\n}\n\n#[no_mangle]\npub extern \"C\" fn mojo_create() -> *mut sqlite3_vfs {\n let mut name_buf = Vec::from(\"mojo\".as_bytes());\n name_buf.push(0);\n\n let fs_name_c = name_buf.as_ptr();\n std::mem::forget(name_buf);\n\n let fs = Box::new(vfs::VFS::default());\n let p_app_data = Box::into_raw(fs) as *mut c_void;\n\n //println!(\"p_app_data={:?}\", p_app_data);\n\n //let sql_vfs = unsafe {\n // let raw_ptr = sqlite3_malloc64(std::mem::size_of::() as u64);\n // raw_ptr as *mut sqlite3_vfs\n //};\n let vfs = Box::into_raw(Box::new(sqlite3_vfs{\n iVersion: 1,\n szOsFile: (std::mem::size_of::()) as i32,\n mxPathname: 512,\n pNext: std::ptr::null_mut(),\n zName: fs_name_c as *const i8,\n pAppData: p_app_data,\n xOpen: Some(mojo_open),\n xDelete: Some(mojo_delete),\n xAccess: Some(mojo_access),\n xFullPathname: Some(mojo_fullname),\n xDlOpen: Some(mojo_dlopen),\n xDlError: Some(mojo_dlerror),\n xDlSym: Some(mojo_dlsym),\n xDlClose: Some(mojo_dlclose),\n xRandomness: Some(mojo_randomness),\n xSleep: Some(mojo_sleep),\n xCurrentTime: Some(mojo_current_time),\n xCurrentTimeInt64: Some(mojo_current_time64),\n xGetLastError: Some(mojo_getlasterr),\n xSetSystemCall: None,\n xGetSystemCall: None,\n xNextSystemCall: None,\n }));\n\n vfs\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_open(vfs: *mut sqlite3_vfs, zname: *const c_char, file: *mut sqlite3_file, flags: c_int, out_flags: *mut c_int) -> c_int {\n let fs = getfs(vfs);\n\n let opt = open_options::OpenOptions::from_flags(flags as i32).unwrap();\n let mut out_opt = opt.clone();\n \n let file_str = if zname.is_null() {\n \"\"\n }else{\n let file_rs = unsafe{std::ffi::CStr::from_ptr(zname)};\n match file_rs.to_str() {\n Ok(file_str) => file_str,\n Err(_err) => {\n println!(\"mojo_open error in filepath\");\n return libsqlite3_sys::SQLITE_CANTOPEN\n },\n }\n };\n\n if opt.kind.is_main() {\n let query_map = match extract_query_params(zname) {\n Ok(map) => map,\n Err(_err) => {\n return libsqlite3_sys::SQLITE_CANTOPEN\n }\n };\n\n if let Err(err) = fs.init(file_str, &query_map, opt.clone()) {\n log::error!(\"mojo_open init path={} err = {:?}\", file_str, err);\n return libsqlite3_sys::SQLITE_CANTOPEN\n }\n }\n\n\n match fs.open(file_str, opt, &mut out_opt) {\n Ok(vfs_file) => {\n let mojo_file = unsafe {(file as *mut MojoFile).as_mut().unwrap()};\n let io_methods = Box::into_raw(Box::new(libsqlite3_sys::sqlite3_io_methods{\n iVersion: 1,\n xClose: Some(mojo_close),\n xRead: Some(mojo_read),\n xWrite: Some(mojo_write),\n xTruncate: Some(mojo_truncate),\n xSync: Some(mojo_sync),\n xFileSize: Some(mojo_filesize),\n xLock: Some(mojo_lock),\n xUnlock: Some(mojo_unlock),\n xCheckReservedLock: Some(mojo_check_reserved_lock),\n xFileControl: Some(mojo_file_control),\n xSectorSize: Some(mojo_sector_size),\n xDeviceCharacteristics: Some(mojo_device_char),\n xShmMap: None,\n xShmLock: None,\n xShmBarrier: None,\n xShmUnmap: None,\n xFetch: None,\n xUnfetch: None,\n }));\n mojo_file.base.pMethods = io_methods as *const libsqlite3_sys::sqlite3_io_methods;\n mojo_file.custom_file = Box::into_raw(vfs_file) as *mut c_void;\n mojo_file.vfs = vfs;\n },\n Err(err) => {\n log::error!(\"mojo_open path={} err = {:?}\", file_str, err);\n return libsqlite3_sys::SQLITE_CANTOPEN;\n }\n }\n\n unsafe {\n if !out_flags.is_null() {\n *out_flags = out_opt.flags;\n }\n };\n\n libsqlite3_sys::SQLITE_OK\n}\n\n\n#[no_mangle]\nextern \"C\" fn mojo_close(sfile: *mut sqlite3_file) -> c_int {\n unsafe {\n let mojo_file = (sfile as *mut MojoFile).as_ref().unwrap();\n let vfs_file = Box::from_raw(mojo_file.custom_file as *mut VFSFile);\n let fs = getfs(mojo_file.vfs);\n match fs.close(*vfs_file) {\n Ok(_) => {},\n Err(_err) => {\n return libsqlite3_sys::SQLITE_IOERR_CLOSE\n }\n }\n }\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_read(sfile: *mut sqlite3_file, ptr: *mut c_void, n: i32, off: i64) -> c_int {\n let file = get_file(sfile);\n let buf = unsafe{ std::slice::from_raw_parts_mut(ptr as *mut u8, n as usize)};\n\n let rc = match file.pread(off as u64, buf) {\n Ok(n) => {\n //let m = n.min(20);\n //log::debug!(\"after read n={} {:?}\", n, &buf[..m]);\n\n if n == buf.len() {\n libsqlite3_sys::SQLITE_OK\n }else{\n //println!(\"short read\");\n libsqlite3_sys::SQLITE_IOERR_SHORT_READ\n }\n }\n Err(err) => {\n log::error!(\"mojo_read id={} off={} blen={} err={:?}\", file.id(), off, buf.len(), err);\n libsqlite3_sys::SQLITE_IOERR_READ\n },\n };\n\n rc\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_write(sfile: *mut sqlite3_file, ptr: *const c_void, n: i32, off: i64) -> c_int {\n let file = get_file_mut(sfile);\n let buf = unsafe{ std::slice::from_raw_parts(ptr as *const u8, n as usize)};\n\n let rc = match file.pwrite(off as u64, buf) {\n Ok(_) => libsqlite3_sys::SQLITE_OK,\n Err(err) => {\n log::error!(\"mojo_write id={} off={} blen={} err={:?}\", file.id(), off, buf.len(), err);\n libsqlite3_sys::SQLITE_IOERR_WRITE\n },\n };\n\n rc\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_truncate(sfile: *mut sqlite3_file, new_sz: i64) -> c_int {\n let file = get_file_mut(sfile);\n\n let rc = match file.truncate(new_sz as u64) {\n Ok(_) => libsqlite3_sys::SQLITE_OK,\n Err(err) => {\n log::error!(\"mojo_truncate id={} new_sz={} err={:?}\", file.id(), new_sz, err);\n libsqlite3_sys::SQLITE_IOERR_WRITE\n },\n };\n\n rc\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_sync(sfile: *mut sqlite3_file, flags: i32) -> c_int {\n let file = get_file_mut(sfile);\n\n let rc = match file.sync(flags) {\n Ok(_) => libsqlite3_sys::SQLITE_OK,\n Err(err) => {\n log::error!(\"mojo_sync id={} err={:?}\", file.id(), err);\n libsqlite3_sys::SQLITE_IOERR_WRITE\n },\n };\n\n rc\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_filesize(sfile: *mut sqlite3_file, out_sz: *mut i64) -> c_int {\n let file = get_file(sfile);\n\n let rc = match file.filesize() {\n Ok(sz) => {\n unsafe{*out_sz = sz as i64;}\n libsqlite3_sys::SQLITE_OK \n }\n Err(_) => libsqlite3_sys::SQLITE_IOERR_WRITE,\n };\n\n rc\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_access(vfs: *mut sqlite3_vfs, zname: *const c_char, flags: c_int, resout: *mut c_int) -> c_int {\n let path = match c_to_path(zname) {\n Ok(path) => path,\n Err(_) => {\n return libsqlite3_sys::SQLITE_IOERR_CONVPATH;\n }\n };\n\n let access_req = if flags == libsqlite3_sys::SQLITE_ACCESS_EXISTS {\n AccessCheck::Exists\n }else{\n AccessCheck::ReadWrite\n };\n\n let fs = getfs(vfs);\n match fs.access(&path, access_req) {\n Ok(status) => {\n unsafe{*resout = if status {1}else{0}}\n },\n Err(_) => {\n return libsqlite3_sys::SQLITE_IOERR_ACCESS;\n }\n }\n\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_delete(vfs: *mut sqlite3_vfs, zname: *const c_char, _syncdir: c_int) -> c_int {\n let path = match c_to_path(zname) {\n Ok(path) => path,\n Err(_) => {\n return libsqlite3_sys::SQLITE_IOERR_DELETE;\n }\n };\n\n let fs = getfs(vfs);\n match fs.delete(&path) {\n Ok(_) => {\n libsqlite3_sys::SQLITE_OK\n }\n Err(err) => {\n log::error!(\"mojo_delete path={:?} err={:?}\", path, err);\n libsqlite3_sys::SQLITE_IOERR_DELETE\n }\n }\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_fullname(vfs: *mut sqlite3_vfs, zname: *const c_char, _nout: c_int, _zout: *mut c_char) -> c_int {\n let fs = getfs(vfs);\n\n let file_rs = unsafe{std::ffi::CStr::from_ptr(zname)};\n let file_str = match file_rs.to_str() {\n Ok(file_str) => file_str,\n Err(_err) => {\n log::error!(\"mojo_fullname error in filepath\");\n return libsqlite3_sys::SQLITE_CANTOPEN;\n }\n };\n\n let _ = fs.fullpath(file_str);\n\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_dlopen(_arg1: *mut sqlite3_vfs, _zfilename: *const c_char) -> *mut c_void {\n std::ptr::null_mut()\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_dlerror(_arg1: *mut sqlite3_vfs, _nbyte: c_int, _zerrmsg: *mut c_char) {\n}\n\n//extern \"C\" fn (_arg1: *mut sqlite3_vfs, _arg2: *mut c_void, _zSymbol: *const c_char)\n#[no_mangle]\nextern \"C\" fn mojo_dlsym(_arg1: *mut sqlite3_vfs, _arg2: *mut c_void, _zsymbol: *const c_char) \n-> Option {\n None\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_dlclose(_arg1: *mut sqlite3_vfs, _arg2: *mut c_void) {\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_randomness(_arg1: *mut sqlite3_vfs, _nbyte: c_int, _zout: *mut c_char) -> c_int {\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_sleep(_arg1: *mut sqlite3_vfs, microseconds: c_int) -> c_int {\n std::thread::sleep(std::time::Duration::from_micros(microseconds as u64));\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_current_time(_arg1: *mut sqlite3_vfs, p: *mut f64) -> c_int {\n let now = (std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap()).as_secs() as f64;\n unsafe {\n *p = (2440587.5 + now / 864.0e5) * 864.0e5;\n }\n\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_current_time64(_arg1: *mut sqlite3_vfs, p: *mut i64) -> c_int {\n let now = (std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).unwrap()).as_secs() as f64;\n unsafe {\n *p = ((2440587.5 + now / 864.0e5) * 864.0e5) as i64;\n }\n\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_getlasterr(_arg1: *mut sqlite3_vfs, _arg2: c_int, _arg3: *mut c_char) -> c_int {\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_lock(_sfile: *mut sqlite3_file, _flags: c_int) -> c_int {\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_unlock(_sfile: *mut sqlite3_file, _flags: c_int) -> c_int {\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_check_reserved_lock(_sfile: *mut sqlite3_file, res_out: *mut c_int) -> c_int {\n unsafe{*res_out = 0;}\n libsqlite3_sys::SQLITE_OK\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_file_control(_sfile: *mut sqlite3_file, _op: c_int, _arg: *mut c_void) -> c_int {\n libsqlite3_sys::SQLITE_NOTFOUND\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_sector_size(_sfile: *mut sqlite3_file) -> c_int {\n 0\n}\n\n#[no_mangle]\nextern \"C\" fn mojo_device_char(_sfile: *mut sqlite3_file) -> c_int {\n libsqlite3_sys::SQLITE_OK\n}\n\nfn getfs(vfs: *mut sqlite3_vfs) -> &'static mut VFS {\n let fs = unsafe{\n ((*vfs).pAppData as *mut VFS).as_mut().unwrap()\n };\n\n fs\n}\n\nfn get_file_mut(sfile: *mut sqlite3_file) -> &'static mut VFSFile {\n let file = unsafe{\n let mojo_file = (sfile as *mut MojoFile).as_mut().unwrap();\n (mojo_file.custom_file as *mut VFSFile).as_mut().unwrap()\n };\n\n file\n}\n\nfn get_file(sfile: *mut sqlite3_file) -> &'static VFSFile {\n let file = unsafe{\n let mojo_file = (sfile as *mut MojoFile).as_ref().unwrap();\n (mojo_file.custom_file as *mut VFSFile).as_ref().unwrap()\n };\n\n file\n}\n\nfn c_to_path(cpath: *const c_char) -> Result {\n let file_rs = unsafe{std::ffi::CStr::from_ptr(cpath)};\n let file_str = file_rs.to_str()?;\n\n Ok(std::path::PathBuf::from(file_str))\n}\n\n#[no_mangle]\npub extern \"C\" fn mojofs_init_log() {\n env_logger::init();\n}\n\n\nfn extract_query_params(filepath: *const c_char) -> Result, Error> {\n let mut map = HashMap::new();\n if filepath.is_null() {\n return Ok(map);\n }\n\n let mut itr: *const c_char = filepath;\n let mut parse_key = true;\n let mut key: &str = \"dummy\";\n let mut value: &str;\n\n unsafe {\n while key.len() > 0 {\n while *itr != 0 {\n itr = itr.add(1);\n }\n itr = itr.add(1);\n\n let s = CStr::from_ptr(itr).to_str()?;\n if parse_key {\n key = s;\n parse_key = false;\n }else{\n value = s;\n parse_key = true;\n map.insert(key.to_owned(), value.to_owned());\n }\n } \n }\n\n Ok(map)\n}" }, { "path": "crates/mojofs/src/native_file.rs", "content": "\nuse std::path::{Path, PathBuf};\nuse nix::fcntl::{self, OFlag};\nuse nix::sys::stat::Mode;\n\nuse crate::open_options::*;\nuse crate::Error;\n\npub struct NativeFile {\n path: PathBuf,\n fd: i32,\n opt: OpenOptions,\n}\n\nimpl NativeFile {\n pub fn open(path: &Path, opt: &OpenOptions) -> Result {\n let open_flags = match opt.access {\n OpenAccess::Read => {\n OFlag::O_RDONLY \n },\n OpenAccess::Write => {\n OFlag::O_RDWR \n },\n OpenAccess::Create => {\n OFlag::O_CREAT|OFlag::O_RDWR\n },\n OpenAccess::CreateNew => {\n OFlag::O_CREAT|OFlag::O_RDWR|OFlag::O_EXCL\n }\n };\n\n let file_perm = Mode::all();\n let fd = fcntl::open(path, open_flags, file_perm)?;\n\n Ok(NativeFile{\n path: path.to_owned(),\n opt: opt.clone(),\n fd,\n })\n }\n\n pub fn pread(&self, buf: &mut [u8], off: i64) -> Result {\n log::debug!(\"native pread fd={} o={}, blen={}\", self.fd, off, buf.len());\n let mut i = 0;\n while i Result<(), Error> {\n log::debug!(\"native pwrite fd={} o={}, blen={}\", self.fd, off, buf.len());\n\n let mut i=0;\n while i Result<(), Error> {\n nix::unistd::close(self.fd)?;\n if self.opt.delete_on_close {\n nix::unistd::unlink(&self.path)?;\n }\n\n Ok(())\n }\n\n pub fn sync(&mut self) -> Result<(), Error> {\n nix::unistd::fsync(self.fd)?;\n Ok(())\n }\n\n pub fn filesize(&self) -> Result {\n let st = nix::sys::stat::fstat(self.fd)?;\n Ok(st.st_size as u64)\n }\n\n pub fn truncate(&mut self, new_sz: u64) -> Result<(), Error> {\n log::debug!(\"truncate id={} {}\", self.fd, new_sz);\n nix::unistd::ftruncate(self.fd, new_sz as i64)?;\n Ok(())\n }\n}" }, { "path": "crates/mojofs/src/open_options.rs", "content": "use libsqlite3_sys as ffi;\n\n#[derive(Debug, Clone, PartialEq)]\npub struct OpenOptions {\n pub flags: i32,\n\n /// The object type that is being opened.\n pub kind: OpenKind,\n\n /// The access an object is opened with.\n pub access: OpenAccess,\n\n /// The file should be deleted when it is closed.\n pub delete_on_close: bool,\n}\n\n/// The object type that is being opened.\n#[derive(Debug, Clone, Copy, PartialEq)]\npub enum OpenKind {\n MainDb,\n MainJournal,\n TempDb,\n TempJournal,\n TransientDb,\n SubJournal,\n SuperJournal,\n Wal,\n}\n\nimpl OpenKind {\n pub fn is_main(&self) -> bool {\n *self == OpenKind::MainDb\n }\n}\n\n/// The access an object is opened with.\n#[derive(Debug, Clone, Copy, PartialEq)]\npub enum OpenAccess {\n /// Read access.\n Read,\n\n /// Write access (includes read access).\n Write,\n\n /// Create the file if it does not exist (includes write and read access).\n Create,\n\n /// Create the file, but throw if it it already exist (includes write and read access).\n CreateNew,\n}\n\n\nimpl OpenOptions {\n pub fn from_flags(flags: i32) -> Option {\n Some(OpenOptions {\n flags,\n kind: OpenKind::from_flags(flags)?,\n access: OpenAccess::from_flags(flags)?,\n delete_on_close: flags & ffi::SQLITE_OPEN_DELETEONCLOSE > 0,\n })\n }\n}\n\nimpl OpenKind {\n fn from_flags(flags: i32) -> Option {\n match flags {\n flags if flags & ffi::SQLITE_OPEN_MAIN_DB > 0 => Some(Self::MainDb),\n flags if flags & ffi::SQLITE_OPEN_MAIN_JOURNAL > 0 => Some(Self::MainJournal),\n flags if flags & ffi::SQLITE_OPEN_TEMP_DB > 0 => Some(Self::TempDb),\n flags if flags & ffi::SQLITE_OPEN_TEMP_JOURNAL > 0 => Some(Self::TempJournal),\n flags if flags & ffi::SQLITE_OPEN_TRANSIENT_DB > 0 => Some(Self::TransientDb),\n flags if flags & ffi::SQLITE_OPEN_SUBJOURNAL > 0 => Some(Self::SubJournal),\n flags if flags & ffi::SQLITE_OPEN_SUPER_JOURNAL > 0 => Some(Self::SuperJournal),\n flags if flags & ffi::SQLITE_OPEN_WAL > 0 => Some(Self::Wal),\n _ => None,\n }\n }\n}\n\nimpl OpenAccess {\n fn from_flags(flags: i32) -> Option {\n match flags {\n flags\n if (flags & ffi::SQLITE_OPEN_CREATE > 0)\n && (flags & ffi::SQLITE_OPEN_EXCLUSIVE > 0) =>\n {\n Some(Self::CreateNew)\n }\n flags if flags & ffi::SQLITE_OPEN_CREATE > 0 => Some(Self::Create),\n flags if flags & ffi::SQLITE_OPEN_READWRITE > 0 => Some(Self::Write),\n flags if flags & ffi::SQLITE_OPEN_READONLY > 0 => Some(Self::Read),\n _ => None,\n }\n }\n}" }, { "path": "crates/mojofs/src/vfs.rs", "content": "\nuse std::path::{PathBuf, Path};\nuse crate::{error, Error};\nuse crate::open_options::*;\nuse std::collections::HashMap;\nuse crate::kvfile::{KVFile, KVFileOpt};\nuse crate::vfsfile::FileImpl;\nuse mojokv::{Store, BucketOpenMode};\n\nuse crate::vfsfile::VFSFile;\n\n#[derive(Debug)]\npub enum AccessCheck {\n Exists,\n ReadWrite,\n}\n\n#[derive(Default)]\npub struct VFS {\n store: Option,\n file_counter: usize,\n fopt: FSOptions,\n}\n\nimpl VFS {\n pub fn name(&self) -> String {\n return \"mojo\".to_owned()\n }\n\n pub fn fs_options(&self) -> FSOptions {\n self.fopt.clone()\n }\n\n pub fn active_ver(&self) -> u32 {\n self.store.as_ref().unwrap().active_ver()\n }\n\n pub fn init(&mut self, root_path: &str, params: &HashMap, opt: OpenOptions) -> Result<(), Error> {\n log::debug!(\"init: root_path={} params={:?} opt={:?}\", root_path, params, opt);\n\n self.fopt = FSOptions::parse(params)?;\n let root_path = Path::new(root_path);\n if opt.access == OpenAccess::Read {\n self.store = Some(Store::readonly(root_path, self.fopt.ver)?);\n log::debug!(\"store opened in readonly mode\");\n }else{\n self.store = Some(Store::writable(root_path, true, Some(self.fopt.pagesz), Some(self.fopt.pps))?);\n log::debug!(\"store opened writable mode\");\n }\n\n Ok(())\n }\n\n pub fn open(&mut self, filepath: &str, opt: OpenOptions, _out_opt: &mut OpenOptions) -> Result, Error> {\n log::debug!(\"open: file={} opt={:?}\", filepath, opt);\n\n self.file_counter += 1;\n let id = self.file_counter;\n let file_path = if filepath.len() == 0{\n std::path::PathBuf::from(format!(\"mojo.tmp.{}\", id))\n }else{\n std::path::PathBuf::from(filepath)\n };\n\n let store = self.store.as_mut().unwrap();\n\n let bucket_name = Self::bucket_name(&file_path);\n let bmode = if let OpenAccess::Read = opt.access {\n BucketOpenMode::Read\n }else{\n BucketOpenMode::Write\n };\n\n let b = store.open(bucket_name, bmode)?;\n let kvfileopt = self.fopt.to_kvfile_opt();\n\n let f = KVFile::open(b, kvfileopt)?;\n let fimpl = FileImpl::KV(f);\n\n log::debug!(\"open: file={} id={} done\", filepath, id);\n Ok(Box::new(VFSFile::new(id, bucket_name, opt, fimpl)))\n }\n\n pub fn fullpath(&mut self, filepath: &str) -> Result {\n log::debug!(\"fullpath filepath={}\", filepath);\n\n let filepath_rs = std::path::Path::new(filepath);\n if filepath_rs.is_absolute() {\n Ok(filepath_rs.to_owned())\n }else{\n let mut cwd = std::env::current_dir()?;\n cwd.push(filepath_rs);\n Ok(cwd)\n }\n }\n\n fn bucket_name(path: &Path) -> &str {\n path.file_name().unwrap().to_str().unwrap()\n }\n\n //TODO: add sync dir\n pub fn delete(&mut self, path: &std::path::Path) -> Result<(), Error> {\n log::debug!(\"delete path={:?}\", path);\n\n let name = Self::bucket_name(path);\n let store = self.store.as_mut().unwrap();\n store.delete(name)?;\n\n Ok(())\n }\n\n pub fn access(&self, path: &std::path::Path, req: AccessCheck) -> Result {\n log::debug!(\"access path={:?} req={:?}\", path, req);\n\n let name = Self::bucket_name(path);\n let store = self.store.as_ref().unwrap();\n let status = store.exists(name);\n\n log::debug!(\"access path={:?} status={}\", path, status);\n Ok(status)\n }\n\n pub fn close(&mut self, f: VFSFile) -> Result<(), Error> {\n let fid = f.id();\n log::debug!(\"close id={}\", f.id());\n\n let bucket_name = f.bucket.clone();\n let opt = f.opt();\n\n f.close()?;\n\n let store = self.store.as_mut().unwrap();\n if opt.delete_on_close {\n log::debug!(\"close_on_delete is set for id={}\", fid);\n store.delete(bucket_name.as_str())?;\n }\n\n Ok(())\n }\n\n pub fn commit(&mut self) -> Result<(), Error> {\n let store = self.store.as_mut().unwrap();\n store.commit()?;\n Ok(())\n }\n\n}\n\n\n#[derive(Default, Clone)]\npub struct FSOptions {\n pub ver: u32,\n pub pagesz: u32,\n pub pps: u32,\n}\n\nimpl FSOptions {\n fn parse(map: &HashMap) -> Result {\n let mut opt = FSOptions {ver: 0, pagesz: 4096, pps: 0};\n\n opt.ver = match map.get(\"ver\") {\n Some(s) => s.parse()?,\n None => 1\n };\n\n let s = map.get(\"pagesz\").ok_or(Error::new(error::MOJOFS_ERR_ARG_PAGESZ_MISSING, \n \"fs arg page size missing\".to_owned()))?;\n opt.pagesz = s.parse()?;\n\n opt.pps = match map.get(\"pps\") {\n Some(s) => s.parse()?,\n None => 65536\n };\n\n Ok(opt)\n }\n\n fn to_kvfile_opt(&self) -> KVFileOpt {\n KVFileOpt {\n ver: self.ver,\n page_sz: self.pagesz,\n pps: self.pps,\n }\n }\n}" }, { "path": "crates/mojofs/src/vfsfile.rs", "content": "use crate::error::Error;\n\nuse crate::native_file::NativeFile;\nuse crate::kvfile::KVFile;\nuse crate::open_options::OpenOptions;\n\npub enum FileImpl {\n Reg(NativeFile),\n KV(KVFile)\n}\n\npub struct VFSFile {\n pub bucket: String,\n id: usize,\n fimpl: FileImpl,\n opt: OpenOptions,\n}\n\n\nimpl VFSFile {\n pub fn new(id: usize, name: &str, opt: OpenOptions, fimpl: FileImpl) -> Self {\n VFSFile{\n bucket: name.to_owned(),\n id,\n fimpl,\n opt,\n }\n }\n\n pub fn id(&self) -> usize {\n self.id\n }\n\n pub fn opt(&self) -> OpenOptions {\n self.opt.clone()\n }\n\n pub fn pread(&self, off: u64, buf: &mut [u8]) -> Result {\n log::debug!(\"pread id={} o={}, blen={}\", self.id, off, buf.len());\n\n let nread = match &self.fimpl {\n FileImpl::Reg(f) => {\n f.pread(buf, off as i64)?\n },\n FileImpl::KV(f) => {\n f.pread(buf, off as i64)?\n }\n };\n\n Ok(nread)\n }\n\n\n pub fn pwrite(&mut self, off: u64, buf: &[u8]) -> Result<(), Error> {\n log::debug!(\"pwrite id={} o={}, blen={}\", self.id, off, buf.len());\n\n match &mut self.fimpl {\n FileImpl::Reg(f) => {\n f.pwrite(off as i64, buf)?;\n },\n FileImpl::KV(f) => {\n f.pwrite(off as i64, buf)?;\n }\n };\n\n Ok(())\n }\n\n pub fn close(self) -> Result<(), Error> {\n log::debug!(\"file close id={}\", self.id);\n\n match self.fimpl {\n FileImpl::Reg(f) => {\n f.close()?;\n },\n FileImpl::KV(f) => {\n f.close()?;\n }\n };\n \n Ok(())\n }\n\n pub fn sync(&mut self, flags: i32) -> Result<(), Error> {\n log::debug!(\"sync id={} flags={}\", self.id, flags);\n\n match &mut self.fimpl {\n FileImpl::Reg(f) => {\n f.sync()?;\n },\n FileImpl::KV(f) => {\n f.sync()?;\n }\n };\n\n Ok(())\n }\n\n pub fn filesize(&self) -> Result {\n let sz = match &self.fimpl {\n FileImpl::Reg(f) => {\n f.filesize()?\n },\n FileImpl::KV(f) => {\n f.filesize()?\n }\n };\n\n Ok(sz)\n }\n\n pub fn truncate(&mut self, new_sz: u64) -> Result<(), Error> {\n log::debug!(\"truncate id={} {}\", self.id, new_sz);\n\n match &mut self.fimpl {\n FileImpl::Reg(f) => {\n f.truncate(new_sz)?;\n },\n FileImpl::KV(f) => {\n f.truncate(new_sz)?;\n }\n };\n\n Ok(())\n }\n\n pub fn lock(&mut self, _flag: i32) -> Result<(), Error> {\n Ok(())\n }\n\n pub fn unlock(&mut self, _flag: i32) -> Result<(), Error> {\n Ok(())\n }\n\n pub fn check_reserved_lock(&self) -> Result {\n Ok(0)\n }\n\n pub fn file_control(&mut self, _op: i32) -> Result<(), Error> {\n Ok(())\n }\n\n pub fn sector_size(&self) -> Result {\n Ok(0)\n }\n\n pub fn device_char(&self) -> Result<(), Error> {\n Ok(())\n }\n}\n" }, { "path": "crates/mojofs/tests/mojofs_test.rs", "content": "use std::path::Path;\nuse anyhow::Error;\nuse std::collections::HashMap;\nuse mojofs::{self, VFS, VFSFile};\n\nfn remove_fs(rootpath: &Path) -> Result<(), Error> {\n if let Err(err) = std::fs::remove_dir_all(rootpath) {\n if err.kind() != std::io::ErrorKind::NotFound {\n return Err(err.into());\n }\n }\n Ok(())\n}\n\nfn setup() -> Result {\n let path = Path::new(\"./testfs\");\n remove_fs(path)?;\n Ok(path.to_owned().to_str().unwrap().to_owned())\n}\n\nfn default_params(pagesz: u32) -> HashMap {\n let mut h = HashMap::new();\n\n h.insert(\"ver\".to_owned(), \"1\".to_owned());\n h.insert(\"pagesz\".to_owned(), format!(\"{}\", pagesz));\n h.insert(\"pps\".to_owned(), \"65536\".to_owned());\n\n h\n}\n\nfn read_test(a: &mut VFSFile, nitems: usize, pagesz: u64, f: fn(usize) -> usize) -> Result<(), Error> {\n let mut buf = [0u8; 8];\n for i in 0usize..nitems {\n let off = i as u64 * pagesz;\n let n = f(i);\n a.pread(off, &mut buf)?;\n assert_eq!(n, usize::from_be_bytes(buf));\n }\n Ok(())\n}\n\nfn write_test(a: &mut VFSFile, nitems: usize, pagesz: u64, f: fn(usize) -> usize) -> Result<(), Error> {\n for i in 0usize..nitems {\n let off = i as u64 * pagesz;\n let n = f(i);\n a.pwrite(off, &n.to_be_bytes())?;\n }\n\n a.sync(1)?;\n Ok(())\n}\n\nfn write_read(a: &mut VFSFile, nitems: usize, pagesz: u64, f: fn(usize) -> usize) -> Result<(), Error> {\n write_test(a, nitems, pagesz, f)?;\n read_test(a, nitems, pagesz, f)?;\n\n Ok(())\n}\n\n#[test]\nfn rw_same_version() -> Result<(), Error> {\n env_logger::init();\n let fspath = setup()?; \n let mut fs_uri_opt = default_params(8);\n let mut opt = mojofs::OpenOptions::from_flags(326).unwrap();\n let nitems = 10;\n\n {\n let mut fs = VFS::default();\n fs.init(&fspath, &fs_uri_opt, opt.clone())?;\n let fsopt = fs.fs_options();\n\n let mut a = fs.open(\"a\", opt.clone(), &mut opt)?;\n\n assert_eq!(fsopt.pagesz, 8);\n\n write_read(&mut a, nitems, fsopt.pagesz as u64, |n| n)?;\n assert_eq!(a.filesize()?, (fsopt.pagesz as u64) * nitems as u64);\n a.close()?;\n fs.commit()?;\n assert_eq!(fs.active_ver(), 2);\n\n let mut a = fs.open(\"a\", opt.clone(), &mut opt)?;\n write_read(&mut a, nitems, fsopt.pagesz as u64, |n| n+10)?;\n assert_eq!(a.filesize()?, (fsopt.pagesz as u64) * nitems as u64);\n a.close()?;\n fs.commit()?;\n assert_eq!(fs.active_ver(), 3);\n }\n\n {\n let mut fs = VFS::default();\n opt.access = mojofs::OpenAccess::Read;\n fs_uri_opt.insert(\"ver\".to_owned(), \"1\".to_owned());\n fs.init(&fspath, &fs_uri_opt, opt.clone())?;\n let fsopt = fs.fs_options();\n let mut a = fs.open(\"a\", opt.clone(), &mut opt)?;\n\n read_test(&mut a, nitems, fsopt.pagesz as u64, |n| n)?;\n }\n \n {\n let mut fs = VFS::default();\n opt.access = mojofs::OpenAccess::Read;\n fs_uri_opt.insert(\"ver\".to_owned(), \"2\".to_owned());\n fs.init(&fspath, &fs_uri_opt, opt.clone())?;\n let fsopt = fs.fs_options();\n let mut a = fs.open(\"a\", opt.clone(), &mut opt)?;\n\n read_test(&mut a, nitems, fsopt.pagesz as u64, |n| n+10)?;\n }\n\n Ok(())\n}" }, { "path": "crates/mojoio/Cargo.toml", "content": "[package]\nname = \"mojoio\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nnix = \"0.24\"\nlog = \"0.4.17\"\nthiserror = \"1.0.31\"\nparking_lot = {version = \"0.12\", features=[\"serde\"]}" }, { "path": "crates/mojoio/src/error.rs", "content": "\n#[derive(thiserror::Error, Debug)]\npub enum Error {\n #[error(\"io error\")]\n IoErr(#[from] std::io::Error),\n\n #[error(\"Unix error\")]\n NixErr(#[from] nix::Error),\n\n #[error(\"Parse int error\")]\n ParseIntErr(#[from] std::num::ParseIntError),\n\n #[error(\"Unknown error `{0}`\")]\n UnknownStr(String),\n\n #[error(\"UTF8 error\")]\n UTF8Err(#[from] std::str::Utf8Error),\n\n #[error(\"Unknown error\")]\n Unknown,\n}" }, { "path": "crates/mojoio/src/lib.rs", "content": "pub mod nix;\nmod error;\n\npub use error::Error;\n\npub const BUFFER_MAGIC: &[u8] = b\"mojo\";\npub const PAGE_HEADER_LEN: usize = 8;\n\n\npub fn add(left: usize, right: usize) -> usize {\n left + right\n}\n\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn it_works() {\n let result = add(2, 2);\n assert_eq!(result, 4);\n }\n}\n" }, { "path": "crates/mojoio/src/nix.rs", "content": "use std::path::Path;\n\nuse nix::fcntl::{self, OFlag};\nuse nix::sys::stat::Mode;\n\n//use crate::value;\nuse crate::Error;\n\npub struct NixFile {\n file_fd: i32,\n curr_off: u64,\n page_header_buf: [u8; crate::PAGE_HEADER_LEN],\n page_header: PageHeader,\n}\n\nimpl NixFile {\n pub fn open(filepath: &Path, _file_no: u32) -> Result {\n let open_flags = OFlag::O_CREAT|OFlag::O_RDWR;\n let file_perm = Mode::all();\n let file_fd = fcntl::open(filepath, open_flags, file_perm)?;\n\n let curr_off = nix::unistd::lseek(file_fd, 0, nix::unistd::Whence::SeekEnd)? as u64;\n\n log::debug!(\"open path={:?} fd={}\", filepath, file_fd);\n\n Ok(NixFile {\n file_fd,\n curr_off,\n page_header_buf: [0; crate::PAGE_HEADER_LEN],\n page_header: PageHeader::new(), \n })\n }\n\n pub fn close(&mut self) -> Result<(), Error> {\n log::debug!(\"close fd={}\", self.file_fd);\n nix::unistd::close(self.file_fd)?;\n Ok(())\n }\n\n pub fn write_buf_at(&mut self, off: u64, block_no: u32, buf: &[u8]) -> Result<(), Error> {\n self.page_header.block_no = block_no;\n self.page_header.encode(&mut self.page_header_buf);\n\n let header_io = std::io::IoSlice::new(&self.page_header_buf);\n let buf_io = std::io::IoSlice::new(buf);\n\n let io_bufs = [header_io, buf_io];\n\n log::debug!(\"file write at fd={} off={} {}\", self.file_fd, off, buf.len());\n let n = nix::sys::uio::pwritev(self.file_fd, &io_bufs, off as i64)?;\n if n < header_io.len() + buf_io.len() {\n return Err(Error::UnknownStr(\"vectored write did not write all data\".to_owned()));\n }\n\n Ok(())\n }\n\n pub fn write_buf(&mut self, block_no: u32, poff: u64, buf: &[u8]) -> Result {\n self.write_buf_at(self.curr_off, block_no, buf)?;\n\n let page_off = self.curr_off;\n //let page_off = self.curr_off;\n self.curr_off += buf.len() as u64 + NixFile::header_len() as u64;\n self.curr_off += poff;\n //self.curr_off += buf.len() as u64;\n\n Ok(page_off)\n }\n\n pub fn header_len() -> usize {\n return crate::PAGE_HEADER_LEN;\n }\n\n fn read_all_at(&self, off: u64, buf: &mut [u8]) -> Result {\n let mut i = 0;\n while i < buf.len() {\n //TODO: Should n==0 be handled?\n let n = nix::sys::uio::pread(self.file_fd, &mut buf[i..], off as i64 + i as i64)?;\n if n == 0 {\n break;\n }\n i += n;\n }\n Ok(i)\n }\n\n pub fn read_buf_at(&self, off: u64, buf: &mut [u8]) -> Result {\n log::debug!(\"file read at fd={} off={} {}\", self.file_fd, off, buf.len());\n let n = self.read_all_at(off, buf)?;\n Ok(n)\n }\n\n pub fn sync(&self) -> Result<(), Error> {\n log::debug!(\"sync fd={}\", self.file_fd);\n nix::unistd::fsync(self.file_fd)?;\n Ok(())\n }\n}\n\n\nstruct PageHeader {\n magic: &'static [u8],\n block_no: u32,\n}\n\nimpl PageHeader {\n fn new() -> PageHeader {\n PageHeader { \n magic: crate::BUFFER_MAGIC,\n block_no: 0,\n }\n }\n\n pub fn encode(&mut self, buf: &mut [u8; crate::PAGE_HEADER_LEN]) {\n let _ = &buf[..4].copy_from_slice(self.magic);\n let _ = &buf[4..8].copy_from_slice(&self.block_no.to_le_bytes());\n //let _ = &buf[12..13].copy_from_slice(&self.flags.to_le_bytes());\n //let _ = &buf[13..17].copy_from_slice(&self.file_no.to_le_bytes());\n }\n\n /*\n pub fn decode(buf: &[u8]) -> Result {\n let magic = &buf[..4];\n if magic != BUFFER_MAGIC {\n return Err(Error::UnknownStr(\"Invalid buffer magic\".to_string()));\n }\n\n let mut tmp_buf = [0u8; 8];\n tmp_buf.copy_from_slice(&buf[4..12]);\n let block_no = u64::from_be_bytes(tmp_buf);\n\n let mut tmp_buf = [0u8; 2];\n tmp_buf.copy_from_slice(&buf[12..14]);\n let size = u16::from_be_bytes(tmp_buf);\n\n tmp_buf.copy_from_slice(&buf[12..14]);\n let flags = buf[15];\n\n let mut tmp_buf = [0u8; 4];\n tmp_buf.copy_from_slice(&buf[15..19]);\n let file_no = u32::from_be_bytes(tmp_buf);\n\n Ok(PageHeader{\n magic: BUFFER_MAGIC,\n block_no,\n size,\n flags,\n file_no,\n })\n }\n */\n}" }, { "path": "crates/mojokv/.gitignore", "content": "/target\n/Cargo.lock\n*.db\nlog*\n*.log\n" }, { "path": "crates/mojokv/Cargo.toml", "content": "[package]\nname = \"mojokv\"\nversion = \"0.2.0\"\nedition = \"2021\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nmojoio = {path=\"../mojoio\"}\nmodular-bitfield = \"0.11\"\nlz4 = \"1.23.3\"\nzstd = \"0.11.2\"\nthiserror = \"1.0.31\"\nlog = \"0.4.17\"\nenv_logger = \"0.9.0\"\nrand = \"0.8.5\"\nclap = {version=\"3.2.6\", features=[\"derive\"] }\nanyhow = \"1.0.58\"\nparking_lot = {version = \"0.12\", features=[\"serde\"]}\nserde = { version = \"1.0\", features = [\"derive\", \"rc\"] }\nserde_json = \"1.0\"\nrmp-serde = \"1.1.0\"\nfslock = \"0.2.1\"\nrustc-hash = \"1.1.0\"" }, { "path": "crates/mojokv/src/bmap.rs", "content": "\nuse std::path::{Path, PathBuf};\nuse std::collections::HashMap;\nuse std::sync::Arc;\nuse crate::bucket::Bucket;\nuse crate::Error;\nuse parking_lot::RwLock;\nuse serde::{Serialize, Deserialize};\n\n#[derive(Clone, Default, Debug, Serialize, Deserialize)]\npub struct BucketMap {\n map: Arc>>,\n}\n\nimpl BucketMap {\n pub fn add(&self, name: &str, ver: u32) {\n log::debug!(\"add name={} ver={} {:?}\", name, ver, self.map);\n let mut map = self.map.write();\n //self.buckets.insert(name.to_owned(), b);\n map.insert(name.to_owned(), ver);\n }\n\n pub fn exists(&self, name: &str) -> bool {\n let map = self.map.read();\n map.contains_key(name)\n } \n\n pub fn get(&self, name: &str) -> Option{\n log::debug!(\"get name={}\", name);\n let map = self.map.read();\n map.get(name).map(|v| *v)\n }\n\n pub fn delete(&self, root_path: &Path, name: &str, ver: u32) -> Result<(), Error> {\n log::debug!(\"delete name={} {:?}\", name, self.map);\n let mut map = self.map.write();\n\n map.remove(name);\n\n Bucket::delete_ver(root_path, name, ver)?;\n\n Ok(())\n }\n\n pub fn map(&self) -> Result, Error> {\n let map = self.map.read();\n\n Ok(map.clone())\n }\n\n pub fn serialize_to_path(&self, path: &Path) -> Result<(), Error> {\n let buf = serde_json::to_vec(&self)?;\n log::debug!(\"serializing bmap={:?}\", std::str::from_utf8(&buf));\n crate::utils::write_file(path, &buf)?;\n Ok(())\n }\n\n pub fn deserialize_from_path(path: &Path) -> Result {\n let mut buf = Vec::new();\n crate::utils::load_file(path, &mut buf)?;\n\n let map = serde_json::from_slice(&buf)?;\n Ok(map)\n }\n\n fn bmap_path(root_path: &Path, ver: u32) -> PathBuf {\n root_path.join(&format!(\"mojo.bmap.{}\", ver))\n }\n\n pub fn load(root_path: &Path, ver: u32) -> Result {\n let bmap_path = Self::bmap_path(root_path, ver);\n log::debug!(\"loading bmap from path={:?}\", bmap_path);\n let bmap = Self::deserialize_from_path(&bmap_path)?;\n\n Ok(bmap)\n }\n}" }, { "path": "crates/mojokv/src/bucket.rs", "content": "\nuse std::collections::HashSet;\nuse std::path::{Path, PathBuf};\nuse crate::{Error, BucketMap};\nuse mojoio::nix::NixFile;\nuse crate::index::mem::MemIndex;\nuse crate::value::Value;\nuse crate::state::State;\n\npub struct BucketInner {\n name: String,\n root_path: PathBuf,\n index: MemIndex,\n file_page_sz: usize,\n fmap: FileMap,\n is_dirty: bool,\n is_modified: bool,\n is_closed: bool,\n active_ver: u32,\n}\n\nimpl BucketInner {\n fn active_file(&mut self, ver: u32) -> &mut NixFile {\n self.fmap.file_mut(ver)\n }\n\n fn sync_index(&mut self, ver: u32) -> Result<(), Error> {\n let non_ref_vers =self.index.update_min_max_ver();\n\n log::debug!(\"closing versions={:?} as they are no longer referenced\", non_ref_vers);\n self.fmap.close_versions(&non_ref_vers, self.active_ver)?;\n\n let index_path = Bucket::index_path(&self.root_path, self.name.as_str(), ver);\n log::debug!(\"syncing index ver={} {:?}\", ver, index_path);\n self.index.serialize_to_path(&index_path)?;\n log::debug!(\"syncing index ver={} done\", ver);\n Ok(())\n }\n}\n\npub struct Bucket {\n state: State,\n //inner: Arc>,\n inner: BucketInner,\n bmap: BucketMap,\n is_write: bool,\n}\n\nimpl Bucket {\n fn with_inner(state: State, inner: BucketInner, bmap: BucketMap) -> Self {\n Bucket {\n state,\n //inner: Arc::new(RwLock::new(inner)),\n inner,\n bmap,\n is_write: false,\n }\n }\n\n pub fn set_writable(&mut self) {\n self.is_write = true\n }\n\n pub fn readonly(root_path: &Path, name: &str, ver: u32, state: State, bmap: BucketMap) -> Result {\n log::debug!(\"bucket name={} readonly at ver={}\", name, ver);\n\n let b = Self::load(root_path, name, state, bmap, ver)?;\n Ok(b)\n }\n\n fn index_path(rootpath: &Path, name: &str, ver: u32) -> PathBuf {\n rootpath.join(&format!(\"{}_i.{}\", name, ver))\n }\n\n pub fn get_key(&self, key: u32) -> Result, Error> {\n //let inner = self.inner.read();\n Ok(self.inner.index.get(key)?.map(|v| v.clone()))\n }\n\n pub fn max_key(&self) -> isize {\n //let inner = self.inner.read();\n self.inner.index.max_key()\n }\n\n pub fn is_modified(&self) -> bool {\n //let inner = self.inner.read();\n self.inner.is_modified\n }\n\n pub fn writable(root_path: &Path, name: &str, state: State, bmap: BucketMap, load_ver: u32) -> Result {\n log::debug!(\"mojo initing bucket pps={}\", state.pps());\n\n let aver = state.active_ver();\n let index_path = Self::index_path(root_path, name, load_ver);\n\n let mut b = if index_path.exists() {\n log::debug!(\"bucket index for version={} exists\", load_ver);\n Self::load(root_path, name, state, bmap, aver)?\n }else{\n log::debug!(\"creating new bucket at ver={}\", aver);\n let mut b = Self::new(root_path, name, state, bmap)?;\n b.sync()?;\n b\n };\n\n b.set_writable();\n\n log::debug!(\"mojo state={:?}\", b.state);\n\n Ok(b)\n }\n\n pub fn load(root_path: &Path, name: &str, state: State, bmap: BucketMap, ver: u32) -> Result {\n log::debug!(\"loading bucket={} version={}\", name, ver);\n\n if ver < state.min_ver() || ver > state.active_ver() {\n return Err(Error::VersionNotFoundErr(ver));\n }\n\n let (_, _, mut index) = Self::load_index(root_path, name, ver)?;\n let fmap = FileMap::init(root_path, name, &index.header().vset, state.active_ver())?;\n index.set_active_ver(state.active_ver());\n\n let file_page_sz = state.page_size() as usize + NixFile::header_len();\n\n let inner = BucketInner {\n name: name.to_owned(),\n root_path: root_path.to_owned(),\n index,\n file_page_sz,\n fmap,\n is_dirty: false,\n is_modified: false,\n is_closed: false,\n active_ver: state.active_ver(),\n };\n\n log::debug!(\"mojo load version done\");\n Ok(Bucket::with_inner(state, inner, bmap))\n }\n\n pub fn load_index(root_path: &Path, name: &str, ver: u32) -> Result<(usize, usize, MemIndex), Error> {\n let index_path = Self::index_path(root_path, name, ver);\n\n log::debug!(\"loading index={:?} for name={} at ver={}\", index_path, name, ver);\n if !index_path.exists() {\n return Err(Error::BucketNotAtVerErr(name.to_owned(), ver));\n }\n\n let index = MemIndex::deserialize_from_path(&index_path)?;\n\n Ok(index)\n }\n\n pub fn new(root_path: &Path, name: &str, state: State, bmap: BucketMap) -> Result {\n log::debug!(\"creating new bucket name={} at ver={}\", name, state.active_ver());\n\n let _ = std::fs::create_dir_all(root_path)?;\n\n let index = MemIndex::new(state.pps() as usize);\n let fmap = FileMap::init(root_path, name, &index.header().vset, state.active_ver())?;\n\n let mut inner = BucketInner {\n name: name.to_owned(),\n root_path: root_path.to_owned(),\n index,\n file_page_sz: state.page_size() as usize + NixFile::header_len(),\n fmap,\n is_dirty: false,\n is_modified: false,\n is_closed: false,\n active_ver: state.active_ver(),\n };\n\n inner.index.set_active_ver(state.active_ver());\n let b = Bucket::with_inner(state, inner, bmap);\n Ok(b)\n }\n\n pub fn logical_size(&self) -> u64 {\n //let inner = self.inner.read();\n (self.state.page_size() as isize * (self.inner.index.max_key() + 1)) as u64\n }\n\n pub fn close(mut self) -> Result<(), Error> {\n //let mut inner = self.inner.write();\n if self.inner.is_closed {\n return Ok(())\n }\n\n self.inner.fmap.close()?;\n self.inner.is_closed = true;\n Ok(())\n }\n\n pub fn truncate(&mut self, new_sz: usize) -> Result<(), Error> {\n let _ = self.state.commit_lock.read();\n\n //let mut inner = self.inner.write();\n log::debug!(\"truncate bucket={} new_sz={}\", self.inner.name, new_sz);\n let pages = new_sz/(self.state.page_size() as usize);\n //let real_sz = pages * self.file_page_sz;\n\n self.inner.index.truncate(pages as u32)?;\n self.inner.is_modified = true;\n //TODO: Delete blocks from file\n //self.active_file().truncate(real_sz)?;\n\n Ok(())\n }\n\n fn put_at(&mut self, key: u32, page_off: u64, buf: &[u8], val: &Value) -> Result<(), Error> {\n\n let mut off = val.get_off() as u64;\n off = off * self.inner.file_page_sz as u64;\n off += page_off;\n let file = self.inner.active_file(self.state.active_ver());\n file.write_buf_at(off, key, buf)?;\n\n Ok(())\n }\n\n pub fn put(&mut self, key: u32, page_off: u64, buf: &[u8]) -> Result<(), Error> {\n if !self.is_write {\n return Err(Error::BucketNotWritableErr);\n }\n\n if self.inner.active_ver < self.state.active_ver() {\n return Err(Error::VerNotWritable(self.inner.active_ver, self.state.active_ver()));\n }\n\n let _ = self.state.commit_lock.read();\n\n log::debug!(\"store put aver={} key={}, buflen={}\", self.state.active_ver(), key, buf.len());\n\n let val_opt = self.get_value_opt(key)?.map(|v| v.clone());\n\n match val_opt {\n Some(val) => {\n //let mut inner = self.inner.write();\n\n log::debug!(\"store put value exists value={:?}\", val);\n if val.get_ver() == self.state.active_ver() {\n self.put_at(key, page_off, buf, &val)?;\n self.inner.index.put(key, val.get_off())?;\n }else{\n let file = self.inner.active_file(self.state.active_ver());\n let write_off = file.write_buf(key, page_off, buf)?;\n let block_no = (write_off/(self.inner.file_page_sz as u64)) as u32;\n log::debug!(\"bucket put was done at block_no={} old value={:?}\", block_no, val);\n self.inner.index.put(key, block_no)?;\n }\n self.inner.is_dirty = true;\n self.inner.is_modified = true;\n },\n None => {\n //let mut inner = self.inner.write();\n \n let file = self.inner.active_file(self.state.active_ver());\n let write_off = file.write_buf(key, page_off, buf)?;\n let block_no = (write_off/(self.inner.file_page_sz as u64)) as u32;\n\n self.inner.index.put(key, block_no)?;\n self.inner.is_dirty = true;\n self.inner.is_modified = true;\n log::debug!(\"store put value not present. value={:?}\", block_no);\n }\n }\n\n Ok(())\n }\n\n pub fn get(&self, key: u32, page_off: u64, out_buf: &mut [u8]) -> Result {\n //let inner = self.inner.read();\n\n let value = self.get_value(key)?;\n\n let mut read_off = (value.get_off() as u64) * (self.inner.file_page_sz as u64);\n read_off += NixFile::header_len() as u64 + page_off;\n let read_ver = value.get_ver();\n\n log::debug!(\"get name={} key={} value: {:?}\", self.inner.name, key, value);\n let file = self.inner.fmap.file(read_ver);\n let n = file.read_buf_at(read_off, out_buf)?;\n log::debug!(\"get name={} key={} n={}\", self.inner.name, key, n);\n\n Ok(n)\n }\n\n fn get_value_opt(&self, key: u32) -> Result, Error> {\n //let inner = self.inner.read();\n\n match self.inner.index.get(key)? {\n None => {\n log::debug!(\"get_value_opt no slot key={}\", key);\n return Ok(None)\n }\n Some(val) => {\n if !val.is_allocated() {\n log::debug!(\"get_value_opt allocated key={}\", key);\n return Ok(None)\n }else{\n return Ok(Some(val.clone()))\n }\n }\n }\n }\n\n fn get_value(&self, key: u32) -> Result {\n self.get_value_opt(key)?.ok_or(Error::KeyNotFoundErr(key))\n }\n\n pub (crate) fn sync_no_commit_lock(&mut self) -> Result<(), Error> {\n if !self.is_write {\n return Err(Error::StoreNotWritableErr);\n }\n\n //let mut inner = self.inner.write();\n\n log::debug!(\"syncing bucket={} at ver={}\", self.inner.name, self.state.active_ver());\n\n self.bmap.add(&self.inner.name, self.state.active_ver());\n self.inner.active_file(self.state.active_ver()).sync()?;\n self.inner.sync_index(self.state.active_ver())?;\n self.inner.is_dirty = false;\n\n log::debug!(\"syncing done\");\n Ok(())\n }\n\n pub fn sync(&mut self) -> Result<(), Error> {\n let _ = self.state.commit_lock.read();\n\n self.sync_no_commit_lock()\n }\n\n pub fn delete_ver(root_path: &Path, name: &str, ver: u32) -> Result<(), Error> {\n log::debug!(\"Deleting bucket name={} ver={}\", name, ver);\n\n let index_path = Self::index_path(root_path, name, ver);\n log::debug!(\"removing index file={:?}\", index_path);\n std::fs::remove_file(index_path)?;\n\n let data_path = FileMap::data_path(&root_path, name, ver);\n log::debug!(\"removing data file={:?}\", data_path);\n std::fs::remove_file(data_path)?;\n\n Ok(())\n }\n\n}\n\n\nstruct FileMap {\n fmap: rustc_hash::FxHashMap,\n}\n\nimpl FileMap {\n fn init(root_path: &Path, name: &str, vset: &HashSet, aver: u32) -> Result {\n //let active_file = Self::open_active_file(root_path, name, active_ver)?;\n log::debug!(\"fmap initing for name={} with vset={:?}\", name, vset);\n\n let mut fmap = FileMap {\n fmap: rustc_hash::FxHashMap::default(),\n };\n\n for ver in vset.iter() {\n if *ver != aver {\n fmap.add_file(root_path, name, *ver)?;\n }\n }\n\n fmap.add_file(root_path, name, aver)?;\n\n Ok(fmap)\n }\n\n fn close(&mut self) -> Result<(), Error> {\n for (_v, f) in &mut self.fmap {\n f.close()?;\n }\n Ok(())\n }\n\n fn close_versions(&mut self, vlist: &Vec, aver: u32) -> Result<(), Error> {\n for v in vlist {\n if *v == aver {\n continue\n }\n if let Some(mut f) = self.fmap.remove(v) {\n f.close()?;\n }\n }\n Ok(())\n }\n\n fn data_path(root_path: &Path, name: &str, ver: u32) -> PathBuf {\n root_path.join(format!(\"{}_d.{}\", name, ver))\n }\n\n fn add_file(&mut self, root_path: &Path, name: &str, ver: u32) -> Result<(), Error> {\n let ver_path = Self::data_path(root_path, name, ver);\n log::debug!(\"adding new file: {:?}\", ver_path);\n\n let f = NixFile::open(&ver_path, ver)?;\n\n self.fmap.insert(ver, f);\n Ok(())\n }\n\n fn file_mut(&mut self, ver: u32) -> &mut NixFile {\n self.fmap.get_mut(&ver).expect(&format!(\"write ver={} not found\", ver))\n }\n\n fn file(&self, ver: u32) -> &NixFile {\n &self.fmap.get(&ver).expect(&format!(\"read ver={} not found\", ver))\n }\n}" }, { "path": "crates/mojokv/src/error.rs", "content": "\n#[derive(thiserror::Error, Debug)]\npub enum Error {\n #[error(\"io error\")]\n IoErr(#[from] std::io::Error),\n\n #[error(\"mojo file error\")]\n MojoFileErr(#[from] mojoio::Error),\n\n #[error(\"Bucket {0} not found at ver={1}\")]\n BucketNotAtVerErr(String, u32),\n\n #[error(\"Bucket not writable\")]\n BucketNotWritableErr,\n\n #[error(\"Version no longer writable bucket ver={0} active ver={1}\")]\n VerNotWritable(u32, u32),\n\n #[error(\"Store not found\")]\n StoreNotFoundErr,\n\n #[error(\"Store not writable\")]\n StoreNotWritableErr,\n\n #[error(\"Missing arguments\")]\n MissingArgsErr,\n\n #[error(\"Commit lock could not be acquired\")]\n CommitLockedErr,\n\n #[error(\"Only single version exists\")]\n SingleVersionErr,\n\n #[error(\"Json serialization error\")]\n SerdeJsonErr(#[from] serde_json::Error),\n\n #[error(\"rmp encode error\")]\n RmpEncodeErr(#[from] rmp_serde::encode::Error),\n\n #[error(\"rmp decode error\")]\n RmpDecodeErr(#[from] rmp_serde::decode::Error),\n\n #[error(\"Key {0} not found\")]\n KeyNotFoundErr(u32),\n\n #[error(\"Key {0} not multiple of page size\")]\n KeyNotMultipleErr(u32),\n\n #[error(\"Version {0} not found\")]\n VersionNotFoundErr(u32),\n\n #[error(\"Parse int error\")]\n ParseIntErr(#[from] std::num::ParseIntError),\n\n #[error(\"Unknown error `{0}`\")]\n UnknownStr(String),\n\n #[error(\"UTF8 error\")]\n UTF8Err(#[from] std::str::Utf8Error),\n\n #[error(\"Unknown error\")]\n Unknown,\n}\n" }, { "path": "crates/mojokv/src/index/mem.rs", "content": "use std::io::Write;\n\nuse serde::Deserialize;\nuse serde::Serialize;\n\nuse crate::value::Value;\nuse crate::keymap::KeyMap;\nuse crate::Error;\nuse crate::utils;\nuse super::IndexHeader;\n\n\n//TODO: Reserve some space for additional data\n#[derive(Serialize, Deserialize)]\npub struct MemIndex {\n header: IndexHeader,\n pub kmap: KeyMap\n}\n\nimpl MemIndex {\n pub fn new(pps: usize) -> Self {\n MemIndex {\n header: IndexHeader::new(pps),\n kmap: KeyMap::new(pps),\n }\n }\n\n pub fn header(&self) -> &IndexHeader {\n &self.header\n }\n\n fn key_map(&self) -> &KeyMap {\n &self.kmap\n }\n\n pub fn set_active_ver(&mut self, ver: u32) {\n self.header.vset.insert(ver);\n self.header.active_ver = ver;\n }\n\n pub fn active_ver(&self) -> u32 {\n self.header.active_ver\n }\n\n pub fn max_key(&self) -> isize {\n self.header.max_key\n }\n\n pub fn update_min_max_ver(&mut self) -> Vec {\n let (min_ver, max_ver, vset) = self.kmap.get_min_max_ver();\n self.header.min_ver = min_ver;\n self.header.max_ver = max_ver;\n\n let non_ref_vers: Vec = self.header.vset.difference(&vset).map(|n| *n).collect();\n non_ref_vers\n }\n\n pub fn put(&mut self, key: u32, off: u32) -> Result<(), Error> {\n let mut val = Value::new();\n val.put_off(off);\n val.put_ver(self.header.active_ver);\n\n log::debug!(\"index put val:{:?}\", val);\n self.header.max_key = self.header.max_key.max(key as isize);\n self.kmap.put(key, val);\n Ok(())\n }\n\n pub fn get(&self, key: u32) -> Result, Error> {\n Ok(self.kmap.get(key))\n }\n\n pub fn truncate(&mut self, key: u32) -> Result<(), Error> {\n self.kmap.truncate(key);\n self.header.max_key = key as isize -1;\n Ok(())\n }\n\n pub fn iter<'a>(&'a self, from_key: u32, to_key: u32) -> Box + 'a > {\n let itr = MemIndexIterator {\n key: from_key,\n to_key,\n index: self\n };\n\n Box::new(itr)\n }\n\n pub fn serialize_to_path(&self, filepath: &std::path::Path) -> Result<(), Error> {\n let tmp_buf = rmp_serde::to_vec(&self)?;\n let cbuf = zstd::bulk::compress(&tmp_buf, 3)?;\n\n let mut f = std::fs::OpenOptions::new()\n .write(true)\n .create(true)\n .truncate(true)\n .open(filepath)?;\n\n let cap_buf = tmp_buf.len().to_le_bytes();\n f.write_all(&cap_buf)?;\n f.write_all(&cbuf)?;\n f.sync_data()?;\n\n Ok(()) \n }\n\n pub fn deserialize_from_path(filepath: &std::path::Path) -> Result<(usize, usize, MemIndex), Error> {\n let mut b = Vec::new();\n utils::load_file(filepath, &mut b)?;\n\n let cap = usize::from_le_bytes([b[0], b[1], b[2], b[3], b[4], b[5], b[6], b[7]]);\n\n let buf = zstd::bulk::decompress(&b[8..], cap)?;\n\n let index = rmp_serde::from_slice(&buf)?;\n Ok((cap, b.len(), index))\n }\n\n}\n\npub struct MemIndexIterator<'a> {\n index: &'a MemIndex,\n key: u32,\n to_key: u32\n}\n\nimpl<'a> MemIndexIterator<'a> {\n pub fn new(from_key: u32, to_key: u32, index: &'a MemIndex) -> Self {\n MemIndexIterator { \n index,\n key: from_key,\n to_key,\n }\n }\n}\n\nimpl<'a> Iterator for MemIndexIterator<'a> {\n type Item = (u32, &'a Value);\n\n fn next(&mut self) -> Option {\n if self.to_key > 0 && self.key >= self.to_key {\n return None;\n }\n\n let kmap_index = self.key/self.index.header.pps as u32;\n\n if kmap_index as usize >= self.index.key_map().slot_map.len() {\n return None;\n }\n\n let slot_map = &self.index.key_map().slot_map[kmap_index as usize];\n\n let ret = match slot_map {\n Some(map) => {\n let slot_index = (self.key as usize) %self.index.header.pps;\n if slot_index >= map.len() {\n return None;\n }\n\n let val = &map[slot_index];\n if val.is_allocated() {\n Some((self.key, val))\n }else{\n None\n }\n },\n None => None\n };\n\n\n self.key += 1;\n\n ret\n }\n}\n" }, { "path": "crates/mojokv/src/index/mod.rs", "content": "pub mod mem;\nuse std::collections::HashSet;\n\nuse crate::Error;\nuse crate::value::Value;\nuse serde::{Serialize, Deserialize};\n\npub const MOJO_INDEX_MAGIC: &'static str = \"mojo_index\";\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct IndexHeader {\n pub magic: String, \n pub format_ver: u32,\n pub min_ver: u32,\n pub max_ver: u32,\n pub vset: HashSet,\n pub active_ver: u32,\n pub max_key: isize,\n pub pps: usize,\n}\n\nimpl IndexHeader {\n pub fn new(pps: usize) -> Self {\n let mut vset = HashSet::new();\n vset.insert(1);\n\n IndexHeader {\n magic: MOJO_INDEX_MAGIC.to_owned(),\n format_ver: 1,\n min_ver: 1,\n max_ver: 1,\n vset,\n active_ver: 1,\n pps,\n max_key: -1,\n }\n }\n}\n\npub trait Index {\n fn put(&mut self, key: u32, off: u32) -> Result<(), Error>;\n fn get(&self, key: u32) -> Result, Error>;\n fn truncate(&mut self, key: u32) -> Result<(), Error>;\n}\n\npub trait IndexSerde {\n fn serialize(idx: &I, w: &mut W) -> Result<(), Error>;\n fn deserialize(idx: &I, r: &mut R) -> Result;\n}" }, { "path": "crates/mojokv/src/keymap.rs", "content": "\nuse std::collections::HashSet;\nuse crate::value::{Value, Slot};\nuse serde::{Serialize, Deserialize};\n\n#[derive(Serialize, Deserialize)]\npub struct KeyMap {\n pub slot_map: Vec,\n pps: usize\n}\n\nimpl KeyMap {\n //TODO: remove this flag\n pub fn new(pps: usize) -> Self {\n KeyMap { \n slot_map: Vec::new(),\n pps,\n }\n }\n\n fn alloc_value_arr(pps: usize) -> Vec {\n let v = vec![Value::new(); pps];\n v\n }\n\n pub fn get_min_max_ver(&self) -> (u32, u32, HashSet) {\n let mut set = HashSet::new();\n let (mut min_ver, mut max_ver) = (u32::MAX,0);\n\n for slot in self.slot_map.iter() {\n if let Some(slot) = slot {\n for val in slot.iter() {\n let v = val.get_ver();\n if v == 0 {\n break;\n }\n set.insert(v);\n min_ver = min_ver.min(v);\n max_ver = max_ver.max(v);\n }\n }\n }\n\n (min_ver, max_ver, set)\n }\n\n pub fn put(&mut self, key: u32, val: Value) {\n let slot = (key as usize)/self.pps;\n if slot >= self.slot_map.len() {\n log::debug!(\"KeyMap put key={}, value= {:?} slot={} kmaplen={}\", key, val, slot, self.slot_map.len());\n for _ in 0..(slot - self.slot_map.len() + 1) {\n self.slot_map.push(None);\n }\n }\n\n let val_arr = self.slot_map[slot].get_or_insert_with(||{\n KeyMap::alloc_value_arr(self.pps)\n });\n\n let slot_key = key % (self.pps as u32);\n log::debug!(\"KeyMap put slot_key={}\", slot_key);\n val_arr[slot_key as usize] = val; \n }\n\n pub fn get(&self, key: u32) -> Option<&Value> {\n let slot = key/self.pps as u32;\n log::debug!(\"KeyMap get key={}, slot={}, kmaplen={}\", key, slot, self.slot_map.len());\n if slot as usize >= self.slot_map.len() {\n return None;\n }\n\n self.slot_map[slot as usize].as_ref().map(|val_arr|{\n let slot_key = key % self.pps as u32;\n &val_arr[slot_key as usize]\n })\n }\n\n pub fn truncate(&mut self, key: u32) {\n let slot = key/self.pps as u32;\n self.slot_map.truncate((slot+1) as usize);\n let slot_key = key % (self.pps as u32);\n\n if let Some(slot_vec) = self.slot_map[slot as usize].as_mut() {\n for i in slot_key as usize ..slot_vec.len() {\n slot_vec[i].deallocate();\n }\n }\n }\n}" }, { "path": "crates/mojokv/src/lib.rs", "content": "//#![feature(write_all_vectored)]\n\npub mod index;\nmod bucket;\nmod error;\nmod value;\nmod state;\nmod keymap;\nmod utils;\nmod store;\nmod bmap;\n\npub use error::Error;\npub use bucket::Bucket;\npub use bmap::BucketMap;\npub use keymap::KeyMap;\npub use value::{Value, Slot};\npub use store::{Store, BucketOpenMode};\n\n\n//TODO: Pass pps from single place" }, { "path": "crates/mojokv/src/state.rs", "content": "\nuse crate::Error;\nuse mojoio::nix::NixFile;\nuse crate::utils;\nuse std::sync::Arc;\nuse parking_lot::RwLock;\nuse serde::{Serialize, Deserialize};\n\n#[derive(Debug, Serialize, Deserialize)]\npub struct StateInner {\n pub format_ver: u32,\n pub min_ver: u32,\n pub max_ver: u32,\n pub active_ver: u32,\n pub pps: u32,\n pub page_sz: u32,\n pub file_header_len: u32,\n pub file_page_sz: u32,\n\n //TODO: add timestamp\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize)]\npub struct State {\n inner: Arc>,\n\n #[serde(skip)]\n pub commit_lock: Arc>,\n}\n\nimpl State {\n\n pub fn new(page_sz: u32, pps: u32) -> Self {\n\n let inner = StateInner {\n format_ver: 1,\n min_ver: 1,\n max_ver: 1,\n active_ver: 1,\n pps,\n page_sz,\n file_header_len: NixFile::header_len() as u32,\n file_page_sz: page_sz + NixFile::header_len() as u32,\n };\n\n let state = State {\n inner: Arc::new(RwLock::new(inner)),\n commit_lock: Arc::new(RwLock::new(false)),\n };\n\n state\n }\n\n pub fn format_ver(&self) -> u32 {\n let inner = self.inner.read();\n inner.format_ver\n }\n\n pub fn active_ver(&self) -> u32 {\n let inner = self.inner.read();\n inner.active_ver\n }\n\n pub fn page_size(&self) -> u32 {\n let inner = self.inner.read();\n inner.page_sz\n }\n\n pub fn file_page_sz(&self) -> u32 {\n let inner = self.inner.read();\n inner.file_page_sz\n }\n\n pub fn pps(&self) -> u32 {\n let inner = self.inner.read();\n inner.pps\n }\n\n pub fn min_ver(&self) -> u32 {\n let inner = self.inner.read();\n inner.min_ver\n }\n\n pub fn max_ver(&self) -> u32 {\n let inner = self.inner.read();\n inner.max_ver\n }\n\n pub fn advance_ver(&self) -> u32 {\n let mut inner = self.inner.write();\n inner.active_ver += 1;\n inner.max_ver = inner.max_ver.max(inner.active_ver);\n\n inner.active_ver\n }\n\n pub fn serialize_to_path(&self, filepath: &std::path::Path) -> Result<(), Error> {\n let buf = rmp_serde::to_vec_named(&self)?;\n\n utils::write_file(filepath, &buf)?;\n\n Ok(()) \n }\n\n pub fn deserialize_from_path(filepath: &std::path::Path) -> Result {\n let mut buf = Vec::new();\n utils::load_file(filepath, &mut buf)?;\n\n let state = rmp_serde::from_slice(&buf)?;\n Ok(state)\n }\n}" }, { "path": "crates/mojokv/src/store.rs", "content": "use std::path::{Path, PathBuf};\nuse std::sync::Arc;\nuse crate::{Error, utils};\nuse crate::state::State;\nuse crate::bucket::Bucket;\nuse crate::bmap::BucketMap;\nuse crate::index::mem::MemIndex;\nuse parking_lot::RwLock;\nuse fslock::LockFile;\n\nstruct StoreInner {\n root_path: PathBuf,\n state: State,\n is_write: bool,\n bmap: BucketMap,\n}\npub struct Store {\n inner: Arc>,\n}\n\nimpl Store {\n pub fn exists(&self, name: &str) -> bool {\n let inner = self.inner.read();\n inner.bmap.exists(name)\n }\n\n pub fn open(&self, name: &str, mode: BucketOpenMode) -> Result {\n let mut inner = self.inner.write();\n\n log::debug!(\"store bucket open name={} mode writable={} store is write: {}\", name, mode.is_write(), inner.is_write);\n\n if !inner.is_write && mode.is_write() {\n return Err(Error::StoreNotWritableErr);\n }\n\n let mut b = match inner.bmap.get(name) {\n Some(v) => {\n log::debug!(\"Bucket name={} exists at ver={}\", name, v);\n Bucket::load(&inner.root_path, name, inner.state.clone(), inner.bmap.clone(), v)?\n },\n None => {\n log::debug!(\"Bucket name={} does not exists\", name);\n if !inner.is_write {\n return Err(Error::StoreNotWritableErr);\n }\n Bucket::new(&inner.root_path, name, inner.state.clone(), inner.bmap.clone())?\n }\n };\n\n if inner.is_write && mode.is_write() {\n log::debug!(\"setting bucket={} to writable\", name);\n b.set_writable();\n b.sync()?;\n }\n\n if mode.is_write() {\n inner.sync_bmap()?;\n }\n\n Ok(b)\n }\n\n pub fn delete(&self, name: &str) -> Result<(), Error> {\n let mut inner = self.inner.write();\n let aver = inner.state.active_ver();\n\n inner.bmap.delete(&inner.root_path, name, aver)?;\n inner.sync_bmap()\n }\n\n pub fn commit(&self) -> Result {\n let mut inner = self.inner.write();\n\n log::debug!(\"committing store ver={}\", inner.state.active_ver());\n\n let _ = inner.state.commit_lock.write();\n\n log::debug!(\"about to acquire commit file lock ver={}\", inner.state.active_ver());\n let mut commit_lock_file = Self::create_lock_file(&inner.root_path)?;\n\n if !commit_lock_file.try_lock_with_pid()? {\n return Err(Error::CommitLockedErr);\n }\n\n let new_ver = inner.state.advance_ver();\n inner.sync_state()?;\n inner.sync_bmap()?;\n\n log::debug!(\"committing store done\");\n Ok(new_ver)\n }\n\n pub fn active_ver(&self) -> u32 {\n let inner = self.inner.read();\n inner.state.active_ver()\n }\n\n pub fn load_state(rootpath: &Path) -> Result {\n let state_path = rootpath.join(\"mojo.state\");\n log::debug!(\"loading state from {:?}\", state_path);\n let state = State::deserialize_from_path(&state_path)?;\n Ok(state)\n }\n\n pub fn readonly(root_path: &Path, ver: u32) -> Result {\n log::debug!(\"opening store in readonly mode at ver={}\", ver);\n let state = Self::load_state(root_path)?;\n Self::load_store(root_path, state, ver)\n }\n\n pub fn writable(rootpath: &Path, create: bool, page_sz: Option, pps: Option) -> Result {\n let init_path = rootpath.join(\"mojo.init\");\n\n if create && (page_sz.is_none() || pps.is_none()) {\n log::debug!(\"Missing mandatory params page_sz:{:?} pps:{:?}\", page_sz, pps);\n return Err(Error::MissingArgsErr);\n }\n\n let store = if !init_path.exists() {\n if !create {\n return Err(Error::StoreNotFoundErr);\n }\n\n log::debug!(\"Store does not exists. Initing now\");\n let mut store = Store::new(rootpath, page_sz.unwrap(), pps.unwrap())?;\n store.init()?;\n log::debug!(\"Store init successfull\");\n store\n }else{\n let state = Self::load_state(rootpath)?;\n let aver = state.active_ver();\n Self::load_store(rootpath, state, aver)?\n };\n\n {\n let mut inner = store.inner.write();\n inner.is_write = true;\n }\n \n Ok(store)\n }\n\n fn load_store(root_path: &Path, state: State, ver: u32) -> Result {\n log::debug!(\"loading store at ver={}\", ver);\n let bmap = BucketMap::load(root_path, ver)?;\n\n let inner = StoreInner {\n root_path: root_path.to_owned(),\n state,\n is_write: false,\n bmap,\n };\n\n let store = Store {inner: Arc::new(RwLock::new(inner))};\n\n Ok(store)\n }\n\n pub fn get_index(&self, name: &str) -> Result, Error> {\n let inner = self.inner.read();\n\n match inner.bmap.get(name) {\n Some(v) => {\n let ret = Bucket::load_index(&inner.root_path, name, v)?;\n Ok(Some(ret))\n },\n None => {\n log::debug!(\"Bucket name={} does not exists\", name);\n return Ok(None)\n }\n }\n }\n\n fn new(root_path: &Path, page_sz: u32, pps: u32) -> Result {\n let state = State::new(page_sz, pps);\n\n let inner = StoreInner {\n root_path: root_path.to_owned(),\n state,\n is_write: false,\n bmap: BucketMap::default(),\n };\n\n let store = Store {\n inner: Arc::new(RwLock::new(inner)),\n };\n\n Ok(store)\n }\n\n fn init(&mut self) -> Result<(), Error> {\n let mut inner = self.inner.write();\n\n std::fs::create_dir_all(&inner.root_path)?;\n inner.sync()?;\n let touch_file = inner.root_path.join(\"mojo.init\");\n utils::touch_file(&touch_file)?;\n Ok(())\n }\n\n\n fn create_lock_file(root_path: &Path) -> Result {\n let lock_path = root_path.join(\"mojo.lock\");\n log::debug!(\"creating lock file: {:?}\", lock_path);\n Ok(LockFile::open(&lock_path)?)\n }\n}\n\nimpl StoreInner {\n fn sync(&mut self) -> Result<(), Error> {\n self.sync_state()?;\n self.sync_bmap()?;\n Ok(())\n }\n\n fn sync_bmap(&mut self) -> Result<(), Error> {\n log::debug!(\"syncing bmap at ver={}\", self.state.active_ver());\n\n let bmap_path = self.root_path.join(&format!(\"mojo.bmap.{}\", self.state.active_ver()));\n\n self.bmap.serialize_to_path(&bmap_path)?;\n\n Ok(())\n }\n\n\n fn sync_state(&mut self) -> Result<(), Error> {\n let file_path = self.root_path.join(\"mojo.state\");\n\n log::debug!(\"syncing state ver={} {:?}\", self.state.active_ver(), file_path);\n \n self.state.serialize_to_path(&file_path)?;\n\n log::debug!(\"syncing state done\");\n Ok(())\n }\n}\n\n\n#[derive(Clone, Debug, PartialEq)]\npub enum BucketOpenMode {\n Read,\n Write,\n}\n\nimpl BucketOpenMode {\n pub fn is_write(&self) -> bool {\n *self == Self::Write\n }\n}" }, { "path": "crates/mojokv/src/utils.rs", "content": "use std::path::Path;\nuse std::io::{Read, Write};\n\nuse crate::Error;\n\npub fn load_file(path: &Path, buf: &mut Vec) -> Result<(), Error> {\n let mut f = std::fs::OpenOptions::new().read(true).open(path)?;\n f.read_to_end(buf)?;\n Ok(())\n}\n\npub fn write_file(path: &Path, buf: &[u8]) -> Result<(), Error> {\n let mut f = std::fs::OpenOptions::new()\n .write(true)\n .create(true)\n .truncate(true)\n .open(path)?;\n\n f.write_all(buf)?;\n f.sync_data()?;\n\n Ok(())\n}\n\npub fn touch_file(path: &Path) -> Result<(), Error> {\n log::debug!(\"creating init file: {:?}\", path);\n let _ = std::fs::File::create(path)?;\n\n log::debug!(\"creating init file done\");\n Ok(())\n}\n\n/*\npub fn read_le_u32(r: &mut R) -> Result {\n let mut buf = [0u8; 4];\n r.read_exact(&mut buf)?;\n Ok(u32::from_le_bytes(buf))\n}\n\npub fn read_le_isize(r: &mut R) -> Result {\n let mut buf = [0u8; std::mem::size_of::()];\n r.read_exact(&mut buf)?;\n Ok(isize::from_le_bytes(buf))\n}\n\npub fn read_le_usize(r: &mut R) -> Result {\n let mut buf = [0u8; std::mem::size_of::()];\n r.read_exact(&mut buf)?;\n Ok(usize::from_le_bytes(buf))\n}\n*/" }, { "path": "crates/mojokv/src/value.rs", "content": "\nuse modular_bitfield::{bitfield, specifiers::*};\nuse serde::{Serialize, Deserialize};\nuse serde::de::Visitor;\n\npub type Slot = Option>;\n\n#[derive(Clone, Copy)]\n#[bitfield]\npub struct Value {\n off: B32,\n ver: B24\n}\n\nimpl std::fmt::Debug for Value {\n fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> Result<(), std::fmt::Error> {\n f.write_fmt(format_args!(\"o={},v={}\", self.off(), self.ver()))\n }\n}\n\nimpl Value {\n pub fn is_allocated(&self) -> bool {\n self.ver() > 0\n }\n\n pub fn deallocate(&mut self) {\n self.set_off(0);\n self.set_ver(0);\n }\n\n pub fn put_off(&mut self, off: u32) {\n self.set_off(off);\n }\n\n pub fn get_off(&self) -> u32 {\n self.off()\n }\n\n pub fn put_ver(&mut self, v: u32) {\n self.set_ver(v);\n }\n\n pub fn get_ver(&self) -> u32 {\n self.ver() as u32\n }\n\n}\n\nimpl Serialize for Value {\n fn serialize(&self, serializer: S) -> Result\n where\n S: serde::Serializer {\n serializer.serialize_bytes(&self.into_bytes())\n }\n}\n\nstruct ValueVisitor {}\n\nimpl<'de> Visitor<'de> for ValueVisitor {\n type Value = Value;\n\n fn visit_bytes(self, v: &[u8]) -> Result\n where\n E: serde::de::Error, {\n\n if v.len() != 7 {\n return Err(serde::de::Error::invalid_length(v.len(), &self));\n }\n \n Ok(Value::from_bytes([v[0], v[1], v[2], v[3], v[4], v[5], v[6]]))\n }\n\n fn expecting(&self, formatter: &mut std::fmt::Formatter) -> std::fmt::Result {\n formatter.write_str(\"byte array of 7 bytes\")\n }\n\n}\n\nimpl<'de> Deserialize<'de> for Value {\n fn deserialize(deserializer: D) -> Result\n where\n D: serde::Deserializer<'de> {\n deserializer.deserialize_bytes(ValueVisitor{})\n }\n}\n\n/*\n\npub fn serialize_valuearr(val_opt: &Option>, w: &mut W) -> Result<(), Error> {\n match val_opt {\n Some(val) => {\n let n_items = val.len() as u32;\n w.write_all(&n_items.to_le_bytes())?;\n\n for elem in val.iter() {\n rmp_serde::encode::write(w, elem)?;\n }\n },\n None => {\n w.write_all(&0u32.to_le_bytes())?;\n }\n }\n\n Ok(())\n}\n\npub fn deserialize_valuearr(r: &mut R, pps: usize) -> Result>, Error> {\n let mut tmp_buf = [0u8; 4];\n \n r.read_exact(&mut tmp_buf)?;\n let count = u32::from_le_bytes(tmp_buf);\n\n if count == 0 {\n Ok(None)\n }else{\n if count as usize != pps {\n return Err(Error::UnknownStr(\"Less number of values than expected\".to_owned()));\n }\n let mut tmp_vec = Vec::new();\n for _ in 0..count {\n let val = rmp_serde::decode::from_read(r)?;\n tmp_vec.push(val);\n }\n\n Ok(Some(tmp_vec))\n }\n\n}\n*/" }, { "path": "docs/design.md", "content": "- [Design choices](#design-choices)\n- [Design](#design)\n - [Layers](#layers)\n - [MojoKV](#mojokv)\n - [Index](#index)\n - [MojoFS](#mojofs)\n\n# Design choices\n\nMojofs is a versioning file-system for sqlite database. It is a completely tailor made for sqlite and is not\nto be used as a general purpose file-system. This allows the fs to make certain assumptions which in turn drives design choices. \n\nFollowing are the assumptions about any sqlite fs, which I think are reasonable:\n\n* Small number of files\n* Large files\n* Very flat folder structure\n* Read/write in terms of fixed page size (or there multiple) dominates compared to any random offset.\n\nThis influences the way we store namespace (i.e. folder/file names), the index (i.e. mapping between pages and offsets) and internal decoupling (e.g mojofs itself doesn't do much but instead uses mojokv as storage layer)\n\n# Design\n\n## Layers\n\nThe layers of mojofs are:\n\nSqlite -> Mojofs extension lib -> mojofs -> mojokv\n\nThe sqlite could be the cli (i.e. sqlite3 binary) or sqlite C API or any of its bindings. \nThe fs is developed as both an sqlite extension and vfs, which is compiled down to a shared library.\nThis shared library is loaded as extension which then registers the VFS=mojo with the sqlite.\nMojofs implements the sqlite's [VFS interface](https://www.sqlite.org/vfs.html) which asks file system like apis to be implemented.\n\nThe fs uses mojokv (Mojo Key-Value store) to actually store the data. The KV has a notion of 'bucket' which the fs creates for each new file asked by sqlite. All the buckets i.e files taken together are versioned.\n\nThe MojoKV is tailor made for the needs of sqlite and as such is not a general purpose key-value store.\n\n## MojoKV\n\nMojoKV is the core storage layer which handles the index and actual data files.\nThe KV has a notion of bucket, on which read/write happens.\n\nEach bucket has an index which is a mapping of (Page No) => (New Page No, version)\n\n### Index\n\nThe write api at [File IO methods](https://www.sqlite.org/c3ref/io_methods.html) looks like below:\n\n```\nint (*xWrite)(sqlite3_file*, const void*, int iAmt, sqlite3_int64 iOfst);\n```\n\nThe `sqlite3_file*` is the file handle, `const void*, int iAmt` are the pointer to data and its length. The `iOfst` is the offset at which data needs to be written. Mojofs has versions/snapshots and the sqlite API does not know anything about it. This means that when write is called, mojofs needs to know to which version the data should be written.\n\nThe sqlite divides the file into pages. This is configurable when db is created for the first time, but assume 4KB for this document. We can logically think of a file and its versions as below:\n\n| |1|2|3|4|\n|------|-|-|-|-|\n|Page 0|1|2|3|4|\n|Page 1|1|2| |4|\n|Page 2|1| | | |\n|Page 3|1| |3|4|\n|Page 4|1| |3| |\n\nThe file above has 5 pages of 4KB each and are depicted as rows. The columns are the version numbers.\nThe value for page 0 and version 2 is 2. This means that the page 0 was modified in version 2.\nFor page 2 and version 2 the value is empty and it means the page was not modified from previous version.\nWhen the file is created new, naturally all the pages will be marked as version 1.\n\nWhen page 2 and version=3 needs to be read, it actually needs to be fetch the page from version=1.\n\nEssentially the page no should map to a certain location on disk.\nFor simple, unversioned file, the page number translates to an offset in file i.e. page no x page size.\nBut for versioned file, the mapping of between the tuple (Pg No, Version) => \\ is needed.\n\nThis nicely yields itself to be stored in a key-value store.\n\nThe actual data-structure used is `Vec>>`. The entire bucket/file is split into slots and the inner/nested vector (`Vec`) is called slot map. Each slot corresponds to a page of sqlite. A slot is a tuple of (physical page number, version). There is a maximum number of slots controlled by a tunable called `pages per slot`. \n \nThe version 3 for the file in the above example is represented as below (with `pages per slot` = 2) \nAssume page size=4KB.\n\n```\nSlot 0:\n index 0 = (0,3)\n index 1 = (1,2)\nSlot 1:\n index 0 = (2,1)\n index 1 = (2,3)\nSlot 2:\n index 0 = (2,3)\n```\n\nThe outer vector consists of 3 vectors, one each corresponding to one slot. The key (which is a page no) is converted to slot vector and index within each slot vector. So the writing and reading from from index is O(1).\n\nThe value is a tuple (physical page no in a version file, version). Since for each version a separate file is created, the page number in the tuple is the physical page no and version in which the page was last modified/created.\n\n## MojoFS\n\nEach sqlite database is created as a directory instead of a single file.\nFor each file name = F and version = V the mojokv stores the file with name F.V.\nThe filename (=F) is chosen by the fs. \n\nSo for a given sqlite db 'a.db' being created for the first time the fs will create the following on disk:\n\n```\nsudeep@local-3 mojo % tree ./a.db \n./a.db\n├── a.db_d.1\n├── a.db_i.1\n├── mojo.bmap.1\n├── mojo.init\n└── mojo.state\n...\n```\n\nSo the fs creates the dir = `a.db`. The sqlite issues open call for the main db file i.e. test.db. \nThe fs adds `_d` (d=data) to the name creates `a.db_d.1`. The `.1` is the version.\n\nThe `a.db_i.1` is the index file, which is internal to the kv. This has the mapping of (page no, version) => Physical offset.\n\nWhen a version 2 is created it will create a file: `a.db_d.2`. This file will contain only those pages which were modified in that version. \nAs a result it will also create the index file `a.db_i.2`.\n\nThe `mojo.*` files are files created/for the mojokv:\n\n`mojo.init` => Presence of this file indicates that filesystem was properly initialized.\n\n`mojo.state` => It stores the current state of the filesystem, like how many versions, current version, etc\n\n`mojo.bmap.1` => Bucket map. This stores namespace of the mojokv." }, { "path": "docs/source.md", "content": "\nWhether you want to just read or contribute to mojo, this document describes the code to get you started.\n\n*__Note__*: This is not a design document.\n\n## Bird's eye view\n\n* All the rust code is under `crates` folder\n* All the docs are under `docs` folder\n* The `sqlite-ext` folder has C code which is compiled down to shared lib\n* The `test-scripts` has various assorted test scripts which includes perf & black-box test\n\n\n## Crates\n\n### mojokv\n\nThis is the KV which powers the mojofs.\n\n* `store.rs` has the main store object. Buckets are \"opened\" using a store object\n* `bucket.rs` has the bucket object. A bucket has get & put methods. Each bucket has an index.\n* `index/mem.rs` has the `MemIndex` which is in-memory index which has the mapping `offset -> (physical offset, version)`. Each index has KeyMap.\n* `keymap.rs` The index is split into slots and a vector such slots are wrapped in KeyMap. \n* `state.rs` has the state object which reflects the current state of the kv\n\n### mojoio\n\nAbstracts out the notion of file. This is the code which does the actual IO. It will have different implementations including remote KV store.\n\n* `nix.rs` implements unix based file\n\n### mojofs\n\nMojofs is the filesystem which is powered by mojokv. Each user file in fs maps to a bucket in mojokv.\n\n* `vfs.rs` has FS like operations like `open`, `delete`, `access`, etc\n* `kvfile.rs` has file like object which is implemented using mojokv, hence the name.\n* `native_file.rs` is the regular passthrough file object (uses std read/write)\n* `vfsfile.rs` has the object VFSFile which either is a kvfile or nativefile. At present everything is kvfile. The native file will be used for transient/temp files which does not need versioning. This is an optimization.\n* `lib.rs` has vfs functions needed by sqlite e.g. `fn mojo_read(sfile: *mut sqlite3_file, ptr: *mut c_void, n: i32, off: i64)`\n\n\n### mojo-cli\n\nThis a CLI utility to control the mojokv. Each command in mojo-cli maps to a `*.rs` file. Example: `commit` command can be found in `commit.rs`\n" }, { "path": "docs/user-guide.md", "content": "\nThis document assumes that you have followed the build instructions and/or have the libmojo shared library.\n\n- [Opening/Creating the database](#openingcreating-the-database)\n- [Committing database](#committing-database)\n- [Committing MojoFS vs Committing Database](#committing-mojofs-vs-committing-database)\n- [Reading old version](#reading-old-version)\n\n\n## Opening/Creating the database\n\nMojofs is compiled down to a shared library. \nIt should be loaded as extension in sqlite before it can be used.\n\n```shell\nsqlite3 <\"\ndb_path=\"a.db\"\n\ncon = sqlite3.connect(\":memory:\")\ncon.enable_load_extension(True)\ncon.execute(f\"select load_extension('{mojo_lib}')\")\ncon.execute(\"pragma page_size=4096\")\ncon.enable_load_extension(False)\ncon.close()\n\nconn_str = f\"file:{db_path}?vfs=mojo&pagesz=4096\"\nconn = sqlite3.Connection(conn_str, uri=True)\n```\n\nLet's decode the URI in the open `.open 'file:a.db?vfs=mojo&pagesz=4096'`:\n\n* `a.db` => Name of the database\n* `vfs=mojo` => Name of the MojoFS\n* `pagesz=4096` => Page size used by the fs.\n This should same as the page size in the pragma `pragma page_size = 4096`\n\nOnce set, the page size cannot be changed.\n\nWhen the database is created for the first time, it starts with version=1.\nVersion numbers are ever incrementing and the highest version number is writable \nand old versions are read-only. \n\nThe fs has to be commited to make the current active version as read-only.\n\n## Committing database\n\nThe `mojo-cli` is the tool for administration of the mojofs. To commit the database `a.db`:\n\n```shell\nmojo-cli ./a.db commit\n```\n\nCommit advances version number by 1. So if current active version=1 then after committing, a new version=2\nwill be created and version=1 will be read-only now.\n\nCommitting FS is really a cheap operation. It only manipulates the metadata of the FS and no data movement\nis involved.\n\n## Committing MojoFS vs Committing Database\n\nCommitting the fs is different than committing the database. You can continue to use the database\nin the active version as long as you want. This means multiple transactions can be initiated with\nall the DB commits and rollbacks, all in a single version. \n\nCommitting the database makes sqlite issue `fsync()` call which makes the data written to disk durable.\nIt is recommended to commit the fs when databases is just committed/rolled-backed.\nThe FS is not aware any uncommitted transactions and committing the fs midway can cause undefined behaviour.\n\n\n## Reading old version\n\nPass the `ver=` and `mode=ro` to open it in readonly mode:\n\n```\n.open 'file:a.db?vfs=mojo&pagesz=4096&ver=2&mode=ro'\n```" }, { "path": "meson.build", "content": "project('mojo', 'c', default_options: ['c_std=c11'], version: '0.1.0')\n\ncompiler = meson.get_compiler('c')\n\nmojokv_lib_path = meson.source_root() + '/target/debug'\nif get_option('buildtype') == 'release'\n mojokv_lib_path = meson.source_root() + '/target/release'\nendif\n\nmojofs_lib = compiler.find_library('mojofs',\n dirs: [mojokv_lib_path])\n\nmojofs_dep = declare_dependency(dependencies: [mojofs_lib])\nsqlite_dep = dependency('sqlite3')\n\nshared_library('mojo',\n name_prefix: '',\n sources: ['sqlite-ext/mojo.c'],\n include_directories: ['sqlite-ext'],\n dependencies: [mojofs_dep, sqlite_dep])\n\n" }, { "path": "sqlite-ext/mojo.c", "content": "#include \n\n#include \nSQLITE_EXTENSION_INIT1\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n/*\n** The maximum pathname length supported by this VFS.\n*/\n#define MAXPATHNAME 512\n\n/*\n** Argument zPath points to a nul-terminated string containing a file path.\n** If zPath is an absolute path, then it is copied as is into the output \n** buffer. Otherwise, if it is a relative path, then the equivalent full\n** path is written to the output buffer.\n**\n** This function assumes that paths are UNIX style. Specifically, that:\n**\n** 1. Path components are separated by a '/'. and \n** 2. Full paths begin with a '/' character.\n*/\nstatic int mojo_fullpath_name(\n sqlite3_vfs *pVfs, /* VFS */\n const char *zPath, /* Input path (possibly a relative path) */\n int nPathOut, /* Size of output buffer in bytes */\n char *zPathOut /* Pointer to output buffer */\n){\n char zDir[MAXPATHNAME+1];\n if( zPath[0]=='/' ){\n zDir[0] = '\\0';\n }else{\n if( getcwd(zDir, sizeof(zDir))==0 ) return SQLITE_IOERR;\n }\n zDir[MAXPATHNAME] = '\\0';\n\n sqlite3_snprintf(nPathOut, zPathOut, \"%s/%s\", zDir, zPath);\n zPathOut[nPathOut-1] = '\\0';\n\n return SQLITE_OK;\n}\n\nint sqlite3_mojo_init(sqlite3 *db, char **pzErrMsg, const sqlite3_api_routines *pApi){\n int rc = SQLITE_OK;\n SQLITE_EXTENSION_INIT2(pApi);\n\n mojofs_init_log();\n\n sqlite3_vfs *vfs = mojo_create();\n vfs->xFullPathname = mojo_fullpath_name;\n rc = sqlite3_vfs_register(vfs, 0);\n //printf(\"extension mojo called with rc=%d\\n\", rc);\n fflush(stdout);\n return rc;\n}" }, { "path": "sqlite-ext/mojofs.h", "content": "#ifndef __mojo_h\n#define __mojo_h\n\n#include \n\ntypedef struct MojoFile {\n sqlite3_vfs base;\n void* custom_file;\n} MojoFile;\n\nsqlite3_vfs* mojo_create();\n\nvoid mojofs_init_log();\n#endif" }, { "path": "test-scripts/commands.py", "content": "\"\"\"Multiple commands for testing\"\"\"\n\nimport subprocess\nimport sqlite3\n\nMOJOKV_CLI=None\n\nclass TestConfig:\n \"\"\"Config for test db\"\"\"\n\n def __init__(self, page_sz=4096, journal_mode=\"WAL\", vac_mode=\"NONE\"):\n self.page_sz = page_sz\n self.journal_mode = journal_mode\n self.vac_mode = vac_mode\n\n def __repr__(self):\n return f\"page_sz={self.page_sz} journal={self.journal_mode}\"\n\ndef vacuum(cur):\n \"\"\"Vacuum\"\"\"\n cur.execute(\"vacuum\")\n\ndef opendb(cfg, db_path, ver=\"1\", mode=\"\"):\n \"\"\"Open sqlite db using cfg\"\"\"\n\n if mode == \"\":\n conn_str = f\"file:{db_path}?vfs=mojo&ver={ver}&pagesz={cfg.page_sz}&pps=65536\"\n else:\n conn_str = f\"file:{db_path}?vfs=mojo&mode=ro&ver={ver}&pagesz={cfg.page_sz}&pps=65536\"\n\n conn = sqlite3.Connection(conn_str, uri=True)\n\n if mode != \"ro\":\n conn.execute(f\"PRAGMA page_size={cfg.page_sz}\")\n conn.execute(f\"PRAGMA journal_mode={cfg.journal_mode}\")\n conn.execute(f\"PRAGMA auto_vacuum={cfg.vac_mode}\")\n\n return conn\n\ndef mkdir(dir_path):\n \"\"\"\"Make dir\"\"\"\n subprocess.run([\"mkdir\", \"-p\", dir_path], check=True, capture_output=True)\n\ndef commit_version(dbpath):\n '''commit_version commits version for given dbpath'''\n subprocess.run([MOJOKV_CLI, dbpath, \"commit\"], check=True, capture_output=True)\n\ndef create_table_person(cur):\n \"\"\"Create table Person\"\"\"\n cur.execute(\"\"\"create table if not exists person(\n name text primary key,\n age integer,\n id integer\n )\"\"\")\n\n cur.execute(\"create index if not exists person_idx_1 on person(id)\")\n\ndef get_row_count(cur, table):\n \"\"\"Get row count of table\"\"\"\n row = cur.execute(f\"select count(*) from {table}\").fetchone()\n if row is None or row[0] is None:\n return 0\n return int(row[0])\n\ndef table_person_count(conn):\n \"\"\"Get row count of Person\"\"\"\n cur = conn.cursor()\n row = cur.execute(\"select count(*) from person\").fetchone()\n return int(row[0])\n\ndef get_max_id_person(cur):\n \"\"\"get max id\"\"\"\n row = cur.execute(\"select max(id) from person\").fetchone()\n if row is None or row[0] is None:\n return 0\n return int(row[0])\n\ndef insert_table_person(cur, count):\n \"\"\"Insert into person table\"\"\"\n\n max_id = get_max_id_person(cur)\n\n for n in range(max_id+1, max_id+count+1):\n name = f\"name-{n}\"\n cur.execute(\"insert into person(name,age,id) values(?,?,?)\", [name,n,n])\n\ndef delete_table_person(cur, from_id, to_id):\n \"\"\"Delete from person table\"\"\"\n\n cur.execute(f\"delete from person where id>={from_id} and id <= {to_id}\")\n\ndef drop_table_person(cur):\n \"\"\"\"Drop table person\"\"\"\n\n cur.execute(\"drop table person\")\n\ndef copy_table_person(cur, new_table):\n \"\"\"\"Copy table from person table\"\"\"\n cur.execute(f\"create table if not exists {new_table} as select name,age,id from person\")\n cur.execute(f\"create index if not exists {new_table}_idx_1 on {new_table}(id)\")" }, { "path": "test-scripts/perftest.py", "content": "\"\"\"\" Perf test for mojo filesystem \"\"\"\n\nimport sqlite3\nimport os\nimport shutil\nimport time\nimport sys\n\ndef rm_dir(path):\n '''Remove dir. Ignore file not found error'''\n try:\n if os.path.isdir(path):\n rm_fr(path)\n else:\n os.unlink(path)\n except FileNotFoundError:\n pass\n\ndef rm_fr(path):\n '''Equivalent of rm -fr'''\n\n journal_path = path + \"-journal\"\n wal_path = path + \"-wal\"\n\n rm_dir(journal_path)\n rm_dir(wal_path)\n\n if not os.path.exists(path):\n return\n if os.path.isfile(path) or os.path.islink(path):\n os.unlink(path)\n else:\n shutil.rmtree(path)\n\ndef _mojo_conn_str(db_path, ver=\"1\", mode=\"\"):\n \"\"\"open database\"\"\"\n if mode == \"\":\n conn_str = f\"file:{db_path}?vfs=mojo&ver={ver}&pagesz=4096&pps=65536\"\n else:\n conn_str = f\"file:{db_path}?vfs=mojo&mode=ro&ver={ver}&pagesz=4096&pps=65536\"\n\n return conn_str\n\ndef _std_conn_str(db_path, mode=\"\"):\n if mode == \"\":\n conn_str = f\"file:{db_path}\"\n else:\n conn_str = f\"file:{db_path}?mode=ro\"\n return conn_str\n\ndef open_db(db_path, vfs=\"mojo\", ver=\"1\", mode=\"\"):\n \"\"\"Open database\"\"\"\n if vfs == \"mojo\":\n conn_str=_mojo_conn_str(db_path, ver=ver, mode=mode)\n else:\n conn_str=_std_conn_str(db_path, mode=mode)\n\n conn = sqlite3.Connection(conn_str, uri=True)\n\n if mode != \"ro\":\n conn.execute(\"PRAGMA page_size=4096\")\n return conn\n\n\ndef create_table(conn):\n \"\"\"create table\"\"\"\n conn.execute(\"create table test(s text primary key)\")\n\ndef insert_rows(conn, row_count, ver, suffix=\"\"):\n \"\"\"create table\"\"\"\n\n cur = conn.cursor()\n for i in range(row_count):\n val = f\"{ver}-text-{i}{suffix}\"\n cur.execute(\"insert into test values(?)\", (val,))\n\ndef count_rows(conn, condition=\"\"):\n \"\"\"count rows\"\"\"\n\n cur = conn.cursor()\n if condition == \"\":\n row = cur.execute(\"select count(*) from test\").fetchone()\n else:\n row = cur.execute(f\"select count(*) from test where {condition}\").fetchone()\n \n return int(row[0])\n\ndef select_rows(conn, condition=\"\"):\n \"\"\"select rows\"\"\"\n\n cur = conn.cursor()\n if condition == \"\":\n rows = cur.execute(\"select * from test\")\n else:\n rows = cur.execute(f\"select * from test where {condition}\")\n \n count = 0\n for _r in rows:\n count += 1\n return count\n\ndef update_text_rows(conn):\n \"\"\"Update rows\"\"\"\n key = \"odd-update-text\"\n conn.execute(\"update test set s = ? where s like 'odd%'\",(key,))\n conn.commit()\n\n\ndef load_extension(mojo_lib):\n \"\"\"load_ext\"\"\"\n print(\"using libpath =\", mojo_lib)\n\n con = sqlite3.connect(\":memory:\")\n # enable extension loading\n con.enable_load_extension(True)\n con.execute(f\"select load_extension('{mojo_lib}')\")\n con.execute(\"pragma page_size=4096\")\n con.enable_load_extension(False)\n con.close()\n\nROW_COUNT=10000000\n\ndef perf_insert(conn):\n \"\"\"\"Perf insert\"\"\"\n\n start = time.time()\n create_table(conn)\n insert_rows(conn, ROW_COUNT, \"1\")\n conn.commit()\n end = time.time()\n return end-start\n\ndef perf_select(conn):\n \"\"\"\"Perf select\"\"\"\n\n start = time.time()\n count = select_rows(conn)\n end = time.time()\n print(\"select iter count:\", count)\n return end-start\n\ndef perf_count_rows(conn):\n \"\"\"\"Perf select\"\"\"\n\n start = time.time()\n count = count_rows(conn, \"s like '%abc%'\")\n print(\"row count:\", count)\n end = time.time()\n return end-start\n\ndef perf_update_rows(conn):\n \"\"\"\"Perf update rows\"\"\"\n\n start = time.time()\n update_text_rows(conn)\n end = time.time()\n return end-start\n\nif __name__ == '__main__':\n if len(sys.argv[1:]) < 1:\n print(\"Error: missing extension library path\")\n sys.exit(1)\n\n ext_path = sys.argv[1]\n load_extension(ext_path)\n\n STD_DBPATH=\"./perf-std.db\"\n MOJO_DBPATH=\"./perf-mojo.db\"\n\n try:\n rm_fr(STD_DBPATH)\n rm_fr(MOJO_DBPATH)\n\n perf_list=[\n (\"insert\", perf_insert),\n (\"update rows\", perf_update_rows),\n (\"select\", perf_select),\n (\"row count\", perf_count_rows),\n ]\n\n for desc, perf_fn in perf_list:\n print(\"Running perf for:\", desc)\n e = []\n for dbpath, vfs in [(STD_DBPATH, \"std\"), (MOJO_DBPATH, \"mojo\")]:\n dbconn = open_db(dbpath, vfs=vfs)\n elapsed = perf_fn(dbconn)\n e.append(elapsed)\n print(f\"\\tvfs={vfs} time elapsed (s):\", elapsed)\n dbconn.close()\n ratio = round(e[1]/e[0], 3)\n print(f\"\\tMojo takes {ratio} times than std vfs\")\n print(\"------------------------\")\n\n except Exception as e:\n raise e\n finally:\n rm_fr(STD_DBPATH)\n rm_fr(MOJO_DBPATH)\n" }, { "path": "test-scripts/test.sql", "content": "\npragma page_size = 4096;\n.load ./build/libmojo\n.open 'file:a.db?vfs=mojo&pagesz=4096'\n\ncreate table if not exists test (\n n int \n);\n\ninsert into test values (1);\ninsert into test values (2);\ninsert into test values (3);" }, { "path": "test-scripts/test2.sql", "content": "\npragma page_size = 4096;\n.load ./build/libmojo\n.open 'file:./testdbs/a_1.db?vfs=mojo&pagesz=4096&mode=ro&ver=2'\n\nselect count(*), max(id) from person;" }, { "path": "test-scripts/testdb.py", "content": "\"\"\"\" Tests for mojo filesystem \"\"\"\n\nimport os\nimport sys\nimport unittest\nimport sqlite3\nimport shutil\nimport commands as c\n\nMOJOKV_CLI=None\n\nclass TestConfig:\n '''Config for test db'''\n\n def __init__(self, page_sz=4096, journal_mode=\"WAL\", vac_mode=\"NONE\", use_tx=True):\n self.page_sz = page_sz\n self.journal_mode = journal_mode\n self.vac_mode = vac_mode\n self.use_tx = use_tx\n\n def __repr__(self):\n return f\"page_sz={self.page_sz} journal={self.journal_mode}\"\n\ndef rm_dir(path):\n '''Remove dir. Ignore file not found error'''\n try:\n if os.path.isdir(path):\n rm_fr(path)\n else:\n os.unlink(path)\n except FileNotFoundError:\n pass\n\ndef rm_fr(path):\n '''Equivalent of rm -fr'''\n\n journal_path = path + \"-journal\"\n wal_path = path + \"-wal\"\n\n rm_dir(journal_path)\n rm_dir(wal_path)\n\n if not os.path.exists(path):\n return\n if os.path.isfile(path) or os.path.islink(path):\n os.unlink(path)\n else:\n shutil.rmtree(path)\n\nclass MojoWritableTest(unittest.TestCase):\n '''MojoWritableTest'''\n\n def __init__(self, cfg, dbpath, *args, **kargs):\n self.cfg = cfg\n self.db_conn = None\n self.db_path = dbpath\n super(MojoWritableTest, self).__init__(*args, **kargs)\n\n def setUp(self):\n rm_fr(self.db_path)\n \n def tearDown(self):\n if self.db_conn:\n self.db_conn.close()\n\n def _subtest_name(self, name):\n return f\"{name}: {self.cfg} {self.db_path}\"\n\n def begin(self, cur):\n \"\"\"begin\"\"\"\n if self.cfg.use_tx:\n cur.execute(\"begin\")\n \n def commit(self, cur):\n \"\"\"commit\"\"\"\n if self.cfg.use_tx:\n cur.execute(\"commit\")\n\n def rollback(self, cur):\n \"\"\"rollback\"\"\"\n if self.cfg.use_tx:\n cur.execute(\"rollback\")\n\n def test_db_use(self):\n '''Tests the general usage of the database '''\n\n db_conn = c.opendb(self.cfg, self.db_path)\n ins_row_count = 100\n\n with self.subTest(self._subtest_name(\"create table\")):\n c.create_table_person(db_conn)\n\n with self.subTest(self._subtest_name(\"insert rows v1\")):\n c.insert_table_person(db_conn, ins_row_count)\n db_conn.commit()\n\n count = c.table_person_count(db_conn)\n self.assertEqual(ins_row_count, count)\n\n db_conn.close()\n\n ### Commit ver=1\n c.commit_version(self.db_path)\n db_conn = c.opendb(self.cfg, self.db_path)\n\n ### active ver=2\n with self.subTest(self._subtest_name(\"insert rows v2\")):\n self.assertEqual(ins_row_count, c.table_person_count(db_conn))\n c.insert_table_person(db_conn, ins_row_count)\n db_conn.commit()\n self.assertEqual(ins_row_count*2, c.table_person_count(db_conn))\n #db_conn.close()\n\n ### Commit ver=2\n db_conn.close()\n c.commit_version(self.db_path)\n db_conn = c.opendb(self.cfg, self.db_path)\n\n ### active ver=3\n with self.subTest(self._subtest_name(\"copy table\")):\n self.assertEqual(ins_row_count*2, c.table_person_count(db_conn))\n\n c.copy_table_person(db_conn, \"person_2\")\n db_conn.commit()\n\n self.assertEqual(ins_row_count*2, c.get_row_count(db_conn, \"person_2\"))\n\n with self.subTest(self._subtest_name(\"read v1\")):\n db_v1 = c.opendb(self.cfg, self.db_path, mode=\"ro\", ver=\"1\")\n self.assertEqual(ins_row_count, c.table_person_count(db_v1))\n db_v1.close()\n\n db_conn.close()\n\n ### Commit ver=3\n c.commit_version(self.db_path)\n db_conn = c.opendb(self.cfg, self.db_path)\n\n ### active ver=4\n with self.subTest(self._subtest_name(\"delete rows in v3\")):\n c.delete_table_person(db_conn, 0, 10000)\n db_conn.commit()\n self.assertEqual(0, c.table_person_count(db_conn))\n db_conn.close()\n\n ### Commit ver=4\n c.commit_version(self.db_path)\n db_conn = c.opendb(self.cfg, self.db_path)\n ### active ver=5\n with self.subTest(self._subtest_name(\"vacuum\")):\n c.vacuum(db_conn)\n db_conn.commit()\n self.assertEqual(0, c.table_person_count(db_conn))\n\n db_v3 = c.opendb(self.cfg, self.db_path, mode=\"ro\", ver=\"4\")\n self.assertEqual(0, c.table_person_count(db_v3))\n db_v3.close()\n\n db_v2 = c.opendb(self.cfg, self.db_path, mode=\"ro\", ver=\"2\")\n self.assertEqual(ins_row_count*2, c.table_person_count(db_v2))\n db_v2.close()\n\n db_conn.close()\n ### Commit ver=5\n c.commit_version(self.db_path)\n\ndef load_extension(mojo_lib):\n \"\"\"load_extension\"\"\"\n print(\"using libpath =\", mojo_lib)\n\n con = sqlite3.connect(\":memory:\")\n # enable extension loading\n con.enable_load_extension(True)\n con.execute(f\"select load_extension('{mojo_lib}')\")\n con.execute(\"pragma page_size=4096\")\n con.enable_load_extension(False)\n con.close()\n\ndef create_suite(full_mode):\n ''' Test suite for mojo '''\n\n suite = unittest.TestSuite()\n\n if not full_mode:\n page_sizes = [4096]\n journal_modes = [\"WAL\"]\n vacuum_modes = [\"INCREMENTAL\"]\n use_tx = [False]\n else:\n page_sizes = [4096]\n journal_modes = [\"OFF\", \"WAL\", \"MEMORY\", \"DELETE\", \"TRUNCATE\", \"PERSIST\"]\n vacuum_modes = [\"NONE\", \"FULL\", \"INCREMENTAL\"]\n use_tx = [False]\n\n\n dbid = 0\n for page_sz in page_sizes:\n for journal_mode in journal_modes:\n for vac_mode in vacuum_modes:\n for tx in use_tx:\n cfg = TestConfig(page_sz=page_sz,\n journal_mode=journal_mode,\n vac_mode=vac_mode,\n use_tx=tx)\n\n dbid += 1\n dbpath = f\"./testdbs/a_{dbid}.db\"\n suite.addTest(MojoWritableTest(cfg, dbpath, 'test_db_use'))\n\n return suite\n\nif __name__ == '__main__':\n if len(sys.argv[1:]) < 1:\n print(\"Error: missing extension library path\")\n sys.exit(1)\n\n if len(sys.argv[2:]) >= 1 and sys.argv[2] == \"full\":\n FULL = True\n else:\n FULL = False\n\n MOJOKV_CLI=os.getenv(\"MOJOKV_CLI\")\n if not MOJOKV_CLI:\n MOJOKV_CLI=\"./build/mojo-cli\"\n\n c.MOJOKV_CLI = MOJOKV_CLI\n\n ext_path = sys.argv[1]\n load_extension(ext_path)\n\n rm_fr(\"./testdbs/*\")\n c.mkdir(\"./testdbs\")\n\n runner = unittest.TextTestRunner(failfast=True)\n runner.run(create_suite(FULL))\n\n\n#conn = sqlite3.connect(\"file:a.db?vfs=mojo&ver=1&pagesz=4096\", uri=True)\n#conn = sqlite3.connect(\"a.db\")\n" } ]